Allow Manila shares to be directly attached to an instance when using libvirt¶

https://blueprints.launchpad.net/nova/+spec/libvirt-virtiofs-attach-manila-shares

Manila is the OpenStack Shared Filesystems service. This spec will outline API, database, compute and libvirt driver changes required in Nova to allow the shares provided by Manila to be associated with and attached to instances.

Problem description¶

At present users must manually connect to and mount shares provided by Manila within their instances. As a result operators need to ensure that Manila backend storage is routable from the guest subnets.

Use Cases¶

As an operator I want the Manila datapath to be separate to any tenant accessible networks.
As a user I want to attach Manila shares directly to my instance and have a simple interface with which to mount them within the instance.
As a user I want to detach a directly attached Manila share from my instance.
As a user I want to track the Manila shares attached to my instance.

Proposed change¶

This initial implementation will only provide support for attaching a share to and later detaching a share from an existing SHUTOFF instance. The ability to express attachments during the initial creation of an instance will not be covered by this spec.

Support for move operations once a share is attached will also not be covered by this spec, any requests to shelve, evacuate, resize, cold migrate or live migrate an instance with a share attached will be rejected with a HTTP409 response for the time being.

A new server shares API will be introduced under a new microversion. This will list current shares, show their details and allow a share to be attached or detached.

A new share_mapping database table and associated ShareMapping versioned objects will be introduced to capture details of the share attachment. A base ShareMapping versioned object will be provided from which virt driver and backend share specific objects can be derived providing specific share attach and detach implementations.

Note

One thing to note here is that no Manila state will be stored within Nova aside from export details used to initially attach the share. These details later being used when detaching the share. If the share is then reattached Nova will request fresh export details from Manila and store these in a new share attachment within Nova.

The libvirt driver will be extended to support the above with initial support for cold attach and detach. Future work will aim to add live attach and detach once support lands in libvirt itself.

This initial libvirt support will target the basic NFS and slightly more complex CephFS backends within Manila. Shares will be mapped through to the underlying libvirt domains using virtio-fs. This will require QEMU >=5.0 and libvirt >= 6.2 on the compute host and a kernel version of >= 5.4 within the instance guest OS.

Additionally this initial implementation will require that the associated instances use file backed memory or huge pages. This is a requirement of virtio-fs as the virtiofsd service uses the vhost-user protocol to communicate directly with the underlying guest. (ref: vhost-user documentation)

Two new compute capability traits and filters will be introduced to model an individual compute’s support for virtio-fs and file backed memory. And while associating a share to an instance, a check will ensure the host running the instance will support the

COMPUTE_STORAGE_VIRTIO_FS trait

and either the

COMPUTE_MEM_BACKING_FILE trait

that the instance is configured with hw:mem_page_size extra spec.

From an operator’s point of view, it means COMPUTE_STORAGE_VIRTIO_FS support requires that operators must upgrade all their compute nodes to the version supporting shares using virtiofs.

COMPUTE_MEM_BACKING_FILE support requires that operators configure one or more hosts with file backed memory. Ensuring the instance will land on one of these hosts can be achieved by creating an AZ englobing these hosts. And then instruct users to deploy their instances in this AZ. Alternatively, operators can guide the scheduler to choose a suitable host by adding trait:COMPUTE_MEM_BACKING_FILE=required as an extra spec or image property.

Users will be able to mount the attached shares using a mount tag, this is either the share UUID from Manila or a string provided by the users with their request to attach the share.

user@instance $ mount -t virtiofs $tag /mnt/mount/path

A previously discussed os-share library will not be created with this initial implementation but could be in the future if the logic required to mount and track shares on the underlying host is also required by other projects. For the time being existing code within the libvirt driver used to track filesystem host mounts used by volumes hosted on remoteFS based storage (such as NFS, SMB etc) will be reused as much as possible.

Share mapping status:

                     +----------------------------------------------------+   Reboot VM
    Start VM         |                                                    | --------------+
    Share mounted    |                       active                       |               |
+------------------> |                                                    | <-------------+
|                    +----------------------------------------------------+
|                      |                   |             |
|                      | Stop VM           |             |
|                      | Fail to umount    |             |
|                      v                   |             |
|                    +------------------+  |             |
|                    |      error       | <+-------------+-------------------+
|                    +------------------+  |             |                   |
|                      |                   |             |                   |
|                      | Detach share or   |             |                   |
|                      | delete VM         | Delete VM   |                   |
|                      v                   |             |                   |
|                    +------------------+  |             |                   |
|    +-------------> |        φ         | <+             |                   | Start VM
|    |               +------------------+                |                   | Fail to mount
|    |                 |                                 |                   |
|    | Detach share    |                                 | Stop VM           |
|    | or delete VM    | Attach share                    | Share unmounted   |
|    |                 v                                 v                   |
|    |               +----------------------------------------------------+  |
|    +-------------- |                      inactive                      | -+
|                    +----------------------------------------------------+
|                      |
+----------------------+

φ: means no entry in the database. No association between a share and a server.
Attach share: means POST /servers/{server_id}/shares
Detach share: means DELETE /servers/{server_id}/shares

This chart describe the share mapping status (nova), this is independent from the status of the Manila share.

Share attachment/detachment can only be done if the VM state is STOPPED or ERROR. These are operations only on the database, and no RPC calls will be required to the compute API. This is an intentional design for this spec. As a result, this could lead to situation where the VM start operation fails as an underlying share attach fails.

Mount operation will be done when the share is not mounted on the compute host. If a previous share would have been mounted on the compute host for another server, then it will attempt to mount it and a warning will be logged that the share was already mounted.

Umount operation will be really done when the share is mounted and not used anymore by another server.

With the above mount and umount operation, the state is stored in memory and do not require a lookup in the database.

The share will be mounted on the compute host using read/write mode. Read-only will not be supported as a share could not be mounted read-only and read/write at the same time. If the user wants to mount the share read-only, it will have to do it in the VM fstab.

Manila share removal issue:

Despite a share being used by instances, it can be removed by the user. As a result, the instances will lose access to the data and might cause difficulties in removing the missing share and fixing the instance. This is an identified issue that requires Manila modifications. A solution was identified with the Manila team to attach metadata to the share access-allow policy that will lock the share and prevent its deletion until the lock is not removed. If the above Manila change can land in the Zed cycle, the proposal here is to use the lock mechanism in Nova. Otherwise, clearly document the known issue as unsupported and warn users that they should take care and avoid this pitfall.

Instance metadata:

Add instace shares in the instance metadata. Extend DeviceMetadata with ShareMetadata object containing shareId and tag used to mount the virtiofs on an instance by the user. See Other end user impact.

Alternatives¶

The only alternative is to continue with the current situation where users must mount the shares within their instances manually. The downside being that these instances must have access to the storage network used by the Manila backends.

REST API impact¶

A new server level shares API will be introduced under a new microversion with the following methods:

GET /servers/{server_id}/shares

List all shares attached to an instance.

Return Code(s): 200,400,401,403,404

{
    "shares": [
        {
            "shareId": "48c16a1a-183f-4052-9dac-0e4fc1e498ad",
            "status": "active",
            "tag": "foo"
        },
        {
            "shareId": "e8debdc0-447a-4376-a10a-4cd9122d7986",
            "status": "active",
            "tag": "bar"
        }
    ]
}

GET /servers/{server_id}/shares/{shareId}

Show details of a specific share attached to an instance.

Return Code(s): 200,400,401,403,404

{
    "share": {
        "shareId": "e8debdc0-447a-4376-a10a-4cd9122d7986",
        "status": "active",
        "tag": "bar"
    }
}

PROJECT_ADMIN will be able to see details of the attachment id and export location stored within Nova:

{
    "share": {
        "attachmentId": "715335c1-7a00-4dfe-82df-9dc2a67bd8bf",
        "shareId": "e8debdc0-447a-4376-a10a-4cd9122d7986",
        "status": "active",
        "tag": "bar",
        "export_location": "server.com/nfs_mount,foo=bar"
    }
}

POST /servers/{server_id}/shares

Attach a share to an instance.

Prerequisite(s):

Instance much be in the SHUTOFF state.
Instance should have the required capabilities to enable virtiofs (see above).

This is a synchronous API. As a result, the VM share attachement state is defined in the database and set as inactive. Then, power on the VM will do the required operations to attach the share and set it as active if there are no errors.

Return Code(s): 202,400,401,403,404,409

Request body:

Note

tag will be an optional request parameter in the request body, when not provided it will be the shareId(UUID) as always provided in the request.

tag if povided by the user must be an ASCII string with a maximum lenght of 64 bytes.

{
    "share": {
        "shareId": "e8debdc0-447a-4376-a10a-4cd9122d7986"
    }
}

Response body:

{
    "share": {
        "shareId": "e8debdc0-447a-4376-a10a-4cd9122d7986",
        "status": "active",
        "tag": "e8debdc0-447a-4376-a10a-4cd9122d7986"
    }
}

DELETE /servers/{server_id}/shares/{shareId}

Detach a share from an instance.

Prerequisite(s): Instance much be in the SHUTOFF or ERROR state.

Return Code(s): 202,400,401,403,404,409

Data model impact¶

A new share_mapping database table will be introduced.

id - Primary key autoincrement
uuid - Unique UUID to identify the particular share attachment
instance_uuid - The UUID of the instance the share will be attached to
share_id - The UUID of the share in Manila
status - The status of the share attachment within Nova (active, inactive, error)
tag - The device tag to be used by users to mount the share within the instance.
export_location - The export location used to attach the share to the underlying host
share_proto - The Shared File Systems protocol (NFS, CEPHFS)

A new base ShareMapping versioned object will be introduced to encapsulate the above database entries and to be used as the parent class of specific virt driver implementations.

The database field status and share_proto values will not be enforced using enums allowing future changes and avoid database migrations. However, to make code more robust, enums will be defined on the object fields.

Fields containing text will use String and not Text type in the database schema to limit the column width and be stored inline in the database.

This base ShareMapping object will provide stub attach and detach methods that will need to be implemented by any child objects.

New ShareMappingLibvirt, ShareMappingLibvirtNFS and ShareMappingLibvirtCephFS objects will be introduced as part of the libvirt implementation.

Security impact¶

The export_location JSON blob returned by Manila and used to mount the share to the host and the host filesystem location should not be logged by Nova and only accessible by default through the API by admins.

Notifications impact¶

New notifications will be added:

One to add new notifications for share attach and share detach.
One to extend the instance update notification with the share mapping information.

Share mapping in the instance payload will be optional and controlled via the include_share_mapping notification configuration parameter. It will be disabled by default.

Proposed payload for attached and detached notification will be the same as the one returned by the show command with admin rights.

{
    "share": {
        "instance_uuid": "7754440a-1cb7-4d5b-b357-9b37151a4f2d",
        "attachmentId": "715335c1-7a00-4dfe-82df-9dc2a67bd8bf",
        "shareId": "e8debdc0-447a-4376-a10a-4cd9122d7986",
        "status": "active",
        "tag": "bar",
        "export_location": "server.com/nfs_mount,foo=bar"
    }
}

Proposed instance payload for instance updade, will be the list of share attached to this instance.

{
    "shares":
    [
        {
            "instance_uuid": "7754440a-1cb7-4d5b-b357-9b37151a4f2d",
            "attachmentId": "715335c1-7a00-4dfe-82df-9dc2a67bd8bf",
            "shareId": "e8debdc0-447a-4376-a10a-4cd9122d7986",
            "status": "active",
            "tag": "bar",
            "export_location": "server.com/nfs_mount,foo=bar"
        },
        {
            "instance_uuid": "7754440a-1cb7-4d5b-b357-9b37151a4f2d",
            "attachmentId": "715335c1-7a00-4dfe-82df-ffffffffffff",
            "shareId": "e8debdc0-447a-4376-a10a-4cd9122d7987",
            "status": "active",
            "tag": "baz",
            "export_location": "server2.com/nfs_mount,foo=bar"
        }
    ]
}

Other end user impact¶

Users will need to mount the shares within their guestOS using the returned tag.

Users could use the instance metadata to discover and auto mount the share.

Performance Impact¶

Through the use of vhost-user virtio-fs should have near local (mounted) file system performance within the guestOS. While there will be near local performance between the vm and host, the actual performance will be limited by the network performance of the network file share protocol and hardware.

Other deployer impact¶

None

Developer impact¶

None

Upgrade impact¶

A new compute service version and capability traits will be introduced to ensure both the compute service and underlying virt stack are new enough to support attaching a share via virtio-fs before the request is accepted.

Implementation¶

Assignee(s)¶

Primary assignee:: uggla (rene.ribaud)
Other contributors:: lyarwood (initial contributor)

Feature Liaison¶

Feature liaison:: uggla

Work Items¶

Add new capability traits within os-traits
Add support within the libvirt driver for cold attach and detach
Add new shares API and microversion

Dependencies¶

None

Testing¶

Functional libvirt driver and API tests
Integration Tempest tests

Documentation Impact¶

Extensive admin and user documentation will be provided.

References¶

History¶

Revisions¶
Release Name	Description
Yoga	Introduced
Zed	Reproposed

Allow Manila shares to be directly attached to an instance when using libvirt

Allow Manila shares to be directly attached to an instance when using libvirt¶

Problem description¶

Use Cases¶

Proposed change¶

Alternatives¶

REST API impact¶

Data model impact¶

Security impact¶

Notifications impact¶

Other end user impact¶

Performance Impact¶

Other deployer impact¶

Developer impact¶

Upgrade impact¶

Implementation¶

Assignee(s)¶

Feature Liaison¶

Work Items¶

Dependencies¶

Testing¶

Documentation Impact¶

References¶

History¶

Nova Specs

Page Contents