Allow Manila shares to be directly attached to an instance when using libvirt¶

https://blueprints.launchpad.net/nova/+spec/libvirt-virtiofs-attach-manila-shares

Manila is the OpenStack Shared Filesystems service. This spec will outline API, database, compute and libvirt driver changes required in Nova to allow the shares provided by Manila to be associated with and attached to instances.

Problem description¶

At present users must manually connect to and mount shares provided by Manila within their instances. As a result of this Manila has to expose details of the backend storage infrastructure to the user and allow the datapath to flow over tenant addressable networks into the users instances.

An alternative to this is for Nova to mount these shares on the underlying compute host and then pass through the provided filesystems to the instance, keeping the datapath outside of any tenant addressable network and without exposing details about the backend storage infrastructure.

Use Cases¶

As a user I want to attach Manila shares directly to my instance and have a simple interface with which to mount them within the instance.
As a user I want to detach a directly attached Manila share from my instance.
As a user I want to track the Manila shares attached to my instance.
As an operator I want the Manila datapath to be separate to any tenant accessible networks.
As an operator I don’t want to expose details of my storage infrastructure to users.

Proposed change¶

This initial implementation will only provide support for attaching a share to and later detaching a share from an existing SHUTOFF instance. The ability to express attachments during the initial creation of an instance will not be covered by this spec.

Support for move operations once a share is attached will also not be covered by this spec, any requests to shelve, evacuate, resize, cold migrate or live migrate an instance with a share attached will be rejected for the time being.

A new server shares API will be introduced under a new microversion. This will list current shares, show their details and allow a share to be attached or detached.

A new share_mapping database table and associated ShareMapping versioned objects will be introduced to capture details of the share attachment. A base ShareMapping versioned object will be provided from which virt driver and backend share specific objects can be derived providing specific share attach and detach implementations.

Note

One thing to note here is that no Manila state will be stored within Nova aside from export details used to initially attach the share. These details later being used when detaching the share. If the share is then reattached Nova will request fresh export details from Manila and store these in a new share attachment within Nova.

The libvirt driver will be extended to support the above with initial support for cold attach and detach. Future work will aim to add live attach and detach once support lands in libvirt itself.

This initial libvirt support will target the basic NFS and slightly more complex CephFS backends within Manila. Shares will be mapped through to the underlying libvirt domains using virtio-fs. This will require QEMU >=5.0 and libvirt >= 6.2 on the compute host and a kernel version of >= 5.4 within the instance guest OS.

Additionally this initial implementation will require that the associated instances use file backed memory or huge pages. This is a requirement of virtio-fs as the virtiofsd service uses the vhost-user protocol to communicate directly with the underlying guest.

Two new compute compatibility traits will be introduced to model an individual compute’s support for virtio-fs and file backed memory.

COMPUTE_STORAGE_VIRTIO_FS
COMPUTE_MEM_BACKING_FILE

These will then be used along with the flavor and image metadata associated with the instance to ensure the above requirements are met during a request to attach a share.

Future work will look into dropping this requirement and introducing a more robust way of allowing access to the underlying guest memory via flavor extra specs and image properties but that is left out of scope for this spec.

Users will be able to mount the attached shares using a mount tag, this is either the share UUID from Manila or a string provided by the user with their request to attach the share.

user@instance $ mount -t virtiofs $tag /mnt/mount/path

A previously discussed os-share library will not be created with this initial implementation but could be in the future if the logic required to mount and track shares on the underlying host is also required by other projects. For the time being existing code within the libvirt driver used to track filesystem host mounts used by volumes hosted on remoteFS based storage (such as NFS, SMB etc) will be reused as much as possible.

Alternatives¶

The only alternative is to continue with the current situation where users must mount the shares within their instances manually. The downside being that these instances must have access to the storage network used by the Manila backends.

REST API impact¶

A new server level shares API will be introduced under a new microversion with the following methods:

GET /servers/{server_id}/shares

List all shares attached to an instance.

Return Code(s): 200,400,401,403,404

{
    "shares": [
        {
            "shareId": "48c16a1a-183f-4052-9dac-0e4fc1e498ad",
            "status": "attached",
            "tag": "foo"
        },
        {
            "shareId": "e8debdc0-447a-4376-a10a-4cd9122d7986",
            "status": "attached",
            "tag": "bar"
        }
    ]
}

GET /servers/{server_id}/shares/{shareId}

Show details of a specific share attached to an instance.

Return Code(s): 200,400,401,403,404

{
    "share": {
        "shareId": "e8debdc0-447a-4376-a10a-4cd9122d7986",
        "status": "attached",
        "tag": "bar"
    }
}

Admins will be able to see details of the attachment id and export location stored within Nova:

{
    "share": {
        "attachmentId": "715335c1-7a00-4dfe-82df-9dc2a67bd8bf",
        "shareId": "e8debdc0-447a-4376-a10a-4cd9122d7986",
        "status": "attached",
        "tag": "bar",
        "export_location": "server.com/nfs_mount,foo=bar"
    }
}

POST /servers/{server_id}/shares

Attach a share to an instance.

Prerequisite(s): Instance much be in the SHUTOFF state.

This is an asynchronous API, callers will need to poll one of the above GET methods to determine when the attachment is complete.

Return Code(s): 202,400,401,403,404,409

Request body:

Note

tag will be an optional request parameter, when not provided it will be the shareId as always provided in the request.

{
    "share": {
        "shareId": "e8debdc0-447a-4376-a10a-4cd9122d7986",
    }
}

Response body:

{
    "share": {
        "shareId": "e8debdc0-447a-4376-a10a-4cd9122d7986",
        "status": "attaching",
        "tag": "e8debdc0-447a-4376-a10a-4cd9122d7986",
    }
}

DELETE /servers/{server_id}/shares/{shareId}

Detach a share from an instance.

Prerequisite(s): Instance much be in the SHUTOFF state.

Return Code(s): 202,400,401,403,404,409

Data model impact¶

A new share_mapping database table will be introduced.

id - Unique UUID to identify the particular share attachment
instance_uuid - The UUID of the instance the share will be attached to
share_id - The UUID of the share in Manila
status - The status of the share attachment within Nova. - detached - attaching - attached - detaching - error
tag - The device tag to be used by users to mount the share within
the instance
export_location - The export location used to attach the share to the
underlying host.

A new base ShareMapping versioned object will be introduced to encapsulate the above database entries and to be used as the parent class of specific virt driver implementations.

This base ShareMapping object will provide stub attach and detach methods that will need to be implemented by any child objects.

New ShareMappingLibvirt, ShareMappingLibvirtNFS and ShareMappingLibvirtCephFS objects will be introduced as part of the libvirt implementation.

Security impact¶

This should improve the security model of using Manila by removing the datapath from tenant networks while also removing the need for users to know details of the underlying storage environment.

The export_location JSON blob returned by Manila and used to attach the share to the underlying host should not be logged by Nova and only accessible by default through the API by admins.

Notifications impact¶

New notifications will be added to represent the attaching and detaching of a share to an instance.

Other end user impact¶

Users will need to mount the shares within their guestOS using the returned tag.

Performance Impact¶

Through the use of vhost-user virtio-fs should have near local (mounted) file system performance within the guestOS.

Other deployer impact¶

None

Developer impact¶

None

Upgrade impact¶

A new compute service version and compatibility traits will be introduced to ensure both the compute service and underlying virt stack are new enough to support attaching a share via virtio-fs before the request is accepted.

Implementation¶

Assignee(s)¶

Primary assignee:: lyarwood

Other contributors:

Feature Liaison¶

Feature liaison:: lyarwood

Work Items¶

Add new compatibility traits within os-traits
Add support within the libvirt driver for cold attach and detach
Add new shares API and microversion

Dependencies¶

None

Testing¶

Functional libvirt driver and API tests
Integration Tempest tests

Documentation Impact¶

Extensive admin and user documentation will be provided.

References¶

History¶

Revisions¶
Release Name	Description
Yoga	Introduced

Allow Manila shares to be directly attached to an instance when using libvirt

Allow Manila shares to be directly attached to an instance when using libvirt¶

Problem description¶

Use Cases¶

Proposed change¶

Alternatives¶

REST API impact¶

Data model impact¶

Security impact¶

Notifications impact¶

Other end user impact¶

Performance Impact¶

Other deployer impact¶

Developer impact¶

Upgrade impact¶

Implementation¶

Assignee(s)¶

Feature Liaison¶

Work Items¶

Dependencies¶

Testing¶

Documentation Impact¶

References¶

History¶

Nova Specs

Page Contents