manila-specs

Example Spec - The title of your blueprint

Tue, 17 Feb 2026 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/manila/+spec/example

Introduction paragraph – why are we doing anything? A single paragraph of prose that operators can understand.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/manila/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox, or see: https://www.siafoo.net/reST.xml
If your specification proposes any changes to the Manila REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query: https://review.openstack.org/#/q/status:open+project:openstack/manila-specs+message:apiimpact,n,z
If you would like to provide a diagram with your spec, text representations are preferred. acsiiflow [1] is a good tool to assist with this. Text representation is preferred because the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
Diagram examples

asciiflow:

+----------+     +-----------+        +----------+
| A        |     |  B        |        |  C       |
|          +-----+           +--------+          |
+----------+     +-----------+        +----------+

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you’re clear about the actors in each use case: Developer, end user, deployer, etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

What are some alternative ways to fix the problem being solved? What are the reasons for using the proposed method as opposed to some of the alternatives? Why aren’t we using those? This doesn’t require an overly detailed write up, instead just demonstrate that thought has been put into the proposed solution and some info on why it’s an appropriate choice.

Data model impact

Changes which require modifications to the data model often have a wider impact on the system. The community often has strong opinions on how the data model should be evolved, from both a functional and performance perspective. It is therefore important to capture and gain agreement as early as possible on any proposed changes to the data model.

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema defintion do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Example JSON schema definitions can be found in the Manila tree http://git.openstack.org/cgit/openstack/manila/tree/manila/api/schemas/v1.1

Note that the schema should be defined as restrictively as possible. Parameters which are required should be marked as such and only under exceptional circumstances should additional parameters which are not defined in the schema be permitted (eg additionaProperties should be False).

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Driver impact

Describe any changes to the syntax or behaviour of driver calls. This includes but is not limited to changes to the ShareDriver class, as well as changes to ShareManager that invoke drivers differently or have different expectations of return values.

What allowance is made for gradual deprecation of removed/changed interfaces?
Is new documentation needed for driver authors?
Will drivers’ tests need updating?

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

For more detailed guidance, please see the OpenStack Security Guidelines as a reference (https://wiki.openstack.org/wiki/Security/Guidelines). These guidelines are a work in progress and are designed to help you identify security best practices. For further information, feel free to reach out to the OpenStack Security Group at openstack-security@lists.openstack.org.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-manilaclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but when considering large scale deployments the proposed call may in fact be performed on hundreds of nodes.
Scheduler filters get called once per host for every volume being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries can have a profound impact on performance, especially in critical sections of code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other volume drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that targets (LVM) are stored in, how do we handle any used directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the volumes in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

New dependencies in development environments
Changes to typical developer workflows
Changes that will require updates to CI
Changes to shared code that will require updates to other areas of code outside this feature.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Work items or tasks – break the feature up into the things that need to be done to implement it. Those parts might end up being done by different people, but we’re mostly trying to understand the timeline for implementation.

Dependencies

Include specific references to specs and/or blueprints in Manila, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Manila (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Please discuss how the change will be tested. We especially want to know what tempest tests will be added. It is assumed that unit test coverage will be added so that doesn’t need to be mentioned explicitly, but discussion of why you think unit tests are sufficient and we don’t need to add more tempest tests would need to be included.

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

What is the impact on the docs team of this change? Some changes might require donating resources to the docs team to have the documentation updated. Don’t repeat details discussed above, but please reference them here.

References

Please add any useful references here. You are not required to have any reference. Moreover, this specification should still make sense when your references are unavailable. Examples of what you could include are:

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. link to any vendor documentation)
Anything else you feel it is worthwhile to refer to

Diagram Tool References

[1] http://asciiflow.com
[2] http://blockdiag.com

Spec Lite: Add human readable export location to share manage

Thu, 27 Nov 2025 00:00:00

problem:

Currently, share supports human readable export locations by mount_point_name field. But, there is no way to determine the export location before the share is managed, and human readable export locations are not available for managed share. With the current logic, the managed share is newly mounted only with the default template, such as volume_name in NetApp driver.

solution:

We could introduce a new optional field for managing shares where users would be able to specify a custom export location just like in the share creation. It should be easier for them to memorize the export location of the share, in case there is need to mount the share again. If this option is not provided, the managed share will be mounted with the default template as the existing workflow. And if this field is specified during the manage share request, the backends that support such functionality would be able to create multiple export locations for a managed share with that value, and users would be able to mount the shares either using the human readable export location, or other if they exist. It’s possible backends that implement this feature may only be able to provide one export path. The new field will be called mount_point_name. This field will not accept special characters other than underscores or hyphens. If special characters other than underscores or hyphens are provided, the manila api service is going to raise HTTP BadRequest, warning that such characters are not allowed. When a request to manage a share is received, if a mount_point_name was provided, share type’s provisioning:mount_point_prefix spec or default_mount_point_prefix configuration will be prepended to it. The number of characters provided by the user added to the number of characters in the prefix can not exceed 255 characters, otherwise the request will be denied. So the manila share service will look up for duplicate mount_point_name values and if it finds any, share manage will fail. It is possible that users coincidentally set the same mount_point_name while managing a share. In such cases, the share will not be managed and its status will be set to manage_error. As same as share creation, administrators will need to create share types with mount_point_name_support extra spec to True. As the manila share manage API will perform validations using this extra spec, it must be always tenant-visible. By having this extra-spec, the scheduler can also filter out backends that do not support such functionality.

impacts:

REST API Impact.
- A microversion bump to the share manage API.
- The API will accept the field mount_point_name as an option.
Documentation Impact
- API Reference
Database Impact
- None
Python-manilaclient and OSClient impact
- The share adopt command will be modified to accept the --mount-point-name parameter as an option.

This implementation is not supposed to impact performance on any aspect.

alternative:

As an alternative, drivers could reuse the name and the description of shares and generate human readable export locations, but names can be duplicated and backends may fail due to that.

timeline:

Include in Gazpacho release.

link:

https://blueprints.launchpad.net/manila/+spec/manage-with-mount-point-name

assignee:

eunkyung(ek121.kim@samsung.com)

Manila qos types

Wed, 01 Oct 2025 00:00:00

Blueprint: https://blueprints.launchpad.net/manila/+spec/qos-types

This blueprint proposes adding new object representing quality of service(QoS). The QoS type object will be having specs representing attributes needed to define quality of service.

Problem description

Manila backend drivers support Quality Of Service (QoS) feature which allows operator to define certain quality parameters on share. The parameters of QoS are passed in share-type extra-specs as driver-specific parameters. Operators must define extra-specs to describe QoS, which creates duplication, lack of reuse, and poor visibility across share types. Backend drivers support DHSS=True and DHSS=False. The way of defining QoS by using share-type extra-specs works for deployments where policy and resource management occur outside of the Openstack environment. However for Manila integrated management, where resources and policies defined during run-time, share-type extra-specs is not the efficient way to define QoS on Manila resources due to reasons mentioned earlier. This specification proposes an approach that enables Manila to create multiple qos_types and optionally mention one of them in share type extra-spec. The qos_type represents a QoS policy which will be applied on resource e.g. share.

Use Cases

Operators want to define reusable QoS profiles (e.g. “Gold-QoS”, “Silver-QoS”) and convert them on QoS policies that will be applied on various Manila resources.
Tenants need visibility into the QoS characteristics of a share type before provisioning a share.
Drivers can implement enforcement of QoS policies by consuming QoS key/value pairs used to define QoS type.

Proposed change

Introduce a new top-level object: qos_type.

qos_type contains an id, name, description, and a set of qos_specs (key/value pairs). The name must be unique.
A qos_type name can be specified in share type extra-spec named as default_qos_type.
qos_type lifecycle management (create, update, delete) will be strictly allowed to admin by the virtue of default RBAC policies. All non-admin users can only list them.

qos_type APIs

qos_type create/delete/update/show/list APIs.
qos_type_specs APIs

qos_type_specs create/delete/update/show/list APIs.
Modify share create API

In order to allow users selecting the desired QoS policy by the virtue of qos_type, the qos_type name must be defined in share-type extra-spec default_qos_type. When share is created from such share-type, the qos_type and its specs are picked and passed to backend driver. The scheduler must validate that backend driver reporting capability qos_type_support will be filtered.
Changes in backend driver

The QoS type and its specs are provided to storage backend driver during share and share-server creation. Backend driver will create the qos policy if its not present and apply to resources.

Alternatives

Continue using extra_specs: leads to duplication and inconsistency. The user can check QoS characteristics of a share type before provisioning a share, but its limited to single QoS policy for resource like share.
Driver-only enforcement: lacks visibility at API and DB layers, inconsistent across backends.

Data model impact

New field in shares table

Field

Type

Null

Key

Default

Extra

qos_type_id

string(36)

YES

NULL
New table qos_types

Field

Type

Null

Key

Default

Extra

id

string(36)

No

PRI

name

string(255)

No

UNIQUE

NULL

description

string(255)

YES

NULL

deleted

string(36)

YES

NULL
New table qos_specs

Field

Type

Null

Key

Default

Extra

id

int(11)

No

PRI

qos_type_id

string(36)

No

NULL

key

string(255)

No

NULL

value

string(1023)

No

NULL

deleted

int(11)

YES

NULL

Field	Type	Null	Key	Default	Extra
qos_type_id	string(36)	YES		NULL

Field	Type	Null	Key	Default
id	string(36)	No	PRI
name	string(255)	No	UNIQUE	NULL
description	string(255)	YES		NULL
deleted	string(36)	YES		NULL

Field	Type	Null	Key	Default
id	int(11)	No	PRI
qos_type_id	string(36)	No		NULL
key	string(255)	No		NULL
value	string(1023)	No		NULL
deleted	int(11)	YES		NULL

CLI API impact

New OpenStackClient(OSC) commands:

openstack share qos type create <name> [--description <desc>]
                                [--spec <key=value>]

openstack share qos type list

openstack share qos type show <qos_type_id>

openstack share qos type set <qos_type_id> [--description <desc>]
                             [--spec <key=value>]

openstack share qos type delete <qos_type_id>

REST API impact

** Create QoS Type**:

POST /v2/qos-types

The create request should not be processed if its non-admin user and the API will respond with 403 Forbidden

Request:

{
    "qos_type": {
        "name": "Gold-QoS",
        "description": "High-performance storage QoS",
        "specs": {
            "expected_iops": "500",
            "peak_iops": "5000",
            "peak_iops_allocation": "used-space",
        }
    }
}

Response(201):

{
    "qos_type": {
        "name": "Gold-QoS",
        "id": "b8b3d83f-96a2-45f2-9e39-ccf3ecf2ff33",
        "description": "High-performance storage QoS",
        "specs": {
            "expected_iops": "500",
            "peak_iops": "5000",
            "peak_iops_allocation": "used-space",
        },
        "created_at": "2025-09-04T10:00:00Z",
        "updated_at": None,
    }
}

** Show QoS Type**:

GET /v2/qos-types/{qos_type_id}

Response(200):

{
    "qos_type": {
        "name": "Gold-QoS",
        "id": "b8b3d83f-96a2-45f2-9e39-ccf3ecf2ff33",
        "description": "High-performance storage QoS",
        "specs": {
            "expected_iops": "500",
            "peak_iops": "5000",
            "peak_iops_allocation": "used-space",
        },
        "created_at": "2025-09-04T10:00:00Z",
        "updated_at": "2025-09-04T10:00:00Z",
    }
}

** List QoS Types.**:

GET /v2/qos-types?{queries}

In detail and index APIs, queries will allow filtering with exact and inexact (“name”, “name~” etc.) attributes.

Response(200):

{
    "qos_types": [
        {
            "name": "Gold-QoS",
            "id": "b8b3d83f-96a2-45f2-9e39-ccf3ecf2ff33",
            "description": "High-performance storage QoS",
            "specs": {
                 "expected_iops": "500",
                 "peak_iops": "5000",
                 "peak_iops_allocation": "used-space",
            },
            "created_at": "2025-09-04T10:00:00Z",
            "updated_at": "2025-09-04T10:00:00Z",
        },
        {
            "name": "Silver-QoS",
            "id": "c8b3d83f-96a2-45f2-9e39-ccf3ecf2ff33",
            "description": "Mid-performance storage QoS",
            "specs": {
                 "expected_iops": "300",
                 "peak_iops": "3000",
                 "peak_iops_allocation": "used-space",
            },
            "created_at": "2025-09-04T11:00:00Z",
            "updated_at": "2025-09-04T11:00:00Z",
        }
    ]
}

** Update QoS Types.**:

PUT /v2/qos-types/{qos_type_id}

Request:

{
    "qos_type": {
        "description": "High-performance storage QoS - Update",
    }
}

The update request should not be processed if its non-admin user and the API will respond with 403 Forbidden

Response(200):

{
    "qos_type": {
        "name": "Gold-QoS",
        "id": "b8b3d83f-96a2-45f2-9e39-ccf3ecf2ff33",
        "description": "High-performance storage QoS - Update",
        "specs": {
            "expected_iops": "500",
            "peak_iops": "5000",
            "peak_iops_allocation": "used-space",
        },
        "created_at": "2025-09-04T10:00:00Z",
        "updated_at": "2025-09-04T11:00:00Z",
    }
}

** Delete QoS Type.**:

DELETE /v2/qos-types/{qos_type_id}

The delete request should not be processed if its non-admin user and the API will respond with 403 Forbidden. Also if qos_type is in use by shares, the API will respond with 400 BadRequest

Response(204):

None

** Create QoS Type Spec**:

POST /v2/qos-types/{qos_type_id}/specs

The create request should not be processed if its non-admin user and the API will respond with 403 Forbidden

Request:

{
    "specs": {
        "expected_iops": "500",
        "peak_iops": "5000",
        "peak_iops_allocation": "used-space",
    }
}

Response(202):

{
    "specs": {
        "expected_iops": "500",
        "peak_iops": "5000",
        "peak_iops_allocation": "used-space",
    }
}

** List QoS Type Specs**:

GET /v2/qos-types/{qos_type_id}/specs

Response(200):

{
    "specs": {
        "expected_iops": "500",
        "peak_iops": "5000",
        "peak_iops_allocation": "used-space",
    }
}

** Show QoS Type Spec**:

GET /v2/qos-types/{qos_type_id}/specs/{key}

Response(200):

{
    "expected_iops": "500",
}

** Update QoS Type Spec.**:

PUT /v2/qos-types/{qos_type_id}/specs

Request:

{
    "expected_iops": "600",
}

The update request should not be processed if its non-admin user and the API will respond with 403 Forbidden

Response(200):

{
    "expected_iops": "600",
}

** Delete QoS Type Spec.**:

DELETE /v2/qos-types/{qos_type_id}/specs/{key}

The delete request should not be processed if its non-admin user and the API will respond with 403 Forbidden.

Response(204):

None

Driver impact

The backend driver needs to implement:

Driver support qos_type will report qos_type_support capability.
The storage backed driver will receive qos_type along-with its specs during creation of resources like share or share-server. The driver then decide how to use those to create and apply QoS policies on resources.
Modifying QoS type specifications after shares are created results in indeterminate behavior and must be avoided. Modifications are not enforced on the storage back end. This means that some drivers can choose to apply the modifications only for new shares and leave the existing shares as is, and some others may alter the behavior for existing shares as well as for new shares. However, this modification is triggered only when a new share is created.

Security impact

The create/update/delete operations for QoS types and specs are available only for admin users by the virtue of default RBAC. All users can only list and use QoS type and its specs.

Notifications impact

“qos_type.create”, “qos_type.update” and “qos_type.delete” notification events will be emitted for the respective actions.

Other end user impact

None

Performance Impact

Minimal overhead when listing QoSTypes.

Other deployer impact

When cloud is upgraded to release that supports this feature and administrator wants to enable this feature, he needs to follow below steps.

Create one or more qos_types
Create valid specs for qos_type
For share-type extra-spec, add extra-spec default_qos_type and mention the qos_type name as value. The share created from such share type will consider qos_type assuming driver supports capability qos_type_support.

Developer impact

Implement feature for Manila-core as well as reference driver.

Implementation

Assignee(s)

Primary assignee:

kpdev(kinpaa@gmail.com)

Work Items

Add DB migrations and ORM models.
Implement DB API functions for CRUD and associations.
Implement REST API controllers for qos_type and qos_type_specs.
Add OSC plugin commands.
Horizon UI updates.
Tempest and unit test coverage.
Documentation and release notes.

Future Work Items

It is possible to support multiple qos_types under single share-type by associating them with share-type. If such use-case arises, current spec can be enhanced.

Dependencies

None

Testing

Unit tests
Tempest tests

Documentation Impact

Docstrings
Devref
User guide
Admin guide
Release notes

References

None

Remove v1 API

Thu, 04 Sep 2025 00:00:00

https://blueprints.launchpad.net/manila/+spec/remove-v1

Remove the long-deprecated v1 API.

Problem description

The v1 API has been deprecated pretty much the entirety of Manila’s time as an independent project. We do not wish to focus on it when adding JSON Schema schemas (as part of the OpenAPI effort) and its continued presence makes that effort harder than is necessary. The removal of this API is long-overdue.

Use Cases

As a developer, I don’t want to have to worry about multiple APIs.

Proposed change

Remove the API, merging any shared code into the v2 API.

Alternatives

None.

Data model impact

None.

REST API impact

Requests to the /v1 API will be rejected.

Driver impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None. Most client tooling already used v2 API.

Performance Impact

None.

Other deployer impact

Deployers will need to delete any endpoint for the v1 API upon upgrade.

Developer impact

None. The API code will be simplified.

Implementation

Assignee(s)

Primary assignee:: stephen.finucane

Work Items

Move or copy any common code from manila.api.v1 to manila.api.v2
Remove tests for v1 API
Delete the v1 API
Update documentation

Dependencies

None.

Testing

We will remove tests for the v1 API. v2 API tests will remain unchanged.

Documentation Impact

References to the v1 API will be removed. A release note will be added.

References

None.

Scenario tests design

Mon, 30 Jun 2025 00:00:00

https://blueprints.launchpad.net/manila/+spec/scenario-tests

Manila has good functional test coverage of its features, but narrow coverage of scenarios. And with addition of new features this coverage becomes even less.

Problem description

Drivers often lack features or implement them incorrectly and it’s very difficult for humans to notice this in code review. Tests ensure that drivers which lack features must explicitly skip tests to get a passing result, and the non-skipped tests ensure that the drivers implement features correctly. Documentation for how features should work is often sparse and driver authors don’t always know what is the correct behavior when implementing a driver feature. Tests allow driver authors to code the feature to pass the test, which simplifies the driver author’s job.

Use Cases

Consider the case when new manila deployment becomes ready. And one wants to verify that main user use cases work in general. Having automatic scenario tests one could test his deployment much faster than manually, as it is now. Also, it could be used in CI systems to test share drivers continuously.

Proposed change

It is proposed to get agreement on scenario tests design (this spec) and implement them in manila plugin for tempest. After that these tests could be used in CI systems as well as on customer deployments. This spec covers only existing features in manila as of last available (Newton) release. All newly added features should be covered separately.

Prerequisites for scenarios:

Depending on share driver mode, it can be required to create share-network with or without security-services.
Depending on protocol, its versions and access type, “mount” operations should be defined explicitly. Scenario tests assume it as predefined and known. Hence, scenarios do not include difference between access type (IP, User, Cert). Due to this, scenario tests should be data driven by “access_type”, “access_proto”, “access_level” and “mount command with all expected options”.
Bold texts depend on specific implementations and can be unsupported by share backends.
Text in brackets depend on share driver configuration and may be optional.
All user machines are separate VMs from share-nodes. They should have network connectivity with share host. User VMs should be built with image that has shared file systems clients.
Share host should have open ports for SSH protocol and protocols of shared file systems.

Here is list of scenarios for implementation:

1) Share access and file operations (Partially implemented)
2) Share access with multiple guests (Partially implemented)
3) Relationships between source shares and child shares (Implemented)
4) Create/extend share and write data (Implemented)
5) Create/shrink share and write data (Implemented)
6) Create/manage share and write data (Implemented)
7) Create/manage share and snapshot and write data (Partially implemented)
8) Replicate ‘writable’ share and write data (Not yet implemented)
9) Replicate and promote ‘readable’ share and write data (Not yet implemented)
10) Replicate and promote ‘dr’ share and write data (Not yet implemented)
11) Get a snapshot of a replicated share (Not yet implemented)
12) Migrate share and write data (Implemented)
13) Mount share through VirtioFS (Not yet implemented)
14) Mount multiple shares through VirtioFS (Not yet implemented)

1) Share access and file operations

Driver mode: any

Involved APIs:

[create share network]
[create share type]
create share
allow share access
deny share access
delete share

Note

Implementation Status

This test case extends manila_tempest_tests.tests.scenario .test_share_basic_ops.ShareBasicOpsBase#test_write_with_ro_access

**Scenario 1 steps**
Step	Action	Result
1	Create user VM (UVM)	ok, created
2	Create share (S)	ok, created
3	SSH to UVM	ok, connected
4	Try mount S to UVM	fail, access denied
5	Provide RO access to S	ok, provided
6	Try mount S to UVM	ok, mounted
7	Try create files on S	fail, access denied
8	Unmount S from UVM	ok, unmounted
9	Remove RO access from S	ok, removed
10	Try mount S to UVM	fail, access denied
11	Provide RW access to S	ok, provided
12	Try mount S to UVM	ok, mounted
13	Try write files to S	ok, written
14	Try read files from S	ok, read
15	Try delete files on S	ok, deleted
16	Unmount S from UVM	ok, unmounted
17	Delete S	ok, deleted
18	Try mount S	fail, not found
19	Delete UVM	ok, deleted

2) Share access with multiple guests

Driver mode: any

Involved APIs:

[create share network]
[create share type]
create share
allow share access
delete share

Note

Implementation Status

This test case extends manila_tempest_tests.tests.scenario .test_share_basic_ops.ShareBasicOpsBase#test_read_write_two_vms

**Scenario 2 steps**
Step	Action	Result
1	Create UVM1	ok, created
2	Create UVM2	ok, created
3	Create share S	ok, created
4	Add RW access to UVM1	ok, added
5	SSH to UVM1	ok, connected
6	Try mount S from UVM1	ok, mounted
7	SSH to UVM2	ok, connected
8	Try mount S from UVM2	fail, access denied
9	Add RW access for UVM2	ok, added
10	Try mount S from UVM2	ok, two VMs have it mounted at once.
11	Create test file in mounted share from UVM1	ok, created. Available from UVM2
12	Write data to test file from UVM2	ok, written. Available from UVM1 too
13	Unmount S on UVM1	ok, unmounted
14	Unmount S on UVM2	ok, unmounted
15	Delete UVM1	ok, deleted
16	Delete UVM2	ok, deleted
17	Delete S	ok, deleted

3) Relationships between source shares and child shares

Driver mode: any

Involved APIs:

[create share network]
[create share type]
create share
allow share access
create share snapshot
delete share snapshot
delete share

Note

Implementation Status

This test case has been implemented as manila_tempest_tests.tests .scenario.test_share_basic_ops .ShareBasicOpsBase#test_write_data_to_share_created_from_snapshot

**Scenario 3 steps**
Step	Action	Result
1	Create UVM	ok, created
2	Create share S1	ok, created
3	Provide RW access to S1	ok, provided
4	SSH to UVM	ok, connected
5	Try mount S1 to UVM	ok, mounted
6	Create “file1”	ok, created
7	Create snapshot SS1 from S1	ok, created
8	Create “file2” in share S1	ok, created. We expect that snapshot will not contain any data created after snapshot creation.
9	Create share S2 from SS1	ok, created
10	Try mount S2	fail, access denied. We test that child share did not get access rules from parent share.
11	Provide RW access to S2	ok, provided
12	Try mount S2	ok, mounted
13	List files on S2	only “file1” exists
14	Create file3 on S2	ok, file created
15	List files on S1	two files exist - “file1” and “file2”
16	List files on S2	two files exist - “file1” and “file3”
17	Unmount S1 and S2	ok, unmounted
18	Delete S2, then SS1, then S1, then UVM	ok, all deleted

4) Create/extend share and write data

Driver mode: any

Involved APIs:

[create share network]
[create share type]
create share
allow share access
extend share
delete share

Note

Implementation Status

This test case has been implemented as manila_tempest_tests.tests .scenario.test_share_extend.ShareExtendBase#test_create_extend_and_write

**Scenario 4 steps**
Step	Action	Result
1	Create UVM	ok, created
2	Create share S1 of 1Gb size	ok, created
3	Provide RW access to S1	ok, provided
4	SSH to UVM	ok, connected
5	Try mount S1 to UVM	ok, mounted
6	Create “file1”	ok, created
7	Fill file1 with data as possible	size of a file does not exceed share size quota
8	Extend share S1 to 2Gb	ok, extended
9	Write additional data to file1	data written, size of a file does not exceed new share size quota and it is more than old one
10	Unmount S1	ok, unmounted
11	Delete share S1	ok, deleted
12	Delete UVM	ok, deleted

5) Create/shrink share and write data

Driver mode: any

Involved APIs:

[create share network]
[create share type]
create share
allow share access
shrink share
delete share

Note

Implementation Status

This test case has been implemented as manila_tempest_tests.tests .scenario.test_share_shrink.ShareShrinkBase#test_create_shrink_and_write

**Scenario 5 steps**
Step	Action	Result
1	Create UVM	ok, created
2	Create share S1 of 2Gb size	ok, created
3	Provide RW access to S1	ok, provided
4	SSH to UVM	ok, connected
5	Try mount S1 to UVM	ok, mounted
6	Write some data for 2 Gb	ok, created
7	Fill file1 with data as possible	size of a file does not exceed share size quota
8	Try shrink share S1 to 1Gb	fail, possible data loss exception
9	Delete data for amount of 1 Gb	data deleted
10	Shrink share S1 to 1Gb	ok, shrinked
11	Try write data more than new size of 1 Gb	fail, cannot write
12	Unmount S1	ok, unmounted
13	Delete share S1	ok, deleted
14	Delete UVM	ok, deleted

6) Create/manage share and write data

Driver mode: any

Involved APIs:

[create share network]
[create share type]
create share
allow share access
manage share
unmanage share
manage share again
delete share

Note

Implementation Status

This test case has been partially implemented as manila_tempest_tests .tests.scenario.test_share_manage_unmanage .ShareManageUnmanageBase#test_create_manage_and_write . It currently tests only DHSS=False back end share drivers. To complete the implementation, this test case needs to support DHSS=True mode of share drivers. Support for managing shares with DHSS=True was added to Manila via API version 2.49. So this test must create a share network if [share]/multitenancy_enabled=True and the API version being tested is >= 2.49, and manage the share into the specific share network.

**Scenario 6 steps**
Step	Action	Result
1	Create UVM	ok, created
2	Create share S1 of 1Gb size	ok, created
3	Provide RW access to S1	ok, provided
4	SSH to UVM	ok, connected
5	Try mount S1 to UVM	ok, mounted
6	Write some data	ok, written
7	Unmount S1	ok, unmounted
8	Unmanage share	ok, unmanaged
9	Try get share S1	fail, 404 code in response
10	Manage share S1	ok, managed.
11	Provide RW access to S1 again	ok, provided. We make sure that even if rule has existed on backend, we do not fail if explicitly try add it again after ‘manage’ operation.
12	Try mount S1 to UVM	ok, mounted. Previously created data still there.
13	Unmount S1	ok, unmounted
14	Delete share	ok, deleted
15	Try manage share again	fail, resource not found
16	Delete UVM	ok, deleted

7) Create/manage share and snapshot and write data

Driver mode: any

Involved APIs:

[create share network]
[create share type]
create share
allow share access
create snapshot
manage share
unmanage share
manage snapshot
unmanage snapshot
delete snapshot
delete share

Note

Implementation Status

This test case is yet to be implemented.

**Scenario 7 steps**
Step	Action	Result
1	Create UVM	ok, created
2	Create share S1 of 1Gb size	ok, created
3	Provide RW access to S1	ok, provided
4	SSH to UVM	ok, connected
5	Try mount S1 to UVM	ok, mounted
6	Write some data	ok, written
7	Create snapshot SS1	ok, created
8	Unmanage snapshot SS1	ok, unmanaged
9	Unmanage share S1	ok, unmanaged
10	Try get share S1	fail, 404 code in response
11	Manage share S1	ok, managed.
12	Provide RW access to S1 again	ok, provided. We make sure that even if rule has existed on backend, we do not fail if explicitly try add it again after ‘manage’ operation.
13	Try mount S1 to UVM	ok, mounted. Previously created data still there.
14	Manage snapshot SS1	ok, managed
15	Delete snapshot SS1	ok, deleted
16	Unmount S1	ok, unmounted
17	Delete share S1	ok, deleted
18	Try manage share S1 again as S2, this should fail asynchronously since the resource is gone	S2 has a status set to ‘error’
19	Delete S2	ok, deleted
20	Delete UVM	ok, deleted

8) Replicate ‘writable’ share and write data

Driver mode: any

Involved APIs:

[create share network]
[create share network subnets in different availability zones]
[create share type]
create share
allow share access
create replica
delete replica
delete share

Note

Implementation Status

This test case is yet to be implemented.

**Scenario 8 steps**
Step	Action	Result
1	Create UVM1	ok, created
2	Create share S1-R1 of 1Gb size	ok, created
3	Provide RW access to S1-R1	ok, provided
4	SSH to UVM1	ok, connected
5	Try mount S1-R1 to UVM1	ok, mounted
6	Create file1	ok, created
7	Create share replica S1-R2	ok, created
8	Create UVM2	ok, created
9	SSH to UVM2	ok, connected
10	Try mount S1-R2 to UVM2	fail, access denied
11	Try mount S1-R2 to UVM1	ok,mounted. Same files exist.
12	Provide RW access to S1-R2	ok, provided
13	Try mount S1-R2 to UVM2	ok, mounted
14	Create file2 in S1-R2	ok, created. S1-R1 has both files too.
15	Create file3 in S1-R1	ok, created. Both replicas have three created files.
16	Unmount both replicas	ok, unmounted
17	Delete original replica S1-R1	ok, deleted. second and the only replica now still exists and has all files that were created.
18	Delete share S1	ok, deleted

9) Replicate and promote ‘readable’ share and write data

Driver mode: any

Involved APIs:

[create share network]
[create share network subnets in different availability zones]
[create share type]
create share
allow share access
create replica
promote replica
delete replica
delete share

Note

Implementation Status

This test case is yet to be implemented.

**Scenario 9 steps**
Step	Action	Result
1	Create UVM1	ok, created
2	Create share S1-R1 of 1Gb size	ok, created
3	Provide RW access to S1-R1	ok, provided
4	SSH to UVM1	ok, connected
5	Try mount S1-R1 to UVM1	ok, mounted
6	Create file1	ok, created
7	Create share replica S1-R2	ok, created
8	Create UVM2	ok, created
9	SSH to UVM2	ok, connected
10	Try mount S1-R2 to UVM2	fail, access denied
11	Try mount S1-R2 to UVM1	ok, mounted. Same files exist.
12	Provide RW access to S1-R2	ok, provided
13	Try mount S1-R2 to UVM2	ok, mounted
14	Try create some file in S1-R2	fail, filesystem is RO only.
15	Create file2 in S1-R1	ok, created. Both replicas have two created files.
16	Promote S1-R2 to active	ok, promoted. S1-R1 became RO.
17	Create file3 in S1-R2	ok, created. S1-R1 has all files too.
18	Try create some file in S1-R1	fail, filesystem is RO
19	Unmount both replicas	ok, unmounted
20	Delete original (now RO) replica S1-R1	ok, deleted. Second and the only replica (active) now still exists and has all files that were created.
21	Delete share S1	ok, deleted

10) Replicate and promote ‘dr’ share and write data

Driver mode: any

Involved APIs:

[create share network]
[create share network subnets in different availability zones]
[create share type]
create share
allow share access
create replica
promote replica
delete replica
delete share

Note

Implementation Status

This test case is yet to be implemented.

**Scenario 10 steps**
Step	Action	Result
1	Create UVM	ok, created
2	Create share S1-R1	ok, created
3	Provide RW access to S1-R1	ok, provided
4	SSH to UVM	ok, connected
5	Try mount S1-R1 to UVM	ok, mounted
6	Create file1	ok, created
7	Create share replica S1-R2	ok, created
8	Unmount S1-R1	ok, unmounted
9	Promote S1-R2	ok, promoted. S1-R1 became ‘dr’-only
10	Try mount S1-R2 to UVM	ok, mounted. ‘file1’ exists
11	Create file2	ok, created
12	Unmount S1-R2	ok, unmounted
13	Promote S1-R1	ok, promoted. S1-R2 became ‘dr’-only.
14	Try mount S1-R1 to UVM	ok, mounted. Files ‘file1’ and ‘file2’ exist.
15	Unmount S1-R1	ok, unmounted
16	Delete S1-R2 (current secondary)	ok, deleted.
17	Delete share	ok, deleted

11) Get a snapshot of a replicated share

Driver mode: any

Involved APIs:

[create share network]
[create share network subnets in different availability zones]
[create share type]
create share
allow share access
create snapshot
create share from snapshot
create replica
promote replica
delete replica
delete snapshot
delete share

Note

Implementation Status

This test case is yet to be implemented.

**Scenario 11 steps**
Step	Action	Result
1	Create UVM	ok, created
2	Create share S1-R1	ok, created
3	Provide RW access to S1-R1	ok, provided
4	SSH to UVM	ok, connected
5	Try mount S1-R1 to UVM	ok, mounted
6	Create ‘file1’	ok, created
7	Create snapshot SS1	ok, created
8	Create replica S1-R2	ok, created
9	Create ‘file2’	ok, created
10	Create snapshot SS2	ok, created
11	Unmount S1-R1	ok, unmounted
12	Promote S1-R2 (For non-’writable’ replication types)	ok, promoted
13	Try mount S1-R2 to UVM	ok, mounted
15	Delete S1-R1	ok, deleted
16	Create share S2 from SS2	ok, created
17	Provide RW access to S2	ok, provided
18	SSH to UVM	ok, connected
19	Try mount S2 to UVM	ok, mounted. All created files exist
20	Unmount S2	ok, unmounted
21	Delete S2, SS2, SS1, S1	ok, deleted

12) Migrate share and write data

Driver mode: any

Involved APIs:

[create share network]
[create share type]
create share
allow share access
migration-start share
migration-complete share
delete share

Note

Implementation Status

This test case has been implemented as manila_tempest_tests.tests .scenario.test_share_basic_ops.ShareBasicOpsBase#test_migration_files

**Scenario 12 steps**
Step	Action	Result
1	Create UVM	ok, created
2	Create share S1 of 1Gb size	ok, created
3	Provide RW access to S1	ok, provided
4	SSH to UVM	ok, connected
5	Try mount S1 to UVM	ok, mounted
6	Create file1	ok, created
7	Unmount share S1	ok, unmounted
8	Do “migration-start”	ok, finished. 1 phase is completed.
9	Do “migration-complete”	ok, share instance only one - new one. it has previously created file1.
10	Try mount S1 to UVM	ok, mounted. Created file1 exists
11	Unmount share S1	ok, unmounted
12	Delete share S1	ok, deleted
13	Delete UVM	ok, deleted

13) Mount share through VirtioFS

Driver mode: any

Involved APIs:

[create share network]
[create share type]
create share
create VM
share attachment
resource locks
delete share

Note

Implementation Status

This test case is yet to be implemented.

**Scenario 13 steps**
Step	Action	Result
1	Create user VM (UVM)	ok, created
2	Create share (S)	ok, created
3	SSH to UVM	ok, connected
4	Try mount S to UVM	fail, access denied
5	Shut UVM off	ok, shut off
6	Attach S to UVM	ok, attached
7	Turn UVM on	ok, active
8	List UVM’s attachments	ok, share listed as an attachment
9	List S resource locks	ok, locks in place
10	List access rule resource locks	ok, locks in place
11	Show access rule access_to added to S using (non-admin user)	fail, access rule visibility is locked
12	SSH to UVM	ok, connected
13	Try mount S to UVM	ok, mounted
14	Try write files to S	ok, written
15	Try read files from S	ok, read
16	Try delete files on S	ok, deleted
17	Try to delete S using (non-admin user)	fail, share is locked
18	Attempt deleting access rule granted to S with non-admin user	fail, access rule deletion is locked
19	Attempt deleting access rule granted to S admin user without unrestricting	fail, access rule deletion is locked
20	Shut UVM off	ok, shut off
21	Detach S from UVM	ok, detached
22	Turn UVM on	ok, active
23	List S resource locks	ok, lock deleted
24	List access rule resource locks	ok, lock deleted
25	Delete S	ok, deleted
26	Delete UVM	ok, deleted

14) Mount multiple shares through VirtioFS

Driver mode: any

Involved APIs:

[create share network]
[create share type]
create share
create VM
share attachment
delete share

Note

Implementation Status

This test case is yet to be implemented.

**Scenario 14 steps**
Step	Action	Result
1	Create user VM (UVM1)	ok, created
2	Create user VM (UVM2)	ok, created
3	Create share (S)	ok, created
4	Shut UVM1 off	ok, shut off
5	Attach S to UVM1	ok, attached
6	Turn UVM1 on	ok, active
7	List UVM1’s attachments	ok, share listed as an attachment
8	SSH to UVM1	ok, connected
9	Try mount S to UVM1	ok, mount succeeded
10	Try write files to S	ok, written
11	SSH to UVM2	ok, connected
12	Try mount S	fail, access denied
13	Shut UVM2 off	ok, shut off
14	Attach S to UVM2	ok, attached
15	Turn UVM2 on	ok, active
16	List UVM2’s attachments	ok, share listed as an attachment
17	SSH to UVM2	ok, connected
18	Try write files to S on UVM2	ok, written
19	Shut UVM1 off	ok, shut off
20	Detach share from UVM1	ok, detached
21	Turn UVM1 on	ok, active
22	Try write files to S on UVM2	ok, written
23	Shut UVM2 off	ok, shut off
24	Detach share from UVM2	ok, detached
25	Turn UVM2 on	ok, active
26	Delete S	ok, deleted
27	Delete UVM1	ok, deleted
28	Delete UVM2	ok, deleted

Alternatives

Alternative is what we have now. It is requirement to test each share driver manually and dependency on presence of detailed docs for each feature share drivers implement.

Data model impact

None

REST API impact

None

Driver impact

None

Security impact

None

Notifications impact

None

Other end user impact

End users will be able to run scenario tests against their manila deployment to test workability of various features.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Original assignee:

vponomaryov

Other contributors:

We’re inviting more contributors to continue to improve scenario tests. Adding new scenario test cases to manila-tempest-plugin does not require adding the test case description to this spec, but it is encouraged if you like feedback for your new test cases.

Work Items

Implement designed scenario tests in manila plugin for tempest.

Dependencies

None

Testing

It is expected that all first-party drivers as well as third-party drivers will be covered in CI systems with designed here scenario tests. Due to big amount of optional features that are covered by scenario tests, only appropriate scenario tests for specific back-end should run in CI systems. Scenarios that include only required features (1-4) are a must for running in CI systems for each share driver.

Documentation Impact

Doc describing usage of manila plugin for tempest [1] should be extended with configuration and usage details of scenario tests.

References

[1] http://docs.openstack.org/developer/manila/devref/tempest_tests.html

Manila - Share backup out of place restore

Tue, 25 Feb 2025 00:00:00

Blueprint: https://blueprints.launchpad.net/manila/+spec/out-of-place-restore

The previously introduced share backup spec and API allows for the creation and restoration of shares through manila, however at the moment, this API is limited to only performing restores to the source share of the given backup. This spec proposes to enhance the share backup API to allow restores to non source share targets.

Problem description

At the moment, it is impossible to restore a given share backup to a location other then the source share used to create the backup. Allowing users to restore to a different share specified by share-id or name would extend the usability of Manila share backups.

Use Cases

As far as business continuity and disaster recovery (BCDR) is concerned, there are a few reasons that make such a feature beneficial:

If a user is limited to performing restores to the source share, they must ensure that the sum of the current shares size and the restores content does not exceed the size / quota of the given share. This might stop users from being able to perform restores. Additionally users may be encouraged to “browse by restore” destructively removing existing share content to see if a backup has valid content they are interested in.

Likewise, if the storage backend that backs the source share type suddenly becomes unavailable, users will not be able to utilise their backup.

Allowing restores to a separate share gives an opportunity for the resumption of a clients service by restoring to a different share backend to that of the source.

This would bring Manila share backup in-line with what is offered by Cinder, and would deliver on the Future Work item detailed “Add support to create share from backup” in the original specification, in an alternative manner.

Proposed change

Extend python-manilaclient ‘share backup restore’ command

The share backup restore command will be extended to allow a optional argument, –target-share to be added by a end user, in the form of a valid share_id or share_name. This will restore a given backup to the targeted share.
Extend python-manilaclient ‘share create’ command

The share create command will be extended to allow a optional argument, –backup-id, to be added by a end user in the form of a valid backup id. this will create a new share and restore the content of the given backup to it.
update API call for targeted share restore
The share backup API call for restores will be extended to allow the appending of a target share to the body of a restore call as part of its information.
- The component side wsgi action for restores will be modifed to consider this new attachment to the body, to check that the supplied share exists in the scope of the clients context, and to pass it to the RPC call component.
- Finally, where relevant, backup drivers that wish to make use of targeted restores should be updated with additional sanity checks such that a difference between the share_protocol of a given backup and target share are handled appropriately. In the case of support, the backup should proceed as expected, whereas if a given matchup of share_protocols isn’t supported, a failure should occur explicitly and be communicated.
backup_driver abstract class update

To ensure compatibility, between different backup drivers, a new flag boolean will be introduced to the backup_driver abstract class: self.support_restore_to_target, in this way a backup driver can signal to the data manager if it is capable of supporting out of place restores to its supported storage backend.
Manila Data-manager update

The Manila data-manager will be modified to allow it to receive a target_share value as part of a restore call. This doesn’t take the form of a new argument, rather the data manager will check to see if a share can be considered a targeted restore by evaluating the backup source shares id agains’t the id of the supplied share.

If these two elements match, a normal restore can be considered to be underway, if they do not, a further check should be performed against the current backup drivers boolean flag to ensure the operation is supported.

If it is the target shares status will be updated to BACKUP_RESTORING, and passed in to the backup driver in place of the original share for a restore.

finally, given these checks or the restore operation succeeds or fails the data_manager will appropriately update the state of the share in contention.
NFSBackupDriver update

The default NFSBackupDriver contained within the data manager will have to be extended to support out of place restores, or otherwise have the support boolean for targeted restores set False so as to disallow such requests.

Alternatives

We could accept that out of place restores are not a valid use case for Manila and cease pursuit of the feature.

Data model impact

None

CLI API impact

expand the restore and create commands in the python manila client OSC plugin:

openstack share backup restore [<backup> –target-share <target-share>]

backup: ID of backup to restore.
target_share: optional Share to target for restore, where value can be A
share name or ID. Default to None, e.g. restore to source)

openstack share create [<share arguments> –backup-id <backup-id>]

share arguments: mandatory parameters expected when creating a share, e.g.
share_protocol, size, etc.
backup-id: optional Optional backup ID to create the share from.

REST API impact

Normal API Restores will still be available as before using:

POST /v2/share-backups/{backup_id}/action

{
    "restore": null
}

while if a targeted restore is desired, the ID can be appended as info:

POST /v2/share-backups/{backup_id}/action

{
    "restore": <target_share_id>
}

In the case the target_share or backup is not known to manila, the API will respond with 404 Not Found as with a normal share.

In the case the target_share or backup is known but the tenant user has no permissions, the API will respond with 403 Unauthorised

The API will respond 202 if request is accepted.

Driver impact

The backup driver will have to be modified to include the new support boolean, eg:

class BackupDriver(object):

    def __init__(self):
        super(BackupDriver, self).__init__()

        # This flag indicates if backup driver implement backup, restore,
        # delete, and get progress operations by its own or uses the data
        # manager.
        self.use_data_manager = True

        # This flag indicates if the backup driver supports out of place
        # restores to a share other then the source of a given backup.
        self.support_restore_to_target = False

Security impact

As with the original share backup specification the data node would have access to read the source and target share data. If the deny access phase fails, the node will continue forever with access to the user’s data.

Notifications impact

None

Other end user impact

If the End user targets an existing share other then the source, it will become unavailable in regards to other share operations during the restore. e.g. no extend/shrink share, replication share, share-group operation, migration share.

Performance Impact

None

Other deployer impact

The deployer will be able to restore a share to a non source target.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

zgoggin(zachary.goggin@cern.ch)

Work Items

Update restore command in python-manilaclient.
Update restore API WSGI handling
Update data manager and backup_driver to support targeted restores
Update tempest support.
Update manila-ui support.
Update manila backup in devstack plugin to support.

Future Work Items

None

Dependencies

None

Testing

Unit tests
Tempest tests

Documentation Impact

Docstrings
Devref
User guide
Admin guide
Release notes

References

None

Manila share encryption

Thu, 30 Jan 2025 00:00:00

Blueprint: https://blueprints.launchpad.net/manila/+spec/share-encryption

Encrypting OpenStack Manila shares is crucial for ensuring the security and confidentiality of users’ data. There are broadly two levels of encryption: “front-end” (data in-transit) and “back-end” (data at-rest). Currently, users can request back-end data encryption via share types that have custom extra-specs. These custom-extra specs direct the back end driver to encrypt the share data at rest, however, there is no mechanism for the user to control much else regarding the encryption process. Ideally, users must be allowed to create and manage their own encryption keys. This specification proposes an approach that enables Manila to coordinate user defined encryption keys for “back-end” (at rest) encryption of share data.

Problem description

While manila users can create encrypted shares with some storage back ends, they cannot create or control their encryption keys via OpenStack Manila. Encryption keys are made up by the storage back end or the Manila driver, and any one with access to the keys could access the data if they gain access to the back end storage. So this spec is addressing is user control of encryption keys.

Here are some reasons why you should consider encrypting Manila shares:

Data Confidentiality: Encryption protects the confidentiality of your data by converting it into unreadable ciphertext. If unauthorized users gain access to the storage, they won’t be able to make sense of the encrypted data without the appropriate decryption key.
Compliance Requirements: Many industries and regulatory standards require the encryption of sensitive data. Encrypting OpenStack Manila share helps you comply with data protection regulations and industry standards, ensuring that your organization meets legal requirements.
Protection Against Unauthorized Access: Encrypting shares adds an extra layer of security against unauthorized access. Even if someone gains access to the underlying storage, they won’t be able to access the data without the encryption key.
Mitigation of Insider Threats: Encryption can help mitigate the risk of insider threats. Even if an authorized user with access to the storage attempts to misuse the data, encryption prevents them from reading or tampering with sensitive information without the proper decryption key.
Protection Against Data Breaches: In the event of a security breach or data leak, encrypted data is much more difficult for attackers to exploit. This can significantly reduce the impact of a data breach, as the stolen information remains unreadable without the encryption key.
Risk Management: Encryption is a fundamental component of a comprehensive risk management strategy. By implementing encryption for Manila shares, you enhance your overall security posture and reduce the potential impact of security incidents.

Use Cases

Users would like to protect the data within shares. This proposal of share encryption provides a manila level support to trigger the data encryption on backend and thus protect the data from attacker.

Users can provide Manila with a reference to an encryption key. This encryption key can be applied by share drivers to share servers that they create, or individually to a specific share. Such keys are stored in through the OpenStack key management service, Barbican. The encryption key ref will be provided in share create request. After key ref is being validated by manila, it will be given to storage back end driver alongside information on how to access the key manager. The key data will be used to encrypt the share’s data within the storage back end.

The actual encryption of the data at-rest is performed by the back end storage system. The scope of Manila’s involvement ends with coordinating the user’s secret with the Key-store.

Proposed change

Modify share create API

In order to support ‘bring your own key’ use case, manila share create API will accept optional parameters i.e. encryption-key-ref.
Share type extra-spec changes

A tenant-visible extra-spec on share types called “encryption_support” will be introduced. It can have a string value. Value can be “share-server” or “share”. When set to “share”, it means that the user can expect encryption keys to apply per-share. Value “share_server” would mean that the user can expect encryption keys to apply at the share server level. Default will be None.
Manila API and Share service changes

Manila api service will allow configuration of a key manager. We will introduce an interface for the Manila API service to communicate with an external key manager interface (OpenStack Castellan), which internally works with a key store (Openstack Barbican). The key manager will be configured via conf file. When encryption key ref is provided, and share type has “encryption_support” set to either “share” or “share_server”, Manila will validate the encryption key ref with barbican and store the encryption key ref in the share instance along with the type of encryption supported. Encryption key ref will be a valid barbican secret_ref. Storage back end devices would need to obtain the key out-of-band in order to perform the encryption. Manila will collate the necessary information that allows a storage back end to identify and retrieve the secret from the key manager. Manila will interact with Barbican as a service. It will use service credentials to retrieve secrets from Barbican on behalf of OpenStack users. Back end storage devices would also need OpenStack credentials to work with Barbican. We will improve Manila’s share manager service to create OpenStack Identity Service (Keystone) application credentials to facilitate this interaction, and hand these to the storage back end device via storage back end drivers. The application credentials will be restricted and so allowed to only fetch key-manager secret for service user. The credentials can not list secrets, or retrieve unrelated secrets even if they belong to the same tenant. To restrict share server creation with different encryption key refs new project level quota called ‘server_encryption_keys’ will be introduced. API service will check the quota limit in case share type extra-spec is ‘share-server’. Once quota limit is reached, API will throw error.
Changes in Manila Scheduler

CapabilitiesFilter will be updated to consider share backend drivers capability encryption_support.
Changes in backend driver

A backend driver will report encryption_support as a capability. It can have any of these values: [“share”], [“share_server”], [“share”, “share_server”] or None. When a share creation request arrives, if DHSS=True, drivers will be asked to provide a compatible share server and the call will include the share’s encryption-key-ref along with all the other network details. If a share server exists with the appropriate key ref and satisfying the network parameters, the driver can present that server. If a compatible share server doesn’t exist, the driver returns [] and the share manager will ask the driver to create a new server. The key ref will be stored in share servers db model. The encryption key ref along-with application credentails will be provided to backend hardware to perform the encryption. The back end storage system will fetch the key directly, out of band of manila using the ref that Manila shares via the back end driver. The fetched key data will be used to encrypt the share data.

Alternatives

If OpenStack Manila doesn’t provide a way for users to manage their own encryption keys, the cloud may need an out-of-band solution, such as:

External or third party key management services that support integration with OpenStack Manila.
Client-Side Encryption: forego data encryption at-rest. Users must encrypt their data locally on their clients before storing it in Manila shares.
File-Level Encryption: encrypting individual files or directories within the clients using tools or libraries instead of encrypting the share data as a whole.
Custom Scripts or Tools: Deployment-local scripts that enable users to manage their encryption keys outside of OpenStack Manila. This may involve creating a user interface or command-line tool that interacts with OpenStack Manila and external key management systems.
OpenStack Manila Extensions: unofficial API extensions that can enhance the functionality of Manila to deal with encryption metadata.

In all, these alternatives are inferior to the convenience that we would provide by implementing the proposal in this specification.

Data model impact

New field in share_servers table

Field

Type

Null

Key

Default

Extra

encryption_key_ref

string(36)

YES

NULL

app_cred_id

string(36)

YES

NULL
New field in share_instances table

Field

Type

Null

Key

Default

Extra

encryption_key_ref

string(36)

YES

NULL

Field	Type	Null	Key	Default	Extra
encryption_key_ref	string(36)	YES		NULL
app_cred_id	string(36)	YES		NULL

Field	Type	Null	Key	Default	Extra
encryption_key_ref	string(36)	YES		NULL

CLI API impact

Add new parameters to commands in openstackclient(OSC):

openstack share create [--encryption-key-ref <key-ref>]
                       <share_protocol> <size>

encryption-key-ref: Valid Barbican (i.e. key-manager) secret ref (UUID) represents share or share server encryption key reference

REST API impact

** To Create share with user defined share encryption key.**:

POST /v2/shares

Request:

{
    "share": {
        "description": "My custom share London",
        "share_type": null,
        "share_proto": "nfs",
        "share_network_id": "713df749-aac0-4a54-af52-10f6c991e80c",
        "share_group_id": null,
        "name": "share_London",
        "snapshot_id": null,
        "is_public": true,
        "size": 1,
        "metadata": {
        },
        "scheduler_hints": {
        },
        "encryption_key_ref": "b7460a86-30ea-4c20-901f-6cee1e945286",
    }
}

The encryption_key_ref should be valid Barbican secret href or UUID, otherwise the API will respond with 400 Not Found

Response(202 Accepted):

{
     "share": {
         "id": "011d21e2-fbc3-4e4a-9993-9ea223f73264",
         "size": 1,
         "description": "My custom share London",
         "status": "creating",
         "progress": null,
         "share_server_id": null,
         "project_id": "16e1ab15c35a457e9c2b2aa189f544e1",
         "name": "share_London",
         "share_type": "25747776-08e5-494f-ab40-a64b9d20d8f7",
         "share_type_name": "default",
         "availability_zone": null,
         "created_at": "2025-02-18T10:25:24.533287",
         "export_location": null,
         "links": [
             {
                 "href": "http://172.18.198.54:8786/v1/16e1ab15c35a457e9c2b2aa189f544e1/shares/011d21e2-fbc3-4e4a-9993-9ea223f73264",
                 "rel": "self"
             },
         ],
         "share_network_id": "713df749-aac0-4a54-af52-10f6c991e80c",
         "share_group_id": null,
         "export_locations": [],
         "share_proto": "NFS",
         "host": null,
         "access_rules_status": "active",
         "has_replicas": false,
         "replication_type": null,
         "task_state": null,
         "snapshot_support": true,
         "volume_type": "default",
         "snapshot_id": null,
         "is_public": true,
         "metadata": {
         },
         "encryption_key_ref": "b7460a86-30ea-4c20-901f-6cee1e945286"
     }
}

Driver impact

The backend driver needs to implement:

The driver needs to report “encryption_support” as a capability. If encryption is supported, the value of this capability can be reported as a list of capabilities with valid keys among “share” and “share_server”.
Return valid list of share servers to Manila-share Manager based on requested parameters. This is supported today, however function needs to be updated to consider additional requirement i.e. encryption key ref. When an encryption key reference is provided by Manila’s share manager, care must be taken to specifically eliminate share servers that don’t match that key.
Instruct the back end storage system to encrypt the share with key data sent from key-store e.g. Barbican

Security impact

After encryption, the shares will be more secure and data is protected from attacker.

Notifications impact

None

Other end user impact

User need to store encryption key/payload with Barbican key-manager and get encryption key ref.

Performance Impact

The share can be encrypted at front-end or back-end. But for Manila we intend to support only back-end encryption and so very less performance penalty in manila services.

Other deployer impact

None.

Developer impact

Implementation

Assignee(s)

Primary assignee:

kpdev(kinpaa@gmail.com)

Work Items

Implement ‘bring you own key’ in share create APIs.
Update ‘openstack share create’ command in python-manilaclient.
Implement tempest support.
Document about create share with encryption

Future Work Items

None

Dependencies

None

Testing

Unit tests
Tempest tests

Documentation Impact

Docstrings
Devref
User guide
Admin guide
Release notes

References

None

Allow force deletion of used subnets

Thu, 17 Oct 2024 00:00:00

Allow deletion of a subnet of a share network used by one or multiple shares. This action is possible only if subnet is not the last subnet of the share network, as it is forbidden to have a network without any subnet.

Problem description

When a share network is attached to a share, manila API allows adding a new subnet to this network, but not deleting it. This means Manila users can add as many network interfaces as they want to their share servers, but these interfaces could never be deleted.

Use Cases

Requirements of clients using a share might change that requires a completely different network configuration of the share server. For example, it might be used from a public network, and then from a private network. In this case, as public IP addresses are limited resources, we want them to be released. Keeping unused network interfaces on a share server may also lead to security issues. Therefore, Manila should provide a way to delete such interfaces or subnets.

Proposed change

Don’t delete share server on share network subnet deletion. A share network might have multiple subnets. So when deleting one of its subnets, the share will still need a share server.
Allow removing network interface on share server. Currently in Manila, there is an API to add network interfaces, but no API allows deleting network interfaces.
Add a delete_share_network_subnet method to share API, RPC API and manager. In the manager, this method relies on update_share_server_network_allocations with remove=True. After the allocation update, if no allocation remains, the share server is deleted.
Add delete parameter to method update_share_server_network_allocations of share manager and driver (that defaults to False). When set to True, allocations are removed from the share server instead of added.

REST API impact

Add a new action endpoint to force subnet deletion that would be admin-only by virtue of default RBAC rules.

URL:

/v2/share-networks/{share_network_id}/subnets/{sns_id}/action

Method: POST
JSON body:
{
  'force_delete': None
}
This API is restricted to administrators only by virtue of default RBAC rules.

When deleting a subnet through this action, check for existence of attached shares is not performed.

Possible response codes: - 202 - Request accepted. - 403 - Forbidden (non-admin access) - 404 - Unrecognized share network ID - 404 - Unrecognized share network subnet ID - 404 - API does not exist in requested microversion - 409 - Conflict (e.g., trying to delete the last subnet)

Introduce a “check-only” mode when attempting to delete a subnet from a share network. This mode would allow users to evaluate whether a subnet can be safely removed from all associated share servers, without actually performing the deletion. This endpoint is also admin-only by virtue of default RBAC rules.

To keep consistency with subnet creation check, this endpoint accept a reset_operation parameter. If set to false API returns cached result. If set to true, it triggers a new asynchronous check operation.

URL:

/v2/share-networks/{share_network_id}/subnets/{sns_id}/action

Method: POST
JSON body:
{
  'share_network_subnet_delete_check': {
    'reset_operation': false
  }
}
This API is restricted to administrators only by virtue of default RBAC rules.

Possible response codes: - 202 - Request accepted. - 403 - Forbidden (non-admin access) - 404 - Unrecognized share network ID - 404 - Unrecognized share network subnet ID - 404 - API does not exist in requested microversion - 409 - Conflict (e.g., trying to delete the last subnet)

End-user impact

None. Default behaviour remains the same. By specifying the force parameter, users indicate that they are aware of consequences and take responsibility for eventual disruption. This API is only available to administrators.

Implementation

Assignee(s)

Primary assignee:

sylvanld (sylvan.le-deunff@ovhcloud.com)

Work Items

Implement the force-delete share subnet API.
Implement the corresponding functionality in SDK.
Implement tempest tests.
Update the documentation.

Testing

Unit tests
Tempest tests

Documentation Impact

User guide
Admin guide
API reference

Spec Lite: Pass resource metadata updates to backend drivers

Sun, 21 Apr 2024 00:00:00

problem:

Currently, Manila allows a user to perform add/update/delete operations on resource metadata. This includes resources such as share, snapshot and share network subnet etc. However these operations only handle changes in db assuming resource metadata can only be used for tagging purposes. So if a customer wants to perform any backend driver operation on a resource based on resource metadata, Manila needs to introduce a new API.

solution:

This effort of adding new APIs for resources can be eliminated by extending existing metadata add/update/delete operations to handle db operations as well as pass metadata updates to backend driver. For resources like snapshot and share network subnet, the updated metadata will be passed to backend driver. For resource like share, in addition to passing updated metadata to backend driver, few changes will be introduced to decide which metadata it can pass to backend driver.

Flow - 1. New driver interface update_share_from_metadata will be added. New config option driver_updatable_metadata will be introduced in default section of manila.conf. This contains comma separated list where each element is of format <driver>:<driver_updatateble_key>. Please note, this contains both admin and non-admin metadata keys. 2. The share type extra-specs will determine the initial configuration of a share. It will contain values in the format of <driver>:<driver_updatateble_key>. During share create API, Manila will copy the share type extra-specs to share metadata. Only keys present in config option driver_updatable_metadata are copied along-with their values in share metadata. 3. If metadata is provided in share create API, possible actions are as below -

If provided metadata is present in admin_only_metadata, Manila will check if policy allows the user to add/update that metadata and if yes, it will check if its present in share metadata (i.e. copied from share type extra-spec). If present in share metadata, db operations are performed and Manila will pass metadata to backend driver. If not present, only db operations are performed.

If provided metadata is not admin-only, the Manila will check if its present in share metadata (i.e. copied from share type extra-spec). If present in share metadata, db operations are performed and Manila will pass metadata to backend driver.

4. During metadata update API as well, first validations and then db operations are performed. After this updated metadata will be passed to backend driver. 5. Finally backend driver will peform some action based on passed metadata. If action results in success or failure, the user will be notified through the user message API. It might be possible that a db change gets applied successfully, but respective backend driver operation results in failure. Thus notifying the end user using message API becomes important.

Use cases -: 1. For example, NetApp ONTAP driver needs to set snapshot policy for a given share. Instead of creating a new API to set/unset snapshot policy, user can add snapshot policy as share metadata i.e. key=value pair. The metadata is then passed to ONTAP driver by manila-api service using rpc calls. The ONTAP driver then sets the snapshot policy for this share and notifies a success/failure. 2. Also, not all share specific operations are allowed for regular user. By using admin_only_metadata along-with driver updatable metadata, admin can control the share behaviour. For example, NetApp ONTAP cloud admin does not want end user to see root path (/) of a share due to security concerns. This is possible by setting the show-mount option of share server NFS. In this case as well, instead of creating new API to set/unset show-mount option, admin can add show-mount option as metadata of share resource. The ONTAP driver then sets the show-mount option on the share server belonging to share.

impacts:

REST API Impact.
- The update metadata behavior will issue an RPC call to the backends when necessary to allow the share manager to process metadata updates via the driver. However, no request/response schema or response error codes will be changed as part of this implementation.
Documentation Impact
- User guide
- Admin guide
Database Impact
- None
Python-manilaclient and OSClient impact
- None

alternative:

As an alternative, Manila could create separate APIs for handling various resource related operations.

timeline:

Include in 2024.2 release.

link:

https://blueprints.launchpad.net/manila/+spec/pass-resource-metadata-updates-to-backend-drivers

assignee:

kpdev

Add new ensure shares API

Thu, 11 Apr 2024 00:00:00

Ensure shares is a functionality in the manila share service that goes over all of the shares within that backend and checks and updates them in case something has changed since the last time it was invoked.

Currently, ensure shares is only called while the manila-share binaries are being restarted, and in case the OpenStack Operator has identified that something has changed in the storage backend.

Ensure shares can not be invoked via the Manila API, so in case a configuration changes, the corresponding manila-share binaries must be restarted to pick up the new configuration.

Problem description

In cloud environments, the share backend configuration can change, and such changes might mean that the share will have outdated mount information until Manila applies the most recent configuration.

Currently, we are only applying such changes when the manila-share binary starts up and invokes the ensure_shares routine for a share driver. In a multi-backend environment, a single share manager service instance spawns multiple processes, each interacting with a single storage backend through a driver. In such an environment, it is possible that “ensure_shares” needs to be run for a single/specific storage backend driver. Today, there is no such provision. So, even if an operator wants to ensure shares belonging to a single backend, they would need to perform a disruptive restart of all backend drivers in such environments.

Ensure shares being called only when the service is restarted provides little flexibility to OpenStack operators to make configuration changes and apply them quickly to all of the shares within that backend, as every time they would need to restart the manila-share service. This operation is also disruptive, and will cause a temporary outage to all of the binaries involved.

Use Cases

OpenStack Operators would like to update the backend configuration and having it applied quickly to the shares without having the cloud facing the downtime of a manila-share service restart, making this a good way to have export locations and other important share information updated easily.

Proposed change

New API for ensure shares

We will introduce an API that allows OpenStack Operators to call the manila share’s ensure shares routine, making this configuration update smooth and requiring no service downtime, so that export locations can be re-calculated and shares updated with the newer config.
Changes to manila shares

Shares will have a different status while the configuration updates are being applied: ensuring. Ensure shares can take a while to complete, as we will be updating all shares in the given backend. After all shares had the configuration applied, we will transition them back to the available status in case there was a status change.
Changes to the manila-share service:

While running ensure shares, the ensuring field for the service will transition to True until the operation is complete. All services which the binary are not manila-share will remain untouched.
Things to consider for this spec

The manila-share service should be up and enabled for being able to run the ensure shares through the API.

We will always allow the ensure shares API to be triggered. The manila share-manager service will decide if a share needs its data to be changed or not based on the status of the share, or if the share is busy or not.

A new configuration option named update_shares_status_on_ensure will be added, and it will default to True. Through this configuration option, OpenStack operators can decide whether they want the shares status to be changed when ensure shares was invoked through the API or not.

Alternatives

Implementing periodic configuration checks, with configurable time for when they would run, and as a result they would pick up the newest configuration and apply them to the shares. However, routine would be run periodically, meaning that we would check for updates even if there are not.

Data model impact

A new status (ensuring) will be added to the shares’ possible status.
New field will be added to the services table +———————–+—————+——+—–+———+——-+ | Field | Type | Null | Key | Default | Extra | +=======================+===============+======+=====+=========+=======+ | ensuring | tinyint(1) | YES | | NULL | | +———————–+—————+——+—–+———+——-+

CLI API impact

A new command will be added to the Manila’s OpenStack CLI:

openstack share service ensure <host@backend>

host@backend: the host and the backend that should update its shares configuration.

REST API impact

To trigger the ensure shares routine:

POST /v2/services/ensure

Request:

{
  "host": "openstackhost@generic",
}

Normal http response code(s):

202 - The share service will start the ensure shares procedure.

Expected http error code(s):

401 - Unauthorized; user has not authenticated
403 - Forbidden; user is forbidden by policy
404 - Not found; API does not exist in micro version
404 - Not found; There is no host and backend matching the provided
409 - Conflict; Not all shares are currently available.
409 - Conflict; Service is not up.

Driver impact

No driver impact, as drivers are currently free to implement what is necessary during the ensure shares routine, we will just allow OpenStack Operators to access it easily.

Security impact

We will implement RBAC rules for this API. Only OpenStack Operators will be able to use it by the virtue of the default RBAC policy.

Notifications impact

We will emit a notification when the “ensure_shares” operation starts and when the operation ends.

Other end user impact

When state changes are enabled, and an OpenStack operator invokes the ensure shares API, end users will notice that their shares have transitioned to “ensuring” status. They will be unable to perform any actions on the share via Manila. Unless the backend driver updates export locations as part of this operation, there is no impact on the data path.

Performance Impact

Shares will be updated by the manila-share service and this can take some time to complete, specially in busy clouds.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:

carloss (ces.eduardo98@gmail.com)

Work Items

Implement the ensure shares API.
Implement the corresponding OSC commands.
Implement the corresponding functionality in SDK.
Implement tempest tests.
Update the documentation.
Add the feature to manila-ui.

Future Work Items

None

Dependencies

None

Testing

Unit tests
Tempest tests

Documentation Impact

User guide
Admin guide

References

[1]: https://bugs.launchpad.net/manila/+bug/1996793 [2]: https://etherpad.opendev.org/p/caracal-ptg-manila-cephfs

Manila share and share snapshot deferred deletion

Thu, 23 Nov 2023 00:00:00

Blueprint: https://blueprints.launchpad.net/manila/+spec/deferred-deletion

Manila does not support the deferred deletion of share and share snapshots. This blueprint is to add support for share and share snapshot deferred deletion. The end user will delete the share and share snapshot using API and then Manila will perform the deferred deletion of the share and share snapshot respectively thereby freeing quota immediately.

Problem description

If users want to remove a share or share snapshot from Manila, it is possible to delete them using API. The manila-API part of deletion procedure finishes quickly, on the other hand the backend driver takes time. The quotas allocated to the share or share snapshot are not freed until share or share snapshot are deleted completely. This increases waiting time for creation of new share or share snapshot. Providing a way to release the quota immediately and perform deferred deletion will solve this issue. Also automated workflows face problem in creation of new shares and/or share snapshots where they have to wait until quota is available after share or share snapshot is deleted by backend driver.

Use Cases

In automated workflows where deletion of share/share-snapshot is an intermediate step, have to wait until share/share-snapshot is deleted. Defer deletion allows automation to proceed without waiting for driver.
The user want to delete share/share-snapshot and check the available share/share-snapshot and their corresponding gigabytes quota.

Proposed change

The following are the changes to be made:

A new status values ‘DELETING_IN_DRIVER’ and ‘DELETING_IN_DRIVER_ERROR’ will be introduced to identify share instances which will be processed in deferred deletion.
Add new configuration option ‘deferred_deletion’, which means if TRUE the shares are deleted in deferred deletion manner, and if FALSE the share will be deleted the way it is happening today.
As soon as share or share-snapshot is deleted by API request, the quota will be reclaimed and share/share-snapshot instance will be moved to ‘DELETING_IN_DRIVER’ state. Any delete request on share/share-snapshot while it is in ‘DELETING_IN_DRIVER’ state, will return error. A periodic task running every 300 seconds will be added in share manager, which gets all share instances in ‘DELETING_IN_DRIVER’ state. The share manager, will then call backend driver to delete the share/share-snapshot instance and move it to ‘DELETED’ state on successful deletion. Any failure during deletion, will move share/share-snapshot instance in the ‘DELETING_IN_DRIVER_ERROR’ state. Only admin can list those shares/share-snapshots and then delete/cleaup them on storage, where they will be marked as ‘DELETED’. The periodic task also checks for share/share-snapshot in the ‘DELETING_IN_DRIVER_ERROR’ state and tries to delete them again after 30 minutes of previous attempt.
No change in backend driver.

Alternatives

Instead of adding new status value, we can mark share/share-snapshot as ‘DELETED’ immediately and move those shares/share-snapshots to another project. This project will contain only deleted shares/share-snapshots by manila API service. The share manager will run a periodic task to delete share and share-snapshot from this project. In this case as well since quota is reclaimed earlier, share manager will make sure share instance must be deleted by backend driver.

An issue with this approach is we will change project id of original share during deletion which can be unwanted data modification.

Data model impact

No change in share instance table except introduction of two more values for ‘status’ field.

CLI API impact

None

REST API impact

Shares and share snapshots in ‘deleting_in_driver_error’ state are not visible to regular user. Only admins can list/show those shares and share snapshots because they can react to such error like cleaning up them.

Driver impact

None

Security impact

None

Notifications impact

None

Other end user impact

If configured, instead of ‘deleting’ and ‘deleting_error’ states, end user will now see ‘deleting_in_driver’ and ‘deleting_in_driver_error’ states respectively. Also, end user will observe that share in ‘deleting_in_driver’ state has freed its quota immediately.

The cloud admins can list shares in the new state(s) via API.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

kpdev(kinpaa@gmail.com)

Work Items

Implement periodic share manager function to process deferred deletion.
Implement tempest support.

Future Work Items

None

Dependencies

None

Testing

Unit tests
Tempest tests

Documentation Impact

Docstrings
Devref
User guide
Admin guide
Release notes

References

None

Allow locking shares against deletion

Mon, 01 May 2023 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/manila/+spec/allow-locking-shares-against-deletion

The default RBAC permits a non-reader project user to create and delete shares under the project’s namespace. Deletion of shares can be dangerous, and we expect that users exercise caution before initiating the action. The Shared File Systems (Manila) API ensures that some pre-conditions are met prior to proceeding with the deletion. A desirable pre-condition would be to check if the share is actively being used by a client workload. Such a check however is not straight forward to perform as it cannot reliably be implemented in a consistent manner across all Network Attached Storage (NAS) protocols or storage system back ends that Manila supports. In other words, Manila does not know who and how many clients have mounted a share, or if data is actively being read or written into the share. So there is a need for a safety mechanism to prevent unintentional consequences.

This specification proposes a new pre-condition, one that allows any non-reader project user to create a deletion lock against a share in the project’s namespace. The deletion lock can be removed by the same user, or by a privileged user.

Problem description

A shared file system is served by a network file server and it allows several simultaneous clients to connect, read and write to it. In OpenStack, Manila shares are collectively owned by the project users that the share belongs to. A user in the project can delete a share that some other user is actively using, and Manila API provides no way to indicate or coordinate communication prior to this deletion.

Further, as part of the protocol, NAS clients are hardened to survive minor network interruptions and server side failures within a degree of tolerance. If the server goes unresponsive for a while, the client can wait, filling up its write cache or retrying its reads until they succeed, or until the tolerance expires. In the most common scenario, the client can be instructed to wait indefinitely (“hard mounts”). So, an extended server failure can be catastrophic to the client. If a file system that is mounted is deleted on the server, the client typically exercises the same waiting behavior and can go unresponsive in the process.

Use Cases

The most recent use case is with the OpenStack Compute feature that allows users to mount their shares to virtual machines via VirtIOFS [1]. With this feature, a compute host can plumb a mounted network filesystem to one or more guests. If the share is deleted while it is mounted, the compute host would be compromised. This would disable all virtual machines on the host, not just the virtual machine that was using the share via VirtIOFS. So while the Compute service orchestrates the mount, a user’s action of deleting the backing share can bring down the shared infrastructure.

Proposed change

Users will have the ability to lock any share in the project. Multiple locks can be placed on the share. A share cannot be deleted, unmanaged or soft-deleted unless all locks have been removed. Only the user that placed the lock, or the administrator user can remove a given lock. If a user attempts to lock a share that is previously locked by them for a specific resource action, the API will not re-create the lock. The lock record can be updated with a new lock reason.

A service user (with the use of X-Service-Token and a service role) can create a lock on a share on behalf of a regular user. A lock user context is also recorded by manila in such a case. So, even when using a user’s token, if a service token is provided, the lock created cannot be removed by the user. Only a service user, or a user with “admin” role can remove or update the lock.

The implementation of this feature will include generalizations for future extensibility. The lock API will accept a resource ID, resource type, and a resource action that must be locked. In the 2023.2 Bobcat release cycle, we will only be implementing deletion locks for shares.

Alternatives

Shares could have an “in-use” state that could prevent adverse manipulation. The presence of access rules can allow a share to transition to this “in-use” status. The problem with this approach is that users could drain access rules prior to deleting the share. This provides a two-step deletion ensuring that the action is deliberate. However, in the use case above, it wouldn’t protect OpenStack Compute service resources from losing the share gracelessly.

Data model impact

A new table will be introduced:

+-------------------+----------------------------------+----------+----------+
|       Field       |               Type               | Nullable | Default  |
+-------------------+----------------------------------+----------+----------+
| ID                | varchar(36)                      | NO       | NULL     |
| USER_ID           | varchar(36)                      | YES      | NULL     |
| PROJECT_ID        | varchar(36)                      | YES      | NULL     |
| RESOURCE_ACTION   | Enum('delete', ..)               | YES      | 'delete' |
| RESOURCE_TYPE     | Enum('share', ..)                | YES      | NULL     |
| RESOURCE_ID       | varchar(36)                      | NO       | NULL     |
| LOCK_USER_CONTEXT | Enum('user', 'service', 'admin') | YES      | NULL     |
| LOCK_REASON       | varchar(1023)                    | YES      | NULL     |
| DELETED           | varchar(36)                      | YES      | 'false'  |
| CREATED_AT        | datetime(6)                      | YES      | NULL     |
| DELETED_AT        | datetime(6)                      | YES      | NULL     |
| UPDATED_AT        | datetime(6)                      | YES      | NULL     |
+-------------------+----------------------------------+----------+----------+

The table will assist storing lock records and will be manipulated as locks are created, updated and removed. A database migration will create this table with no initial data. The LOCK_USER_CONTEXT field will permit “user”, “service” and “admin” via enum. RESOURCE_TYPE and RESOURCE_ACTION fields will also be limited by enum constants to supported resource types and resource actions.

REST API impact

The APIs using resource lock endpoints and methods will only be available in a new API micro version. However, if resource locks exist, they cannot be circumvented by using an older API micro version to perform the action that they are preventing.

Create a resource lock on a particular action:

POST /v2/resource-locks

Normal http response code(s):

200 - Lock created successfully

Expected http error code(s):

401 - Unauthorized; user has not authenticated
400 - Bad Request
400 - Unrecognized action on resource
400 - Unrecognized resource (no such resource in project namespace)
403 - Forbidden; user is forbidden by policy
404 - API does not exist in micro version

Request example:

{
    'resource_lock': {
        'resource_action': 'delete',
        'resource_type': 'share',
        'resource_id': 'a448e0d2-7501-4b99-a447-1b89e3961e39',
        'lock_reason': 'share is used by audit team'
    }
}

Response example:

{
    'resource_lock': {
        'id': 'be0871e8-742e-4c19-8567-7016fa0e2235',
        'user_id': 'cec1dd3e297b45348228f4fc3f5dba38',
        'project_id': '2e47ac4e2cf04a5b8b8509de8177d65d',
        'resource_action': 'delete',
        'resource_type': 'share',
        'resource_id': 'a448e0d2-7501-4b99-a447-1b89e3961e39',
        'lock_reason': 'share is used by audit team',
        'created_at': '2023-04-28T09:49:58-05:00',
        'updated_at': None
    }
}

Update a resource lock:

PUT /v2/resource-locks/{id}

Updatable fields include “resource_action” and “lock_reason”. “lock_reason” can be nullified on update. Only the user that created the lock or a user with “admin” role will be allowed to update a lock per default RBAC policy.

Normal http response code(s):

200 - Lock updated successfully

Expected http error code(s):

401 - Unauthorized; user has not authenticated
400 - Bad Request
400 - Unrecognized action on resource
403 - Forbidden; user is forbidden by policy
404 - API does not exist in micro version
404 - lock does not exist in project namespace

Request example:

{
    'resource_lock': {
        'lock_reason': 'share will be used by audit team until 2024'
    }
}

Response example:

{
    'resource_lock': {
        'id': 'be0871e8-742e-4c19-8567-7016fa0e2235',
        'user_id': 'cec1dd3e297b45348228f4fc3f5dba38',
        'project_id': '2e47ac4e2cf04a5b8b8509de8177d65d',
        'resource_action': 'delete',
        'resource_type': 'share',
        'resource_id': 'a448e0d2-7501-4b99-a447-1b89e3961e39',
        'lock_reason': 'share will be used by audit team until 2024',
        'created_at': '2023-04-28T09:49:58.231919',
        'updated_at': '2023-04-28T20:01:13.12106'
    }
}

Delete a resource lock:

DELETE /v2/resource-locks/{id}

Only the user that created the lock or a user with “admin” role will be allowed to delete a lock per default RBAC policy.

Normal http response code(s):

204 - Lock deleted successfully

Expected http error code(s):

401 - Unauthorized; user has not authenticated
400 - Bad Request
403 - Forbidden; user is forbidden by policy
404 - API does not exist in microversion
404 - lock does not exist in project namespace

Request and response do not contain any data

List resource locks:

GET /v2/resource-locks?{queries}

Queries will allow filtering with exact and inexact (“created_since”, “created_before”) attributes. Querying with “project_id” or “all_projects” will only be allowed for a user with “admin” role per default RBAC policy.

Normal http response code(s):

200 - List of locks in project namespace

Expected http error code(s):

401 - Unauthorized; user has not authenticated
403 - Forbidden; user is forbidden by policy
404 - API does not exist in microversion

Response example:

{
    'resource_locks': [
        {
            'id': 'be0871e8-742e-4c19-8567-7016fa0e2235',
            'user_id': 'cec1dd3e297b45348228f4fc3f5dba38',
            'project_id': '2e47ac4e2cf04a5b8b8509de8177d65d',
            'resource_action': 'delete',
            'resource_type': 'share',
            'resource_id': 'a448e0d2-7501-4b99-a447-1b89e3961e39',
            'lock_reason': 'share will be used by audit team until 2024'
        },
        {
            'id': '4945b04e-cdda-4308-9cfd-1483e7f9dd8c',
            'user_id': '80b789450540431db23575b333059ca8',
            'project_id': '2e47ac4e2cf04a5b8b8509de8177d65d',
            'resource_action': 'shrink',
            'resource_type': 'share',
            'resource_id': '4227fbd2-7f55-4ff4-9239-2cfc700d9fdf',
            'lock_reason': 'space is reserved for in place snapshots'
        }
    ]
}

Show lock:

GET /v2/resource-locks/{id}

Normal http response code(s):

200 - Details of a lock in the project namespace

Expected http error code(s):

401 - Unauthorized; user has not authenticated
403 - Forbidden; user is forbidden by policy
404 - API does not exist in micro version
404 - lock does not exist in project namespace

Response example:

{
    'resource_lock': {
        'id': 'be0871e8-742e-4c19-8567-7016fa0e2235',
        'user_id': 'cec1dd3e297b45348228f4fc3f5dba38',
        'project_id': '2e47ac4e2cf04a5b8b8509de8177d65d',
        'resource_action': 'delete',
        'resource_type': 'share',
        'resource_id': 'a448e0d2-7501-4b99-a447-1b89e3961e39',
        'lock_reason': 'share will be used by audit team until 2024',
        'created_at': '2023-04-28T09:49:58.231919',
        'updated_at': '2023-04-28T20:01:13.12106'
    }
}

Deleting a share that has locks:

DELETE /v2/shares/{id}

Normal http response code(s):

202 - No locks exist and all other pre-conditions allow, accepted

Expected http error code(s):

401 - Unauthorized; user has not authenticated
403 - Forbidden; user is forbidden by policy
404 - API does not exist in microversion
404 - share does not exist in project namespace
409 - share deletion precondition failed, perhaps there’s a lock

A “delete” lock will also prevent soft deletion or un-manage operations on the share in a similar fashion.

New RBAC policies will be introduced:

"""Policy defaults that are used in specific policies below:"""

RULE_ADMIN = "role:admin"
RULE_SERVICE = "role:service"
RULE_ADMIN_OR_SERVICE = f'({RULE_ADMIN}) or ({RULE_SERVICE})'
PROJECT_MEMBER = "rule:project-member"
PROJECT_READER = "rule:project-reader"
PROJECT_OWNER_USER = "rule:project-owner-user"

ADMIN_OR_SERVICE_OR_PROJECT_MEMBER = f'({RULE_ADMIN_OR_SERVICE}) or ({PROJECT_MEMBER})'
ADMIN_OR_SERVICE_OR_PROJECT_MEMBER = f'({RULE_ADMIN_OR_SERVICE}) or ({PROJECT_READER})'
ADMIN_OR_SERVICE_OR_PROJECT_OWNER_USER = f'({RULE_ADMIN_OR_SERVICE}) or ({PROJECT_OWNER_USER})'

rules = [
    policy.RuleDefault(
        name='project-member',
        check_str='role:member and '
                  'project_id:%(project_id)s',
        description='Project scoped Member',
        scope_types=['project']),
    policy.RuleDefault(
        name='project-reader',
        check_str='role:reader and '
                  'project_id:%(project_id)s',
        description='Project scoped Reader',
        scope_types=['project']),
    policy.RuleDefault(
        name='project-owner-user',
        check_str='role:member and '
                  'project_id:%(project_id)s and '
                  'user_id:%(user_id)s',
        description='Project scoped Member who owns a resource',
        scope_types=['project']),
]

Create a lock

policy.DocumentedRuleDefault(
     name= 'resource_locks:create',
     check_str=ADMIN_OR_SERVICE_OR_PROJECT_MEMBER,
     scope_types=['project'],
     description="Create a resource lock.",
     operations=[
         {
             'method': 'POST',
             'path': '/resource-locks',
         },
     ],
 )

Update a lock

policy.DocumentedRuleDefault(
     name= 'resource_locks:update',
     check_str=ADMIN_OR_SERVICE_OR_PROJECT_OWNER_USER,
     scope_types=['project'],
     description="Update a resource lock.",
     operations=[
         {
             'method': 'PUT',
             'path': '/resource-locks/{id}',
         },
     ],
 )

Delete a lock

policy.DocumentedRuleDefault(
     name= 'resource_locks:delete',
     check_str=ADMIN_OR_SERVICE_OR_PROJECT_OWNER_USER,
     scope_types=['project'],
     description="Delete a resource lock.",
     operations=[
         {
             'method': 'DELETE',
             'path': '/resource-locks/{id}',
         },
     ],
 )

List locks

policy.DocumentedRuleDefault(
     name= 'resource_locks:index',
     check_str=ADMIN_OR_SERVICE_OR_PROJECT_READER,
     scope_types=['project'],
     description="List all resource locks.",
     operations=[
         {
             'method': 'GET',
             'path': '/resource-locks?{queries}',
         },
     ],
 )

List locks with project queries

policy.DocumentedRuleDefault(
     name= 'resource_locks:get_all_projects',
     check_str=RULE_ADMIN,
     scope_types=['project'],
     description="Get resource locks across projects.",
     operations=[
         {
             'method': 'GET',
             'path': '/resource-locks?all_projects=1&project_id={project_id}',
         },
     ],
 )

Get lock

policy.DocumentedRuleDefault(
     name= 'resource_locks:get',
     check_str=ADMIN_OR_SERVICE_OR_PROJECT_READER,
     scope_types=['project'],
     description="Get details about a resource lock.",
     operations=[
         {
             'method': 'GET',
             'path': '/resource-locks/{id}',
         },
     ],
 )

Driver impact

None. This is an API only feature.

Security impact

Default RBAC policies will allow users with “admin” role to create, view or delete user locks. The “admin” role is presumed to be given to the operator user. If a lock must be created on behalf of the user by a service or an application, it is advised that the service or application is configured with a user that has the “service” role and not “admin”.

No further security impact, positive or negative is noted.

Notifications impact

“lock.create” and “lock.delete” notification events will be emitted for the respective actions.

Other end user impact

User Interface improvements will be introduced in OpenStackClient (python-manilaclient plugin) and the OpenStack Dashboard (manila-ui plugin). The OpenStackClient addition will be accompanied by manilaclient and openstacksdk interfaces:

Create a resource lock:

openstack share lock create <resource_id> \
  [--resource-action <resource_action>] \
  [--resource-type <resource_type>] \
  [--reason <lock_reason>}]

The “resource-action” defaults to “delete”.

Update a resource lock:

openstack share lock update <id> \
  [--resource-action <resource_action>] \
  [--reason <lock_reason>}]

Delete a resource lock:

openstack share lock delete <id>

List resource locks:

openstack share lock list

Show a resource lock:

openstack share lock show <id>

Performance Impact

As we’re introducing a new pre-condition on share deletion (or unmanage/soft-deletion), the share delete API will suffer performance degradation due to the additional lookup. It’s not possible to avoid this lookup even when locks are not used in the environment. We’ll optimize the query by using appropriate indices. In the future, as more resources and resource actions use this approach, we will be impacting the existing performance of these APIs. It’s a trade-off for the feature functionality.

Other deployer impact

None.

Developer impact

Consider allowing locks via this interface when defining or manipulating actions.

Implementation

Assignee(s)

Primary assignee:: gouthamr

Work Items

Manila API changes
support in manilaclient, openstackclient, manila UI
support in openstacksdk
e2e tests with manila-tempest-plugin
API Reference, user and administrator documentation

Dependencies

This feature doesn’t depend on work elsewhere, but, the VirtIOFS integration effort in Nova requires this feature.

Testing

New tests will be added to create locks, list locks, show locks, delete locks. Test cases will cover use of multiple locks and involve validation of request and response schema and codes. RBAC policies will also be tested via tempest.

Documentation Impact

API Reference will be updated alongside the API changes. User and administrator documentation will follow alongside the UX changes in respective repositories.

References

[1] VirtIOFS Specification

[2] 2023.2 Bobcat PTG Discussion

Prevent Access rules from being viewed or manipulated by non-owners

Mon, 01 May 2023 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/manila/+spec/protect-access-rules

In the OpenStack Shared File Systems service (Manila), by virtue of default RBAC, share access rules (a.k.a “Access Control Lists”) can be created, viewed and deleted by all users of the project that owns the share. An access rule can contain sensitive information. This specification proposes an approach to prevent sharing of this sensitive information amongst all users of the project.

Problem description

When creating an access rule for a share, a user can provide type of access requested (access_type), a client identifier (access_to), the level of access (access_level) and metadata. All of this information is namespaced to the share’s project. So, all project users can view the access rules associated with a share within the project. Sometimes this may not be desirable. When a user creates a CephX access rule, they are able to retrieve the CephX access secret key via Manila, this key can be private information. Sharing this key among other users of the project would mean that a CephX user key is visible and usable by multiple OpenStack project users. If the goal was to allow only a subset of the OpenStack users to access the share, the purpose is defeated with this level of visibility. The same applies if individual CephX users had to have different access levels (“read-only” vs “read-write”); since the CephX users (access_to) and CephX access secret keys (access_key) are easily available, nothing prevents unauthorized access amongst project users.

In case of NFS with IP access control, users may not want to “leak” the IP address of the client for security reasons, even amongst other users of the same project.

Use Cases

When creating and using CephFS shares, different OpenStack users would like to protect their own CephX access keys from being visible to other users in the project.

The OpenStack Compute service (Nova) creates an access rule to mount the share on a compute host and make it available to a virtual machine via VirtIOFS. It expects that the access rule data (access_to and access_key) are not visible to other users. [1] It’s important to clarify here that Nova would be performing the API interactions with the user’s own token. However, Manila would rely on nova also supplying the X-Service-Token header. Doing so would allow manila to create any visibility restrictions in favor of the nova service user, instead of the project user.

Nova would also like to prevent users from deleting an access rule that was created since it would hard mount the share to the compute host.

Proposed change

When creating an access rule, users will be able to restrict the visibility of the access_to and access_key fields to themselves. A restricted access rule is still visible to all project users, however, these sensitive fields are only visible to the user that imposed the restriction. A restricted access rule cannot be deleted by other users in a project. Only the user, or a more privileged user (“admin”) can remove the restrictions. Once a restriction is removed, other users in the project can view all details regarding the previously restricted access rule.

These restrictions will be facilitated through resource locks. Through the rest of this specification, for the sake of brevity, we’ll refer to a resource lock that imposes restrictions as a “access rule lock”. The access rules API will be enhanced to create such an access rule lock automatically, and drop it based on new API input parameters.

An OpenStack service (like nova) can use the X-Service-Token and proxy the user’s own token. The restrictions created by a service user will be imposed in favor of the service user in this case. This means that sensitive access rule fields can only be viewed by the service user, and the rule can only be deleted by the service user. Internally, manila will enforce this by recording a “lock user context”. In consequence, even when using a user’s token, if a service token is provided, the lock created cannot be updated or removed by the user. Only a service user, or a user with “admin” role can remove or update the lock. Manila will validate that the X-Service-Token corresponds to a user with a service role.

This ability to restrict an access rule will have an important caveat. While the deletion of the access rule is restricted, the corresponding share can still be deleted. To prevent deletion of shares, a dedicated resource deletion lock can be used.

Alternatives

This feature can be limited to “admin” and “service” users only. It’d be easier to implement the visibility and manipulation restrictions directly via RBAC in this case. However, such an approach does not lend itself to the self-service nature of the cloud. Regular users would have to work through the administrator user to gain the ability to impose these restrictions.

Data model impact

None.

REST API impact

Create an access rule with restrictions:

POST /shares/{share_id}/action

Normal http response code(s):

202 - Access rule creation accepted

Expected http error code(s):

401 - Unauthorized; user has not authenticated
400 - Bad Request
403 - Forbidden; user is forbidden by policy
404 - API does not exist in micro version

Request example:

{
    "allow_access": {
        "access_type": "ip",
        "access_to": "203.0.113.10",
        "restrict": "True"
    }
}

Response example:

{
    "access": {
        "share_id": "406ea93b-32e9-4907-a117-148b3945749f",
        "created_at": "2023-04-30T09:14:48.000000",
        "updated_at": null,
        "access_type": "ip",
        "access_to": "203.0.113.10",
        "access_level": "rw",
        "access_key": null,
        "id": "a25b2df3-90bd-4add-afa6-5f0dbbd50452",
        "metadata": null
    }
}

When an access rule has restrictions applied, a different user without the “admin” or “service” roles will see the access_to and access_secret fields set to ******.

Delete an access rule with restrictions:

POST /shares/{share_id}/action

Normal http response code(s):

202 - Access rule deletion accepted

Expected http error code(s):

401 - Unauthorized; user has not authenticated
400 - Bad Request
403 - Forbidden; user is forbidden by policy
404 - API does not exist in micro version

Request example:

{
    "deny_access": {
        "access_id": "a25b2df3-90bd-4add-afa6-5f0dbbd50452",
        "unrestrict": "True",
    }
}

The API provides no response body. If the “unrestrict” field is not specified, a restricted access rule cannot be deleted. The API will respond with HTTP 400 in that case. If the “unrestrict” field is specified, but, the user doing so isn’t the one that restricted the rule; or isn’t a “service” user if the lock user context is “service”, the API will respond with error code HTTP 403.

Create a restriction on an existing access rule

POST /v2/resource-locks

Normal http response code(s):

200 - Lock created successfully

Expected http error code(s):

401 - Unauthorized; user has not authenticated
400 - Bad Request
400 - Unrecognized action on access rule
400 - Unrecognized access rule (no such rule in project namespace)
403 - Forbidden; user is forbidden by policy
404 - API does not exist in micro version

Request example:

{
    'resource_lock': {
        'resource_action': 'view,delete',
        'resource_type': 'access_rule',
        'resource_id': '222e1229-3de7-4678-9cb9-20cfd4ca9776',
        'lock_reason': 'infra host rule managed by fancyuser1'
    }
}

Response example:

{
    'resource_lock': {
        'id': '413080b6-1a20-48e1-9516-b2c509d034ec',
        'user_id': 'cec1dd3e297b45348228f4fc3f5dba38',
        'project_id': '2e47ac4e2cf04a5b8b8509de8177d65d',
        'resource_action': 'view,delete',
        'resource_type': 'access_rule',
        'resource_id': 'a448e0d2-7501-4b99-a447-1b89e3961e39',
        'lock_reason': 'infra host rule managed by fancyuser1',
        'created_at': '2023-04-28T09:49:58-05:00',
        'updated_at': None
    }
}

Update access rule restrictions

To remove or update an access rule restriction without deleting the access rule, see allow-locking-shares-against-deletion.html#update-resource-locks.

Other API impact

When access rule restrictions are present on a share, a transfer request to move it across OpenStack projects cannot be accepted with keeping the rules intact. The API will respond with 409 Conflict.

The REST API micro version will be increased indicating these new features. Access rule restrictions can only be applied when using an API micro version that is equal to or later than the micro version at which these API changes appear. However, when access rule restrictions exist, they cannot be circumvented by using a lower API micro version.

Driver impact

None. This change affects API behavior and is back end driver agnostic.

Security impact

The proposed change enhances the security model of the access control lists. Restrictions can be created with new and existing access rules. The enhancement is opt-in.

Notifications impact

None

Other end user impact

This feature will be supported by the OpenStack CLI plugin and manilaclient SDK in python-manilaclient. The CLI interactions will look like the following:

Create an access rule with restrictions:

openstack share access create <share> <access_type> <access_to> \
   [--access-level <access_level> \
   [--restrict] \
   [--properties [<key=value> ...]] \
   [--wait]

Set the restrictions on an access rule:

openstack share lock create <access_id> \
--resource-action view-or-delete \
--resource-type access \
[--reason <lock_reason>}]

Unset the restrictions on an access rule:
```
openstack share lock delete <lock_id>
```

Delete an access rule with restrictions:

openstack share access delete <access_rule_id> --unrestrict

Performance Impact

Access rule restrictions introduce pre-conditions in API methods to list and delete access rules. Evaluating these pre-conditions will introduce a minor performance degradation. This should be made better with appropriate indices in the resource_locks database table.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: gouthamr
Other assignee:: carloss

Work Items

Testing with manila-tempest-tests
Support in python-manilaclient, openstacksdk, manila-ui
Documentation

Dependencies

This feature is required by the Nova/VirtioFS feature [1].

Testing

Code changes will be covered with unit and tempest tests.

Documentation Impact

API Reference
User Guide documentation

References

[1] VirtIOFS Specification

[2] Manila Specs: Allow locking shares against deletion

[3] 2023.2 Bobcat PTG Discussion

Spec Lite: Scheduler Weigher based on Netapp Active IQ

Thu, 27 Apr 2023 00:00:00

problem:

The Manila Scheduler weighers are generic, it is designed to weigh based on the backend stats, being restricted to that set of information. For some NetApp environments, it could be more helpful to have a more specific tool that tracks the storages status (space, cpu usage and several others) and use statistics and artificial intelligence techniques for weighing the hosts in a more efficient manner.

solution:

To make a better weight evaluation, NetApp has its own software called Active IQ (AIQ) [1]. The AIQ connects to the NetApp storages collecting and processing the status of each device dynamically. Using REST API endpoints, external services can request the AIQ to weigh the available storages for a given share creation. API request example:

{
     "method": "POST",
     "url": "https://10.63.167.192/api/storage-provider/data-placement/balance",
     "args": [],
     "headers": {
         "Accept": "application/json",
         "Content-Type": "application/json"
     },
     "json": {
         "capacity": "10GB",
         "eval_method": 0,
         "opt_method": 0,
         "priority_order": [
             "ops",
             "latency",
             "volume_count",
             "size"
         ],
     "ssl_key": "bbda43aa-aee9-11ed-a4cd-005056bd7087",
     "separate_flag": false,
     "resource_keys": [
         "1c54fcf0-2adb-11ec-86b0-d039ea2ef942:type=aggregate,uuid=b22ea874-eada-4c31-b6f9-1cf95e2bdacc",
         "1c54fcf0-2adb-11ec-86b0-d039ea2ef942:type=aggregate,uuid=a39c5d4f-2ca9-4910-b57f-71a7936656c9",
         "1c54fcf0-2adb-11ec-86b0-d039ea2ef942:type=aggregate,uuid=ec6a23a2-c98f-4640-8b4b-4615c1969751"
         ]
     }
}

The API response:

{
    "request_ok": true,
    "status_code": 200,
    "headers": {
        "Expires": "0",
        "Cache-Control": "no-cache, no-store, must-revalidate",
        "X-Powered-By": "NetApp Application Server",
        "Server": "NetApp Application Server",
        "X-XSS-Protection": "1; mode=block",
        "Pragma": "no-cache",
        "X-Frame-Options": "SAMEORIGIN"
    },
    "data": [
        {
            "scores": {
                "total_weighted_score": 30.0
            },
            "key": "1c54fcf0-2adb-11ec-86b0-d039ea2ef942:type=aggregate,uuid=a39c5d4f-2ca9-4910-b57f-71a7936656c9"
        },
        {
            "scores": {
                "total_weighed_score": 20.0
            },
            "key": "1c54fcf0-2adb-11ec-86b0-d039ea2ef942:type=aggregate,uuid=b22ea874-eada-4c31-b6f9-1cf95e2bdacc"
        },
        {
            "scores": {
                "total_weighted_score": 10.0
            },
            "key": "1c54fcf0-2adb-11ec-86b0-d039ea2ef942:type=aggregate,uuid=ec6a23a2-c98f-4640-8b4b-4615c1969751"
        }
    ]
}

The idea is proposing a vendor specific Scheduler weigher called NetAppActiveIQWeigher, which will call the NetApp AIQ for getting the weight of each host. If the list of host contains at least one non NetApp host, the weigher is skipped. In order to connect to the external service, the new weigher requires connection configurations set by the cloud administrator, they are: aiq_username, aiq_password, aiq_hostname and aiq_port. For more weighing flexibility, the weigher will contain the following optional configurations: aiq_eval_method, aiq_opt_method, aiq_priority_order and aiq_separate_flag (see the Active IQ documentation). The cloud administrator may want to filter the storages according to its performance and storage objectives for a new workload. The AIQ has the concept of performance service level that provide this functionality and it can be informed during the weight request. As result, this spec is proposing to add a new NetApp scoped extra-spec called netapp:aiq_performance_level as a UUID string representing the AIQ performance service level. The weight request requires the set of aggregates UUIDs that will be evaluated. Unfortunately, the NetApp driver stats does not contain it, so the driver have to start to report the UUID of each pool. It could be collected once during driver start up, not affecting the driver life cycle.

impacts:

NetApp Driver Impact.
- The driver will start reporting the NetApp aggregate UUID
Documentation Impact
- Admin guide
- Contributor guide
Scheduler Impact
- Propose a vendor weigher: NetAppActiveIQWeigher
CI Impact
- Add the new weigher for the dummy driver jobs
- Add the new weigher for the NetApp jobs

This implementation may impact the performance of the scheduler weigher phase, since the NetApp weigher will run a network request.

alternative:

There is no alternative other than keep running with generic weighers.

timeline:

Include in Bobcat release.

link:

[1]: https://docs.netapp.com/us-en/active-iq-unified-manager/index.html

assignee:

felipe_rodrigues

Manila oversubscription enhancements

Tue, 19 Apr 2022 00:00:00

https://blueprints.launchpad.net/manila/+spec/manila-oversubscription-enhancements

About allocated_capacity_gb and provisioned_capacity_gb, allocated_capacity_gb corresponds to shares_gb created on a storage pool via manila. provisioned_capacity_gb is equal to the sum of allocated_capacity_gb add shares_gb_created_manually (or another manila) on the same storage pool. It implies that provisioned_capacity_gb greater than or equal to allocated_capacity_gb. It is important for administrators to know these two values in order to obtain information about back-end allocated capacity.

For a storage backend that supports thin provisioning,The following describes the reported capacity information.

total_capacity_gb = Total physical disk capacity of the storage pool used by Manila.

free_capacity_gb = total_capacity_gb - Used physical disk capacity of the storage pool.

provisioned_capacity_gb = Total capacity allocated by all shares (include shares created manually or other manila) in the storage pool.

allocated_capacity_gb = Total capacity allocated by all shares created by current manila.

Problem Description

Problem One: Here’s an example. Storage pool pool_A physical disk capacity is 10G. There are four shares in this pool. Share A created by openstack O_A(current manila), share size is 5G. There is a 124MB file in this share. Share B created by openstack O_A(current manila), share size is 4G. There is a 0MB file in this share. Share C created by openstack O_B, share size is 3G. There is a 400MB file in this share. Share D created manually, share size is 2G. There is a 500MB file in this share.

The total physical capacity in use is 1G(124MB+0MB+400MB+500MB), the result is: total_capacity_gb = 10G free_capacity_gb = 9G provisioned_capacity_gb = 14G (5G+4G+3G+2G) allocated_capacity_gb = 9G (5G+4G, only share A and B is created by current manila)

In fact, the storage back end doesn’t know how many Manila services are using the storage pool or whether the administrator has manually created shares in the pool. So the allocated_capacity_gb reported by backend storage driver is not correct. Only provisioned_capacity_gb reported by backend storage driver is correct.

Currently, some drivers (inspur, huawei, netapp, qnap, zadara) report allocated_capacity_gb and some drivers (infinibox, hpe_3par, nexenta) report provisioned_capacity_gb, which is chaotic.

Problem Two: About scheduler service supports Active/Active HA, The consume_from_share function is executed each time a share is created or extend, in this function allocated_capacity_gb and free_capacity_gb must be updated. Let’s suppose 100 1gb shares are created, 50 of which are executed on node 1 and 30 on node 2 and 20 on node 3. Since allocated_capacity_gb is maintained in the memory of the scheduling service. As a result, the allocated_capacity_gb values of the three scheduling nodes are inconsistent. This is problematic. We need to calibrate allocated_capacity_gb periodically.

Use Cases

The administrator can view the capacity allocation of the back-end storage by allocated_capacity_gb and provisioned_capacity_gb.
Manila scheduler use provisioned_capacity_gb to filter and weigh back-end.

Proposed Change

This spec proposes to:

allocated_capacity_gb reported by driver may be correct if the driver is capable of tracking it accurately, but allocated_capacity_gb calculated from database must be correct. So for uniformity, allocated_capacity_gb will use calculated by database rather than reported by driver.
we strongly recommend that the driver report provisioned_capacity_gb, if it is not reported, Manila is considered exclusive storage pool, will use allocated_capacity_gb as the provisioned_capacity_gb value.
for problem two, Add a periodic task in Manila Share to periodically collect allocated_capacity_gb statistics from the database and record it in self._allocated_capacity_gb(a member variable of class ShareManager). Read self._allocated_capacity_gb and update allocated_capacity_gb in share_stats[pools] when executing _report_driver_status. Because Manila doesn’t require very much real-time data for allocated_capacity_gb, Therefore, the interval of periodic tasks can be set to 5 minutes to reduce system and database pressure.

Alternatives

The current model is an alternative, however, the calculation occuring in the scheduler is problematic because it’s happening in each scheduler process for each backend and for each pool, which is inefficient and wrong

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

we’d be removing a huge performance bottleneck from the scheduler service!

Other deployer impact

None

Developer impact

Drivers should update to report provisioned_capacity_gb instead of allocated_capacity_gb.

Implementation

Assignee(s)

Primary assignee:: haixin <haix09@chinatelecom.cn>

Work Items

Update share manager and scheduler’s host manager.
Update unittest.
Update related documents.

Dependencies

None

Testing

Add the unit tests

Documentation Impact

The following OpenStack documentations will be updated to reflect this change:

OpenStack Contributor Guide

References

None

Transfer share between project

Tue, 23 Nov 2021 00:00:00

https://blueprints.launchpad.net/manila/+spec/transfer-share-between-project

Add support for share transfer that allows a user to transfer a share to another user under other project, just like cinder transfer-create, transfer-accept, transfer-delete. There are two scenarios, the first is DHSS=False, the second is DHSS=True. In the first scenario, will only support transferring shares that are not tied to a share network. The object of the transfer is Share. In the second scenario, we would support transferring share networks, which would cause all of its shares to be migrated within. This will be completed in another spec.

Problem Description

In an actual OpenStack production environment, it is common for a user to transfer a storage resource to another user for collaborative pipeline work. Manila Share also needs a secure way to transfer shares so that other users can receive them safely.

Use Cases

The user in project A pre-populate data for another user in project B.
When a project is “ending”/being decommissioned, share data can be transferred to a project that’s still needed.
Transfer share to specific user to collaborate on the rest of the work.

Proposed Change

This spec proposes to:

Add a new table, which named transfers. Includes the following fields:
- id the primary key of table.
- resource_type the type of resource which is being transferred, this can be share, share network and so on.
- resource_id the uuid of resource which is being transferred.
- name the display name of transfer.
- salt a short random string used to calculate crypt_hash, after create transfer will return auth_key(also a short random string), salt and auth_key calculate crypt_hash by sha1.
- crypt_hash the hash value for transfer.
- expires_at the expire time of transfer, one hour(default) later after created.
- source_project_id the source project id of share.
- destination_project_id the destination project id of share.
- accepted whether the share has been accepted.
Add a new status awaiting_transfer for shares, which is being transferred.
Check the following before create share transfer:
- check the status of share and share_snapshot and share_replica before create transfer, share and all share_snapshot must be available.
- check whether the share in share group, share cannot be in a share group.
- check whether share has share network, this spec just supports the first scenario (DHSS=False), share without share network will allowed to be transferred.
Check the following before accept transfer:
- check quota of destination project, include project quota, project user quota, project share type quota, per_share_gigabytes, make sure destination project has enough remaining quota.
- if the share type of source share is not public, and the share type access of share type does not include the destination project, it will fail to accept transfer.
For security reasons, a share transfer will be cancelled automatically if it is not accepted one hour(default) after creation. There will be a periodic task that checks for expire transfers every 5 minutes.
Add a new configure item wait_transfer_timeout_seconds, default is 3600 seconds.
“transfer-accept” need to call driver to complete project_id change, such as CephFS driver stores the project_id as metadata. The parent class driver only defines the function, but does not implement it.
When accepting a transfer, users can choose whether need to clear the share access rule. By default, the share access rule will not be cleared.
Add new API for transfer-create, transfer-accept, transfer-delete, transfer- list, transfer-show.
The above API changes will bump a new microversion.

Alternatives

An alternative here is manage/unamanage, a share in Project A can be “unmanaged” by a privileged user and “managed” into another project. However two issues with that approach:
- manage/unmanage is meant to be disruptive - pre-existing access rules will be flushed and the share would be re-exported.
- you’d need to be a privileged user by virtue of the default RBAC because this approach gives away some of teh abstraction of the cloud.

Data model impact

A new table transfers will be added.

REST API impact

Create a share transfer

POST /v2/share-transfers

Request Example:

{
    "transfer":
    {
        "share_id": "da8eb12e-123c-49ea-ae2b-5d42f02fa00e",
        "name": "share transfer"
    }
}

Response Example:

{
    "transfer":
    {
        "auth_key": "qbgcabtdbad29r08",
        "created_at": "2021-12-27T11:29:46.743632",
        "name": "share transfer",
        "share_id": "da8eb12e-123c-49ea-ae2b-5d42f02fa00e",
    }
}

auth_key is necessary in accept share transfer, please keep records properly, since it cannot be retrieved later on.

Accept a share transfer

POST /v2/share-transfers/{transfer_id}/accept

Request Example:

{
    "accept":
    {
        "auth_key": "6461646164641397",
        "clear_access_rules": "false",
    }
}

List share transfers for a project

GET /v2/share-transfers

List share transfers and details

GET /v2/share-transfers/detail

Show share transfer detail

GET /v2/share-transfers/{transfer_id}

Delete share transfer

DELETE /v2/share-transfers/{transfer_id}

Security impact

Each of these transfer APIs will be protected by RBAC.
Users can only transfer shares belonging to their project.
The transfer ID and transfer auth key are expected to be conveyed out of band. The transfer auth key is only returned by the API during creation of a transfer. There will be no way to retrieve the transfer auth key after a transfer has been created.
the choice of cryptographic algorithms to generate an auth_key and the fact that transfer IDs are non-guessable UUIDs will provide additional security to the process.
Since transfers expire in an hour(default), there’s a time bound protection to this critical change of namespaces.

Notifications impact

We must notify to external listeners when a volume transfer has been initiated and when it has been accepted.

Other end user impact

The Manila client, CLI will be extended to support share transfer.

The command of create share transfer will be like:
```
openstack share transfer create <share>
```

The command of accept share transfer will be like:

openstack share transfer accept [--clear-rules] <transfer> <auth_key>

The command of list share transfer will be like:
```
openstack share transfer list
```
The command of show share transfer detail will be like:
```
openstack share transfer show <transfer>
```
The command of delete share transfer will be like:
```
openstack share transfer delete <transfer>
```

Performance Impact

None

Other deployer impact

None

Developer impact

Drivers that use project and user based metadata must implement the new driver interface to complete “transfer_accept”.

Implementation

Assignee(s)

Primary assignee:: haixin <haix09@chinatelecom.cn>

Work Items

Update API.
Update Database
Update Manager.
Update Manila CLI commands.
Update unit and tempest test.
Update related documents.
Update Manila UI.

Dependencies

None

Testing

Add the unit tests
Add the tempest tests

Documentation Impact

The following OpenStack documentations will be updated to reflect this change:

OpenStack User Guide
OpenStack Contributor Guide
OpenStack API Reference

References

Based on cinder transfer strategy. Includes the following:

Metadata for all user facing resources

Tue, 19 Oct 2021 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/manila/+spec/metadata-for-share-resources

Manila stores information regarding user-owned tangible technical assets in its database as resources. These assets include file shares, snapshots, access control rules, export locations, share networks and security services among others. Resources are designed to accommodate distinguishable identifiers geared at machines and humans such as ID, Name and Description. However these identifiers may be insufficient or inflexible to end users’ needs such as being able to tag resources to a purpose, or attach labels that drive some automation. This specification proposes a design allowing for user modifiable metadata on all user facing resources.

Problem description

End users interact with manila API to manage their shared file system storage resources. These resources include:

Shares

Share Export Locations

Share Access Rules

Share Instances

Share Instance Export Locations

Share Replicas

Share Replica Export Locations

Share Snapshots

Share Groups

Share Group Snapshots

Security Services

Share Networks

Share Network Subnets

All of these resources are identifiable by ID. Some of them even allow setting free form text as Name and/or Description. Free form text is usually hard to construct, breakdown or query, so these fields are mostly useful to humans rather than machines. Many times, users want to be able to store much more auxiliary contextual information.

One way they do that today with shares and access rules is with the help of “metadata”. Metadata is a term used to describe auxiliary information associated with some data. For shares and access rules, users today use metadata as flexible key=value pairs to attach useful optional/additional information. This sort of interaction isn’t possible with the other resources enumerated above.

Metadata isn’t just helpful for user interactions. It could also provide additional representational space for manila itself. For example, the service currently uses database objects to represent export location metadata to relay helpful hints regarding the nature of specific exports to end users, such as the preference and accessibility. This metadata is owned by the service, and end users cannot modify this metadata.

Use Cases

Users may want to use metadata to store contextual information such as:

the purpose of the resource, e.g.: “usedfor=fileserver”
details regarding the provisioner, e.g.: “createdby=manila-csi”
grouping information, e.g.: “department=physics”
additional chronological information, e.g.: “last_audited=2020-12-24T00:22:29”
mount point tag for clients, e.g.: “fstag=website”

Metadata interactions are expected to be uniform and consistent across resources, which makes it easier for building automation on top of these APIs.

Proposed change

A new metadata controller mixin will be created in manila’s API modules that can be inherited by all the user facing API controllers. This mixin will implement API handler code to:

get resource metadata (retrieve either all of it, or specific item)
set/unset all resource metadata (create, update or delete metadata items as key=value pairs)
set a single resource metadata item (update a specific key=value pair)
unset resource metadata item (delete a specific key=value pair)

To distinguish and protect service owned or admin only metadata, there will be an ignore_keys parameter and ignored keys dictionary to prevent these metadata items from being manipulated by the end user.

Alternatives

Alternatives to key-value metadata include Name and Description fields. Resources that don’t currently support name and/or description can be updated to include these fields. This alternative suffers from the inflexibility problems described above and doesn’t cover all the use cases.

We could also just add metadata controllers one at a time, based on user feedback instead of adding metadata controllers for each user facing resource. This also allows us to test each metadata controller in isolation and build incrementally. This is pretty non-controversial, however, each resource API update will have to be made in a new microversion, and doing things one at a time instead of using the inheritence pattern to subsume common code will be costlier. Testing effort is significant, and unavoidable.

Instead of distinguishing “user” owned and “service” owned metadata, we could allow all metadata to be modifiable by end users - even data that’s created by the service. Doing so would simplify user interactions, however, it may harm the service and cause misbehavior.

Another option to including a user_modifiable boolean field to metadata is to introduce granular RBAC. However, this may produce too many RBAC rules that may be unnecessary. If this turns out to be a use case, it is possible to easily build on top of the proposed design.

Data model impact

To store resource metadata, new tables and corresponding ORM models are necessary. These tables will be created during a database upgrade, and nullified and destroyed during a database downgrade.

Existing metadata tables for Shares (“share_metadata”), Export Locations (“share_instance_export_locations_metadata”) will be modified to include a string attribute called deleted. The default value for this attribute is set to False.

Existing metadata tables will also be modified to replace the datatype of the “id” field to a string of length 36 for UUIDs. This will be done to avoid integer overflows and provide scalability.

A general ORM schema for a metadata table will be as follows (where ‘Resource’ is a stand-in for the real resource name):

class ResourceMetadata(BASE, ManilaBase):
  """Represents a metadata key/value pair for a Resource."""
  __tablename__ = 'resource_metadata'
  id = Column(String(36), primary_key=True)
  key = Column(String(255), nullable=False)
  value = Column(String(1023), nullable=False)
  resource_id = Column(String(36), ForeignKey('resources.id'), nullable=False)
  deleted = Column(String(36), default='False')
  resource = orm.relationship(Resource, backref="resource_metadata",
                              foreign_keys=resource_id,
                              primaryjoin='and_('
                              'ResourceMetadata.resource_id == Resource.id,'
                              'ResourceMetadata.deleted == "False)')

Metadata items are not soft deleted when they are unset by the service or by end users. The metadata table is not loaded alongside the resource unless the resource has been queried with metadata, or a detailed view of the resource has been requested.

REST API impact

New API endpoints will be created to index metadata, show metadata item, create metadata, update metadata item, update_all metadata (delete all existing metadata and update with requested metadata), and delete metadata item for each resource. The general structure of these APIs is as follows:

Index Metadata

Retrieve all metadata key=value pairs as JSON:

GET /v2/{resource}/metadata

Sample request body: null
Success Codes: 200
Default API policy role: Project Reader
Error Codes: 401 (Unauthorized), 403 (Policy Not Authorized), 404 (Invalid resource)

Sample response body:

{
   "metadata": {
       "project": "my_app",
       "aim": "doc"
   }
}

Show specific metadata item

Retrieve a single metadata key=value pair:

GET /v2/{resource}/metadata/{key}

Sample request body: null
Success Codes: 200
Default API policy role: Project Reader
Error Codes: 401 (Unauthorized), 403 (Policy Not Authorized), 404 (Invalid resource)

Sample response body:

{
   "metadata": {
       "project": "my_app",
   }
}

Update all metadata

Replace all metadata with the updated set, can also be used to delete all metadata:

PUT /v2/{resource}/metadata

Sample request body:

{
    "metadata": {
       "aim": "changed_doc",
       "project": "my_app",
       "new_metadata_key": "new_information"
    }
}

Success Codes: 200
Default API policy role: Project Member
Error Codes: 401 (Unauthorized), 403 (Policy Not Authorized), 404 (Invalid resource), 400 (Malformed request)

Sample response body:

{
   "metadata": {
      "aim": "changed_doc",
      "project": "my_app",
      "new_metadata_key": "new_information"
   }
}

Note: Metadata keys that are part of the admin-only dictionary will not be deleted and updated if the user requesting this is not an system or project admin.

Update specific metadata item

Update a specific metadata item, leaving the rest unmodified:

POST /v2/{resource}/metadata/{key}

Sample request body:

{
   "metadata": {
      "aim": "updated_doc",
   }
}

Success Codes: 200
Default API policy role: Project Member
Error Codes: 401 (Unauthorized), 403 (Policy Not Authorized), 404 (Invalid resource), 400 (Malformed request)

Sample response body:

{
   "metadata": {
      "aim": "updated_doc",
      "project": "my_app",
      "new_metadata_key": "new_information"
   }
}

Important

Currently, the POST /v2/{share}/metadata API currently expects a meta object. However, the other metadata APIs expect a metadata object. For the sake of consistency, this error will be fixed in a new API microversion.

Delete specific metadata item

Hard delete a single metadata key=value pair:

DELETE /v2/{resource}/metadata/{key}

Sample request body: null
Success Codes: 200
Default API policy role: Project Member
Error Codes: 401 (Unauthorized), 403 (Policy Not Authorized), 404 (Invalid resource)
Sample response body: null

Query resources by metadata items

URL encoding can be performed by the client:

GET /v2/{resource}?metadata=%7B%27foo%27%3A%27bar%27%2C%27clemson%27%3A%27tigers%27%7D

or the request can be made in a decoded format as well:

GET /v2/{resource}?metadata={'foo':'bar','clemson':'tigers'}

Driver impact

None. Metadata manipulation is directly performed on the manila database and shared file system back end drivers are not invoked during the creation, modification or deletion of resource metadata.

Security impact

It is advised that metadata operations are rate limited to prevent bad actors or automation from adding a large number of metadata items to a resource.

Notifications impact

None

Other end user impact

Python-manilaclient SDK will include support for the new APIs and we’ll ensure that there are corresponding CLI commands in the new OSC plugin shell. Manila UI’s support for share, export location and access rule metadata is limited. This specification doesn’t seek to address all the UI gaps; but all effort will be made to close the feature parity between the CLI utilities and the UI. Eventually users will be able to perform all metadata interactions via the UI as well.

Performance Impact

API performance is bound to suffer when resource queries include metadata items. Since we’ll be providing metadata along with detail retrievals of resources, the performance of those APIs will also be affected negatively because of the new database joins that will be necessary. Over time, as the number of shares and the metadata tables grow, performance degradation can be severe. This impact will be documented; as best practice, it is recommended that a resource have no more than a handful of metadata items.

Other deployer impact

New APIs introduced may require tweaking of policy files if the default RBAC policy is not acceptable.

Developer impact

When adding any new user facing metadata, the metadata mixin controller can be inherited and extended by developers.

Implementation

Assignee(s)

Primary assignee:: ashrod98 <ashrod98@gmail.com>
Other contributors:: gouthamr

Work Items

Add database migration to convert the id field of share, export location and access rule metadata to a string from integer and populate the field with UUIDs
Add database migrations to introduce the “deleted” field to share, export location and access rule metadata tables.
Add database migrations to create new metadata tables for all other resources
Add MetadataControllerMixin, inherit and extend in all resources and bump up the API microversion.
Add unit and integration tests
Add support for metadata APIs in manilaclient SDK, and OSC CLI and SDK
Add support for metadata interactions in the UI
Add documentation

A further enhancement to this API would be to provide an interface for administrators to create admin-only metadata values. Currently, admin-only metadata values are delinated in a dictionary in manila/manila/common/constants.py. Such a fixated list is sufficient for the implementation of backend filtering for scheduling according to share affinities. To implement this customization, we could revisit adjusting the data-model to include an admin-only boolean value identifying which keys are adjustable by admins or non-admins. Or we could provide some configuration option in manila/manila/data/. Such an enhancement requires further thought, and can be implemented at a later point.

Dependencies

Not a direct dependency, but this API change incorporates the metadata changes necessary to implement this spec: Affinity and anti-affinity scheduler filter

Testing

Extensive unit tests will be written to test the API and database methods being added. A database “walk migrations” test will be added for all the database changes. Tempest tests will be added to cover the new metadata operations across resources.

Documentation Impact

API reference
End user guide
Release notes

References

Spec Lite: Add human readable export location support

Wed, 23 Jun 2021 00:00:00

problem:

Export locations are usually too difficult to memorize. Currently, there is no way to determine the export location before the share is created, so users wait until the share creation request gets completed, and then they check the export locations to mount the share. The generated export locations are often not human readable and it is hard to memorize and control them.

solution:

We could introduce a new field for shares where users would be able to specify a custom export location in the share creation. It should be easier for them to memorize the export location of the share, in case there is need to mount the share again. If this field is specified during the creation request, the backends that support such functionality would be able to create multiple export locations for a given share, and users would be able to mount the shares either using the human readable export location, or other if they exist. It’s possible backends that implement this feature may only be able to provide one export path. The new field will be called mount_point_name and it will be added to the shares model. This field will not accept special characters other than underscores. If special characters other than underscores are provided, the manila api service is going to raise HTTP BadRequest, warning that such characters are not allowed. An export location must be unique in a share server. So, when a request to create a share is received, if a mount_point_name was provided, the project name will be prepended to it. The number of characters provided by the user added to the number of characters in the project name can not exceed 255 characters, otherwise the request will be denied. It is reasonable to think that the project name can be easily remembered by the users, so it is still a better option compared to an id, and we can be sure that appending the project name to the custom mount point name will not drift apart from this proposal main goal. So the manila share service will look up for duplicate mount_point_name values and if it finds any, the creation will fail. It is possible that two projects in different domains have the same name, and users coincidentally set the same mount_point_name while creating a share. In such cases, the share will not be created and its status will be set to error. A user message will be created for both scenarios. A new back end capability called human_readable_export_location_support will be added and drivers that support such capability should report it as True. Administrators will need to create share types with such extra spec set to True. As the manila share API will perform validations using this extra spec, it must be always tenant-visible. By having this extra-spec, the scheduler can also filter out backends that do not support such functionality. In case of a share migration, if a new share type is provided, and it has human_readable_export_location_support=True, the migration will fail in the scheduler if the chosen destination backend doesn’t support it. If a new share type was specified and it differently from the former does not support custom names for export locations, the migration will succeed, the custom export location won’t be created and the mount_point_name field value will be set to None. Administrators will be able to specify a custom mount point name to be configured in the migrated share through a new parameter called --new-mount-point-name, in the migration-start command. This will help administrators to avoid possible failures caused by duplicated custom export locations in the migration. So in a share creation scenario, the users will be able to create shares like:

$ manila create nfs 1 --name share_name --mount-point-name custom_export_path

And users will be able to mount shares using the custom export location, as in this example:

$ sudo mount -t nfs 10.1.0.2:/project_name_custom_export_path /mnt/my_share

impacts:

REST API Impact.
- A microversion bump to the share API.
- The API will accept the field mount_point_name.
Documentation Impact
- User guide
- API Reference
- Contributor guide
Database Impact
- A new field called mount_point_name will be added to the shares model.
Python-manilaclient and OSClient impact
- The share create command will be modified to accept the --mount-point-name parameter in the shares creation.
- The migration-start command will be modified to accept the --new-mount-point-name parameter.

This implementation is not supposed to impact performance on any aspect.

alternative:

As an alternative, drivers could reuse the name and the description of shares and generate human readable export locations, but names can be duplicated and backends may fail due to that.

timeline:

Include in Xena release.

link:

https://blueprints.launchpad.net/manila/+spec/human-readable-export-locations

assignee:

carloss

Support Share Recycle Bin

Sat, 08 May 2021 00:00:00

https://blueprints.launchpad.net/manila/+spec/manila-share-support-Recycle-Bin

Manila doesn’t support share Recycle Bin. This BP is to add support for share Recycle Bin. The end user can soft delete share to Recycle Bin, and can restore it.

Problem Description

If users want to remove a share from Manila, it is possible to delete a given share. Or if they want the share only not being managed by Manila anymore, they can unmanage it and the share would still exist in the share backend. If users later wants to restore the share, they must record export_location_path, host, share_proto and so on, and end users do not have permission to manage a share. If the user never uses the share again, this share data will remain permanently on the back-end storage, becoming garbage data. So users need a way to temporarily delete and easily restore.

Use Cases

The user wants to soft delete shares to Recycle Bin.
The user wants to list shares in Recycle Bin.
The user wants to restore shares from Recycle Bin.
The user wants to completely delete share from Recycle Bin.

Proposed Change

The following are the changes to be made:

A new column is_soft_deleted will be added in shares table to identify shares placed in the Recycle Bin.
A new column scheduled_to_be_deleted_at will be added in shares table, the shares in Recycle Bin will be automatically and completely deleted once the expire time reached. Shares not in Recycle Bin scheduled_to_be_deleted_at will be None.
A new property is_soft_deleted will be added in ShareInstance model. This property will inherit the parent share’s value.
Add new configuration item soft_deleted_share_hold_time, which means the maximum time cloud administrators want to keep a share in the recycle bin. The default value is 604800 seconds (7 days).
A new share action will be added for soft delete share. Once the share has been deleted to the Recycle Bin, will set is_soft_deleted of share to be True. and calculate the scheduled_to_be_deleted_at of the share. scheduled_to_be_deleted_at = now_time + soft_deleted_share_hold_time.
A new request parameter is_soft_deleted will be added to the original list shares API. is_soft_deleted default is False, if set True, it will only list shares that were soft deleted.
the project quota will remain allocated after the share is soft deleted, and it will not be released until the shares are deleted from the Recycle Bin.
A new periodic task will be added to the share manager layer to check if the shares in the Recycle Bin have reached their scheduled_to_be_deleted_at. If the scheduled_to_be_deleted_at gets reached, the periodic task will delete the share automatically.
Add new API to restore share from Recycle Bin.
List share instances will filter the share’s is_soft_deleted is False.
The above API changes will bump a new microversion.

Alternatives

None

Data model impact

Two new columns is_soft_deleted and scheduled_to_be_deleted_at will be added to model of shares.
A new property is_soft_deleted will be added in ShareInstance model. This property will inherit the parent share’s value.

REST API impact

Soft delete a share to Recycle Bin

POST /v2/shares/{share_id}/action

{“soft_delete”: null}

Preconditions

(1)Share status must be available, error or inactive
(2)You cannot soft delete share already in Recycle Bin.
(3)You cannot already have a snapshot of the share.
(4)You cannot already have a group snapshot of the share.
(5)You cannot already have a replica of the share.
(6)You cannot soft delete a share that doesn’t belong to your project.

If the provided share_id doesn’t exist, the API will respond with 404 Not Found. If the operation can’t be performed due to not meet the (1)(3)(4) constraints, the API will respond with 400 Bad Request. If the operation can’t be performed due to not meet the (2)(5) constraints, the API will respond with 409 Conflict Request. If the operation can’t be performed due to not meet the (6) constraints, the API will respond with 403 forbidden.

List shares in Recycle Bin

GET /v2/shares?is_soft_deleted=true

List shares in Recycle Bin with details

GET /v2/shares/detail?is_soft_deleted=true

Delete share completely from Recycle Bin same as delete share not in Recycle Bin

DELETE /v2/shares/{share_id}

If the provided share_id doesn’t exist, the API will respond with 404 Not Found.

Restore share from Recycle Bin

POST /v2/shares/{share_id}/restore

{‘restore’: null}

If the provided share_id doesn’t exist, the API will respond with 404 Not Found. If the share not exist in Recycle Bin, the API will return success directly.

Security impact

None

Notifications impact

None.

Other end user impact

The Manila client, CLI will be extended to support share Recycle Bin.

The command of soft delete share will be like:
```
manila soft-delete <share_id>
```
The command of list shares in Recycle Bin, the supported parameters are the same as the Manila list, it will be like:
```
manila list --soft-deleted
```
The command of restore share from Recycle Bin will be like:
```
manila restore <share_id>
```

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: haixin<haixin@inspur.com>

Work Items

Update API.
Update Manager.
Update Manila CLI commands.
Update unit and tempest test.
Update related documents.
Update Manila UI.

Dependencies

None

Testing

Add the unit tests
Add the tempest tests

Documentation Impact

The following OpenStack documentations will be updated to reflect this change:

Openstack Admin Guide
OpenStack User Guide
OpenStack API Reference

References

None

Affinity and anti-affinity scheduler filter

Thu, 11 Feb 2021 00:00:00

https://blueprints.launchpad.net/manila/+spec/affinity-antiaffinity-filter

Copy of cinder feature https://blueprints.launchpad.net/cinder/+spec/affinity-antiaffinity-filter

To add scheduler filter to manila that allows scheduler to make placement decision based on affinity relationship between existing shares and new share (the one being scheduled). The affinity relationship here means the location of shares (‘host’ of share).

Problem description

Manila has done a good job hiding the details of storage back ends from end users by using share types. However there are use cases where users who build their application on top of shares would like to be able to ‘choose’ where a share be created on. How can manila provide such capability without hurting the simplicity we have been keeping? Affinity/anti-affinity is one of the flexibility we can provide without exposing back end details.

The term affinity/anti-affinity here is to describe the relationship between two sets of shares in terms of location. To limit the scope, we describe one share has affinity with the other one only when they reside in the same share back end; on the contrary, ‘anti-affinity’ relation between two sets of shares simply implies they are on different manila back ends (hosts).

This affinity/anti-affinity filter filters manila back end based on hint specified by end user. The hint expresses the affinity or anti-affinity relation between new shares and existing share(s). This allows end users to provide hints like ‘please put this share to a place that is different from where share-XYZ resides in’.

Use Cases

DB team builds MySQL master onto one share, they’d prefer to put new shares for slave DBs to different back ends from where the master DB resides in, for the sake of high availability.

Proposed change

Add two new filters to Manila - AffinityFilter and AntiAffinityFilter. These two filters will look at the scheduler hint provided by end users and filter back ends by checking the ‘host’ of old and new shares see if a back end meets the requirement (being on the same back end as existing share or not being on the same back end(s) as existing share(s)). Shares in different pools located on the same host are still considered as based on the same host.

The scheduler_hints (same_host, different_host) have to be saved as a share metadata as soon as new Share record is created in the database. The shares referenced in those hints will receive the metadata update too, for instance:

There are three shares exist with: UUID1, UUID2, UUID3.
User provisions a new share with UUID4 and requests it with a hint to have different_host than UUID1, UUID2 and UUID3 shares.
Share UUID4 gets this hint stored with all other three shares UUIDs.
The metadata of each of three shares is also updated to reflect that they have anti-affinity to a new share with UUID4.

The respective share metadata is also automatically updated on share deletion in the similar fashion.

This extra metadata should be marked as only “admin_modifiable” to prevent non-admin users from accidential deletion or modification of the values.

Hint infomation is needed to allow user to understand how were the shares scheduled and also to be able to respect the hints in case share is migrated during its lifetime.

Alternatives

There had been one proposal to allow admin user to directly specify the back end for new shares. It doesn’t really provide similar functionality as affinity filter because it was admin only and it itself has a few drawbacks (security concern, for example).

Affinity is also possible with share groups. However, there are some limitations of share groups to the use case proposed:

1) Affinity/Anti-affinity is only required at the time of provisioning the resource, and not at any other point during its lifetime. Shares created within share groups today are tied to the share group throughout their lifetime. 2) Current feature limitations with share groups include the inability to migrate or replicate shares within a share group. 3) While achieving affinity is possible with share groups,anti-affinity is not.

There is an option to have a soft OR semantics for both filters instead of hard AND. That means, the share creation should succeed if at least 1 condition has been met. The drawback is that there won’t be a clear way to present this information to the user since the share creation is async. (When a user “fires and forgets” a creation request with hints where not all conditions are met).

Alternative to storing scheduler_hints in share metadata can be extension of Share DB model for saving the hints there. This approach causes more modifications.

Data model impact

None. The hints will be saved as additional share metadata.

REST API impact

Parameter ‘scheduler_hints’ added to share create, for example:

`{
    "share": {
        "scheduler_hints": {
            "different_host": [
                "share_uuid_1",
                "share_uuid_2"
            ],
            "same_host": ["share_uuid_3"]
        }
    }
}`

In this particular example, the share should be scheduled to the same host where ‘share_uuid_3’ is located, yet only if both ‘share_uuid_1’ and ‘share_uuid_2’ are not located on that host.

All uuid-s given will be validated if exist and share creation will fail if at least one share is not existing or uuid malformed.

The semantics for both filters is hard AND, meaning that the share won’t get created, unless all placement conditions are satisfied.

If the desired scheduling combination is not possible (e.g. affinity with two shares located on two different hosts) - the share creation should fail. The same applies for anti-affinity filter - if there are only two hosts and each has a share with a uuid that was given in the ‘different_host’ hint for a new, third share - creation will fail.

The scheduler hints should also be respected if a share is migrated between hosts, however there should be an option to override the hints manually and force migration even if affinity/anti-affinity conditions won’t be met after the migration (e.g add ‘force: true’ option).

Microversion of the API is incremented.

Security impact

Although this change involves using or parsing user-provided - scheduler hints. This doesn’t put Manila in any more danger as it is now.

Notifications impact

None

Other end user impact

None

Performance Impact

New filters would query DB once per request, it only adds slightly latency to the system and the latency has nothing to do with the size of the system.

The share metadata update might take extra time, especially when many shares are specified in the hints. Yet share creation or deletion is an async process and this impact should not be noticeable by the end user.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Dmitry Galkin (galkindmitrii in Gerrit) Kiran Pawar (kpdev in Gerrit)

Work Items

Filter implementation
Add scheduler hints parameter
Add hints argument for python-manilaclient
Add hints support in manila-ui

Dependencies

None

Testing

Test against AffinityFilter (Same host):

Create one share A;
Create another share B with uuid of A and ‘same_host’ as hint;
Checks if B is created on same back end as A;

Test against AntiAffinityFilter (Different host):

Create one share A;
Create another share C with uuid of A and ‘different_host’ as hint;
Checks if C is created on different back end as A;

Documentation Impact

Need to document the usage of new filters.

References

Nova has been offering similar feature called SameHostFilter and DifferentHostFilter since Diablo.

https://github.com/openstack/nova/blob/master/nova/scheduler/filters/affinity_filter.py

Cinder has been offering similar feature called AffinityFilter and AntiAffinityFilter since Juno.

https://specs.openstack.org/openstack/cinder-specs/specs/juno/affinity-antiaffinity-filter.html

Support Manila APIs in the OpenStackSDK

Fri, 25 Dec 2020 00:00:00

Blueprint link:

https://tree.taiga.io/project/ashrod98-openstacksdk-manila-support/kanban

Problem Description

Manila is an open source OpenStack Shared File System service that is a software that exists within the OpenStack cloud. It is a collection of microservices and other components that provide self-service management of elastic file system storage infrastructure that can allow you to work with and provision storage via 30+ storage technologies over file system protocols. It is unique in how it is able to add RESTful semantics to consuming shared storage in a reliable and scalable manner. We aim to improve the common user experience by adding Manila support to the OpenStackSDK, and to reuse OpenStack Identity, Logging, Configuration integration modules without duplicating them into a separate SDK. This project is part of the resolution defined by the TC Stance on OpenStackSDK and the OpenStackClient.

Use Cases

OpenStack Client will be affected as current manila commands are being implemented using python-manilaclient SDK. OpenStack’s Ansible Collections will be affected as it directly uses of the OpenStack SDK. Developers that use automation tooling will also benefit from incorporating Manila into OpenStack SDK.

Proposed Changes

Resources and methods are listed in order of priority

The following tasks will be a part of the Wallaby Cycle

Priority 1

Shares
Share Export Locations
Share networks
Share network subnets
Share types

Priority 2

Share actions - grant access, revoke access, list access rules, change share size, revert to snapshot
Security services
User messages
Share access rules

The following will be part of the cycle(s) following Wallaby

Priority 3

Share metadata
Share snapshots - List share snapshots, Show share snapshot details, Create share snapshot, * Update share snapshot, Delete share snapshot
Scheduler stats
Availability zones
Quota sets
Quota class sets

Priority 4

Share replicas
Share replica export locations
Share servers
Services
Share access rule metadata
Share groups
Share group types
Share group snapshots

Priority 5

Share Actions - Reset share state, force delete share, unmanage share
Share Snapshot - manage share snapshot, unmanage share snapshot, reset share snapshot state, force delete share snapshot
Share Replicas - resync share replica, reset status of share replica, reset state of share replica, force delete share replica
Share Servers - reset status of share server

Priority 6

Share snapshot instances
Share instances
Share instance export location
Experimental api/share migration

Alternatives

Continue using python-manilaclient SDK. Continue maintenance for python-manilaclient SDK. Do not implement support for manila in OpenStackSDK.

These alternative are not inline with TC resolution: https://review.opendev.org/c/openstack/governance/+/759904

Data model impact

No impact on the data model.

Security impact

No security impact as authentication is already provided by OpenStackCloud before OpenStackSDK is used.

Notifications impact

No notifications impact.

Other end user impact

Improves user experience by providing user the ability to interact with Manila API through OpenStackSDK.

Performance Impact

No current performance impact. Future impact may include mitigating the use of plug-in clients.

Other deployer impact

No other deployer impact.

Developer impact

Implementing Manila support may make it easier for developers to code scripts using the SDK. Manila will not stand out within the OpenStackSDK as it is part of a cohesive design.

Implementation

Assignee(s)

Primary assignee: * Ashley Rodriguez <ashrod98> * Nicole Chen <arkaruki> * Mark Tony

Other contributors:

Work Items

Implement proposed changes
Write unit tests
Write documentation
Write functional tests
Write end user guide

Dependencies

No dependencies at the moment of writing for this project.

Testing

Unit tests will be required for each resource. Functional tests will also be written.

Documentation Impact

For each resource coded, we will concurrently write its documentation. We will also write an End User Guide.

References

End User Guide OSSDK: https://docs.openstack.org/openstacksdk/latest/user/guides
Manila API Ref: https://docs.openstack.org/api-ref/shared-file-system/
TC Stance: https://review.opendev.org/c/openstack/governance/+/759904

The share size can be limited by share type

Fri, 27 Nov 2020 00:00:00

https://blueprints.launchpad.net/manila/+spec/share-size-limited-by-share-type

Add support for limit the size of share through the share type, the share created by the user shall not be greater than the maximum value set in the share type and shall not be less than the minimum value set. Of course, depending on the usage scenario, only a maximum or a minimum can be set.

Problem description

In some usage scenarios, we do not want users to create too large share size, then affect the number of shares that can be created, maybe limited by project quota or backend storage, or in some usage scenarios, creating a share that is too small makes no sense. So administrators desperately need a way to limit the size of share created by users.

Use Cases

In scenarios where storage backend capacity is small, or project quota is limited and the number of users is high, In order to enable more users to create their own share, Administrators need to limit the maximum size of each share. In other scenarios, if the share is too small to ensure the normal operation of the business, the administrator needs to set the minimum value of each share. Administrator can also set the maximum and minimum values of share according to actual needs.

Proposed change

The following changes are proposed:

Two new extra_specs keys of share type are added for support set the minimum share size and maximum share size.
- provisioning:min_share_size Set the minimum size of share by the share type.
- provisioning:max_share_size Set the maximum size of share by the share type.
The value of these keys is a positive integer, the unit defaults to GB.
These two extra specifications will be visible to end users. If they’re not set, it is assumed that there are no share size limits enforced by the share type, but limits may be enforced by the service and the user’s quotas.

Alternatives

None

Data model impact

None

REST API impact

The size restrictions will be checked at the API level as part of share creation, extend, shrink or migration.

Security impact

None

Notifications impact

None.

Other end user impact

The size of the share created by the end user will be limited by the share type if set specified keys in extra_specs in share type. Operations including extend, shrink, and migration operations are also subject to this restriction.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: haixin<haixin@inspur.com>

Work Items

Add size check in API level
Add related unit and functional tests
Add docs update

Dependencies

None

Testing

Unit test to test whether these keys restrictions are in effect.
Tempest test whether set keys work correctly from API perspective.

Documentation Impact

The manila API documentation will need to be updated to reflect extra_specs of share type support two more keys.

References

None

Security Service Updates for In Use Share Networks

Fri, 06 Nov 2020 00:00:00

https://blueprints.launchpad.net/manila/+spec/improve-security-service-update

https://blueprints.launchpad.net/manila/+spec/add-security-service-in-use-share-networks

Manila security service is an entity that stores client configuration used for authentication and authorization. It abstracts a set of configuration data that can be used by back ends that support it, to configure a server that can join specific authentication domains.

Manila supports different types of security services such as LDAP, Kerberos and Microsoft Active Directory [1]. Users can create, update, view and delete security services, however, in order to apply its configuration, manila needs to maintain an association between security services and share networks. Although, there is no way of providing such association or update any of its parameters after a share network starts being used by a share server.

This spec proposes an improvement on both security service update and share network association with security services, that will allow users update their share networks to replace or to add new security services, which can reflect on modifications in all related share servers.

Problem Description

Security service has configurations that might change while it’s still associated to share servers, leaving the latter outdated and possibly with problems to authenticate within domains. Furthermore, users won’t be able to grant access to shares that live on these share servers, and Manila doesn’t provide an easy way of updating such configurations, since the current implementation only supports update of name and description attributes [2] since they do not affect share server’s configuration.

Likewise, share network association with security services can’t be changed after a share server has been provisioned, and new authentication server configurations can’t be added to the already deployed share servers. In this scenario, users can only create new share networks that will be used to create new share servers.

Use Cases

Security service attributes such as dns and password might change due to network changes or password update policies and hence affect one or more share servers. Users must be able to modify these attributes in all affected security services which must reflect such configuration change in all associated share servers. A security service is qualified to be updated only if all its associated share servers have support of such operation, otherwise it must be denied.

Moreover, new security services might need to be associated with in use share networks, in order to bring more flexibility to users that want to create new shares using different authentication methods. Users must be able to include new security service associations to their share networks, which must reflect on security service updates in all associated share servers. A new security service association is qualified to be configured only if all affected share servers support such capability.

This specification focuses on both scenarios where share networks are being used by share servers and an update or a new association of security service is requested by the user, which will end up with one or more share servers being updated in the back end storages.

Proposed Solution

To solve these use cases, the share network entity will need be improved to support updates and association of new security services, even when it is already in use. A new status field will be added to represent the current state of a share network. While under modification, the share network will stay in a maintenance mode, meaning that back end configurations are being changed. The update will finish when all affected resources finish their respective configurations. Users will be able to check the current status of a share network and will be blocked of requesting new operations that can be affected by the current modification in progress.

To have these new operations working, the following changes are being proposed:

Share network will have a new status attribute;
Share network API will be updated to allow association and update of security services for in use share networks. The API will also be improved to accept share network reset-status operation.
Update share server model to include a new capability, for supporting security service updates.
Add a new share server state, indicating that it’s under network modification.
New share network states will be added to cover all scenarios.
Add a new driver interface for share server update;
Add a new back end capability to identify drivers that support such functionality;
A new share network property will be added to indicate if it supports security service updates based on its share servers’ capabilities.

API Changes

When receiving a request for security service update or association, the API will check if there are share servers associated with the share network. If the list of share servers is not empty, the API will need to check the new share network’s property, security_service_update_support, to see if the request can be completed or not and fail earlier if doesn’t support it.

A new API will be create to handle security service update, which will allow the replacement of an already configured security service by another one, that should be of the same type.

The new share network’s status will need to be validated in many other resources’ operations, which end up on backend changes. These operations should wait until the associated share network become available again, in order to proceed with the new modifications.

Scheduler Capability

Backends that support this functionality will be able to report the security_service_update_support capability as True to the scheduler, which can further be used to select backend pools that support it.

Manila Manage

By adding a new capability to the share server model, it’s important to consider that existing share servers will need to update this field in the future, based on driver’s support, to have this functionality enabled. This will be achieved by providing a new management command for manila-manage tool that will let administrators update their share servers accordingly.

Alternatives

Alternatively, the security service resource could handle the update operation and trigger back end modification for all share servers associated with the affected share networks. This alternative can lead to scenarios with lots of back end modification at once, and as possible result, many failed share servers if the new parameters or associations are invalid.

Impacts

Data model impact

A new security_service_update_support capability field will be added to manila.db.sqlalchemy.models.ShareServer indicating if the driver and the back end where this share server resides support the new security service update operations. In database migration upgrade, the new column will be added with a default value set to False, meaning that all share servers already deployed won’t be able to update their security service configuration even if the driver supports it. In database migration downgrade the column will be dropped.

A new status field will be added to the manila.db.sqlalchemy.models.ShareNetwork to hold new states that will assist on different share network operations. At this moment, only three status could be assigned to share networks: active, error and network_change. The share network is considered healthy and is available to be used only if its status is active. The status network_change represents a share network that is under modification and can’t be changed or used until it becomes active again. For specific failure scenarios, that can’t be recovered without administrator intervention, the share network will receive an error status.

A new security_service_update_support property will be added to manila.db.sqlalchemy.models.ShareNetwork to indicate whether a share network supports or not the new security service update operations. This property will inherited its value from all current associated share servers. If all associated share servers support security_service_update_support, the share network property will be set to True, otherwise it will be set to False.

REST API impact

Both share server and share network view will include a new response parameter, the security_service_update_support that will indicate if the share server (or share network) is capable of updating security service configuration.
Share network view will include a new status parameter that will indicate its current status.
All share network dependant operations will be blocked in the API for share networks with status different from active, to avoid conflicting backend configurations.
Semantic changes are expected in the share network API. Associating new security services for in use share networks will require that all share servers affected by the change support such operation, or the API will will respond with 403 Forbidden. If the destination share service is unavailable, the API will respond with 409 Conflict.
New share network APIs will be added:

share-network-security-service-update

Updates an existing security service association:

POST /v2/{project_id}/share-networks/{share_network_id}/action

Body:

{
  "update_security_service": {
    "current_service_id": "bab0debd-fa50-4ade-80c5-ce85e2fc2614",
    "new_service_id": "1ec35b2b-5d0c-4c46-a6ca-71f9eab49ce2"
  }
}

Response:

Code: 200 OK

{
  "name": "my_network",
  "id": "620a1050-1711-4961-908a-bd6f7c0b1d00",
  "project_id": "f227b9e2cbdc40e78ed761e5e22c5fb4",
  "description": "This is my share network",
  "created_at": "2020-11-27T11:26:10.000000"
  "updated_at": null,
  "status": "network_change",
  "security_service_update_support": true,
  "share_network_subnets": [
    {
      "created_at": "2020-11-27T11:26:10.000000",
      "id": "aa7a1269-703b-4832-a3df-17ed954c276c",
      "availability_zone": null,
      "segmentation_id": null,
      "neutron_subnet_id": "12c1490a-e82c-4f5e-bcb1-fb267a58cf10",
      "updated_at": null,
      "neutron_net_id": "998b42ee-2cee-4d36-8b95-67b5ca1f2109",
      "ip_version": null,
      "cidr": null,
      "network_type": null,
      "gateway": null,
      "mtu": null
    }
  ]
}

If the provided new_service_id doesn’t have the same type as the current one, or one of the security services’ ID don’t exist, the API will respond with 400 Bad Request. If the provided share-network-id doesn’t exist, the API will respond with 404 Not Found. If at least one of the destinations share services is unavailable, the API will respond with 409 Conflict.

share-network-reset-status:

Reset the status of a share network:
```
POST /v2/{project_id}/share-networks/{share_network_id}/action
```
Body:
```
{
  "reset_status": {
  "status": "active"
  }
}
```
If the provided share-network-id doesn’t exist, the API will respond with 404 Not Found. If the user doesn’t have permission to execute this operation, the API will respond 403 Forbidden.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance impact

None.

Other deployer impact

No new configurations are expected to be added.
The back end capability will help deployers to identify pools that support new security service association.
All existing share servers will have their security_service_update_support set to False, even if the driver supports it. New share servers will have the correspondent capability set according to the back end capability reported by the drivers. Administrators will need to manually update share server’s capability using manila manage commands.

Developer impact

None

Driver impact

There is no impact for drivers that don’t want to support the proposed feature.

A new driver interface will be included to support share server security service update. Both share server and network info will be provided during this call. Drivers that want to support this operation will need to implement this call and report the capability security_service_update_support as True.

Implementation

Assignee(s)

Primary assignee:: dviroel

Work Items

Implement core changes that must include:
- Add share network and share server model attributes;
- Change API behavior when associating new security services with a share network;
- Create new share network APIs to update security service and reset share network status;
- New Share API interface for updating security service information;
- New driver interface will be included to update share server network configuration;
- New back end capability for supporting security service association will be added to share driver.
Implementation in a first party driver
Add new functional test in manila-tempest-plugin
Add new command to manila-manage for share server capability update
Update manila documentation

Dependencies

None.

Testing

Unit test coverage will be added/maintained as per community standards. New tempest tests will be added to cover new security service association scenarios. The container or the dummy driver will be improved to properly configure security services and be used to validate the proposed changes.

Documentation Impact

The following OpenStack documentation will be updated to reflect this change:

Admin Guide: document the premises for having new security service association applied to share servers;
API Reference: include share server new status and new capability field;
Developer Reference: add information about how to implement and add support for this new functionality.

References

[1]: https://docs.openstack.org/manila/latest/admin/shared-file-systems-security-services.html

[2]: https://docs.openstack.org/api-ref/shared-file-system/#update-security-service

Spec Lite: Limit number of allowed replicas per share

Tue, 27 Oct 2020 00:00:00

problem:

Currently, Manila allows the creation of unlimited share replicas of a given share. It is a problem for both DHSS=True and DHSS=False modes. It allows the user to deplete the storage resources and have troubles depending on the configured backend. There is also a problem when it comes to the user visibility side on what backend do or do not support. Some backends do not support more than a limited number of share replicas, and it is not noticeable until the backend asynchronously denies the operation.

solution:

We should let the administrator specify the max number of replicas that a given share is able to have according to the backend capacity. In this way, the API would be able to check the number of existent share replicas during a new replica creation request, and be able to fail faster and synchronously. The proposed solution is to add a new extra spec to the share types called max_replicas_per_share. Then, when creating a share replica, Manila will check the max_replicas_per_share value of the source share’s share type, and will verify the number of existent replicas of the source share. If the number of existent share replicas has already reached the max_replicas_per_share value, the request to create a new replica will be denied. While creating the new extra-spec for the share type the administrator must specify an integer value. A database migration will be created in order to add the new share type extra spec for all existent share types. In the database migration, a default of two replicas will be set due to some backends capacity. All administrators will need to update the share type according to their backend capacity, and will set the extra spec according to the number of replicas that shares created under the given share type are able to have. When creating a new share type, if the user does not specify the new extra spec, the service will automatically set this value to 2 and in case the administrators need to change the value, they will need to update the extra spec. The administrator will also be able to provide a new backend property called supported_replicas_per_share for each backend. The scheduler will be improved in order to filter backends considering this backend property. This solution can be combined with improvements, which consists of defining a project quota for share replicas using the existent quota system, and allow the administrator to set a default value for the max_replicas_per_share extra spec. So at the end, the whole solution would have a way to avoid this bug, alongside with an improvement that will help administrators to manage their resources. The solution for the bug will require the administrator intervention, since the database migration script will update all the share types with a default value, and it may block users while creating new share replicas after migrating the database. It is not currently possible to set per-share replica limits as proposed in this specification, but the manila project quota controls can be applied to share replicas since Ussuri release.

impacts:

REST API Impact.
- When creating a share replica, if the share has already reached the number of allowed share replicas, the API will return 403 Forbidden with the following message: “The share with the id ‘share_id’ already reached the limit of replicas according to its share type”.
- The API will expect an integer value while setting the max_replicas_per_share extra spec, otherwise it will return 400 BadRequest with the following message: “The specified max replicas value ‘non_integer_value’ must be an integer and greater than or equal to 2.”.
Documentation Impact
- Admin guide
- API Reference

timeline:

Include in Ussuri release.

link:

https://blueprints.launchpad.net/manila/+spec/limit-share-replicas-per-share

assignee:

carloss

Spec Lite: Add new share server limits

Mon, 26 Oct 2020 00:00:00

problem:: An administrator is not able to specify how many shares can be created in a given share server, nor the maximum size that a share server can hit. The current behavior allows the system to allocate a bunch of shares in a single share server and it allows the share server to reach a huge size, making it a bit harder to manage.
solution:: We should add a limit for the amount of shares that a share server can hold. Administrators can configure this limit for each backend, and have more control over the size of share servers, which helps them to manage the cloud resources. The proposed solution introduces two new backend properties called max_shares_per_share_server and max_share_server_size, whereby the administrators are able to determine the amount of shares that can be created upon a given share server. Configuring the max_shares_per_share_server property means that a share server which reached the limit of shares will be filtered out of the compatible share servers list in the share manager. For the max_share_server_size backend capability, when a given share server hits the specified amount of gigabytes, the share manager will filter the share server out of the compatible share server list, and a new share server is going to be provided. In this case, not only shares are going to be considered in the final gigabytes amount, but share replicas and snapshots will be considered as well. If one of these backend capabilities is not set in the chosen backend, the share manager will understand that there is no limit for shares or gigabytes in that backend share servers. The share manager provide_share_server_for_share and provide_share_server_for_share_group methods will be modified to also check if there is a configured limit in the corresponding backend session in the Manila configuration file. These new limits can be implemented as mutable, which means an administrator won’t need to restart any service to have the changes applied. If at least one of the properties was specified, the share manager will query the amount of existent share server resources using the database layer for performance purposes, and finally, if needed, request the creation of a new share server to place the received request. When the limits are reached, no exceptions are being raised and the users or admins won’t be impacted by this. The share manager itself will handle with the situation by providing a new share server.
impacts:: This implementation slightly impacts performance only when at least one of the backend capabilities was set, but there is no expectation to increase significantly the share creation time, specially compared to another possible approach where we ask the drivers to calculate the amount of resources in a given share server. Administrators must be aware that when configuring one of these limits more network allocations may be needed.
alternative:: As an alternative we could use share type extra specs to provide driver specific limits for share server shares and max share server size in gigabytes. This approach would work in a project level, but we should consider that it may require the share manager to forward more information to the driver than we do today, in order to avoid driver calls to the storage which would cause a performance deterioration.
timeline:: Include in Victoria release.
link:: https://etherpad.opendev.org/p/victoria-ptg-manila https://review.opendev.org/#/c/510542/ https://blueprints.launchpad.net/manila/+spec/new-share-server-limits
assignee:: carloss

Share Server Migration

Tue, 16 Jun 2020 00:00:00

https://blueprints.launchpad.net/manila/+spec/share-server-migration

Manila supports the deployment model where share drivers are able to handle the creation and the management of share servers as well as shares and their capabilities[1]. By managing different share servers per tenant level, Manila leverages its capability of configuring storage entities and provides more manageability for administrators. As presented in Liberty release, and later improved on Mitaka, Newton and Ocata releases, share migration operation allows administrators to move a share across backends, in a non-disruptive manner, by implementing a 2-phase migration approach. This spec now proposes to extend this migration concept to the share server entity, relying on share drivers that can do this operation in an atomic and efficient way.

Problem description

Administrators might need to handle situations like back end evacuation or rebalancing, and face the problem of migrating lots of shares, one by one, to a specific, and probably common, destination. Even with additional tools or scripts this task can be hard to manage and mainly, to recover from failure states. The lack of a feature that helps administrators to rebalance/evacuate large storage systems is the reason for proposing the following solution.

Use Cases

There are several scenarios where share server migration comes handy and provides benefits to cloud administrators:

Rebalance: move shares to a back end that has more free capacity, freeing up space for other shares to grow over the time;
Optimization: move shares and spare a back end in order to conserve power. Move data closer to the hosts for a better network performance;
Evacuation: evacuate a back end that is too old or that is experiencing failures;
Maintenance: move shares to a newer hardware version/model;
Others: change shares’ configuration like: share network, security services, etc.

Proposed change

As designed for share migration on Newton release[2], the 2-phase migration logic will be also implemented for share servers. By invoking share-server-migration-start, the share server migration can start to copy all data, from source to destination, including all shares, snapshots and shares’ access, if supported by the driver that implements it.

After finishing the 1st phase, administrators can plan and start the 2nd phase, by invoking ‘share-server-migration-complete’ to finish the operation, that usually causes the disruption of share’s access, since share’s export locations might be updated.

It is important to note that when migrating a share server, many share attributes won’t be modified during the process, while share server attributes might change depending on the provided parameters. Administrators will be able to provide a new ‘Share Network’ to associate to the new share server, but won’t be able to change its shares’ attributes like ‘Share Type’ since this is a share level entity and different ‘Share Types’ can live in the same share server.

Share API and Manager Changes

The share API will hold all validations needed before proceeding with driver’s calls and database updates. The API will check if any of the shares within the share server being migrated are in an invalid state or have any dependent resource that cannot be migrated together with the share. The migration can fail earlier if one of those validations cannot be satisfied.

Before starting the migration, the share server and all its shares will have their status updated to reflect the operation that is being executed and to block any other operation that could be triggered after this one started. The source share server and all its shares will have their status updated to server_migrating while the destination share server will be updated to server_migrating_to. By changing all shares’ status, users will be able to identify that a group of shares is blocked for receiving any other operation.

After running through all validations with success, the share server’s new attribute called task_state will be updated to server_migration_starting and the scheduler will be invoked to validate if the host matches with the provided share types.

By reaching share manager’s migration start method, a driver’s call will be triggered to analyze if the destination back end can handle such operation before starting the migration. If one of the required options can’t be satisfied, the migration will fail.

The share manager will update the share server’s task_state to server_migrating and all its instances’ status to server_migrating. A new share server might be requested in the destination back end to hold all the data from source. It is expected that drivers will be able to identify that a new server is being requested for migration purposes. After that, the driver will be called to start the share server migration and to return immediately.

A share manager periodic task will continuously check share servers that have the task_state set to server_migrating to invoke the driver’s call share_server_migration_continue to track the progress of share servers that are in the 1st phase of the migration. After successfully finishing the 1st phase, the share server task_state will be updated to server_migrating_phase1_done.

Finally, share manager’s share_server_migration_complete method can be invoked for share servers that already completed the 1st phase, to finish the migration. In this phase, the driver is called to finish the share server migration and perform the last steps in the back end and return the list of export locations for all its shares. The task_state of the share server is set to server_migration_completed and all its shares have their export paths updated before they become available again.

Before moving to the 2nd phase, during the data copy or at the 1st phase completed, administrators can cancel the operation by invoking the share_server_migration_cancel API. If supported by the driver, the cancel operation will delete everything new that was created during the process, and the share server and all its shares will go back to the initial state.

Scheduler Changes

The scheduler filters can be used to validate if the destination host can hold all shares associated to the share server being migrated. Share API will need to provide the share server’s total size along with all associated share types’ capabilities in order to validate if the destination host is suitable for the new share server. However, the Scheduler won’t be able to validate share servers that spans across multiple pools, and for this type of scenario, share server migration will need to rely on driver’s checks to validate the feasibility of such operation.

Alternatives

The alternative is to use scripts or any other automation tool to move all shares to a new destination, one by one, using share migration feature.

Data model impact

A new field will be added to Share Server table to help tracking the states of a share server migration. The new field task_state will work like the same field that already exists on Share table. Administrator will be able to reset the task_state by issuing the API share-server-reset-task-state, as shown in the next section.

REST API impact

For admin-only, new API methods will be implemented:

share-server-migration-start

Migrates a share server:

POST /share-servers/{share_server_id}/action

Body:

{
  "migration_start": {
    "writable": true,
    "nondisruptive": true,
    "preserve_snapshots": true,
    "host": "host@dummy1#pool2",
    "new_share_network_id": "new_share_network_id"
  }
}

The host contains the string host where the share server will be migrated to. The capabilities preserve_metadata, writable, nondisruptive and preserve_snapshots, if enabled, must be supported by the drivers that implement such feature. If one of the capabilities isn’t supported, the migration will fail later in the driver’s compatibility check.

By setting writable to true it’s expected that all shares remain writable during the first phase of the migration, where the data copy usually occurs. However it doesn’t guarantee that will remain writable during the second phase, where the cutover usually happens for drivers that don’t support a nondisruptive migration.

By specifying nondisruptive equal to true, the migration will be performed without disrupting clients during the entire process, which usually means that export locations won’t be modified, and hence new network allocations won’t be made for the new share server.

If preserve_snapshots is set, it’s expected that all snapshots from all shares will be migrated together with the share server. If not supported by the driver, users will need to consider unmanaging or deleting all snapshots before proceeding with the migration.

The only optional parameters is ‘new_share_network_id’, which may need to be provided to fit destination network requirements.

If the provided share_server_id doesn’t exist, the API will respond with 404 Not Found. If one of the optional parameters is invalid or doesn’t exist, the API will respond with 400 Bad Request. If during the initial validations in the Share API, one of the resources is busy or has an invalid status, the API will respond with 409 Conflict.

Upon a failure, the share server and all its share will have their status updated to available and their task_state set to server_migration_error.

share-server-migration-complete

Start the 2nd phase of migration:

POST /share-servers/{share_server_id}/action

Body:

{"migration_complete": {}}

Triggers the start of the 2nd phase of migration on a share server that already finished the 1st phase.

If the provided share_server_id doesn’t exist, the API will respond with 404 Not Found. If the operation can’t be performed due to unsupported migration state, the API will respond with 400 Bad Request.

Upon a failure in the second phase of the migration, the share server and all its shares will have their status updated to error and their task_state set to server_migration_error. At this point, it won’t be possible to determine the status of the share server and its shares, and it will be up to the administrator to manually fix this problem.

share-server-migration-cancel

Attempts to cancel migration:

POST /share-servers/{share_server_id}/action

Body:

{"migration_cancel": {}}

To cancel a migration in progress, the operation must not be in the 2nd phase and the driver must support such operation.

If the provided share_server_id doesn’t exist, the API will respond with with 404 Not Found. If the operation can’t be performed due to unsupported migration state or unsupported operation within the driver, the API will respond with 400 Bad Request.

After a successful migration cancellation operation, the share server and all its shares will have their status updated to available and their task_state set to server_migration_cancelled.

share-server-migration-get-progress

Attempts to obtain migration progress:

POST /share-servers/{share_server_id}/action

Body:

{"migration_get_progress": {}}

Response:

{"total_progress": 30}

Gives the current migration progress in a percentage value. Drivers might also provide additional information together with total_progress info.

If the provided share_server_id doesn’t exist, the API will respond with 404 Not Found. If the provided share_server_id isn’t performing a migration, the API will respond with 400 Bad Request.

share-server-reset-task-state

Reset task state field value:

POST /share-servers/{share_server_id}/action

Body:

{
  "reset_task_state": {
    "task_state": "migration_error"
   }
}

If the provided share_server_id doesn’t exist, the API will respond with 404 Not Found.

share-server-migration-check

Check if a share server can be migrated to a destination host:

POST /share-servers/{share_server_id}/action

Body:

{
  "migration_check": {
    "writable": true,
    "nondisruptive": true,
    "preserve_snapshots": true,
    "host": "host@dummy1#pool2",
    "new_share_network_id": "new_share_network_id"
  }
}

Response:

{
  "compatible": true,
  "requested_capabilities": {
    "writable": true,
    "nondisruptive": true,
    "preserve_snapshots": true,
    "host": "host@dummy1#pool2",
    "new_share_network_id": "new_share_network_id"
  }
  "supported_capabilities": {
    "writable": true,
    "nondisruptive": false,
    "preserve_snapshots": true,
    "new_share_network_id": "new_share_network_id"
    "migration_cancel": true,
    "migration_get_progress" false,
  }
}

Checks the feasibility of migrating a share server to a destination host. Drivers will be able to check if the provided destination host can hold the share server and which migration options will be available for this operation.

By answering compatible equal to true or false, the admin will know if the provided host is a feasible destination for the share server.

The migration options writable, nondisruptive and preserve_snapshots show if the driver supports such options while migrating the share server. If supported, the current share network or, if provided, the new_share_network_id will also appear in the supported_capabilities field.

The migration operations migration_cancel and migration_get_progress may also be available depending on the driver implementation.

Driver impact

Vendors that want to support share server migration must implement the following interfaces:

choose_share_server_compatible_for_migration: interface needed to tell the share manager which compatible share server can be used as destination in a migration operation;
share_server_migration_check_compatibility: it will be always called before starting the migration to check if the driver supports migrating the share server to the required destination, and answer which kind of capabilities will be supported on such operation;
share_server_migration_start: called to start the first phase of migration. The procedure should be started in the back end and return immediately.
share_server_migration_continue: will be called to monitor the progress of a share server migration. Drivers will answer if the 1st phase was already finished or raise an exception in case of failure.
share_server_migration_complete: starts the 2nd phase of the migration, to complete the operation by cutting over the access from the source and providing access through the destination.
share_server_migration_cancel: drivers will implement this call if they support the cancellation of a migration operation that is already in progress. The migration cancellation won’t be available for share servers that already started the 2nd phase;
share_server_migration_get_progress: drivers will implement this call to provide the total progress of the migration.

As implemented in share migration approach, drivers will be invoked to check the compatibility with the destination back end before starting the migration. During this validation, drivers will be able to return the capabilities supported for migrating a share server to the provided destination, such as remaining writable, preserving snapshots and others.

After that, share_server_migration_start will take place and ask drivers to start the 1st phase of the migration, that should be answered asynchronously. Manila will reuse the same periodic task from share migration to continuously check if the 1st phase is already completed by calling the driver interface share_server_migration_continue.

Finally, the driver will need to perform the last steps to complete the share server migration when the share_server_migration_complete is invoked. At this moment, the access to the source share server shares may be interrupted, depending on driver’s capabilities, and moved to the new destination.

Security impact

None

Notifications impact

None

Other end user impact

During the migration process users won’t be able to perform any management operation in all shares that belong to the share server being migrated. Depending on driver’s capabilities, users may also lose write access to those shares.

Performance Impact

No performance impact is expected on implementing this feature. However, depending on how many shares are placed within a share server, other operations can be impacted due to the number of database operations triggered by a share server migration, during sanity checks and status updates on all affected resources (shares, snapshots, access, etc).

Other deployer impact

Drivers that implement share server migration might need to retrieve the configuration from other back ends in order to access it and provide a way of copying all the data. Administrators will need to keep these files up to date in all its share service instances.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: dviroel

Work Items

Implement main patch that contains:
- New API methods for share server migration;
- New Scheduler call for share server migration start;
- Share Manager implementation for share server migration;
- Database updates for Share Server model;
- New driver interfaces for migration of share servers.
Update python-manilaclient with new share server’s CLI commands.
For testing:
- Improve and implement both container and dummy drivers to support share server migration across different back ends.
- New functional tests in manila-tempest-plugin.
Documentation updates.

Dependencies

None.

Testing

The container driver will need to be improved to support share server migration across different back ends.

New functional tests will be added to perform share server migration on the same back end and across different back ends. Vendors that implement support for this feature will be encouraged to run these tests in their CI.

Documentation Impact

The following documentation will be updated:

API reference: Will update the Share Server API by adding the new actions for share server migration procedure.
Admin reference: Will add information on how the functionality works and which drivers supports it.
Developer reference: Will add information on how the new functionality works, and which interfaces need to be implemented.

References

[1] https://docs.openstack.org/manila/ussuri/admin/shared-file-systems-share-server-management.html

[2] https://opendev.org/openstack/manila-specs/src/branch/master/specs/newton/newton-migration-improvements.rst

[3] https://etherpad.opendev.org/p/share-server-migration

Support to query user messages filtering by timestamp comparison

Wed, 01 Apr 2020 00:00:00

https://blueprints.launchpad.net/manila/+spec/query-user-message-by-timestamp

Add support for querying user messages by specifying a timestamp, which will be compared to the created_at field, and Manila will return all the messages that match to the given time condition.

Problem description

Manila API only support filtering user messages by discrete values (such as message_id or resource_id), even though user messages have timestamp fields as created_at, users can not query by a given period. Users may be interested in user messages in specific periods in order to analyze causes of asynchronous errors. Currently they need to retrieve and filter the resource messages by themselves. This change will improve the usability when searching for user messages and the efficiency of timestamp based queries.

Use Cases

In large scale environment, lots of asynchronous error messages may be created . In order to quickly locate the cause of the error, user or system manager only need to get user messages corresponding to the resource which was created during a specified time period, instead of querying all asynchronous error messages of the resource.

Proposed change

The following changes are proposed:

Introduce two additional parameters in the url that searches for user messages, being both timestamps.
- created_before Return results older than the specified time.
- created_since Return results since the specified time.

Alternatives

None

Data model impact

None

REST API impact

List user messages API will accept new query string parameters. User can add created_before to the url to query user messages older than the specified time and created_since to query user messages newer than the specified time, or use both of them to query user messages that were created during the specified time interval. This changes also need to bump the microversion of API to keep forward compatibility.

GET /v2/{project_id}/messages?created_since=2019-11-01T01:00:00&created_before=2019-11-02T01:00:00

The time format is ISO 8601, it can be as follows:

2019-11-01
2019-11-01T01:00
2019-11-01T01:00:00Z
2019-11-01T01:00:00+05:00

Security impact

None

Notifications impact

None.

Other end user impact

One command will be updated:

manila message-list [--resource_id <resource_id>]
                    [--resource_type <type>] [--action_id <id>]
                    [--detail_id <id>] [--request_id <request_id>]
                    [--level <level>] [--limit <limit>]
                    [--offset <offset>] [--sort-key <sort_key>]
                    [--sort-dir <sort_dir>] [--columns <columns>]
                    [--before <b_time>] [--since <s_time>]

Python client may add help to inform users this new filter, and will update the corresponding cli command.

Performance Impact

Performance can be improved for timestamp based queries while using the database engine. Additionally, using indexes, the query performance will be enhanced.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: haixin<haixin@inspur.com>

Work Items

Add API filter
Add querying support in sql
Add related unit and functional tests
Add python-manilaclient support
Add docs update

Dependencies

None

Testing

Unit test to test if those filters can be correctly applied.
Tempest test if change filter work correctly from API perspective.

Documentation Impact

The manila API documentation will need to be updated to reflect the REST API changes.

References

None

OSC support for Manila

Wed, 18 Mar 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/python-manilaclient/+spec/openstack-client-support

Problem description

Python-Openstackclient is the default command line client for many OpenStack projects.

Use Cases

An end user can interact with manila using the same client they use for other services in OpenStack through the python-openstackclient.

Proposed change

The intent of this spec is to identify the commands to be implemented and establish conventions for command and argument names. This spec is not intended to be a full and correct specification of command and argument names. The details can be left to the code reviews for the commands themselves.

The following conventions will be adopted for argument flags:

We are not planning to implement any different / new arguments to existing ones in python-manilaclient.

The following manila commands will be implemented for openstack

Limits

manila absolute-limits
openstack limits show --absolute

manila rate-limits
openstack limits show --rate

manila create
openstack share create

manila delete
openstack share delete

manila force-delete
openstack share delete --force

manila list
openstack share list

manila show
openstack share show

manila update
openstack share set
openstack share unset

Security services

manila security-service-create
openstack share security service create

manila security-service-delete
openstack share security service delete

manila security-service-list
openstack share security service list

manila security-service-show
openstack share security service show

manila security-service-update
openstack share security service set
openstack share security service unset

Storage pools

manila pool-list
openstack share pool list

Services

manila service-enable
manila service-disable
openstack share service set

manila service-list
openstack share service list

Availability zones

manila availability-zone-list

We must implement this as a subcommand to the existing openstack availability zone list command.

Manage and unmanage shares

manila manage
openstack share adopt

manila unmanage
openstack share abandon

Quota sets

manila quota-defaults
openstack quota defaults

manila quota-delete
openstack quota delete

manila quota-show
openstack quota show

manila quota-update
openstack quota set

Quota class set

manila quota-class-show
openstack share quota class show

manila quota-class-update
openstack share quota class set

User messages

manila message-delete
openstack share message delete

manila message-list
openstack share message list

manila message-show
openstack share message show

Experimental APIs

Alternatives

Continue using python-manilaclient as the manila client. Continue maintenance for python-manilaclient. Do not implement any openstack command.

Data model impact

No impact on the data model.

REST API impact

No impact on the REST API.

Driver impact

No impact on the drivers.

Security impact

No impact on security.

Notifications impact

No impact on notifications.

Other end user impact

Users will be able to interact with Manila through python-openstackcli. On the other hand, users may keep using manila outside of traditional openstackcli.

Performance Impact

No impact on performance.

Other deployer impact

No deployer impact.

Developer impact

No developer impact.

Implementation

Assignee(s)

Primary assignee:

Soledad Kuczala <sol.kuczala@gmail.com>

Other contributors:

Goutham Pacha Ravi <gouthampravi@gmail.com>
Sofia Enriquez <senrique@redhat.com>
Victoria Martinez de la Cruz <victoria@redhat.com>

Work Items

Implement basic python-openstackclient shell support
Implement shares and share-types support
Implement limits
Implement storage pools
Implement services
Implement share export locations, share metadata and share actions
Implement share snapshots and share snapshot instances
Implement share networks
Implement security services
Implement share servers
Implement share instances and share instance export locations
Implement availability zones
Implement manage and unmanage shares
Implement quota and quota class sets
Implement user messages
Implement share access rules and share access rule metadata
Implement share migration (experimental)
Implement share replicas and share replicas export locations (experimental)
Implement share groups, share groups types and share group snapshots (experimental)

Dependencies

No dependencies at the moment of writing for this project.

Testing

Unit tests will be required as part of the implementation for each of the openstack commands.

Documentation Impact

End User guide
Documentation in which there is CLI usage will need to be updated. For consistency sake, we expect to have good coverage before changing those docs. In order words, at least non-experimental functionality needs to be implemented before changing manila commands docs for OpenStack commands.

References

http://lists.openstack.org/pipermail/openstack-discuss/2019-January/002271.html

Improve share and back end capabilities in Stein

Wed, 18 Mar 2020 00:00:00

https://blueprints.launchpad.net/manila/+spec/storage-proto-enhancement

https://blueprints.launchpad.net/manila/+spec/make-share-shrinking-support-a-capability

Manila’s back-end storage drivers offer a wide range of capabilities. The variation in these capabilities allows cloud administrators to provide a storage service catalog to their end users.

Sometimes back-end capabilities are very specific to the storage system, and are opaque to manila or the end users - they provide programmatic directives to the concerned storage driver to do something during share creation or share manipulation. Cloud administrators know of the opaque capabilities through driver documentation and they configure these capabilities within share types as scoped extra-specs (e.g.: hpe3par:nfs_options). The manila scheduler ignores scoped extra-specs during its quest to find the right back end to provision shares.

There are some back-end capabilities in manila that do matter to the scheduler. For our understanding, lets call these non-opaque capabilities. The documentation for these capabilities is here [1] and the share back end feature support classification is provided here [2]. All non-opaque capabilities can be used within share types as non-scoped extra-specs. The non-opaque capabilities can be of three types:

Capabilities pertaining to a specific back end storage system driver: For example, huawei_smartcache, these are considered by the scheduler’s capabilities filter (and any custom filter defined by deployers). These capabilities are reported via the scheduler-stats API, however, no other manila API relies on them.
Common capabilities that are not tenant visible: The manila community has standardized some cross-platform capabilities like thin_provisioning, dedupe, compression, qos, ipv6_support and ipv4_support. Values of these options do not matter to any manila APIs, however, they can signify something to the manila services themselves. For example when a back end supports thin_provisioning, the scheduler service performs over-provisioning, and if a back end does not report ipv6_support as True, the share-manager service drops IPv6 access rules before invoking the storage driver to allow/deny access rules.

Although these capabilities are not tenant visible, administrators may use them as extra-specs in share types and the scheduler will match these capabilities in the various filters, including the Capabilities filter included with manila.
Common capabilities that are tenant visible: Some capabilities affect functionality exposed via the manila API. For example, not all back ends support snapshots, and even if they do, they may not support all of the snapshot operations. For example, snapshot_support, create_share_from_snapshot_support, revert_to_snapshot_support, mount_snapshot_support and replication_type, etc. The support for these extra-specs determines whether users would be able to perform certain control-plane operations with manila. For example, the CEPHFS back end may report snapshot_support as True allowing end users to invoke the snapshot API, however, CEPHFS snapshots cannot be used by the driver to create a new share, so it reports create_share_from_snapshot_support as False. This reporting allows cloud administrators to create a share type that supports snapshots but not creating shares from snapshots. When a user uses such a share type and their share is created on a CEPH cluster, they cannot invoke the snapshot API successfully because the share type specifies that the feature be disabled for that specific share.

Tenant-visible capabilities aid manila in validating requests and failing fast on requests it cannot accommodate. They also help level set the user expectations on some failures. For example, if snapshot_support is set to False on the share type, since users can see this, they will not invoke the create snapshot API, and even if they do, they will understand the HTTP 400 (and error message) in better context.

Synchronous failures can be acted upon by automation as well as end users interacting manually with manila. Therefore, to the extent possible, all validation that can be performed in the API must be performed because failing early and fast is better than failing asynchronously.

Asynchronous failures may be a bit more painful to react to. With user messages [3] tenants can examine the cause of asynchronous failures. However not all user messages are actionable by end users.

Problem description

This specification focuses on two problem areas: shrinking of shares and storage protocol:

Shrinking shares

Shared File Systems are meant to be elastic. However, not all storage systems can support reducing the size of the allocated filesystems. Consider the case where manila is deployed with a back end that does not support shrinking of shares. When a user tries to shrink a share provisioned on that back end, the call reaches the share-manager process and fails at the driver with a NotImplementedError. The result of the failure causes a state change on the share to shrinking_error. There is no user message either [4]. Even though the share itself is not manipulated in any way, the status on the share makes it impossible to do anything else with it. Typically, users would have to contact the cloud administrator to recover from this status change.

Storage Protocols supported

Currently there is a configuration option enabled_share_protocols that is used by the manila API service to validate if a user’s request can be accepted for scheduling. If this option is mis-configured, we expect a scheduling failure to occur (See Bug 1783736 [5]). Users have no way of knowing what storage protocols are have been configured and what protocols are supported in a given deployment. Drivers themselves may support one or more storage protocols that they report to the scheduler. We introduced capability lists in manila [6] several releases ago. However, drivers that support multiple protocols are reporting a fused string to the scheduler (ex: NFS_CIFS).

Use Cases

Users must be able to determine beforehand if shrinking a share is possible
Users must receive a synchronous API failure if shrinking is not supported for a given share, but they attempt it anyway.
Cloud administrators must be able to configure storage protocols supported on a share type as extra-specs.
Users must be able to determine what storage protocols are supported on a manila deployment through share types.
The UI should be able to filter share protocols supported and enabled through the share types and vice versa.

Proposed change

A new common capability called shrink_support will be introduced. The base driver will determine if the shrink_share() interface is implemented and report the capability shrink_support appropriately.
Cloud administrators will be able to configure shrink_support as an extra-spec in share types. This extra-spec will be tenant visible, and can have one of three values: True, False or legacy (reserved value).
The scheduler will match only back ends that report shrink_support=True if the extra-spec’s value is set to True. If the extra-spec’s value is set to False or legacy, the share could be scheduled to a back end that may or may not support shrinking shares.
If the extra-spec’s value is set to False, manila will set a boolean flag (also called shrink_support) on the share object to False. This flag is inspected by the shrink API and if the value is False, shrinking will be disallowed right off the bat.
If the extra-spec’s value is set to legacy, the scheduler disregards the capability and allows the share to be provisioned on any back end (as filtered and weighed by all other factors). On successful scheduling and share creation, the real value of shrink_support will be determined and set on the share by the share-manager process.

Note

The extra-spec value legacy preserves backwards compatibility to schedule shares with pre-existing share types. See the Upgrade impact section for more discussion on this proposal.

Administrators can only set the value of shrink_support to a boolean expression. In consequence, they will be prevented from setting the value to the reserved value legacy.
Cloud administrators already have the ability to configure storage_protocol as an extra-spec. This extra spec will become tenant visible. If administrators do not configure this extra-spec, it will have a default value of '*' which indicates that shares of all storage protocols are allowed to be created. The value of this extra-spec can be a single protocol or list form, ex: <in> NFS <in> CIFS.
If the extra-spec storage_protocol is present and not set to '*', the API will validate if the requested protocol for the share being created is supported by the share type.
The UI will be modified to parse the share type chosen and populate the allowed protocols in the drop down on the create share form.
Drivers reporting multiple protocols will be updated to report them as a capability list instead of a string.
The base driver will evaluate [DEFAULT]/enabled_share_protocols and intersect this list with the capability reported by the driver code to report the correct protocol/s supported in the deployment to the scheduler.

Alternatives

An alternative to not implementing shrink_support as a capability is to disable the share shrinking API with the help of API policy (share:shrink). However, this mechanism cannot be used if the cloud has multiple back ends, one or more of which support shrinking shares while others don’t.

Users today cannot discover storage protocols they can use. We can live with this situation by expecting cloud administrators to communicate what is supported out-of-band of manila. This strategy may work for small deployments where there is a tight communication between end users and cloud administrators. For many others this inconvenience may be annoying, for instance, consider a cross-cloud application that prefers to discover capabilities and run on any OpenStack cloud.

Data model impact

The manila.db.sqlalchemy.models.Share model will be modified to include a nullable field called shrink_support.

Upgrade impact

There will be no data migration to populate the shrink_support capability field on pre-existing shares. The initial value of this field will be null. When users try to shrink a share that has its shrink_support attribute set to NULL, the API will continue to work as it does today, with no change in behavior, whether or not the share can really be shrunk by the back end.

In the share-manager service if NotImplementedError is raised by the driver, the field is set to False and a user message is generated. The status of the share will be set to shrinking_error. After resetting the share’s status, if users attempt to shrink the same share again, the API will be able to prevent it because the shrink_support has been appropriately determined. If the share-manager service does not detect a NotImplementedError, the shrink_support field will be set to True.

All existing share types will be updated to include the extra-spec shrink_support with its value set to legacy. This upgrade impact will be clearly called out in administrator documentation and release notes. By virtue of setting this extra-spec, we preserve backwards compatibility. The manila scheduler will ignore the shrink_support capability if its value is legacy. If a user uses such a share type to create a share, a warning message will be logged in the scheduler service suggesting that the administrator must override the legacy value to enhance user experience.

The share’s shrink_support attribute is determined at the driver after share creation by the virtue of the capability known by the driver. If cloud administrators do not change the value from legacy, their deployments will continue to provide a bad user experience. Users will not know until the share is created whether they can shrink the share or not. The experience further degrades if the same legacy share type matches two back ends: one that supports shrinking shares, and one that doesn’t. Users will see that some shares are shrinkable while some others are not even when using the same legacy share type.

REST API impact

Please note the current state of these APIs in our API reference.

Create and Manage share APIs:

POST /v2/{tenant_id}/shares
POST /v2/{tenant_id}/shares/manage

Changes to request schema / headers / authorization / endpoint: None

Changes to API behavior / response schema / headers / authorization:

Response schema:

{
    "share": {
        ...
        "shrink_support": true,
        ...
    }
}

The API will set the share attribute of shrink_support to True if the tenant visible extra-spec shrink_support has been set to a value that evaluates to True. If shrink_support extra-spec has not been configured, or if its value has been set to False, the shrink_support attribute on the share is set to False.

Note

Please see Upgrade impact to understand how pre-existing share types will be dealt with.

The API will evaluate if the requested protocol is present in the storage_protocol extra-spec of the share type used. If not present, HTTP 400 is sent back to the requester suggesting that they are using a protocol that is not supported.

Shrink share API:

POST /v2/{tenant_id}/shares/{share_id}/action

Changes to request schema / headers / authorization / endpoint: None

Changes to API behavior / response schema / headers / authorization:

In a new API version, the shrink_support attribute of the share object will be evaluated, and HTTP 400 will be returned to the requester if shrinking is not allowed. Since this action is bound to fail if allowed, API backwards compatibility will not be preserved to provide for a better user experience.

Other APIs:

GET /v2/{tenant_id}/shares
GET /v2/{tenant_id}/shares/detail
GET /v2/{tenant_id}/shares/{share_id}
PUT /v2/{tenant_id}/shares/{share_id}

All of the above APIs will include a response schema change to include the shrink_support in a new API version.

The share types APIs:

POST /v2/{tenant_id}/types
POST /v2/{tenant_id}/types/{share_type_id}/extra_specs

already disallow non-boolean values for a list of tenant visible common extra-specs. shrink_support will be added to this list.

Security impact

None

Notifications impact

A fix for LP 1783736 [4] will add a notification for a share shrink operation that can fail asynchronously.

Other end user impact

None

Performance impact

None

Other deployer impact

See Upgrade impact for the deployer’s action when this feature lands.

Developer impact

None

Driver impact

The driver capability storage_protocol will be changed as part of this effort to cleanup support for multiple protocols. From an underscore separated string it will be converted into a capability list.

Implementation

Assignee(s)

Primary assignee:: gouthamr

Work Items

Introduce shrink_support capability. Modify create/manage share APIs and other share API view builders. Add database migration to introduce share capability attribute. Add base driver implementation to detect support for shrinking shares.
Introduce storage_protocol tenant visibe extra-spec, modify driver reporting
Add support for shrink_support in python-manilaclient and manila-ui
Add support for storage_protocol in python-manilaclient and manila-ui
Add tempest tests for both changes

Dependencies

None

Testing

Unit test coverage will be added/maintained as per community standards. Tempest tests will be modified/added to cover new API changes. Share shrink tests will now create a new share type with shrink_support set to True to test shrinking shares.

Documentation Impact

The following OpenStack documentation will be updated to reflect this change:

OpenStack User Guide: Capabilities documentation will be improved
OpenStack Admin Guide: Share types changes will be documented and upgrade impact will be re-emphasized (Release notes will contain the impact and the actions in detail).
OpenStack API Reference: New APIs will be documented
Manila Developer Reference: No new documentation expected
OpenStack Security Guide: No new documentation expected

References

Create share from snapshots in another pool or back end

Mon, 30 Dec 2019 00:00:00

https://blueprints.launchpad.net/manila/+spec/create-share-from-snapshot-in-another-pool-or-backend

One of the current limitations present in most Manila drivers is the inability to create new shares from snapshots in pools other than the source share’s.

Given that some storage back ends have the ability to fast clone a share from a snapshot to another pool or back end, having this feature working in Manila will greatly improve the user experience.

This spec proposes changes to the current design of scheduling new shares created from snapshots. We will introduce a new scheduler capability and improve the behavior of an existing API parameter. By allowing the user to choose the desired availability zone and the admin to optimize its placement, we can improve our scheduler to be smarter for better load balancing and space efficiency results, while also preventing erroneous behavior.

One of the result of the improvements proposed is that drivers will be able to rely on a proper scheduling mechanism that makes sure new shares created from snapshots can land on compatible destinations.

Problem description

Manila has the scheduler’s configuration option use_scheduler_creating_share_from_snapshot that controls whether the scheduler should consider all available pools or just the source one as candidates to schedule a new share created from a snapshot. This option is disabled by default and is crucial to be set this way for drivers that cannot create shares anywhere other than the source pool. When enabled, this option allows different results:

The new share from snapshot may land on the same pool. This works for all drivers.
The new share from snapshot may land on another pool in the same back end. This works for the generic driver and may work on others (we do not know which). In case it does not work, it will result in an error in the driver.
The new share from snapshot may land on another compatible back end. This can work for the generic driver and is unlikely to work on others (we do not know which). In case it does not work, it will result in an error in the destination pool’s driver.
The new share from snapshot may land on another incompatible back end. This will certainly not work for any of the drivers in manila.

The main benefit from enabling this option is that the scheduler can select a different compatible back end to place the new share, even when the source share’s back end has space enough to handle it. In order to have it working properly, we need to address the following:

Guarantee that only compatible back ends will receive requests;
Guarantee that, when more than one compatible back end is available, the scheduler will choose the best fit;
No regressions are inserted to the current behavior when the option is disabled.

Use Cases

Given that there are storage back ends that support (or may want to support) new shares from snapshots to be efficiently created in pools or back ends different than the source share’s, a cloud administrator may want to enable this functionality, in order to balance the space usage across the several configured different compatible back ends in the environment.

Furthermore, if a new share from a snapshot may be placed on a back end other than the source share’s, it may as well be placed in a different availability zone. Since AZs are user-facing and are already included as a parameter in the Create share API, users could greatly benefit from this to create independent copies of their shares in AZs that are closer to their application.

From the user’s perspective there are two different ways of requesting new shares from snapshot:

User doesn’t provide the destination AZ. The user doesn’t specify an AZ to place the new share from snapshot. The scheduler won’t filter out the back ends based on a specific AZ, will filter out only the back ends that are not compatible with source share’s pool. The list of compatible back ends will be weighed based on configured weigher classes.
User provides the destination AZ. The scheduler will filter out all back ends that do not belong to the user’s specified AZ and all back ends that are not compatible with source share’s pool. The list of compatible back ends will be weighed based on configured weigher classes.

Proposed Changes

The proposed changes will only affect the existing behavior if the option use_scheduler_creating_share_from_snapshot is enabled. Otherwise, the API will skip the scheduler’s processing, with the new share being created into the same source share’s pool. Even so, if the user specifies a destination AZ to place the share from snapshot with the scheduler’s processing disabled, the API will return an error to the user about the erroneous usage of the parameter.

This spec proposes changes in order to have this functionality working as expected:

1. Change how the API handles the availability zone parameter, which is currently ignored when creating new shares from snapshots;

2. Create a new scheduler’s filter CreateFromSnapshotFilter, to identify compatible back ends;

3. Create a new weigher, to improve the scheduler’s selection of the best destination pool;

4. Update the Share Manager’s workflow when requesting a share creation from a snapshot. The driver interface for create_share_from_snapshot will be updated to return a model update when an asynchronous operation is needed.

5. Create a new periodic task to track and update the status of shares that were created from snapshots.

Provide a proper configuration and guidelines for the feature.

The required changes are detailed in the following subsections.

API’s Semantic

Currently, the AZ parameter is ignored in case a new share is created from a snapshot (potentially a bug, as it should return an error stating that a different AZ should not be specified). With the option use_scheduler_creating_share_from_snapshot enabled, the API will need to forward the AZ parameter into the request sent to the scheduler, otherwise, the request will fail and the user will be notified about the reasons of the failure.

Scheduler’s Improvement

The scheduler should be able to distinguish drivers that support creating new shares from snapshots in pools or back ends different than the source share’s. This ability will be addressed by existing filters and the use of an already implemented property:replication_domain.

The replication feature is tightly related to the ability of creating a share in another pool. If two or more back ends are coupled for replication, they commonly have a fast data link between each other, that can be used by some storage vendors to move, copy and replicate data among the back ends in the replication domain.

This spec proposes to create a new filter, CreateFromSnapshotFilter, to use the property replication_domain to identify compatible storage back ends, i.e., remote pools and back ends capable of creating shares from snapshots in an efficient way.

In this proposal it is considered that back ends that can replicate data between each other should also be able to create new shares in an efficient way, instead of depending on generic data copy methods. Although, drivers still need to implement mechanisms to perform this kind of operation efficiently based on back end capabilities.

At this point, this proposal would have solved the filtering problem, but not efficiently or with good performance, considering that some back ends that support this functionality have to create the new share from snapshot locally and move it over to the destination. Moreover, consider that some back ends are able to create these new shares from snapshots as COW (Copy-On-Write) clones, using no significant additional space.

Therefore, we could enhance the aforementioned proposal by creating a new weigher to assist on new share’s placement when creating from a snapshot. A new weigher called HostAffinityWeigher will always prefer to create the new share, from a snapshot, closer to the source’s pool, whenever it has space available and the AZ is the same as the source’s pool. The weigher will classify the compatible back ends based on their relative location with the source share’s location. If the source and destination hosts are located on:

same back ends and pools, the destination host is a perfect choice (e.g. 100 points)

same back ends and different pools, the destination host is a very good choice (e.g. 75 points)

different back ends with the same AZ: the destination host is a good choice (e.g. 50 points)

different back ends and AZ’s: the destination host isn’t a good choice (e.g. 25 points)

Even so, this strategy still doesn’t solve the used space balance use case at all, i.e., if no AZ is specified, the source pool will still be used until it runs out of space. This limitation won’t be handled by the new weigher itself, but it can be mitigated by the cloud administrator, who can specify filters or goodness functions to fulfill her/his own needs. An example of a possible configuration is presented below:

If the administrator wants to use the source pool until the free capacity goes under 500 GiB, he/she can provide a filter function to avoid the selection of this pool:
- filter_function = “share.snapshot_id and capabilities.free_capacity_gb < 500”

If COW clones are available at the source share’s pool, it’s very unlikely that the administrator will want to place the new share in another pool when working within the same AZ. But at the end, it is up to him/her to decide when a host should be filtered when creating shares from snapshot.

We can finally combine these approaches presented here and describe our proposed scheduling algorithm improvement:

The scheduler request will be updated to include source share’s host location needed by the new filter and weigher functions. It will be submitted through a list of scheduler filters, including the AvailabilityZoneFilter which will exclude back ends that don’t satisfy an availability zone restriction, if specified by the user during the share creation.
Alongside the scope of back ends determined by regular filters, we submit them through the CreateFromSnapshotFilter in order to filter incompatible back ends that don’t fulfill at least one of the restrictions:
1. Pools located in the same parent’s share back end.
2. Back ends and respective pools that match the same parent’s share string set in replication_domain.
After that, we submit them through the regular weighers. If there are no valid weighed back ends, we error out with the usual No valid host found message.
In case there is more than one candidate, the new HostAffinityWeigher will be used to weigh hosts based on their proximity to the source’s pool.

In this proposal, we assume that drivers that implement replication features between compatible back ends shall also be capable of creating new shares from snapshots in another pool or back end different from the source’s share.

Alternatives

There are no known alternatives that could fully solve the before-mentioned use cases. However, the alternatives below can address some of the current problems and partially solve the use cases:

Remove the config option ``use_scheduler_creating_share_from_snapshot`` and thus make the functionality restrictive and consistent across all drivers: This would also remove an existing functionality (even if it does not work as intended at all times). The API would be changed to return an error if the AZ parameter is specified when creating new shares from snapshots. This will not address the case of spreading out the creation of new shares from snapshots.
Use the Data Service for a generic implementation: This could partially address the use case, as it would allow drivers that are compatible with the Data Service to be seamlessly compatible with each other. However, any driver-specific optimizations that could allow the operation to be performed more efficiently between compatible drivers would not be able to be used. This is an approach that could be used as a fallback from the main proposal, in case the AZ parameter is specified but there are no drivers at the destination AZ that are compatible with the source share’s driver.

Data model impact

A new progress field will be added to manila.db.sqlalchemy.models.ShareInstance indicating the progress of a share creation.

REST API impact

Currently the AZ parameter is erroneously ignored when creating new shares from snapshots. The proposal changes the API behavior to use the AZ parameter to determine the AZ to schedule the new share from snapshot to, therefore the AZ parameter will not be ignored anymore. In the same way, if the requested AZ isn’t the same as the source share’s AZ when the configuration option [DEFAULT]/use_scheduler_creating_share_from_snapshot is set to False, the API will return a HTTP 400, Bad Request.

Driver impact

There is no impact for drivers that do not support or want to support the proposed feature. If the cloud administrator enables the option use_scheduler_creating_share_from_snapshot, the scheduler filter will guarantee that unsupported back ends will not receive these kind of requests. Vendors that want to support this feature will need to change their drivers implementation to properly handle this use case. In order to have it working, some prerequisites are needed:

1. Use the configuration option replication_domain and supply it to the scheduler together with the storage pool stats.

2. The driver implementation for create_share_from_snapshot will need to be modified to accept a destination pool different from the source share’s pool, and be prepared to return asynchronously if a slow copy operation is needed to complete the share creation.

3. Implement a new driver interface, get_share_status, that will be used by the share manager to periodically check the status of shares created from snapshots. The driver will be able to provide the current status, the creation progress information in a percentage value, and the export locations for shares that become available. If omitted, the progress information will be set to 0% for shares in STATUS_CREATING_FROM_SNAPSHOT and 100% for shares in STATUS_AVAILABLE.

Security impact

None.

Notifications impact

New scheduler code introduced will include error notifications for when a new share from snapshot fails to be scheduled. The message and code for those notifications will continue to be No valid host found. New manager code will include share status notifications when an asynchronous creation mode is used to create a new share from snapshot. The user will be notified that the share will take more time to be created and be notified when the share status is updated by the share manager.

Other end user impact

No changes to python-manilaclient are necessary. End users will be able to create new shares from snapshots in AZs other than the source share’s using the current python-manilaclient.

Performance Impact

The performance impact should be minimal. The share manager will need to check in regular intervals whether the driver has finished the share creation for shares that remain with status equal to STATUS_CREATING_FROM_SNAPSHOT.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: dviroel

Work Items

Implement main patch for manila that includes:
- Share API adjustments to pass AZ parameter;
- ShareInstance’s new progress field will be included to provide the total progress of a share creation.
- Scheduler’s new weigher will be added to rate host based on their proximity to the source’s share;
- Scheduler’s new filter CreateFromSnapshotFilter will be added to filter out incompatible back ends, only when the source share’s snapshot_id is provided.
- Share manager will introduce new periodic checks for asynchronous operations, update share’s status in the database and notify users about share’s status changes;
- New driver interface to check for updates on shares that were created from snapshots in an asynchronous mode.
Testing:
- Implement functionality in ZFSonLinux driver to validate correct scheduling and share creation from snapshot on different pools.
Functional tests in manila-tempest-plugin.
Docs update.

Dependencies

None.

Testing

New functional tests will be added to create a new share from a snapshot in a given AZ different than the existing one. Negative tests will check if user’s requested AZ is available and if the operation is compatible with the configured option use_scheduler_creating_share_from_snapshot.

The new tests will run at the gate for the ZFSonLinux driver configured with at least two AZs. Vendors that implement support for the new capabilities in their drivers will be encouraged to run the tests in their third party CI.

Documentation Impact

The following documentation sections will be updated:

API reference: Will update the Create Share API information, adding some detail on the impact of the availability_zone parameter when creating new shares from snapshots.
Admin reference: Will add detailed information on how the Create Share API behaves according to the AZ parameter when creating new shares from snapshots.
Developer reference: Will add information on how the functionality works, the optimizations and new capabilities.

References

[1] https://etherpad.openstack.org/p/manila-ptg-planning-denver-2018 [2] https://etherpad.openstack.org/p/manila-ptg-train [3] https://etherpad.openstack.org/p/shanghai-ptg-manila-virtual

Update share-type’s metadata

Tue, 23 Jul 2019 00:00:00

https://blueprints.launchpad.net/manila/+spec/update-share-type-name-or-description

This blueprint proposed to update the name, description and/or share_type_access:is_public attribute of share-type after create a share type, that support update the name, description and/or share_type_access:is_public attribute of a share-type.

Problem description

Currently, only the name, description and/or share_type_access:is_public of share-type is set when the share-type is created, and not allowed to be edited after the share-type is created. We can only set extra spec for share-type. But not name, description and/or share_type_access:is_public for share-type.

Use Cases

As a user, I would like to update the attribute of share-type, and to update its name, description and/or share_type_access:is_public of share-type after created. I would prefer this when I need a new share-type and do not have to create a new one.

Proposed change

Add a new microversion to the share type API.

Add a new function API to the Share types.

Add a update share-type API, available in a new micro version of the manila API:
- Add update method to ShareTypesController class.
- Add name, description and/or share_type_access:is_public parameters to share_type of the request body.
- Make the name and description parameters optional. And their length not more than 255. share_type_access:is_public parameters is also optional, but it’s a boolean type.
  
  If name is NULL, the name of share type will not be updated.
  
  If description is NULL, the description of share type will not be updated.
  
  If share_type_access:is_public is NULL, the public access of share type will not be updated.
  
  If name is “”, the name of share type is not valid.
  
  If description is “”, the description of share type will be empty.

Alternatives

None

Data model impact

None

REST API impact

URL: * /v2/{project_id}/types/{share_type_id}
Request method: * PUT
Normal http response code(s): * 200(OK)
Expected error http response code(s): * 400(Bad Request), 401(Unauthorized), 403(Forbidden), 404(Not Found), 409(Conflict)

Update share-type request will accept the following payload

{
    "share_type": {
        "name": "testing",
        "description": "share type description",
        "share_type_access:is_public": true
    }
}

Security impact

None

Notifications impact

None

Other end user impact

User will be able to update share-type attribute after created. Manila client may add help to inform users about this new function.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Upgrade impact

None

Implementation

Assignee(s)

Primary assignee:: haixin
Other contributors:: brinzhang

Work Items

Add update share-type API to the Share types.
Add update share-type API to manilaclient, and it will be supported by ‘manila type-update’.
Add update share-type API to manila-ui project.
Add functional tests.
Add units tests.
Add tempest tests.

Dependencies

None

Testing

Add related unittest
Add related functional test
Add tempest tests

Documentation Impact

Add update share-type API information to docs.

References

None

History

Revisions :header-rows: 1
Release Name	Description
Train	Introduced

Making manila share networks span multiple subnets (with AZs)

Tue, 14 May 2019 00:00:00

https://blueprints.launchpad.net/manila/+spec/share-network-multiple-subnets

Manila can be deployed with share drivers that handle the creation and management of share servers. In such a deployment, users create share networks by specifying network and subnet IDs from neutron. If the share driver is configured accordingly, a share server will be created with network information from the specified neutron subnet. This network information includes the IP addresses for the share server creation which are reflected in the export locations of shares created by such drivers.

If the share driver is configured with a standalone network or with a single network configuration [1], the share server is created with the network details from the manila.conf file. Network information from the user specified share network is (currently) ignored in this case. There is future work envisioned to provide network tunneling and hence, connectivity between the configured and user specified networks in such cases.

Users use share networks to create shares. A share is associated with a single share network and is exported off a share server that is also associated with the share network. Users can also associate security services such as LDAP, Active Directory or Kerberos with a share network, thereby affecting all the shares on the share network.

Problem description

When share replication [2] was designed in the Mitaka release of OpenStack, we decided to make replication work within and across availability zones. However, this feature was limited only to drivers with pre-configured share servers (DHSS = False mode). We realized that we could not support replication of shares created with drivers that support share server handling without making changes to the design of share networks.

With share replication, we recommend the use of availability zones as failure domains within an OpenStack cloud. manila-share services would be running across different availability zones when share replication is used for disaster recovery workloads. Users would create their primary share in one availability zone and replicas in different availability zones. Therefore, share servers that are created to handle and export these shares need to be created within respective availability zones.

Availability zones are used across other OpenStack services, such as nova and cinder as a way of logically partitioning resources within an OpenStack deployment. This partitioning can be motivated by the fact that these resources are meant for a particular task, or because they are co-located. However, the most common criterion is because these resources are associated with a common point of failure, such as being connected to a common power source. In the Mitaka release of OpenStack, neutron added support to availability zones [3][4]. With this, it is effectively possible for users to allocate network resources to availability zones for high availability.

In the Liberty release, when share instances were added to support features such as share replication and share migration, we acknowledged that the user’s share resources can live in multiple places simultaneously. These places may be across different network subnets and/or availability zones.

Currently, there is no provision for users to isolate their manila network resources, i.e share networks and share servers within availability zones that they create their compute host aggregates or block/shared file system storage resources within.

Use Cases

Providing a way for a share network to span multiple subnets allows for a more realistic representation of an internal data center network when the user means for a share network to signify network connectivity between different instances of his/her resources.

Currently, when a user brings up a compute instance or a shared file system on a subnet, all other resources on that subnet have network connectivity to this resource. Resources on other subnets can “access” this resource if there is a router that can send and receive traffic between these subnets.

When users create their shared file systems in a distributed manner, there is a very plausible chance that they mean for their mirrors (share replicas) or migration targets (share instances) to be on different subnets.

At the same time, there is a possibility that deployers would choose to make giant subnets that span all the availability zones.
Share servers in manila consume network allocations from a subnet. For advanced workloads such as share replication and share migration, they might often need to communicate between each other.

Therefore, it makes semantic sense for users to create one share network and assign subnets to the share network. This would essentially tag all share servers that are managing share instances of a particular share with one network. Each individual share instance however is exported off a particular subnet.

Proposed change

To solve the problem of allowing users to manage their share instances (and consequently share servers) within failure domains and to allow for a better representation of the instances of a shared file system across the user’s OpenStack cloud:

Share networks will span multiple subnets. These subnets would be associated with an availability zone each.
There can be a subnet associated with a share network that is associated with all storage availability zones. For the purposes of this specification, we will call this a default subnet. While not being associated with any particular availability zone, this default subnet is meant to service all availability zones.
Each share network can have no more than one default subnet.
A share network need not have a default subnet.
Users will have the ability to assign one subnet per configured storage availability zone per share network. They can assign the same subnet to different AZs if their setup permits, but they cannot have more than one subnet serving an AZ in particular, per share network.

Note

We could consider allowing more than one fall-back or default subnet per share network; Similarly, we could allow more than one subnet to be specified per share network for a given availability zone.

However, when a user requests to create a resource in a particular AZ, manila will then have no way of picking the right subnet of the choices it would have.

To avoid any indeterminate (and hence wrong) behavior, we will not allow more than one default subnet per share network; or more than one subnet per availability zone.

API changes would involve extending the ability to modify a share network by adding and removing subnets (with pre-conditions related to existing shares and share servers).
Users would need to pick the share network and an AZ during share creation if they expect a share to be provisioned with a specific subnet. They only need to pick the AZ during share replica creation.

Things that will not change:

A share would still only be associated with one share network.
A share server would still be allocated IPs from a single subnet’s allocation pool.
A security service would continue to be associated with the share network. When share servers are created on new subnets, they will derive all the security server information from the share network.

Work-flows affected

Creating a share network

Users can continue to create a share network with a single subnet, but they would have to provide an availability zone to assign the subnet to. The user may also intend to create a subnet that spans all availability zones. This subnet could act as the default or fallback subnet when a dedicated subnet is not available to service a given availability zone. However, the user may not have more than one of these default subnets per share network.

Alternatively, users can also create “empty” share networks, as is the case today.

Creation of a share server

Share servers will still be created by manila-share service with network allocations from a single subnet.

Creating a share

Users can provide the share network to create the share with, as before. However, if there is more than one subnet associated with the share network, they would ideally provide an availability zone as well to create it on a particular subnet (perhaps for co-location purposes with other resources they care about on their OpenStack cloud).

If the user provides a share network and an availability zone to create the share within, the API will validate that the share network supports that AZ (i.e, either has a subnet assigned to the specified AZ or has a default subnet). The API will respond with a 400 BadRequest if this condition fails.

Creating a share replica

Users would provide the share network and AZ during creation of the primary share. They would only need to provide an availability zone to create the share replica within. The share network is inherited from the parent share object and the specific subnet is chosen in association with the AZ provided.

Note

Currently, the API to create a share replica accepts share_network_id as an optional parameter. This parameter will be removed to support this workflow. Since there are no share drivers that work in DHSS = True mode and support share replication, we can remove this unused parameter. See associated launchpad bug [5].

Assigning a security service

Users will continue to assign security services to share networks, no changes are proposed to this workflow.

Alternatives

We can keep the existing design of share networks, but allow related resources to be created in multiple share networks. A share network will not span subnets, and users would require to use a different share network for each instance of a share. They would have to configure security services to each of these networks. For share replication, effectively, the share network of the share would be the share network in which the active replica instance is created.

There are limitations of this approach when the scope of future expansion is to be considered. Ideally, we would like users to be able to control their resources with respect to availability zones, and this approach lends itself poorly to that goal.

Quotas are currently enforced on the number of share networks. If a user needs to deal with multiple subnets as multiple share networks, quota limitations would be a problem. As a user, I might want to divide my network resources by availability zones, without consuming the network quotas, because they are all being used to manage the same share (and its replicas or instances), or the same security service.

Data model impact

The manila.db.sqlalchemy.models.ShareNetwork model will no longer contain the subnet specific information: neutron_net_id, neutron_subnet_id, network_type, segmentation_id, cidr and ip_version. These keys will be part of a new model manila.db .sqlalchemy.models.ShareNetworkSubnet which has a primary key: id (we’ll refer to this as share_network_subnet_id for the purposes of this document) and foreign keys, availability_zone_id and share_network_id. A share network subnet is only associated with exactly one share network and one availability zone.

The manila.db.sqlalchemy.models.ShareServer model will stop having a foreign key binding to share_network_id and switch to having a foreign key reference to share_network_subnet_id instead.

No key changes are going to be made to manila.db.sqlalchemy.models.ShareNetworkSecurityServiceAssociation, manila.db.sqlalchemy.models.SecurityService, manila.db.sqlalchemy .models.ShareInstance or manila.db.sqlalchemy.models.ConsistencyGroup models. All these resources will continue to contain a foreign key reference to the manila.db.sqlalchemy.models.ShareNetwork model.

The database upgrade step will create a new share network subnet per existing share network. The availability_zone_id field will be null indicating that these are default subnets, unconfined to a given availability zone.

The database downgrade step will collapse existing share network subnets into the share network table. This step may result in loss of information if multiple subnets exist per share network. Hence, it is not recommended in a production cloud.

REST API impact

Please note the current state of these APIs in our API reference.

Creating a share network:

POST /v2/{tenant_id}/share-networks

Request:

{
    "share_network": {
        "neutron_net_id": "998b42ee-2cee-4d36-8b95-67b5ca1f2109",
        "neutron_subnet_id": "53482b62-2c84-4a53-b6ab-30d9d9800d06",
        "name": "my_network",
        "description": "This is my share network",
        "availability_zone": "london",
    }
}

Subnet details neutron_net_id and neutron_subnet_id are optional. Although, whenever one of them is specified, both must to be specified, otherwise the API will respond with 400 Bad Request.

availability_zone is optional if one of the following conditions are met:

user is creating a subnet that is meant to span all availability zones.
manila-share services are only configured with a single availability zone, or,
when subnet details are not provided.

If the availability_zone is not known to manila, the API will respond with 400 Bad Request.

If the tenant’s share network quota has exceeded, the API will respond with 403 Forbidden.

The API will not validate the subnet information with the network provider (neutron). The validation will be performed at the share manager layer.

Response:

Code: 202 Accepted

{
    "share_network": {
        "name": "my_network",
        "created_at": "2016-06-01T21:12:12.617687",
        "id": "77eb3421-4549-4789-ac39-0d5185d68c29",
        "project_id": "e10a683c20da41248cfd5e1ab3d88c62",
        "description": "This is my share network",
        "updated_at": null
    }
}

Adding a subnet to a share network:

POST /v2/{tenant_id}/share-networks/{share_network_id}/subnets

Request:

{
    "share-network-subnet" : {
        "neutron_net_id": "998b42ee-2cee-4d36-8b95-67b5ca1f2109",
        "neutron_subnet_id": "12c1490a-e82c-4f5e-bcb1-fb267a58cf10",
        "availability_zone": "paris"
    }
}

The same conditions exist for these parameters as with the create API above.

If the share network ID is invalid, the API will respond with 404 NotFound. If there already is a subnet defined for the availability zone specified, the API will respond with 409 Conflict.

Response:

Code: 202 Accepted

{
    "share_network_subnet": {
        "created_at": "2016-06-01T21:12:14.843836",
        "id": "aa7a1269-703b-4832-a3df-17ed954c276c",
        "share_network_id": "77eb3421-4549-4789-ac39-0d5185d68c29",
        "availability_zone": "paris",
        "segmentation_id": null,
        "neutron_subnet_id": "12c1490a-e82c-4f5e-bcb1-fb267a58cf10",
        "updated_at": null,
        "neutron_net_id": "998b42ee-2cee-4d36-8b95-67b5ca1f2109",
        "ip_version": null,
        "cidr": null,
        "network_type": null
    }
}

Removing a subnet from a share network:

DELETE /v2/​{tenant_id}​/share-networks/{share_network_id}/subnets/{share-network-subnet-id}

If the share network ID or the share network subnet ID is invalid, the API will respond with 404 NotFound. If there is a share server on the share network subnet, it cannot be removed; the API will respond with 409 Conflict.

Response:

202 Accepted

Summary listing of share networks:

GET /v2/{tenant_id}/share-networks

Response:

Code: 200 OK

{
    "share_networks": [
        {
            "id": "77eb3421-4549-4789-ac39-0d5185d68c29",
            "name": "my_network"
        },
        {
            "id": "00cc3770-2558-4370-a088-de03b055dcff",
            "name": "my_network_2"
        }
    ]
}

No changes from the previous version of this API

Detailed listing of share networks:

GET /v2/{tenant_id}/share-networks/detail

Response:

Code: 200 OK

{
    "share_networks": [
        {
            "name": "my_network",
            "id": "77eb3421-4549-4789-ac39-0d5185d68c29",
            "project_id": "e10a683c20da41248cfd5e1ab3d88c62",
            "description": "This is my share network",
            "updated_at": null,
            "share_network_subnets": [
                {
                    "created_at": "2016-06-01T21:12:14.843836",
                    "id": "aa7a1269-703b-4832-a3df-17ed954c276c",
                    "availability_zone": "paris",
                    "segmentation_id": null,
                    "neutron_subnet_id": "12c1490a-e82c-4f5e-bcb1-fb267a58cf10",
                    "updated_at": null,
                    "neutron_net_id": "998b42ee-2cee-4d36-8b95-67b5ca1f2109",
                    "ip_version": null,
                    "cidr": null,
                    "network_type": null
                },
                {
                    "created_at": "2016-06-01T21:12:14.843836",
                    "id": "9c5dbe11-cdf6-48a4-b6ca-9582ef5af193",
                    "availability_zone": "london",
                    "segmentation_id": null,
                    "neutron_subnet_id": "53482b62-2c84-4a53-b6ab-30d9d9800d06",
                    "updated_at": null,
                    "neutron_net_id": "998b42ee-2cee-4d36-8b95-67b5ca1f2109",
                    "ip_version": null,
                    "cidr": null,
                    "network_type": null
                }
            ]
        },
        {
            "name": "my_network_2",
            "id": "00cc3770-2558-4370-a088-de03b055dcff",
            "project_id": "e10a683c20da41248cfd5e1ab3d88c62",
            "description": "This is also my share network",
            "updated_at": null,
            "share_network_subnets": [
                {
                    "created_at": "2016-06-01T21:12:14.843836",
                    "id": "1feffbd9-9747-413d-b162-83b97981f0ba",
                    "availability_zone": "paris",
                    "segmentation_id": null,
                    "neutron_subnet_id": "647cf190-e439-4159-98f9-17cb266d6d00",
                    "updated_at": null,
                    "neutron_net_id": "3b49335d-273e-4829-9b08-b7b1df157e69",
                    "ip_version": null,
                    "cidr": null,
                    "network_type": null
                }
            ]
        }
    ]
}

Showing details of a single share network:

GET /v2/{tenant_id}/share-networks/{share_network_id}/detail

If the share network ID is invalid, the API will respond with 404 NotFound.

Response:

Code: 200 OK

{
    "name": "my_network",
    "id": "77eb3421-4549-4789-ac39-0d5185d68c29",
    "project_id": "e10a683c20da41248cfd5e1ab3d88c62",
    "description": "This is my share network",
    "updated_at": null,
    "share_network_subnets": [
        {
            "created_at": "2016-06-01T21:12:14.843836",
            "id": "aa7a1269-703b-4832-a3df-17ed954c276c",
            "availability_zone": "paris",
            "segmentation_id": null,
            "neutron_subnet_id": "12c1490a-e82c-4f5e-bcb1-fb267a58cf10",
            "updated_at": null,
            "neutron_net_id": "998b42ee-2cee-4d36-8b95-67b5ca1f2109",
            "ip_version": null,
            "cidr": null,
            "network_type": null
        },
        {
            "created_at": "2016-06-01T21:12:14.843836",
            "id": "9c5dbe11-cdf6-48a4-b6ca-9582ef5af193",
            "availability_zone": "london",
            "segmentation_id": null,
            "neutron_subnet_id": "53482b62-2c84-4a53-b6ab-30d9d9800d06",
            "updated_at": null,
            "neutron_net_id": "998b42ee-2cee-4d36-8b95-67b5ca1f2109",
            "ip_version": null,
            "cidr": null,
            "network_type": null
        }
    ]
}

Showing details of a subnet:

GET /v2/​{tenant_id}​/share-networks/{share_network_id}/subnets/{share_network_subnet_id}

If the share network ID or the share network subnet ID is invalid, the API will respond with 404 NotFound.

Response:

Code: 200 OK

{
    "share_network_subnet": {
        "created_at": "2016-06-01T21:12:14.843836",
        "id": "aa7a1269-703b-4832-a3df-17ed954c276c",
        "share_network_id": "77eb3421-4549-4789-ac39-0d5185d68c29",
        "availability_zone": "paris",
        "segmentation_id": null,
        "neutron_subnet_id": "12c1490a-e82c-4f5e-bcb1-fb267a58cf10",
        "updated_at": null,
        "neutron_net_id": "998b42ee-2cee-4d36-8b95-67b5ca1f2109",
        "ip_version": null,
        "cidr": null,
        "network_type": null
    }
}

Deleting a share network:

DELETE /v2/{tenant_id}/share-networks/{share_network_id}

A share network with multiple subnets cannot be deleted atomically. The API will respond with 409 Conflict. Users would have to iteratively remove subnets until one or lesser subnets remain in the share network before attempting to delete the share network.

Response:

Code: 202 Accepted

Security impact

Security services are still associated with share networks, but the coverage of the share network is increasing to span multiple subnets. So for an external user, the only visible change is that the share network could now be “bigger” than it previously was. Hence, due care must be taken in adding subnets to existing share networks. Users may assign as many subnets to a particular share network as there are availability zones in a deployment. They can also define a subnet that can span all availability zones.

Notifications impact

We will asynchronously validate the network information during share creation, and if there is a mismatch with what is specified in the configured network plugin, we will raise an error user message.

Other end user impact

End users will have to specify an availability zone parameter during share network creation unless they would like to create the share network with a subnet that would span all availability zones.
End users would have to add new subnets to an existing network iteratively to extend the share network across AZs.

Performance impact

Validating that users provide the correct storage availability zones will be performed at the API layer. However, the validation of the availability zones with respect to their configured networks will be done at the network plugin layer, as existing network checks are done today. This is not expected to cause a performance impact, but will keep our existing share network validation isolated from the API allowing for further enhancements instead of gating changes at the API layer.

In other words, to minimize the performance impact, no further validation of network information at the API is recommended by the design introduced in this spec.

Other deployer impact

No new configuration options are expected to be added. However, neutron performs AZ validation when users create networks with availability_zone_hints. Deployers must ensure that neutron services are running in the desired availability_zones to allow for the network creation to succeed.
All existing share networks will have their subnet (if existing) designated as default after the database upgrade to this version of the database changes.

Developer impact

None

Driver impact

None. Drivers must be agnostic to all these changes since they will be handled at the manila API service and the share manager services.

Implementation

Assignee(s)

Primary assignee:: lseki

Work Items

Introduce share network subnets in the database and map them to availability zones.
Changes to share network APIs - create, list, modify
CLI implementation
manila-ui support

Dependencies

The evolution of share replication to cover the DHSS = True use case is dependant on this change. Once this work is completed, share drivers that support DHSS = True mode can support share replication. As a pre-requisite correction, the Share replica API should no longer support specifying the share_network_id. [5]

Testing

Unit test coverage will be added/maintained as per community standards. Tempest tests will be modified/added to cover new share network API changes. AZ awareness will be coded into the tests, with fall-backs like the share replication tempest tests.

Setting up multiple AZs on the gate is not in the scope of this work. However, these tests can be run manually with multi-AZ configurations.

Documentation Impact

The following OpenStack documentation will be updated to reflect this change:

OpenStack User Guide: will document the changes in creation and modification of share networks.
OpenStack Admin Guide: will document the APIs to list/delete share network instances.
OpenStack API Reference: All API changes will be documented
Manila Developer Reference: the low level implementation considerations and design of this feature will be documented.
OpenStack Security Guide: Readers will be made aware of the high level picture of adding AZ awareness to share networks.

References

[1]: http://docs.openstack.org/admin-guide/shared_file_systems_share_networks.html

[2]: https://wiki.openstack.org/wiki/Manila/design/manila-mitaka-data-replication

[3]: http://specs.openstack.org/openstack/neutron-specs/specs/liberty/availability-zone.html

[4]: http://docs.openstack.org/mitaka/networking-guide/adv-config-availability-zone.html

[5]: https://bugs.launchpad.net/manila/+bug/1588144

Mechanism to Prevent Race Conditions

Wed, 08 May 2019 00:00:00

https://blueprints.launchpad.net/manila/+spec/eliminate-race-conditions

This proposal is to develop a general solution for preventing race conditions which will work across services and in deployments where there are multiple copies of the same services (commonly known as Active/Active HA deployments).

The focus is on keeping all state in the database, and protecting changes to database state using briefly-held locks. Also concurrent operations which are mutually exclusive should fail as early as possible with a helpful error code to simplify the retry logic of upper layers.

Problem description

Certain operations in Manila should not be allowed to proceed in parallel, because the result of one operation would prevent the other operation from completing successfully.

For example, taking a snapshot of a share cannot happen simultaneously while deleting that share. Either the snapshot must occur first, which prevents the delete – or the delete must occur first, which prevents the snapshot. Unfortunately, not enough state is stored in the database to prevent these operations from racing with each other, so in practice two API calls can both proceed through the API service to the share manager, where eventually an error will occur and one or both operations will fail mysteriously.

There are multiple scenarios like the above where undefined behavior results. This specification does not attempt to enumerate all of them because the goal is to describe a mechanism for fixing these kinds of issues rather than explicitly fixing all such issues. Generally speaking, race conditions should be treated as bugs, but up until now Manila has lacked the tools to fix these bugs reliably.

Use cases

Specific cases:

Two snapshot operations should not be able to occur at the same time. One must complete before the second can begin. This ensures that snapshots occur in a known order, and prevents the useless situation of having 2 identical snapshots.
Taking a snapshot of a share should prevent a delete of that share.
Valid changes to access rules should always be accepted regardless of the state of the existing access rules. Although rules are applied to the backend asynchronously, it’s valid to add multiple rules faster than the system can apply them and expect Manila to catch up.

General cases:

These guarantees must be enforceable using a database running in a clustered configuration such as Galera. This prevents obvious solutions such as relying on DB row-level locking.
These guarantees must be enforceable while running multiple copies of Manila services, including multiple API, scheduler, and share manager services. This prevents obvious solutions like in-process locks.
These guarantees must be enforceable while running in a distributed configuration where cooperating services are on different nodes (physical, VMs, or containers). This prevents our existing approach of using file locks. Even though network-based file locking solutions exist, they represent a single point of failure and are unacceptable in properly distributed environments.
The Manila services should be able to automatically and gracefully recover from crashes and other unplanned downtime. This means that implicit state should be avoided and because long-held locks are implicit state they should be avoided in favor of short-held locks with explicit state.

Proposed change

More transitional states will be added so that operations which can conflict with other operations can be explicitly detected by looking at the share state. This spec only proposes one specific new state to address the races involving snapshots, but more generally provides a framework for resolving similar races as they are discovered.

Transitions between states will always be done while holding a distributed lock – a lock implemented by a distributed lock manager (DLM). Use of distributed locks ensures that all services see the same locking state even if services run on different nodes, and even across transient failures such as node failures and network partitions. The lock will be held only for the duration of the database test-and-set operation to minimize lock contention.

No locks will be held during calls from the share manager to the share driver. Mutual exclusion between driver calls will be achieved with state checks.

No locks will be held during RPC calls or casts.

Alternatives

The approach used by Cinder which relies on elaborate SQL calls to compare-and-swap fields was considered but rejected for the following reasons:

The code in Cinder can’t be shared with Manila because it relies on OVO (Oslo Versioned Objects)
Not enough people understand how it works so it’s likely to be hard to maintain.
Cinder’s compare-and-swap approach limits the kind of state changes you can make because updating multiple tables atomically is impossible. Locks don’t suffer from this restriction.

Data model impact

New states will be added:

Snapshotting
States for access rules covered in Access rules spec

digraph share_states { label="Share States" // Transitional States creating[shape=hexagon]; manage_starting[shape=hexagon]; deleting[shape=hexagon]; snapshotting[shape=hexagon,color=gold4, fontcolor=gold4]; migrating[shape=hexagon]; shrinking[shape=hexagon]; extending[shape=hexagon]; unmanage_starting[shape=hexagon]; replication_change[shape=hexagon]; // Error states error[color=red4, fontcolor=red4]; shrinking_error[color=red4, fontcolor=red4]; shrinking_possible_data_loss_error[color=red4, fontcolor=red4]; extending_error[color=red4, fontcolor=red4]; unmanage_error[color=red4, fontcolor=red4]; manage_error[color=red4, fontcolor=red4]; error_deleting[color=red4, fontcolor=red4]; // Other states new[color=blue, fontcolor=blue]; available[color=darkgreen, fontcolor=darkgreen]; deleted[shape=box, color=navy, fontcolor=navy]; unmanaged[shape=box, color=navy, fontcolor=navy]; // User requested transitions new -> creating[label="create"]; new -> manage_starting[label="manage"]; available -> deleting[label="delete"]; available -> snapshotting[label="create snapshot", color=gold4, fontcolor=gold4]; available -> migrating[label="migrate"]; available -> shrinking[label="shrink"]; available -> extending[label="extend"]; available -> unmanage_starting[label="unmanage"]; available -> replication_change[label="add replica"]; // Automatic transitions creating -> available[label="success", color=darkgreen, fontcolor=darkgreen]; deleting -> deleted[label="success", color=darkgreen, fontcolor=darkgreen]; snapshotting -> available[label="success", color=darkgreen, fontcolor=darkgreen]; manage_starting -> available[label="success", color=darkgreen, fontcolor=darkgreen]; unmanage_starting -> unmanaged[label="success", color=darkgreen, fontcolor=darkgreen]; extending -> available[label="success", color=darkgreen, fontcolor=darkgreen]; shrinking -> available[label="success", color=darkgreen, fontcolor=darkgreen]; replication_change -> available[label="success", color=darkgreen, fontcolor=darkgreen]; // Reset transitions error -> available[label="reset"]; shrinking_error -> available[label="reset"]; extending_error -> available[label="reset"]; unmanage_error -> available[label="reset"]; manage_error -> available[label="reset"]; error_deleting -> available[label="reset"]; // Error transitions creating -> error[label="fail", color=red4, fontcolor=red4]; migrating -> error[label="fail", color=red4, fontcolor=red4]; shrinking -> shrinking_error[label="fail", color=red4, fontcolor=red4]; shrinking -> shrinking_possible_data_loss_error[label="fail", color=red4, fontcolor=red4]; extending -> extending_error[label="fail", color=red4, fontcolor=red4]; unmanage_starting -> unmanage_error[label="fail", color=red4, fontcolor=red4]; manage_starting -> manage_error[label="fail", color=red4, fontcolor=red4]; snapshotting -> error[label="fail", color=red4, fontcolor=red4]; deleting -> error_deleting[label="fail", color=red4, fontcolor=red4]; }

REST API impact

New states will be visible through any API that shows states. Also new error conditions will become possible as we detect races earlier and report them directly.

The behavioral changes related to locking will not be microversioned, as it won’t be possible or desirable to emulate the old behavior once the changes are implemented. However in cases where new states are added, those changes will be microversioned so that clients which depend on the new states can detect that the server supports them.

Driver impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Distributed locking is expected to moderately slow down state changes. Also adding more state changes will slow down operations that require them.

Other deployer impact

Requirement to deploy and configure suitable Tooz backend. Since Manila will depend on tooz for correctness, tooz backends that fail to meet the API contract won’t be suitable.

Developer impact

This will be significant. Developers will need to follow the new model for all features that involve state changes. Also care will be needed with locks to avoid deadlock situations. Holding locks for very limited time will help avoid deadlocks but in case 2 locks are ever held at the same time, they need to be deadlock-proofed by establishing a lock order.

Implementation

Assignee(s)

bswartz

Work Items

Add snapshotting state
Complete tooz integration
Wrap state changes with tooz locks

Dependencies

Tooz

Testing

Existing tests will help ensure no regressions, but to detect race conditions we need rally tests or similarly high-concurrency tests.

Documentation Impact

Admin guide - need to document tooz requirements.

Developer reference - need to document state machines and locking protocol

References

Access rules spec: https://review.openstack.org/#/c/399049/

Tooz integration: https://review.openstack.org/#/c/318336/

Add new priority attribute for access rules

Wed, 08 May 2019 00:00:00

https://blueprints.launchpad.net/manila/+spec/add-priority-for-access-rule

Currently allow access rule API only supports access-level, which has “ro” and “rw” as valid values. This is not enough when users want to specify priority to individual access rules.

Problem description

The user is not able to reorder their access rules, and different drivers have different access rule sorts. Looking at the current behavior of share drivers, we can broadly classify them into three categories for our understanding:

The “last-rule” drivers add the access rules one by one, the latest access rule always has higher priority.
The “first-rule” drivers always let the prior access rules have higher priority.
The “most specific rule” drivers make sure individual addresses always sort higher than the subnets they are part of, and smaller subnets always sort above larger subnets.
Sometimes, after updating the access rules in driver, the access rules order won’t always be the same as the last time we update the rules.

For the moment Manila API of share access allow does not allow specifying extra attributes of access like “priority”.

It only makes sense for access rules which could possibly overlap. Which would include anything that uses IP-based access. Perhaps user-based access doesn’t need priorities. Although with groups, one could imagine overlap with user-based access too. So, if the user rules don’t need it, they don’t need to use the priority parameter when the user creates an access rule.

Use Cases

The user wants to set which access have higher priority to write or read the data of share, which access have lower priority to write or read the data of share. Such as: A 192.168.1.10 RW rule would be in effect against a 192.168.1.0/24 RO rule if the 192.168.1.10 RW one has higher priority. Then if the user wants to hide the 192.168.1.10 RW rule, they could simply set it to lower priority.

Proposed change

The following are the changes to be made:

A new parameter ‘priority’ will be added to access-allow API parameters.
The ‘priority’ value supports an integer value in the range 1-200, where a lower number means a higher priority. This means the number ‘1’ is the highest priority. The maximum value of ‘priority’ is set to 200. We never need more priorities than there are bits in a netmask for ip-based rules. Because 2 different rule with the same netmask are either for the same address range or nonconflicting, and the prefix length can be all the way long to 128 in manila access rule allow API, it could add 129 different conflicting access rules. Since we don’t treat 10.1.0.1/32 and 10.1.0.1 as conflicts, so perhaps even a few more different conflicting access rules. For non-IP-based rules, such as user-based access, the number of possible conflicts has to do with the number of groups a user could be in, and it seems unlikely that a user would be a member of more than 200 groups all of which had different levels of accesses to the same share. The range is not for different access rules, just for overlapping ones, If we leave it unbound people will wonder about negative numbers and numbers maybe higher than 4.2 billion, so the validation is important and the number 200 will result in better usability.
The access rules will be sorted through the ‘priority’ value in the share manager before they are sent down to the driver. The rules with highest priority will be stored in the front of the access rule list that will be sent down to the driver.
- If two of more rules conflict and have the same priority, then the behavior is undefined. Where possible, the behavior in this case should remain unchanged from previous manila versions, where the behavior in this case is also undefined.
Add the new API to support modifying the access rule ‘priority’. Modifying rule priority results in the manager invoking update access method in the driver.
The Manila API terminology treats ‘deny’ the access rules as rules removal and not in the classic ACL deny (where some IPs/users can access a resource and some are denied from it/just not allowed). So prioritizing ‘deny’ means prioritizing rule removal, thus doesn’t mean anything. The deny access rule API and command won’t be changed.

Alternatives

Instead of adding a separate parameter, we could overload the “access_to” parameter to allow specifying the priority by using ‘#’ as a separator. We will parse the value of access_to in driver. such as: “manila access-allow test_share_id user#priority=1”.
Prioritization only matters to rules whose clients can overlap. The most important use case of this is with NFS rules to client IP addresses. So, instead of allowing user-defined priorities, we could adopt a behavior in manila where access rules are always sorted in the share manager in the following order: Rules for Individual IP addresses sort higher than those for subnets containing them, and smaller subnets sort higher than larger subnets containing these smaller subnets. For example: If the user allows access in the following way:
1. allow ‘ro’ access to 192.168.17.16
2. allow ‘rw’ access to 192.168.17.0/22
3. allow ‘ro’ access to 192.168.17.0/24
4. allow ‘rw’ access to 192.160.16.15
The priority that manila ensures would be as follows:
```
+----------+-----------------+--------------+
| Priority |    Access To    | Access Level |
+----------+-----------------+--------------+
|        1 | 192.168.17.16   | ro           |
|        2 | 192.160.16.15   | rw           |
|        3 | 192.168.17.0/24 | rw           |
|        4 | 192.168.17.0/22 | ro           |
+----------+-----------------+--------------+
```
- In this way, if we want to disabled the Individual IP addresses rules or smaller subnets access rules, we have to delete those access rules. For example, if I want to force all my shares to read only for a short period while I fix something, but I don’t want to have to delete all my rules out of manila to achieve that.
- This way will also make upgrades a bit harder for backwards compatibility. Because the access rules will have the different order after upgrades to the new version.

Data model impact

The manila.db.sqlalchemy.models.ShareInstanceAccessMapping model will have a new field called priority.

The priority of all pre-existing rules will be set to 100 after upgrading manila.

REST API impact

The new parameter will be added in access API. We will bump the micro-version to expose the ‘priority’ parameter:

Adding an access rule

POST /v2/{tenant-id}/shares/{share-id}/action BODY:
{
    'allow_access': {
            "access_level": "rw",
            "access_type": "ip",
            "access_to": "0.0.0.0/0",
            "priority": "1"
    }
}
The “priority” is an optional parameter. If the user doesn’t input it, it will be set to 1.

Updating access rules

PATCH /v2/{tenant-id}/share-access-rules/{access-id} BODY:
{
    "priority": "1"
}

Listing access rules

GET /v2/{project_id}/share-access-rules?share_id={share-id}&sort_dir=desc&sort_key=priority Response:

{
    "accesses": [
        {
                "access_level": "rw",
                "state": "active",
                "id": "507bf114-36f2-4f56-8cf4-857985ca87c1",
                "access_type": "cert",
                "access_to": "example.com",
                "access_key": null,
                "priority": "1",
        },
        {
                "access_level": "rw",
                "state": "error",
                "id": "329bf795-2cd5-69s2-cs8d-857985ca3652",
                "access_type": "ip",
                "access_to": "10.0.0.2",
                "access_key": null,
                "priority": "3",
        },
    ]
}

Adding the “priority” field in a micro-version change to this new API. The “share_id” is a mandatory query key, and the API will respond with HTTP 400 if the “share_id” is not provided. Adding “sort_dir” and “sort_key” filter in list API. The “sort_dir” means sort direction, and the value of “sort_dir” should be ‘desc’ or ‘asc’.

Note

The current access rules list API accepts HTTP POST requests. To ensure correct HTTP semantics around idempotent and safe information retrieval, we will introduce a new API that accepts GET requests. The old API will be capped with a maximum micro-version, i.e, it will not be available from the micro-version that this new API is introduced within.

Security impact

None

Notifications impact

Add the “priority” field in user error notifcations when we create an access rule or update the access rule.

Other end user impact

The Manila client, CLI will be extended to support access rule priority.

The access-allow command with access priority supported will be like:

manila access-allow [--priority <priority>]
                    [--access-level <access_level>]
                    <share> <access_type> <access_to>

Optional arguments:

--priority  The 'priority' value supports an integer value in the
            range 1-200, where a lower number means a higher priority.
            The default value is set to '100'.

The new access-update command with priority supported will be like:

manila access-update <access> [--priority <priority>]

Optional arguments:

--priority  The 'priority' value supports an integer value in the
            range 1-200, where a lower number means a higher priority.
            OPTIONAL: Default=None.

The access-list command with access priority sorting supported will be like:

manila access-list [--columns <columns>] [--sort-key <sort_key>]
                   [--sort-dir <sort_dir>]
                   <share>

Optional arguments:
--sort-dir <sort_dir>, --sort_dir <sort_dir>
                    Sort direction, available values are ('asc', 'desc').
                    OPTIONAL: Default=None.
--sort-key <sort_key>, --sort_key <sort_key>
                    Key to be sorted, available keys are (priority).
                    OPTIONAL: Default=None.

We will also perform client side validation for value of “priority” limit range from 1 to 200.

Performance impact

Sorting access rules have a negative service performance impact.

Other deployer impact

None

Developer impact

None

Driver impact

The access rules will be ordered in the share manager before being sent down to drivers in update_access function, so the drivers need not see the priority field and need not change their current behavior. If some drivers are already reordering the rules, we will audit drivers that are reordering rules and report bugs right away. They have to be updated to comply with the order determined by the share manager. If your back end can’t correctly implement a broad rule overriding a more narrow rule when the broad rule is earlier in the list, then the driver must drop the more narrow rule and not send it to the backend at all.

Implementation

Assignee(s)

Primary assignee:: zhongjun

Work Items

Add priority property to access rule object and bump the API microversion.
Add a new parameter in “access_rules” table and add db upgrade script.
Add a new update-access API.
Add new unit and tempest tests for access rule priority.
Update python-manilaclient and the UI, Allow the end user to move rules around in a list and figure out the priorities, like other public clouds do [1].

Dependencies

None

Testing

Add the unit tests
Add the tempest tests

Documentation Impact

The following OpenStack documentation will be updated to reflect this change:

OpenStack User Guide
OpenStack API Reference
Manila Developer Reference
The Admin Guide

References

Support for access rule priority in Alibaba Cloud:
- [1] https://www.alibabacloud.com/help/doc-detail/27534.htm?spm=a3c0i.o27518en.b99.23.371c253bJ2y4HY
Support for access rule priority in Tencent Cloud:

https://intl.cloud.tencent.com/document/product/582/13778

Stochastic Weighing Scheduler

Wed, 08 May 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/manila/+spec/stochastic-weighing-scheduler

The filter scheduler is the de-facto standard scheduler for Manila and has a lot of desirable properties. However there are certain scenarios where it’s hard or impossible to get it to do the right thing. I think some small tweaks could help make admin’s lives easier.

Problem description

I’m concerned about 2 specific problems:

When there are multiple backends that are not identical, it can be hard to ensure that load is spread across all the backends. Consider the case of a few “large” backends mixed with some “small” backends. Even if they’re from the same vendor by default new shares will go exclusively to the large backends until free space decreases to the same level as the small backends. This can be worked around by using something other than free space to weigh hosts, but no matter what you choose, you’ll have a similar issue whenever the backends aren’t homogeneous.
Even if the admin is able to ensure that all the backends are identical in every way, at some point the cloud will probably need to grow, by adding new storage backends. When this happens there will be a mix of brand new empty backends and mostly full backends. No matter what kind of weighing function you use, initially 100% of new requests will be scheduled on the new backends. Depending on how good or bad the weighing function is, it could take a long time before the old backends start receiving new requests and during this period system performance is likely to drop dramatically. The problem is particularly bad if the upgrade is a small one: consider adding 1 new backend to a system with 10 existing backends. If 100% of new shares go to the new backend, then for some period, there will be 10x load on the single backend.

There is one existing partial solution to the above problems – the goodness weigher – but that has some limitations worth mentioning. Consider an ideal goodness function – an oracle that always returns the right value such that the best backend for new shares is sorted to the top. Because the inputs to the goodness function (other than available space) are only evaluated every 60 seconds, bursts of creation requests will nearly always go to the same backend within a 60 second window. While we could shrink the time window of this problem by sending more frequent updates, that has its own costs and also has diminishing returns. In the more realistic case of a goodness function that’s non-ideal, it may take longer than 60 seconds for the goodness function output to reflect changes based on recent creation requests.

Use Cases

The existing scheduler handles homogeneous backends best, and relies on a relatively low rate of creation requests compared to the capacity of the whole system, so that it can get keep up to date information with which to make optimal decisions. It also deals best with cases when you don’t add capacity over time.

I’m interested in making the scheduler perform well across a broad range of deployment scenarios:

Mixed vendor scenarios
A mix of generations of hardware from a single vendor
A mix of capacities of hardware (small vs. large configs)
Adding new capacity to a running cloud to deal with growth

These are all deployer/administrator concerns. Part of the proposed solution is to enable certain things which are impossible today, but mostly the goal is to make the average case “just work” so that administrators don’t have to babysit the system to get reasonable behavior.

Proposed change

Currently the filter scheduler does 3 things:

Takes a list of all pools and filters out the ones that are unsuitable for a given creation request.
Generates a weight for each pool based on one of the available weighers.
Sorts the pools and chooses the one with the highest weight.

I call the above system “winner-take-all” because whether the top 2 weights are 100 and 0 or 49 and 51, the winner gets the request 100% of the time.

I propose renaming the existing HostWeightHandler to OrderedHostWeightHandler and adding a new weight handler to the filter scheduler called StochasticHostWeightHandler. The OrderedHostWeightHandler would continue to be the default and would not change behavior. The new weight handler would be implement different behavior as follows:

In step 3 above, rather than simply selecting the highest weight, the weight handler would sum up the weight of all choices, assign each pool a subset of that range with a size equal to that host’s weight, then generate a random number across the whole range and choose the pool mapped to that range.

An easier way to visualize the above algorithm is to imagine a raffle drawing. Each pool is given a number of raffle tickets equal to the pool’s weight (assume weights normalized from 0-100). The winning pool is chosen by a raffle drawing. Every creation request results in a new raffle being held.

Pools with higher weights get more raffle tickets and thus have a higher chance to win, but any pool with a weight higher than 0 has some chance to win.

The advantage to the above algorithm is that it distinguishes between weights that are close (49 and 51) vs weights that are far (0 and 100) so just because one pools is slightly better than another pool, it doesn’t always win. Also, it can give different results within a 60 second window of time when the inputs to the weigher aren’t updated, significantly decreasing the pain of slow share stats updates.

It should be pointed out that this algorithm not only requires that weights are properly normalized (the current goodness weigher also requires this) but that the weight should be roughly linear across the range of possible values. Any deviation from linear “goodness” can result in bad decisions being made, due to the randomness inherent in this approach.

Alternatives

There aren’t many good options to deal with the problem of bursty requests relative to the update frequency of share stats. You can update stats faster but there’s a limit. The limit is to have the scheduler synchronously request absolute latest share stats from every backend for every request. Clearly that approach won’t scale.

To deal with the heterogeneous backends problem, we have the goodness function, but it’s challenging to pick a goodness function that yields acceptable results across all types of variation in backends. This proposal keeps the goodness function and builds upon it to both make it stronger, and also more tolerant to imperfection.

Data model impact

No database changes.

REST API impact

No REST API changes.

Security impact

No security impact.

Notifications impact

No notification impact.

Other end user impact

End users may indirectly experience better (or conceivably worse) scheduling choices made by the modified scheduler.

Performance Impact

No performance impact. In fact this approach is proposed expressly because alternative solutions would have a performance impact and I want to avoid that.

Other deployer impact

I propose a single new weigher class for the scheduler. The default weigher would continue to be the existing weighter. An administrator would need to intentionally modify the weigher class config option to observe changed behavior.

Developer impact

Developers wouldn’t be directly impacted, but anyone working on goodness functions or other weighers would need to be aware of the linearity requirement for getting good behavior out of this new scheduler mode.

In order to avoid accidentally feeding nonlinear goodness values into the stochastic weighing scheduler, we may want to create alternatively-named version of the various weights or weighers, forcing driver authors to explicitly opt-in to the new scheme and thus indicate that the weights they’re returning are suitably linear.

Implementation

Assignee(s)

Primary assignee:: bswartz

Work Items

This should be doable in a single patch.

Dependencies

Filter scheduler (manila)
Goodness weigher (manila)

Testing

Testing this feature will require a multibackend configuration (otherwise scheduling is just a no-op).

Because randomness is inherently required for the correctness of the algorithm, it will be challenging to write automated functional test cases without subverting the random number generation. I propose that we rely on unit tests to ensure correctness because it’s easy to “fake” random numbers in unit tests.

Documentation Impact

Dev docs need updated to explain to driver authors what the expectations are for goodness functions.

Config ref needs to explain to deployers what the new config option does.

References

This spec is a copy of any idea accepted by the Cinder community: https://github.com/openstack/cinder-specs/blob/master/specs/newton/stochastic-weighing-scheduler.rst

API Validation

Tue, 23 Apr 2019 00:00:00

https://blueprints.launchpad.net/manila/+spec/json-schema-validation

Currently, Manila has different implementations for validating request bodies. The purpose of this blueprint is to track the progress of validating the request bodies sent to the Manila server, accepting requests that fit the resource schema and rejecting requests that do not fit the schema. Depending on the content of the request body, the request should be accepted or rejected consistently.

Problem description

Currently Manila doesn’t have a consistent request validation layer. Some resources validate input at the resource controller and some fail out in the backend. Ideally, Manila would have some validation in place to catch disallowed parameters and return a validation error to the user.

The end user will benefit from having consistent and helpful feedback, regardless of which resource they are interacting with.

Use Cases

As a user or developer, I want to observe consistent API validation and values passed to the Manila API server.

Proposed change

One possible way to validate the Manila API is to use jsonschema similar to Nova, Cinder, Keystone and Glance (https://pypi.python.org/project/jsonschema). A jsonschema validator object can be used to check each resource against an appropriate schema for that resource. If the validation passes, the request can follow the existing flow of control through the resource manager to the backend. If the request body parameters fails the validation specified by the resource schema, a validation error wrapped in HTTPBadRequest will be returned from the server.

Example: “Invalid input for field ‘name’. The value is ‘some invalid name value’.

Each API definition should be added with the following ways:

Create definition files under ./manila/api/schemas/.
Each definition should be described with JSON Schema.
Each parameter of definitions(type, minLength, etc.) can be defined from current validation code, DB schema, unit tests, Tempest code, or so on.

Some notes on doing this implementation:

Common parameter types can be leveraged across all Manila resources. An example of this would be as follows:

# share create schema

manila/api/schema/shares.py:

from manila.api.validation import parameter_types
create_v231 = {
    'type': 'object',
    'properties': {
        'share': {
            'type': 'object',
            'properties': {
                'description': parameter_types.description,
                'share_type': {
                    'format': 'uuid'
                },
                'share_proto': parameter_types.proto
                'share_network_id': {
                    'format': 'uuid'
                },
                'share_group_id': {
                    'format': 'uuid'
                },
                'name': parameter_types.name,
                'snapshot_id': {
                    'format': 'uuid'
                },
                'size': parameter_types.positive_integer,
                'metadata': {
                    'type': 'object'
                },
            },
            'required': ['size'],
            'additionalProperties': False,
        }
        'required': ['share'],
        'additionalProperties': False,
    }
}

manila/api/validation/parameter_types.py:

description = {
    'type': 'string', 'minLength': 0, 'maxLength': 255,
}

name = description

# This registers a FormatChecker on the jsonschema module.
# It might appear that nothing is using the decorated method but it gets
# used in JSON schema validations to check uuid formatted strings.
from oslo_utils import uuidutils

@jsonschema.FormatChecker.cls_checks('uuid')
def _validate_uuid_format(instance):
    return uuidutils.is_uuid_like(instance)

positive_integer = {
    'type': ['integer', 'string'],
    'pattern': '^[0-9]*$', 'minimum': 1, 'minLength': 1
}

proto = {
    'type': ['string'],
    'enum': ['NFS', 'CIFS', 'GlusterFS', 'HDFS', 'CephFS']
}

The validation can take place at the controller layer using below decorator:

manila/api/v2/shares.py:

from manila.api.schemas import share

@wsgi.Controller.api_version("2.31")
@validation.schema(shares.create_v231)
def create(self, req, body):
    """creates a share."""

Initial work will include capturing the Shared File API Spec for existing resources in a schema. This should be a one time operation for each major version of the API. This will be applied to the Shared File V2 API.
When adding a new extension to Manila, the new extension must be proposed with its appropriate schema.

Alternatives

Before the API validation framework, we need to add the validation code into each API method in ad-hoc. These changes would make the API method code messy and we needed to create multiple patches due to incomplete validation.

If using JSON Schema definitions instead, acceptable request formats are clear and we don’t need to do ad-hoc works in the future.

Pecan Framework: Pecan Some projects(Ironic, Ceilometer, etc.) are implemented with Pecan/WSME frameworks and we can get API documents automatically from the frameworks. In WSME implementation, the developers should define API parameters for each API. Pecan would make the implementations of API routes(URL, METHOD) easy.

Data model impact

None

REST API impact

In Manila, since Liberty, we strive to maintain complete API backwards-compatibility. So, if we have a client that’s talking to different manila releases, they should experience the same behavior if using the same API version. We break this rule rarely, but have consistently maintained that if we were correcting buggy behavior, we will consider breaking backwards-compatibility.

API Response code changes:

There are some occurrences where API response code will change while adding schema layer for them. For example, On current master ‘services’ table has ‘host’ and ‘binary’ of maximum 255 characters in database table. While updating service user can pass ‘host’ and ‘binary’ of more than 255 characters which obviously fails with 404 ServiceNotFound wasting a database call. For this we can restrict the ‘host’ and ‘binary’ of maximum 255 characters only in schema definition of ‘services’. If user passes more than 255 characters, he/she will get 400 BadRequest in response.

API Response error messages:

There will be change in the error message returned to user. For example, On current master if user passes more than 255 characters for share name then below error message is returned to user from manila-api:

Invalid input received: name has <actual no of characters user passed> characters, more than 255.

With schema validation below error message will be returned to user for this case:

Invalid input for field/attribute name. Value: <value passed by user>. ‘<value passed by user>’ is too long.

Security impact

The output from the request validation layer should not compromise data or expose private data to an external user. Request validation should not return information upon successful validation. In the event a request body is not valid, the validation layer should return the invalid values and/or the values required by the request, of which the end user should know. The parameters of the resources being validated are public information, described in the Shared File API spec, with the exception of private data. In the event the user’s private data fails validation, a check can be built into the error handling of the validator to not return the actual value of the private data.

jsonschema documentation notes security considerations for both schemas and instances: http://json-schema.org/latest/json-schema-core.html#anchor21

Better up front input validation will reduce the ability for malicious user input to exploit security bugs.

Notifications impact

None

Other end user impact

None

Performance Impact

Manila will pay some performance cost for this comprehensive request parameters validation, because the checks will be increased for API parameters which are not validated now.

Other deployer impact

None

Developer impact

This will require developers contributing new extensions to Manila to have a proper schema representing the extension’s API.

Implementation

Assignee(s)

Primary assignee: dongdongpei (Dongdong Pei <peidongdong121@163.com>)

Other contributors: wosunoozzy (Yang Zhang <wosunoozzy@gmail.com>)

Work Items

Initial validator implementation, which will contain common validator code designed to be shared across all resource controllers validating request bodies.
Introduce validation schemas for existing core API resources.
Introduce validation schemas for existing API extensions.
Enforce validation on proposed core API additions and extensions.
Remove duplicated ad-hoc validation code.
Add unit and end-to-end tests of related API’s.
Add/Update Manila documentation.

Dependencies

The code under manila/api/v1 are getting called by v2. So if we add schema validation for v2 then we will have to remove the existing validation of parameters which is there inside of controller methods which will again break the v2 apis.

Solution:

Do the schema validation for v2 apis using the @validation.schema decorator similar to Nova and also keep the validation code which is there inside of method to keep v1 working.
Once the decision is made to remove the support to v1 we should remove the validation code from inside of method.

Testing

Tempest tests can be added as each resource is validated against its schema. These tests should walk through invalid request types.

We can follow some of the validation work already done in the Nova V3 API:

Negative validation tests should use tempest.test.NegativeAutoTest

Documentation Impact

The Manila API documentation will need to be updated to reflect the REST API changes.
The Manila developer documentation will need to be updated to explain how the schema validation will work and how to add json schema for new API’s.

References

Understanding JSON Schema:

https://json-schema.org/understanding-json-schema/index.html
Nova Validation Examples:

https://opendev.org/openstack/nova/src/branch/master/nova/api/validation
JSON Schema on PyPI:

https://pypi.python.org/project/jsonschema
JSON Schema core definitions and terminology:

https://tools.ietf.org/html/draft-zyp-json-schema-04
JSON Schema Documentation:

https://json-schema.org/specification.html

Share servers with multiple subnets

Sun, 25 Nov 2018 00:00:00

https://blueprints.launchpad.net/manila/+spec/multiple-subnet-share-servers

This spec proposes changes to Manila with the purpose of addressing the need for having share servers with network allocations on multiple subnets. To do so, the spec is proposing allowing share network to span multiple subnets in the same AZ.

Problem description

One of two modes that a manila driver can operate in, driver_handles_share_servers = True, requires a share server to be provisioned in a neutron subnet that can be either IPv4 or IPv6. The share server provisioned there has its shares accessible from hosts connected to same subnet. When IPv6 support was added to Manila, it allowed for new use cases and some limitations became apparent, namely that a share server could not serve shares with IPv4 and IPv6 export locations at the same time.

In a world that has transitioning infrastructure from IPv4 to IPv6, it is no longer unusual to expect connectivity in both address families. The fundamental concept of file services is to have data shared across multiple hosts, thus it is sensible to expect shares to be accessible from both IPv4 and IPv6 address families simultaneously as well, as storage back ends are capable of accomplishing that.

Manila drivers implement support for share servers by receiving a list of network allocations that the back end should create. The amount of network allocations supplied varies by driver, as each driver reports the amount it needs before the allocation definitions are obtained from Neutron.

Prior to the implementation of [1], the share network entity held information from Neutron networks and subnets, and Manila referred to its data when trying to dynamically allocate addresses. Since the implementation of [1], the share network entity was modified to do not hold Neutron information, it was instead modified to span multiple share network subnets in different availability zones. Since then, share servers are created and associated with a single share network subnet, which contains neutron network and subnet information.

A share network subnet is created by the cloud-user and its neutron network and subnet associations are also defined at creation time. No driver implementation was required to handle more than one subnet. The result is that, if more than the expected amount of allocations are provided, all drivers as currently implemented would fail to handle the additional allocations, which would be needed to meet the requirements for serving a share in both IP versions.

Since the implementation of [2] , Manila has the ability to change the share networks’ security services even if such networks were being used by share servers. The subnet associations still cannot be changed, which could result in scenarios where users end up creating subnets to just serve shares, while resources reside in other more flexible networks, having traffic routed between them with reduced performance.

Use Cases

By allowing share servers to be provisioned in more than one subnet, the following use cases can be addressed:

Ability to serve shares with both IPv4 and IPv6 export locations simultaneously.
Ability to serve shares in multiple subnets. In an overloaded environment, the Neutron subnet may be running out of IPs (to assign to clients), requiring a new subnet to client access, instead of:
- Having to route data between them.
- Having to create one share network for each subnet, which will result in spawning new share servers each with their own shares.
Update network allocations of existing share server by adding share network neutron subnet associations.

Proposed change

New boolean driver capabilities will be introduced:

``share_server_multiple_subnet_support``: determines whether the driver supports share server configuration on multiple subnets. If so, this capability is reported as True. Otherwise, the value is False.
``network_allocation_update_support``: determines whether the driver implements a new interfaces update_share_server_network_allocations and check_update_share_server_network_allocations. If so, this capability is reported as True. Otherwise, the value is False.

The purpose of the new capability share_server_multiple_subnet_support is so that the scheduler can determine the proper back end when creating new shares in share networks that already have more than one subnet per AZ.

The new driver interface update_share_server_network_allocations enables drivers update share server network allocations whenever neutron subnets are added from share networks. Also, the interface must return the new share’s export locations, since they will change together with share server subnet change.

Before calling the update driver interface, the Manila will call the check_update_share_server_network_allocations to check that the update is available across all drivers that have servers in the share network. If one of the drivers deny the update, the entire update operation will fail. In this case, no resource goes to error state, only logging why the operation was denied.

A new boolean column network_allocation_update_support will be added to the share server model to distinguish existing share servers that can support the update functionality. It will inherit from its driver capability.

The workflows affected by this proposal are:

When adding share network subnets to existing share network that results in more than one subnet per AZ (or more than one default subnet), the restriction of one subnet per AZ will only exist if there are any existing associated share servers that have network_allocation_update_support field set to False. In such case, the API call will fail with 409 Conflict. Otherwise, the API call will succeed and the share manager will be invoked to update the share server network allocations.
When creating shares in share networks with more than one subnet per AZ, the scheduler will attempt to find a valid back end that has share_server_multiple_subnet_support set to True. If that is not the case, the request will result in No valid host scheduling error.

It is possible for a driver to support these capabilities independent of each other. If a driver reports share_server_multiple_subnet_support=True and network_allocation_update_support=False, the share network will be immutable after the first share server has been provisioned on it. So if a cloud administrator wants both capabilities they must explicitly request each via share type extra specifications.

Existing share servers whose drivers implement neither interface will continue to work without changes. The API will never allow their associated share networks to have neutron subnets added if it results in more than one subnet per AZ. New shares will be allowed to be scheduled to them only when the associated share network does not have more than one neutron subnet per AZ.

If a driver decides to implement update_share_server_network_allocations in later releases, after the network_allocation_update_support field has already been set to either True or False, the share server’s network_allocation_update_support field will need to be reset using the manila-manage tool.

Share network updates can be an one-to-many operation, which might trigger multiple share server updates, for different back ends in different share services. To add a subnet associated with a share network, the share API will need to validate if all affected resources are healthy before proceeding with the request. After that, both share network and share servers status will be updated to network_change.

If the share server setup using multiple subnet fails to configure the server for one of the given subnets, the entire setup will fail. Likewise, if the update_share_server_network_allocations fails, the share server status will be set to error along with all affected shares access rules. The network status won’t be affected by errors raised by back end drivers, share network will be active and its resources will be updated (subnets added).

Manila Manage

Alternatives

Since there is no way to address the use cases introducing changes in manila while re-using the same driver interfaces, the only alternative is to not implement the proposed changes and continue with the limitations.

Data model impact

A new network_allocation_update_support capability column will be added to manila.db.sqlalchemy.models.ShareServer indicating if the driver and the back end where this share server resides support the update multiple subnet per AZ operations. In database migration upgrade, the new column will be added with a default value set to False, meaning that all share servers already deployed won’t be able to update subnets per AZ configuration even if the driver supports it. In database migration downgrade the column will be dropped.

Given that the ShareServer can have multiple ShareNetworkSubnet and the ShareNetworkSubnet can also be in multiple ShareServer, a new table ShareServerShareNetworkSubnetMapping will be added. The field share_network_subnet_id in the ShareServer will be removed. The database migration upgrade will build this new table based on the current ShareServer table and remove the field. Downgrade is not recommended, though, since the mapping could have more than one entry per ShareServer.

A new network_allocation_update_support property will be added to manila.db.sqlalchemy.models.ShareNetwork to indicate whether a share network supports or not the update multiple subnet per AZ operations. This property will inherited its value from all current associated share servers. If all associated share servers support network_allocation_update_support, the share network property will be set to True, otherwise it will be set to False.

Since the current Manila data model design has the entity manila.db.sqlalchemy.models.ShareNetworkSubnet containing the foreign key availability_zone_id, not the inverse, nothing is blocking the data model to have multiple subnets in the same AZ.

REST API impact

There are no new API introduced, but the changes to the existing APIs will be microversioned. Namely:

Displaying a share server: after the microversion bump, the share server view will include the field network_allocation_update_support. Also, the share_network_subnet_id will be a list of IDs.
Adding a neutron subnet to a share network: after the microversion bump, this API will no longer always return an error when the operation result would be more than one subnet per AZ. Instead, it will check if the existing associated share servers have network_allocation_update_support field set to True. If that is not the case, it will return 409 Conflict.

Security impact

None.

Notifications impact

There are no new notifications introduced. The existing notifications already accommodate the changes proposed.

Other end user impact

None other than the restriction changes when adding or removing subnets from a share network.

Performance impact

None.

Other deployer impact

No new configurations are expected to be added.
The back end capability will help deployers to identify pools that support multiple subnet share server support.
All existing share servers will have their network_allocation_update_support set to False, even if the driver supports it. New share servers will have the correspondent capability set according to the back end capability reported by the drivers. Administrators will need to manually update share server’s capability using manila manage commands.

Developer impact

None.

Driver impact

Drivers that wish to support the update functionality must implement the new check and update driver interfaces:

def check_update_share_server_network_allocations(
    self, network_info, server_details, shares, snapshots):
"""Updates a share server's network allocations.

:param network_info: Dictionary containing network parameters for share
    server update, with the map of network allocations and security
    services among them.
:param server_details: Back end details for share server being updated.
:param shares: All shares in the share server.
:param snapshots: All snapshots in the share server.
:return Boolean indicating whether the update is possible or not. It is
    the driver responsibility to log the reason why not accepting the
    update.

def update_share_server_network_allocations(
    self, network_info, server_details, shares, snapshots):
"""Updates a share server's network allocations.

:param network_info: Dictionary containing network parameters for share
    server update, with the map of network allocations and security
    services among them.
:param server_details: Back end details for share server being updated.
:param shares: All shares in the share server.
:param snapshots: All snapshots in the share server.
:return If the update changes the shares export locations or snapshots
        export locations, this method should return a dictionary containing
        a list of share instances and snapshot instances indexed by their
        id's, where each instance should provide a dict with the relevant
        information that need to be updated. Also, the returned dict
        contains the updated back end details to be saved in the database.

        Example::

            {
                'share_updates':
                {
                    '4363eb92-23ca-4888-9e24-502387816e2a':
                    {
                    'export_locations':
                    [
                        {
                        'path': '1.2.3.4:/foo',
                        'metadata': {},
                        'is_admin_only': False
                        },
                        {
                        'path': '5.6.7.8:/foo',
                        'metadata': {},
                        'is_admin_only': True
                        },
                    ],
                    'pool_name': 'poolA',
                    },
                },
                'snapshot_updates':
                {
                    'bc4e3b28-0832-4168-b688-67fdc3e9d408':
                    {
                    'provider_location': '/snapshots/foo/bar_1',
                    'export_locations':
                    [
                        {
                        'path': '1.2.3.4:/snapshots/foo/bar_1',
                        'is_admin_only': False,
                        },
                        {
                        'path': '5.6.7.8:/snapshots/foo/bar_1',
                        'is_admin_only': True,
                        },
                    ],
                    },
                    '2e62b7ea-4e30-445f-bc05-fd523ca62941':
                    {
                    'provider_location': '/snapshots/foo/bar_2',
                    'export_locations':
                    [
                        {
                        'path': '1.2.3.4:/snapshots/foo/bar_2',
                        'is_admin_only': False,
                        },
                        {
                        'path': '5.6.7.8:/snapshots/foo/bar_2',
                        'is_admin_only': True,
                        },
                    ],
                    },
                }
                'backend_details':
                {
                    'new_share_server_info_key':
                        'new_share_server_info_value',
                },
            }
    Example::

        {'server_name': 'my_share_server'}

    """
    raise NotImplementedError()

Drivers should expect to receive multiple network allocations. The total number will be the number of subnets associated with the same AZ multiplied by the number of allocations reported by the driver.

The pre-existent driver interface _setup_server will be modified. The network allocations dictionary entry network_info will now be a list of dictionary, representing allocations in each subnet. To keep the compatibility, the drivers will be changed to consume the first element of the list and an email informing this interface change will be sent in the openstack email list.

Implementation

Assignee(s)

Primary assignee:: felipe_rodrigues

Work Items

Implement core changes that must include:
- Add share network and share server model attributes;
- Add database migration.
- Update Add Share network neutron subnet API validation based on new Share Server model field.
- Add new driver interfaces.
- Add new capability.
- Add scheduler capability check.
Implementation in a first party driver
Add new functional test in manila-tempest-plugin
Add new command to manila-manage for share server capability update
Update manila documentation

This spec may require more than one release to be delivered covering all use cases. So, it can be splitted in two major deliverables, being:

Add ability to define multiple subnets in the same share network AZ
Add ability to “update” subnets in a share network AZ

This deliverable approach would split the test and validation effort a long the releases.

Dependencies

None.

Testing

Unit test coverage will be added/maintained as per community standards. New tempest tests will be added to cover multiple subnets per AZ scenarios. The container or the dummy driver will be improved to properly configure security services and be used to validate the proposed changes..

Documentation Impact

The following OpenStack documentation will be updated to reflect this change:

OpenStack User Guide: will document the changes to Share Networks APIs.
OpenStack Admin Guide: will document the changes to Share Servers.
OpenStack API Reference: All API changes will be documented.
Manila Developer Reference: the low level implementation considerations, feature design and guidelines for driver implementation will be documented.

References

[1]: https://specs.openstack.org/openstack/manila-specs/specs/train/share-network-multiple-subnets.html

[2]: https://specs.openstack.org/openstack/manila-specs/specs/wallaby/security-service-updates-in-use-share-network.html

Storage Availability Zone improvements in Stein

Thu, 08 Nov 2018 00:00:00

https://blueprints.launchpad.net/manila/+spec/per-backend-availability-zones

https://blueprints.launchpad.net/manila/+spec/export-locations-az

https://blueprints.launchpad.net/manila/+spec/share-type-supported-azs

Manila supports the concept of storage availability zones (AZ) (configuration option: storage_availability_zone). A storage AZ is a string that provides a loose, flexible and conceptual representation of physical storage resources that share a common fate in terms of failure. Manila provides an API to discover what AZs have been configured. A user can request for their share to be scheduled to a specific AZ. Shares can also be replicated between AZs.

Availability zones are also used across other OpenStack services, such as nova and cinder, as a way of logically partitioning resources within an OpenStack deployment. In Nova, this partitioning can be motivated by the fact that these resources are meant for a particular task, or because they are co-located. However, the most common criterion is because these resources are associated with a common point of failure, such as being connected to a common power source. In the Mitaka release of OpenStack, neutron added support to availability zones. With this, it is effectively possible for users to allocate network resources to AZs for high availability.

As new use cases of OpenStack evolve, the concept of storage availability zones becomes more important to build features around. OpenStack’s use in Edge computing is built around the concept of stretch OpenStack clusters and split control planes spanning multiple “zones” in a single “region” [1]. It is important for manila to clearly define its availability zones construct so it can be used in an extensible manner in all use cases.

Problem description

This spec highlights three problems with the existing support for storage AZs:

The coupling between Service Availability and Storage Availability

Manila expects the storage_availability_zone config option to be specified in the [DEFAULT] section of the configuration file for the share manager service. At service startup, all back ends configured within the same configuration file register with the AZ specified. There are two instances where this is not desirable:

Service availability is not always the same thing as storage availability. The manila-share manager process can be run directly on dedicated “storage” nodes and this flavor is usually seen when using cloud-local storage such as LVM, ZFSOnLinux or cinder-backed block storage (Generic Driver). In such cases, it is expected that the service AZ is the same as storage AZ.

However, in many deployments, manila’s control plane is not in the same failure domain as the storage it manages. For example, consider an OpenStack cloud that has 3 “controller” nodes, and manila’s control plane (api, scheduler, share-manager and data processes) runs on all three controller nodes. This cloud can have a third party storage system that is external to the cloud as a manila back end. In this model, the share-manager process and its storage back end are in distinct failure domains. Typically, to achieve high availability of the control plane, cloud administrators run multiple copies of the manila share-manager service (e.g. on every OpenStack controller node) and orchestrate the high availability outside of manila with tools such as Pacemaker/Corosync or just by configuring each service to listen to the same service message queue channel/topic (configuration option: host).

The second limitation is the inflexibility in case of multi-backend. Consider a case of a deployment with both cloud-local storage and third-party external storage managed by the same share-manager service. The share-manager service forks a separate process to manage each back end. However, since the parent service consumes a single configuration file, administrators can only specify one storage_availability_zone for both back ends, even while that is truly not the case.

Discovering the right exports to use

This problem is specific to the user experience with replicated shares. There may sometimes be data path connectivity across AZs. However, more often than not, high bandwidth data networks (which shares are exported on) are local and distinct to AZs. Regardless, users must be able to discover which export path should be used within a given AZ when consuming a replicated share. Currently, users do not have a way to query manila for export locations that pertain to a specific AZ.

Use Cases

Cloud Administrators must be able to configure storage_availability_zone for each back end
Users must be able to discover export locations for a given availability zone
Users must be able to discover share types available within a given availability zone and availability zones supported by a given share type

Proposed change

Allow configuring AZ per back end

We will move the configuration option storage_availability_zone from the [DEFAULT] section to the driver specific section. Configuring storage_availability_zone in the [DEFAULT] section will be deprecated, but not unsupported, keeping in mind upgrade impact to the deployers. See corresponding cinder change here: [2]

Export Locations changes

The export locations API (GET shares/{share_id}/export_locations) will be modified to only present export locations from active replicas of a share. This API change will be micro-versioned. This implies that when the API is invoked on a non-replicated share, it will present all export locations of the share. Users of replicated shares can query export locations of replicas via a new API. The new API will present the export locations, their existing driver driven “metadata” and relevant replica information (replica ID, replica state, availability zone) to discern which exports are appropriate to be used. These API changes make the APIs leaner, but the UI will be designed to collate information from different APIs where necessary to present relevant information together.

Share Type changes

A new optional user/tenant-visible extra-spec will be created, called “availability_zones”. The default value of this extra-spec when not specified will be ‘*’, and this value is shown to users and administrators. Cloud administrators can override this default with a comma separated list of availability zones that the share type is supported within. Share types can be filtered with availability_zones by specifying a comma separated list of availability zones.

Note

Share types are mutable. Any changes to the extra-specs associated with a share type do not affect existing shares of that type. Cloud administrators can update the value of the availability_zones extra-spec at any time, but they must be aware that modifying tenant-visible extra-specs (without also modifying affected properties of pre-existing shares) may confuse end users.

AZ Scheduling changes

If an AZ is not chosen by the user to create a new share, the Availability Zone scheduler filter (manila.scheduler.filters.availability_zone .AvailabilityZoneFilter) will consider the availability_zones extra-spec to filter backends to scheduler the share.

See the corresponding changes to cinder’s volume type here [3] and here [4]

Workflows affected

Retrieving Share Types

The share types API will be modified to support availability_zones as a user-visible “optional” extra-spec. When not set, its value is set to “*” signifying that all availability zones are supported. This field can be filtered with one or more availability zones. Share types can also be filtered by availability_zones=* which will retrieve only those share types that support all availability zones. On the UI, if a share type is chosen, only a list of supported availability zones will be displayed and vice-versa.

Creating a share

The share API will validate the share type’s support of an availability zone if the share is requested to be created within an availability zone. If the share type does not support the requested availability zone, the API will return HTTP 400. When using the CLI, users typically list share types and availability zones and pick a share type and availability zone to invoke the manila create command. With this change, the manila share-type-list command will display the availability_zones extra-spec so they can make a wise choice. There will be no validation on the CLI with respect to the share-types and AZs to prevent a performance regression, the API will carry a clear error message that should suffice.

Retrieving export locations

Users will no longer be able to retrieve the export locations of replicas when using the export locations API. The share instance export locations API will not be altered, so consumers of that API will not be affected. The share replica export locations API can be used to retrieve details of all replica export locations for a given share, or export locations for a specific replica of a share or detailed export location information for a specific export.

Creating a share group

A share group can be created within a specified availability zone. The share group API will check whether the availability zone is supported within the share types used when specified. On the UI, when an availability zone is chosen to schedule a share group, the list of share types will be filtered based on support for that availability zone.

Alternatives

Currently, when administrators want to configure multiple availability zones, they configure multiple manila share-manager services, each with its own configuration file. This requires deployer side tooling and duplicates the effort of the share-manager service itself spawning processes to manage its multiple back ends.

Users currently have hacky/non-documented ways of figuring out if export locations are optimized (or even work) in a specific AZ. One approach is to rely on the possibility that export locations contains the share “instance” ID as a substring. However, this approach doesn’t work for non-replicated shares, since users (by virtue of default policy) cannot list share instances. They can only see IDs of replicas of a share (which are share instances under the hood). Another approach is to attempt to mount the share with each export location in the list, and sticking with the one that connects, or one that is the fastest (as determined by some user-driven test).

Currently share types being unavailable in specific AZs causes an asynchronous failure which can be diagnosed through user messages. We could live with this user experience.

Data model impact

No database schema changes are proposed. Therefore, no database migrations will be committed.

REST API impact

Please note the current state of these APIs in our API reference.

List Share types:

GET /v2/{tenant_id}/types?availability_zones=az1

Response:

Code: 200 OK

{
    "volume_types": [
        ..
    ],
    "share_types": [
        {
           "required_extra_specs": {
                "driver_handles_share_servers": "True"
            },
           "share_type_access:is_public": true,
           "extra_specs": {
                "driver_handles_share_servers": "True",
                "mount_snapshot_support": "False",
                "revert_to_snapshot_support": "False",
                "create_share_from_snapshot_support": "True",
                "snapshot_support": "True",
                "availability_zones": "az1,az4"
            },
            "id": "7fa1342b-de9d-4d89-bdc8-af67795c0e52",
            "name": "testing",
            "is_default": false,
            "description": "share type description"
        }
    ]
}

Get share type:

GET /v2/{tenant_id}/types/{share_type_id}

Response:

Code: 200 OK
{
    "share_type": {
        "required_extra_specs": {
            "driver_handles_share_servers": "True"
        },
        "share_type_access:is_public": true,
        "extra_specs": {
            "driver_handles_share_servers": "True",
            "mount_snapshot_support": "False",
            "revert_to_snapshot_support": "False",
            "create_share_from_snapshot_support": "True",
            "snapshot_support": "True",
            "availability_zones": "*"
        },
        "id": "2780fc88-526b-464a-a72c-ecb83f0e3929",
        "name": "default-share-type",
        "is_default": true,
        "description": "default share type"
    },
    "volume_type": {
        ..
    }
}

A similar schema change will be done for the following APIs:

GET /v2/{tenant_id}/types/default
GET /v2/{tenant_id}/types/{share_type_id}/extra_specs
POST /v2/{tenant_id}/types

List share export locations:

The change to this API is to remove non-active replica locations from the response schema:

GET /v2/{tenant_id}/shares/{share_id}/export_locations

Response:

Code: 200 OK
{
    "export_locations": [
        {
            "path": "10.254.0.3:/shares/share-e1c2d35e-fe67-4028-ad7a-45f668732b1d",
            "id": "b6bd76ce-12a2-42a9-a30a-8a43b503867d",
            "preferred": false,
        },
        {
            "path": "[db9f:6954::7766]:/shares/share-e1c2d35e-fe67-4028-ad7a-45f668732b1d",
            "id": "a16ef0be-9181-40af-a61c-7764816bc08d",
            "preferred": true,
        }
    ]
}

Share replica’s export locations:

GET /v2/{tenant_id}/share-replicas/{share_replica_id}/export_locations

Response:

Code: 200 OK
{
    "export_locations": [
        {
            "path": "10.254.0.3:/shares/share-8acb2fc3-7139-434f-8637-1ad7f49ee881",
            "share_replica_id": "8acb2fc3-7139-434f-8637-1ad7f49ee881",
            "replica_state": "in_sync",
            "id": "8da0a189-8365-4c28-919b-9f07c4f06c65",
            "preferred": false,
            "availability_zone": "northYVZ"
        },
        {
            "path": "[db9f:6954::7766]:/shares/share-e1c2d35e-fe67-4028-ad7a-45f668732b1d",
            "share_replica_id": "e1c2d35e-fe67-4028-ad7a-45f668732b1d",
            "replica_state": "in_sync",
            "id": "ad19597c-4d04-4869-af5f-c8173c2bcd51",
            "preferred": true,
            "availability_zone": "northYVZ"
        }
    ]
}

Share replica export location:

GET /v2/{tenant_id}/share-replicas/{share_replica_id}/export_locations/{export_location_id}

Response:

Code: 200 OK
{
    "export_location": {
            "path": "10.254.0.3:/shares/share-8acb2fc3-7139-434f-8637-1ad7f49ee881",
            "share_replica_id": "8acb2fc3-7139-434f-8637-1ad7f49ee881",
            "replica_state": "in_sync",
            "id": "8da0a189-8365-4c28-919b-9f07c4f06c65",
            "preferred": false,
            "availability_zone": "northYVZ",
            "created_at": 2018-11-27T22:54:32.000000,
            "updated_at": 2018-11-27T22:54:38.000000
        }
}

Security impact

None

Notifications impact

None

Other end user impact

If administrators do not configure the availability_zones extra-spec, no API change will be observed by the end user. CLI and UI impact are as follows:

manilaclient and CLI: manila show and manila share-export-location-list will no longer show export locations of non-active replicas of a given share. A new command will be added manila share-replica-export-location-list which will accept the replica ID as a parameter to retrieve the replica export locations. Users can also use the manila share-replica-export-location-show command along with the replica ID and export location ID to retrieve details about a specific replica export location. These commands will have corresponding manilaclient implementations that will allow users of the python package to retrieve share replica export locations.
manila UI: Users will only see active replica export locations on the UI in the share details page. The UI will use the newly created client integrations to invoke the share replica export locations API to retrieve export locations and display them in the share replica details page. This approach will resolve LP 1787016 [5].

Performance impact

Filtering the non-active replicas out of the export locations API is expected to add a slight/negligible performance regression since we will be performing a joined load of the export locations and the share instances associated with them.

Other deployer impact

storage_availability_zone will be deprecated from the [DEFAULT] group.

Developer impact

API microversion will be bumped to expose new functionality. Backwards compatibility will be strictly maintained.

Driver impact

There is no third-party driver change anticipated. The availability zone configuration changes will be done in a generic fashion within the base share driver. All other proposed changes do not affect share drivers directly.

Implementation

Assignee(s)

Primary assignee:: gouthamr

Work Items

Move config-opt storage_availability_zone to driver/back end sections
Modify the export locations API and introduce share replica export locations APIs.
Add support for configuring availability_zones in share types
Filter share types in the UI by availability zones and vice-versa during share and share group creation phases.

Dependencies

None

Testing

Unit test coverage will be added/maintained as per community standards. Tempest tests will be modified/added to cover new API changes. Allowing multiple AZs in a single multi-backend style configuration file will simplify test environment setup. The DevStack plugin already supports multi-backend, it can support multi-AZ without significant changes to either the plugin or the scripts invoking it.

The dummy driver and LVM driver job configuration on the gate will be modified to support multiple availability zones on a single share-manager service.

Documentation Impact

The following OpenStack documentation will be updated to reflect this change:

User Guide: document the changes in export location APIs, CLI and GUI and the changes to share types
Admin Guide: document the theory of supporting multiple availability zones per share-manager service.
API Reference: All API changes will be documented
Manila Developer Reference: the low level implementation considerations and design of this feature will be documented.

References

Manage / Unmanage with Share Servers

Tue, 02 Oct 2018 00:00:00

https://blueprints.launchpad.net/manila/+spec/manage-unmanage-with-share-servers

This spec proposes enhancement to manila’s Manage/Unmanage functionality so that drivers running in DHSS=True mode can:

import existing shares and snapshots, bringing them under manila’s management.
release shares and their snapshots from manila’s management without destroying them.

Problem description

Cloud administrators cannot bring pre-existing shares under manila’s management when those shares pertain to a back end driver operating in driver_handles_share_servers = True mode [1]. For brevity, we will refer to this mode as DHSS = True in this document. Since DHSS = True is the mode in which manila guarantees secure multi-tenant isolation, cloud-users end up having to make an unfortunate tradeoff, using their imported shares in DHSS = False mode.

We lack manage/unmanage for DHSS=True mode drivers today because of the following complexities:

Share server setup and networking: In DHSS = True mode, manila provisions and manages the lifecycle of share servers for each of the tenant’s networks. When creating the share servers, manila has to allocate network ports (and later de-allocate them when share servers are deleted). However, in DHSS = False mode the concept of share servers do not apply, it is up to the administrator and the driver to do whatever is necessary for the back end’s export locations to be accessible from the client hosts.
Managing shares: In DHSS = False, manila requests the driver to find a pre-existing share in the configured back end based on an export location given at the API call. Shares being managed are expected to be accessible from client hosts as any share created from manila would. In DHSS = True mode, any existing share would be in a share server, therefore to manage a share, one would have to manage its share server first and then manage the share within the share server.
Managing snapshots: Since currently there is no way to manage a share in manila in DHSS = True mode, there is no way to manage snapshots as well. Additionally, unmanaging of snapshots is currently disallowed.
Lifecycle of a share server: In DHSS = True mode, manila provisions the share servers and network resources associated with it, therefore it also deletes them when appropriate (like when the share server is not serving any shares). In DHSS = False, manila only manages the lifecycle of the resources it creates, such as shares, snapshots, replicas and access rules.

As per the aspects mentioned previously, it becomes clear that to manage a share or snapshot in DHSS = True mode, one needs an API to manage a share server first.

Use Cases

Cloud administrators that have been using storage devices to provision shares and their snapshots should have the ability to import the existing shares when migrating over to manila. Manila drivers that support DHSS = True mode could accomplish that if there was such API implemented in Manila.

Similarly, manila shares and snapshots could be unmanaged in DHSS = True mode to be migrated to another system or to have maintenance performed, but this would only make sense if they could be managed back.

Proposed change

The following changes are proposed:

Add a Manage Share Server API: Through this API, share servers shall be managed. Their state transitions will be similar to when managing shares. In other words, they will be created with the status manage_starting and transition either to available in case of success or to manage_error otherwise. The following parameters are expected to be supplied:
- Host: back end name (“<node_hostname>@<backend_stanza>”).
- Share Network: share network associated with the neutron network the share server is connected to.
- Identifier: a driver-specific share server identifier required by the driver to manage the share server.
- Driver Options: optional list of driver-specific key-value pairs that may be necessary to assist the driver managing the share server.
- Network allocations pre-requisites: Since connectivity is expected between an existing share server and the client hosts, the share server will already have interfaces with MAC and IP addresses but neutron and manila know nothing of them. Before taking the share server into manila’s management, the cloud administrator must create neutron ports corresponding to these share server interfaces, allocate the proper addresses to these ports, and set each of the port’s device_owner to manila:share. When managing the share server, the share server IP addresses will be requested by the driver and once retrieved, matched with neutron ports owned by manila:share present in the neutron subnet associated with the share network provided or configured network plugin. If not all allocations are found, the operation is aborted with an error. When using the Standalone Network Plugin, this step is not required since Neutron is not involved.
- Security services pre-requisites: Any pre-existing authentication services set up with share servers to be managed must be configured as security services associated with the share network before managing a share server. Manila offers no capability to update security services on existing manila provisioned share servers today. This specification does not add an ability to update security services on managed share servers.
- Implementation: When a request to manage a share server is received, the API parameters are validated and then a share server model with the is_imported field set is created. The share service that runs the back end stanza specified in the host specified is invoked through RPC, and the driver method manage_server is invoked to obtain the back end details and network allocations of the share server to be managed. The back end details returned are saved in the database and the network allocations retrieved are then passed to the configured network plugins to validate the network allocations previously created by the admin. After the validation, the network allocation is saved in manila’s network_allocations database table.
Add a Unmanage Share Server API: Through this API, share servers shall be unmanaged. No parameters are required beyond the ID of the share server to be unmanaged. The share server specified is removed from manila database. We are not going to remove the allocations, as for any existing untracked resource in a network, it will be bound to result in conflicts in the future if the ports are de-allocated. It will be up to the administrator to de-allocate the ports if desired. The state transitions will be similar to unmanaging a share, thus it would transition from available to unmanage_starting, then either to deleted in case of success or to unmanage_error otherwise.
- Implementation: When a request to unmanage a share server is received, the API parameters are validated and then the share service responsible by the given share server is invoked through RPC. The driver is then invoked to perform any operation that may be necessary to proceed with unmanaging, and finally the network allocations are deleted from manila’s network_allocations database table.
Add share_server_id parameter to Manage Share API: Whenever managing a share and passing a share type defined with driver_handles_share_servers set to True, the parameter share_server_id will be required, else the API will return 400 BadRequest. The Manage Snapshot API does not require this parameter since it will read it from the parent share’s model.
Allow unmanaging of shares and snapshots in ``DHSS = True`` mode: Currently it is not allowed to unmanage shares and snapshots that were created in this mode. We will change the API to allow it (and thus no longer return an error) when the newer microversion is specified. There are no behavioral changes required other than this one at the API layer.
Update driver interface of Manage/Unmanage Share and Snapshot to pass the share server: Since the existing implementation has never expected to work in DHSS = True driver mode, the driver interfaces do not include a share server parameter. The driver interfaces will be updated to receive the share server model to perform for manage and unmanage operations of shares and snapshots. The parameter is optional, so it will not affect existing DHSS = False driver implementations in any way.
Prevent automatic deletion of managed share servers: As opposed to share servers created by manila, we will not attempt to automatically delete managed share servers that have no manila shares, as there may be existing shares unknown to manila within it.
Manual deletion of managed share servers: Share servers managed by manila may contain existing shares and not all of those shares may be managed by manila. If admins decide to delete the share server, it will be up to the admin to try to delete it along with any existing shares that are unknown to manila. However, it will be up to the driver to allow the operation to succeed, as some back ends do not allow the share server to be deleted if there are remaining shares. If that is the case, the share server will go to error state and the admin will have to either unmanage it, or delete the remaining shares (that are unknown to manila) either manually in the back end or by managing them.

Alternatives

Instead of adding the Manage Share Server API, a previously discussed approach was a Manage Share API that would have two phases in DHSS = True mode. Such approach presents the following characteristics:

Same complexity and amount of technical work as the proposed solution, but done by a single API.
Several additional parameters that make sense only for DHSS = True mode.
More complex error handling, as there would need to be statuses and errors specific to each phase (managing a share server or managing a share within).
The user experience degrades when we have a single API behaving so differently, as the user can get confused when using it on both driver modes.

Alternatively to requiring the neutron ports to be created by the admin before managing a share server, we could have manila create the network allocations when managing the share servers, as this would be easier for the admin when doing a “bulk” manage of share servers. The downsides to this approach are:

It is reasonable to assume that the share servers we are managing are already connected to the share networks and accessible by hosts. Therefore, it is correct to also assume that a port in neutron should already exist to prevent the network allocations of those share servers from conflicting with other resources in the same subnet.
When unmanaging a share server, we would not remove the network allocations, since it is assumed these share servers may remain connected to the share networks and would need those network allocations to prevent conflicts with other resources in the same subnet. This behavior would be asymmetrical to creating the ports when managing share servers.

If resorting to not change the APIs, there is no way to import existing resources to be managed by manila. If this is not implemented, the administrators would need to manage their storage devices outside of manila, or accept the limitation of using only DHSS = False mode, lacking the benefits present in the DHSS = True.

Data model impact

Since there is the need to distinguish share servers created by manila from managed ones, we proposed the addition of a boolean column in the ShareServer table named is_imported defaulting to False. The database schema upgrade will add the column with the value False for all existing share servers, while the database schema downgrade will remove the column.

REST API impact

There are a couple of new APIs introduced and a few others changed. The policy for both new APIs are admin-only like the existing manage/unmanage APIs. There are no changes for the existing policies. The API microversion will be bumped to the next one for the listed changes.

Managing a share server:

POST /v2/{tenant-id}/share-servers/manage

Request parameters:

{
    "share_server": {
        "host": "host@backend",
        "share_network_id": "e76be4e9-4054-4df3-9e5c-178e68fb0949",
        "identifier": "0e73a5e1-e233-4635-b6df-db568307385f",
        "driver_options": {
            "key1": "value1",
            "key2": "value2",
            "key3": "value3"
        }
    }
}

Parameter driver_options is optional. If any of the other parameters is missing or invalid, the API will return 400 BadRequest.

Response:

Code: 202 Accepted

{
    "share_server":
        "status": "manage_starting",
        "created_at": 2018-09-17T18:05:34.000000,
        "updated_at": 2018-09-17T18:05:34.000000,
        "share_network_id": "e76be4e9-4054-4df3-9e5c-178e68fb0949",
        "share_network_name": "my_share_net",
        "host": "host@backend",
        "project_id": "0ebbe03068554da9b9d9ad11983bb08a",
        "id": "fa4e4d78-d3d9-46cf-8e61-514da5008cee",
        "is_imported": "True",
        "backend_details": {
            "key1": "value1",
            "key2": "value2",
            "key3": "value3"
        }
    }
}

Unmanaging a share server:

POST /v2/{tenant-id}/share-servers/{share_server_id}/action

Request parameters:

{
    "unmanage": null
}

Response:

Code: 202 Accepted

If the share server does not exist the API will return 404 NotFound.
If the share server status is not in error, active, inactive, manage_error or unmanage_error, the API will return 400 BadRequest.
If the share server has shares registered in manila, it will return 409 Conflict.

Displaying a share server:

After the microversion bump, the share server view will include the field is_imported whenever a newer microversion is used.

Managing a share:

POST /v2/{tenant-id}/shares/manage

Request parameters:

{
    "share": {
        "protocol": "NFS",
        "name": "my_share",
        "share_type": "my_type",
        "description": null,
        "driver_options": {},
        "is_public": false,
        "service_host": "host@backend#pool",
        "export_path": "192.168.10.100/my_export",
        "share_server_id": "fa4e4d78-d3d9-46cf-8e61-514da5008cee"
    }
}

The new parameter share_server_id is required if the share type given specifies DHSS = True mode. If share_server_id``is not given, or is given while the given share type specifies ``DHSS = False mode, the API will return 400 BadRequest.

Response:

Code: 202 Accepted

There are no changes proposed to the response body.

Unmanaging a share:

POST /v2/{tenant-id}/shares/{share_id}/action

Request parameters:

{
    "unmanage": null
}

Response:

Code: 202 Accepted

This API will no longer return 403 Forbiddden when attempting to unmanage a share that was created in DHSS = True mode.

Managing a snapshot:

No API changes are required. The API currently does not validate if the snapshot being managed is associated with a share that was created in DHSS = True mode.

Unmanaging a snapshot:

POST /v2/{tenant-id}/snapshots/{snapshot_id}/action

Request parameters:

{
    "unmanage": null
}

Response:

Code: 202 Accepted

This API will no longer return 403 Forbiddden when attempting to unmanage a snapshot of a share that was created in DHSS = True mode.

Driver impact

A new driver interface is introduced in order to get share server network allocation data. This data shall be used to validate the network allocations previously created by the admin. Drivers that support DHSS = True mode must implement this interface to support the Manage Share Servers functionality:

def manage_server(self, context, share_server, identifier, driver_options):
    """Return compiled back end details and network allocations.

    :param context: Current context.
    :param share_server: Share server model.
    :param identifier: A driver-specific share server identifier
    :param driver-options: Dictionary of driver options to assist managing
        the share server
    :return Dictionary with back end details to be saved in the database
        and a list containing IP addresses allocated in the back end.

    Example::

        {'server_name': 'my_old_server'},['192.168.10.10', 'fd11::2000']

    """
    raise NotImplementedError()

If the driver does not implement this interface, an exception will be raised and the operation will be aborted.

The driver interfaces manage_existing, unmanage, manage_existing_snapshot and unmanage_snapshot will be updated to receive the share server model parameter. All drivers which implement those interface will have their method definition updated to avoid issues. In detail:

def manage_existing(self, share, driver_options, share_server=None):
    """Brings an existing share under Manila management.

    If the provided share is not valid, then raise a
    ManageInvalidShare exception, specifying a reason for the failure.

    If the provided share is not in a state that can be managed, such as
    being replicated on the backend, the driver *MUST* raise
    ManageInvalidShare exception with an appropriate message.

    The share has a share_type, and the driver can inspect that and
    compare against the properties of the referenced backend share.
    If they are incompatible, raise a
    ManageExistingShareTypeMismatch, specifying a reason for the failure.

    :param share: Share model
    :param driver_options: Driver-specific options provided by admin.
    :param share_server: Share server model or None.
    :return: share_update dictionary with required key 'size',
             which should contain size of the share.
    """
    raise NotImplementedError()

def unmanage(self, share, share_server=None):
    """Removes the specified share from Manila management.

    Does not delete the underlying backend share.

    For most drivers, this will not need to do anything.  However, some
    drivers might use this call as an opportunity to clean up any
    Manila-specific configuration that they have associated with the
    backend share.

    If provided share cannot be unmanaged, then raise an
    UnmanageInvalidShare exception, specifying a reason for the failure.
    """

def manage_existing_snapshot(self, snapshot, driver_options,
                             share_server=None):
    """Brings an existing snapshot under Manila management.

    If provided snapshot is not valid, then raise a
    ManageInvalidShareSnapshot exception, specifying a reason for
    the failure.

    :param snapshot: ShareSnapshotInstance model with ShareSnapshot data.

    Example::
        {
        'id': <instance id>,
        'snapshot_id': < snapshot id>,
        'provider_location': <location>,
        ...
        }

    :param driver_options: Optional driver-specific options provided
        by admin.

    Example::

        {
        'key': 'value',
        ...
        }

    :param share_server: Share server model or None.
    :return: model_update dictionary with required key 'size',
        which should contain size of the share snapshot, and key
        'export_locations' containing a list of export locations, if
        snapshots can be mounted.
    """
    raise NotImplementedError()

def unmanage_snapshot(self, snapshot, share_server=None):
    """Removes the specified snapshot from Manila management.

    Does not delete the underlying backend share snapshot.

    For most drivers, this will not need to do anything.  However, some
    drivers might use this call as an opportunity to clean up any
    Manila-specific configuration that they have associated with the
    backend share snapshot.

    If provided share snapshot cannot be unmanaged, then raise an
    UnmanageInvalidShareSnapshot exception, specifying a reason for
    the failure.
    """

Security impact

As admin-only APIs, all the information necessary for managing share servers and the shares within is restricted to the admin by virtue of default policy.

Notifications impact

Since the new API proposed is an admin-only API, there is no need for a user message notification.

Other end user impact

New commands will be added to python-manilaclient:

manila share-server-manage <host> <share_network_id> <identifier> [--driver_options key1=value1 [key2=value2] ...]

manila share-server-unmanage <share_server_id>

One command will be updated:

manila manage [--name <name>] [--description <description>]
              [--share_type <share-type>]
              [--share_server_id <share-server-id>]
              [--driver_options [<key=value> [<key=value> ...]]]
              [--public]
              <service_host> <protocol> <export_path>

As for manila-ui, there will be a new button “Manage Share Server” and a context option “Unmanage Share Server” which will only be displayed if there are no manila shares associated with the given share server.

Performance Impact

No significant performance impact is expected.

Other deployer impact

None.

Developer impact

The work proposed by this spec is impacted by [2]. The share_network_id parameter for the Manage Share Server will need to be replaced in favor of share_network_subnet_id.

Implementation

Assignee(s)

Primary assignee:: ganso

Work Items

Implement main patch for manila that includes:
- Database schema migration
- Manage Share Server API
- Unmanage Share Server API
- Updates to Manage Share, Unmanage Share, Manage Snapshot and Unmanage Snapshot APIs
- Add manage_server driver interface
- Update existing affected driver interfaces
- Share Manager adjustments to prevent automatic deletion of managed share servers
- Adaptations to network plugins
Implementation in a First Party Driver
Functional Tests in manila-tempest-plugin
Python-manilaclient update
Docs update
Manila-UI update

Dependencies

None.

Testing

The functional tests of this change will consist of creating a share (which will create a regular share server), obtain its details, unmanage it, manage it, create another share in it, and then delete the share and the server.

The existing config option “run_manage_unmanage_tests”, now in combination with “multitenancy_enabled” will control whether those tests will run. If any of those are disabled the tests will be skipped.

Documentation Impact

The following documentation sections will be updated:

API reference: Will add the Manage Share Server API information and parameter details. The Manage Share API will be updated to include the share_server_id parameter as well.
Admin reference: Will add instructions on how about to manage shares in DHSS = True mode (including pre-requisite steps) and how to the use the new/updated CLI commands.
Developer reference: Will add information on how the functionality works and how to implement support for it in drivers.

References

[1] https://docs.openstack.org/manila/latest/contributor/driver_requirements.html#at-least-one-driver-mode-dhss-true-false

[2] https://blueprints.launchpad.net/manila/+spec/share-replication-enhancements-for-dhss

[3] https://etherpad.openstack.org/p/manila-ptg-planning-denver-2018

Support metadata for access rule

Fri, 09 Mar 2018 00:00:00

https://blueprints.launchpad.net/manila/+spec/metadata-for-access-rule

Add a new “metadata” property for access rule resource.

Problem description

Access rule resource lacks the ability for getting/setting metadata property.

Use Cases

The metadata here for access rule is the descriptive metadata. It’s used for discovery and identification. Users could add key-value pairs for the access rules to describe them. Users could also filter access rules with specified metadata while requesting a list of access rules.

Proposed change

The “metadata” property will be added to access rule object.
A new DB table “share_access_rules_metadata” will be created.
The access rule create API will be updated to support “metadata”.
A set of APIs will be created for access rule metadata’s CRUD.

Alternatives

We can only get the desired access rule value by access_to. There is no way to tag what the access rule is by the user or add some special info to access.

Data model impact

The new “share_access_rules_metadata” table will be created in DB and it will include access rule id. The primary key is “id”. We could get information of the access_rules table by the access rule id.

+-------------------+--------------+-------------+-------------+-----------------+
|     key           | value        | id          | deleted     |    access_id    |
+-------------------+--------------+-------------+-------------+-----------------+
|    varchar(255)   | varchar(255) | varchar(36) | varchar(36) |    varchar(36)  |
+-------------------+--------------+-------------+-------------+-----------------+
|    varchar(255)   | varchar(255) | varchar(36) | varchar(36) |    varchar(36)  |
+-------------------+--------------+-------------+-------------+-----------------+`

REST API impact

It will follow the microversion rules.

The access rule create API’s request body will be updated to support “metadata”.

POST /v2/{project_id}/shares/{share_id}/action

the request body can contain "metadata".
{
    "allow_access":{
        "metadata":{
            "key1": "value1",
            "key2": "value2"
        }
        ...
    }
}

A set of new APIs related to access rule metadata will be created.

Show an access rule’s metadata

GET    /v2/{project_id}/share-access-rules/{access_id}

Response
{
    "access": {
        "access_level": "rw",
        "state": "error",
        "id": "507bf114-36f2-4f56-8cf4-857985ca87c1",
        "access_type": "cert",
        "access_to": "example.com",
        "access_key": null,
        "metadata": {
            "key1": "value1",
            "key2": "value2"
        }
    }
}

List access rules filtered by access rule metadata

GET   /v2/{project_id}/share-access-rules?share_id={share-id}&key1=value1&key2=value2

Response
{
    "accesses": [
        {
                "access_level": "rw",
                "state": "active",
                "id": "507bf114-36f2-4f56-8cf4-857985ca87c1",
                "access_type": "cert",
                "access_to": "example.com",
                "access_key": null,
                "metadata": {
                    "key1": "value1",
                    "key2": "value2"
                }
        },
        {
                "access_level": "rw",
                "state": "error",
                "id": "329bf795-2cd5-69s2-cs8d-857985ca3652",
                "access_type": "ip",
                "access_to": "10.0.0.2",
                "access_key": null,
                "metadata": {
                    "key1": "value1",
                    "key2": "value2"
                }
        },
    ]
}

The “share_id” is a mandatory query key, and the API will respond with HTTP 400 if the “share_id” is not provided.

Note

The current access rules list API accepts HTTP POST requests. To ensure correct HTTP semantics around idempotent and safe information retrieval, we will introduce a new API that accepts GET requests. The old API will be capped with a maximum micro-version, i.e, it will not be available from the micro-version that this new API is introduced within.

Remove one specified metadata
```
DELETE   /v2/{project_id}/share-access-rules/{access_id}/metadata/{key}
```
- If we don’t input the “key” value, manila won’t remove any metadata and return HTTP 400.

Update a specified metadata

PUT   /v2/{project_id}/share-access-rules/{access_id}/metadata

Request
{
    "metadata":{
        "key1": "value1",
        "key2": "value2"
    }
}

If we don’t input the “key” value, it won’t update any metadata and return error.

Security impact

None

Notifications impact

The new APIs will send new notifications as well.

Other end user impact

The Manila client, CLI will be extended to support access metadata.

The access-allow command with access metadata supported will be like:

manila access-allow [--metadata <key=value> [<key=value> ...]]
                    [--access-level <access_level>]
                    <share> <access_type> <access_to>

The new access-metadata command will be like:

manila access-metadata <access_id> <action> <key=value> [<key=value> ...]

Set or delete metadata on a access rule.

Positional arguments:
<access_id>     ID of the access rule to update metadata on.
<action>     Actions: "set" or "unset" to set or delete metadata on a access rule.
<key=value>  Metadata to set or unset, only key is necessary to unset.

The new access-show command with access metadata supported will be like:

manila access-show <access_id>

Show information of given access rule.

Positional arguments:
<access_id>       ID of the access rule to update metadata on.

The access-deny command will delete all access rule metadata. The command syntax won’t be changed:
```
manila access-deny <share> <id>
```

The access-list command will add metadata filter. The command will be like:

manila access-list [--columns <columns>] <share>
                   [--metadata [<key=value> [<key=value> ...]]]
Show access list for share.

Positional arguments:

<share> Name or ID of the share.

Optional arguments:

--columns  Comma separated list of columns to be displayed
example –columns “access_type,access_to”.

--metadata Filters results by a metadata key and value.
OPTIONAL: Default=None.

Performance Impact

A new “share_access_rules_metadata” table will be created. The DB join action may cause the search performance to reduce on the existing access rules APIs.

Other deployer impact

None

Developer impact

Drivers will not have access to the metadata, and the driver interfaces for update_access will not be modified.

Implementation

Assignee(s)

Primary assignee:: zhongjun

Work Items

Add metadata property to access rule object and bump the APIs version.
Create a new DB table “share_access_rules_metadata” and add db upgrade script.
Update access rule create/list API.
Add update/delete APIs for access rule metadata.
Add new tests within openstack/manila-tempest-plugin for the new APIs.
Allow adding/updating/deleting access rule metadata in Manila UI and python-manilaclient.

Dependencies

None

Testing

Unit tests
Tempest tests

Documentation Impact

Api-ref needs update.
update user documentation and CLI documentation

References

None

manila networking support for hierarchical port bindings for neutron

Wed, 14 Feb 2018 00:00:00

https://blueprints.launchpad.net/manila/+spec/manila-hpb-support

Many manila drivers are capable of supporting VLAN networking but this technology limits the number of actual networks in the cloud to 4096. Other overlay technologies are often not supported by vendor drivers. With HPB (Hierarchical port binding) this barrier can be reduced by using multiple (in the hierarchy of the physical network) port bindings. For example this allows the usage of VXLAN on top of VLAN. In general this is transparent for the underlying storage since this hierarchical binding is all done by neutron and in the end it’s just a VLAN that will be visible to the backend storage.

Problem description

Manila with the current design (manila/network) uses neutron only to reserve an IP and to retrieve the dedicated segmentation id (VLAN ID) out of it. The port itself stays in status inactive.

This feature will cover three aspects of implementation:

The support of neutron port binding in manila

The support of baremetal provisioning

The support of multi-segment network / HPB support

Use Cases

As admin I want to create a share-network with share-server that will be automatically provisioned to the underlaying network infrastructure. This covers neutron ML2 private and provider networks and multi-segmented networks (like HPB networks).

This feature is only useful for drivers with DHSS support (DHSS=True). Otherwise networking setup is a manual work an admin needs to do.

Proposed change

manila should be able to benefit from the neutron port-binding extension [1]. This API extension allows ports to do a real binding also with physical port configuration on a switch.

Enabling port binding support

To enable port binding it’s necessary to specify one additional field within the neutron port create request: binding:host_id

The host_id is often used in neutron ML2 drivers to identify the agent that can do the binding. An agent is not necessarily needed for manila case, so the host_id should be set to a value that is not managed by an OVS-agent. It can be set to the name (or IP address) of the storage box.

While binding the port, neutron will iterate through all ML2 mech drivers. It’s important that one of the drivers signals that the binding can be fulfilled. Available mech drivers from Cisco [2] and upcoming Arista mech driver already support binding for such cases.

Part of the feature is also a small neutron manila mech driver. Which can fulfill the binding generically. This also enables drivers that do a partial binding and would work in neutron also in the gate.

Baremetal vNIC type / Ironic ML2 integration

Ironic has a very similar problem when connecting physical devices/servers to a neutron managed network. This feature can reuse the ML2 Ironic interface described here: [3]

To reuse the feature, the vnic_type must be set to baremetal during port creation. Furthermore it’s needed to add some network information to the port create, like:

"binding:profile": {
    "local_link_information": [{
        "switch_id": 00-12-ff-e1-0d,
        "port_id": dd013:12:33:4,
        "switch_info": {"switch_ip": 10.0.0.1}
    }]
}

This can be done with static configuration in manila.conf per backend:

[manila_storage_drv1]
port_binding_profiles=phy_conn1,phy_conn2

[phy_conn1]
switch_id = 00-12-ff-e1-0d
port_id = dd013:12:33:4
switch_info = switch_ip=10.0.0.1

Multi-segment binding

A multi-segment binding behaves differently than binding a single segment network. API-wise a single segment looks like:

$ neutron net-show <>

provider:network_type: vlan
provider:physical_network: mynet1
provider:segmentation_id: 1029

A multi-segment network looks like:

$ neutron net-show <>

segments: [
    provider:network_type: vxlan, provider:segmentation_id: 123, ..
    provider:network_type: vlan, provider:segmentation_id: 544, ..
    ]

It’s also possible for mech drivers to dynamically allocate segments during binding.

For manila, this means, the ports must be created before identifying the used segment. This can be done with a dedicated neutron-manila-mech driver that adds needed information in binding:vif_details or by using the physical_network field in manila configuration.

Later, neutron should support an API interface to retrieve the binding information in a better way. This work will be tracked here: [5]

Alternatives

Introduce a neutron ML2 agent that does the actual binding following the concept that neutron is doing all network related actions. This would mean all the agent needs to have a driver concept to support multiple vendors and APIs.

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance impact

The share server creation will take longer since manila needs to wait for the neutron port to become active. This can be enhanced later, e.g. by introducing multi-processing and proceeding with share server creation like Nova is doing.

Other deployer impact

Configuration files need to be enhanced to activate the feature. Old functionality / old configuration will work as before.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: m-koderer

Other contributors:

tpatzig

dgonzalez

Work Items

The support of neutron port binding in manila

The support of baremetal provisioning

The support of multi-segment network / HPB support

Adding a manila mech driver in contrib

Dependencies

None.

Testing

Code will be tested by unit tests. Functional testing must be done in a separate CI job:

Binding can be tested using a manila mech driver

Multi-segment (only static) can potentially be tested using a neutron network

A potential test candidate could be the container driver [6], since it needs a binding done by neutron. A full end-to-end test with dynamic multi-segments would need a 3rd-party CI. Currently in discussion is to add those tests in Netapp-CI system.

Documentation Impact

Documentation for new configuration switches and possible deployment types.

References

Manila db purge utility

Wed, 14 Feb 2018 00:00:00

https://blueprints.launchpad.net/manila/+spec/clean-deleted-row-in-db

This spec adds the ability to sanely and safely purge soft-deleted rows from the manila database for all relevant tables. Presently, we keep all deleted rows. And this is unmaintainable as we move towards more upgradable releases. Today, most operators depend on manual DB queries to delete this data, but this opens up to human errors.

The goal is to have this be an extension to the manila-manage db command. Similar specs are being submitted to all the various projects that touch a database.

Problem description

Very long lived OpenStack installations will carry around database rows for years and years. This brings the following problems:

If deleted data is kept in the DB, the number of rows can grow very large taking up the disk space of nodes. Larger disk space means more worry for disaster recovery, long running non-differential backups, etc.
Large number of deleted rows also means that an admin or authorized owner querying for the corresponding rows will get 5xx responses timing out on the DB, eventually slowing down other queries and API performance.
DB upgrade ability is a big challenge if the older data style are less or inconsistent with the latest formats. An example would be the image locations string where older location string styles are different from the latest.

To date, there is no “mechanism” to programmatically purge the deleted data.

Use cases

Operators should have the ability to purge deleted rows, possibly on a schedule (cron job) or as needed (Before an upgrade, prior to maintenance). The intended use would be to specify a number of days prior to today for deletion, e.g. “manila-manage db purge 10” would purge deleted rows that have the “deleted_at” column greater than 10 days ago

Proposed change

The proposal is to add a “purge” command to manila-manage db command collection. This will take a non-negative number (0 will delete the rows “up to now”) of days argument and use that for a data match. Like:

DELETE FROM shares
WHERE deleted_at <= NOW() - INTERVAL <specified_days> DAY

Note: row with attribute(s) used as a foreign key will not be deleted even if it satisfies the deleted_at time requirement.

To accomplish this, we rearrange the table list in case of foreign key constraint, To make the logic simple and direct, we would hard code the whole table list with variable ‘PURGE_TABLE_LIST’, the existing tables in order are below (30 in total, ‘manila_nodes’ is excluded as it does not exist in db):

availability_zones
services
quotas
project_user_quotas
quota_classes
reservations
quota_usages
cgsnapshots
cgsnapshot_members
share_instance_access_map
share_access_map
share_snapshot_instances,
share_instance_export_locations_metadata,
share_instance_export_locations
share_snapshots
share_instances
share_type_projects
share_type_extra_specs
share_types
share_metadata
shares
consistency_groups
consistency_group_share_type_mappings
share_network_security_service_association
security_services
share_server_backend_details
network_allocations
share_servers
share_networks
drivers_private_data

This list should be synchronized when the database schema is changed.

Alternatives

Today, this can be accomplished manually with SQL commands, or via script.

Data model impact

None

REST API impact

None

CLI impact

A new manila-manage command will be added:

manila-manage db purge <age_in_days>

Driver impact

None

Security impact

Low, This only touches already soft deleted rows.

Notifications impact

None

Other end user impact

None

Performance impact

This has the potential to improve performance for very large databases. Very long-lived installations can suffer from inefficient operations on large tables. This would have negative DB performance impact while the purge is running.

Other deployer impact

None

Developer impact

Developer should update the table list whenever the tables’ relationship changed.

Implementation

Assignee(s)

Primary assignee:

zhongjun(jun.zhongjun2@gmail.com)
TommyLike(tommylikehu@gmail.com)

Work items

Implement ‘db purge’ command.
Add related unit tests.
Add documentation of this feature.

Dependencies

None

Testing

Unit testcases which focus on algorithm:

Single table with rows vary in ‘deleted_at’ time.
Multiple tables with inner relationship and their rows vary in ‘deleted_at’ time.

Unit testcases which focus on the hard code list (PURGE_TABLE_LIST)’s consistency:

All tables in manila should be added to list (exception to this also can be added here in purpose).
Each child table that uses foreign key(s) should come before its parent table(s).

Documentation impact

Documentation of this feature will be added to the admin guide and developer reference.

References

This is already discussed and accepted in other OpenStack components, such as Glance [1] and Cinder [2].

[1] https://specs.openstack.org/openstack/glance-specs/specs/mitaka/database-purge.html [2] https://specs.openstack.org/openstack/cinder-specs/specs/kilo/database-purge.html

Ensure share

Mon, 11 Sep 2017 00:00:00

https://blueprints.launchpad.net/manila/+spec/ensure-share

It’s not reasonable to update all shares on every share manager when we restart manila share service. This driver interface is currently being used wrong and needs a rewrite. This spec adds the ability to solve the potential problem of slow start up, and deal with non-user-initiated state changes to shares.

Problem description

Manila-share service has share driver interface called “ensure_share”, that, depending on share driver, performs some actions on existing shares. This method is called on each manila-share service start and none of RPC requests are handled while this method does not finish processing existing shares. It makes life of cloud administrator painful, because each manila-share service restart takes more and more time with growth of shares amount. Also, in most cases, this processing is redundant.

Use cases

Consider the following reasonable use cases:

Manila-share node going to be shut down for maintenance and then enabled back when we want to upgrade software or something else. In this case admin expects fast start of service, no matter how many shares do exist and managed by manila, and no matter they change any config options or not.
Allows admins to update shares automatically when the admin restarts the service and they changed some config options or updates storage somehow (such as: we have an IPv4-only controller and we want to add an IPv6 address (any export location)).

In the future we will solve following use cases:

Allows admins to automatically tag some shares or shares belonging to particular backends as needing ‘ensure’.
Allows admins to update other resouces(such as: snapshot, share group, etc) for share services when the admin restarts the service.
Allows admins to update shares and don’t need to restart manila share services when the admin changes something on array (such as: export location IP). (depends on other spec)

Proposed change

To accomplish this change, we will do the following:

When the share manager starts up, it will call the get_backend_info() driver method to obtain a dictionary of values which could affect shares. The share manager will compute a hash of the returned dictionary and compare that hash to the previously computed value. If the value hasn’t changed then the ensure_shares logic will be skipped, otherwise the ensure_shares logic will execute as normal but the new hash value will be stored in the DB.

The get_backend_info() driver method is a new driver method which will be called after driver initialization but before ensure_shares, that returns a dictionary. Drivers will have complete control of the contents of the dictionary to be hashed. Driver can include any of the following:

The ID of the most recent DB migration. The manager will add new get_recent_db_migration_id interface to supply this value to the driver. Then this value guarantees that ensure_shares will be called after DB schema changes.
Any of the keys and values in the oslo config object. Including these values guarantees that ensure_shares is called whenever the config file changes in a way that impacts the driver.
Hardcoded values inside the driver, such as version numbers. Including these values allows the driver to force ensure_shares to be called when a code changes occurs.
Values collected from the storage controller itself. In case the administrator upgrades or reconfigures the storage controller, this allows Manila to detect the change and to run ensure_shares, allowing Manila to learn any new important details or perform any needed maintenance.

The reason for hashing the values is because Manila doesn’t care what the old values were, only that something changed. Storing a hash of the values is simpler from a DB schema perspective. In order to avoid accidental hash collisions MD5 will be used to generate 128 bits worth of data. Dictionaries will be sorted lexicographically by key, and all values will be converted to strings, then UTF-8 encoded into bytes, and fed into the hash algorithm.

The ensure_share() method will be replaced by a method called ensure_shares() which takes a list of shares in and returns a dictionary of model updates. This is because there are no cases where it makes sense to call ensure_share() on less than all of the shares on a particular backend (or share_server, if DHSS=true). Combining all the calls into one makes it easier for driver implementers to reuse data that will be common to all shares.

Alternatives

None

Data model impact

We will add a new table and model to store the hash information per-backend.

The model will be called manila.db.sqlalchemy.models.BackendInfo and will contain 2 columns: host and info_hash. The host field will be just the “hostname” of the backend with no pool suffix.

REST API impact

None

Driver impact

Add driver interfaces:

def get_backend_info(self, context):
    """Get driver and array configuration parameters and return for
       assessment.
    :return A dictionary containing driver-specific info::
         {
          'version': '2.23'
          'port': '80',
          'logicalportip': '1.1.1.1',
           ...
         }
    """

Replace ensure_share() with ensure_shares()

def ensure_shares(self, context, shares, share_server=None):
“””Invoked to ensure that shares are exported.

Driver can use this method to update the list of export locations of the shares if it changes. To do that, a dictionary of shares should be returned. :shares: None or a list of all shares for updates. :return None or a dictionary of updates in the format:
{
    '09960614-8574-4e03-89cf-7cf267b0bd08': {
        'export_locations': [{...}, {...}],
        'status': 'error',
    },

    '28f6eabb-4342-486a-a7f4-45688f0c0295': {
        'export_locations': [{...}, {...}],
        'status': 'available',
    },

}
“””

Note that drivers that don’t override the parent class’s implementation of get_backend_info() would get the parent class implementation which would return an empty dictionary and thus prevent ensure_shares() from being called, a change from current behavior. We can optionally add an override implementation of that method to drivers that need ensure_shares() to be called more often.

As part of this change we will determine which drivers have required logic in ensure_shares() and implement get_backend_info() for them in such a way that no functionality is lost.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance impact

The effect of this change would be to make the manila-share service start up faster in cases where nothing important has changed (the common case). The effect will be small when ensure_shares doesn’t do anything or when the number of shares is small, but could be very large for backends with a large number of shares and expensive ensure_shares operations, reducing a O(n) startup time to O(1).

Other deployer impact

None

Developer impact

Drivers will be strongly encouraged to implement get_backend_info(), but won’t be required. Any drivers implementing ensure_shares() will need to update their logic to not assume that ensure_shares is called every time the driver starts.

Most importantly, drivers will be able to implement potentially more expensive operations in ensure_shares() without creating large scalability problems.

Implementation

Assignee(s)

Primary assignee:

zhongjun(jun.zhongjun2@gmail.com)

Work items

Implement the core feature with unit tests
Convert all ensure_share() methods to ensure_shares()
Implement get_backend_info() in first party drivers.
Add documentation of this feature

Dependencies

None

Testing

Due to the difficulty of restarting services during functional tests, it’s only practical to test this change with unit tests.

Documentation impact

Documentation of this feature will be added to the developer reference.

References

None

Show resource’s total count info in share list APIs

Fri, 08 Sep 2017 00:00:00

https://blueprints.launchpad.net/manila/+spec/add-amount-info-in-list-api

This blueprint proposes adding total count info in Manila’s /shares and /shares/detail APIs.

Problem description

Show how many resources a user or tenant has is usually required in the web portal’s summary section, but we cannot get this info now. In order to show this kind of total number to the end user, many clouds have to collect all of the resources.

Use Cases

The administrators or users to know how many resources they have in total without retrieving and accumulating them all when it return just a slice of paginated list.

Proposed change

According to the openstack API-WG [1] proposal. It provide guidance on returning the total size of a resource collection in a project’s public REST API.

So this bp proposes to add new attribute count in our /shares and /shares/detail APIs to having the total count information in /shares and /shares/detail APIs response. If we add the count attribute into our response body in default, it could has a performance impact if we have a mount of resources, considering not every request requires this kind of info, the additional query parameter is required to turn this on when listing resource.

Alternatives

There are few alternative solutions for this requirement, let’s list and compare them all here.

Add amount information in response header:
```
X-resource-count: 200
```

The disadvantage of this is it will add some burden to the API customers, we don’t have any history for adding useful content in response headers. Also it makes more difficult for documentation.

Add explicit API for each resources:

POST: /v2/{tenant_id}/{resources}/action {os-count}

This change involves a lot of modifications and creates several new APIs. Adding this amount of APIs for such a simple API change is overdesign.

Create a new API for different resources:

POST: /V3/{tenant_id}/action {os-count}
BODY:
    {
        "resource": "share",
        "user": "user_1",
        "other_filter": "other_value"
    }

For this change, one more API request is required if the end user wants to know how many resources in total when listing resources.

Data model impact

None

REST API impact

Microversion bump is required for this change.

Add the ‘with_count’ parameter in query string in share list API, it is used to indicate if the total count of resources should or should not be returned from a GET REST API request. Any value that equates to True indicates that the count should be returned. Conversely, any value that equates to False indicates that the count should not be returned.

List shares

URL: /v2/{tenant_id}/shares?with_count=true
Method: POST

JSON body:

{
    'shares': [{
        'name': 'test',
        ...
    }]
    'count': <int_count>,
}

List shares with details

URL: /v2/{tenant_id}/shares/detail?with_count=true
Method: POST

JSON body:

{
    'shares': [{
        'name': 'test',
        ...
    }]
    'count': <int_count>,
}

Client impact

The share list command will be upgraded to support this.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Since we will add additional COUNT() statement if the list APIs are requested with the with_count option, there would be a performance impact on those APIs, especially when there are a lot of data in database.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: zhongjun(jun.zhongjun2@gmail.com)

Work Items

Add with_count option support in share list APIs.
Add related unit testcases.
Update manila-client.
Update manila-ui.

Dependencies

None

Testing

Add unit tests and tempest test to cover this change.

Documentation Impact

Update API documentation.

References

[1]: http://specs.openstack.org/openstack/api-wg/guidelines/counting.html

Flexible API filters

Thu, 18 May 2017 00:00:00

https://blueprints.launchpad.net/manila/+spec/like-filter

This spec proposes changing API filter behavior to inexact matching of filter values.

Problem description

Currently, we could only filter manila resource by exact match. This is not flexible enough, because customer needs to remember full share name, and can not get shares quickly by key words. However for customer, if partly alike filters are supported to retrieve the shares via user input, that should be useful for user to filter shares by name flexibly, especially for the users who have a lot of shares which are managed by manila.

By the way, it’s already introduced in many other projects, we can take advantage of those some existing mechanism, nova [1] and ironic [2]. In nova, it change for regex filter matching in sqlalchemy. In ironic, it add a way to filter nodes in the API by their name (regex and wildcard).

Use Cases

User has really big amount of shares. He named them using some different templates based on his needs. And he may want to filter out shares based on some part of share names. The same situation with “description”.

Proposed change

It’s worth to mention here, that this feature is also been proposed in cinder [4] and it’s depended on the generalized resource filter [5] , but making the API behavior configurable is still open to question in manila, so we would only pick the part we’d like to have while keep the API consistency with cinder.

Use ‘~’ operator after the filter key to trigger the like filtering operation, this is also how our API will look like:

GET /v2/<tenant_id>/shares?name~=test

Support like filter only on specified resources (cinder let the administrator to have that decision by configuration file):

list shares (name, description)

list snapshots (name, description)

list share-networks (name, description)

list share-groups (name, description)

The Manila client will add a new command argument ‘–name~’ and ‘–description~’ to ‘list’ APIs to filter manila resources by inexact match.

We can provide resource filtering based on regex filter, but there is a possibility that we could have ReDos [3] attack. Considering the filter is flexible and safe enough for this case. We could introduce ‘LIKE’ operator. And we can easily apply this filter at the existing sqlalchemy filtering function, and use ‘LIKE’ operator to filtering resource.

Alternatives

There is an option that we can deploy searchlight which mainly focus on various cloud resource querying, and that’s a more wider topic. But we should also consider the cloud environment that don’t contain searchlight.

Also, there is another option that user can gather all the raw data and do the filtering on their own, but it’s obvious that is what we try to avoid, cause it costs a lot of unnecessary resource expense.

Data model impact

None

REST API impact

Microversion bump is required for this change.

Add name~ and description~ parameter to list API interfaces. It means we can use this new parameter to inexact filtering:

List shares with share name

GET /v2/<tenant_id>/shares?name~=test

Accept: application/json

JSON Response

{
   "shares": [
         {
            "status": "available",
            "export_location": %ip%:/opt/stack/data/manila/mnt/share-%share_id%,
            "name": "test_1",
            ...
       },
       {
            "status": "available",
            "export_location": null,
            "name": "2_test",
            ...
       }
   ]
}


List snapshot with share snapshot name

GET /v2/<tenant_id>/snapshots?name~=snap_test

Accept: application/json

JSON Response

{
   "shares": [
       {
            "status": "available",
            "name": "snap_test_1",
            ...
       },
       {
            "status": "available",
            "name": "snap_test_xxxxx",
            ...
       }
   ]
}

Client impact

The Manila client will add a new command argument ‘–name~’ and ‘–description~’ to ‘list’ to filter manila resource by inexact match:

manila list [--name~ <name~>] [--description~ <description~>]

We assuming we have two shares in the name of’test_1’, ‘2_test’, usually we would get none of them if type this command:

manila list --name test

But within this feature merged, we would have both of them with the identical command if type this command:

manila list --name~ test

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Developer should update the filtering function to support exact/inexact filter (use ‘LIKE’ operator to filtering resource) in DB API whenever the list/show API changed.

Implementation

Assignee(s)

Primary assignee:: tommylikehu(tommylikehu@gmail.com) jun.zhongjun(jun.zhongjun2@gmail.com)

Work Items

Functional tests
Implement core feature
Add related unit tests
Update python-manilaclient

Dependencies

None

Testing

Add unit tests and functional tests to cover filter process change.

Documentation Impact

Update API documentation.

References

It is proposing essentially the same spec for cinder [5].

[1]: https://review.openstack.org/#/c/45026/ [2]: https://review.openstack.org/#/c/266688/ [3]: https://en.wikipedia.org/wiki/ReDoS [4]: https://review.openstack.org/#/c/442982/ [5]: https://review.openstack.org/#/c/441516/

Integration with ceilometer

Tue, 16 May 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/manila/+spec/ceilometer-integration

At the moment of writing it is not possible to send usage notifications from manila to ceilometer. Having this possiblity, users would be able to monitor the status of the share service and also the shares themselves.

Problem description

Currently there is no way to receive notifications about events in the life-cycle of share service resources, precisely, information such as:

Share size
Share capacity usage
Share create (start/end)
Share delete (start/end)
Share update (start/end)
Share extend (start/end)
Share type create (start/end)
Share type delete (start/end)
Snapshot size
Snapshot create (start/end)
Snapshot delete (start/end)
Revert to snapshot (start/end)

Size metrics are discrete values (gauge meters) while create/delete/update operations are changing over time values (delta meters).

And also it’s not possible to know performance information about the available shares, that is, information like:

ops/s (number of operations per second issued to the filesystem)
rops/s (number of read operations per second issued to the filesystem)
wops/s (number of write operations per second issued to the filesystem)

The latter has been requested by users in the past [1].

For now, the community has shown interest on implementing the required logic to retrieve metrics for the resources in manila [2]. And hence, this spec will focus on this.

Depending on the project plans in the future, we will decide if it is on manila’s scope to put hands on metrics for the different shares.

Use cases

Monitoring usage of resources would allow us to keep a better control on the health of the deployments and see when capacity limits are being approached.
Operators would be able to act over the collected data and trigger actions when defined criteria are met.

Proposed change

The proposed change consists of the following

Add notifiers for all the resources we want to measure. Right now we have notifiers in share_types.py [3] but this is a remainder of refactoring from cinder’s volumes_types.py in previous versions. We would need to add notifiers to shares and snapshots, and check that the notifiers in share-types publish the data we expect.
Add meters in ceilometer for the desired resources. For this, we would need to follow ceilometer’s guidelines [4]. The resources we are considering in this spec are shares, share types and snapshots. Meters for other resources such as share groups will be added in follow up changes.

Alternatives

Do not integrate with ceilometer and don’t provide telemetry for manila.

Data model impact

This change has no impact over the data model.

REST API impact

This change has no impact over the REST API.

Driver impact

This change has no impact over any driver.

Security impact

Notifications impact

This change will allow sending notifications about the consumed resources. This is something that operators should be able to enable or disable. By default, it should be disabled.

Other end user impact

End users will be able to see the notifications in ceilometer if enabled.

Performance impact

We need to consider in here the overhead produced by emiting notifications to the messaging queue for every resource we will be measuring and for all the actions we will be covering. Whereas it only retrieves data that is being accessed already (i.e. we don’t need to go to the control database in order to retrieve more data), is an action that will be performed in every interaction with the resource.

Other deployer impact

We will need to add an option for enabling and disabling sending notifications. This flag will affect manila deployed with any backend. By default, this option will be set to false and manila won’t be sending notifications, keeping the behavior it currently has.

If operators want to start retrieving metrics on the manila resources, they will need to enable sending notifications on the configuration file. They also need to make sure they have a specified version of ceilometer deployed. After this, the feature here specified will be available for use.

Developer impact

After the framework is added, developers may need to add notifications accordingly when adding new functionalities.

Implementation

Assignee(s)

Primary assignee:: Victoria Martinez de la Cruz <victoria@redhat.com>

Work Items

Add notifiers for shares. Add unit tests. Add meters for shares. Add dev docs for shares meters.
Add or update notifiers for share-types. Add unit tests. Add meters for share-types. Add dev docs for share-types meters.
Add notifiers for snapshots. Add unit tests. Add meters for snapshots. Add dev docs for snapshot meters.

Dependencies

The blueprint on the ceilometer side can be accessed in [5]. We will add the desired meters as part of that blueprint.

No other dependencies considered at the moment of writing.

Testing

Whereas having tempest coverage for this feature is desired, it’s not a priority.

Unit tests will be added for main functionality.

A CI job with manila and ceilometer enabled will be added to exercise integration and compatibility of both services.

Documentation Impact

With regard to documentation, we will need to add explicit docs on how to configure manila to enable this feature and we will need to add explicit instructions on which kind of data will be available. Whereas this is something that we will cover on the developers docs, it will be important to add some subsection under manila on the operations manuals.

References

Share usage size tracking

Tue, 16 May 2017 00:00:00

https://blueprints.launchpad.net/manila/+spec/share-usage-size

We need to have a way to gather information about actual share storage usages, so cloud operators could use this information for billing, health checks and/or other purposes.

Problem description

Currently, it is impossible to get actual storage usage of some specific share, hence, it is impossible to bill for storage usage. It is only possible to bill for quota limits.

Use cases

Manila could report storage usages to metering services like ceilometer [1].
Some storage backends do not reserve whole requested share size right away after share creation and just set quota/limit for shares, for such backends it may be more suitable to bill for usage, not limit/quota.

Proposed change

Add the periodic task in share service for gathering the shares usage size. The periodic task will be invoked after the share manager starts up. We can set the interval time that determines how often the share manager will poll the driver to perform the next step of get share usage size. Administrators and driver developers can “opt-in” to enable this feature in their drivers and in their clouds.

Add notifiers for shares usage size we want to measure. Right now we intend to add notifiers to resources (such as: shares, share groups, snapshots and etc) and publish the resources we expect [1].

The update_share_usage_size() driver method is a new driver method which will be called after driver initialization and in the periodic task, that returns a dictionary about shares id and shares usage size. The shares usage size value info will be notified to ceilometer or other project, otherwise the share info doesn’t need to be notified.

Alternatives

Why we don’t gather the space usage when we need to use it (such as: list, show APIs)?

Because we could publish such real time information to ceilometer.
Why we don’t store the space usage?

Because we can’t implement this inside manila without duplicating functionality already in other projects like ceilometer. It’s a number that changes over time. If the number is too old, it might be wildly inaccurate. So we’re going to push the storage/retrieval parts of the problem outside of manila to keep manila’s scope small and focused.

Data model impact

None

REST API impact

None

CLI impact

None

Driver impact

Get real share provisioned capacity from driver:

def update_share_usage_size(self, context, shares):
    """Invoked to get the usage size of given shares.

    Driver can use this method to update the share usage size of
    the shares. To do that, a dictionary of shares should be
    returned.
    :shares: None or a list of all shares for updates.
    :return None or a dictionary of updates in the format::

        {
            '09960614-8574-4e03-89cf-7cf267b0bd08': {
                'used_size': '200',
            },
        }

    """
    raise NotImplementedError()

If an administrator has configured a backend for monitoring and the driver cannot support the update_share_usage_size method, then the share manager will catch the raise exception (NotImplementedError) from the driver, and we will later not invoke the periodic task to update share usage size for shares of this backend. We will LOG that it is unsupported instead.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance impact

Collecting share usage have a negative service performance impact. When the size of a share is known at the moment of creation and does not change without an explicit extend or shrink, we don’t have to collect usage at all. Now we have to collect usage. It could be costly.

Administrators may set the periodic interval configuration option value to a large interval or disable it if necessary. It could be less costly or not costly.

Other deployer impact

If operators want to start retrieving metrics on the manila shares usage size, they will need to enable sending notifications on the configuration file after upgrade to or installation of Pike. They also need to make sure they have a specified version of ceilometer or other software deployed. After this, the feature here specified will be available for use.

Developer impact

This will require a change on the driver interface. To support this feature, drivers will need to implement the update_share_usage_size feature.

Implementation

Assignee(s)

Primary assignee:

zhongjun(jun.zhongjun2@gmail.com)

Work items

Implement the core feature with functional tempest test coverage
Add documentation of this feature

Dependencies

None

Testing

Unit tests
Functional tempest tests

Documentation impact

Devref
API reference
User guide and Admin guide

References

[1] https://specs.openstack.org/openstack/manila-specs/specs/pike/ceilometer-integration.html

Add share groups to Manila

Wed, 12 Apr 2017 00:00:00

Blueprint: https://blueprints.launchpad.net/manila/+spec/manila-share-groups

Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by share groups, which we think is a clean extension to the original architecture of Manila.

Problem description

Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features are added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.

Consistency groups

Consistency groups (CGs) are the only construct currently in Manila that operate on groups of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them.

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs that totaled over 9900 lines. None of these are reusable for anything else, and none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low and the limited availability of CGs ensures the user experience is inconsistent between clouds.

Replication

Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way.

The proposed approach would be to have group extra specs that dictate whether the shares in the group are replicated in a consistent way, individually, or neither.

Migration

Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the migration engine to be CG aware.

Use Cases

Consider the following reasonable use cases:

Replicate a consistency group
Snapshot a consistency group
Snapshot a replication group
Migrate a replication group
Retype shares in a consistency group
Retype shares in a replication group
Backup a consistency group
Backup a replication group

On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward.

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

Proposed change

So to arrive at a solution, let’s enumerate what we have:

Primitive objects (shares) with supported operations controlled by share types that vary by backend
A very specific group object (CGs) with limited support potential by backends
A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project

And what would we prefer:

A uniform API and user experience that varies as little as possible by backend
A clean way to group shares so we can do CGs, multi-object replication, migration, etc.

How do we get there?

Universally available groups of primitive objects with supported operations that vary by backend

Share Group Types

Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:

Ordered writes
Consistent snapshots
Group replication
Group backup

As with share types, the cloud administrator predefines share group types that may contain group specs corresponding to the group capabilities reported by the backends. The admin also specifies which share type(s) a given group type may contain. If admin does not specify it, then manila will use ‘default’ share type. The scheduler then creates the group on one of the backends that match the specified share type(s) and share group type.

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment.

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

Share Action	Share Group Action
Create (share type)	Create (share types, group type)
Delete	Delete (group)
Snapshot	Snapshot (may or may not be “CG” snapshot)
Create from snapshot	Create from group snapshot
Clone	Clone group (and all members) (planned)
Replicate	Replicate
Backup	Backup (planned)
Retype	Retype (planned)
Migrate	Migrate
Extend/shrink	N/A

Details

There are few things to note, several of which were already solved during the CG work.

Groups are first-class objects in Manila, and operations on groups are treated as atomic. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs.

The capabilities of a group will all be public group specs, similar to snapshot_support in share types. Users will need to know what a group can do, as will tools like manila-ui.

Group quotas could be added later and described with separate spec. They are out of scope for this spec, because this spec is, mostly, about porting CGs to share groups and CGs haven’t had quota implementation.

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but this would not be a use-case covered initially. Any actions in this category must remain available to group members, and other actions such as taking snapshots of group members can be allowed, but actions such as migration or replication would be available only at the group level and not on its members.

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit.

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility.

Consistent with CGs, a grouped share must spend its entire lifecycle in the group. Adding or removing shares from groups at other than creation/deletion time might be possible for some backends, while others might have to move the data. The migration engine is envisioned as the means of moving shares into or out of a group.

We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use.

The scheduler treats multiple extra specs as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately.

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently.

Alternatives

One possible alternative is just removal of CGs. Where advantage is simplification of code base and disadvantage is losing of a feature.

Other alternative is keeping CGs as already implemented and add other groups for replication, etc., as needed. As noted above, the downside is greater complexity, larger codebase, and inconsistent user experience.

A higher-level, looser association construct could be added using tags, which would allow overlapping membership as well as members in multiple backends, but this would preclude the functionality sought in this feature that is only available for shares on a single backend. The tag-based association could be added later as a parallel feature.

Data model impact

The data model for groups should be virtually identical to that for CGs. In fact, generic share groups supersedes CGs, so the CG tables will be replaced by those for share groups.

The data model for group types should be virtually identical to that for share types, with the additional list of share types allowed for a group type.

REST API impact

It is possible to design a REST API that seamlessly handles both shares and share groups with little duplication of APIs. But at this point in Manila’s development, it is arguably too late to radically redesign the API.

It may be possible to overload some of the existing APIs to handle both shares and share groups. For example, POST /shares/{id}/action could accept the ID of a share or share group and just do the right thing. But other APIs, such as GET /shares are less practical to overload, since a single endpoint would be returning objects of different types.

It seems better to merely duplicate a few APIs with group versions as needed. For example:

POST /shares –> POST /share-groups
POST /shares/{share_id}/action –> POST /share-groups/{group_id}/action
POST /snapshots –> POST /share-group-snapshots

Share group APIs:

Create share group

URL: /share-groups
Method: POST

JSON body:

{
  'share_group': {
    'name': 'fake_name',
    'description': 'fake_description',
    'availability_zone': 'fake_az',
    'group_type_id': 'fake_group_type_id',
    'share_network_id': 'fake_sn_id',
    'source_group_snapshot_id': 'fake_snap_id',
    'source_group_backup_id': 'fake_backup_id',
    'source_group_clone_id': 'fake_clone_id',
    'share_types': ['fake_st_id_1', 'fake_st_id_2', 'fake_st_id_n']
  }
}

Notes: Keys ‘source_group_backup_id’ and ‘source_group_clone_id’ will be implemented only when appropriate features appear in manila.

List share groups
- URL: /share-groups
- Method: GET
- URL args:
  - all_tenants - integer, either 0 or 1
  - offset - integer, 0 or bigger
  - limit - integer, 1 or bigger
  - sort_key - string, key to be sorted (i.e. ‘created_at’ or ‘status’)
  - sort_dir - string, sort direction, should be either ‘asc’ or ‘desc’.
List share groups (detailed)
- URL: /share-groups/detail
- Method: GET
- URL args:
  - all_tenants - integer, either 0 or 1
  - offset - integer, 0 or bigger
  - limit - integer, 1 or bigger
  - sort_key - string, key to be sorted (i.e. ‘created_at’ or ‘status’)
  - sort_dir - string, sort direction, should be either ‘asc’ or ‘desc’.
Show share group
- URL: /share-groups/{share_group_id}
- Method: GET
Delete share group
- URL: /share-groups/{share_group_id}
- Method: DELETE
- Note: Share group can only be deleted if it does not contain any shares.
Force delete share group
- URL: /share-groups/{share_group_id}/action
- Method: POST
- JSON body:
```
{
  'force_delete': None
}
```
- Notes: it is admin-only API. Difference between this and common ‘delete’ command is that this one ignores status and raised exceptions.
Reset share group status
- URL: /share-groups/{share_group_id}/action
- Method: POST
- JSON body:
```
{
  'reset_status': '%status%'
}
```
- Notes: it is admin-only API.

Update share group

URL: /share-groups/{share_group_id}/action
Method: POST

JSON body:

{
  'share_group': {
    'name': 'new_name',
    'description': 'new description'
  }
}

Share group snapshot APIs:

Create share group snapshot

URL: /share-group-snapshots
Method: POST

JSON body:

{
  'share_group_snapshot': {
    'name': 'fake_name',
    'description': 'fake_description',
    'share_group_id': 'fake_share_group_id'
  }
}

Show share group snapshot
- URL: /share-group-snapshots/{share_group_snapshot_id}
- Method: GET
List share group snapshots
- URL: /share-group-snapshots
- Method: GET
- URL args:
  - offset - integer, 0 or bigger
  - limit - integer, 1 or bigger
  - sort_key - string, key to be sorted (i.e. ‘created_at’ or ‘status’)
  - sort_dir - string, sort direction, should be either ‘asc’ or ‘desc’.
List share group snapshots (detailed)
- URL: /share-group-snapshots/detail
- Method: GET
- URL args:
  - offset - integer, 0 or bigger
  - limit - integer, 1 or bigger
  - sort_key - string, key to be sorted (i.e. ‘created_at’ or ‘status’)
  - sort_dir - string, sort direction, should be either ‘asc’ or ‘desc’.
Delete share group snapshot
- URL: /share-group-snapshots/{share_group_snapshot_id}
- Method: DELETE
Force delete share group snapshot
- URL: /share-group-snapshots/{share_group_snapshot_id}/action
- Method: POST
- JSON body:
```
{
  'force_delete': None
}
```
- Notes: it is admin-only API. Difference between this and common ‘delete’ command is that this one ignores status and raised exceptions.
Reset share group snapshot state
- URL: /share-group-snapshots/{share_group_snapshot_id}/action
- Method: POST
- JSON body:
```
{
  'reset_status': '%status%'
}
```
- Notes: it is admin-only API.

Update share group snapshot

URL: /share-group-snapshots/{share_group_snapshot_id}/action
Method: POST

JSON body:

{
  'share_group_snapshot': {
    'name': 'new_name',
    'description': 'new description'
  }
}

Share group type APIs:

Create share group type

URL: /share-group-types
Method: POST

JSON body:

{
  'share_group_type': {
    'name': '%name%',
    'is_public': '%boolean%',
    'group_specs': {
      'foo_key': 'foo_value',
      'bar_key': 'bar_value'
    },
    'share_types': [
      'fake_share_type_id_1',
      'fake_share_type_id_2',
      '..',
      'fake_share_type_id_n',
    ]
  }
}

Show share group type
- URL: /share-group-types/{share_group_type_id}
- Method: GET

Set share group type specs

URL: /share-group-types/{share_type_id}/group-specs
Method: POST

JSON body:

{
  'group_specs': {
    'driver_handles_share_servers': True,
    'snapshot_support': True,
    'storage_protocol': 'NFS'
  }
}

Unset share group type specs
- URL: /share-group-types/{share_group_type_id}/group-specs/{group_spec}
- Method: DELETE
List share group type specs
- URL: /share-group-types/{share_group_type_id}/group-specs
- Method: GET
List share group types
- URL: /share-group-types
- Method: GET
- URL args:
  - is_public - string, that contains either boolean-like value or ‘all’. It will have the same behavior as in ‘list share types’ API. If not set or set to True, then admin/user sees only public share group types. If set to False, then admin sees only private share group types and user sees only those private share group types that are allowed to user’s project. If set to ‘all’, then admin sees all share group types and user sees all except those private share group types that are allowed to user’s project.
Delete share group type
- URL: /share-group-types/{share_group_type_id}
- Method: DELETE

Share group type access APIs:

Add project to access list
- URL: /share-group-types/{share_group_type_id}/action
- Method: POST
- JSON body:
```
{
  'addProjectAccess': {
    'project': '%project_id%'
  }
}
```
- Notes: applicable only to private group types
Remove project from access list
- URL: /share-group-types/{share_group_type_id}/action
- Method: POST
- JSON body:
```
{
  'removeProjectAccess': {
    'project': '%project_id%'
  }
}
```
- Notes: applicable only to private group types
List allowed projects
- URL: /share-group-types/{share_group_type_id}/group-type-access
- Method: GET
- Notes: applicable only to private group types
Changes to existing ‘Create share’ API:
- New optional ‘share_group_id’ param will be added.
Changes to existing ‘Delete share’ API:
- New ‘share_group_id’ param will be required in URL as GET argument, when share is part of a share group.

The share group APIs will initially be experimental.

Security impact

None

Notifications impact

None

Other end user impact

The Manila client, CLI, and GUI should ultimately be extended to support share groups, ideally all in Ocata. The client and CLI will be updated first, by removing CG commands and adding group commands, to enable Tempest coverage.

Performance impact

A group action may take longer if the share manager must iterate over several individual shares, and drivers and storage backends must be able to handle such requests. Otherwise, performance should be comparable to the existing consistency groups implementation.

Other deployer impact

Share groups feature is optional and can be ignored by deployers in case they do not need it. Also, they can disable all group-related APIs using policy.json file.

If using groups, administrators must define group types and ensure their cloud supports the combinations of group/share types they advertise.

Developer impact

One of the goals of this feature is to simplify maintenance and development of Manila. New feature authors will have to ensure any new feature works with groups as well as individual shares (possibly in a phased implementation). Driver authors will want to consider supporting group actions in an advantaged way if possible on their storage systems, with the generic implementation attempting to function with existing drivers.

Implementation

Assignee(s)

Primary assignee:

alex-meade

Other contributors:

clintonk
dustin-schoenbrun
vponomaryov

Work Items

Implementing generic groups in Manila should be a straightforward series of steps:

Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
Add the share group type feature by duplicating and customizing the share type code.
Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
Update Tempest to cover all of the above.
Add functional tests to manila client.
Add support of share group driver interfaces to dummy driver.
Update Manila client to change CGs to share groups.
Enhance Manila client with share group types.
Add share group support to manila-ui. We never built CG support into the UI, so this is all-new work that now has much broader appeal and applicability. It should be optional, but enabled by default, in the same way as share migration and replication feature support is implemented.
Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.

Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.

Dependencies

None

Testing

Because share groups require no explicit driver support, they should be fully testable using Tempest tests in the gate. Because the minimal viable product for this feature in Ocata is feature parity with CGs, the existing CG tests should be adaptable to share groups. Manilaclient tests will be written from scratch, because there is no CG tests for it right now.

Documentation Impact

Share groups is a major feature that should be fully documented. At a minimum, user-guide, admin-guide, in-tree API-ref and in-tree devref must be covered, as with other experimental features so far.

References

Add mountable snapshots

Wed, 12 Apr 2017 00:00:00

https://blueprints.launchpad.net/manila/+spec/manila-mountable-snapshots

Currently, manila only allows the user to create new shares from snapshots. This limitation raised some discussions on Tokyo and Austin summits to add new snapshot semantics. One of the new features agreed to be implemented is the capability to mount a snapshot in read-only mode.

Problem description

Some drivers can expose snapshots to be mountable, but the current implementation demands that the only action that the user can do with a snapshot is to create a new share from it. While this is useful for several situations, there are use cases in which creating a whole new share from a snapshot is overkill, or in some cases, cannot be done by existing drivers.

Use Cases

There are situations in which the user only needs a single file or a set of files to be restored. In these cases it would be simpler to just mount the snapshot and retrieve the desired files, instead of needing to create a whole new share to copy the files and then delete it. It could also be used by an administrator or an external system, like manila-data.

This feature could also be used to check the contents of the snapshot before reverting the share to its state, using the revert-to-snapshot feature.

Proposed change

The proposal is to add export locations to the snapshots, so that the user can allow and deny access to the snapshots to mount them in read-only mode. Some users or systems that do not have access to the parent share might need access to the snapshot, so the access rules for the snapshot must be separated from its parent share. This separation also provides more security, since it avoids unwanted access to the share. It is important to note that in order to support this feature, a driver must be able to support this separation of access rules.

The snapshot export locations will be provided by the driver when creating a snapshot. The APIs for this proposal will follow the current implementation of share allow and deny access, including the new protocol-specific access rule restrictions agreed in Barcelona summit. In order to support this feature, a new extra spec called mount_snapshot_support will be added.

Alternatives

The main alternative is to create a new share from the snapshot, so the files could be retrieved from this new share. While this is already implemented and usable, this alternative consumes the user quotas and in most cases the new share will just be used to retrieve a single file, which makes this overly complicated. Another alternative is to implement a read-only share that could be used in these situations. Being a read-only share, some of the operations that the user can do with a share in manila might not make sense (e.g. create snapshot, extend, shrink), so this new type of share might not comply with the current concept of shares in manila. There is also the possibility that the snapshot inherits the access rules from the share, but in read-only. While this alternative is simpler, it is also more restrictive because it does not allow systems or users that do not originally have access to the share to access the snapshot.

Data model impact

Three new tables will be needed:

ShareSnapshotInstanceExportLocations
ShareSnapshotAccessMapping
ShareSnapshotInstanceAccessMapping

A new export_locations property will be added to the existing ShareSnapshotInstance table and a mount_snapshot_support property will be added to the existing Share table.

REST API impact

The following new API methods will be implemented:

(202, POST) snapshot-access-allow: allows access to the snapshot

URL: /snapshots/<id>/action Body: {“access-allow”: {“access_type”: <value>, “access_to”: <value>}}

(202, POST) snapshot-access-deny: denies access to the snapshot

URL: /snapshots/<id>/action Body: {“access-deny”: {“access_id”: <value>}}

(200, GET) snapshot-access-list: list the access of the snapshot

URL: /snapshots/<id>/access-list

(200, GET) snapshot-export-location-list

URL: /snapshots/<id>/export-locations

(200, GET) snapshot-instance-export-location-list

URL: /snapshot-instances/<id>/export-locations

(200), GET) snapshot-export-location-show

URL: /snapshots/<snap-id>/export-locations/<el-id>

(200, GET) snapshot-instance-export-location-show

URL: /snapshot-instances/<si-id>/export-locations/<el-id>

Driver impact

Add driver interfaces:

def snapshot_allow_access(self, context, snapshot, access,
                          share_server=None):
"""Allow access to the snapshot."""

def snapshot_deny_access(self, context, snapshot, access,
                         share_server=None):
"""Deny access to the snapshot."""

To support this feature, the create_snapshot method must be changed to return the snapshot export locations, and the drivers must report mount_snapshot_support as True.

Security impact

None

Notifications impact

None

Other end user impact

This feature will be implemented in python-manilaclient and manila-ui. The commands for allowing and denying access to snapshots will follow the current implementation for allow and deny access to shares and getting the snapshot export locations will be similar to the shares implementation.

Performance Impact

None

Other deployer impact

None

Developer impact

This will require a change on the driver interface. To support this feature, drivers will need to implement the new methods.

Implementation

Assignee(s)

Primary assignee:: tiago.pasqualini

Work Items

Implement the core feature with functional tempest and scenario test coverage
Implement snapshot-access-allow and snapshot-access-deny commands in python-manilaclient as well as python-manilaclient functional tests
Implement mountable snapshots in one of the first-party drivers
Implement manila-ui changes (snapshot allow/deny access and expose snapshot export locations)
Create mountable snapshots documentation

Dependencies

This feature depends on Create share from snapshot extra spec, since it will remove overload that currently exists on the ‘snapshot_support’ extra spec.

Testing

Unit tests
Functional tempest tests
Scenario tests

Documentation Impact

Docstrings
Devref
API reference
User guide

References

Etherpad: https://etherpad.openstack.org/p/mitaka-manila-mountable-snapshots

Add share_groups quota

Wed, 12 Apr 2017 00:00:00

https://blueprints.launchpad.net/manila/+spec/add-share-groups-quota

Manila is able to create several different kinds of resources, which use real back-end resources. Since real resources are always finite, it is practical to have different kinds of limits/quotas for them. And one of not covered resources by limits is amount of share groups that user is able to create. Share Groups were added in Ocata timeframe and were, originally, consistency-groups in older than Ocata releases.

Problem description

The problem is in possibility to schedule creation of unlimited amount of share groups, that can lead to 2 bad things:

garbaging of DB with any amount of DB table records;
waste of real back-end resources.

So, implementation of quotas for share_group resource will solve above mentioned problems.

Use Cases

Control of used manila resources will now be more flexible, because admin will be able to set additional quotas for previously untracked manila resource.

Proposed change

Implement new quota resource called “share_groups” in existing quota mechanism. This quota will track amount of created share groups resources and restrict creation of them after limits are exceeded. Make default value be equal to amount of shares.

Alternatives

Do not implement quotas for share groups. In this case any user will be able to create unlimited amount of share groups.

Data model impact

Existing DB schema does not require any changes. Hence, no new DB migrations are required. We will reuse existing logic, where we can track any amount of resources by their names.

REST API impact

Only one (1) API will be changed, it is “quota-update”. It will be able to handle one more key provided in POST data, called “share_groups”. Also, “quota-show” command will return dict with quota resources [and usages] for this key too. Both these changes require bump of microversion and will be available only with this or later one microverisons. If for moment of this spec implementation share-groups feature stays ‘experimental’, then these API changes should be available only if ‘X-OpenStack-Manila-API-Experimental’ header provided and set to ‘true’ boolean value.

Update quotas

URL: /quota-sets
Method: PUT

JSON body:

{
  'quotas': {
    'share_groups': 100,
  }
}

Driver impact

None

Security impact

None

Notifications impact

None

Other end user impact

Users will now be restricted in amount of share groups compared to previous behavior.

Manila client will have additional following optional key to “quota-update” command:

$ manila quota-update <tenant_id> --share-groups %amount_as_integer_value%

In general, quota should be able to be set to any value and in case usage exceeds current quota, request for creation of new resource will be refused.

Performance Impact

Resource usages will be checked each time we create/delete resources, but it is relatively cheap operation.

Other deployer impact

Deployers will be able to set quotas for share groups using either configuration file or “quota-update” API. It is optional possibility, not a mandatory action. New option will be named in the same way as all existing ones - “quota_share_groups”. It will accept only integer values.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

vponomaryov ( vponomaryov@mirantis.com )

Work Items

Server side API changes, including unit and functional tests
Manila Client side changes, including unit and functional tests
Manila UI side changes, including unit tests

Dependencies

None

Testing

unit tests in ‘manila’, ‘python-manilaclient’ and ‘manila-ui’ projects
functional tests in ‘manila’ and ‘python-manilaclient’ projects

Documentation Impact

admin guide: add new ‘–share-groups’ parameter to ‘update-quota’ command.
devref: reflect the proposed change

References

Share Groups feature spec: https://github.com/openstack/manila-specs/blob/master/specs/ocata/manila-share-groups.rst

Support IPv6 access in manila

Wed, 12 Apr 2017 00:00:00

https://blueprints.launchpad.net/manila/+spec/support-ipv6-access

Problem description

Currently ip access rule type only supports IPv4. A valid format is xxx.xxx.xxx.xxx or xxx.xxx.xxx.xxx/xx. This is not enough when users want to enable IPv6 in manila. The most affected areas are API (access rules) and the drivers (access rules & export locations).

Use cases

End users with IPv6 interfaces also want to access their shares, and manila currently only supports IPv4 protocol when access is IP-based.

Proposed change

Support IPv6 in access rule

As agreed by the manila community in IRC meeting (topic IPv6) [1], manila can address this requirement by adding the IPv6 format validation for ip access rule type. Change the validation approach from our own function (_validate_ip_range) to a standard module ‘ipaddress’ (both IPv4 and IPv6 are supported).

With the ipaddress module, the IPv6 formats below are all considered acceptable:

0DB8:02de:0000:0000:0000:0000:0e13   #uncompressed
DB8:2de:0:0:0:0:0e13                 #leading zeros removed
DB8:2de::0e13                #consecutive sections of zeroes omitted
dB8:2De::0E13                #mixed case
dB8::/32                     #network range

and manila will convert all of these into lower and compressed format, which means the result of first example will be like (it’s also a canonical format which is recommended in RFC5952 [2]):

2001:db8:2de::e13

Support IPv6 in network plugins

The standalone and neutron network plugins (nova network plugins are ignored as they are being deprecated) will have to be modified to support IPv6.

Network plugins will add new boolean configuration options ‘network_plugin _ipv4_enabled’ and ‘network_plugin_ipv6_enabled’ to indicate its enabled ip versions. The default values are ‘True’ and ‘False’ respectively. We will ultimately support both IPv4 and IPv6 enabled for single network plugin, but for this phase, restricted to the current manila network plugin architecture, only one option should be configured True (either IPv4 or IPv6) in Ocata. We will also deprecate the ‘standalone_network_plugin_ip _version’ which will be equivalent to ‘network_plugin_ipv6_enabled=True’ and ‘network_plugin_ipv4_enabled=False’, or on the contrary.
Network plugins should be updated to allocate network resource based on their ip version’s configuration (both for user network and admin network). For example, the neutron network plugins will only allocate IPv6 ports if ‘network_plugin_ipv6_enabled = True’. If no suitable resource is found with the configured or specified network resource the allocation process would fail with exception.

Support IPv6 in drivers

As the export location support’s status varies in different drivers, manila will introduce two extra_specs (and driver capabilities) ‘ipv4_support’, ‘ipv6_support’ to make backends and users easy to support and configure.

Add new optional boolean extra_specs ‘ipv4_support’, ‘ipv6_support’ for share type that indicates the required ip version (Omitting either extra_spec means “don’t care” about a match for that capability).
Drivers have to report the capabilities with the ip version which they can support, the capability is reported in the same format with extra_specs. There are two cases that driver should consider: - DHSS=false, the driver should report IPv4 is True only if it is actually configured with IPv4 export locations, and IPv6 is True only if it is actually configured to support IPv6 export locations. - DHSS=true, the driver should report IPv4 is True only if both driver and the configured network plugins (both user and admin) have the IPv4 capability, so does the IPv6.
The manila scheduler will choose the correct backend with capability filter when ip version extra_specs are involved.
Driver must return export locations according to its IP version capabilities, and only host addresses are allowed in export locations, not hostnames.

Alternatives

None

Data model impact

None

REST API impact

The API microversion will have to be bumped for this.

The IPv6 address and IPv6 network are supported in allow_access action.

Driver impact

Drivers will be updated so that existing drivers that use IPv4 addresses will report ipv4_support=True. When drivers support both IPv4 and IPv6, they need report their support versions based on their capabilities and configuration of the storage backend. Also the following functions might need to be changed:

_update_share_stats
create_share
delete_share
create_share_from_snapshot
update_access
setup_server
manage_existing
create_replica
promote_replica
migration_start
migration_complete
create_consistency_group
create_consistency_group_from_cgsnapshot

Also, drivers must export locations as addresses rather than as host names since addresses are specifically IPv4 or IPv6 whereas host names could resolve to either an IPv4 or an IPv6 address at any particular time.

Security impact

None

Notifications impact

None

Other end user impact

python-manilaclient will be changed to reflect the validation approach changes for ip access type.

The access-allow command with IPv6 supported will be like:
```
manila access-allow test_share ip "AD80:0:0:0:ABAA:0:C2:2"
```
manila-ui will have corresponding UI changes to deal with the valid value for ip access rule type after python-manilaclient implementation.

Performance impact

None

Other deployer impact

Deployers must ensure their IP version preferences are reflected on the storage backends, such that data interfaces have IPv4 and IPv6 addresses as desired.

Developer impact

Changes to the driver interface are noted above. Third party backends will implement these changes and report their IPv6’s ability if they want to enable IPv6 in their drivers (the support matrix can be found here [3]).

Implementation

Assignee(s)

Primary assignee:

Work items

Implement core feature
Report capability ipv4_support=True for existing IPv4 drivers
Implement IPv6 in network plugins.
Implement IPv6 ACL in generic, huawei and dummy drivers
Implement tempest tests
Implement Scenario tests
Implement manila-ui support
Implement python-manilaclient support
Update manila API documentation
Update manila guide documentation on IPv6

Dependencies

None

Testing

Unit test
Functional test
Tempest test

Documentation impact

Update user guide
Update manila API documentation
Update manila developer documentation on IPv6

References

[1] http://eavesdrop.openstack.org/meetings/manila/2016/manila.2016-08-25-15.00.log.html [2] https://tools.ietf.org/html/rfc5952#section-4 [3] http://docs.openstack.org/developer/manila/devref/share_back_ends_feature_support_mapping.html

Support retrieving shares filtered by export-location

Wed, 12 Apr 2017 00:00:00

https://blueprints.launchpad.net/manila/+spec/support-filter-share-by-export-location

Currently we can only filter the shares by share name, status, share type, extra specs, share server id, metadata, snapshot, host, share network, project id, and consistency group, but this is too limited for user’s various cases. Manila doesn’t support filtering the shares by export location. This spec intends to add the ability for Manila.

Problem description

It’s a common use case to list NAS shares with filters, and Manila has the API to cover this, but it does not satisfy the situation when export location filter is desired. The main problem is that having export location we don’t have possibility to get to know which manila share owns it.

Use Cases

When user has a share that it has been mounted in VM, and user only knows the share export location path. User often wants to quickly filter a share by share export location in Manila. This change will help him achieve this.

Proposed change

The modified version of shares and share-instances list will be identical at the most part, except for the parts mentioned below:

A query parameter ‘export_location’ will be introduced to shares and share-instances list API (GET shares and GET shares/detail, GET share-instances and GET share-instances/detail).
In the API service, the export_location will be added in share and share instance search options. In Manila DB’s method, will add a new export location filter in SQLAlchemy API, and add a new logic which would filter all the shares or share instances with the export location.
The Manila client will also add one argument ‘–export_location’ to support this.

Alternatives

We could get share instance ID from share export location path, and use “share-instance-show <instance>” command to find out associated share.

Data model impact

None

REST API impact

The API microversion will have to be bumped up for the new APIs below.

Share and share-instances list API will accept new query string parameter ‘export_location’. Administrators can pass path and id of export_location to retrieve shares filtered.

GET /v2/{tenant_id}/shares?export_location=test_path
GET /v2/{tenant_id}/share-instances?export_location=test_path

Detailed share and share-instance list API will also accept new query string parameter ‘export_location’. Administrator can pass path and id of the export location to retrieve shares filtered.

GET /v2/{tenant_id}/shares/detail?export_location=test_path
GET /v2/{tenant_id}/share-instances/detail?export_location=test_path

The search will be among “allowed”, for user, shares - all his shares and public ones.

Manila client impact

Manila client will add a new parameter ‘–export_location <export-location>’ to command ‘list’, the modified version of command will be like:

* list --export_location <export-location> <other existing command arguments>
* share-instance-list --export_location <export-location> <other existing command arguments>

Security impact

None

Notifications impact

None.

Other end user impact

None.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: zhongjun2(jun.zhongjun2@gmail.com)

Work Items

Add ‘export_location’ parameter to ‘list’ collection.
Add related tests in Manila.
Add documentation in Manila.
Add new argument for command ‘list’ in Manila client.
Add related tests in Manila client.

Dependencies

None

Testing

Unit and tempest tests whether new API works correctly.
Manila client’s unit tests and functional tests on new added argument.

Documentation Impact

The Manila API documentation will need to be updated to reflect the REST API changes.

References

Support quotas per share type

Wed, 12 Apr 2017 00:00:00

Blueprint: https://blueprints.launchpad.net/manila/+spec/support-quotas-per-share-type

Currently, quotas in Manila are configurable on a per tenant and per user basis. They allow admins to set the values for the number of shares, the number of snapshots, or the amount of gigabytes for a project and allow a project member to use resources within these given limits. This spec describes the extension of the current quota scheme: in addition to an overall quota for, say, the number of shares, admins will have the additonal option to define the quota for the number of shares for a given share type. This functionality is similar to the one offered by Cinder for quotas per volume type.

Problem description

Quotas are meant to help administrators to control the provisioning of available resources, such as the number of shares or the space shares take up on the configured backends. While Manila supports quotas, the current quota scheme does not take share types into account. Considering setups with multiple backends (accessible via different share types), administrators have hence no means to control the resource consumption on the various backends.

Use Cases

In setups with multiple share-types, per share type quota will allow resource provider to have better control over the provisioned resources.

Proposed change

The proposal is to add per share type quotas in addition to project and user quotas. When a resource is requested, the share type specific quota is checked first. If this check passes, the request is validated against the project/user quota for that resource. It will be possible to set share type quotas only per project, but not per user in scope of single project. It is intended restriction to avoid creation of one more quota layer (third), that will be useful only in case when we need to set different share type quotas for users that are related to one single project.

Alternatives

An alternative would be to make all share types private, require users to create a new tenant for each share type they’d like to access, and then grant access to the corresponding tenants. This will become impracticable for the users when the number of share types goes up, or when quota schemes for other projects and resources would follow the same approach.

Other possible solution is to port Cinder’s implementation. Where instead of additional quota layer Cinder creates additional quota resources for each created volume type and names them using following template:

%RESOURCE_NAME%_%VOLUME_TYPE_NAME%

But this approach provides two inconveniences:

“quota-show” command’s response grows with creation of each new share/volume type

names of new share-type-based quota resources can be confusing depending on names of share types. For example, share type ‘default’ and ‘shares’ quota resource will produce additional quota resource named ‘shares_default’ in parallel to existing ‘shares’ project quota resource that is not really intuitive for understanding.

Data model impact

New share type quotas will have its own DB model similar to existing DB model that is used for “user” quotas with the same relation to “project” quotas. So, it will behave completely the same way as “user quotas”. We will have following structure:

digraph quotas { center=true; node_pr [label = "Project Quotas", shape = ellipse, width = 2.2]; node_prs[label = "<f0> PROJECT_1|<f1> PROJECT_2|<f2> \.\.|<f3> PROJECT_N", shape = record, fontsize = 10]; "node_prs":f0 -> "node_pr" [dir=back]; "node_prs":f1 -> "node_pr" [dir=back]; "node_prs":f3 -> "node_pr" [dir=back]; node_u [label = "Users Quotas", shape = ellipse, width = 2.2]; node_us[label = "<f0> USER_1|<f1> USER_2|<f2> \.\.|<f3> USER_N", shape = record, fontsize = 10]; "node_u" -> "node_us":f0; "node_u" -> "node_us":f1; "node_u" -> "node_us":f3; node_st [label = "Share Type Quotas", shape = ellipse, width = 2.2]; node_sts[label = "<f0> ST_1|<f1> ST_2|<f2> \.\.|<f3> ST_N", shape = record, fontsize = 10]; "node_st" -> "node_sts":f0; "node_st" -> "node_sts":f1; "node_st" -> "node_sts":f3; "node_pr" -> "node_u"; "node_pr" -> "node_st"; }

REST API impact

Three (3) quota APIs from four (4) will be changed - ‘update’, ‘delete’ and ‘get’ API. ‘defaults’ will stay unchanged and will return project default quotas as they will be default for each share type. All three APIs will start handling additional ‘share_type’ GET argument, which will be mutually exclusive with existing ‘user_id’ argument. Response structure will stay the same, with the only exception - quota ‘share_networks’ will be absent in ‘quota-show’ response body, because this kind of quota has no value in case of share types. Also, it will not be possible to set/update this quota per share type. New ‘share_type’ argument should be able to accept both - name and UUID of a share type. API changes will require microversion bump. Old microversions will behave as did before.

Get quotas

URL: /quota-sets?share_type=%name_or_id%
Method: GET

Response Body:

{
    'quota_set': {
        'shares': -1,
        'gigabytes': -1,
        'snapshots': -1,
        'snapshot_gigabytes': -1,
    }
}

Note

‘share_networks’ quota is absent in response, because it is user/project quota only.

Delete quotas
- URL: /quota-sets?share_type=%name_or_id%
- Method: DELETE

Update quotas

URL: /quota-sets?share_type=%name_or_id%
Method: PUT

JSON body:

{
    'quotas': {
        'shares': -1,
        'gigabytes': -1,
        'snapshots': -1,
        'snapshot_gigabytes': -1,
    }
}

Driver impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

The Manila client will get a new optional ‘–share-type’ argument for the ‘quota-update’, ‘quota-show’ and ‘quota-delete’ commands, so the modified version of the client will be:

$ quota-update ... [--share-type <name_or_id>] ... <tenant_id>
$ quota-show ... [--share-type <name_or_id>] ... --tenant <tenant_id>
$ quota-delete ... [--share-type <name_or_id>] ... --tenant <tenant_id>

When the ‘–share-type’ argument is omitted, the project quota will be updated, just as now. If the ‘–share-type’ argument is passed, the quota of the corresponding resource(s) for the specified share type will be set.

Performance Impact

Additional layer of quota usages calculation will slow down things at some level.

Other deployer impact

When creating new share types, the initial quotas for the new share type will be the project quota. Hence, for new tenants, the initial quota must be set for the share types if they shall differ from the project quota.

Developer impact

Developers of new features that consume quota will need to provide additional “share_type_id” argument to ‘reserve’, ‘commit’ and ‘rollback’ quota commands. Underlying DB layer will handle both quota layers automatically.

Implementation

Assignee(s)

Primary assignee:

vponomaryov ( vponomaryov@mirantis.com )

Work Items

Server side API changes
Manila Client side changes
Manila UI side changes

Dependencies

None.

Testing

unit tests in ‘manila’, ‘python-manilaclient’ and ‘manila-ui’ projects
functional tests in ‘manila’ and ‘python-manilaclient’ projects

Documentation Impact

admin guide: add new ‘–share-type’ parameter to ‘update-quota’, ‘quota-show’ and ‘quota-delete’ commands.
devref: reflect the proposed change

User Messages

Wed, 12 Apr 2017 00:00:00

https://blueprints.launchpad.net/manila/+spec/user-messages

For quite some time, OpenStack services have been wanting to be able to send messages to API end users (by user I do not mean the Operator, but the user that is interacting with the client).

At the Mitaka cinder mid-cycle [6], we decided to begin solving this problem by adding a /messages resource to the API in order for end-users to query for more information about errors. We should follow suit in manila.

Problem description

If manila operations fail, e.g. create share or create share group, user gets no detailed information whatsoever. Only the operation status is updated, e.g. error or failed, without any update to user.

Use Cases

Provide more information to user about failed async operations in following situations. Below is a list of reasons/messages we could create when a resource (e.g. a share) goes to error state:

Shares:
- Creation fails because no valid host is found - if chain of filters returned no hosts, the last filter name is in the message.
- Creation fails because driver does not expect share-network to be provided with current configuration.
- Creation fails because failed to get share server.
- Creation fails for an unknown reason.
- Deletion of access rules failed.
- Extending error (represented by status too).
- Shrinking fails because quota update fails.
- Shrinking fails because of possible data loss.
Share group:
- Creation fails because can not choose compatible share-server.
- Creation fails because of share instance failed to get share server.
- Creation fails for an unknown reason.
- Creation fails because driver does not expect share-network to be provided with current configuration.
Share group snapshot:
- Creation fails for an unknown reason.
- Deletion fails for an unknown reason.
Share replica:
- Creation fails because an ‘ACTIVE’ replica must exist in ‘AVAILABLE’ state to create a new replica for share.
- Creation fails because driver does not expect share-network to be provided with current configuration.
- Creation fails because failed to get share server for share replica creation.
- Creation fails for an unknown reason.
- Deletion of access rules failed.
- Update of share replica fails because of a driver error.
Snapshot:
- Creation fails for an unknown reason.
- Creation of replicated snapshot fails for an unknown reason.
- Deletion of access rules failed.
- Revert to snapshot fails for an unknown reason.
Snapshot replica:
- The driver was unable to delete some or all of the share replica snapshots on the backend/s.
- Update fails because replica snapshot was not found on the backend.
- Update fails because of a driver error.

The above list reflects a subset of situations where error state is set for a resource in the share service [3]. [4] shows creation of scheduling error messages.

Proposed change

When an error occurs during an async operation, following details about the error are stored in database: * resource type and ID for the resource on which the error occurred * action during which the error occurred * error detail (user-friendly error explanation) * error level * expiration time * request ID * project ID

Because exceptions raised by drivers or manila services may contain sensitive information (e.g. about backend infrastructure, host names…), this exception is not exposed to user, instead only sanitized error detail is exposed.

User can get messages either directly through the new /messages API or through manila CLI which allows showing a message, listing all messages and deleting existing messages. This spec doesn’t address integration with Horizon.

The same approach is already implemented in cinder.

Alternatives

User facing notifications - use the existing notification framework in combination with an AMQP consumer to pull messages off and provide an endpoint for the user. Faults with this approach are that we do not want to display the current information in notifications to the user and it will require many more services as dependencies.
Retrieving user messages via a separate service (such as zaqar) This approach suggests storing user messages in another service that the user could query or the service could utilize webhooks to notify the user. One major drawback to this approach is the complexity in writing bindings for the separate service(s) and the need for a separate service as a dependency.

What this specification does not solve

State change notifications. This solution does not intend to solve the use-case of alerting users when a share or any other resource changes state. For example, when a share changes its status from CREATING to AVAILABLE.

Data model impact

New messages table in database to store all messages. This table may prove to grow large in a cloud with lots of errors. The admin will be able to utilize the expires_at column to delete old messages by a manila-manage CLI command:

id = Column(String(36), primary_key=True, nullable=False)
project_id = Column(String(36), nullable=False)
message_level = Column(String(32), nullable=False)
request_id = Column(String(255), nullable=True)
resource_type = Column(String(255))
resource_uuid = Column(String(36), nullable=True)
action = Column(String(255), nullable=False)
expires_at = Column(DateTime, nullable=True)
detail = Column(String(255), nullable=True)
deleted = Column(String(36), default=’False’)

Only a constant will be stored in ‘detail’ column and there will be a mapping

(in a file) from these constants to user friendly messages:

message_map = {

MessageIds.UNEXPECTED_NETWORK: _(“Current back end configuration does not “: “support creating shares within project ” “defined share-networks.”),

MessageIds.QUOTA_UPDATE: _(“Failed to update quota.”)

}

REST API impact

Message APIs:

List messages

URL: /v2/<tenant>/messages
Method: GET (200, 400)
URL args:
- offset - integer, set offset to define start point of message listing, 0 or bigger
- limit - integer, maximum number of messages to return, 1 or bigger
- sort_key - string, key to be sorted (i.e. ‘created_at’ or ‘resource_type’)
- sort_dir - string, sort direction, should be either ‘asc’ or ‘desc’.

JSON body:

{
  "messages": [
    {
     "id": "5429fffa-5c76-4d68-a671-37a8e24f37cf",
     "action": "ALLOCATE_HOST",
     "user_message": "No storage could be allocated for this share
                     request. Trying again with a different size or
                     share type may succeed.",
     "message_level": "ERROR",
     "resource_type": "SHARE",
     "resource_uuid": "f292cc0c-54a7-4b3b-8174-d2ff82d87008",
     "created_at": 2015-08-27T09:49:58-05:00,
     "expires_at": 2015-09-27T09:49:58-05:00,
     "request_id": "req-936666d2-4c8f-4e41-9ac9-237b43f8b848",
    }
  ]
}

Show message

URL: /v2/<tenant>/messages/<message_id>
Method: GET (200, 403 Forbidden, 404 Not Found)

JSON body:

{
  "message": {
    "id": "5429fffa-5c76-4d68-a671-37a8e24f37cf",
    "action": "ALLOCATE_HOST",
    "user_message": "No storage could be allocated for this share
                    request. Trying again with a different size or
                    share type may succeed.",
    "message_level": "ERROR",
    "resource_type": "SHARE",
    "resource_uuid": "f292cc0c-54a7-4b3b-8174-d2ff82d87008",
    "created_at": 2015-08-27T09:49:58-05:00,
    "expires_at": 2015-09-27T09:49:58-05:00,
    "request_id": "req-936666d2-4c8f-4e41-9ac9-237b43f8b848",
  }
}

Delete message
- URL: /v2/<tenant>/messages/<message_id>
- Method: DELETE (204, 403 Forbidden, 404 Not Found)

Driver impact

None

Security impact

Only message type (instead of internal error message) is kept in database and exposed to user to assure that no sensitive information is exposed. For example back-end driver might expose sensitive information or infrastructure hostnames.

Notifications impact

None

Other end user impact

Users will be able to list, show and delete user messages through the new API or CLI commands.

Performance Impact

None

Other deployer impact

New configuration option message_ttl that will dictate the number of seconds after the messages creation time to set the ‘expires_at’ attribute on generated messages.
The messages table will be potentially large and may be reaped based on the ‘expires_at’ column. Where all messages with a expires_at date earlier than the current time can be safely deleted.

Developer impact

Developers should be aware of use-cases where the user needs information about an error. In these situations, an appropriate user message should be written and creation of the message added in the specific code path(s).

Implementation

Assignee(s)

Primary assignee:: Jan Provaznik Alex Meade

Alex Meade is author of the messages approach, this spec is mostly based on Alex’s existing patch (see references).

Work Items

Add new database table and migration for storing messages.
Add /messages API [1].
Extend CLI with message commands [2].
Extend Horizon UI with an interface for listing, showing and deleting messages.
Add “message create” calls on proper places in scheduler and share services [3] and [4].
Add a new “manila-manage message delete-expired” command for deleting expired messages from database.

Dependencies

None

Testing

Tempest tests should be written and run in the gate. It may prove difficult to implement complete functional testing of the feature as messages will not be created unless there is an error, which may be difficult to trigger. However, some operations are easy to trigger failure with unlimited quotas. One example is creating a share too big to be stored on the backend.

Documentation Impact

REST API documentation.
New config option, message_ttl (time to live).
New API policies for messages.

References

[1] Alex Meade’s existing patch (not up-to-date, it doesn’t reflect this spec precisely): https://review.openstack.org/#/c/313549/
[2] CLI message commands: https://review.openstack.org/#/c/429614/
[3] messages added into share service: https://review.openstack.org/#/c/443101/
[4] messages added into scheduler service: https://review.openstack.org/443102
[5] A related cinder spec (which is more generic though): https://specs.openstack.org/openstack/cinder-specs/specs/newton/summarymessage.html
[6] cinder Mitaka midcycle notes: https://etherpad.openstack.org/p/mitaka-cinder-midcycle-user-notifications
[7] cinder user messages improvement spec: https://review.openstack.org/#/c/451761/

Manila backup & restore share

Tue, 28 Mar 2017 00:00:00

Blueprint: https://blueprints.launchpad.net/manila/+spec/share-backup

Backup and restore share is a valuable feature for most of storage users, especially for NAS user, but currently, manila itself doesn’t support backup and restore share features, this spec proposes a backup and restore solution based on the existing one from cinder.

Problem description

Today we can’t use manila commands to backup shares. These shares reside on the storage backend itself. Providing a way to backup shares directly will allow the user to protect the shares on a backup device, separately from the same storage backend.

Use Cases

There are users who have many shares and would like to protect these shares. This proposal of share backup provides a manila level of data protection.

There are other projects in OpenStack focusing on data protection such as Freezer, Karbor, Raksha, etc. They are all in different stages of design, development, or adoption. The backup API in manila is not a replacement of those projects. Instead, manila APIs can be consumed by those higher level projects for data protection and can also be used directly by users who do not need those higher level projects.

When user backing up files from a share in a naive way, we lose metadata. By having backup API in Manila, we can retain metadata (when backend is identical)

Also when talking about share backup, most of the users would like to enforce backup policies on them. Such as incremental backup every two days, create backup local or remote, only keep latest 5 copies, etc. The policies are not part of this spec and will be considered for future.

Proposed change

New API collection for backups

In order to support backup, We will introduce the basic operations create /update/delete/list/show/restore for backups. We also allow to reset the backup state in some scenarios.
New database resource backup

When creating a backup, user can specify share_id and availability_zone. Information related to backup i.e. backup_id, share_id, status, AZ and host are stored in database.
New backup driver

A new type of driver called backup driver will be introduced to take responsibility for share related operations. The backup driver will configured at the manila data service node, and provide the basic abilities
1. Create a backup.
2. Delete a backup.
3. Restore a backup in specified share.
Correspondingly, we will implement these with a new and simple backup driver called ‘NFSBackupDriver’. This driver is copied from cinder [1] and aims to provide the basic backup ability. The vendor that supports nfs, must provide space for nfs to interconnect with nfs backup drivers. The implementation of nfs backup driver will be generic though. The backup process for this driver would be:
1. Make sure share is in available state and not busy.
2. Allow read access to share and write access to backup share.
3. Mount the share and backend driver’s share(i.e. backup share) to the data service node.
4. Copy data from share to backup share.
5. Unmount the share and backup share.
6. Deny access to share and backup share.
Also, several new configuration options will be added to support backup driver, at present, only one backup backend is allowed for simplicity (we don’t have to report the backup driver’s capabilities and filtering the backends). By default no backup driver will be enabled:
```
# OPTION1: enabled backup backend
enabled_backup_backend = backup1
[backup1]
# OPTION2: which backup driver will be used for this backend
backup_driver = manila.data.drivers.test.TestBackupDriver
```
New status for backup and share:
1. backup(
  creating, available, deleting, deleted, error_deleting backup_restoring, error)
2. share(
  backing_creating, backup_restoring, backup_restoring_error)
During backup, share will be marked as busy and other operations on share such as delete, soft_delete, migration, extend, shrink, ummanage, revert_to_snapshot, crate_snapshot, create_replica etc can not be performed unless share become available. Finally, whether or not the share is successfully backed up, the state of the share is rolled back to the available state. In case backup fails, share task_state will contain the failure information. Also, share message will be recorded.
New clean up actions

The backup and restore actions could break when service is down, so new clean up action will be added to reset the status and clean temporary files (if involved).
New quotas for backup
1. quota_backups, indicate the share backups allowed per project.
2. quota_backup_gigabytes, indicate the total amount of storage, in gigabytes, allowed for backups per project.
Correspondingly, we will add new configuration options.

Alternatives

We could use the third-party projects to backup the file shares.

Data model impact

Add table for backup

Field	Type	Null	Key	Default
created_at	datetime	YES		NULL
updated_at	datetime	YES		NULL
deleted_at	datetime	YES		NULL
deleted	tinyint(1)	YES		NULL
id	varchar(36)	NO	PRI	NULL
share_id	varchar(36)	NO		NULL
user_id	varchar(255)	YES		NULL
project_id	varchar(255)	YES		NULL
host	varchar(255)	YES		NULL
availability_zone	varchar(255)	YES		NULL
display_name	varchar(255)	YES		NULL
display_description	varchar(255)	YES		NULL
status	varchar(255)	YES		NULL
size	int(11)	YES		NULL

New field in shares table

Field

Type

Null

Key

Default

Extra

source_backup_id

varchar(255)

YES

NULL

Field	Type	Null	Key	Default	Extra
source_backup_id	varchar(255)	YES		NULL

CLI API impact

Add new commands to openstackclient(OSC):

openstack share backup create [–name <name>]: [–description <description>] [–availabity-zone <availability-zone>] <share>

name: Name of backup. Default=None
description: Backup description. Default=None.
availability-zone: Availability-zone of backup.
share: Name or ID of share to backup.

openstack share backup restore <backup>

backup: ID of backup to restore.

openstack share backup list [–share <share>]

share: Filter backups by a share name or ID.

openstack share backup show <backup>

backup: ID or name of backup

openstack share backup set <backup>: [–name <name>] [–description <description>]

backup: ID or name of backup
name: Name of backup. Default=None
description: Backup description. Default=None.

openstack share backup delete <backup>: [–force <force>]

backup: ID or name of a backup.
force: Allows deleting backup of a share when its status is other than “available” or “error”. Default=False.

REST API impact

APIs will be experimental, until some cycles of testing, and the eventual graduation of them.

Creating a share backup:

POST /v2/share-backups

Request:

{
    "share_backup": {
        "share": "77eb3421-4549-4789-ac39-0d5185d68c28",
        "name":  "backup_share",
        "description": "This is my backup",
    }
}

Backup details name and description are optional.

If the share is not known to manila, the API will respond with 404 Not Found. If the share is not in available state or share has snapshots, the API will respond with 400 Invalid Share. If the project’s share backup quota has exceeded, the API will respond with 413 QuotaError.

Response(202 Accepted):

{
    "share_backup": {
        "share_id": "77eb3421-4549-4789-ac39-0d5185d68c28",
        "created_at": "2016-06-01T21:12:12.617687",
        "updated_at": "2016-06-01T21:12:12.617687",
        "id": "77eb3421-4549-4789-ac39-0d5185d68c29",
        "project_id": "e10a683c20da41248cfd5e1ab3d88c62",
        "display_name": "backup_share",
        "display_description": "This is my backup",
        "status": "creating"
    }
}

Delete a backup:

DELETE /v2/share-backups/{backup_id}

If the share backup is not known to manila, the API will respond with 404 Not Found. If the share backup state not in available or error, the API will respond with 400 Invalid State. API will respond 202 if request is accepted.

Detailed listing of share backups:

GET /v2/share-backups/detail

Response(200 OK):

{
    "share_backups": [
        {
            "availability_zone": "az1",
            "created_at": "2016-04-02T10:35:27.000000",
            "id": "2ef47aee-8844-490c-804d-2a8efe561c65",
            "display_name": "my_backup",
            "display_description": "this is description",
            "size": 1,
            "status": "available",
            "share_id": "e5185058-943a-4cb4-96d9-72c184c337d6",
        },
        {
            "availability_zone": "az2",
            "created_at": "2016-05-02T10:35:27.000000",
            "id": "2ef47aee-8844-423c-804d-2a8efe561623",
            "display_name": "my_backup_1",
            "display_description": "this is description",
            "size": 2,
            "status": "available",
            "share_id": "e5185058-943a-4cb4-96d9-72c184c33dsd",
        }
    ]
}

Restore a share backup:

POST /v2/share-backups/{backup_id}/action

Request:

{
    "restore": null
}

The backup will be restored in source share (i.e. share from which backup was created) will be used for restore. In case, source share or share backup is not known to manila, API will repsond with 404 Not Found. In case, source share size is different than size of backup, API will respond with 400 Invalid Share. However, API will respond 202 if request is accepted.

Update a share backup information:

PUT /v2/share-backups/{backup_id}

Request:

{
    "share_backup": {
        "display_name": "test share backup",
    }
}

If the share backup is not know to manila, the API will respond with 404 Not Found. API will respond 200 if request is accepted.

Driver impact

The backup driver needs to implement these functions:

def backup(self, backup, share):
    """Create a backup of a specified share.

    The driver should return the backup model with new created
    backup content if creation is successful, manila
    will update the model after this. For example::
    ```
    {"backup":
        {
            "name": "backup_one",
            "id": "e5185058-943a-4cb4-96d9-72c184c33d12",
            "created_at": "2016-04-02T10:35:27.000000",
        }
    }
    ```

    :param backup: backup object.
    :param share: share object.
    :returns: backup: backup object with backup object updated.
    """
    return


def restore(self, backup, share):
    """Restore a shared backup.

    Driver will restore the specified backup.

    :param backup: backup object.
    :param share: share object.
    """
    return

def delete(self, backup):
    """Delete a saved backup.

    Driver will delete the share backup.

    :param backup: backup object.
    """
    return

Security impact

During backup process the data node would have access to read the share data. If the deny access phase fails, the node will continue forever with access to the user’s data.

Notifications impact

None

Other end user impact

End user will be unavailable or restricted to other share operations while it is backing up. Such as: extend/shrink share, replication share, share-group operation, migration share.

Performance Impact

None

Other deployer impact

The deployer will be able to backup a share.

Developer impact

Implementation

Assignee(s)

Primary assignee:

kpdev(kinpaa@gmail.com)

Work Items

Implement core feature.
Implement backup in NFSBackupDriver.
Implement backup command in python-manilaclient.
Implement tempest support.
Implement manila-ui support.
Support manila backup in devstack plugin.

Future Work Items

New table ‘backup_metadata’ where the data service or the user can store some useful metadata such as the location of the backup. The metadata can possibly used to restore the backup, even in the event of a catastrophic database failure.
Add support to create share from backup, where ‘openstack share create’ API will accept –backup <backup_id> option and create share of the size as that of backup. In addition, backup data will be copied to share.
Add support to handle backup failover in case manila service is interrupted, i.e. when service is back online, monitor the backups, check on their completion and recover if posssible.
Restore operation can be enhanced to consider restore to share of different size than backup. If restore share is smaller in size, it will be expanded before restore and if its larger in size, it will be shrinked before restore.

Dependencies

None

Testing

Unit tests
Tempest tests

Documentation Impact

Docstrings
Devref
User guide
Admin guide
Release notes

References

[1]: https://specs.openstack.org/openstack/cinder-specs/specs/kilo/nfs-backup.html

Share migration Ocata improvements

Mon, 05 Dec 2016 00:00:00

https://blueprints.launchpad.net/manila/+spec/ocata-migration-improvements

Proposal for Share Migration improvements and changes, based on Barcelona Summit feedback.

Problem description

At Barcelona Summit we gathered feedback on Share Migration Experimental API feature [1]. There were a few improvements suggested:

For driver-assisted migration, drivers may be able to migrate a share and its snapshots, but the API layer is blocking the request when there are snapshots. This should only be the case when “force-host-assisted-migration” parameter is used.
There has been a lot of debate on whether the default value of “nondisruptive” parameter should be True or False. Currently it is also inconsistent with all other driver-assisted related parameters, defaulting to False while all the other driver-assisted parameters default to True.
The driver-assisted migration API parameters could be greyed out in the manila-ui “Start Migration” form so users better understand that “Force Host-assisted migration” negates the effects of the others.
Currently the API does not immediately return an error on the obvious incorrect combination of “force-host-assisted-migration” and other driver-assisted parameters set to True. This will be useful for when the manila-ui is not used.
The driver-assisted migration API parameters’ descriptions in manila-ui and python-manilaclient could be improved to state that these parameters are “Enforced”, as opposed to “not preserving metadata” when these properties are not selected.
Currently the Share Migration API prevents migrations to the same storage pool. This prevents the cases of migrating shares between share servers or simply retyping. In such cases, it may be desired for the share to remain in the same storage pool, while being able to change the share’s type or share network (and consequently the share server in the latter).

Use Cases

The implementation of the changes suggested addresses existing problems and enables use cases as seen below:

Migrating shares along with snapshots
Better user experience when choosing the API parameters
Load balancing among share servers
Retyping a share

Proposed change

This spec proposes:

Removing the snapshot restriction from the API layer when “force-host-assisted-migration” is False, changing it to work just like the other parameters’ validation (writable, preserve_metadata, etc), thus adding an additional entry for migration_check_compatibility driver interface. Add the parameter to the API layer, named “preserve_snapshots”. Also, this parameter, as the other ones, is not applicable for the host-assisted migration, preventing it from running if enabled.
Changing all REST API parameters to be mandatory, except for “force-host-assisted-migration”. The purpose is to have better REST API support in the future, defaulting non-existing new REST API parameters to False when older microversions are used, and to avoid data loss in cases where the administrator does not specify a parameter by accident. For more information, see reference [2].
Return an error when “force-host-assisted-migration” API parameter is set along with other driver-assisted migration API parameters.
Improve the driver-assisted API parameters’ descriptions in manila-ui and python-manilaclient to include “Enforce …” in the name. Example: “Enforce Writable”. Please note that this will not change the parameter name.
Changing the validation that rejects the API request when the parameters are the same as the source share. The correct behavior should be to return HTTP 200 “SUCCESS” if the parameters requested match the share’s properties, already having the same “share network”, “share type” and “pool”.
Migration should not be allowed to start when “access_rules_status” is in “ERROR” state. This is cause any migration to fail and should be blocked at the API layer. This spec proposes adding a validation for this.
Export locations should not be shown for the destination share instance during a migration, due to the fact that this instance cannot be mounted (because new access rules cannot be added during a migration) thus confusing the user that can see two export locations while being unable to determine which one is mountable.
With the addition of optional extra_specs saved in the Share model such as mount_snapshot_support, revert_to_snapshot_support and create_share_from_snapshot_support, it created the need to update these properties after a migration to match the destination instance share_type if it has been changed.

Alternatives

An alternative solution has been discussed with regards to proposal #2 above, where all options would be changed to False. This will imply that the default migration-start command will be able to run the host-assisted migration if the driver-assisted approach is unsupported or fails. We decided not to take this approach because we wanted above all else to avoid surprises when performing migration. We also considered defaulting all the driver-assisted parameters to True, to avoid the surprise situation, but that would compromise the sanity of the REST API with regards to future updates. We addressed these concerns by insisting that clients always send a value of True or False for each option. For more details see [2].

Data model impact

None

REST API impact

The microversion will be bumped to include the changes proposed. Support to previous microversion of migration-start will be dropped as we will only support the one with mandatory parameters.

The changes proposed will update the migration_start API as follows:

(POST, 200, 202, 400, 409) migration-start: migrates share URL: /shares/<share_id>/action Body:

{
  'migration_start': {
    'force_host_assisted_migration': false,
    'preserve_metadata': true,
    'writable': true,
    'nondisruptive': true,
    'preserve_snapshots': true,
    'host': 'ubuntu@generic2#GENERIC2',
    'new_share_type_id': 'foo_share_type_id',
    'new_share_network_id': 'bar_share_network_id'
  }
}

This API will not return an error immediately anymore when a share has snapshots, unless the “Force host-assisted migration” API parameter option is set.
Not specifying any value for “nondisruptive”, “preserve_snapshots”, “writable” or “preserve_metadata” will now return error 400.
The same pool API restriction will be changed to return 200 when the combination of “destination host”, “share network” and “share type” is the same as the source share.
Specifying “force_host_assisted_migration” as True, along with any of “writable”, “preserve_metadata”, “preserve_snapshots” or “nondisruptive” being True, will also result in error 400, as it would be an incompatible combination.
Attempting a migration while “access_rules_status” is in “ERROR” will return error 400.
Export locations for the destination migration instance are no longer shown until migration is complete.

Security impact

None

Notifications impact

None

Other end user impact

This proposal will require updates to python-manilaclient. See example:

manila migration-start <share> <host> --nondisruptive <True|False>
--writable <True|False> --preserve-metadata <True|False>
--preserve-snapshots <True|False> [--new-share-network <share_net>]
[--new-share-type <share_type>]
[--force-host-assisted-migration <True|False>]

manila migration-start share_1 ubuntu@generic1#GENERIC1 --writable True
--nondisruptive False --preserve-metadata True --preserve-snapshots False

As for manila-ui, there will be a new checkbox “Preserve Snapshots”.

Performance impact

None

Other deployer impact

None

Developer impact

None

Driver impact

Driver maintainers will be prompted to update their driver-assisted migration “migration_check_compatibility” implementation according to the new API parameter ‘preserve_snapshots’ introduced. In this change, it will be added to the existing implementations as “False”.

All existing migration driver interfaces will be updated to include snapshot-related parameters. See the new updated interfaces below:

def migration_start(
        self, context, source_share, destination_share,
        source_snapshots, snapshot_mappings, share_server=None,
        destination_share_server=None):
    """Starts migration of a given share to another host.

    .. note::
       Is called in source share's backend to start migration.

    Driver should implement this method if willing to perform migration
    in a driver-assisted way, useful for when source share's backend driver
    is compatible with destination backend driver. This method should
    start the migration procedure in the backend and end. Following steps
    should be done in 'migration_continue'.

    :param context: The 'context.RequestContext' object for the request.
    :param source_share: Reference to the original share model.
    :param destination_share: Reference to the share model to be used by
        migrated share.
    :param source_snapshots: List of snapshots owned by the source share.
    :param snapshot_mappings: Mapping of source snapshot IDs to
        destination snapshot models.
    :param share_server: Share server model or None.
    :param destination_share_server: Destination Share server model or
        None.
    """
    raise NotImplementedError()

def migration_continue(
        self, context, source_share, destination_share, source_snapshots,
        snapshot_mappings, share_server=None,
        destination_share_server=None):
    """Continues migration of a given share to another host.

    .. note::
        Is called in source share's backend to continue migration.

    Driver should implement this method to continue monitor the migration
    progress in storage and perform following steps until 1st phase is
    completed.

    :param context: The 'context.RequestContext' object for the request.
    :param source_share: Reference to the original share model.
    :param destination_share: Reference to the share model to be used by
        migrated share.
    :param source_snapshots: List of snapshots owned by the source share.
    :param snapshot_mappings: Mapping of source snapshot IDs to
        destination snapshot models.
    :param share_server: Share server model or None.
    :param destination_share_server: Destination Share server model or
        None.
    :return: Boolean value to indicate if 1st phase is finished.
    """
    raise NotImplementedError()

def migration_complete(
        self, context, source_share, destination_share, source_snapshots,
        snapshot_mappings, share_server=None,
        destination_share_server=None):
    """Completes migration of a given share to another host.

    .. note::
        Is called in source share's backend to complete migration.

    If driver is implementing 2-phase migration, this method should
    perform the disruptive tasks related to the 2nd phase of migration,
    thus completing it. Driver should also delete all original share data
    from source backend.

    :param context: The 'context.RequestContext' object for the request.
    :param source_share: Reference to the original share model.
    :param destination_share: Reference to the share model to be used by
        migrated share.
    :param source_snapshots: List of snapshots owned by the source share.
    :param snapshot_mappings: Mapping of source snapshot IDs to
        destination snapshot models.
    :param share_server: Share server model or None.
    :param destination_share_server: Destination Share server model or
        None.
    :return: If the migration changes the export locations or snapshot
        provider locations, this method should return a dictionary with
        the relevant info. In such case, a dictionary containing a list of
        export locations and a list of model updates for each snapshot
        indexed by their IDs.

        Example::

            {
                'export_locations':
                [
                    {
                    'path': '1.2.3.4:/foo',
                    'metadata': {},
                    'is_admin_only': False
                    },
                    {
                    'path': '5.6.7.8:/foo',
                    'metadata': {},
                    'is_admin_only': True
                    },
                ],
                'snapshot_updates':
                {
                    'bc4e3b28-0832-4168-b688-67fdc3e9d408':
                    {
                    'provider_location': '/snapshots/foo/bar_1'
                    },
                    '2e62b7ea-4e30-445f-bc05-fd523ca62941':
                    {
                    'provider_location': '/snapshots/foo/bar_2'
                    },
                },
            }

    """
    raise NotImplementedError()

def migration_cancel(
        self, context, source_share, destination_share, source_snapshots,
        snapshot_mappings, share_server=None,
        destination_share_server=None):
    """Cancels migration of a given share to another host.

    .. note::
       Is called in source share's backend to cancel migration.

    If possible, driver can implement a way to cancel an in-progress
    migration.

    :param context: The 'context.RequestContext' object for the request.
    :param source_share: Reference to the original share model.
    :param destination_share: Reference to the share model to be used by
        migrated share.
    :param source_snapshots: List of snapshots owned by the source share.
    :param snapshot_mappings: Mapping of source snapshot IDs to
        destination snapshot models.
    :param share_server: Share server model or None.
    :param destination_share_server: Destination Share server model or
        None.
    """
    raise NotImplementedError()

def migration_get_progress(
        self, context, source_share, destination_share, source_snapshots,
        snapshot_mappings, share_server=None,
        destination_share_server=None):
    """Obtains progress of migration of a given share to another host.

    .. note::
        Is called in source share's backend to obtain migration progress.

    If possible, driver can implement a way to return migration progress
    information.
    :param context: The 'context.RequestContext' object for the request.
    :param source_share: Reference to the original share model.
    :param destination_share: Reference to the share model to be used by
        migrated share.
    :param source_snapshots: List of snapshots owned by the source share.
    :param snapshot_mappings: Mapping of source snapshot IDs to
        destination snapshot models.
    :param share_server: Share server model or None.
    :param destination_share_server: Destination Share server model or
        None.
    :return: A dictionary with at least 'total_progress' field containing
        the percentage value.
    """
    raise NotImplementedError()

As can be noted above, the migration_complete driver interfaces had its return value changed to return a dictionary structure containing the export locations and a dictionary of snapshot updates, containing model updates for each snapshot, in order to update the provider location in manila’s database.

Implementation

When starting a driver-assisted migration, it will be checked if drivers can support “preserve_snapshots” regardless of the API option specified, due to the fact that the migrating share may have existing snapshots. In case the driver does not support “preserve_snapshots”, an error message will be raised, stating that driver-assisted migration cannot proceed while the share has snapshots.

If the driver does support “preserve_snapshots”, it will be checked if all existing snapshots have ‘available’ status. If so, destination snapshot instances respective to each source snapshot instance will be created in the database. Finally, a list of source snapshot instances and a mapping dictionary, that comprises of destination snapshot instances indexed by source snapshot instance IDs, will be passed to drivers in the updated driver interfaces.

This mapping and list of snapshots will be easily retrieved from the database at later stages such as when invoking migration_continue and migration_complete. After migration is complete, the snapshot instances will be updated according to the model updates returned by the driver, with fields such as “provider_location” and “export_locations”.

As for host-assisted migration, the validation of existing snapshots present in in the API layer, has been copied to before starting a host-assisted migration, as it will prevent the host-assisted migration from running.

Finally, the optional extra_specs of the share model are updated according to the destination share type.

Assignee(s)

Primary assignee:: ganso

Work Items

Implement main patch for manila that includes:
- Updated Tempest tests
- Updated Unit tests
- API changes described in this proposal
- Share migration host-assisted and driver-assisted changes required
Implement additions and changes in python-manilaclient with:
- Unit tests
- Functional tests
Implement additions and changes in manila-ui with:
- Unit tests
Update documentation for this feature (see Documentation Impact section)

Dependencies

No previous dependencies on other Ocata patches so far

Testing

Unit tests in manila, manila-ui and python-manilaclient
Tempest API tests in manila and python-manilaclient

Documentation Impact

Docstrings
Release notes
Developer reference
Admin reference
API reference

References

[1] https://etherpad.openstack.org/p/ocata-manila-contributor-meetup

[2] http://lists.openstack.org/pipermail/openstack-dev/2016-November/107186.html

Fix and improve access rules and interactions with share drivers

Mon, 31 Oct 2016 00:00:00

https://blueprints.launchpad.net/manila/+spec/fix-and-improve-access-rules

One of the major benefits that shared file systems offer is access control. Manila’s implementation of access control offers an easy-to-use interface to allow and revoke client access to shared file systems. Access is controlled on the basis of the respective type of access that makes sense to individual storage protocols. For example, access to NFS Shares is controlled with IP based rules, and to CEPH FS shares is controlled with CEPHX access rules, etc. For many releases now, we have had to fix many issues with access control APIs biggest of which were incorrect API behavior [1] and race conditions [2].

Problem description

Two major design changes in manila affected the access rules implementation: Introduction of Share Instances [3] and bulk access rule updates on share back ends [4]. Prior to these changes the behavior was as follows:

Tenant could request (or deny) access for a given share to a client
Tenant could send x number of such requests in succession
Tenant could list access rules on a given share and each rule had a distinct state that would inform the tenant if the rule was applied or denied successfully.

This API behavior was easy to write scripts around and even perform bulk access rule operations on a given share. Since we had a per share per rule state and every access rule was individually applied or denied by a share driver, there was better error reporting and lesser scope for a race condition to occur.

The current behavior is as follows:

Shares have instances. There is no state for an access rule for a given share instance, instead, the share instance has an access_rules_status attribute that is a combined state across all the access rules of a given share instance.
Tenants are not aware of share instances; access control is still performed at the share level. They can add or remove rules for a given share via the access control API. We do not support modifying multiple access rules at once, so API calls to allow_access or deny_access only ever impact one access rule at a time.
Tenants can send x number of such requests in succession, however, if new requests come in when rules are still being processed for the given share instance, the access_rules_status on the share instance will transition from ‘active’ to ‘updating’ and from ‘updating’ to ‘updating_multiple’.
If there was a request that the back end or the share driver could not honor, the exception is raised and the access_rules_status would be set to ‘error’. No further rule additions would be allowed, until all “bad” rules are removed.
Identifying “bad rules” would require a careful amount of supervision on the tenant’s part by noting each state transition; otherwise, the server would pretend that all rules are in ‘error’ state - owing to the fact that there is no tracking of the per share instance, per access rule state in the manila database.
When a bunch of access requests are made on the same share, after a point, the share manager ignores the requests (i.e, share instance’s access_rules_status transitions from ‘active’ to ‘updating’ on the first new rule, and from ‘updating’ to ‘updating_multiple’ when the next rule comes in and the status is still ‘updating’. If any further requests come in when the status is ‘updating_multiple’, they are ignored.) [5]
Further, in an attempt to avoid race conditions in the share manager, the whole interaction to the share back end is synchronized with a lock. If the share driver or the back end takes a long time, the lock would prevent any further operations.
If the share manager service crashes or is bounced when a share instance’s access_rules_status is ‘updating’ or ‘updating_multiple’, users are stuck. No new rules can be added and because of poor error reporting. To recover from this state, users would have to perform another access rules API request (deny access to an existing client) to trigger an update of the access rules.
When share instances were introduced as the underlying database representation for share replicas and migration instances, the design was conscious of the fact that “instances” would be visible to administrators and not end users (replicas are visible to tenants, however, they needn’t know that replicas are share instances). However, error messages [6] seem to break the abstraction.
When a share has multiple instances, the API sends an RPC request to the share manager host corresponding to each instance. Currently, the logic is as follows:
- user requests access to a share for a client
- manila-api performs basic validation on the state of the share
- It then commits the rule to the database
- It loops over the instances of the share, validating each one and shooting off the RPC request.
- manila-api responds with a 202 Accepted once all the RPC requests have been sent out.
Here, if the validation fails for the second or subsequent share instance, the RPC call has already been made for the first share instance, resulting in the user receiving a 400 Bad Request and the rule transitioning to ‘error’, but the share might be accessible via export paths of the first instance.

Use Cases

For a user of shared file systems, access control is arguably the most basic requirement. The proposed change would allow preserving the good portions and enhancing the user experience for access control in manila:

Users can add any number of access rules to a given share in quick succession as far as the API is concerned.
Users can identify which rule/s failed to apply.
Users may continue to apply or deny rules while some rules are in ‘error’ state.
Users may be able to deny a rule that has not yet been applied.

Elimination of the currently existing race conditions will benefit both users and cloud administrators.

Proposed change

Per share instance rule, per access rule state will be re-introduced.
Four new access rules states will be introduced: ‘queued_to_apply’ (is currently ‘new’), ‘queued_to_deny’, ‘applying’ and ‘denying’, the state transition for this:
- Adding a new rule:
  
  When the request is received by the API
  
  share_instance.access_rules_status:
  - ‘active’ —-> ‘out_of_sync’
  - ‘error’ —-> ‘error’ (state will be preserved)
  - ‘out_of_sync’ —-> ‘out_of_sync’ (state will be preserved)
  share_instance_access_map.state
  - ‘queued_to_apply’ —-> ‘queued_to_apply’ (rule starts out in this state)
  In the share manager, before the request is sent to the share driver
  
  share_instance.access_rules_status:
  - ‘error’ / ‘out_of_sync’ —-> (state will be preserved)
  share_instance_access_map.state
  - ‘queued_to_apply’ —-> ‘applying’
  When the share driver returns with the response
  
  share_instance.access_rules_status:
  - ‘error’ —-> ‘error’ (state will be preserved)
  - ‘out_of_sync’ —-> ‘active’ (if no other ‘queued_to_apply’ or ‘queued_to_deny’ rules)
  share_instance_access_map.state
  - ‘applying’ —-> ‘active’ or ‘error’
- Deleting an existing rule:
  
  When the request is received by the API
  
  share_instance.access_rules_status:
  - ‘active’ —-> ‘out_of_sync’
  - ‘error’ —-> ‘error’ (state will be preserved)
  - ‘out_of_sync’ —-> ‘out_of_sync’ (state will be preserved)
  share_instance_access_map.state
  - ‘active’ —-> ‘queued_to_deny’
  - ‘applying’ —-> ‘queued_to_deny’
  - ‘error’ —-> ‘queued_to_deny’
  - ‘queued_to_apply’ —-> ‘queued_to_deny’
  In the share manager, before the request is sent to the share driver
  - ‘queued_to_deny’ —-> ‘denying’
  When the share driver returns with the response
  
  share_instance.access_rules_status:
  - ‘error’ —-> ‘error’ (state will be preserved)
  - ‘out_of_sync’ —-> ‘active’ (if no other ‘queued_to_apply’ or ‘queued_to_deny’ rules)
  share_instance_access_map.state
  - ‘denying’ —-> ‘deleted’ or ‘error’

Note

When a share has multiple share instances, all instances of the share are expected to have the same access rules.

In case of share migration, while existing access rules from the source share instance are eventually applied on the destination share instance, it begins out with no access rules.

Also, when migrating a share, certain back ends may not be able to allow write operations on the source share during the migration, for a variety of reasons. Host based migration cannot handle new data being written into the share when existing data is being copied over. To ensure that all write operations are fenced off, manila casts existing rules on the source share to read-only prior to these kinds of migrations.

In case of share replication, the expectation that clients have access to all replicas still holds with a clarification of semantics: For ‘dr’ replicas, we track all the access rules for the replica in the database, however, there are no export locations for the replica, hence, though the database contains these rules as being ‘active’ for the given replica, the replica is not accessible.

In case of ‘readable’ replicas, any read/write rules are cast by the drivers to read-only for the secondary replicas; i.e, a rule’s database representation does not change from ‘r/w’, the ‘r/o’ semantics on the replica are inherently ensured by the share drivers themselves [7] [8]. This expectation off the drivers is an exception that exists only within this feature, i.e, the logic to ensure read-only semantics currently needs to live in each driver, though it is something that the share manager could handle uniformly, as we have in case of share migration.

See further discussion within Read-only access semantics.

Important

If all share instances are expected to have the same access rules, why would we still maintain per access rule per share instance states?

While this may be clear with the purpose and design of share instances, let’s clarify this once more for posterity: Each share instance is associated with a manila host. They can each be created at different times and acted upon independently, i.e, consider creating a share, allowing access and at a later time, creating a replica for it, or migrating it. Therefore, the state of access control should be tracked per rule, per share instance. This is the benefit of having an access_rules_status as an attribute of each share instance as well. The access_rules_status of an active replica can be ‘active’, while for a secondary replica where rules have not yet been synced, the attribute can be “out_of_sync”. When the user lists the access rules however, the per share instance access rule states will be aggregated. See Aggregate status fields for more details.

Rule status updates will be coordinated between the API service and the share manager service. The only state transition for an existing rule in the API service, as noted above, is performed while denying rules. The API service will acquire the same locks as the share manager service in order to make this transition.
Manila-api and manila-share both have access to the database; the access rules payload is not necessary since these services can individually read the rules from the database and perform necessary state transitions. Thus, the RPCs: manila.share.rpcapi.ShareAPI.allow_access and manila.share.rpcapi.ShareAPI.deny_access will be collapsed into manila.share.rpcapi.ShareAPI.update_access.
When the share manager receives the RPC request to update_access, it will react as follows:
- Acquire a lock, look into the database for any rules in ‘applying’ or ‘denying’ states for the given share instance. If there are any rules in these states, the driver is currently processing rule changes, any ‘queued_to_apply’ or ‘queued_to_deny’ rules are batched to be applied or denied when the driver is done with its current task. Steps below are not executed right away. If there are no rules in ‘applying’ or ‘denying’ states, set the state of any ‘queued_to_apply’ rules to ‘applying’, and ‘queued_to_deny’ rules to ‘denying’. Release the lock.

Note

When the driver is processing an access rules update, any ‘queued_to_apply’ or ‘denying’ rules are left alone until the driver returns from its current task. The update_access interface is designed for a bulk update. There is no order for processing rules. If access rules have to be processed in any particular order, it would be up to the share driver to do so.

Call the driver interface update_access passing existing rules and the “changed” rules (‘applying’ or ‘denying’ rules).
Accept state attribute to be set by driver, allowing for rules in transitional states to be updated by the driver. If the driver does not return a state for each rule in transitional state, transition ‘applying’ rules to ‘active’ and soft delete rules in ‘denying’ state. Perform these actions by acquiring a lock to read the current state, and releasing it at the end of the transaction.
Acquire a lock and read any ‘queued_to_apply’ rules that may have shown up, if any, repeat the last three steps, else continue to the next step.
Acquire a lock and transition the share instance’s access_rules_status from out_of_sync to active. error state will be preserved if any access rules are in error state for the given share instance.

The database API for retrieving a specific rule or all rules for a given share, or a given share instance will be refactored.
The coarse lock around the update_access driver interface (or the fall back interfaces) will be removed. A reader-writer lock around database calls for these access rules will be introduced as pointed out above. This is because both manila-share and manila-api services read from and write to the database. For correct behavior, deployers should prefer a distributed lock [9] or a file lock living on a shared file system.
On restarting the share-manager service, the ‘recovery’ step for access rules will be updated. All rules in ‘applying’ state will be reverted to ‘queued_to_apply’ before requesting the driver to sync access rules for a given share instance.
Read-only access semantics:

For share migration and share replication, access rules of a given instance, source instance for share migration or secondary replicas of a share, may require to be set to r/o (read-only).

Currently, there is code to ensure the read-only semantics on the fly for the source share instance of a migration. This spec proposes adding a field to the database representation of share instances.

The field will be called cast_rules_to_readonly. It will be set and unset as necessary in migration and replication workflows where applicable.

When we have the field, performing a check in the update_access workflow, before invoking the driver would be as simple as:
```
if share_instance['cast_rules_to_readonly']:
    # The share manager will know to cast all rules to 'r/o'
    # before calling the share driver as long as this condition holds.
```
The cast_rules_to_readonly field will be False by default for any share instance.

The field will be set to True when creating a new replica on a share supporting readable style of replication. It will be unset on the replica when it gets promoted to being the primary (active replica) and set to True on the replica that gets demoted to being a secondary.

The cast_rules_to_readonly field will be set to True at the beginning of each migration where necessary and unset always if any migration is canceled.

Note that migration is not supported for replicated shares, replicas have to be deleted before migrating the share.

The cast_rules_to_readonly field will not be exposed to tenants. It will be exposed to administrators in all APIs that return the share instance in “detail”. It will not be present in the share model, not even as a @property because it solely belongs to a particular share instance.

Setting and unsetting of the cast_rules_to_readonly will be synchronized by a lock.

Supporting read-only rules is a minimum requirement within manila. If drivers do not support them, this and future evolution of manila features will expose differences and break the abstraction. There have been plans to act on such discrepancies, in terms of policy changes. Any action on these drivers is out of scope of this spec.

Aggregate status fields

To preserve the abstraction at the API as far as users are concerned, we need to aggregate status fields for both the share instance access_rules_status as well an per instance access rule state attributes:

share_access_map.state
non persistent, but reflects aggregate state from share_instance_access_map.state

Aggregation: Read all share_instance_access_map.state, prefer in the following order: ‘error’, ‘queued_to_apply’, ‘queued_to_deny’, ‘applying’, ‘denying’, ‘active’

share.access_rules_status:
non persistent, but reflects aggregate state from share_instance.access_rules_status

Aggregation: Read all share_instance.access_rules_status, prefer in the following order: ‘error’, ‘out_of_sync’, ‘active’

Alternatives

Not having per share instance per access rule state: Live outside the reality that processes fail and things go wrong once in a while and even genuine user errors can upset well written driver or storage system code (There are drivers that error out entire requests on not understanding a rule type). If we don’t separate the status field now, we would have to support annoyed users and document how to get out of messy situations.
Not having transitional states: Allow coarse transitions as occurring today and live without the added benefit of concurrency control and poor user experience, i.e, allow all rules to go to ‘error’.
Keep the coarse lock in the share manager: Holding locks for long durations increases the chances of things going wrong when processes fail while holding locks and has a higher chance of running into deadlock situations [10].
Don’t refactor the API, allow the error messages to remain as they are: We need user documentation and awareness around what share instances are and how they work in manila. This breaks the abstraction and design, but maybe is not the worst thing in the complicated OpenStack ecosystem.
Don’t combine RPCs - reasonable, but no real benefit of separating them.
Don’t refactor the database API code - reasonable, it is a lot of code churn, but again, the number of database calls increases and we would be compromising on performance because we don’t want to address this problem.
Don’t require a shared/distributed lock: The alternative would be to make state transitions only occur in one service (either manila-api or manila-share). However, the proposed design is superior in terms of correctness. We would need to compromise on the behavior changes: i.e, disallow denying a rule until it is ‘active’; or the manila-data service must make requests via the manila-share or the manila-api to create new rules. The resulting code may be simpler, but the loss of functionality would be a backwards-incompatible API change. (This is perfect reasoning for a manila-conductor service, a la nova-conductor.)

On the flip side, if a deployer does not deploy a DLM lock as suggested, and still distributes manila-api and manila-share services, the chances of running into race conditions is higher when the proposed implementation merges. However, this may not occur in some clouds where high volume of access requests on the same share are rarely performed, if ever. Note that any scripting of the access API may uncover these race conditions. Disclaimer: The developer driving this effort does not condone wrong deployment practices.
Instead of the cast_rules_to_readonly field being added to the share instances, we can evolve a set of conditions within the code to ensure the read-only semantics that we desire for share instances.
- For a host assisted migration, we can refer to the configuration parameter that specifies whether the host can support read-only access to a given share, and in the update_access workflow, we can change all access rules to read-only rules before invoking the driver.
- Similarly, we can ascertain any readable replicas, within the update_access workflow and cast access rules to read-only before invoking the driver.
- For writable and nondisruptive migrations, we can carefully ensure we never invoke update-access while migrating.
While this seems straight-forward, it seems much complicated to maintain in the light of future feature functionality. It also overloads the use of multiple status fields. This creates a bottleneck, and a potential race condition when these status fields are updated by any process within manila. Also, we need to ensure a consistent behavior on restarting manila services. With this motivation, a field in the database seems necessary.
Require that driver update the state attribute of rules. This is a reasonable ask. Allowing the drivers to update which rules were not acted upon successfully is more beneficial than the share manager setting all the transitional rule states to ‘error’ during a bulk update operation. We need to have code in the share manager that performs rule updates conditional to whether or not the driver has decided to return updates for each access rule. This is a similar pattern as seen with other existing driver entry-points. Therefore, while we will encourage drivers to revisit the update_access interface and return updates to the state field. It is not feasible to make these changes en-masse as part of this effort, not even for the first party drivers.

Data model impact

The manila.db.sqlalchemy.models.ShareInstanceAccessMapping model will have a new field called state.

The database upgrade step will add this column by populating it with the value of existing manila.db.sqlalchemy.models.ShareInstance.access_rules_status column.

Values of the manila.db.sqlalchemy.models.ShareInstance.access_rules_status column will be re-mapped to remove ‘updating’ and ‘updating_multiple’ as valid access_rules_status values.

The manila.db.sqlalchemy.models.ShareInstance model will have a new field called cast_rules_to_readonly.

The database upgrade step will add this column with a default value of True for all share instances that have their replication_type attribute set to readable and whose replica_state is not active (secondary replicas in a readable type of replication). For all other share instances, the value will be set to False.

In the ORM layer, manila.db.sqlalchemy.models.ShareInstanceAccessMapping model will now contain a back-ref to the access rules model (manila.db.sqlalchemy.models.ShareAccessMapping). That way, the database APIs associated with reading a given share instance access mapping row will have access to the related access rule data (such as access_type and access_level).

The database downgrade step will drop the state column from manila.db.sqlalchemy.models.ShareInstanceAccessMapping. It will not alter the manila.db.sqlalchemy.models.ShareInstance.access_rules_status column.

The database downgrade step will also drop the cast_rules_to_readonly column from manila.db.sqlalchemy.models.ShareInstance.

As always, this downgrade is not recommended in a production cloud.

REST API impact

No new APIs will be added. While we will bump the micro-version to expose the ‘queued_to_deny’ state, transitional ‘applying’ and ‘denying’ states, some other changes will be made without bumping the micro-version, since the behavior is currently broken in the API and it is hard to warrant requiring backwards compatibility given the unpredictable/undesirable behavior:

Adding an access rule

POST /v2/{tenant-id}/shares/{share-id}/action BODY:

{
    'allow_access': {
            "access_level": "rw",
            "access_type": "ip",
            "access_to": "0.0.0.0/0"
    }
}

Policy check will be completed first before any other validation
If any share instance is invalid, the API will return without creating the rule in the database.
Adding access rules while the share is being migrated will be possible, as long as all share instances have a valid host. This behavior will be exposed only from a new micro-version of the allow-access API.

Denying an access rule

POST /v2/{tenant-id}/shares/{share-id}/action BODY:

{
    'deny_access': {
           "access_id": "a25b2df3-90bd-4add-afa6-5f0dbbd50452"
    }
}

Policy check will be completed first before any other validation
If any share instance is invalid, the API will return without denying the rule for any other valid instance.
Access rules cannot be denied when the share instance status is not ‘available’.

Listing access rules

POST /v2/{tenant-id}/shares/{share-id}/action BODY:
{
    'access_list': null
}

Policy check will be completed first before any other validation
Transitional statuses introduced will be mapped to pre-existing states for API requests with a micro-version lesser than the micro-version where they were introduced. Mapping ‘applying’ and ‘queued_to_apply’ is easy, they will be set to ‘new’; mapping ‘queued_to_deny’ and ‘denying’ is tough because the state could have been anything prior to transitioning to ‘denying’. So, we will read the share’s aggregate access_rules_status, if it is ‘error’, we will map the state of this access rule to ‘error’, else if the access_rules_status is ‘out_of_sync’, the rule’s state will be mapped as ‘new’, and if the access_rules_status is ‘active’, the rule’s state will be mapped to ‘active’. Note that this was the behavior that we are trying to change by re-introducing the per share instance per access rule state.

Share Instance APIs

The cast_rules_to_readonly field will be exposed in the “detail” views of share instances (admin-only) as a micro-versioned change.

Security impact

Bulk access rule updates will no longer be ignored by manila. We will also support denying access in case access was accidentally granted. Since tracking of state changes would be improved, we expect a positive security impact.

Notifications impact

None until [11] merges and is fully supported in manila. The work to add user messages will be proposed with a new blueprint and not part of this work.

Other end user impact

End users will be prompted to prefer the new micro-version in case of writing applications against a newer manila release, so as to gain full benefit of the more predictable state transitions. When manila is upgraded in their clouds, they will already benefit from faster API failures as opposed to the previous versions, no matter what API version they use.

Performance impact

Changes to the ORM and the database API will reduce redundant calls to the database, and is expected to have a positive impact on the performance. Instead of reacting to every RPC request and triggering update_access calls on the driver, the proposed implementation defers un-applied rule requests if a driver is processing rule requests. This could lead to batching of requests to take effect at once, thereby reducing the number of calls to the storage back end, hopefully improving performance.

Other deployer impact

Until tooz [9] support is added in manila, we expect deployers to support manila with file locks created with oslo_concurrency. In multi-node deployments, these file locks must be accessible to all nodes where all manila services are run. Typically, this is achieved with storing these locks on a shared file system.

Once tooz support is added, synchronization introduced in this patch will automatically benefit from the locking abstraction introduced by tooz; and deployers may choose to configure a distributed lock management system underneath manila/tooz.

Since no state is saved in the services, this proposed change does not introduce any regression to affect active-active deployment choices.

Developer impact

None

Driver impact

As always, when denying a rule, if the rule does not exist on the back end, the drivers must not raise an exception, they may log that the access rule never existed, and return None allowing for the rule to be deleted from the database.

Raising exceptions will set all rules in transitional states, ‘applying’ or ‘denying’ to ‘error’. So, drivers must carefully consider exception handling.

As part of this work, we will accept ‘error’ rule states from the driver in the response, so, drivers can tell the share manager which exact rules failed to be applied. This needs to be added to each driver considering how each back end can handle error reporting.

Implementation

Assignee(s)

Primary assignee:: gouthamr
Other contributors:: yogesh

rooneym

Work Items

Code - For ease of implementation and review, this work will be done in multiple related change sets, each will partially implement the blueprint until all of the proposed items are implemented.
Functional tests in server and client

Dependencies

None

Testing

Negative/API tests will be added for the changes in the API including scheduling invalid shares or invalid replicas and testing the access rules interaction at the API.
Existing access rules tests will be modified to validate the access_rules_status but wait on the access rule’s state attribute in both manila and python-manilaclient.
The scenario tests spec [12] proposes to add tests around the read-only semantics. This simplification should pass the tests proposed.

Documentation Impact

The following OpenStack documentation will be updated to reflect this change:

OpenStack User Guide: will document the changes in state transitions.
OpenStack API Reference: All API changes will be documented
Manila Developer Reference: A state transition diagram will be added for access rules’ state and share instances’ access_rules_status.

References

[1]: https://bugs.launchpad.net/manila/+bug/1550258

[2]: https://bugs.launchpad.net/manila/+bug/1626249

[3]: https://blueprints.launchpad.net/manila/+spec/share-instances

[4]: https://blueprints.launchpad.net/manila/+spec/new-share-access-driver-interface

[5]: https://bugs.launchpad.net/manila/+bug/1626249

[6]: https://github.com/openstack/manila/blob/4c6ce2f/manila/share/api.py#L1417

[7]: http://docs.openstack.org/developer/manila/devref/share_replication.html#access-rules

[8]: http://docs.openstack.org/admin-guide/shared-file-systems-share-replication.html#access-rules

[9]: https://review.openstack.org/#/c/318336/

[10]: https://en.wikipedia.org/wiki/Dining_philosophers_problem

[11]: https://blueprints.launchpad.net/manila/+spec/user-messages

[12]: https://review.openstack.org/#/c/374731/

Create share from snapshot extra spec

Thu, 27 Oct 2016 00:00:00

https://blueprints.launchpad.net/manila/+spec/add-create-share-from-snapshot-extra-spec

This spec describes why and how we must separate the overloaded meaning of ‘snapshot_support’ into two independent standard extra specs. Furthermore, to prevent the proliferation of required extra specs as well as to make the creation of new share types as simple and flexible as possible, we can make snapshot_support and each of the new snapshot-related extra specs optional.

Problem description

For a long time, manila has had the ‘snapshot_support’ standard extra spec, which has been overloaded to mean two things:

a driver can take a snapshot of a share, and
a driver can create new shares from a snapshot

With additional snapshot semantics being proposed, including share revert and mountable snapshots, we expect that some drivers may support the new semantics without being able to create new shares from snapshots. It is therefore necessary to break the overload of ‘snapshot_support’.

We can add a new standard extra spec, create_share_from_snapshot_support, to indicate whether a backend can create new shares from share snapshots.

The snapshot_support extra spec is currently required, which means an admin cannot create a type where snapshot_support is a “don’t care”, such that shares of that type will not offer snapshot features even if they exist on storage controllers than can take snapshots. By making snapshot_support and the other new snapshot-related extra specs optional, we can support “don’t care” semantics on share types.

Use cases

A driver that can take snapshots of shares, but not create new shares from snapshots, will benefit from this change. For example, such a driver could report snapshot_support=True, create_share_from_snapshot_support=False, and revert_to_snapshot_support=True.

Proposed change

To accomplish this change, we will do the following:

Define the new extra spec, create_share_from_snapshot_support, at the next available microversion.
Add a database migration to add create_share_from_snapshot_support to each share type that contains snapshot_support. Because snapshot_support was overloaded prior to this change, we can simply copy the value of snapshot_support for each type to the new create_share_from_snapshot_support extra spec.
Add a database migration to add the new create_share_from_snapshot_support capability to the share record, so that manila has a record of each share’s relevant snapshot capability at the time the share was created.
Copy the value of create_share_from_snapshot_support to each share upon share creation. If create_share_from_snapshot_support is not present on the share type, set the corresponding value on the share to False.
Continue copying the value of snapshot_support to each share upon share creation. Because snapshot_support is now optional, if snapshot_support is not present on the share type, set the corresponding value on the share to False.
Ensure create_share_from_snapshot_support is True when creating shares from snapshots.
Ensure that previous microversions cannot delete the snapshot_support extra spec from share types.
Ensure that previous microversions continue setting a default value of True for snapshot_support as well as create_share_from_snapshot_support.

Alternatives

We could leave the overload in place, but this would lead to confusing and incorrect behavior for drivers that can create snapshots but not create new shares from them.

We could continue requiring snapshot_support and all new snapshot-related extra specs to be present on each share type, but that would prevent admins from creating share types with “don’t care” semantics.

Data model impact

Add create_share_from_snapshot_support field to share table, and copy value of snapshot_support to the new create_share_from_snapshot_support field for each share.
Add create_share_from_snapshot_support rows to the share_type_extra_specs table for each type that has snapshot_support, with the value copied from snapshot_support.

REST API impact

At the microversion for this change:

The snapshot_support extra spec becomes deletable, as are the new snapshot-related extra specs.
Default values for snapshot_support are no longer provided. If not specified on a share type, snapshot_support and create_share_from_snapshot_support are not returned from the share types API for admin contexts. However, users see both values with a nominal value of False so that they know how shares of that type will behave.
The value of create_share_from_snapshot_support is checked on a share prior to creating a share from a snapshot (else HTTPBadRequest is returned).

Driver impact

The value of create_share_from_snapshot_support is inferred for each driver based on the methods it contains, much as is already done for snapshot_support.

Security impact

None

Notifications impact

None

Other end user impact

Prior to this change, python-manilaclient provided default values for the required extra spec snapshot_support. As of the new microversion, it will not provide a default for snapshot_support because the extra spec is no longer required.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

clintonk (manila & python-manilaclient)

Other contributors:

gouthamr (tempest & db-related tests)

Work Items

Code is all in one patch and has already been made available.

Dependencies

None

Testing

New unit and tempest test coverage will be added for all of the added or changed functionality.

Documentation Impact

API Ref: Add content about the API.
User Guide: Add content about the share types.
Admin Guide: Add content regarding the common capabilities and the don’t care behavior.
Developer Ref: Add create_share_from_snapshot_support to common capabilities and the infamous driver matrix.

References

None

Add share revert-to-snapshot feature to manila

Thu, 27 Oct 2016 00:00:00

https://blueprints.launchpad.net/manila/+spec/manila-share-revert-to-snapshot

In manila, the only thing users can do with snapshots is to create new shares from them. Alternate snapshot semantics were discussed in Tokyo and Austin. At the Austin Summit, the community agreed one of the new snapshot semantics would be to revert a share (in place) by restoring the most recent snapshot of that share taken by manila.

Problem description

There are some drivers which can take snapshots but are unable to create new shares from those snapshots (or doing so is prohibitively expensive). And even if creating a share from a snapshot is possible, doing so consumes quota and in some cases real space on a backend storage system.

Use cases

Consider the case where a file system becomes corrupted or even unusable, perhaps due to a virus or accidental installation of libraries or other dependencies, or even a malicious “rm -rf” attack. In such cases, it would be valuable to be able to turn back the clock to a point before the corruption occurred, without having to do so using a new share or restoring files from a mounted snapshot.

Proposed change

As agreed by the manila community in Austin, manila can address this need by adding the ability to revert a share (in place) from a snapshot.

Theoretically, if multiple snapshots exist, a storage system could restore a share to any one of those snapshots. But reverting a share to a snapshot which is not the most recent one typically deletes the later snapshots, which is a form of data loss and may not be what the user intends. So this feature will limit manila to restoring the most recent share snapshot known to manila.

For this use case, reverting backwards in time starting with the most recent snapshot makes sense. Only the user can determine whether a share is usable or must be reverted, so getting back to a good state is a manual iterative process:

Observe corruption in active file system
Revert to most recent snapshot
While corruption is present in active file system:
  Delete most recent snapshot
  Revert to most recent snapshot

An early suggestion to keep the REST interface simple was to merely include the share ID and allow manila to determine the most recent snapshot, but that opens a potential concurrency bug. If two users call share-revert-to-snapshot and snapshot-create at the same time, the two calls race and one user could be surprised about which snapshot was used to revert the share. Similarly, if two users call share-revert-to-snapshot and snapshot-delete at the same time, the calls race and someone may be surprised when the share is reverted to an even earlier snapshot than expected (which is a form of data loss).

To keep the REST interface explicit, and to avoid the race conditions described above, the manila community decided the interface will accept the ID of the snapshot to be restored. Manila will verify that the snapshot is the most recent one via the created_at field on the snapshot object, returning an error if not. At no time will manila delete any snapshots during the revert operation, and it will not operate on any snapshot other than the one specified in the REST call. Once user messages are available in manila, it would be helpful to include a message indicating that the snapshot restoration was successful.

The share size might have changed after the snapshot to be restored was taken. The size of any share at the time it is snapshotted is stored in the DB in the snapshot record. This should be used to ensure a quota will not be exceeded by restoring the snapshot, and it may be used to update the share record as well as the user’s share size quota with the original size after the restoration has succeeded.

The CLI would look like:

usage: manila revert-to-snapshot <snapshot>

Revert a share to the specified snapshot.

Positional arguments:
  <snapshot>  Name or ID of the snapshot to restore.

Note that this feature is proposed in the short term for individual shares, but it could apply equally well to share groups. Since groups aren’t yet fully baked, revert-to-group-snapshot should come later.

Alternatives

Manila could simply rely on creating new shares from snapshots. But not all snapshot-capable drivers can do this, and it involves either copying data from the new share to the old one, or updating an application to point to the new share.

Manila could also add a backup feature to address use cases such as this one (complete restoration to an earlier point in time to resolve share-wide corruption, etc.). But backups typically involve an off-site data copy (either full or differential) and so are inherently slower than a snapshot restore operation performed in place by a storage controller.

Data model impact

No changes to the database schema are required.

During a snapshot restoration, there are two objects that are affected whose states must reflect the operation. The share is being “reverted”, while the snapshot is being “restored”. Other operations must not be allowed to either object during the restoration, so each will be updated with a transitional status:

Share —> ‘reverting’
Snapshot —> ‘restoring’

After a successful restoration, both objects’ states will again be set to ‘available’. If the restoration fails, the share will be set to ‘reverting_error’, and the snapshot will be set to ‘available’.

For replicated shares, all “active” replicas must be reverted immediately, and other replicas will be “out-of-sync” until they are again consistent with the “active” replicas. For the purpose of determining the most recent snapshot, the created_at field on the snapshot object will be used; it would be incorrect to look at the snapshot_instance objects because they may be created much later during replica creation.

REST API impact

A single API method is required for revert-to-snapshot. The REST API is modeled after other share actions, such as deny_access:

Method = POST
URL = /shares/{share_id}/action
Body = {'revert': {'snapshot_id': <snapshot_id>}}

Calling this method reverts a share to the specified snapshot. It is intended for both tenants and admins to use, and the policy.json file will be updated to reflect allowed use by all. If the snapshot is not the most recent one known to manila, or the state of either share or snapshot is not ‘available’, the API will return the HTTP error code 409 (Conflict). If the share isn’t found, 404. If the snapshot doesn’t exist, 400 (because it isn’t explicitly referenced in the URL).

Driver impact

There will be one new driver entry point to revert a share to a snapshot, and another to revert a replicated share to a snapshot. Drivers may explicitly advertise support for the revert feature using the ‘revert_to_snapshot_support’ pool attribute, but the share manager will be able to discern that automatically as it already does for ‘snapshot_support’ by looking for the presence of the entry point(s).

Security impact

None

Notifications impact

None

Other end user impact

This feature will be available in python-manilaclient, and it should be straightforward to implement it in manila-ui as well. In the latter case, the GUI should only present the action on the snapshot known to be the most recent one.

Performance impact

To correctly identify the latest snapshot known to manila, the existing snapshot-create workflow must be protected with locks to ensure no races occur that can cause the snapshot record timestamps to be out of order relative to the order in which the snapshots were taken on the storage controller. This could cause delays in cases where multiple snapshots of a share are taken in rapid succession, such as in automated tests.

The community decided to place a share in a ‘snapshotting’ state while taking a snapshot in order to prevent multiple simultaneous snapshot operations. Complete correctness of the revert-to-snapshot feature depends on this work being completed, although they could merge in any order.

Also, determining which snapshot is the latest requires a database query that sorts by a timestamp (the created_at field on the snapshot object). This would be slightly slower than a query that does not care about result ordering.

Other deployer impact

None

Developer impact

As with other snapshot semantics, including the ability to take a snapshot, a driver must advertise its ability to restore a snapshot. This will be done using the driver method discovery code that exists today to report the share revert capability to the scheduler. The new field shall be ‘revert_to_snapshot_support’. It will also be reported as an optional public extra spec on the share type to enable user-facing tools to selectively offer the feature on a per-snapshot basis, and the value will be copied from the share type to the share at the time of share creation. Shares created before this feature is released will not have the attribute set, so the revert-to-snapshot action will not be available on those even if the backend support is present. The default value of ‘revert_to_snapshot_support’ on a share object will be False if the corresponding key is not set on the share type.

Implementation

Assignee(s)

Primary assignee:

clintonk (manila & python-manilaclient)

Other contributors:

vponomaryov (manila-ui)
bswartz (LVM driver)
TBD (generic driver)
vponomaryov (ZFS driver)
akerr (functional tests)
TBD (scenario tests)

Work items

Implement snapshot locks to remove race conditions that cause snapshot timestamps to be unreliable for the purpose of finding the most recent one. These can be file locks as provided by Oslo concurrency, but they should become distributed locks once the Tooz adoption code is available.
Implement revert-to-snapshot command in python-manilaclient
Implement core feature
Implement revert-to-snapshot in at least one first-party driver, covering both normal and replicated snapshots
Implement tempest support
Implement manila-ui support

Dependencies

For the purpose of determining the most recent snapshot, the created_at field on the snapshot object will be used. The snapshot-create API must place a share in the ‘snapshotting’ state to ensure only one snapshot operation occurs on a given share at the same time. Otherwise, manila’s view of the latest snapshot on a share with multiple snapshots could differ from the actual latest snapshot on the storage controller.

Testing

Tempest coverage may be added that checks for the existence of the ‘revert_to_snapshot_support’ capability and exercises the corresponding API. Tests should include cases where the share size is changed after the original snapshot is taken to ensure the share size and quotas are correct after the revert operation. Negative tests should include attempts to restore snapshots other than the most recent one.

To guarantee that a snapshot restore actually took place, a new scenario test will be needed that writes data to a share, creates a snapshot, modifies the share, restores the snapshot, and ensures the original data is present in the share.

Documentation impact

API Ref: Add content about the API.
User Guide: Add content about the revert feature and new public extra spec.
Admin Guide: Add content regarding the revert_to_snapshot_support common capability.
Developer Ref: Add revert_to_snapshot_support to common capabilities and the infamous driver matrix.

References

https://etherpad.openstack.org/p/newton-manila-snapshot-semantics

Support retrieve pools filtered by share-type

Sun, 23 Oct 2016 00:00:00

https://blueprints.launchpad.net/manila/+spec/pool-list-by-share-type

Add feature that allows administrators can get back-end storage pools filtered by share-type, with that Manila will return the pools filtered by share-type’s extra-specs.

Problem description

Currently Manila’s pools list API(detail API included) only supports filter pools by ‘host_name’, ‘pool_name’ and so on. This is not enough when administrators want to know the pool’s status which also can match the specified share type’s requirements. This information is useful when administrators want to track the usage of each share type. Administrators also can get all pools and filter them on their own, but it’s more complicated and inefficient. This change intends to cover this situation and bring more convenience to administrators.

Use Cases

In production environments, administrators often need to have an overall pools availability statistics grouped by share type, this will help them to make an adjustment before resources run out.

Proposed change

The modified version of pools list will be identical at the most part, except for the parts mentioned below:

A query parameter ‘share_type’ will be introduced to pools list API(GET pools and GET pools/detail).
In the API service, the share_type’s ‘extra_specs’ property will be retrieved and wrapped in ‘capabilities’ as a filter parameter for method ‘get_pools’ in scheduler [1].
In Manila scheduler’s host_manager which currently retrieves the pools dict, will add a new logic which would filter all the pools with the ‘capabilities’ parameter.
The logic mentioned above is already existing in Manila [2] (_satisfies_extra_specs) which focus on the extra_spec’s comparison, so it would be better to add an util function in order to reduce code redundancy.
The Manila client will also add two arguments ‘–share_type’ and ‘–detail’ to support this(the ‘–detail’ argument intends to cover the existing feature in Manila).

Alternatives

Administrators also can retrieve and filter the pools on their own, but it’s more complicated and inefficient. This change can reduce the request amount and filter unnecessary data transmitted from server to client.

Data model impact

None

REST API impact

The API microversion will have to be bumped up for the new APIs below.

Pools list API will accept new query string parameter ‘share_type’. Administrators can pass name or id of share type to retrieve pools filtered.

GET /v2/{tenant_id}/scheduler-stats/pools?share_type=share_type1

Detailed pools list API will also accept new query string parameter ‘share_type’. Administrator can pass name or id of the share type to retrieve pools filtered.

GET /v2/{tenant_id}/scheduler-stats/pools/detail?share_type=share_type1

Manila client impact

As the Manila client already has the pool_list command without ‘detail’ feature supported, this spec will add two new command options, which are ‘–detail’ and ‘–share_type’:

‘–detail’ supports show pools information in detail.
‘–share_type’ supports filter pools by share_type’s extra_specs.

The modified version of command will be like:

pool_list --share_type s_type1 --detail <other existing command arguments>

Security impact

None

Notifications impact

None.

Other end user impact

None.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: zhongjun2(jun.zhongjun2@gmail.com) TommyLike(tommylikehu@gmail.com)

Work Items

Add share type filter to pools list and detailed pools list API.
Add unit tests and tempest tests in Manila.
Add documentation in Manila.
Add new command arguments in Manila client.
Add unit tests and functional tests in Manila client.

Dependencies

None

Testing

Unit test to test whether share_type filter can be correctly applied.
Tempest test whether share_type filter works correctly from API perspective.
Manila client’s unit test to test whether new arguments work correctly.
Manila client’s functional tests.

Documentation Impact

The Manila API documentation will need to be updated to reflect the REST API changes.

References

[1] https://github.com/openstack/manila/blob/master/manila/scheduler/rpcapi.py#L72 [2] https://github.com/openstack/manila/blob/master/manila/scheduler/filters/capabilities.py

Add detail API to quota-set API collection

Sun, 23 Oct 2016 00:00:00

https://blueprints.launchpad.net/manila/+spec/admin-check-tenant-quota-usage

Currently we can only get the quota-set information with ‘limit’ attribute returned both for user and tenant in Manila, but this is too limited for user’s various cases. In other OpenStack components such as Nova, it both has the ability to check the ‘quota-show’(only ‘limit’) and ‘quota-show –detail’(‘in_use’, ‘limit’ and ‘reserved’), Cinder also has the same ability with command ‘quota-usage’. This change intends to add the same ability for Manila just like Nova.

Problem description

It’s a common use case to check the quota-set information, and Manila has the API ‘quota-sets’ to cover this, but it does not satisfy the situation when overall ‘quota-sets’ information is desired. In effect, Manila already supported this inside the quota-set’s code [1] at present, but this hasn’t been exposed to user. This change intends to support this.

Use Cases

As described above, user often wants to check the quota-set’s overall information (both tenant and user of this tenant), this change will help them achieve this.

Proposed change

The following are the changes to be made:

A new API ‘…/quota-sets/{tenant_id}/detail’ will be added to quota-sets API collection.
The new API will share same policy with the existing GET ‘quota-sets’ API.
The subsequent logic will be identical with existed API ‘GET quota-sets’ except that the parameter ‘usage’ in ‘QuotaSetsController._get_quotas’ is set True.
The ‘_get_quotas’ will return a dictionary with ‘in_use’, ‘limit’, ‘reserved’ attributes contained.
The Manila client will add a new command argument ‘–detail’ to ‘quota-show’ to indicate whether to return the quota-sets information in detail.

Alternatives

Currently Manila does not support this.

Data model impact

None

REST API impact

The API microversion will have to be bumped up for the new APIs below.

(GET 200 403) quota-sets detail: show quota-sets in detail

URL: /v2/{tenant_id}/quota-sets/{tenant_id}/detail?user_id={u_id_query}

Response Body:

{
    'quota_set':{
        'id': '1e1cc7d3-c524-4b05-85b4-5d548916b11b',
        'shares': {'in_use': 0, 'limit': 23, 'reserved': 0},
        'gigabytes': {'in_use': 0, 'limit': 45, 'reserved': 0},
        'snapshots': {'in_use': 0, 'limit': 34, 'reserved': 0},
        'snapshot_gigabytes': {'in_use': 0,'limit': 56,'reserved': 0},
        'share_networks': {'in_use': 0,'limit': 67,'reserved': 0}
    }
}

Manila client impact

Manila client will add a new command argument ‘–detail’ to command ‘quota-show’, the modified version of command will be like:

* quota-show --detail <other existing command arguments>

Security impact

None

Notifications impact

None.

Other end user impact

None.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: zhongjun2(jun.zhongjun2@gmail.com) TommyLike(tommylikehu@gmail.com)

Work Items

Add ‘detail’ API to ‘quota-sets’ collection.
Add related tests in Manila.
Add documentation in Manila.
Add new argument for command ‘quota-show’ in Manila client.
Add related tests in Manila client.

Dependencies

None

Testing

Unit and tempest tests whether new API works correctly.
Manila client’s unit tests and functional tests on new added argument.

Documentation Impact

The Manila API documentation will need to be updated to reflect the REST API changes.

References

[1] https://github.com/openstack/manila/blob/master/manila/quota.py

Process And Deadlines For Specs

Thu, 22 Sep 2016 00:00:00

The Manila spec repo was created for the Newton release and did not get used very effectively. Many specs did not get reviews, even fewer specs were merged during the release, and some features with merged specs did not get code review priority.

In order to ensure that specs do get reviews, and also to help focus the efforts of the reviewers, I propose adding some official deadlines and rules for handling of specs.

Problem description

We have a few specific problems, some new in Newton, and some which have grown worse over time:

Some specs get minimal reviews, or don’t get reviewed at all.
There is no reason to merge a spec because we didn’t attach any meaning to the action of merging a spec, so most specs were not merged.
The core reviewer team has too little resources to review all of the proposed features in a release.
Code reviews are turning up architectural issues which could have been caught during a spec review, and the problems are being found very late in the release cycle.
Contributors have very little idea whether their changes will get merged until right before feature freeze, because we rarely say “no” to anything unless it misses a deadline.

Use Cases

The purpose of the proposed change is to focus the core reviewer team on a smaller set of features, and to give contributors earlier feedback on their proposals. Rather than having a lot of things with small chances of success, we would like to have a few things with large chances of success.

Proposed change

First, I propose defining what it means to merge a spec. Merging a spec during a release will mean:

We are happy with the definition for the proposed feature, and it fits with the direction for the project and the current release.
The feature will be targeted at the current release, and will receive code review attention and testing.
The team intends to merge the completed feature before feature freeze, if the submitter has completed code and is responsive to feedback.

Note that merging a spec will not guarantee that the code for a feature will merge only that we will try. More importantly, NOT merging a spec will mean that we don’t intend to work on that feature, and it will be reconsidered for the following release.

Next, I propose a specs deadline:

Low priority specs must merge by the milestone-1 date. Specs not deemed high priority and not merged by milestone-1 will be taken out of consideration for the release, to help focus the team.
High priority specs must be merged by the milestone-2 date. Any spec not merged by the milestone-2 date will be taken out of consideration for the release, to help focus the team.
Specs must be deemed high priority by the team before the milestone-1 date. In many cases high priority specs will be decided earlier but the milestone-1 will be the final deadline so we can start narrowing our scope for a release.
High priority specs will be designated by adding them to an index file called high_priority.rst in the appropriate release directory in the specs tree. Modifications to this file will be controlled by gerrit.

Because merging a spec will imply that the team will spend sufficient time reviewing that feature, the team will need to limit the number of merged specs, even if specs meet all of the other requirements to be merged. The team will need to prioritize the proposed features and merge the higher priority ones first, and stop merging specs when there’s no more available review bandwidth.

Because the number of proposed specs can be more than the whole team can reasonably review in the time before the first milestone, the team should designate a subset of the specs as “review focus” specs which everyone is expected to review, while the remaining specs will be reviewed on a volunteer basis. Review focus specs will be listed on an etherpad agreed to by the core reviewer team.

Lastly, I propose that we require specs for any kind of change with broad impacts. It’s better to agree on the design before we get too deep into implementation.

New drivers and changes to drivers should not ever require specs, and thus driver changes are not affected by these deadlines. The exception would be a change to a driver that implemented something unusual or controversial and thus required community discussion.

Decisions

This specs process requires several decisions to be made:

Review focus designation for specs
Prioritization for specs, including stack ranking and low/high priority designation
Cut line for merging specs
Merge decisions for individual specs

It’s important for the contributors involved with Manila to have influence over the direction for the project, so I propose that none of these decisions be made solely by the PTL, and that decisions should be made as democratically as practical (with PTL resolving deadlocks).

The whole community should provide input on the decision for which specs to focus on for reviews, and also the prioritization of specs. We want to work on things which the largest number of people will find valuable so we will gather input during weekly meetings and using etherpads about what we should focus on.

Determination of the cut line has to be made by the core reviewer team because the commitment to review new featutes comes from the core reviewers and the goal of this change is to avoid overcommitting.

Merging of individual specs should be done as usual with two +2s from core reviewers from different companies if the spec is low priority, and with +2 from a majority of the core reviewers for high priority specs.

Designation of a spec as “high priority” which changes the merge deadline from milestone-1 to milestone-2 should be done by agreement by the core reviewer team, because of the additional review commitment and increased risk such a decision carries.

The purpose of giving “high priority” specs more time than “low priority” specs is not to divert attention away from the high priority stuff, but to make the decision to push stuff out of the release as early as possible. This process can’t make people review specs earlier if they’re not inclined to, but we can reduce the set of specs that need reviewing by drawing a cut line fairly early, and hopefully that will focus our limited review time more productively.

Store authentication secrets in access mappings

Tue, 31 May 2016 00:00:00

Blueprint: https://blueprints.launchpad.net/manila/+spec/auth-access-keys

In storage systems such as Ceph with a built-in authentication system that generates user credentials, granting share access to a user involves creating a credential, a secret key, that the user henceforth uses to authenticate oneself. This spec proposes to allow drivers of such storage systems to provide secret keys to the users by storing the keys in manila’s database, and exposing them through a user facing API that provides access rule information.

Problem description

Manila’s access control APIs and underlying mechanisms were designed for storage backends that rely on external authentication systems to identify clients. The share drivers let the storage back ends know the clients that need to be authorized to access a share.

It was not possible for a driver of a storage back end with its own authentication system to return a credential to the client through a manila API. This meant that the clients had to be provided with the requisite credentials out of band of manila, making the admin and user workflows cumbersome.

Use Cases

When a user requests access to a share, a driver will be able to provide an authentication credential, a secret key, to the user through a manila API. For now, cephfs_native driver stands to benefit. It will be able to share the secret key generated by the Ceph storage back end with the user.

Proposed change

Receive the access credentials, secret keys, generated by a storage back end for the newly added share rules, as the return value of the back-end driver’s update_access().
Store the secret keys in the ShareAccessMapping model.
Expose them in the manila.share.api.access_get_all API, used to list the access rules of shares.

Alternatives

Expect the authentication credentials generated by the storage backend to be shared with the users out of band of manila.

Data model impact

Add access_key attribute to the existing ShareAccessMapping model for storing secret keys:
```
access_key = Column(String(255), nullable=True)
```
Add a new DB API, share_access_update_access_key() that updates the attribute access_key of the ShareAccessMapping model.
DB migration:

Upgrade will add the new column access_key to the share_access_map table.

Downgrade will drop the column access_key from the share_access_map table.

REST API impact

Add a response parameter access_key of type string that will display the secret key when listing the access rules.

Method: POST

URL: /shares/{share_id}/action Normal response code: 200

Action body:

{
    "access_list": null
}

Example response:

{
    "access_list": [
        {
            "access_level": "rw",
            "state": "active",
            "id": "507bf114-36f2-4f56-8cf4-857985ca87c1",
            "access_type": "cephx",
            "access_to": "alice",
            "access_key": "AQC7fRhXbQXxHxAApF58+AmP6a3zBpwYWNIBbA=="
        }
    ]
}

Driver impact

ShareDriver:

A driver’s update_access() may choose to return a dictionary of access_id, access_key as key, value pairs to the ShareManager’s access_helper, for the rules that it added. In the case of recovery/maintenance mode of update_access(), the driver will have to return secret keys for all the access IDs that its ordered to sync, i.e., the access IDs that are in the access_rules parameter.
ShareManager:

The update_access() of the ShareManager’s access_helper calls the update_access() of the driver to add access rules. After adding rules, the driver may return a dictionary, {‘access_id’: ‘access_key’, …}. The update_access of the access_helper will use this dictionary to call the new DB API, share_access_update_access_key() iteratively, to store the secret keys for the various access rules in the share_access_map table.

Security impact

The access keys required to access shares will be visible to the users when listing the share access rules.

Notifications impact

None

Other end user impact

python-manilaclient:

When listing access rules of a share, a new column, access_key will display the access credential (if supplied by a driver). The user will be able to selectively view it.
manila-ui:

A new column access_key will be seen in the RulesTable.

Performance Impact

When a driver adds access rules and returns corresponding access keys, the access keys will be updated for the various access IDs in the share_access_map table.

Other deployer impact

None

Developer impact

Only those drivers that can make use of the feature added by this spec would need to be modified along with the tempest tests run by their CIs.

Implementation

Assignee(s)

Primary assignee:: rraja

Work Items

Enable cephfs_native driver to return access keys when adding share access rules.
Implement core changes to receive access keys from the driver, store them in the share_access_map table, and expose them via access_get_all API.
Allow python-manilaclient and manila-ui to display the access keys.

Dependencies

Work will depend on changes to be made to bring back monitoring of access status per access rule instead of per share. This was discussed at the Austin summit, https://etherpad.openstack.org/p/newton-manila-update-access

Testing

Update the unit tests in manila, python-manilaclient, and manila-ui repositories.
Update the tempest tests in manila repository.
Update the functional tests in python-manilaclient repository.

Documentation Impact

Update the API reference guide.
Update the configuration reference guide mentioning the changes in the cephfs_native driver.
Update the development reference guide.
Update the user guide.

References

Mailing list: http://lists.openstack.org/pipermail/openstack-dev/2015-October/077602.html

Newton Share Migration Improvements

Thu, 12 May 2016 00:00:00

https://blueprints.launchpad.net/manila/+spec/newton-migration-improvements

Share migration is a feature that allows an administrator to move a share across backends. Ideally, the data moved should be exactly the same as before. This operation is expected to be disruptive in most cases, because if data is moved to another place or backend, the export location may need to change, thus the client may need to re-mount the share. In order to handle this disruptiveness, this spec proposes a 2-phase migration approach.

Problem description

Whenever a share is created on a backend, it cannot be moved to another backend in the use cases scenarios. The administrators would have to do it manually by creating an empty share in the destination backend, mounting both shares, copying data, and handling all possible difficulties by themselves, while also being inefficient, because in this approach it would download and re-upload all the data being copied. Since several administrators from several cloud environments are prone to facing these scenarios, it justifies having a feature that performs this in a common way.

Use Cases

There are several scenarios for which a share may need to be migrated:

Administrator-oriented

Maintenance/Evacuation
- Evacuate a backend for hardware/software upgrades
- Evacuate a backend experiencing failures
- Evacuate a backend which is EOL
Optimization
- Defragment backends to create empty ones which can be taken offline to conserve power
- Rebalance backends to maximize available performance
- Move data and compute closer together to reduce network utilization and decrease latency/increase bandwidth
True migration
- Migrate from old hardware generation to a newer generation
- Migrate from one vendor to another

User-oriented (through another feature, such as share-modify or share groups)

Change share type
Change AZ
Change share protocol
Change share network (for DHSS=true) shares
Change share group
Expand share when there’s no available space

Proposed change

This spec aims to re-evaluate the Share Migration design, including the improvements discussed at Austin Summit 2016.

There are two possible ways of doing migration:

Driver understands destination backend and is able to move at backend level.
Manila migrates the share’s data to another share created in the destination backend, using the Data Service.

In (1), the backend may use different mechanisms such as replication or snapshots, may not require share to be mounted, may be able to do so without changing the share to read-only, changing the export location and may also not be disruptive. When the operation is completed, it should return a new list of export locations to update the database, when necessary. This approach is referred to as driver-assisted approach.

In (2), share must be changed to read-only while we are using non-incremental copy approach, which is expected to cause downtime to tenant applications. This approach may not be able to preserve file metadata and is also referred to as host-assisted approach.

It is very important to note that when migrating a share from a backend to another, the destination backend needs to support exactly the same share protocol as the source share’s. Migration itself does not change share protocols, another feature called “Share Modify” that is yet to be implemented should be responsible for doing this operation and may use share migration feature to do so.

Some attributes such as ‘Share Network’, ‘Share Type’ and ‘Availability Zone’ may be modified through migration, the administrator can do so through the API.

Since migration is expected to be disruptive, both if performed by the driver (as in #1 above) or if performed by manila code (as in #2 above), it was designed with a 2-phase possibility in mind. When invoking “migrationg-start”, a share will be migrated but will pause when the 1st phase is completed without any disruptiveness, so the administrator can prepare for when to invoke “migration-complete” to finish the migration, which may be disruptive.

Alternatives

Do not have the feature implemented and administrator will need to do this manually and less efficiently. Multiple cloud providers would need to create scripts for doing this and handle each failure scenario themselves. This spec proposes a common way of doing this.

Data model impact

In order to track share migration’s operations, support 2-phase migration and properly handle errors, a field in the database is required. An additional field in the “Share” table is able to meet these requirements until the Jobs table (see [1]) is implemented. The field, named “task_state”, works in a similar way as a status and it can be reset via API “reset-task-state”.

In order to have change types during a migration, at a certain point, a share needs to have two instances with different types. At the moment, this is not possible due to the ‘share_type_id’ field being within the ‘Shares’ table in the database. So this spec includes a database migration to move the ‘share_type_id’ field to the ‘ShareInstances’ table.

REST API impact

Five admin-only new API methods are introduced:

(POST, 202, 400, 409) migration-start: migrates share

URL: /shares/<share_id>/action

Body:

{
  'migration_start': {
    'force_host_assisted_migration': false,
    'preserve_metadata': true,
    'writable': true,
    'nondisruptive': true,
    'host': 'ubuntu@generic2#GENERIC2',
    'new_share_type_id': 'foo_share_type_id',
    'new_share_network_id': 'bar_share_network_id'
  }
}

(POST, 202, 400) migration-complete: triggers 2nd phase of migration

URL: /shares/<share_id>/action

Body:

{"migration_complete": {}}

(POST, 200, 400) migration-get-progress: attempts to obtain migration progress

URL: /shares/<share_id>/action

Body:

{"migration_get_progress": {}}

Example response:

RESP BODY: {
  "task_state": "data_copying_in_progress",
  "total_progress": 50,
}

(POST, 202, 400) migration-cancel: attempts to cancel migration

URL: /shares/<share_id>/action

Body:

{"migration_cancel": {}}

(POST, 202, 400) reset-task-state: reset task state field value to desired one

URL: /shares/<share_id>/action

Body:

{"reset_task_state": {"task_state": "migration_error}}

API details:

migration-start [--force-host-assisted-migration <True/False>] [--preserve-metadata <True/False>] [--writable <True/False>] [--non-disruptive <True/False>] [--new-share-type <new_share_type>] [--new-share-network <new_share_network>] <share> <host@backend#pool>

force-host-assisted-migration (defaults False):: forces the host-assisted approach to be used, thus using the Data Service to move copy data across backends. This skips the driver-assisted approach which would otherwise be run attempted first.
preserve-metadata (defaults True):: whether migration should enforce the preservation of metadata. If set to True, this will prevent host-assisted migration from running. Drivers are queried to validate this capability, and if not capable, driver-assisted approach will be skipped and migration will fail.
writable (defaults to True):: whether migration should only be performed if share remains writable. If set to True, this will prevent host-assisted migration from running. Drivers are queried to validate this capability, and if not capable, driver-assisted approach will be skipped and migration will fail.
non-disruptive (defaults to False):: whether migration should only be performed if share access is not disrupted during migration. For such, it is also expected that the export location does not change. If set to True, this will prevent host-assisted migration from running. Drivers are queried to validate this capability, and if not capable, driver-assisted approach will be skipped and migration will fail.
new-share-type (defaults to None):: the new share type that should be set in the migrated share.
new-share-network (defaults to None):: the new share network that should be set in the migrated share.
share:: share to be moved.
host@backend#pool:: string that combines host@backend#pool combination where share should be migrated to.

migration-complete <share>

share:: share on which migration should be completed. Share must be in host-assisted’s “copy completed” or driver-assisted’s “driver phase 1 completed” task state to have its phase 2 migration invoked.

migration-get-progress <share>

share:: share from which migration progress should be obtained. The total progress is displayed along with the current task state value. If the share is not being migrated or the driver cannot obtain progress then an error message is returned.

migration-cancel <share>

share:: share from which migration should be cancelled. Share must be in host-assisted’s “copy completed” or driver-assisted’s “driver phase 1 completed” task state to be cancellable.

reset-task-state [--task-state <state>] <share>

task-state (defaults to None):: value to reset the task state field to.
share:: share from which the task state field should be reset to the value provided.

Driver impact

Vendors can implement the driver-assisted migration in their drivers in order to migrate data efficiently across backends from the same vendor or using vendor-compatible protocols.

In order to support host-assisted migration, existing drivers which run in DHSS=False mode should not need to implement any additional code, while code to handle their share protocol should be present in the Data Service and the networks need to be manually set up. However, it is highly recommend to support admin network to require less network configuration effort.

For DHSS=True mode drivers, if existing drivers do not have admin network support to allow connectivity between shares and nodes in admin network, the host-assisted approach will not work.

Add driver interfaces:

def migration_check_compatibility(
        self, context, source_share, destination_share,
        share_server=None, destination_share_server=None):
    """Checks destination compatibility for migration of a given share.

    .. note::
        Is called to test compatibility with destination backend.

    Driver should check if it is compatible with destination backend so
    driver-assisted migration can proceed.

    :param context: The 'context.RequestContext' object for the request.
    :param source_share: Reference to the share to be migrated.
    :param destination_share: Reference to the share model to be used by
        migrated share.
    :param share_server: Share server model or None.
    :param destination_share_server: Destination Share server model or
        None.
    :return: A dictionary containing values indicating if destination
        backend is compatible, if share can remain writable during
        migration, if it can preserve all file metadata and if it can
        perform migration of given share non-disruptively.

        Example::

        {
            'compatible': True,
            'writable': True,
            'preserve_metadata': True,
            'nondisruptive': True,
        }
    """
    return {
        'compatible': False,
        'writable': False,
        'preserve_metadata': False,
        'nondisruptive': False,
    }

def migration_start(
        self, context, source_share, destination_share,
        share_server=None, destination_share_server=None):
    """Starts migration of a given share to another host.

    .. note::
       Is called in source share's backend to start migration.

    Driver should implement this method if willing to perform migration
    in a driver-assisted way, useful for when source share's backend driver
    is compatible with destination backend driver. This method should
    start the migration procedure in the backend and end. Following steps
    should be done in 'migration_continue'.

    :param context: The 'context.RequestContext' object for the request.
    :param source_share: Reference to the original share model.
    :param destination_share: Reference to the share model to be used by
        migrated share.
    :param share_server: Share server model or None.
    :param destination_share_server: Destination Share server model or
        None.
    """
    raise NotImplementedError()

def migration_continue(
        self, context, source_share, destination_share,
        share_server=None, destination_share_server=None):
    """Continues migration of a given share to another host.

    .. note::
        Is called in source share's backend to continue migration.

    Driver should implement this method to continue monitor the migration
    progress in storage and perform following steps until 1st phase is
    completed.

    :param context: The 'context.RequestContext' object for the request.
    :param source_share: Reference to the original share model.
    :param destination_share: Reference to the share model to be used by
        migrated share.
    :param share_server: Share server model or None.
    :param destination_share_server: Destination Share server model or
        None.
    :return: Boolean value to indicate if 1st phase is finished.
    """
    raise NotImplementedError()

def migration_complete(
        self, context, source_share, destination_share,
        share_server=None, destination_share_server=None):
    """Completes migration of a given share to another host.

    .. note::
        Is called in source share's backend to complete migration.

    If driver is implementing 2-phase migration, this method should
    perform the disruptive tasks related to the 2nd phase of migration,
    thus completing it. Driver should also delete all original share data
    from source backend.

    :param context: The 'context.RequestContext' object for the request.
    :param source_share: Reference to the original share model.
    :param destination_share: Reference to the share model to be used by
        migrated share.
    :param share_server: Share server model or None.
    :param destination_share_server: Destination Share server model or
        None.
    :return: List of export locations to update the share with.
    """
    raise NotImplementedError()

def migration_cancel(
        self, context, source_share, destination_share,
        share_server=None, destination_share_server=None):
    """Cancels migration of a given share to another host.

    .. note::
       Is called in source share's backend to cancel migration.

    If possible, driver can implement a way to cancel an in-progress
    migration.

    :param context: The 'context.RequestContext' object for the request.
    :param source_share: Reference to the original share model.
    :param destination_share: Reference to the share model to be used by
        migrated share.
    :param share_server: Share server model or None.
    :param destination_share_server: Destination Share server model or
        None.
    """
    raise NotImplementedError()

def migration_get_progress(
        self, context, source_share, destination_share,
        share_server=None, destination_share_server=None):
    """Obtains progress of migration of a given share to another host.

    .. note::
        Is called in source share's backend to obtain migration progress.

    If possible, driver can implement a way to return migration progress
    information.
    :param context: The 'context.RequestContext' object for the request.
    :param source_share: Reference to the original share model.
    :param destination_share: Reference to the share model to be used by
        migrated share.
    :param share_server: Share server model or None.
    :param destination_share_server: Destination Share server model or
        None.
    :return: A dictionary with at least 'total_progress' field containing
        the percentage value.
    """
    raise NotImplementedError()

def connection_get_info(self, context, share, share_server=None):
    """Is called to provide necessary generic migration logic.

    :param context: The 'context.RequestContext' object for the request.
    :param share: Reference to the share being migrated.
    :param share_server: Share server model or None.
    :return: A dictionary with migration information.
    """

    Has a default implementation that can be overridden.

The general approach for a driver-assisted migration is that drivers will be invoked to analyze compatibility with the destination backend and return a dictionary containing information that describes whether they are compatible and which capabilities, such as perserving metadata, remaining writable and being non-disruptive, are supported. To obtain information to perform this analysis, drivers are advised to read other related backends data from manila configuration file. Ideally, they should be talking to each other through RPC calls, but sensitive data such as passwords should not be included in RPC responses, so such approach cannot be taken at this moment. At this point, it is recommend that drivers also test for connectivity with the destination backend.

If the destination share has a share network ID defined, it is implied that the share requires a share server, so manila code will send a request to the destination backend so it can provide a share server, where if one does not exist, it will be created on the destination backend, invoked by manila.

Then, the migration_start method of the source backend driver is invoked, to perform migration. This method should start the migration job in the storage and return. Manila will invoke the method migration_continue according to a periodic task so the driver can perform subsequent steps to continue migration until the first phase is completed, in which the driver should return so.

The driver should also make sure that while migration is not completed, the source share instance must be revertible to and its data intact.

At last, the administrator will invoke migration-complete to perform the last, possibly disruptive, steps of migration. Drivers should remove the source share at this moment. Manila will also apply the existing access rules to the destination instance using update_access driver interface.

Additionally, if the current base driver class implementation for several methods used by the host-assisted migration is not supported by a driver, the driver can override those methods adding a special behavior to support the host-assisted migration approach.

Security impact

In order to access a share’s data, it must be mounted by an entity. The entity mounting the share is responsible for the data. During migration, if the host-assisted approach is used, the Data Service will be mounting the migrating share and copying data, thus it will expose the share’s contents to the Data Service node during a limited period of time. The Data Service is accessible through the administrator network, thus it grants access to data to whoever is able to connect to the Data Service. Restricted access to this node is advised.

This change includes new entries to rootwrap permission file, for following commands that must be run as root:

ls -pA1 –group-directories-first %s
touch –reference=%s %s

Notifications impact

None

Other end user impact

None

Performance Impact

All the copy-related commands are resource-intensive and should be run in a separate node where only the Data Service is installed, thus not disrupting the other services.

Other deployer impact

New configuration options are introduced. Most have default values, while a few others require an administrator to input them. In order for the Data Service to handle mounting shares of several different protocols, it needs to be configured:

The node must be set up in the admin network, and the config option ‘data_node_access_ip’ must be set with the IP value of this node interface that connects it to the admin network. This is enough to mount shares which access rules are IP-based.
Protocol libraries like for NFS and CIFS need to be installed in this node.
For protocols which access rules are certificate-based, the certificate needs to be installed and the config option ‘data_node_access_cert’ must be set.
For protocols which access rules are user-based, the user must be configured in the node and backend security service as an administrator. The username must be set in the ‘data_node_access_admin_user’ config option and the ‘data_node_mount_options’ config option must be set with the command parameters that include the username, password and domains required to mount as the admin user.
Other protocols other than NFS and CIFS have not been tested, they may work if their access type is included among the supported ones.
In order to properly check compatibility with destination backends, drivers will rely on their local configuration files to read information about other backends, so it is advisable that deployers try to keep configuration files of multiple manila-share nodes synchronized and the latest values loaded in the services’ memory.

Developer impact

Driver vendors and CI maintainers are advised to enable migration tests to validate whether the host-assisted approach works for their respective drivers and share protocols.

Implementation

Upon receiving the API request to migrate a share, the Share API layer will perform the following validations:

Check if share has replicas:

if True, return error 409 (Conflict). Migration of a share with replicas is not handled at this moment.

Check share’s status:

If not available, return error 400 (Invalid).

Check if share is busy with another task:

If busy, return error 400 (Invalid).

Check if destination host is different:

If it is the same, return error 400 (Invalid).

Check if there are snapshots:

If there are, return error 400 (Invalid). Migration of a share with snapshots is not handled at this moment.

Check if destination host is available:

If it is not, return error 400 (Invalid).

Check if the new_share_type and share_network_id supplied exist:

If not, return error 400 (Invalid).

If all validations succeed, it should set task_state to MIGRATION_STARTING and invoke the scheduler asynchronously to validate the host against the share type. If host validation fails, scheduler will set task_state to MIGRATION_ERROR (no notification). Else it will invoke the source share’s manager also asynchronously to proceed with migration.

If new_share_type is supplied, it will be used when validating the host in the scheduler instead of the share’s original one. The new_share_network, if supplied, will be used when creating the share instance model that will be used by the migrated share, thus triggering the creation of a new share server, if necessary.

At the share manager, it will change the share’s task_state to MIGRATION_IN_PROGRESS and instance status to MIGRATING. Then, it will prepare to invoke the driver-assisted migration if the force_host_assisted_migration API parameter is set to False.

First, it will attempt to perform the driver-assisted migration, by creating a destination share instance model, obtaining a share server for it and invoking a method that checks for compatibility. If it succeeds and returned capabilities correspond to the supplied API values for ‘writable’, ‘preserve_metadata’ and ‘non-disruptive’, the task_state is set to MIGRATION_DRIVER_STARTING and the driver’s migration_start method is invoked, in which the driver is expected to start the migration job and return a list of export locations to access the destination instance, if possible at this point.

At this point, the task state is set to MIGRATION_DRIVER_IN_PROGRESS and a period task runs to invoke the driver’s migration_continue method to perform the next steps of migration until it returns True, signaling that the first migration phase has completed, allowing the task state to be set to MIGRATION_DRIVER_PHASE1_DONE.

If any exception is raised before the first migration phase is completed, all data allocated, such as the destination share instance model and share server is cleaned up. If an exception is raised during the second migration phase, data is not cleaned up so the administrator can analyze the failure and possibly fix manually.

If the driver-assisted migration fails up to the migration_start driver call, the host-assisted approach takes over, if the variables ‘preserve_metadata’, ‘writable’ and ‘nondisruptive’ supplied API values are all False.

The host-assisted approach code consists in changing all of share’s access rules to read-only through the driver (rules are not changed in DB), creating a new share in the destination host through RPCAPI asynchronously and waiting for it to have “available” status, obtaining the connection_info dictionaries for the source and destination backends and invoking the Data Service asynchronously to perform the migration.

The RPCAPIs for the Data Service include “migration_start” which perform the data copy with regards to the logic for migration (like setting proper statuses and notifying the source backend when necessary), “data_copy_cancel” and “data_copy_get_progress” which are detailed below and are applicable to any data copy job.

The connection_info dictionary consists in the access mapping compatible with the share being migrated and two templates, one for the mount command, and one for the unmount command. The base driver class has a default implementation for these templates, and can be overridden if the driver requires a particular custom behavior. Both command templates can be customized through the manila.conf configuration file for each backend section, although the Data Service expects at least the “%(path)s” section to be present in the template so it can be replaced by the appropriate export location. Other template elements need to be overridden by customizing the “connection_get_info” method.

The default mount template is: mount -vt %(proto)s %(options) %(export)s %(path)s The default unmount template is: umount -v %(path)s

The following access mapping is predefined in the driver base class:

{
  'ip': ['nfs'],
  'user': ['cifs'],
}

The Data Service does the hard work of migration, it is responsible for calling the API to add the proper access rule to be able to mount both the source and destination shares, mount, copy, unmount, and delete the access rules.

The way to determine the proper access rule type is according to the share’s protocol and access mapping configured by the driver. The share’s protocol will select the access types where the protocol entry is present and then intersect with the other backend’s access mapping to add an access rule that does not cause errors for any of the involved drivers.

All access rules related to the access types present in the intersected access mapping will be added and later removed after migration. To properly fill the ‘access_to’ field of the access rule entries, the Data Service reads a config option for each type, as follows:

If access type is ‘user’, it will read ‘data_node_admin_user’ config option. In this case, it is also expected that the administrator has filled the ‘data_node_mount_options’ such as ‘-o user=foo,pass=bar’ if that is necessary to mount the share.
If access type is ‘cert’, it will read ‘data_node_access_cert’ config option.
If access type is ‘ip’, it will read ‘data_node_access_ip’ config option.
Else it will throw an error message that the access type provided is not supported.

It is expected that the Data Service node is configured properly in the admin network, has proper libraries and certificates installed and security service user configured.

The copy work is done by specialized copy class. This class is responsible for iterating through files recursively, copy them attempting to preserve all metadata (cp -P –preserve=all), optionally verify if the SHA-2 hashes of the source and destination match, and finally attempt to apply the metadata of all the files. All operations are performed as root so user restrictions can be bypassed and set back to files after copy. Since all copy operations are performed by running linux commands through rootwrap, the Data Service thread sleeps until the process exits, thus allowing the Data Service to process multiple RPC requests while the node is copying bytes. If any of the above-mentioned operations fail or are not validated, it will be retried once, and if it fails a second time, it will return an error and migration will fail.

After copying, the Data Service will set the task state to DATA_COPYING_COMPLETED, allowing the admin to invoke migration-complete.

The share’s manager migration_complete method will first check the migrating share’s task_state value to decide whether to invoke the driver’s migration_complete method or host-assisted approach one. The driver is expected to perform the last disruptive steps of migration and return the list of export locations pertaining to the migrated instance. The host-assisted migration_complete applies the access rules to the new share according to the DB (the original rules), sets the destination share status to “available”, deletes the source share and sets the task_state to MIGRATION_SUCCESS.

The migration_cancel API can only be invoked during copying or first phase is completed. The migration_get_progress API can be invoked at any time, but it will only query the driver or the Data Service for the progress if they are at the step of migrating or copying files, respectively.

Assignee(s)

Primary assignee:: ganso

Work Items

Implement changes agreed in this spec. Since they are improvements to existing code, they could be implemented as a single patch.
Update python-manilaclient with the CLI commands.
Update manila-ui.
Document the implementation (see below).

Dependencies

Update Access interface implemented in drivers.

Testing

Unit tests
Tempest tests

Documentation Impact

Docstrings
Devref
Security guide
User guide
Release notes

References

[1] Newton design summit etherpad discussion:

https://etherpad.openstack.org/p/newton-manila-data-service-migration

[2] Mitaka design summit etherpad discussion:

https://etherpad.openstack.org/p/mitaka-manila-migration-improvements

[3] Mitaka merged main patches:

https://review.openstack.org/#/c/244286/
https://review.openstack.org/#/c/250515/

[4] Liberty design summit etherpad discussion:

https://etherpad.openstack.org/p/YVR-manila-liberty-share-migration

[5] Liberty merged main patch:

https://review.openstack.org/#/c/179790/

[6] Access support mapping:

http://docs.openstack.org/developer/manila/devref/share_back_ends_feature_support_mapping.html

Contributing to: manila-specs

Wed, 04 May 2016 00:00:00

If you would like to contribute to the development of OpenStack, you must follow the steps in this page:

http://docs.openstack.org/infra/manual/developers.html

If you already have a good understanding of how the system works and your OpenStack accounts are set up, you can skip to the development workflow section of this documentation to learn how changes to OpenStack should be submitted for review via the Gerrit tool:

http://docs.openstack.org/infra/manual/developers.html#development-workflow

Pull requests submitted through GitHub will be ignored.

Bugs should be filed on Launchpad, not GitHub:

https://bugs.launchpad.net/manila