Metadata for all user facing resources

Launchpad blueprint:

https://blueprints.launchpad.net/manila/+spec/metadata-for-share-resources

Manila stores information regarding user-owned tangible technical assets in its database as resources. These assets include file shares, snapshots, access control rules, export locations, share networks and security services among others. Resources are designed to accommodate distinguishable identifiers geared at machines and humans such as ID, Name and Description. However these identifiers may be insufficient or inflexible to end users’ needs such as being able to tag resources to a purpose, or attach labels that drive some automation. This specification proposes a design allowing for user modifiable metadata on all user facing resources.

Problem description

End users interact with manila API to manage their shared file system storage resources. These resources include:

  • Shares

  • Share Export Locations

  • Share Access Rules

  • Share Instances

  • Share Instance Export Locations

  • Share Replicas

  • Share Replica Export Locations

  • Share Snapshots

  • Share Groups

  • Share Group Snapshots

  • Security Services

  • Share Networks

  • Share Network Subnets

All of these resources are identifiable by ID. Some of them even allow setting free form text as Name and/or Description. Free form text is usually hard to construct, breakdown or query, so these fields are mostly useful to humans rather than machines. Many times, users want to be able to store much more auxiliary contextual information.

One way they do that today with shares and access rules is with the help of “metadata”. Metadata is a term used to describe auxiliary information associated with some data. For shares and access rules, users today use metadata as flexible key=value pairs to attach useful optional/additional information. This sort of interaction isn’t possible with the other resources enumerated above.

Metadata isn’t just helpful for user interactions. It could also provide additional representational space for manila itself. For example, the service currently uses database objects to represent export location metadata to relay helpful hints regarding the nature of specific exports to end users, such as the preference and accessibility. This metadata is owned by the service, and end users cannot modify this metadata.

Use Cases

Users may want to use metadata to store contextual information such as:

  • the purpose of the resource, e.g.: “usedfor=fileserver”

  • details regarding the provisioner, e.g.: “createdby=manila-csi”

  • grouping information, e.g.: “department=physics”

  • additional chronological information, e.g.: “last_audited=2020-12-24T00:22:29”

  • mount point tag for clients, e.g.: “fstag=website”

Metadata interactions are expected to be uniform and consistent across resources, which makes it easier for building automation on top of these APIs.

Proposed change

A new metadata controller mixin will be created in manila’s API modules that can be inherited by all the user facing API controllers. This mixin will implement API handler code to:

  • get resource metadata (retrieve either all of it, or specific item)

  • set/unset all resource metadata (create, update or delete metadata items as key=value pairs)

  • set a single resource metadata item (update a specific key=value pair)

  • unset resource metadata item (delete a specific key=value pair)

To distinguish and protect service owned or admin only metadata, there will be an ignore_keys parameter and ignored keys dictionary to prevent these metadata items from being manipulated by the end user.

Alternatives

Alternatives to key-value metadata include Name and Description fields. Resources that don’t currently support name and/or description can be updated to include these fields. This alternative suffers from the inflexibility problems described above and doesn’t cover all the use cases.

We could also just add metadata controllers one at a time, based on user feedback instead of adding metadata controllers for each user facing resource. This also allows us to test each metadata controller in isolation and build incrementally. This is pretty non-controversial, however, each resource API update will have to be made in a new microversion, and doing things one at a time instead of using the inheritence pattern to subsume common code will be costlier. Testing effort is significant, and unavoidable.

Instead of distinguishing “user” owned and “service” owned metadata, we could allow all metadata to be modifiable by end users - even data that’s created by the service. Doing so would simplify user interactions, however, it may harm the service and cause misbehavior.

Another option to including a user_modifiable boolean field to metadata is to introduce granular RBAC. However, this may produce too many RBAC rules that may be unnecessary. If this turns out to be a use case, it is possible to easily build on top of the proposed design.

Data model impact

To store resource metadata, new tables and corresponding ORM models are necessary. These tables will be created during a database upgrade, and nullified and destroyed during a database downgrade.

Existing metadata tables for Shares (“share_metadata”), Export Locations (“share_instance_export_locations_metadata”) will be modified to include a string attribute called deleted. The default value for this attribute is set to False.

Existing metadata tables will also be modified to replace the datatype of the “id” field to a string of length 36 for UUIDs. This will be done to avoid integer overflows and provide scalability.

A general ORM schema for a metadata table will be as follows (where ‘Resource’ is a stand-in for the real resource name):

class ResourceMetadata(BASE, ManilaBase):
  """Represents a metadata key/value pair for a Resource."""
  __tablename__ = 'resource_metadata'
  id = Column(String(36), primary_key=True)
  key = Column(String(255), nullable=False)
  value = Column(String(1023), nullable=False)
  resource_id = Column(String(36), ForeignKey('resources.id'), nullable=False)
  deleted = Column(String(36), default='False')
  resource = orm.relationship(Resource, backref="resource_metadata",
                              foreign_keys=resource_id,
                              primaryjoin='and_('
                              'ResourceMetadata.resource_id == Resource.id,'
                              'ResourceMetadata.deleted == "False)')

Metadata items are not soft deleted when they are unset by the service or by end users. The metadata table is not loaded alongside the resource unless the resource has been queried with metadata, or a detailed view of the resource has been requested.

REST API impact

New API endpoints will be created to index metadata, show metadata item, create metadata, update metadata item, update_all metadata (delete all existing metadata and update with requested metadata), and delete metadata item for each resource. The general structure of these APIs is as follows:

Index Metadata

Retrieve all metadata key=value pairs as JSON:

GET /v2/{resource}/metadata

  • Sample request body: null

  • Success Codes: 200

  • Default API policy role: Project Reader

  • Error Codes: 401 (Unauthorized), 403 (Policy Not Authorized), 404 (Invalid resource)

  • Sample response body:

    {
   "metadata": {
       "project": "my_app",
       "aim": "doc"
   }
}

Show specific metadata item

Retrieve a single metadata key=value pair:

GET /v2/{resource}/metadata/{key}

  • Sample request body: null

  • Success Codes: 200

  • Default API policy role: Project Reader

  • Error Codes: 401 (Unauthorized), 403 (Policy Not Authorized), 404 (Invalid resource)

  • Sample response body:

    {
   "metadata": {
       "project": "my_app",
   }
}

Update all metadata

Replace all metadata with the updated set, can also be used to delete all metadata:

PUT /v2/{resource}/metadata

  • Sample request body:

    {
    "metadata": {
       "aim": "changed_doc",
       "project": "my_app",
       "new_metadata_key": "new_information"
    }
}

  • Success Codes: 200

  • Default API policy role: Project Member

  • Error Codes: 401 (Unauthorized), 403 (Policy Not Authorized), 404 (Invalid resource), 400 (Malformed request)

  • Sample response body:

    {
   "metadata": {
      "aim": "changed_doc",
      "project": "my_app",
      "new_metadata_key": "new_information"
   }
}

Note: Metadata keys that are part of the admin-only dictionary will not be deleted and updated if the user requesting this is not an system or project admin.

Update specific metadata item

Update a specific metadata item, leaving the rest unmodified:

POST /v2/{resource}/metadata/{key}

  • Sample request body:

    {
   "metadata": {
      "aim": "updated_doc",
   }
}

  • Success Codes: 200

  • Default API policy role: Project Member

  • Error Codes: 401 (Unauthorized), 403 (Policy Not Authorized), 404 (Invalid resource), 400 (Malformed request)

  • Sample response body:

    {
   "metadata": {
      "aim": "updated_doc",
      "project": "my_app",
      "new_metadata_key": "new_information"
   }
}

Important

Currently, the POST /v2/{share}/metadata API currently expects a meta object. However, the other metadata APIs expect a metadata object. For the sake of consistency, this error will be fixed in a new API microversion.

Delete specific metadata item

Hard delete a single metadata key=value pair:

DELETE /v2/{resource}/metadata/{key}

  • Sample request body: null

  • Success Codes: 200

  • Default API policy role: Project Member

  • Error Codes: 401 (Unauthorized), 403 (Policy Not Authorized), 404 (Invalid resource)

  • Sample response body: null

Query resources by metadata items

URL encoding can be performed by the client:

GET /v2/{resource}?metadata=%7B%27foo%27%3A%27bar%27%2C%27clemson%27%3A%27tigers%27%7D

or the request can be made in a decoded format as well:

GET /v2/{resource}?metadata={'foo':'bar','clemson':'tigers'}

Driver impact

None. Metadata manipulation is directly performed on the manila database and shared file system back end drivers are not invoked during the creation, modification or deletion of resource metadata.

Security impact

It is advised that metadata operations are rate limited to prevent bad actors or automation from adding a large number of metadata items to a resource.

Notifications impact

None

Other end user impact

Python-manilaclient SDK will include support for the new APIs and we’ll ensure that there are corresponding CLI commands in the new OSC plugin shell. Manila UI’s support for share, export location and access rule metadata is limited. This specification doesn’t seek to address all the UI gaps; but all effort will be made to close the feature parity between the CLI utilities and the UI. Eventually users will be able to perform all metadata interactions via the UI as well.

Performance Impact

API performance is bound to suffer when resource queries include metadata items. Since we’ll be providing metadata along with detail retrievals of resources, the performance of those APIs will also be affected negatively because of the new database joins that will be necessary. Over time, as the number of shares and the metadata tables grow, performance degradation can be severe. This impact will be documented; as best practice, it is recommended that a resource have no more than a handful of metadata items.

Other deployer impact

New APIs introduced may require tweaking of policy files if the default RBAC policy is not acceptable.

Developer impact

When adding any new user facing metadata, the metadata mixin controller can be inherited and extended by developers.

Implementation

Assignee(s)

Primary assignee:

ashrod98 <ashrod98@gmail.com>

Other contributors:

gouthamr

Work Items

  • Add database migration to convert the id field of share, export location and access rule metadata to a string from integer and populate the field with UUIDs

  • Add database migrations to introduce the “deleted” field to share, export location and access rule metadata tables.

  • Add database migrations to create new metadata tables for all other resources

  • Add MetadataControllerMixin, inherit and extend in all resources and bump up the API microversion.

  • Add unit and integration tests

  • Add support for metadata APIs in manilaclient SDK, and OSC CLI and SDK

  • Add support for metadata interactions in the UI

  • Add documentation

A further enhancement to this API would be to provide an interface for administrators to create admin-only metadata values. Currently, admin-only metadata values are delinated in a dictionary in manila/manila/common/constants.py. Such a fixated list is sufficient for the implementation of backend filtering for scheduling according to share affinities. To implement this customization, we could revisit adjusting the data-model to include an admin-only boolean value identifying which keys are adjustable by admins or non-admins. Or we could provide some configuration option in manila/manila/data/. Such an enhancement requires further thought, and can be implemented at a later point.

Dependencies

Not a direct dependency, but this API change incorporates the metadata changes necessary to implement this spec: Affinity and anti-affinity scheduler filter

Testing

Extensive unit tests will be written to test the API and database methods being added. A database “walk migrations” test will be added for all the database changes. Tempest tests will be added to cover the new metadata operations across resources.

Documentation Impact

  • API reference

  • End user guide

  • Release notes

References