Tag Catalog Support For Metadata Definitions

Tag Catalog Support For Metadata Definitions

https://blueprints.launchpad.net/glance/+spec/metadefs-tags

This blueprint adds the basic tag catalog back after it was deferred from Juno due to time constraints. The original, approved spec included supporting tag libraries in the metadata definitions catalog. However, at the end of Juno, the original spec was updated to remove tags since they weren’t implemented.

This spec is actually a reduction of the tag concepts in the original approved Juno spec. In the original Juno spec, tags had a dynamic hierarchy capability. This spec does not include the tag hierarchy in order to simplify this spec. That aspect of tags will be deferred to a later spec.

The implemented juno spec is here for reference:

https://github.com/openstack/glance-specs/blob/master/specs/juno/metadata-schema-catalog.rst

Problem description

A challenge with using OpenStack is discovering, sharing, and correlating tags across services and different types of resources. We believe this affects both end users and administrators.

For example, a cloud operator or vendor may have a predefined set of “tags” they want to be used as a starting point for images and instances. Currently, OpenStack does not have a facility for the cloud operator to include that base set of tags. This means that every deployment and every project may end up with its own disparate set of tags. This leads to inconsistencies, but also is extra hassle for end users who end up reinventing all the “tags” in every project.

For example, is the tag “postgres” the same as “PostgreSQL”? If a base library of tags is used, any user typing “pos” would be prompted with the one that already exists in the tag library and would choose it. Future searches based on tags would ensure consistent results.

Terminology

The term metadata can become very overloaded and confusing. This proposed enhancement is about the additional metadata that is set as “tags” (name only) across various artifacts and OpenStack services.

Different APIs may use tags and key / value pairs differently. Tags typically are not used to drive runtime behavior. However, key / value pairs are often used by the system to potentially drive runtime, such as scheduling, quality of service, or driver behavior.

A few examples of metadata today:

Nova Cinder Glance
Flavor
  • extra specs
Host Aggregate
  • metadata
Instances
  • metadata
  • tags
Volume & Snapshot
  • image metadata
  • metadata
VolumeType
  • extra specs
  • qos specs
Image & Snapshot
  • properties
  • tags

Proposed change

We are proposing enhancements to the Metadata Definitions Catalog. The following subsections detail the enhancements to the catalog.

Tags

A catalog of possible tags that can be used to help ensure tag name consistency across users, resource types, and services. So, when a user goes to apply a tag on a resource, they will be able to either create new tags or choose from tags that have been used elsewhere in the system on different types of resources. For example, the same tag could be used for Images, Volumes, and Instances. Tags are not case sensitive (BigData is equivalent to bigdata but is different from Big-Data).

Alternatives

A key use case is the collaboration on tags using a common catalog. This is complementary to tags being added ad-hoc across all services. We think the metadata API could also be backed by a search indexer across services to include ad-hoc metadata as well as defined metadata. However, that is not the focus of this blueprint.

Data model impact

This will use a relational database and exist in the same database as the existing Glance Metadata Definitions Catalog. It will be additive to the existing schema.

The following DB schema is the initial suggested schema. We will improve and take comments during code review. Constraints not shown for readability.

Suggested Basic Schema:

CREATE TABLE `metadef_tags` (
  `id`                     int(11) NOT NULL AUTO_INCREMENT,
  `namespace_id`           int(11) NOT NULL,
  `name`                   varchar(80) NOT NULL,
  `created_at`             timestamp NOT NULL,
  `updated_at`             timestamp
)

This will not include Tag descriptions in this revision.

REST API impact

In the REST API everything is referred by namespace and name rather than synthetic IDs. This helps to achieve portability (import / export using JSON).

APIs should allow coarse grain and fine grain access to information in order to control data transfer bandwidth requirements.

Working with Namespaces Basic interaction is:

  1. Get list of namespaces with overview info based on the desired filters.
  2. Get tags

Common Response Codes

  • Create Success: 201 Created
  • Modify Success: 200 OK
  • Delete Success: 204 No Content
  • Failure: 400 Bad Request with details.
  • Forbidden: 403 Forbidden
  • Not found: 404 Not found if specific entity not found

API Version

All URLS will be under the v2 Glance API. If it is not explicitly specified assume /v2/<url>

Namespace may optionally contain the following in addition to basic fields.

  • resource_types
  • properties
  • objects
  • tags

This spec adds Tags.

Tags

List All Tags in a namespace:
GET /metadefs/namespace/{namespace}/tags

Filters by adding query parameters:

limit          = Use to request a specific page size. Expect a response
                 to a limited request to return between zero and limit items.
marker         = Specifies the name of the last-seen tag.
                 The typical pattern of limit and marker is to make an initial
                 limited request and then to use the last tag from the
                 response as the marker parameter in a subsequent limited
                 request.

Design note: We want a format that allows for additional information such as description to be added without changing the base response. For this reason, we used a dictionary for each tag rather than just a flat list of tags.

Example Body:

{
    "tags": [
        {
            "name": "Databases"
        },
        {
            "name": "BigData"
        },
        {
            "name": "MySQL",
        },
        {
            "name": "PostgreSQL",
        },
        {
            "name": "MongoDB",
        }
    ]
}
Create / Replace all tags in a specific namespace:
POST /metadefs/namespaces/{namespace}/tags/
Add tag in a specific namespace:
POST /metadefs/namespaces/{namespace}/tags/{tag}
Delete tag in specific namespace:
DELETE /metadefs/namespaces/{namespace}/tags/{tag}
Delete all tags in specific namespace:
DELETE /metadefs/namespaces/{namespace}/tags

Security impact

None

Notifications impact

None

Other end user impact

We intend to expose this via Horizon and are working on related blueprints.

Update python-glanceclient as needed.

Performance Impact

None anticipated.

This is expected to be called from Horizon when an admin wants to annotate tags onto things likes images and instances. This API would be hit for them to get available tags or create new ones.

Other deployer impact

DB Schema Creation for new API

Default / Sample tag libraries will be checked into Glance.

Deployers can customize these and provide additional definition files suitable to their cloud deployment.

glance-manage will include loading tags.

Developer impact

None (New API)

Implementation

Assignee(s)

Primary assignee:
wayne-okuma
Other contributors:
None

Reviewers

Core reviewer(s):
zhiyan
Other reviewer(s):
lakshmi-sampath travis-tripp

Work Items

Changes would be made to:

  1. The database API layer to add support for CRUD operations on tags
  2. The REST API for CRUD operations on the namespaces (add tags)
  3. The REST API for CRUD operations on the tags
  4. The python-glanceClient to support operations
  5. glance-manage to handle tags

Dependencies

Same dependencies as Glance.

Testing

Unit tests will be added for all possible code with a goal of being able to isolate functionality as much as possible.

Tempest tests will be added wherever possible.

Documentation Impact

Docs needed for new API extension and usage

References

Youtube summit recap of Graffiti Juno POC demo that included tags.

Current glance metadata definition catalog documentation.

Simple application category tags (no hierarchy)

Images, volumes, software applications can be assigned to a category. Similarly, a flavor or host aggregate could be “tagged” with supporting a category of application, such as “BigData” or “Encryption”. Using the matching of categories, flavors or host aggregates that support that category of application can be easily paired up.

Note: If a resource type doesn’t provide a “Tag” mechanism (only key value pairs), a blueprint should be added to support tags on that type of resource. In lieu of that, a key of “tags” with a comma separated list of tags as the value be set on the resource

Creative Commons Attribution 3.0 License

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.

glance-specs 0.0.1.dev331