Metadata Assignment And Propagation¶
https://blueprints.launchpad.net/murano/+spec/metadata-assignment-and-propagation
Cloud users and operators may need to assign some arbitrary key-value pairs to environments and applications deployed by murano to associate some metadata information with them. If applicable this metadata attributes should be propagated to OpenStack resources provisioned by Murano.
Problem description¶
The meta information may describe various aspects of the deployed applications, for example:
Deployed software: names, versions, descriptions and license terms of the software components deployed on the VMs;
VM-related configuration features, like listening ports or interfaces;
Hardware specs of provisioned virtual machines;
Guest OS info, like the type of operating system, its family, version etc;
Information about the deployments purpose: type of the deployed environment (development, staging, production), usage purpose, security consideration etc;
Information about the user who deployed the app or environment: name, department, contact information, etc.
This information may be consumed directly by inspecting murano applications with dashboard or CLI, by indexing these metadata attributes into specialized search software, such as Searchlight [1] or by building custom reports with grouping or aggregation by these metadata.
For the latter case it is especially important to propagate the metadata from the initial objects it is assigned to (application or environment) to the resource objects spawned by them (VMs, Volumes, Networks etc).
Proposed change¶
UI change¶
It is proposed to utilize the functionality provided by Glance Meta Definition
Catalog [2] to define the schemas of the applicable metadata attributes. It is
proposed to add two types of resources for murano applications and murano
environments: OS::Murano::Application
and OS::Murano::Environment
accordingly (ability to add metadata attributes for murano packages is out of
scope for this spec, however in future this may also become possible, so
OS::Murano::Package
is likely to be appropriate resource type).
The standard Angular-based Metadata assignment dialog (the one being currently used to assign metadata to Instances, Images and Volumes) should be extended to support Murano Dashboard and these new resource types. This extended dialog will load the metadata definition schemas for attributes which are assignable to appropriate type of resources and will submit them to Murano API for persistence once the input is saved.
Murano Object Model change¶
For the applications it is proposed to persist the metadata attributes in the object model, in the new block called ‘metadata’, which should be placed under the ‘?’ section of each object. For example, the ‘?’ block of Apache application tagged with a couple of attributes may look like this:
"?": {
"_26411a1861294160833743e45d0eaad9": {
"name": "Apache HTTP Server"
},
"type": "com.example.apache.ApacheHttpServer",
"id": "19dfb844-6eee-48bd-826a-c6206d065e85",
"metadata": {
"sw_webserver_apache_version": "2.1",
"sw_webserver_apache_http_port": 80
}
}
Since the ?-headers of objects placed in the environment’s object model may be modified with the existing Murano API no changes in the API is needed.
For the Environments a similar approach may be followed — to store the metadata attributes under the /?/metadata key of the Environment object in the object model. However an alternative may be considered: since each Environments have a dedicated entry in murano’s database, the metadata attributes for environments may be stored in database in normalized form. See the “Alternatives” section below for more details.
Regardless of the chosen approach it is required to introduce a new API call to assign metadata attributes to the Environment, since the existing API does not have capabilities to edit the ?-headers of Environment object. The appropriate change into API is out of scope here and is covered with a separate spec [3]
MuranoPL Change¶
MuranoPL language should be extended with an ability to read the metadata
attributes assigned to any given object. It is proposed to create a global yaql
function named metadata()
for this purpose (similar to id()
and
name()
which read appropriate information from the same ?-header).
Murano’s serializer/deserializer should be properly updated to load the metadata attributes and persist them back.
Core Library Change¶
Core library should be extended with an ability to communicate with Glance
Metadata Definition API to query the namespaces applicable to a given resource
type and extract all the names of attributes from those namespaces. This will
allow the resource class to check if the given metadata attribute attached to
the application the class belongs to is applicable for the resource itself. The
binding to the Glance API should be done via a python-backed class
MetadefBrowser
, while the rest of the logic may be implemented in MuranoPL.
A new MuranoPL class called MetadataAware
should be implemented. This class
will have two methods: collectMetadata
and getResourceType
.
collectMetadata
method will return all the metadata attributes assigned to
the object itself and all the attributes assigned to the owners of the object,
traversing the ownership tree to the root (i.e. to the Environment). All the
attributes from the traversed objects may be checked for applicability to the
particular resource type. To do this check, the code should call the
getResourceType
method which is supposed to return the resource type as
defined in Glance (OS::Nova::Server
for VMs, OS::Cinder::Volume
for
volumes etc) and then use the MetadefBrowser
described above to check if
the attribute is applicable to the objects of this type.
The resource-specific classes of the Core Library (Instance
,
CinderVolume
etc) have to extend the MetadataAware
and override the
getResourceType
method.
Alternatives¶
There are a number of alternatives to consider.
First, instead of using Glance Metadef catalog to store the possible attribute names and constraints, we may assign attributes without any predefined schema. This will reduce our dependency on other OpenStack components (Glance in this case), but will introduce extra development burden (since we won’t be able to reuse the existing metadata-assignment dialog) and will prevent us from using convenient schema-based validators of tags which are made for metadefs. So, it is proposed to stay with metadef-compliant implementation.
Second option to consider is the storage of metadata attributes on environment level. Most other OpenStack projects store metadata attributes in normalized form: they have the separate table mapped to the primary object table as one-to-many. The approach suggested above uses denormalized storage: the attributes are stored inside the json-document describing the complete environment, which is stored as a single value in the database. This is much easier to implement, also having metadata in object model of environment object is still needed, so duplicating data storage seems like an extra work.
Third option to consider: not to check for attribute applicability on resource
types. This simplifies the development (no need to create MetadefBrowser
class and much simpler logic in MetadataAware
), however this will degrade
user experience, as resources may get inappropriate attributes assigned, which
will be confusing.
Data model impact¶
This proposal adds new fields to the ?-headers of objects in object model. If
the normalized alternative is chosen, it will also introduce extra table in
database and a relationship with the Environments
table. This will require
to create a database migration as well.
REST API impact¶
A modification of Murano API is required to allow assigning of metadata attributes to the Environment objects, however it is out of scope here. See [3] for details.
Versioning impact¶
Since the proposed change requires to add a new yaql function to the DSL and modifies the serializer/deserializer logic this requires to bump the MuranoPL format version.
Other end user impact¶
The users will see new action buttons “Update Metadata” in Murano Dashboard. This buttons will be available as row actions for Environment Components and Environments.
Deployer impact¶
The proposed change introduces a dependency on Glance API. To configure the connectivity to this API (endpoint details, encryption etc) a new optional configuration block should be added to Murano config.
Developer impact¶
Developers may implement the MetadataAware
class to add metadata handling
logic for their classes.
Murano-dashboard / Horizon impact¶
This change will add a custom version of metadata.service javascript module to Murano Dashboard. It will have the similar logic as the standard Horizon’s module, but will add the references to murano api’s, so the Metadata dialog can update the attributes assigned to Murano entities.
Implementation¶
Assignee(s)¶
- Primary assignee:
ativelkov
- Other contributors:
TBD
Work Items¶
Add a rest api handlers for murano-dashboard to handle metadata updates and fetching;
Add a javascript API to this handlers;
Create a custom version of metadata.service, so the metadata modal dialog may utilize murano’s javascript APIs to metadata;
Modify Murano Dashboard to call metadata modal dialog as row handlers for Environments and Services;
Modify Murano API to allow edits of ?-section of Environment objects;
Add a
MetadefBrowser
class to Core Library;Implement a
MetadataAware
class and extendInstance
andCinderVolume
classes with it.
Dependencies¶
Glance Metadefinition Catalog [2]
Environment modification API changes [3]
Testing¶
The integration tests should verify that the attributes are propagated properly
Documentation Impact¶
The new capabilities to add attributes should be documented in user manual.
The changes in core library and the usage of MetadataAware
class should be
reflected in developers’ manual.
References¶
[1] https://wiki.openstack.org/wiki/Searchlight [2] http://docs.openstack.org/developer/glance/glancemetadefcatalogapi.html [3] https://review.opendev.org/#/c/378602