Upload serialized deployment facts to ConfigDB service API

https://blueprints.launchpad.net/fuel/+spec/upload-deployment-facts-to-configdb

There are multiple levels of hierarchy in Hiera used by deployment tasks on nodes. Some of those data exist only in YaML files on a node and can’t be accessed by 3rd party components.

With configuration database service, we can store serialized deployment data for later use. It requires though that we can upload the data from nodes to the service as a part of deployment process.

We propose to develop a deployment task to upload the facts to external HTTP-based API and add it to a plugin that enables 3rd-party LCM application with Fuel [1].

Problem description

The store for serialized deployment information (e.g. ConfigDB API extension in Nailgun [2]) allows 3rd party applications to access it. It also allows alternative deployment/lifecycle management solutions to synchronize their configrations with Fuel installer.

However, it doesn’t solve the problem of getting the information into the service, since the extension itself is a more or less passive store.

The solution is needed to perform actual upload of required information into the configuration database. It also must keep the data up to date by synchronizing them upon every change applied to the environment.

Synchronization required in the following cases:

1. Deployment settings changed in Nailgun via UI/CLI/API. In this case, Nailgun DB will have the latest changes, and Nailgun API will respond with properly updated serialized deployment data [2]. This data can be imported into ConfigDB directly by requesting the Nailgun API and sending result to ConfigDB API. They are accessible by Hiera as astute data source [3].
2. Deployment data changed due to changes made to the node (e.g. hardware updated, versions of packages updated, etc) outside the Fuel context. These changes are reflected in the serialized data generated and stored on the node itself, in YaML files:

• /etc/hiera/globals.yaml - global configurations calculated by deployment task globals.
• /etc/hiera/override/plugins.yaml - plugin-specific overrides of parameters defined in data sources on higher levels (i.e. astute and globals).
• /etc/hiera/override/configuration.yaml - specific overrides for OpenStack configuration parameters which are not exposed by Nailgun directly.
• /etc/hiera/override/<hostname>.yaml - node-level configurations that override the basic parameters from other sources.

1. Deployment data changed in 3rd party deployment/lifecycle management application (e.g. in Puppet Master’s top-level manifests or in External Nodes Classifier application for the Puppet Master). Here we need to import data from the 3rd party application. This case is out of scope of the current proposal.

In this specification, we will focus on the use cases #1 and #2.

Proposed changes

The current deployment process implemented in Fuel installer assumes that all actual deployment data is available to Puppet agent on a target node locally as a set of YaML files in /etc/hiera directory.

By the time Puppet agent starts to execute actual deployment tasks, all the configuration settings must be up to date. It means that we can import the actual set of the deployment configuration data from those files as a part of deployment process.

The following changes are proposed in scope of this specification:

• Create a deployment task that uploads the serialized deployment data from files in /etc/hiera of a target node to the corresponding resources in ConfigDB API endpoint (e.g. /environment/<:id>/node/<:id>/resource/<:datasource>/values). See Orchestration section for details.
• Integrate the a task into the deployment graph using plugins mechanisms of Fuel. The task must run in the end of the deployment process to make sure that all the tasks that impact the deployment settings files are already executed, including:
  • globals
  • override_configuration
  • any plugin task that updates/overrides basic deployment settings
• Implement a pass of auth information for ConfigDB API extension of Nailgun API to the deployment task in question in a secure way.

Deployment task details

The deployment task shall have the following ID:

upload_data_to_configdb

The task shall depend on the following tasks:

globals
upload_configuration
pre_deployment_end

The task shall run at the Fuel Master node. Auth credentials for the ConfigDB API shall be made available for it upon LCM plugin installation from the following file:

/var/www/nailgun/plugins/<lcm-plugin-name>/environment_config.yaml

That file will be created by the plugin builder and will contain metadata of the LCM application, including auth credentials in question.

The task shall perform the following operations using ConfigDB API:

• Verify if the environment env_id exists in the ConfigDB API.
  • If not, create an environment with POST request. It should contain a list of data sources to create for the environment. See details in ConfigDB API specification [2].
  • The list of data sources is fetched from astute.yaml file, from parameter data_sources of Puppet Master LCM plugin’s metadata.
• Read data from files in /etc/hiera directory via mcollective client into internal variable of the deployment task.
• Upload data to ConfigDB API’s data sources based on the filenames from which the data was read.
• Read the effective data from ConfigDB API (i.e. values calculated by merging uploaded data with configured overrides at all levels or hierarchy) and write the data to files in /etc/hiera directory via mcollective client.

The latter function ensures that the deployment process accounts for data overriden in ConfigDB. This is required for 2nd day operations based on the Fuel workflow, specifically for scale-out opertaions (adding a node to the cluster) and replacement of failed nodes.

Auth mechanism details

ConfigDB API extension as a part of Nailgun API uses Keystone to verify and authorize users.

Before installing an environment with the 3rd-party LCM plugin, user must add a service account for the plugin in that environment with Keystone CLI. For example:

$ keystone user-create --name=lcm-plugin --tenant=admin

While configuring environment, the user enables the LCM plugin and configures the service account credentials for the plugin via Fuel UI or API.

Plugin uses these access creates credentials to configure deployment task upload_data_to_configdb and the custom Hiera backend.

Example workflow

The following example illustrates the workflow of the solution:

• Assume that the User intends to use 3rd-party application to perform some tasks, for example, LCM operations, on an OpenStack environment deployed by Fuel.

  • User installs the Fuel Master node with the ConfigDB extension. The extension is installed as an RPM on top of the existing system.
  • User installs a plugin for LCM operations that should include components to upload deployment data to ConfigDB API (e.g. deployment task upload_data_to_configdb) and to perform lookup for certain parameters in ConfigDB API (e.g. custom Hiera backend).

• User configures OpenStack environment using Fuel UI. Nailgun creates metadata for the environment and individual nodes.

  • The deployment data for the environment and nodes is accessible via Nailgun API by URIs /cluster/<:id>/orchestrator/deployment and /cluster/<:id>/orchestrator/deployment/default.
  • Deployment data for specific node are exposed via the same URIs with addition of parameter ?node=<:node_id> to the URI path.

• User deploys the environment as usual via Fuel UI or CLI.

  • Deployment task upload_data_to_configdb runs on every node in the environment and uploads serialized deployment data from YaML files in /etc/hiera/ directory to ConfigDB API.
  • Another deployment task configures the node to work with 3rd party LCM tool. This might or might not include disable of the ordinary Fuel means of deployment.

• Afterwards the User makes changes to the environment configuration using 3rd-party LCM tool.

  • User changes or extends the deployment settings by assigning values to parameters via ConfigDB API, for example, changes keystone_url parameter in globals data source.
  • ConfigDB saves the override data to an override for the data source globals.
  • User triggers 3rd party application which reads the effective data (i.e. raw uploaded values with applied overrides) from ConfigDB API and applies changes to all affected nodes.

• In future, the User adds another node to the environment and deploys it using standard Fuel methods.

  • Deployment data for the new node provided by Nailgun’s standard serializers.
  • When the deployment is initiated, the task upload_data_to_configdb synchronizes astute.yaml file created at the node by Astute with data overrides created in ConfigDB API: it uploads the contents of the file to ConfigDB API and then downloads effective data for the asuste data source from there.
  • After the pre-deployment finishes, the prepared deployment data are synchronized to the ConfigDB API by task upload_data_to_configdb. This ensures that override settings from all data sources in the ConfigDB API are applied to all files in /etc/hiera at the node.
  • Deployment is done by Fuel standard deployment tasks, but with settings adjusted with overrides from ConfigDB.
  • After successful deployment, the LCM plugin reconfigures the node to work with 3rd-party LCM tool.

Web UI

None.

Nailgun

None.

Data model

None.

REST API

None.

Orchestration

A new deployment task shall be added to ensure that all changes to files in /etc/hiera directory are synchronized with the ConfigDB.

The task shall send a series of requests to the URI of the resource in ConfigDB based on the parameters of the deployment:

<:service_uri>/environments/<:env_id>/nodes/<:node_id>/resources/<:datasource>/values

• service_uri is a endpoint from Keystone Service Catalog, defaults to /api/v1/config.
• env_id is an identifier of cluster the node belongs to. The ID of environment shall be fetched from deployment fact deployment_id.
• node_id is an identifier of the node. It should be matched in Nailgun API by the node’s FQDN scope recieved from Puppet Master.
• datasource is a name of the data source.

The task will: #. upload data to /values of the data source’s resource and

1. download effective data from /values?effective and write it to files.

See detailed description of the API in corresponding specification. [2]

RPC Protocol

None.

Fuel Client

None.

Fuel Library

None.

Alternatives

The alternative way to keep deployment data from nodes in sync with ConfigDB is to upload data to API from deployment tasks.

While it is possible to adjust globals and openstack_config tasks to upload configuration data to external service, it is generally impossible to do with all supported plugins.

A plugin can override default values in astute.yaml generated by the Nailgun-provided serialized data. However, this overrides are configured by plugin tasks on a per-node basis. Override information is not available to Nailgun or even Astute directly. So, to ensure sync of plugins’ override data we need to modify each and every plugin, which apparently is not an option.

Another way to keep data in sync is to upload it from some bottom-level catch-all Astute post-deployment task. This would allow to keep Nailgun/ConfigDB credentials limited to the Master node and not expose them to target nodes in the deployment.

On the other hand, there was a work done on Astute to convert its tasks into standard deployment tasks in fuel-library. Thus, we should net add new tasks to Astute in this proposal.

Upgrade impact

None.

Security impact

Sensitive configuration data, such as passwords and access credentials, shall be uploaded to the ConfigDB API using proposed functions. It is recommended to use encrypted HTTP protocol to transfer these data.

The approach to authentication of the plugin’s application with Nailgun API assumes that the user is responsible for configuring access credentials for the plugin applications in Keystone. The user is also responsible for configuring proper credentials for the plugin when installing an environment with 3rd-party LCM application.

Notifications impact

None.

End user impact

None.

Performance impact

The deployment task proposed in this spec will take some time to upload all data to the ConfigDB API. Moreover, if many nodes trying to write to the same API endpoint at the same time, it might significantly affect the overall duration of deployment.

Deployment impact

None.

Developer impact

None.

Infrastructure impact

The deployment task is packaged as a part of 3rd-party LCM plugin.

Documentation impact

None.

Implementation

Assignee(s)

Primary assignee:

<gelbuhos> Oleg S. Gelbukh

Other contributors:

<sryabin> Sergey Ryabin

Mandatory design review:

<rustyrobot> Evgeniy Li <ikalnitsky> Igor Kalnitsky <vsharshov> Vladimir Sharshov <vkuklin> Vladimir Kuklin

Work Items

• Develop deployment task as a part of Puppet Master LCM plugin code base [1].
• Develop unit tests for the deployment task in the plugin’s code base.
• Develop automated integration tests for the plugin in openstack/fuel-qa repository.

Dependencies

1. ConfigDB API implementation as Nailgun extension [2]

Testing, QA

• The feature shall be tested in conjunction with ConfigDB API feature [2]
• Tests shall verify that contents of data sources are consistent with contents of files in /etc/hiera at nodes after the deployment finishes.

Acceptance criteria

• Deployment data from nodes uploaded to corresponding data sources in ConfigDB API upon successful deployment of the OpenStack environment.

References

[1]	(1, 2) Puppet Master LCM plugin specification TBD

[2]	(1, 2, 3, 4, 5, 6) Nailgun API extension for serialized deployment facts https://review.openstack.org/#/c/284109/

[3]	Nailgun API for Deployment Information https://github.com/openstack/fuel-web/blob/master/nailgun/nailgun/api/v1/handlers/orchestrator.py#L190