Cinder/Neutron plugins in fuel

https://blueprints.launchpad.net/fuel/+spec/cinder-neutron-plugins-in-fuel

Cloud operators want to extend and change behavior of Fuel in order to do that, Fuel should provide plugin mechanism.

Problem description

Sometimes Fuel user wants to extend Fuel to install Cinder/Neutron plugin. Right now user changes the code of Fuel services rebuilds and adds repositories manually.

The current approach causes a lot of problems:

  • user has to support all the patches
  • also he has to apply all the patches manually after Fuel upgrade

Proposed change

List of terms

  • plugin - archive which contains all required data, like repositories for ubuntu, centos, metadata file with description of plugin, scripts, etc
  • fpb or fuel-plugin-builder - is fuel plugin builder, command line tool which helps user to develop plugin, the code will be in fuel-web repository

Requirements

  • user should be able to install simple Cinder/Neutron plugins, “simple” means that plugin doesn’t require additional business logic, user can configure only static data for settings tab
  • plugin developer in the most cases should know nothing about python/js/css/html
  • plugin developer should have easy way to test his plugin (he shouldn’t reinstall his master node again and again to test his plugin)
  • plugin should be environment specific, it means that user should be able to enable or disable plugins for specific environment, plugin should be disabled for deployed environments without possibility to enable it

Plugins constraints

For the current release we have the next constraints:

  • plugin cannot change business logic and should not contain any python code
  • plugin can provide additional attributes for environment, it cannot remove or change existing information which we provide for orchestrator
  • plugin cannot add new kernel
  • plugin cannot change provisioning data
  • user will not be able to enable plugin on deployed environment
  • plugin cannot change or add new bootstrap image
  • plugin cannot be uninstalled in the first feature release
  • plugin cannot change existing database schema
  • plugins will work on openstack releases after 6.0 Fuel release, plugins won’t work on 5.X openstack releases. This constraint is related to changes in MCollective plugins

Plugins examples

Plugin development process

Plugin developer will be able to develop plugin on his machine, we will specify all requirements for environment, like version of OS and additional dependencies.

  • plugin developer installs all of the dependencies which are mentioned in development document to prepare his env, like python, rpm, createrepo, dpkg-dev
  • plugin developer installs fpb command line tool pip install fuel-plugin-builder
  • plugin developer runs fpb –create plugin-name
  • fpb creates new directory plugin-name, where he can see the a basic structure of the plugin with in place documentation
  • plugin developer adds his packages with all required dependencies for ubuntu and centos
  • sets the metadata, like version of the plugin, its description, and versions of openstack releases
  • then he runs fpb –build plugin-name from the plugin directory, fpb checks, that all required fields are valid and all required files are there, builds the repositories and generates tar-ball

Note, fpb should provide –debug key to turn on debug information.

Checks and validation

fpb should perform checks during the plugin build

  • check that metadata is correct
  • deploymetn_scripts_path is exists for each release
  • repositories_path is exists for each release
  • tasks.yaml, check the structure

Plugin installation process

From user point of view:

  • user downloads a fuel plugin
  • runs fuel plugins –install fuel-plugin-name-1.0.0.fp, and plugin is installed

Install script workflow:

  • check if current fuel version is compatible with the plugin
  • copy all the files in /var/www/plugins/plugin_name-plugin_version
  • via rest api create plugin in nailgun

Plugin archive structure

This structure should be generated by fpb script.

.
|-- deployment_scripts
|   `-- deploy.sh
|-- environment_config.yaml
|-- LICENSE
|-- metadata.yaml
|-- pre_build_hook
|-- README.md
|-- repositories
|   |-- centos
|   |   `-- .gitkeep
|   `-- ubuntu
|       `-- .gitkeep
`-- tasks.yaml

Here is detailed description of some of the files:

metadata.yaml file

# Plugin name
name: fuel_awesome_plugin
# Human-readable name for your plugin
title: Awesome plugin
# Plugin version
version: 1.0.0
# Description
description: Enable to use plugin X for Neutron
# Required fuel version
fuel_version: ['6.0']

# The plugin is compatible with releases in the list
releases:
  - os: ubuntu
    version: 2014.2-6.0
    # User can specify if his plugin is ha compatible or not
    mode: ['ha', 'multinode']
    deployment_scripts_path: deployment_scripts/
    repository_path: repositories/ubuntu
  - os: centos
    version: 2014.2-6.0
    mode: ['ha', 'multinode']
    deployment_scripts_path: deployment_scripts/
    repository_path: repositories/centos
    # If plugin can work with several openstack releases
    # user can define different directories with packages
    # and deployment scripts, at the same time he can specify
    # the same directory for all of the versions, it depends
    # on plugin implementation
  - os: centos
    version: 2014.2-7.0
    mode: ['multinode']
    deployment_scripts_path: 7.0/deployment_scripts/
    repository_path: 7.0/repositories/centos

# Version of package format
package_version: '1.0.0'

environment_config.yaml

attributes:
  fuel_simple_port:
    value: 2333
    label: 'Port'
    description: 'Port which be used for service binding'
    weight: 25
    type: "text"

  fuel_simple_host:
    value: 0.0.0.0
    label: 'Host'
    description: 'Host which be used for service binding'
    weight: 10
    type: "text"

tasks format description

# Roles which the task should be applied on
- role: ['controller', 'cinder']
  stage: pre_deployment
  type: shell
  parameters:
    cmd: configure_glusterfs.sh
    timeout: 42
# Task is applied for all roles
- role: "*"
  stage: post_deployment
  type: puppet
  parameters:
    puppet_manifest: cinder_glusterfs.pp
    puppet_modules: modules
    timeout: 42

Directories structure on the master node

Directory /var/www/plugins which contains all of the plugins, should be mounted to the next containers.

  • rsync - for puppet manifests
  • nailgun - to extend nailgun
  • nginx - is required for repositories

Plugins upgrade

User wants to be able to upgrade his plugin, if there will be some new plugin with updated version of package or other bug fixes. For the current version we don’t provide any upgrade mechanism for plugins. In theory we could use this mechanism if openstack patching feature was not experimental.

Alternatives

There are a lot of alternatives, the best of them are described in Future improvements section and will be implemented later.

Future improvements (not for 6.0)

Plugin manager

Separate services which keeps information about all of the plugins in the system, it should know how to install or delete plugins. We will use this service instead of install script to install the plugins.

Plugins which change business logic

Nailgun drivers and hooks which will provide a way to change deployment/provisioning data for orchestrator. Also it will be possible to add new role.

UI plugins

Add new step in wizard, add new tab, for cluster env, add new settings window for node configuration.

Plugins which implement separate service

User will be able to install any service on the master node, the good example of such kind of plugins is OSTF.

Users requirements for Fuel plugins

General use cases:

  • ability to execute custom puppet code during deployment state (ideally on any stage not only as a post deployment step)
  • ability to execute custom python code in Nailgun
    • Define custom roles and node priorities
    • Provisioning serialization
    • Deployment serialization
    • Post deployment orchestration
  • ability to execute custom java script code
  • ability to modify UI
  • ability to add custom deb/rpm packages
  • ability to change and extend node specific parameters

More specific use cases:

  • Swift standalone installation: custom roles, priorities, UI changes
  • Add neutron plugin: custom puppet modules, UI changes
  • Custom monitoring schema: UI, priorities, puppet
  • Custom Cinder driver: UI, puppet
  • Cinder multibackend: UI, puppet
  • Add package that require reboot: provisioning customization

Plugins distribution and management

  • user should be able to define dependencies between plugins, it means that one plugin can require another to be installed
  • user should be able to define conflicts between plugins, it means that particular plugin cannot be installed on the same master node with another plugin
  • plugin system should be able recursively retrieve all of the dependency and check that all of the subplugins are compatibele with each other and with the current version of master node
  • plugins update

Nodes management hooks

  • post_node_deletion - execute after node is deleted
  • pre_node_deletion - execute before node is deleted

fpb command line interface

  • before build check that packages dependencies are compatibele with openstack releases dependencies, in order to do that, fpb should have access to all of the repositories

Data model impact

There will be new model in nailgun, Plugins with many to many relation to Cluster model. Model for many to many relation ClustersPlugins will be used in order to disable or enable plugin for specific environment.

Plugins

  • id - unique identificator
  • name - plugin name
  • version - plugin version
  • description - plugin description
  • fuel_version - requires specified fuel version
  • openstack_releases - is a list of strings with releases

ClustersPlugins

  • id - record id
  • plugins.id - plugin id
  • clusters.id - cluster id

REST API impact

GET /api/v1/plugins/

Returns the list of plugins

[
    {
        "id": 1,
        "name": "plugin_name",
        "version": "1.0",
        "description": "Enable to add X plugin to Neutron",
        "title": "Plugin name",
        "fuel_version": ["6.0"],
        "package_version": "1",
        "releases": [
            {
                "os": "ubuntu",
                "version": "2014.2-6.0"
            },
            {
                "os": "centos",
                "version": "2014.2-6.0"
            }
        ]
    }
]

POST /api/v1/plugins/

{
    "id": 1,
    "name": "plugin_name",
    "version": "1.0",
    "description": "Enable to add X plugin to Neutron",
    "title": "Plugin name",
    "fuel_version": ["6.0"],
    "package_version": "1",
    "releases": [
        {
            "os": "ubuntu",
            "version": "2014.2-6.0"
        },
        {
            "os": "centos",
            "version": "2014.2-6.0"
        }
    ]
}

GET /api/v1/plugins/1/

Get the information about specific plugin, where 1 is id of the plugin

{
    "id": 1,
    "name": "plugin_name",
    "version": "1.0",
    "description": "Enable to add X plugin to Neutron",
    "title": "Plugin name",
    "fuel_version": ["6.0"],
    "package_version": "1",
    "releases": [
        {
            "os": "ubuntu",
            "version": "2014.2-6.0"
        },
        {
            "os": "centos",
            "version": "2014.2-6.0"
        }
    ]
}

PATCH /api/v1/plugins/1/

Update specified attributes for plugin

Accepts the same format as response from GET request.

PUT /api/v1/plugins/1/

Update all of the attributes

Accepts the same format as response from GET request.

DELETE /api/v1/plugins/1/

Remove a plugin from DB, should have validation which returns the error, if plugin is used by some environment.

Validation should be disabled if plugin deletion is performed with force parameter in url. It will be required for development.

Orchestration (astute) RPC format

As it was described above, user specifies the structure like this

- role: ['controller', 'cinder']
  stage: pre_deployment
  type: shell
  parameters:
    cmd: configure_glusterfs.sh
    timeout: 42
- role: *
  stage: post_deployment
  type: puppet
  parameters:
    puppet_manifest: cinder_glusterfs.pp
    puppet_modules: modules
    timeout: 42

Then nailgun configures this data in the next format

# This stages should be run after astute yaml for role
# and repositories are on the slaves
pre_deployment:
  # Add new repo
  - # This task will be autogenerated by nailgun
    type: upload_file
    uids: [1, 2, 3]
    priority: 0
    parameters:
      path: /etc/apt/sources.list.d/plugin_name-1.0
      data: the file data
      # Overwrite already existed file?
      overwrite: true
      # Create intermediate directories as required
      parents: true
      # File permission
      permissions: '0644'
      # User owner
      user_owner: 'root'
      # Group owner
      group_owner: 'root'
      # What permissions should be set for folder
      dir_permissions: '0644'
  - # This task will be autogenerated by nailgun
    type: sync
    uids: [1, 2, 3]
    priority: 1
    parameters:
      src: rsync:///var/www/nailgun/plugins/plugin_name-1.0/scripts
      dst: /etc/fuel/plugins/plugin_name-1.0/scripts
  - type: shell
    uids: [1, 2, 3]
    priority: 10
    parameters:
      cmd: configure_glusterfs.sh
      timeout: 42
      # This parameter should be autogenerated by nailgun
      cwd: /etc/fuel/plugins/plugin_name-1.0
post_deployment:
  - type: puppet
    uids: [1, 2, 3, 4, 5, 6]
    priority: 20
    parameters:
      puppet_manifest: cinder_glusterfs.pp
      puppet_modules: modules
      timeout: 42
      # This parameter should be autogenerated by nailgun
      cwd: /etc/fuel/plugins/plugin_name-1.0
deployment_info:
  # Here is deployment information in the same format
  # as it is now

In the first release orchestrator should fail deployment if one of the tasks is not executed successfully.

Deployment scripts

Plugin developer can use any bash scripts or puppet manifests in order to perform plugin installation, here is a list of requirements for the scripts

  • if user wants the script to be executed it should has right permission and executable flag
  • if user uses puppet for plugins installation he should provide puppet manifests and modules in his plugin
  • scripts should not break anything if they were run several times

Nailgun implementation

Nailgun should provide ability to mix new environment attributes which are required for plugin configuration

Also nailgun should extend default deployment/patching tasks with tasks for pre and post deployment hooks, where should be specified paths to scripts directory on the master node

For each plugin nailgun generates separate section with checkbox to show it on UI.

UI implementation

It is not required to add new logic on UI tab, nailgun generates checkbox for each plugin on settings tab, so user can enable or disable particular plugin and configure it.

Upgrade impact

Current release

Because we don’t have any python code in our plugins, plugin will depend on openstack release, we don’t delete releases, as result it’s not necessary to check if plugin is compatible with the current version of fuel. Also plugin is stored on shared volume which we mount to nailgun container.

Future releases

When we get plugins with python code, in upgrade script we will have to check if plugins are compatible with the new version of fuel, if they aren’t compatible, upgrade script should show the message with the list of incompatible plugins and it should fail the upgrade. If user wants to perform upgrade, he should provide the directory with new plugins, which will be updated during the upgrade, or user should delete plugins which he doesn’t use.

Security impact

This feature has a huge security impact because the user will be able to execute any command on slave nodes. Security is included in acceptance criteria of plugins certification, see Plugins certification section.

Notifications impact

Installation script will create notification after plugin is installed.

Other end user impact

User should be able to disable or enable plugin for specific environment.

Performance Impact

Deployment

  • there will not be any impacts if user doesn’t have enabled plugins
  • if user has enabled plugins for environment, there will be performance impact, the time of deployment will be increased, the increasing time depends on the way how plugin is written

Nailgun

  • we assume that there will not be any notable performance impact, in hooks we will have to enable merging of custom attributes in case if plugin is enabled for environment, the list of the plugins can be gotten within a single database query

Also performance is added as acceptance criteria for core plugins, see Plugins certification section.

Other deployer impact

Plugin developer will be able to execute pre/post deployment hooks for the environment.

Changes which are required in astute:

  • add several repositories (should be ready, testing is required)
  • add posibility to rsync specific directories from master to slave
  • add hooks execution before and after puppet run

Plugins certification

The topic isn’t covered by this document, separate document needs to be created.

Items which should be reviewed during plugin certification:

  • Security review
  • Performance review
  • Compatibility with other plugins in core
  • Plugins upgrade
  • Check that plugin works fine in case of openstack patching

After plugin is certified user should be able to add plugin in our plugins repository.

Cerified plugin code repository

User should not follow fuel’s workflow in development, as result they can have their own repositories with code

Cerified plugin repository

We should provide repository with built plugins where user will be able to download plugin.

Core plugins

Core plugin is a plugin which is developed and supported by fuel team. They can or cannot be included in an iso. Build system should has config with a list of built-in plugins.

Fuel CI

  • generate plugins on each patch to fuel-plugins repository
  • generate plugins after patch is merged to the master
  • run system tests with master’s plugins

Developer impact

Features design impacts:

  • any new feature should be considered to be a plugin
  • features should be designed to be extendable

Development impacts:

  • we should try not to break compatibility with plugins, it should be very easy for plugins developer to make migration from previous version of Fuel to new one

Implementation

Assignee(s)

Primary assignee:

Other contributors:

Work Items

  • Plugin creation tools - creates plugin skeleton, builds the plugin, also it should provide installation script
  • Nailgun - should provide ability to enable/disable plugins for specific environments, also it should read plugin’s attributes and merge them on the fly
  • Nailgun/Orchestrator - nailgun should provide post/pre deploy tasks for orchestrator, orchestrator should provide post/pre deploy hooks
  • UI - ability to enable/disable plugin for specific environment
  • Fuel CLI - list/enable/disable/configure plugins for environment

Dependencies

Nailgun dependencies:

  • SQLAlchemy==0.9.4

Testing

There will be several core plugins, which QA team will be able to install and test.

For neutron it will be LBaaS plugin, for Cinder it will be GlusterFS backend.

Also it will be required to have infrastructure, where plugin developer will be able to test his plugins. He should have ability to specify plugin url and the set of plugins, which he would like to run tests with.

Also we can have core plugins, which should be included in our testing cycle, it means that we should run system tests with plugins, and also run plugins specific tests.

Documentation Impact

  • how to create a plugin
  • how to test a plugin
  • how to debug a plugin
  • how to add a plugin in core repository and how to perform testing
  • documentation for plugin user, where will be the information where to take a plugin
  • how to install a plugin