Promote Ansible deploy interface to ironic¶
This spec presents a new deploy driver interface for provisioning nodes via Ansible playbooks.
Sometime it may be required to treat (some) baremetal nodes more like pets than cattle, providing per-node custom logic of deploy process.
Currently such customization in Ironic is not an easy task, as developer/cloud administrator would have to do one or more of the following:
Modify the code of Ironic deploy driver.
Modify the code of Ironic Python Agent.
Modify the code of ironic-lib.
Rebuild IPA ramdisk.
Upload new ramdisk to Glance or HTTP server(s).
Update the nodes’ info with new ramdisk.
Update deploy driver on Ironic conductors.
This problem can be partially solved in deploy driver based on external templates for a configuration management tool.
Possible use cases and advantages:
Custom actions with hardware at any stage of deploy process with vendor’s utilities, in-band or out-of-band.
Easy replace usage of one Linux utility with other for deploy.
Deep tuning of deploy process.
Changing behavior of deploy process without restart the conductors.
Long life time of deploy ramdisk - vendor’s utilities can be downloaded from external sources during deploy process.
Separation of concerns - ironic manages the ‘what’ part, Ansible the ‘how’ part
Allows BMs to be treated more like pets
This spec proposes such deploy interface based on Ansible 1 as configuration management solution. Effectively it uses ironic for its power, management and boot capabilities, and shifts the logic of deployment itself to Ansible playbooks.
Such deploy interface is already implemented
and is available as part of
ironic-staging-drivers project 2.
It can be used as possible deploy interface for
(and possibly other hardware as well).
Therefore this spec is proposing promotion of the
ansible deploy interface
for inclusion in the core ironic as one of available deploy interfaces.
Below is a short description of this deploy interface architecture
and current capabilities.
More information is available in the
documentation at 3.
This deploy interface is mostly suitable for ‘undercloud’-like or
standalone ironic usage, where the operator of ironic service is usually the
‘owner’ of the images to be deployed and/or the deployed instances themselves.
It does however already include set of Ansible playbooks that try to mimic the
capabilities of the standard
direct deploy interface of ironic
as close as possible.
This deploy interface is really useful when, during provisioning, there is a need to perform some low-level system configuration that would be hard to do in the already provisioned instance (like placing the root partition on a LVM/software RAID) or would require an extra reboot (like changing kernel parameters).
We chose Ansible for the following reason:
open source (GPLv3 + parts as MIT), mature and popular enough, including OpenStack ecosystem itself
written and extendable in Python, fits nicely in the OpenStack ecosystem
configuration files are human-readable YAML
agent-less by design, with minimal requirements on managed node, only Python and SSH server are required.
There are two ways to utilize Ansible from Python’s program:
use Ansible Python API or run CLI utility with parameters.
Ansible Python API currently does mandatory fork of caller’s process,
this behaviour is not suitable for Ironic conductor
(oslo.messaging does not allow forks at least).
Besides, Ansible licensing choice (GPLv3) prohibits usage of Ansible Python API
from ironic (Apache 2.0).
ansible-playbook CLI utility is used.
Each action (deploy, clean) is described by a single Ansible playbook with roles, which is run as a whole during deployment, or tag-wise during cleaning. Control of cleaning steps is through Ansible tags and auxiliary clean steps file. The playbooks for actions can be set per-node, as is cleaning steps file.
The dreploy interface tries to re-use as much code from ironic as possible,
and interface-wise is quite similar to the
direct deploy interface.
Currently this interface supports two modes for continuing deployment or cleaning:
having the deploy ramdisk calling back to ironic API’s heartbeat endpoint (default)
polling the node until the ssh port is open as part of a playbook
We propose to remove the latter from the interface when moving it to ironic, as support for it is not tested in gates, it decreases performance due to polling, and in general makes code more complicated.
A custom Ansible callback plugin is used for logging. It can read the logging configuration from ironic configuration file, and thus emit log entries for Ansible events directly to the logging backend ironic is also using (works best with journald backend).
The interface prepares a set of parameters which are needed for deploy,
including access info for the node to be deployed.
It then executes the
ansible-playbook script passing it all the collected
information, with node access info being used to register the node in Ansible
inventory at runtime.
Supported image types are whole-disk images and partition images with “local” boot option, “netboot” is currently not supported. Compressed images are downloaded to deploy ramdisk and converted to actual disk device/partition, RAW images are streamed to the target directly.
Creating a configdrive partition is supported for both whole disk and partition images, on both msdos and GPT labeled disks.
Root device hints are currently supported in their basic form only
(with exact matches, without oslo.utils operators), there are patches
to add full support on review.
If no root device hint is provided for the node, first device returned
as part of
ansible_devices Ansible fact is used as root device
to create partitions on or write the whole disk image to.
Cleaning procedure for Ansible deploy interface:
Each cleaning step is a tag in the Ansible playbook file used for cleaning. Available steps, their priorities and corresponding tags are defined in a auxiliary cleaning steps configuration file.
get_clean_steps()method returns a list of cleaning steps defined in mentioned configuration file.
prepare_cleaning()method loads the same ramdisk as for deploying.
execute_clean_step()method does synchronous execution of cleaning step via Ansible, executing only tasks with Ansible tags assigned to the cleaning step.
Default cleaning playbook provided with the interface supports
erase_devices clean steps of
deploy interface by executing shallow disk metadata cleanup
and shredding of disk devices respectively, honoring
priorities of those steps set in ironic’s configuration file.
Use a different deployment customization mechanism or don’t support the pet-like treatment.
The short rundown of main pros and cons of current
ansible deploy interface
compared to already available and standard
direct deploy interface:
easier to extend for custom provision logic
is not async
does not support netboot
Data model impact¶
State Machine Impact¶
REST API impact¶
Client (CLI) impact¶
RPC API impact¶
Driver API impact¶
Nova driver impact¶
To support this new deploy interface the deploy ramdisk should include:
a user with password-less sudo permissions - required
running SSH server configured for access to this user via SSH keys - required
Python interpreter (Python2 >= 2.6 or Python3 >=3.5)
currently tested with Python2 2.7 only
actual Python version required depends on Ansible version used and the version of Python interpreter that executes Ansible on ironic-conductor host
current Ansible support of Python3 on managed node is still considered experimental (and is not supported by Ansible 2.1.x at all)
a software component that upon boot of the ramdisk will make a
lookupAPI request to ironic API and make
default choice for such component is IPA
other system utilities used by deploy or clean playbooks
All or part of those (except the SSH server) can in principle be installed at runtime through additional Ansible tasks in playbooks (reducing the memory footprint and download/boot time of the deploy ramdisk), but also can be provided with deploy ramdisk to shorten the provisioning time.
Re-using IPA as
heartbeat ironic API client
makes it possible to have a unified deploy ramdisk for
ansible deploy interfaces.
ironic-staging-drivers project includes a set of scripts to perform
a simple rebuild of TinyCore Linux based deploy ramdisk (TinyIPA),
as well as
diskimage-builder element to build a suitable ramdisk with this
utility, basing it on
ironic-agent element and thus also containing IPA.
Those will be promoted to the new
Rebuild of CoreOS-based deploy ramdisk is currently not supported but such support can be added in the future.
Ansible communicates with remote machines over SSH. Deploy process is secure if private SSH key is properly secured (can be accessed only by the user running ironic-conductor service).
Other end user impact¶
The driver is not async and is blocking.
One conductor’s worker thread per node being provisioned or cleaned
Most of the time the thread waits in blocking state for the completion
Possible thread pool exhaustion must be accounted for when planning deployment
that allows usage of this deploy interface and configuring thread pool size
in ironic configuration file (
There are ideas on how to make the driver async / not blocking, those will be proposed in further specs/RFEs.
We have conducted some tests to measure performance impact of running
ansible-playbook processes on ironic-conductor host 5.
The results show that while there is indeed a performance overhead introduced
by using the
ansible deploy interface, it is well within possibilities
of quite standard hardware - we were able to provision 50 and 100 nodes
concurrently via single ironic-conductor service using this deploy interface,
with total provisioning time similar to
direct deploy interface.
Please see the blog post referenced above for mode details on test setup
Other deployer impact¶
these are defined in the
[ansible] section of ironic configuration file
None, 0-4. Corresponds to number of ‘v’s passed to
ansible-playbook. Default (None) will pass ‘vvvv’ when global debug is enabled in ironic, and nothing otherwise.
Full path to the
ansible-playbookscript. Useful mostly for testing environments when you e.g. run Ansible from source instead of installing it. Default (None) will search in
$PATHof the user running ironic-conductor service.
Path to folder that contains all the Ansible-related files (Ansible inventory, deployment/cleaning playbooks, roles etc). Default is to use the playbooks provided with the package from where it is installed.
Path to Ansible’s config file. When set to None will use global system default (usually
/etc/ansible/ansible.cfg). Default is
Extra arguments to pass to
ansible-playbookon each invocation. Default is None.
Name of the user to use for Ansible when connecting to the ramdisk over SSH. Default is ‘ansible’. It may be overridden by per-node
ansible_usernameoption in node’s
Absolute path to the private SSH key file to use by Ansible by default when connecting to the ramdisk over SSH. If none is provided (default), Ansible will use the default SSH keys configured for the user running ironic-conductor service. Also note that private keys with password must be pre-loaded into
ssh-agent. It may be overridden by per-node
ansible_key_fileoption in node’s
Path (relative to $playbooks_path or absolute) to the default playbook used for deployment. Default is ‘deploy.yaml’. It may be overridden by per-node
ansible_deploy_playbookoption in node’s
Path (relative to $playbooks_path or absolute) to the default playbook used for graceful in-band node shutdown. Default is ‘shutdown.yaml’. It may be overridden by per-node
ansible_shutdown_playbookoption in node’s
Path (relative to $playbooks_path or absolute) to the default playbook used for node cleaning. Default is ‘clean.yaml’. It may be overridden by per-node
ansible_clean_playbookoption in node’s
Path (relative to $playbooks_path or absolute) to the default auxiliary cleaning steps file used during the node cleaning. Default is ‘clean_steps.yaml’. It may be overridden by per-node
ansible_clean_steps_configoption in node’s
Memory (in MiB) consumed by the Ansible-related processes in the deploy ramdisk. Affects decision if the downloaded user image will fit into RAM of the node. Default is 10.
Number of times to retry getting power state to check if bare metal node has been powered off after a soft poweroff. Default is 6.
Amount of time (in seconds) to wait between polling power state after triggering soft poweroff. Default is 5.
The last 3 options are effectively copies of similar options in
section of configuration file.
We could use single option for (some of) those for all deploy interfaces
that make use of them,
especially if we rename/move them from
[agent] section to a section with
more general name (like
Per-node fields in driver_info¶
These parameters can be provided with driver_info, all are optional and their default values can be set in the ironic configuration file:
User name to use for Ansible to access the node (default is
Deprecated in favor of
Private SSH key used to access the node. If none is provided (default), Ansible will use the default SSH keys configured for the user running ironic-conductor service. Also note that private keys with password must be pre-loaded into
Deprecated in favor of
Name of the playbook file inside the
playbooks_pathfolder to use when deploying this node.
Name of the playbook file inside the
playbooks_pathfolder to use to gracefully shutdown the node in-band.
Name of the playbook file inside the
playbooks_pathfolder to use when cleaning the node.
Name of the YAML file inside the
playbooks_pathfolder that holds description of cleaning steps used by this node, and defines playbook tags in
ansible_clean_playbookfile corresponding to each cleaning step.
Developers may use this deploy interface for drivers.
- Primary assignee:
Pavlo Shchelokovskyy - pas-ha (IRC), pshchelo (Launchpad)
- Other contributors:
Yurii Zveryanskyy - yuriyz (IRC), yzveryanskyy (Launchpad)
ansibledeploy interface from
most changes would happen in imports of unit test modules
ansibledeploy interface entrypoint, add it to the list of supported deploy interfaces for
ipmihardware type, do not enable it by default in the configuration file.
ironic-staging-driversproject to the new
Install and use scripts from this new project in DevStack plugins and gate jobs
Amend ironic’s DevStack plugin to be able to set up nodes with this deploy interface.
Currently will require small rebuild of TinyIPA image during DevStack install.
Copy and modify the
gate-tempest-dsvm-ironic-staging-drivers-ansible-wholedisk-ubuntu-xenial-nvgate job, enable it in non-voting mode on ironic project.
Ansible has to be installed on the host running
This deploy interface was developed and tested with Ansible 2.1, and targets Ansible >= 2.1 (with some intermediate versions being excluded as incompatible). Currently the gate job testing this deploy interface passes with latest released Ansible version (18.104.22.168 as of this writing).
Also see the Ramdisk impact section for changes necessary to the deploy ramdisk.
Unit tests are already in place.
CI testing is already in place
as this is a vendor-agnostic deploy interface, it can be tested with virtual hardware in DevStack on upstream gates
gate-tempest-dsvm-ironic-staging-drivers-ansible-wholedisk-ubuntu-xenial-nvis already running in non-voting mode on all changes to
it would have to be copied and modified after appropriate changes are made to ironic’s DevStack plugin.
Upgrades and Backwards Compatibility¶
Documentation is already available and will have to be moved to ironic code tree as well.