Add a virt driver for Ironic

https://blueprints.launchpad.net/nova/+spec/add-ironic-driver

This specification proposes to add a virt driver to enable Nova to deploy images to bare metal resources by using the OpenStack Bare Metal Provisioning Service (“Ironic”).

Problem description

The community has split out the functionality of provisioning bare metal servers into a separate program, which includes the ironic and python-ironicclient projects. The original intent of the nova.virt.baremetal driver was two-fold:

  • to provide physical machines more suitable to HPC-style workloads, eg where virtualization overhead is too high;

  • to be an experimental proof-of-concept for enabling the TripleO project.

In order to address scalability and architectural concerns affecting both use-cases, this driver was split out into a separate OpenStack Program, and developed over the last year as such.

Proposed change

This proposal aims to enable Nova to use Ironic to perform the same functions which it is currently able to perform via the nova.virt.baremetal driver. This abstracts the details of physical hardware within Ironic, such that the user interacts with Nova in the same way when deploying instances to virtual or physical machines. The hardware-specific details are only exposed to the cloud operators.

Specifically, this will:

  • add the nova.virt.ironic driver, which will use the python-ironicclient library to interact with Ironic’s REST API for the purpose of provisioning physical machines.

  • add new IronicHostManager class, similar to BaremetalHostManager, which fills the same purpose but is specific to Ironic. Namely, this provides several customizations to Nova’s HostManager, tailoring it to consuming discrete and non-subdivisible physical resources.

  • add exact-match scheduler filters, to facilitate users who wish to match nova flavor to hardware specifications exactly. The best matching possible today is greater-than-or-equal, which is often undesirable (eg. because a machine with 128GB of RAM could be selected to fulfil a request for an instance with 16GB of RAM).

This driver will initially implement a subset of the Nova virt driver API sufficient to support the same functionality that the nova.virt.baremetal driver supported. Over time, additional functionality will be added, as appropriate and possible for physical hardware. It is expected that some operations may never be added to this driver, eg when the operation is not possible where there is no local hypervisor.

This driver will expose the complete resources of the ironic service it is connected to. Therefor, running multiple nova-compute processes within a single cell or region will not be possible, and HA for the nova-compute service must be achieved externally, eg. via pacemaker+corosync. Scale-out may be achieved by running multiple ironic clusters, with a single n-cpu connected to each ironic end-point. This is not optimal, and is a result of a current limitation within Nova. See the Alternatives section below for a summary of the discussion which has occurred around this limitation.

Alternatives

One alternative would be for users to directly interact with Ironic’s API, circumventing Nova when deploying instances to bare metal. This would require Ironic to duplicate a significant amount of functionality present in Nova, and violate the abstraction layer. Note that giving end-users direct access to Ironic’s API may present security concerns for some operators; see the Security Impact section below for a discussion of this.

Instead of creating a new IronicHostManager class, the existing BaremetalHostManager class could be refactored to support both drivers.

Instead of adding exact-match scheduler filters, we could create a new scheduler that is specifically geared towards non-divisible resources. However, this approach is sufficient for many use cases, and does not prevent the later creation of another scheduler.

An alternative was proposed which would allow multiple nova-compute processes to proxy for the same Ironic service end-point at the same time. This could be done by setting the same ‘host’ property on each nova-compute service, such that they expose the same set of resources. Therefor, certain operations would need to be skipped when starting the nova-compute process (eg, so it doesn’t trample over an ongoing operation on another compute host). This would be accomplished by creating a new ClusteredComputeManager class (subclassed from ComputeManager) which would override init_host() to avoid the call to InstanceList.get_by_host(), self._destroy_evacuated_instances() and self._init_instance(). This proposal was denied due to architectural concerns within Nova, specifically around @utils.synchronize(instance[‘uuid’]) calls, and event callbacks that could be routed to a host other than the one waiting for the callback.

Instead of overriding ComputeManager.init_host(), a significant rewrite of Nova’s internal resource model could be undertaken – eg, to remove the (host, hypervisor_hostname) tuple from all places within the code and make the nova-conductor process handle resource locking for clustered hypervisors, such as Ironic. This would be a significant undertaking, and while merited, it was agreed that this work would not block the Ironic driver.

Data model impact

Adding the nova.virt.ironic driver will not impact the db model.

Some additional extra_specs may be leveraged in faciliating better scheduling in the future. There is precedent in the way that the nova.virt.baremetal driver leverages extra_specs:baremetal:cpu_arch. Ironic may extend this to support additional hardware metadata in the future.

REST API impact

The nova.virt.ironic driver will not add any REST API extensions or require changes to any of Nova’s APIs.

Security impact

Allowing Nova to provision physical hardware has significant security implications. The nova.virt.baremetal driver required direct access to the OOB management (IPMI) network of the hardware it managed. A compromise of nova-compute would expose that hardware’s management interface.

In a properly-secured OpenStack deployment, security will be improved by moving this functionality out of nova-compute and into ironic-conductor, because there is a strict API between the two services.

The ironic-api should not be reachable or discoverable by end-users, and only the ironic-conductor service should have access to the hardware management interface. The user of Nova who requests an instance on bare metal will thus have no direct access to the services managing that bare metal host. Should nova-compute be compromised, the malicious user would still need to gain access to the ironic-conductor host before having any access to the hardware management interface.

Considering how often IPMI is not properly secured, and that in many cases it can not be secured, the OOB management network should be as isolated from users as possible.

Notifications impact

None

Other end user impact

None

Performance Impact

No impact on Nova itself.

The performance profile of the nova.virt.ironic driver will be different than other virt drivers due to the nature of managing physical machines. For example, power cycling bare metal often takes more than five minutes as the hardware must complete a POST cycle. Thus, a deploy may be expected to take a minimum of ten minutes, though depending on the hardware, it may be more or less.

Other deployer impact

Deploying Nova with the nova.virt.ironic driver will be considerably different to deploying Nova with other virt drivers, and also different from the nova.virt.baremetal driver. Main areas of difference are:

  • different system libraries will be required. No local hypervisor needs be installed, and none of the system libraries to enable baremetal need to be installed on the compute host itself.

  • the OpenStack Ironic services must be properly set up and discoverable via Keystone in order for the nova.virt.ironic driver to function properly.

  • Nova must be supplied with admin credentials capable of interacting with Ironic.

An upgrade path from the nova.virt.baremetal driver to the nova.virt.ironic driver will be provided. The details of that are proposed in another document:

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

devananda

Other contributors:

lucasagomes nobodycam

Work Items

  • Merge auxiliary components: HostManager and exact-match scheduler filters

  • Delete auxiliary components from Ironic’s tree

  • Split the nova.virt.ironic driver into a series of patches, the sum of which will pass unit and functional tests.

  • Delete driver from Ironic’s tree after it has merged in Nova.

Dependencies

None

Testing

There is already tempest testing being done upstream against changes in ironic, nova, devstack, and tempest. However, it is non-voting today. The following paragraph describes how it works.

Devstack creates a “mock” bare metal node, enrolls it with Ironic, and configures Nova appropriately to use the nova.virt.ironic driver. A tempest scenario test is then run against that devstack instance, which allows tempest to test functionality appropriate for this driver. Certain tests may be excluded when the functionality does not apply to bare metal (eg, live migrate). The current test is fairly simple: validate the boot process, network connectivity of the instance, and validate destroy. Additional tests have been proposed for more coverage, eg. “rebuild –preserve-ephemeral”.

Testing of functionality not exposed via the nova virt driver interface is done directly in Tempest via the Ironic API (eg, management operations) and is mentioned here only for completeness.

Documentation Impact

Documentation should be added to Nova stating the existence of the new driver, and should include links to the Ironic project’s developer and deployer documentation.

References

Current code, in Ironic’s git tree::

http://git.openstack.org/cgit/openstack/ironic/tree/ironic/nova

Devstack support for testing this driver::

http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/ironic http://git.openstack.org/cgit/openstack-dev/devstack/tree/tools/ironic

Tempest test which deploys using the nova.virt.ironic driver::

http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/test_baremetal_basic_ops.py

Juno summit etherpad discussing this::

https://etherpad.openstack.org/p/juno-nova-deprecating-baremetal

Some best practices for IPMI sanity::

http://fish2.com/ipmi/bp.pdf

Discussions of IPMI vulnerabilities::

http://fish2.com/ipmi/itrain.pdf http://fish2.com/ipmi/river.pdf