Virtual Bare Metal Cluster Manager

StoryBoard RFE

Testing Ironic has always been challenging. Bare metal compute resources are expensive and inflexible. The typical solution to this issue is to emulate bare metal compute resources using virtual machines (VMs), allowing us to improve the utilisation of test infrastructure, and test various scenarios using general purpose test infrastructure. Support for creating virtual bare metal environments is available in DevStack, and also in Bifrost. These are used in OpenStack CI. This specification covers creation of a new tool for managing virtual bare metal clusters.

Problem description

As Ironic and bare metal cloud use cases grow in popularity, it becomes necessary to ensure that tooling is available for performing representative tests. The DevStack plugin and Bifrost playbooks are helpful, but not applicable in every environment, as they pull in additional dependencies, and are quite tied to the specifics of those environments.

In particular, testing of OpenStack deployment tools that make use of Ironic such as Kayobe, Kolla Ansible and TripleO would benefit from a shared tool for managing the configuration of virtual bare metal testing environments.

It would be desirable for the new tool to be extensible and support more complex configuration in future - for example, networking configuration. Shell script-based tools such as the DevStack plugin inherently risk becoming monolithic with new feature additions. It is important that the new tool supports more complex use cases without sacrificing its ease of use or maintenance for simpler configurations.

User Stories

  • As an OpenStack developer, I want to be able to easily create a generic all-in-one virtual bare metal cluster at will, so that I can use it as a development environment.

  • As an OpenStack developer, I want to be able to tear down my virtual cluster once I am finished with it, so that my system is brought back to a clean state.

  • As an Ironic developer, I want to be able to create a virtual bare metal cluster to a particular specification of VMs and virtual networks, so that I can simulate a given system architecture for my own development.

  • As an Ironic developer, I want to be able to reconfigure my virtual cluster, so that I can test changes in a variety of bare metal architectures.

  • As a maintainer of a CI toolchain, I want there be a de facto virtual cluster management tool for CI jobs, so that I don’t have to use a bespoke/non-extensible utility.

  • As an Ironic developer, I want CI jobs to set up job environments with the same tool that I use to set up my development environment, so that I can easily reproduce the results of CI jobs on my own machine.

  • As a systems deployer, I want to be able to perform bare metal scale testing locally/virtually, so I can diagnose scaling issues before deploying onto a real system.

Proposed change

We propose to add a new tool for managing virtual bare metal clusters, named Tenks. The scope of Tenks includes:

  • Creation and deletion of VMs representing bare metal nodes

  • Orchestration of tools used to emulate Baseboard Management Controllers (BMCs), e.g. virtual BMC and sushy-tools <https://github.com/openstack/sushy-tools>.

  • Creation and deletion of virtual networks representing bare metal networks

Tenks will initially be targeted at the typical Libvirt/QEMU/KVM environment, but in the typical OpenStack way, will be written in such a way as to allow support for other virtualisation providers to be added in future.

N.B. Libvirt provides an extremely extensive API for configuration of VM properties. It is not Tenks’ aim to expose anything but the smallest common subset of this. This offers the advantages of easier cross-compatibility with multiple virtualisation providers, and looser coupling to each of these. Tenks is intended for development and testing purposes, rather than for fine-grained tuning of virtual cluster configuration.

Once built, we propose to migrate the Ironic DevStack plugin and/or Bifrost test playbooks to use Tenks.

The source code for Tenks is available on GitHub. A blog post about Tenks’ initial development can be found on the StackHPC website. Documentation can be found on Read the Docs.

Etymological Side Note

  • Tenks

  • ten-k-s

  • 10-thousand-spoons

  • Alanis Morissette - Ironic: lyrics L27

Name idea courtesy of mgoddard.

Proposed Implementation

It is proposed to implement Tenks using Ansible. Ansible is a good fit for such a tool because:

  • It provides a good platform upon which to write an idempotent application. Since Tenks will be configured declaratively, idempotence is important because it allows the system to be brought to the declared state regardless of its initial state (for example, if some of the declared VMs already exist).

  • It allows easy reference to pre-existing Ansible roles to help with configuration.

  • It encourages development of an opinioniated tool that can be easily configured as required. This means that Tenks’ default Ansible configuration would be able to create a basic virtual bare metal cluster with minimal set-up required (due to Ansible role defaults), but any options can be overridden to customise the configuration as required, taking advantage of Ansible’s variable precedence and scoping logic.

Dependent on the size that Tenks grows to, it will probably be necessary to write an OpenStack-like CLI to wrap around the Ansible plays.

Multi-Stage Process

  • Host setup:

    • Installing and configuring the relevant software on each host:

      • Open vSwitch (containerised?)

      • Virtualisation tookit (e.g. Libvirt/QEMU/KVM)

      • Virtual platform management toolkit: Virtual BMC for IPMI at the moment. Scope for extension to include other tools in future (e.g. VirtualPDU for SNMP, sushy-tools for Redfish).

      • LVM - set up storage pools for VM volumes

    • Networking setup:

      • Creation of bridges for each physical network, and connection to physical interfaces.

  • VM configuration:

    • Schedule requests of different VM specifications to the most appropriate BM hypervisor. The scheduling algorithm could initially be naïve (or even random), and could be iteratively improved. Tenks could even be configured to prioritise certain scheduling heuristics more or less.

    • Create the specified VMs through the virtualisation provider.

    • Attach the VMs’ NICs to the relevant bridges, based on the physical networks they were configured to be connected to.

    • Register each VM with a platform management tool suitable for its driver (e.g. Virtual BMC for IPMI).

  • VM enrolment (these steps are optional if introspection is to be used):

    • Enrol each VM with Ironic, using a specified deployment ramdisk and kernel.

    • Set any additional properties on the node. This could include boot-from-volume details, capabilities and boot mode (for boot modes supported by Tenks).

    • Set any traits on the node.

    • Create a port in Ironic for each of the VMs’ NICs.

    • Make Ironic nodes available for deployment.

  • Post-deployment:

    • Create Nova flavors as required. These can specify node traits that are either desired or forbidden.

Tenks should also support a ‘tear-down’ mode which would clean up all created resources and restore the system (more or less) to its initial state.

Configuration

A declarative configuration style would be appropriate to describe the virtual infrastructure provisioned by Tenks. This could include:

  • Host inventory. Tenks will need a list of bare metal hypervisors, i.e. hosts of virtual machines simulating bare metal nodes. Could use an Ansible inventory for this.

  • Physical networks. Tenks would be configured with a list of physical networks that are shared by all hosts in the inventory. A per-host mapping of physical networks to source interfaces/bridges would be required. Tenks would create a bridge for each physical network on each host.

  • Desired virtual bare metal VM configuration. Tenks would be configured with ‘flavours’ of VM, and a count of how many VMs of each flavour to create. Flavours would ideally be agnostic of virtualisation provider, but should have the following properties:

    • Physical networks. A virtual NIC would be added to the VM for each physical network, and the NICs would be plugged into the respective bridge on the hypervisor.

    • Generic VM attributes. These would include number of CPUs and amount of RAM.

    • Volumes to be attached to the VM. Creation of blank volumes and volumes from existing images should be supported, in addition to use of existing volumes.

    These properties could be used to influence VM placement during scheduling. Initial flavour mappings for the Libvirt provider may be facilitated using the StackHPC libvirt-vm Ansible role as an interface.

Alternatives

  • Continue using specific tools in each environment

Data model impact

None

State Machine Impact

None

REST API impact

None

Client (CLI) impact

None

RPC API impact

None

Driver API impact

None

Nova driver impact

None

Ramdisk impact

None

Security impact

None

Other end user impact

None

Scalability impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

Will Miller: willm@stackhpc.com

Other contributors:

Work Items

  • Develop proof-of-concept Ansible playbooks

  • Flesh out Tenks to include more advanced features, for example:

    • Tear-down of clusters

    • Improved scheduling heuristics

    • Reconfiguration of clusters without need for tear-down

    • Providers other than Libvirt

    • Command-line interface

    • Tests (unit, functional, integration) as necessary

  • Manual testing of Tenks with various environments/configurations

  • Submit Tenks to PyPI (if the extent of Python code requires this)

  • Adapt CI pipelines to use Tenks for ephemeral cluster management

Dependencies

None

Testing

TODO

Upgrades and Backwards Compatibility

None

Documentation Impact

TODO

References