Pick guest CPU architecture based on host arch in libvirt driver

https://blueprints.launchpad.net/nova/+spec/pick-guest-arch-based-on-host-arch-in-libvirt-driver

Implement new image meta property that allows for the selection of the correct QEMU binary, cpu architecture, and machine type for a guest architecture that is different than the host architecture; An x86_64 guest running on an AArch64 host, and vice versa.

Problem description

Currently, in many places, Nova’s libvirt driver makes decisions on how to configure guest XML based on host CPU architecture caps.host.cpu.arch. That is not optimal in all cases where physical hardware support is limited for non-traditional architectures.

So all of the said code needs to be reworked to make those decisions based on guest CPU architecture (i.e. guest.arch, which should be set based on the image metadata property, hw_emulation_architecture).

A related piece of work is to distinguish between hosts that can do AArch64, PPC64, Etc. via KVM (which is hardware-accelerated) vs. those that can only do it via plain emulation TCG — this is to ensure that guests are not arbitrarily scheduled on hosts that are incapable of hardware acceleration, thus losing out on performance-related benefits.

Use Cases

  • As an admin/operator I want to allow for cpu architecture emulation due to constraints of or lack with alternate physical architecture types.

  • As an admin/operator I want to deploy AArch64, PPC64, MIPs, RISC-V, and s390x as an emulated architecture on x86_64.

  • As an admin/operator I want to deploy x86_64, PPC64, MIPs, RISC-V, and s390x as an emulated architecture on AArch64.

Proposed change

To enable this new cpu architecture spec, an image property will be introduced and an additional function which allows for checks and comparisions between the host architecture and desired emulation architecture

Note

The following hw_architecture image property relates to the physical architecture of the compute hosts. If physical nodes are not present for the desired architecture then the instance will not be provisioned.

Retrieve OS architecture for LibvirtConfigGuest

This leverages nova virt libvirt config to grab the os_arch and update the hw_architecture image meta property with the retrieved value. With this change we can perform the required comparisons within nova virt libvirt driver for the hw_architecture and hw_emulation_architecture values.

if self.os_arch is not None:
    type_node.set("arch", self.os_arch)

Allow emulation architecture to be defined by image property

To enable defining the guest architecture the following string based image meta property will be introduced:

  • hw_emulation_architecture

When this image property is not defined then instance provisioning will occur as normal. The process is demonstrated below via the 3 examples.

Example 1 When both image meta properties are set the emulation architecture will take precedent, and it will build on a X86_64 host that supports emulatating AARCH64 or whatever supported architecture is inputted in place of AARCH64.

  • hw_emulation_architecture = AARCH64

  • hw_architecture = X86_64

Example 2 When the emulation image meta property is set the emulation architecture will take precedent, and it will build on any host that supports emulating X86_64 or whatever supported architecture is inputted in place of X86_64.

  • hw_emulation_architecture = X86_64

  • hw_architecture = <unset>

Example 3 When the hw_emulation_architecture property is unset it will build on any host that natively supports the specified architecture.

  • hw_emulation_architecture = <unset>

  • hw_architecture = AARCH64 OR hw_architecture = X86_64

Update scheduler request_filter to handle both architecture fields

Within the transform_image_metadata function, we will add the two architecture properties to the prefix_map. this in itself also requires additional os-traits to be added for both hw and compute.

def transform_image_metadata(ctxt, request_spec):
    """Transform image metadata to required traits.

    This will modify the request_spec to request hosts that support
    virtualisation capabilities based on the image metadata properties.
    """
    if not CONF.scheduler.image_metadata_prefilter:
        return False

    prefix_map = {
        'hw_cdrom_bus': 'COMPUTE_STORAGE_BUS',
        'hw_disk_bus': 'COMPUTE_STORAGE_BUS',
        'hw_video_model': 'COMPUTE_GRAPHICS_MODEL',
        'hw_vif_model': 'COMPUTE_NET_VIF_MODEL',
        'hw_architecture': 'HW_ARCH',
        'hw_emulation_architecture': 'COMPUTE_ARCH',
    }

Update os-traits

Below are the os-traits proposed for the compute cpu architectures to be supported for emulatation, where as the hardware architecture includes all current nova supported architectures within nova objects fields.

TRAITS = [
    'AARCH64',
    'PPC64LE',
    'MIPSEL',
    'S390X',
    'RISCV64',
    'X86_64',
]

To account for the emulation of these architectures, updates will be made to the nova virt libvirt driver ensuring that compute capability traits are reported for each architecture emulator that is available on the hosts.

Perform architecture test against emulation

To facilitate a simple check throughout the nova virt libvirt driver the following function does a check and will set the appropriate guest architecture based on emulation, if defined.

def _check_emulation_arch(self, image_meta):
    emulation_arch = image_meta.properties.get("hw_emulation_architecture")
    if emulation_arch:
        arch = emulation_arch
    else:
        arch = libvirt_utils.get_arch(image_meta)
    return arch

Utilization of the actual check performed through processing the image_meta dictionary values.

arch = self._check_emulation_arch(image_meta)

Proposed emulated architectures and current support level

All testing performed with changes proposed in this spec demonstrated that the emulated guests maintain current support for all basic lifecycle actions. Listed below are the proposed architectures and there current functional level with the spec, with the plan of all being Tested and validated for functional support.

  • X86_64 - Tested and validated for functional support

  • AARCH64 - Tested and validated for functional support

  • PPC64LE - Tested and validated for functional support

  • MIPSEL - Awaiting libvirt patch for PCI support

  • S390X - Troubleshooting guest kernel crash for functional support

  • RISCV64 - To be Tested

Alternatives

Other attempts have been made leverage existing image meta properties such as hw_architecture only; however, this opens various other issues with conflicting check and alterations of core code. This also runs into issues during the scheduling of instances as there will be no matching physical host architectures, which is what this spec aims to solves.

While the best option is providing actual physical support for the cpu architectures you want to test, this opens the ability to a wider audience to perform the same type of local emulation they can with QEMU within an openstack environment.

Data model impact

  • Adds a new set of standard traits to os-traits.

  • Adds new property to image_meta objects.

  • The OS arch value will be pulled into the LibvirtConfigGuest.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

This is expected to improve boot performance in a heterogeneous cloud by reducing reschedules. By passing a more constrained request to placement this feature should also reduce the resulting set of allocation_candidates that are returned.

This will also ensure that native support is handled first over emulation as it requires a specific property to be set in order to perform the required checks.

Other deployer impact

Ensure that all the desired QEMU binaries are installed on the physical nodes for the cpu architectures that you would like to support.

Developer impact

None

Upgrade impact

None

Implementation

Assignee(s)

Primary assignee:

chateaulav - Jonathan Race

Feature Liaison

Feature liaison:

Liaison Needed

Work Items

  • Add new traits

  • Update prefilter

  • Modify nova libvirt virt driver to perform checks for emulation architecture

  • Add new property to image_meta objects

  • Modify nova libvirt virt config to pull OS arch into LibvirtConfigGuest

  • Tests

Dependencies

Blueprint

Project Changesets

Libvirt MIPs PCI Bug

Testing

Unit tests will be added for validation of the following proposed changes:

  • nova virt libvirt driver to validate handling of the hw_emulation_architecture image property value and associated checks.

  • nova scheduler request_filter to ensure proper handling of the prefilter, with added the two new values.

Proposed updates to tempest will account for the non-native architectures being supported through emulation.

  • AARCH64 architecture will be tested with every patch

  • Remaining architectures will be tested with the periodic-weekly and experimental pipelines.

Documentation Impact

A release note will be added. As there is enduser impact, user facing documentation will be required for the supported emulation architecture types and the required image meta properties to need to be set.

References

History

Revisions

Release Name

Description

Yoga

Introduced