Add maxphysaddr support for Libvirt¶
This blueprint propose new flavor extra_specs to control the physical address bits of vCPUs in Libvirt guests.
When booting a guest with 1TB+ RAM, the default physical address bits are too small and the boot fails . So a knob is needed to specify the appropriate physical address bits.
Booting a guest with large RAM.
In Libvirt v8.7.0+ and QEMU v2.7.0+, physical address bits can be specified with following XML elements  . The former means to adopt any physical address bits, the latter means to adopt the physical address bits of the host CPU.
<maxphysaddr mode='emulate' bits='42'/>
Here I suggest the following two flavor extra_specs. Of course, if these are omitted, the behavior is the same as before.
hw:maxphysaddr_modecan be either
hw:maxphysaddr_bitstakes a positive integer value. Only meaningful and must be specified if
Nova scheduler changes¶
Nova scheduler also needs to be modified to take these two properties into account.
There can be a mix of supported and unsupported hosts depending
on Libvirt and QEMU versions. So add new traits
to check the scheduled host supports this feature.
trait:COMPUTE_ADDRESS_SPACE_PASSTHROUGH=required is automatically added
hw:maxphysaddr_mode=passthrough is specified in flavor extra_specs.
And same for
Passthrough and emulate modes have different properties. So let’s consider the two separately.
The case of
hw:maxphysaddr_mode=passthrough. In this case,
cpu_mode=host-passthrough is a requirement, which is already taken
into account in nova scheduling, and no additional modifications are
required in this proposal. It is not guaranteed whether the instance
can be migrated by nova. So the admin needs to make sure that targets
of cold and live migration have similar hardware and software.
This restriction is similar for
The case of
hw:maxphysaddr_mode=emulate. In nova scheduling,
it is necessary to check that the hypervisor supports at least
hw:maxphysaddr_bits. The maximum number of bits supported by
hypervisor can be obtained by using libvirt capabilities . Therefore,
ComputeCapabilitiesFilter can be used to compare the number of bits in
scheduling. For example, this can be accomplished by adding
Cold migration and live migration can also be realized with this filter
So the overall flavor extra_specs look like the following:
openstack flavor set <flavor> \ --property hw:maxphysaddr_mode=emulate \ --property hw:maxphysaddr_bits=42
Since ComputeCapabilitiesFilter only supports flavor extra_specs and not image properties , this proposal is out of scope for image properties.
maxphysaddr option was introduced into Libvirt, it was specified
as a workaround with the QEMU comanndline parameter. But this alternative is
not allowed in nova.
Also, some Linux distributions may have machine types with
host-phys-bits=true . For example,
pc-q35-bionic-hpb. However, this alternative has following two issues and
cannot be adopted for general-purpose use cases.
Ubuntu package maintainers are applying a patch to QEMU . It means this is not included in vanilla QEMU and is not available in other distributions.
This is only the case for
hw:maxphysaddr_mode=passthroughand does not include
cpu_mode=host-passthroughto be used , this alternative cannot be used with
cpu_mode=host-model. So, this alternative is not sufficient for a cloud with many different CPU models.
As for scheduling, placement does not currently support numeric traits, so the maximum number of bits supported by hypervisor cannot be checked by this mechanism.
Data model impact¶
REST API impact¶
Other end user impact¶
Other deployer impact¶
Operators should specify appropriate flavor extra_specs as needed.
As described earlier, the new traits
COMPUTE_ADDRESS_SPACE_EMULATED signal if the upgraded compute nodes support
- Primary assignee:
- Other contributors:
- Feature liaison:
Add new guest configs
Add new fileds in nova/api/validation/extra_specs/hw.py
Add new fields in LibvirtConfigCPU in nova/virt/livbirt/config.py
Add new traits to check Libvirt and QEMU versions
Add new field
Add docs and release notes for new flavor extra_specs
Libivrt v8.7.0+. QEMU v2.7.0+.
Add the following unit tests:
check that proposed flavor extra_specs are properly validated
check that intended XML elements are output
check that traits are properly added and used
check that new field in
ComputeCapabilitiesFilteris property added and used
For operators, the documentation describes what proposed flavor extra_specs mean and how they should be set.