Overhead option to differ hypervisor process on a global set of pCPUs

https://blueprints.launchpad.net/nova/+spec/overhead-pin-set

The Nova scheduler and the placement API determine CPU resource utilization and instance CPU placement based on the number of vCPUs in the flavor. A number of hypervisors have operations that are being performed on behalf of the guest instance in the host OS. These operations should be accounted and scheduled separately, as well as have their own placement policy controls applied.

Problem description

Previously was introduced option hw:emulator_threads_policy which adds additional pCPU per guest to run emulator threads.

While it resolves issues related to spike latency caused by emulator threads running on same pCPUs that vCPUs are pinned on, some operators have desire to pack all emulator threads on a specific set of pCPUs in order to allow more pCPUs running vCPUs.

Use Cases

As an operator I want all the emulator threads of all the instances running in a specific set pCPUs.

Project Priority

None

Proposed change

To extend flexibility and address use-cases where resources on hosts are limited a separate “Standardize CPU resource tracking” spec that discusses a change to how we would like to both simplify the configuration of a compute node with regards to CPU resource inventory as well as make the quantitative tracking of dedicated CPU resources consistent with the tracking of shared CPU resources via the placement API, introduces a CONF option cpu_shared_set which stores a pinset string that indicates the physical processors that should be used for the VCPU resource requests.

The proposed change is to run the emulator threads work on these shared host CPUs. The admin who would like to take advantage of such improvement for its flavors will have to configure the flavor extra-specs hw:emulator_threads_policy=share.

It is has noted that, the hw:emulator_threads_policy=share already exists but its default behavior where CONF.cpu_shared_set is not configured on host will remain the same, meaning that the emulator threads will float on the set of pCPUs dedicated for the guest. As for hw:emulator_threads_policy=isolate its behavior will remain the same, meaning that an additional pCPU is reserved to run guest emulator threads on.

Alternatives

An alternative would be to always pin emulator threads on CONF.cpu_shared_set. It has been noted that, removing the actual flexibility provided to users to isolate guest emulator threads on an dedicated pCPU or let the guest emulator threads floating across the whole set of pCPUs dedicated for guest are still valid option and we should not remove such flexibility.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

For end users, using option emulator_threads_policy=share with hosts have CONF.cpu_shared_set configured is going to improve the latency of guests running sensitive workloads.

Performance Impact

None

Other deployer impact

Operators who want to configure some flavors to run their guest emulator threads outside of the pCPUs pinned for running vCPUs threads will be able to achieve that by specifying a range of pCPUs using CONF.cpu_shared_set and setting hw:emulator_threads=share.

Developer impact

• Developers of other virtualization drivers may wish to make use of the new flavor extra spec property and scheduler accounting. This will be of particular interest to the Xen hypervisor, if using the stub domain feature.

Upgrade impact

N/A

Implementation

Assignee(s)

Primary assignee:

Sahid Orentino Ferdjaoui <sahid-ferdjaoui>

Work Items

• Introducing CONF.compute.cpu_shared_set option for compute nodes

• Configuring guest to pin its emulator threads on the CONF.compute.cpu_shared_set when emulator_threads_policy=share

Dependencies

• The CONF.compute.cpu_shared_set is also defined in “Standardize CPU resource tracking” spec. This option could be introduced by both of the spec.

Testing

This can be tested in any CI system that is capable of testing the current NUMA and dedicated CPUs policy. i.e. It requires ability to use KVM and not merely QEMU. Functional tests for the scheduling and driver bits (libvirt) are going to be added.

Documentation Impact

The documentation detailing NUMA and dedicated CPU policy usage will need to be extended to also describe the new options this work introduces.

References

History

Revisions

Queen	Proposed
Rocky	Re-proposed