Overhead option to differ hypervisor process on a global set of pCPUs¶
The Nova scheduler and the placement API determine CPU resource utilization and instance CPU placement based on the number of vCPUs in the flavor. A number of hypervisors have operations that are being performed on behalf of the guest instance in the host OS. These operations should be accounted and scheduled separately, as well as have their own placement policy controls applied.
Previously was introduced option
adds additional pCPU per guest to run emulator threads.
While it resolves issues related to spike latency caused by emulator threads running on same pCPUs that vCPUs are pinned on, some operators have desire to pack all emulator threads on a specific set of pCPUs in order to allow more pCPUs running vCPUs.
As an operator I want all the emulator threads of all the instances running in a specific set pCPUs.
To extend flexibility and address use-cases where resources on hosts
are limited a separate “Standardize CPU resource tracking” spec that
discusses a change to how we would like to both simplify the
configuration of a compute node with regards to CPU resource inventory
as well as make the quantitative tracking of dedicated CPU resources
consistent with the tracking of shared CPU resources via the placement
API, introduces a CONF option
cpu_shared_set which stores a pinset
string that indicates the physical processors that should be used for
VCPU resource requests.
The proposed change is to run the emulator threads work on these
shared host CPUs. The admin who would like to take advantage of such
improvement for its flavors will have to configure the flavor
It is has noted that, the
exists but its default behavior where
CONF.cpu_shared_set is not
configured on host will remain the same, meaning that the emulator
threads will float on the set of pCPUs dedicated for the guest. As for
hw:emulator_threads_policy=isolate its behavior will remain the
same, meaning that an additional pCPU is reserved to run guest
emulator threads on.
An alternative would be to always pin emulator threads on
CONF.cpu_shared_set. It has been noted that, removing the actual
flexibility provided to users to isolate guest emulator threads on an
dedicated pCPU or let the guest emulator threads floating across the
whole set of pCPUs dedicated for guest are still valid option and we
should not remove such flexibility.
Data model impact¶
REST API impact¶
Other end user impact¶
For end users, using option
CONF.cpu_shared_set configured is going to improve the
latency of guests running sensitive workloads.
Other deployer impact¶
Operators who want to configure some flavors to run their guest
emulator threads outside of the pCPUs pinned for running vCPUs threads
will be able to achieve that by specifying a range of pCPUs using
CONF.cpu_shared_set and setting
Developers of other virtualization drivers may wish to make use of the new flavor extra spec property and scheduler accounting. This will be of particular interest to the Xen hypervisor, if using the stub domain feature.
- Primary assignee:
Sahid Orentino Ferdjaoui <sahid-ferdjaoui>
CONF.compute.cpu_shared_setoption for compute nodes
Configuring guest to pin its emulator threads on the
CONF.compute.cpu_shared_setis also defined in “Standardize CPU resource tracking” spec. This option could be introduced by both of the spec.
This can be tested in any CI system that is capable of testing the current NUMA and dedicated CPUs policy. i.e. It requires ability to use KVM and not merely QEMU. Functional tests for the scheduling and driver bits (libvirt) are going to be added.
The documentation detailing NUMA and dedicated CPU policy usage will need to be extended to also describe the new options this work introduces.