VCPU in One Instance¶
The spec CPU resource tracking splits host CPUs into
resources, making it possible to run instances of
dedicated CPU allocation
policy and instances of
shared CPU allocation policy in the same host.
This spec aims to create such kind of instance that some of the CPUs are
PCPU) CPUs and the remaining CPUs are shared (
and expose this information via the metadata API.
The current CPU allocation policy,
shared, is applied to
all CPUs of an instance. However, with the introduction of
CPU resource tracking, it is possible to propose a more fine-grained CPU
allocation policy, which is based on the control over individual instance CPU,
and specifying the
shared CPU allocation policy to each
As an operator, I would like to have an instance with some realtime CPUs for high performance, and at the same time, in order to increase instance density, I wish to make the remaining CPUs, which do not demand high performance, shared with other instances because I only care about the performance of realtime CPUs. One example is deploying the NFV task that is enhanced with DPDK framework in the instance, in which the data plane threads could be processed with the realtime CPUs and the control-plane tasks are scheduled on CPUs that may be shared with other instances.
As a Kubernetes administrator, I wish to run a multi-tier or auto-scaling application in Kubernetes, which is running in single OpenStack VM, with the expectation that using dedicated high-performance CPUs for application itself and deploying the containers on shared cores.
Introduce a new CPU allocation policy
shared are the existing instance CPU allocation policies
that determine how instance CPU is scheduled on host CPU. This specification
proposes a new CPU allocation policy, with the name
create a CPU mixed instance in such way that some instance CPUs are
allocated from computing node’s
PCPU resource, and the rest of instance
CPUs are allocated from the
VCPU resources. The CPU allocated from
PCPU resource will be pinned on particular host CPUs which are defined in
CONF.compute.dedicated_cpu_set, and the CPU from
VCPU resource will be
floating on the host CPUs which are defined in
In this proposal, we call these two kinds of CPUs as dedicated vCPU and
shared vCPU respectively.
Instance CPU policy matrix¶
Nova operator may set the instance CPU allocation policy through the
hw_cpu_policy interfaces, which may raise conflict.
The CPU policy conflict is proposed to be solved with the following policy
INSTANCE CPU POLICY
For example, if a
dedicated CPU policy is specified in instance flavor
hw:cpu_policy, then the instance CPU policy is
of the setting specified in image property
is explicitly set in
hw:cpu_policy, then a
mixed policy specified
hw_cpu_policy is conflict, which will throw an exception, the instance
booting request will be rejected.
If there is no explicit instance CPU policy specified in flavor or image
property, the flavor matrix result would be ‘undefined’, and the final
instance policy is further determined and resolved by
resources:VCPU specified in flavor extra specs. Refer to
section and the spec
CPU resource tracking.
Affect over real-time vCPUs¶
It’s also possible to set some mixed instance vCPUs as real-time vCPU, realtime CPUs must be chosen from the instance dedicated CPU set.
Affect over emulator thread policy¶
If emulator thread policy is
ISOLATE, the mixed instance will look for
a dedicated host CPU for instance emulator thread, which is very similar
to the case introduced by
dedicated policy instance.
If the emulator thread policy is
SHARE, then the instance emulator thread
will float over the host CPUs defined in configuration
Set dedicated CPU bit-mask in
As an interface to create the
mixed policy instance through legacy flavor
extra specs or image properties, the flavor extra spec
introduced. If the extra spec
hw:pinned_cpus is found in the instance
flavor, then the information of the dedicated CPU could be found through
Here is the example to create an instance with
$ openstack flavor set <flavor_id> \ --property hw:cpu_policy=mixed \ --property hw:pinned_cpus=0-3,7
And, following is the proposing command to create a mixed instance which
consists of multiple NUMA nodes by setting the dedicated CPUs in
$ openstack flavor set <flavor_id> \ --property hw:cpu_policy=mixed \ --property hw:pinned_cpus=2,7 \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-2 \ --property hw:numa_cpus.1=3-7 \ --property hw:numa_mem.0=1024 \ --property hw:numa_mem.1=2048
Please be aware that there is no equivalent setting in image properties
for flavor extra spec
hw:pinned_cpus. It will not be supported to
create mixed instance through image properties.
Create mixed instance via
CPU resource tracking introduced a way to create an instance with
shared CPU allocation policy through
resources:VCPU interfaces, but did not allow requesting both
VCPU resource for one instance.
This specification proposes to let an instance request
PCPU resource along
VCPU, and effectively applying for the
mixed CPU allocation
policy if the
cpu_policy is not explicitly specified in the flavor list.
So an instance with such flavors potentially creates a
$ openstack flavor set \ --property "resources:PCPU"="<dedicated CPU number>" \ --property "resources:VCPU"="<shared CPU number>" \ <flavor_id>
For mixed instance created in such way, both <shared CPU number> and
<dedicated CPU number> must be greater than zero. Otherwise, it effectively
shared policy instance, that all CPUs in the
instance is in a same allocation policy.
resources::VCPU interfaces only put the request
Placement service for how many
VCPU resources are
required to fulfill the instance vCPU thread and emulator thread requirement.
VCPU distribution on the instance, especially on the
instance with multiple NUMA nodes, will be spread across the NUMA nodes in the
round-robin way, and
VCPU will be put ahead of
PCPU. Here is one
example and the instance is created with flavor below:
flavor: vcpus:8 memory_mb=512 extra_specs: hw:numa_nodes:2 resources:VCPU=3 resources:PCPU=5
Instance emulator thread policy is not specified in the flavor, so it does not
occupy any dedicated
PCPU resource for it, all
resources will be used by vCPU threads, and the expected distribution on NUMA
NUMA node 0: VCPU VCPU PCPU PCPU NUMA node 1: VCPU PCPU PCPU PCPU
The demanding instance CPU number is the number of vCPU, specified by
flavor.vcpus, plus the number of CPU that is special for emulator
thread, and if the emulator thread policy is
ISOLATE, the instance
flavor.vcpus + 1 vCPUs, if the policy is not
the instance just requests
Creating CPU mixed instance by extending the
Instead of adding a special instance CPU allocation policy, the CPU mixed
instance is supported by extending the existing
dedicated policy and
specifying the CPUs that are pinned to the host CPUs chosen from
Following extra spec and the image property are defined to keep the
dedicated CPUs of a
mixed policy instance:
hw:cpu_dedicated_mask=<cpu set string> hw_cpu_dedicated_mask=<cpu set string>
<cpu set string> shares the same definition defined above.
This was rejected at it overloads the
dedicated policy to mean two things,
depending on the value of another configuration option.
mixed instance with
Following commands was proposed as an example to create a mixed instance by
an explicit request of
PCPU resources, and infer the
VCPU count by
$ openstack flavor create mixed_vmf --vcpus 4 --ram 512 --disk 1 $ openstack flavor set mixed_vmf \ --property hw:cpu_policy=mixed \ --property resources:PCPU=2
This was rejected due to the mixing use of
resources:PCPU. It is not recommended to mix placement style syntax with
traditional extra specs.
Data model impact¶
No change to data objects.
InstanceNUMACell.cpu_pinningfield keeps the CPU pinning information with the instance CPU ID and the host CPU ID that the instance CPU is pinning on. This field is used and filled after the CPU pair information is determined for the instance taking the
dedicatedpolicy. This field will also be used for
mixedpolicy instance with the same purpose, but will be initialized with a dictionary keyed by the instance CPU IDs that wants to be pinned on host CPUs at the instance object creating stage. The value of this field will be initialized with -1, which means host CPU ID is not valid because the host accommodating the instance is not determined in instance creating stage. The value is replaced with appropriate host CPU ID by nova-scheduler.
REST API impact¶
The metadata API will be extended with the dedicated vCPU info and a new OpenStack metadata version will be added to indicate this is a new metadata API.
The new field will be added to the
dedicated_cpus=<cpu set string>
<cpu set string> indicated the vCPU IDs which are running on dedicated
The new cpu policy
mixed is added to extra spec
Other end user impact¶
If the end user wants to create an instance with a
mixed CPU allocation
policy, the user is required to set corresponding flavor extra specs or image
This proposal affects the selection of instance CPU allocation policy, but the performance impact is trivial.
Other deployer impact¶
mixed cpu policy is only available when the whole cluster upgrade
finished. A service version will be bumped for detecting the upgrade.
- Primary assignee:
Wang, Huaqiang <firstname.lastname@example.org>
- Feature liaison:
Stephen Finucane <email@example.com>
Add new instance CPU allocation policy
mixedproperty and resolve conflicts
Bump nova service version.
Add flavor extra spec
hw:pinned_cpusand create mixed instance
Translate dedicated and shared CPU request to placement
Change libvirt driver to create
Add nova metadata service by offering final pCPU layout in
Validate real-time CPU mask for
Functional and unit tests are required to cover:
Ensure to solve the conflicts between the CPU policy matrix
Ensure only dedicated vCPUs are possible to be real-time vCPUs
mixedpolicy instance properly either by flavor settings or by
Ensure shared vCPUs is placed before the
Ensure the emulator CPU is properly scheduled according to its policy.
The documents should be changed to introduce the usage of new
allocation policy and the new flavor extra specs.
Metadata service will be updated accordingly.