VCPU in One Instance¶
The spec CPU resource tracking splits host CPUs into
resources, making it possible to run instances of
dedicated CPU allocation
policy and instances of
shared CPU allocation policy in the same host.
This spec aims to create such kind of instance that some of the vCPUs are
PCPU) CPUs and the remaining vCPUs are shared (
and expose this information via the metadata API.
The current CPU allocation policy,
shared, is applied to
all vCPUs of an instance. However, with the introduction of
CPU resource tracking, it is possible to propose a more fine-grained CPU
allocation policy, which is based on the control over individual instance vCPU,
and specifying the
shared CPU allocation policy to each
As an operator, I would like to have an instance with some realtime CPUs for high performance, and at the same time, in order to increase instance density, I wish to make the remaining CPUs, which do not demand high performance, shared with other instances because I only care about the performance of realtime CPUs. One example is deploying the NFV task that is enhanced with DPDK framework in the instance, in which the data plane threads could be processed with the realtime CPUs and the control-plane tasks are scheduled on CPUs that may be shared with other instances.
As a Kubernetes administrator, I wish to run a multi-tier or auto-scaling application in Kubernetes, which is running in single OpenStack VM, with the expectation that using dedicated high-performance CPUs for application itself and deploying the containers on shared cores.
Introduce a new CPU allocation policy
shared are the existing instance CPU allocation policies
that determine how instance CPU is scheduled on host CPU. This specification
proposes a new CPU allocation policy, with the name
create a CPU mixed instance in such way that some instance vCPUs are
allocated from computing node’s
PCPU resource, and the rest of instance
vCPUs are allocated from the
VCPU resources. The CPU allocated from
PCPU resource will be pinned on particular host CPUs which are defined in
CONF.compute.dedicated_cpu_set, and the CPU from
VCPU resource will be
floating on the host CPUs which are defined in
In this proposal, we call these two kinds of vCPUs as dedicated vCPU and
shared vCPU respectively.
Instance CPU policy matrix¶
Nova operator may set the instance CPU allocation policy through the
hw_cpu_policy interfaces, which may raise conflict.
The CPU policy conflict is proposed to be solved with the following policy
INSTANCE CPU POLICY
For example, if a
dedicated CPU policy is specified in instance flavor
hw:cpu_policy, then the instance CPU policy is
of the setting specified in image property
is explicitly set in
hw:cpu_policy, then a
mixed policy specified
hw_cpu_policy is conflict, which will throw an exception, the instance
booting request will be rejected.
If there is no explicit instance CPU policy specified in flavor or image
property, the flavor matrix result would be ‘undefined’, and the final
instance policy is further determined and resolved by
resources:VCPU specified in flavor extra specs. Refer to
section and the spec
CPU resource tracking.
Affect over real-time vCPUs¶
Real-time vCPU also occupies the host CPU exclusively and does not share CPU with other instances, all real-time vCPUs are dedicated vCPUs. For a mixed instance with some real-time vCPUs, with this proposal, the vCPUs not in the instance real-time vCPU list are shared vCPUs.
Affect over emulator thread policy¶
If emulator thread policy is
ISOLATE, the mixed instance will look for
a dedicated host CPU for instance emulator thread, which is very similar
to the case introduced by
dedicated policy instance.
If the emulator thread policy is
SHARE, then the instance emulator thread
will float over the host CPUs defined in configuration
Set dedicated CPU bit-mask in
As an interface to create the
mixed policy instance through legacy flavor
extra specs or image properties, the flavor extra spec
hw:cpu_dedicated_mask is introduced. If the extra spec
hw:cpu_dedicated_mask is found in the instance flavor, then the
information of the dedicated CPU could be found through
Here is the example to create an instance with
$ openstack flavor set <flavor_id> \ --property hw:cpu_policy=mixed \ --property hw:cpu_dedicated_mask=0-3,7
And, following is the proposing command to create a mixed instance which
consists of multiple NUMA nodes by setting the dedicated vCPUs in
$ openstack flavor set <flavor_id> \ --property hw:cpu_policy=mixed \ --property hw:cpu_dedicated_mask=2,7 \ --property hw:numa_nodes=2 \ --property hw:numa_cpus.0=0-2 \ --property hw:numa_cpus.1=3-7 \ --property hw:numa_mem.0=1024 \ --property hw:numa_mem.1=2048
Please be aware that there is no equivalent setting in image properties
for flavor extra spec
hw:cpu_dedicated_mask. It will not be supported
to create mixed instance through image properties.
The dedicated vCPU list of a mixed instance could be specified through
the newly introduced dedicated CPU mask or the cpu-time CPU mask, the
hw_cpu_realtime_mask, you cannot set it
by setting dedicated CPU mask extra spec and real-time CPU mask at the
Create mixed instance via
CPU resource tracking introduced a way to create an instance with
shared CPU allocation policy through
resources:VCPU interfaces, but did not allow requesting both
VCPU resource for one instance.
This specification proposes to let an instance request
PCPU resource along
VCPU, and effectively applying for the
mixed CPU allocation
policy if the
cpu_policy is not explicitly specified in the flavor list.
So an instance with such flavors potentially creates a
$ openstack flavor set \ --property "resources:PCPU"="<dedicated CPU number>" \ --property "resources:VCPU"="<shared CPU number>" \ <flavor_id>
For mixed instance created in such way, both <shared CPU number> and
<dedicated CPU number> must be greater than zero. Otherwise, it effectively
shared policy instance, that all vCPUs in the
instance is in a same allocation policy.
resources::VCPU interfaces only put the request
Placement service for how many
VCPU resources are
required to fulfill the instance vCPU thread and emulator thread requirement.
VCPU distribution on the instance, especially on the
instance with multiple NUMA nodes, will be spread across the NUMA nodes in the
round-robin way, and
VCPU will be put ahead of
PCPU. Here is one
example and the instance is created with flavor below:
flavor: vcpus:8 memory_mb=512 extra_specs: hw:numa_nodes:2 resources:VCPU=3 resources:PCPU=5
Instance emulator thread policy is not specified in the flavor, so it does not
occupy any dedicated
PCPU resource for it, all
resources will be used by vCPU threads, and the expected distribution on NUMA
NUMA node 0: VCPU VCPU PCPU PCPU NUMA node 1: VCPU PCPU PCPU PCPU
The demanding instance CPU number is the number of vCPU, specified by
flavor.vcpus, plus the number of CPU that is special for emulator
thread, and if the emulator thread policy is
ISOLATE, the instance
flavor.vcpus + 1 vCPUs, if the policy is not
the instance just requests
Creating CPU mixed instance by extending the
Instead of adding a special instance CPU allocation policy, the CPU mixed
instance is supported by extending the existing
dedicated policy and
specifying the vCPUs that are pinned to the host CPUs chosen from
Following extra spec and the image property are defined to keep the
dedicated vCPUs of a
mixed policy instance:
hw:cpu_dedicated_mask=<cpu set string> hw_cpu_dedicated_mask=<cpu set string>
<cpu set string> shares the same definition defined above.
This was rejected at it overloads the
dedicated policy to mean two things,
depending on the value of another configuration option.
mixed instance with
Following commands was proposed as an example to create a mixed instance by
an explicit request of
PCPU resources, and infer the
VCPU count by
$ openstack flavor create mixed_vmf --vcpus 4 --ram 512 --disk 1 $ openstack flavor set mixed_vmf \ --property hw:cpu_policy=mixed \ --property resources:PCPU=2
This was rejected due to the mixing use of
resources:PCPU. It is not recommended to mix placement style syntax with
traditional extra specs.
Data model impact¶
pcpuset field in
InstanceNUMACell object to track the dedicated
vCPUs of the instance NUMA cell, and the original
is special for shared vCPU then.
This change will introduce some database migration work for the existing
instance in a
dedicated CPU allocation policy, since all vCPUs in such an
instance are dedicated vCPUs which should be kept in
pcpuset field, but
they are stored in
REST API impact¶
The metadata API will be extended with the dedicated vCPU info and a new OpenStack metadata version will be added to indicate this is a new metadata API.
The new field will be added to the
dedicated_cpus=<cpu set string>
<cpu set string> lists the dedicated vCPU set of the instance, which
might be the content of
hw_cpu_realtime_mask or the CPU list
generated with the round-robin policy as described in
The new cpu policy
mixed is added to extra spec
Other end user impact¶
If the end user wants to create an instance with a
mixed CPU allocation
policy, the user is required to set corresponding flavor extra specs or image
This proposal affects the selection of instance CPU allocation policy, but the performance impact is trivial.
Other deployer impact¶
mixed cpu policy is only available when the whole cluster upgrade
finished. A service version will be bumped for detecting the upgrade.
InstanceNUMACell.pcpuset is introduced for dedicated vCPUs and the
InstanceNUMACell.cpuset is special for shared vCPUs, all existing
instances in a
dedicated CPU allocation policy should be updated by moving
InstanceNUMACell.cpuset filed to
InstanceNUMACell.pcpuset field. The underlying database keeping the
InstanceNUACell object also need be updated to reflect this change.
- Primary assignee:
Wang, Huaqiang <firstname.lastname@example.org>
- Feature liaison:
Stephen Finucane <email@example.com>
Add a new field, the
InstanceNUMACellfor dedicated vCPUs.
Add new instance CPU allocation policy
mixedproperty and resolve conflicts
Bump nova service version to indicate the new CPU policy in nova-compute
Add flavor extra spec
hw:cpu_dedicated_maskand create mixed instance
Translate dedicated and shared CPU request to placement
Change libvirt driver to create
Add nova metadata service by offering final pCPU layout in
Validate real-time CPU mask for
Functional and unit tests are required to cover:
Ensure to solve the conflicts between the CPU policy matrix
Ensure only dedicated vCPUs are possible to be real-time vCPUs
mixedpolicy instance properly either by flavor settings or by
Ensure shared vCPUs is placed before the
Ensure the emulator CPU is properly scheduled according to its policy.
The documents should be changed to introduce the usage of new
allocation policy and the new flavor extra specs.
Metadata service will be updated accordingly.