Libvirt: tenant control of qemu performance monitoring unit (vPMU)

https://blueprints.launchpad.net/nova/+spec/libvirt-pmu-configuration

qemu/kvm supports emulation of a vPMU to enable standard performance monitoring tools such as Perf to be used within a virtualisation environment. The vPMU which is available on x86 cpus emulates the hardware PMU found on Intel processors and was introduced in kvm in kernel 3.3.1.

libvirt introduced support for vPMU control in 1.2.12 see https://libvirt.org/formatdomain.html#elementsFeatures so this feature is available in nova minimum supported libvirt of 1.3.1.

This spec aims to allow tenants to control when the vPMU is enabled.

Problem description

While kvm/qemu support for a vPMU is generally a useful feature the requirement to collect and maintain virtual performance counter introduces additional latency of ~10us which is about 1% of the total budget for 5G end to end traffic processing latency. While this might seem small it is an appreciable portion of the total latency introduced by virtulisation and is therefore an important factor in achieving the end to end system latency target.

As the provision of a vPMU is not currently controllable by an operator or tenant directly this creates a problem for those that want to enable or disable the vPMU to either avoid the latency overhead or rely on it to monitor the performance of their workload.

Use Cases

As a telecoms operator building a 5G network I wish to be able to deploy a virtualised Radio access network (vRAN) applicance with minimal latency impact from my virtualisation stack.

As a tenant I wish to be able to monitor the performace of my application using standard tools like perf in a virtualized environment to enable development, tuning, and profiling of my application.

Proposed change

This spec proposes adding a boolean image metadata key hw_pmu=True|False and a corresponding flavor extra spec hw:pmu=True|False to enable/disable the pmu explicitly.

The default value will be unset meaning the property is not present in either the image or flavor. This will preserve the current behavior.

If the pmu property is set to true then the pmu feature element in the libvirt xml will be set to on. Similarly if it the pmu property is set to false the pmu feature element will be set to off. If the pmu property is not specified no pmu element will be emitted in the xml allowing qemu to determine if the pmu should be enabled or not.

Note

Currently when not set the pmu is enabled/disabled based on the cpu mode and model. If cpu_mode=host-passthough then it will be enabled; if a custom cpu model is set it will be disabled.

In addition to the above minimum changes the libvirt driver could be modifed to report support for a vPMU to improve scheduling. As the vPMU feature is supported by Nova’s minimum required qemu/libvirt this would only be useful in a heterogeneous cloud. As such the desicion to expose this feature as a trait is left to the implementation and will be enabled as part of https://blueprints.launchpad.net/nova/+spec/image-metadata-prefiltering if desired.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

The image metadata versioned notification will be extended to contain the newly added field.

Other end user impact

None

Performance Impact

None

Other deployer impact

If the operator wants to consume this feature they will need to update their flavors and/or images accordingly.

Developer impact

None

Upgrade impact

None

The default behavior when the flavor extra_spec and image metadata value is unset was chosen to keep backwards compatiblity on upgrade.

Implementation

Assignee(s)

Primary assignee:

sean-k-mooney

Work Items

  • Extend libvirt driver config module to support the pmu element

  • Extend libvirt dirver to enable/disable the feature based on flavor/image

  • optionally enable vPMU trait.

Dependencies

If we chose to enable the reporting of vPMU emulation as a trait then the consumption of that trait would depend on the completion of https://blueprints.launchpad.net/nova/+spec/image-metadata-prefiltering

The general feature has no dependencies

Testing

This will primarily be tested via unit tests of the xml generation and flavor/image handling code. If the traits support is added functional test using the libvirt fake driver can also be implemented.

Documentation Impact

The flavor and image docs will need to be extended to document the new extra_spec. The Glance metadefs will also be updated to document their use and supported values.

References

None

History

Revisions

Release Name

Description

Train

Introduced