Real-time guest VMs require compute nodes with a specific configuration to control the sources of latency spikes.
Manual configuration of compute nodes to support real-time guests is possible. However this is complex and time consuming where there is large number of compute nodes to configure.
On a real-time compute node a subset of the available physical CPUs (pCPUs) are isolated and dedicated to real-time tasks. The remaining pCPUs are dedicated to general housekeeping tasks. This requires a real-time Linux Kernel and real-time KVM that allow their housekeeping tasks to be isolated. The real-time and housekeeping pCPUs typically reside on different NUMA nodes.
Huge pages are also reserved for guest VMs to prevent page faults, either via the kernel command line or via sysfs. Sysfs is preferable as it allows the reservation on each individual NUMA node to be set.
A real-time Linux guest VM is partitioned in a similar manner, having one or more real-time virtual CPUs (vCPUs) and one or more general vCPUs to handle the non real-time housekeeping tasks.
A real-time vCPU is pinned to a real-time pCPU while a housekeeping vCPU is pinned to a housekeeping pCPUS.
It is expected that operators would require both real-time and non real-time compute nodes on the same overcloud.
The primary use-case is NFV appliances deployed by the telco community which require strict latency guarantees. Other latency sensitive applications should also benefit.
This spec proposes changes to automate the deployment of real-time capable compute nodes using TripleO.
Worse-case latency in real-time guest VMs should be significantly reduced. However a real-time configuration potentially reduces the overall throughput of a compute node.
The operator will remain responsible for:
Real-time overcloud-full image creation:
create a disk-image-builder element to include the real-time packages
add support for multiple overcloud images in python-tripleoclient CLIs:
openstack overcloud image build openstack overcloud image upload
Real-time compute role:
Ideally this can be implemented outside of TripleO in the Tuned profiles, where it is possible to set the kernel command line and manage sysfs. TripleO would then manage the Tuned profile config files. Alternatively the grub and systemd config files can be managed directly.
This requirement is shared with OVS-DPDK. The development should be coordinated to ensure a single implementation is implemented for both use-cases. Managing the grub config via a UserData script is the current approach used for OVS-DPDK. See OVS-DPDK documentation in References.
As outlined in the proposed changes.
Genuine real-time guests are unlikely to be testable in CI:
However the workflow to deploy these guest should be testable in CI.
Manual steps performed by the operator shall be documented:
BIOS settings for low latency
Real-time overcloud image creation
CentOS repos do not include RT packages. The CERN CentOS RT repository is an alternative.
Flavor and profile creation
Parameters required in a TripleO environment file
Nova blueprint “Libvirt real time instances”
The requirements are similar to Adding OVS-DPDK to Tripleo
CERN CentOS 7 RT repo http://linuxsoft.cern.ch/cern/centos/7/rt/
NoveVcpuPinSet parameter added: https://review.openstack.org/#/c/343770/
OVS-DPDK documentation (work-in-progress): https://review.openstack.org/#/c/395431/