Libvirt file backed memory

Spec for blueprint libvirt-file-backed-memory https://blueprints.launchpad.net/nova/+spec/libvirt-file-backed-memory

With the advent of large capacity memory devices, it’s now reasonable to run virtual machines with file backed memory. This enables a much larger total memory area per compute node

Problem description

New memory technology and modern NVMe SSDs have progressed to the point where it is reasonable to utilize these devices as a backing store for VM memory to expand the memory capacity of a compute host.

Use Cases

As an operator, I want to be able to leverage libvirt’s support for file-backed memory technologies to expand my compute node memory capacity.

Proposed change

The proposed change is to add a new option in nova.conf under the libvirt section to configure file backed memory:

  • file_backed_memory

file_backed_memory will default to 0, indicating file backed memory is disabled. When set to non-zero, the libvirt driver will include elements to enable file backed memory within instances.

When file_backed_memory is set to non-zero, the libvirt driver will include a MemoryBacking element for all instances on the compute node, with a source subelement with type file, and an access subelement with mode shared.

The value configured in file_backed_memory will be reported by the libvirt driver as the total memory capacity of the compute node in MiB. As the memory capacity is dependent on the backing store(s) in use, libvirt must report a value other than the real system memory capacity. Available capacity will then be calculated from the value in file_backed_memory minus the currently used memory for instances. System memory will be used as a cache for the file backed memory through the kernel pagecache.

As overcommit is not expected to work with file backed memory, enabling this option requires the value of ram_allocation_ratio in nova.conf, to be set to the value 1.0, and will block Nova startup if this is not configured properly.

Note

shared access mode is required, as private access will not utilize an underlying backing store for pages in-use by the instance, but will keep those pages within main system memory.

Note

During migration, the destination compute node must be checked for the file_backed_memory option, and add or remove the MemoryBacking element and subelements as appropriate, to ensure memory is appropriately allocated during the migration process. This can be in nova/virt/libvirt/migration.py, within the get_updated_guest_xml method, similar to how graphics, serial, volume, and perf events are handled today.

Note

Migration from compute nodes running versions of Nova without this feature will not include the appropriate libvirt XML for file backed memory. Nova will block these migrations, to ensure all instances migrated to a node with file_backed_memory enabled are actually using file backed memory. Migrations will be blocked within check_can_live_migrate_destination

Alternatives

In Nova, the available memory capacity could be detected dynamically. Due to the variety of memory and SSD technologies and ways the memory backing directory could be configured within libvirt and on the system, this would require implementing an external call point to determine the available capacity, or require vendor-specific code included in Nova. Either option would significantly increase the scope of this spec.

In Nova, file backed memory could be enabled via a flavor extra-spec, allowing for control per individual instance. This results in an inconsistent use of RAM and backing devices, leading to a confusing / conflicting memory capacity calculation on the compute node.

Not implement this spec at all. In this case, a compute node is limited to the capacity of the standard DRAM in the compute node.

Data model impact

None

REST API impact

None

Security impact

As the instance memory will now be contained within a file on a filesystem, instance memory will be accessible to any process owned by a user with permissions necessary to access the files. Currently, the root user on a compute node already has capability to read system memory and VM memory.

Notifications impact

None

Other end user impact

None

Performance Impact

When the file_backed_memory option is enabled, instance memory performance will be dependent on the backing store, as configured by the operator.

Other deployer impact

New config options would need to be explicitly enabled to take effect.

Prior to enabling new config options, an operator should configure the libvirt memory_backing_dir configuration setting to point to their selected backing store, such as a filesystem on an NVMe device.

Enabling file_backed_memory will reject migrations from compute nodes running versions of Nova that do not support file backed memory.

Developer impact

None

Upgrade impact

Enabling file_backed_memory will reject migrations from compute nodes running versions of Nova that do not support file backed memory.

It’s recommended to only enable file_backed_memory after all compute nodes are upgraded.

Implementation

Assignee(s)

Primary assignee:

Zack Cornelius <zcornelius>

Other contributors:

None

Work Items

  • Add new configuration options to nova.conf

  • Check for configuration options within libvirt driver and report the file-backed capacity value instead of system memory

  • Generate additional libvirt domain XML as needed

  • Validate / correct libvirt domain XML during migration process

Dependencies

  • Qemu >= 2.6.0

  • Libvirt >= 4.0.0

Testing

Unit tests will be added to validate the instances booted on host have files in the libvirt memory_backing_dir on the host.

A test will be needed for the edge cases of migrating between a host with the new options enabled, and a host without the new options enabled, to ensure the memory is allocated from the correct source.

We will investigate adding an integration test for this by creating a large ramdisk, enabling these settings, and validating that an instance is utilizing memory within files on that ramdisk. We believe this test layout should be possible, but if not, will fall back to relying on unit tests.

Documentation Impact

The documentation for nova.conf should be updated with the new configuration options.

References

History

Revisions

Release Name

Description

Rocky

Introduced