Libvirt: AIO mode for disk devices¶
Libvirt and qemu provide two different modes for asynchronous IO (AIO), “native” and “threads”. Right now nova always uses the default thread mode. Depending on the disk type that is used for backing guest disks, it can be beneficial to use the native IO mode instead.
Storage devices that are presented to instances can be backed by a variety of different storage backends. The storage device can be an image residing in the file system of the hypervisor, it can be a block device which is passed to the guest or it can be provided via a network. Images can have different formats (raw, qcow2 etc.) and block devices can be backed by different hardware (ceph, iSCSI, fibre channel etc.).
These different image formats and block devices require different settings in the hypervisor for optimizing IO performance. Libvirt/qemu offers a configurable asynchronous IO mode which increases performance when it is set correctly for the underlying image/block device type.
Right now nova sticks with the default setting, using userspace threads for asynchronous IO.
A deployer or operator wants to make sure that the users get the best possible IO performance based on the hardware and software stack that is used.
Users may have workloads that depend on optimal disk performance.
Both users and deployers would prefer that the nova libvirt driver automatically picks the asynchronous IO mode that best fits the underlying hardware and software.
The goal is to enhance the nova libvirt driver to let it choose the disk IO mode based on the knowledge it already has about the device in use.
For cinder volumes, different LibvirtVolumeDriver implementations exist for the different storage types. A new interface will be added to let the respective LibvirtVolumeDriver choose the AIO mode.
For ephemeral storage, the XML is generated by LibvirtConfigGuestDisk, which also allows to distinguish between file, block and network attachment of the guest disk.
Restrictions on when to use native AIO mode¶
Native AIO mode will not be enabled for sparse images as it can cause Qemu threads to be blocked when filesystem metadata need to be updated. This issue is much more unlikely to appear when using preallocated images. For the full discussion, see the IRC log in .
AIO mode has no effect if using the in-qemu network clients (any disks that use <disk type=’network’>). It is only relevant if using the in-kernel network drivers (source: danpb)
In the scenarios above, the default AIO mode (threads) will be used.
Cases where AIO mode is beneficial¶
Raw images and pre-allocated images in qcow2 format
Cinder volumes that are located on iSCSI, NFS or FC devices.
Quobyte (reported by Silvan Kaiser)
An alternative implementation would be to let the user specify the AIO mode for disks, similar to the current configurable caching mode which allows to distinguish between file and block devices. However, the AIO mode that fits best for a given storage type does not depend on the workload running in the guest, and it would be beneficial not to bother the operator with additional configuration parameters.
Another option would be to stick with the current approach - using the libvirt/qemu defaults. As there is no single AIO mode that fits best for all storage types, this would leave many users with inefficient settings.
Data model impact¶
No changes to the data model are expected, code changes would only impact the libvirt/qemu driver and persistent data are not affected.
REST API impact¶
Other end user impact¶
IO performance for instances that run on backends which allow to exploit native IO mode will be improved. No adverse effect on other components.
Other deployer impact¶
- Primary assignee:
Collect performance data for comparing AIO modes on different storage types
Implement AIO mode selection for cinder volumes
Implement AIO mode selection for ephemeral storage
Unit tests will be provided that verify the libvirt XML changes generated by this feature.
Also, CI systems that run libvirt/qemu would use the new AIO mode configuration automatically.
Wiki pages that cover IO configuration with libvirt/qemu as a hypervsior should be updated.
 General overview on AIO: http://www.ibm.com/developerworks/library/l-async/
 Best practices: Asynchronous I/O model for KVM guests https://www-01.ibm.com/support/knowledgecenter/linuxonibm/liaat/liaatbpkvmasynchio.htm
 Libvirt and QEMU Performance Tweaks for KVM Guests “http://wiki.mikejung.biz/KVM/_Xen#AIO_Modes”
 qemu irc log http://paste.openstack.org/show/480498/