Allow Nova to download Glance images directly via RBD

Problem description

When using compute-local storage with qcow2 based VM root disks, Glance images are downloaded into the libvirt image store by way of the Glance HTTP API. For images in the 10s-100s of GB, this download can be _very_ slow. If the compute node has access to Ceph, it can instead perform an ‘rbd export’ on the Glance image, bypassing the Glance API entirely and directly download the image from Ceph. This direct download can result in a drastic reduction in download time, from tens of minutes to tens of seconds.

Use Cases

As a user with a Ceph-backed image storage, I want to configure some compute hosts for qcow2 images local to the compute host but quickly get the images from Ceph rather than slow downloads from the Glance API.

Proposed change

A special download handler will be registered for Glance images when the ‘rbd’ value is present in allowed_direct_url_schemes option.

This download handler will be called only when a VM is scheduled on a node and the required Glance image is not already present in the local libvirt image cache. It will execute the OS native ‘rbd export’ command, using privsep, in order to perform the download operation instead of using the Glance HTTP API.

The mechanism for per-scheme download handlers was previously available as a plugin point, which is now deprecated, along with the allowed_direct_url_schemes config option. This effort will close out on that deprecation by moving the per-scheme support into the nova.images.glance module itself, undeprecating the allowed_direct_url_schemes config, and removing the old plug point.

The glance module also never used to perform image signature verification when the per-scheme module was used. Since we are moving this into core code, we will also fix this so that per-scheme images are verified like all the rest.


VM root disks can be run directly within Ceph as creation of these VM root disks are fast as they are COW clones for the Glance image, also in Ceph. However, running the VM root disks from Ceph introduces additional latency to the running VM and needlessly wastes network bandwidth and Ceph IOPS. This specific functionality was added in Mitaka but is aimed at a different use case where the VM root disks remain in Ceph and are not run as qcow2 local disks.

The other alternative is to continue with existing approach only.

Data model impact


REST API impact


Security impact


Notifications impact


Other end user impact


Performance Impact


Other deployer impact

As proposed, there are no new configuration items, simply configuration of existing items.

The following configuration options are required to ensure qcow2 local images are downloaded from Ceph and cached on the local compute host:

On the Glance API node in glance-api.conf:


On the Nova compute node in nova.conf:





glance.allowed_direct_url_schemes = rbd

Looking ahead, it may be desired to create additional entries in the libvirt section of nova.conf for this feature as the current implementation assumes that the rbd_user will have access to the Glance images. This may not be the case depending upon how the Ceph pool permissions are configured.

Developer impact

The allowed_direct_url_schemes option was deprecated in Queens. Proposed implementation of this feature would halt the deprecation of this option and we would need to “un-deprecate” it.

Upgrade impact




Primary assignee:

Jiri Suchomel <>

Feature Liaison

Feature liaison:

Dan Smith (danms)

Work Items

  • Refactor existing in-house out-of-tree implementation and integrate it fully into current codebase

  • Write tests for implementation

  • Update the admin guide with the description of how to set up the config if the new feature is required.




  • Unit tests

  • Add an experimental on-demand queue job which uses Ceph with local qcow2 images and ‘direct from rbd’ feature enabled

Documentation Impact

The admin guide should be updated to call out this use case and how it differs from the Ceph-native snapshot feature. A good place to document this may be:




Release Name