Make cinder driver compatible with multiple stores¶
https://blueprints.launchpad.net/glance_store/+spec/multiple-cinder-backend-support
From Train onwards Glance is fully supporting configuring multiple glance stores. Glance can configure to use cinder as a store using available cinder driver of glance_store. Cinder makes available volume-types, which describe the characteristics of volumes. Currently, the cinder glance_store can only use one volume-type. The point of glance multi-store implemented in Train is to give operators the ability to expose glance stores of differing characteristics to end users. Even though all images will wind up in a single cinder installation, it is possible for operators to expose different categories of storage in cinder by creating different volume-types. So when a cinder installation exposes multiple volume-types of differing characteristics, what we want to do here is to be able to map different glance ‘stores’ to cinder volume-types.
Problem description¶
1. As of now cinder can configure different volume-types but glance will be able
to use only one of those available volume-types. If glance store is
set to use cinder then every time new image is created it will always be
uploaded to default volume-type unless operator has configured
cinder_volume_type in glance-api.conf file.
2. If cinder store is configured to use in glance then while creating the
image cinder store of glance creates location URL with cinder:// prefix
only. When cinder has configured multiple backends and glance has also
configured multiple cinder stores then it is difficult to analyse
the image is stored in which of the cinder store as the location url
for the image as cinder://<image_id>.
Proposed change¶
We propose that Glance be able to expose multiple cinder stores that differ
by what volume-type each store uses. These would be defined in the normal way
in the glance configuration file using the enabled_backends option
(see example below). Further, when using multiple glance stores, each cinder
store must have the cinder_volume_type option set.
While initializing the store object, glance will validate that volume type
set using cinder_volume_type is exist in cinder. If it’s not then that
store will be excluded by disabling ‘add’ and ‘delete’ operations. To
connect to cinder from glance operator needs to specify
cinder_store_auth_address, cinder_store_user_name,
cinder_store_password and cinder_catalog_info for each of the store
in glance-api.conf file.
Below are some multiple cinder store configuration examples.
Example 1: Fresh deployment
For example, if cinder has configured 2 volume types glance-fast and
glance-slow then glance configuration should look like;:
[DEFAULT]
# list of enabled stores identified by their property group name
enabled_backends = fast:cinder,slow:cinder
# the default store, if not set glance-api service will not start
[glance_store]
default_backend = fast
# conf props for fast store instance
[fast]
rootwrap_config = /etc/glance/rootwrap.conf
cinder_volume_type = glance-fast
description = Really fast and expensive storage
cinder_catalog_info = volumev2::publicURL
cinder_store_auth_address = http://localhost/identity/v3
cinder_store_user_name = glance
cinder_store_password = admin
cinder_store_project_name = service
# etc..
# conf props for slow store instance
[slow]
rootwrap_config = /etc/glance/rootwrap.conf
cinder_volume_type = glance-slow
description = Slower but less expensive storage
cinder_catalog_info = volumev2::publicURL
cinder_store_auth_address = http://localhost/identity/v3
cinder_store_user_name = glance
cinder_store_password = admin
cinder_store_project_name = service
# etc..
Example 2: Upgrade from single cinder store to multiple cinder stores, if default_volume_type is set in cinder.conf and cinder_volume_type is also set in glance-api.conf then administrator needs to create one store in glance where cinder_volume_type same as old glance configuration:
# cinder.conf
The glance administrator has to find out what the default volume-type is
in the cinder installation, so he/she needs to discuss with either cinder
admin or cloud admin to identify default volume-type from cinder and then
explicitly configure that as the value of ``cinder_volume_type``.
Example config before upgrade:
[glance_store]
stores = cinder, file, http
default_store = cinder
cinder_state_transition_timeout = 300
rootwrap_config = /etc/glance/rootwrap.conf
cinder_catalog_info = volumev2::publicURL
cinder_volume_type = glance-old
Example config after upgrade:
[DEFAULT]
enabled_backends = old:cinder, new:cinder
[glance_store]
default_backend = new
[new]
rootwrap_config = /etc/glance/rootwrap.conf
cinder_volume_type = glance-new
description = Newly defined second (cinder) store
cinder_catalog_info = volumev2::publicURL
cinder_store_auth_address = http://localhost/identity/v3
cinder_store_user_name = glance
cinder_store_password = admin
cinder_store_project_name = service
# etc..
[old]
rootwrap_config = /etc/glance/rootwrap.conf
cinder_volume_type = glance-old # as per old cinder.conf
description = Previously existing (cinder) store
cinder_catalog_info = volumev2::publicURL
cinder_store_auth_address = http://localhost/identity/v3
cinder_store_user_name = glance
cinder_store_password = admin
cinder_store_project_name = service
# etc..
Operator can decide on the basis of deployment strategy which volume type they wants to use by coordinating with cinder admin or cloud operator.
We also propose to modify location url for cinder and use
store identifier in location url so that user or operator will
identify in which cinder store of glance image is stored. The new
location URL should looked like cinder://store-name/image-id.
For legacy images stored in cinder backend we will modify the lazy loading mechanism of glance which will update the location URL of existing images as per new format. The lazy loading operation is a check before GET API call which traverse through image location and based on location URI it identifies in which glance store image data is stored and updates that information in location metadata. This mechanism is also useful in a way that if in future operator decides to change the name of the glance store or retire one of the configured store by migrating the images to new stores.
Alternatives¶
None
Data model impact¶
None
REST API impact¶
None
Security impact¶
The security impact is same as it was with single store but we’re just
pointing it out here; The image-volume is stored in the configured
project cinder_store_project_name and can be accessed with configured
user cinder_store_user_name.
There could be a potential risk if someone was able to get a hold of these credentials and access the image-volumes. Worst case is someone could alter the image-volumes if they had permission to perform any cinder operation on it such as retype, attach etc.
Care will have to be taken to ensure it isn’t accessible by normal users.
Notifications impact¶
None
Other end user impact¶
None
Performance Impact¶
After upgrade from single cinder store to use multiple cinder stores first image-list or first get call for image will take additional time as we are performing the lazy loading operation to update legacy image location url to use new image location urls. Subsequent get or list calls will perform as they were performing earlier.
Other deployer impact¶
Operators should be aware of different volume types available in cinder. They
can either use type-list command of cinder client or coordinate with cinder
admin and decide which volume-type of cinder should be configured in
glance-api.conf.
Developer impact¶
None
Implementation¶
Assignee(s)¶
Primary assignee: * whoami-rajat * abhishek-kekane
- Other contributors:
None
Reviewers¶
Core reviewer(s):
jokke
rosmaita
smcginnis
Work Items¶
Modify cinder driver initialization to set new cinder location url
Modify usage of cinder location url
Modify lazy loading mechanism to update legacy image location URLs
Unit tests
Dependencies¶
None
Testing¶
Appropriate unit and functional tests to ensure the changes to glance function correctly. Aslo we could add a job which will run tests using cinder stores in glance.
Documentation Impact¶
We’ll need to ensure that glance/glance_store docs are updated for:
Usage of cinder volume types as cinder stores of glance.
We should also document that, if cinder store is used as glance backend, Only the Image Service API should be used to manipulate images. Manipulating image data directly via the Block Storage Service API is not supported and may lead to adverse consequences, including data loss.
How to upgrade from single cinder store to multiple cinder stores.
References¶
None
