Glance Specs

2026.2 Project Priorities

Mon, 09 Mar 2026 00:00:00

TODO(glance-ptl): fill this in after the PTG

Example Spec - The title of your blueprint

Wed, 14 Jan 2026 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/glance/+spec/example

Introduction paragraph – why are we doing anything? A single paragraph of prose that operators can understand.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/glance/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox, or see: https://www.siafoo.net/reST.xml
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Glance REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.openstack.org/#/q/status:open+project:openstack/glance-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem:

For a new feature this might be use cases. Ensure you are clear about the actors in each use case: End User vs Deployer
For a major reworking of something existing it would describe the problems in that feature that are being addressed.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

What other ways could we do this thing? Why aren’t we using those? This doesn’t have to be a full literature review, but it should demonstrate that thought has been put into why the proposed solution is an appropriate one.

Data model impact

Changes which require modifications to the data model often have a wider impact on the system. The community often has strong opinions on how the data model should be evolved, from both a functional and performance perspective. It is therefore important to capture and gain agreement as early as possible on any proposed changes to the data model.

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
Glance is committed to zero-downtime database migrations and has adopted an E-M-C migration strategy to achieve this. Address the following in sufficient detail to make it clear that the intended database change will be achieved.
- Will this change require database triggers? If yes, describe them.
- Explain what your expand migrations will look like.
- Explain what your data migrations will look like.
- Explain what your contract migrations will look like.
- Finally, do these changes have the potential to interfere with the database migrations for other specs that have been approved for this cycle?
How will the initial set of new data objects be generated? For example if you need to take into account existing images, or modify other existing data, describe how that will work.

REST API impact

An /api directory is now included for REST API updates. Each API method which is either added or changed should have the following:

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE/PATCH)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the URL
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Example JSON schema definitions can be found in the Glance tree https://opendev.org/openstack/glance/src/branch/master/glance/api/v2/tasks.py#L342

Note that the schema should be defined as restrictively as possible. Parameters which are required should be marked as such and only under exceptional circumstances should additional parameters which are not defined in the schema be permitted (i.e. additionalProperties should be False).

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

For more detailed guidance, please see the OpenStack Security Guidelines as a reference (https://wiki.openstack.org/wiki/Security/Guidelines). These guidelines are a work in progress and are designed to help you identify security best practices. For further information, feel free to reach out to the OpenStack Security Group at openstack-security@lists.openstack.org.

Notifications impact

Please specify any changes to notifications. This includes introduction of a new notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-glanceclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system. How often will new code be called? Is there a major change to the calling pattern of existing code?

Examples of things to consider here include:

A small change in a utility function or a commonly used decorator can have a large impact on performance.
Calls which result in database queries can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that widgets are stored in, how do we handle widget directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the widgets in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the store API, discussion of how stores would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Work items or tasks – break the feature up into the things that need to be done to implement it. Those parts might end up being done by different people, but we’re mostly trying to understand the timeline for implementation.

Dependencies

Include specific references to specs and/or blueprints in glance, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Glance: document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Please discuss how the change will be tested. We especially want to know what tempest tests will be added. It is assumed that unit test coverage will be added so that doesn’t need to be mentioned explicitly, but discussion of why you think unit tests are sufficient and we don’t need to add more tempest tests would need to be included.

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

What is the impact on the docs team of this change? Some changes might require donating resources to the docs team to have the documentation updated. Don’t repeat details discussed above, but please reference them here.

References

Please add any useful references here. You are not required to have any reference. Moreover, this specification should still make sense when your references are unavailable. Examples of what you could include are:

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g., if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

Example Spec - The title of your blueprint

Wed, 14 Jan 2026 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/glance/+spec/example

Introduction paragraph – why are we doing anything? A single paragraph of prose that operators can understand.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/glance/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox, or see: https://www.siafoo.net/reST.xml
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Glance REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.openstack.org/#/q/status:open+project:openstack/glance-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem:

For a new feature this might be use cases. Ensure you are clear about the actors in each use case: End User vs Deployer
For a major reworking of something existing it would describe the problems in that feature that are being addressed.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
Glance is committed to zero-downtime database migrations and has adopted an E-M-C migration strategy to achieve this. Address the following in sufficient detail to make it clear that the intended database change will be achieved.
- Will this change require database triggers? If yes, describe them.
- Explain what your expand migrations will look like.
- Explain what your data migrations will look like.
- Explain what your contract migrations will look like.
- Finally, do these changes have the potential to interfere with the database migrations for other specs that have been approved for this cycle?
How will the initial set of new data objects be generated? For example if you need to take into account existing images, or modify other existing data, describe how that will work.

REST API impact

An /api directory is now included for REST API updates. Each API method which is either added or changed should have the following:

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE/PATCH)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the URL
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Example JSON schema definitions can be found in the Glance tree https://opendev.org/openstack/glance/src/branch/master/glance/api/v2/tasks.py#L342

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. This includes introduction of a new notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-glanceclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system. How often will new code be called? Is there a major change to the calling pattern of existing code?

Examples of things to consider here include:

A small change in a utility function or a commonly used decorator can have a large impact on performance.
Calls which result in database queries can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that widgets are stored in, how do we handle widget directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the widgets in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the store API, discussion of how stores would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in glance, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Glance: document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g., if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

Add support to configure weight for each store

Sun, 11 Jan 2026 00:00:00

https://blueprints.launchpad.net/glance-store/+spec/store-weight

Add support to configure weight to each store. The store with highest weight will be given preference to download the image.

Problem description

Since introduction of importing single image into multiple stores or copying existing image into multiple stores, single image can be stored at multiple locations or different stores configured by glance. Current download image is based on the default location_strategy which is location_order, traverse through the image locations one by one if there are multiple locations (in this case it will return the image stored at the first location). If user prefers to download the image from a specific store then they can use store_type as location strategy to give preference to download the image from that store only. For example if location_strategy is set as store_type and store_type has rbd as preference then the image will be downloaded from the rbd store only. Now the problem with store_type location strategy is that there can be multiple stores of same type (multiple rbd or file stores). So again user will not able to download the image from the specific store even if location_strategy is set to store_type.

Consider the following use cases for providing download from specific store support:

I have a large image and want to download it from the SSD store since it’s fast.
I do multiple concurrent downloads on a particular image so want to download it from the RELIABLE store since the i/o handling is better.

Proposed change

This proposal requires changes in glance_store as well as in glance.

Glance store side change:

Add new configuration option weight default to zero for each store. Operator can change it for each store if they wish. The store with highest weight will be given preference to download the image from.

[default]
enabled_backends = robust:rbd,cheap:file

[robust]
rbd_store_pool=images
weight=10

[cheap]
filesystem_store_datadir=/opt/stack/data/glance/images/
weight=5

In above example, robust store will always be a preferred store to download the image from. If image is not available in robust store then it will be searched in cheap store which is next inline.

If no weight is provided for stores (i.e. all stores have default weight 0) then the image will be searched based on image creation order (similar as location_order).

Glance side change:

Once glance_store side changes are implemented then we need to modify GET API of image to sort the image locations based on the wegith assigned to each store. If weight is default then the location order will not be changed.

Alternatives

Add new location strategy ‘store_identifier’ to existing default two strategies. This will add comma separated list of store identifiers which will be given preference to download the image from. New configuration option store_identifier_preference under group store_identifier_location_strategy will be added where user/deployer can reference their preferences based on store identifiers.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Deployer/Operator need to configure weight for each glance-store.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: dansmith
Other contributors:: abhishek-kekane

Work Items

Add new configuration option weight for each store
GET API change to sort locations based on store weight
Unit and functional tests
Tempest coverage

Dependencies

None

Testing

Sufficient unit/functional and tempest tests will be added.

Documentation Impact

Need to document how location order will be changed based on weight assigned to each store.

References

None

Glance as first-line defense for image format attacks

Sun, 11 Jan 2026 00:00:00

https://blueprints.launchpad.net/glance/+spec/glance-as-defender

Glance is the point of entry for images into the cloud. It is the orifice through which an untrusted (but authenticated) user brings an image into the system, after which it will be processed by backend routines either for format conversion or preparation for boot.

It is at this stage in the pipeline where we are best positioned to do sanity checking about the images we accept, and the point at which we can validate some of the metadata provided by the user to ensure that downstream services (such as nova, glance, ironic and others) can reasonably assume that metadata to be correct.

Problem description

Glance will (for the most part) allow a user to upload literally content and declare its format to be any of the valid values we have for disk_format and container_format. This is certainly surprising to downstream services, external consumers, and humans which expect the format stated on the image to be in line with the actual content.

A further problem is in the use of the raw value for disk_format. In general, we use raw to mean “a byte-for-byte image of a block device”, usually with a partition table and often a bootloader (in the case of a root disk). However, in reality raw has come to mean “anything in a format for which we do not have another name”. This catch-all behavior means that if we want to support images of full-disk formats, we also need to support “anything else we don’t know about” to some degree.

Proposed change

This spec proposes two major changes to glance:

First, we will start enforcing that the format of the uploaded content matches the disk_format declared on the image. Since our format_inspector module is already in the data pipeline for both upload and import we simply need to remove the “never fail” behavior we currently have and abort the process if we determine that the format does not match what was claimed. Consider the following two examples:

An image is declared to be of format qcow2 but the content uploaded is something else (either another complex format such as vmdk or something we do not recognize).
An image is declared to be of format raw but the content uploaded is detected as a qcow2.

Ideally, we would reject both of these cases. However, the second is more complex because there are situations where a service handling a disk image that it does not assume any particular format may be stopped from storing it in glance if the user of that image has given it a specific format. Thus, for the first iteration of this work, we will only enforce the first case, which means that raw could contain a qcow2, but a vmdk could not.

The use of [image_format]/disk_formats will effectively allow an admin to limit the types of disks they accept. Today, that only limits “honest” users, but this change will make it enforced on content as well.

The second major change is significant in terms of the model and behavior of existing users, but is small in absolute terms and impact to glance itself. A new disk_format option of gpt will be added, which will henceforth serve to signify that “this is an image of a raw block device with a partition table” thus removing our need to overlap that definition with “this is something we do not recognize” for raw. The definition of gpt is actually a superset of the legacy PC MBR (Master Boot Record) format, which means an inspector for this format should have no problem detecting and allowing images of very old (read: Windows XP/2003 vintage) disk images. Thus, many of the images we now legitimately use raw for will be (and need to be converted to be) a disk_format of gpt going forward.

We will add a configuration option to disable this behavior as a relief valve to support migration to this stricter model and/or to account for false positive detections.

require_image_format_match: Default to true, but allow setting to false to avoid aborting the upload/import if the format does not match the content.

Alternatives

Glance could continue to be ambivalent about the content uploaded and the mismatch between that and the metadata it stores. Services like nova and cinder will have to continue treating glance as untrustworthy and remain highly suspicious of its metadata.

Data model impact

The only data model change that should be required here is one to allow the new disk_format value of gpt to be specified in the API and stored in the database.

There will, of course, be a need to convert existing raw images in the database to gpt, and thus some tooling will be required. Options for that could be a glance-manage command to automatically (or manually) do this, or allow it to be done through the API. Alternately, we could also annotate existing images and have the API report them as gpt if the client is determined to be new enough.

REST API impact

The main REST API impact comes from allowing gpt as one of the valid options for disk_format. Additional impact could come if we decide to provide format conversion (or reporting) through the API.

Security impact

In general, this will improve security for the entire cloud by allowing nova, cinder, and other users of glance some amount of trust in the image content and associated metadata. It is important to avoid the other services thinking they no longer need to inspect image content entirely. Security for this sort of thing is best provided in layers and services need to continue to be vigilant about images they download from glance, certainly applying context-specific checks before using them.

Notifications impact

None (aside from the new format).

Other end user impact

Users will defintitely be impacted as the muscle memory of (over) using raw as both a catch-all and as meaning “an image of a whole disk” will take some time to un-learn.

Performance Impact

We are already using format_inspector in the data pipeline. We will need to run all the inspectors in parallel instead of just the declared-format one we are currently using. However, these are designed to be as memory-efficient as possible, and thus the overhead should be minimal. Actual performance of the upload itself should not be impacted.

Other deployer impact

Deployers will have work to do here for sure, specifically as existing disks marked as raw will (mostly) need to be converted to gpt. We can not just convert any raw, as many of those may be kernel images, or other formats that we do not (but probably need to) support identifying.

Options for that would be:

Tell operators to just do it themselves and provide some way to change the disk_format of an image in the database.
Provide a tool to detect and convert the images based on content. We can, in many cases, do this without seeing the whole image, as things that should be gpt will be identified within the first sector or two of the content. This could be used only for converting raw to gpt, but could be written generically in a way that allows operators to audit all their images to make sure they are in the format they claim to be.

Developer impact

No specific impact, although as more formats need to be supported, additional inspector modules will need to be written.

Implementation

Assignee(s)

Primary assignee:: danms

Work Items

Make the format_inspector in the data pipeline detect all formats and abort the upload if the format is determined and does not match
Make glance depend on oslo.utils for format_inspector
Add gpt as a valid disk_format
Write tooling for converting raw disks to gpt in the database
Add config options for new/fallback behaviors

Dependencies

The gpt part of this depends on the oslo port of the inspector code.

Testing

We will need negative tempest tests for format mismatches, which can be written without much drama. Most of the formats require just a few 512-byte sectors of data to be detected and we can generate those inline in tempest tests to make sure that glance rejects mismatches.

Documentation Impact

We will definitely need documentation about the raw-to-gpt behavior change, and we could definitely use better documentation about when to use raw, which will be easier to explain in the context of gpt.

References

S3 Store Boto3 Checksum Behavior Configuration

Sun, 11 Jan 2026 00:00:00

https://blueprints.launchpad.net/glance-store/+spec/s3-boto3-checksum-improvements

Add config options to control boto3 S3 checksum behavior. This fix problems with S3-compatible storage that came with boto3 1.36.x. Operators can now keep using old S3-compatible storage and also use new AWS S3 data integrity features.

Problem description

Boto3 1.36.0 added new S3 “data integrity” protections that run by default. This means boto3 now calculates checksums for requests and validates checksums in responses. This is good for data integrity but many S3-compatible storage systems do not support this yet. For example, Ceph RGW does not work with these new features.

This breaks Glance image uploads to S3-compatible storage. Users see errors like this:

botocore.exceptions.ClientError: An error occurred (MissingContentLength)
when calling the PutObject operation: The internal error code is -2011

This problem is in Launchpad bug #2121144. Users who upgrade from Dalmatian to Epoxy see this problem because boto3 upgraded from 1.35.x to 1.36.x.

Current workaround is to set environment variables:

export AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED
export AWS_RESPONSE_CHECKSUM_VALIDATION=WHEN_REQUIRED

But this is not good because it affects all boto3 usage in the system, not only Glance. This is not good for production with different storage backends.

Proposed change

Add three new config options to S3 store driver to control boto3 checksum behavior:

s3_store_enable_data_integrity_protection (boolean, default: false) - Turn on/off boto3 data integrity features - When false: Use ‘when_required’ behavior for compatibility - When true: Use the configured checksum settings
s3_store_request_checksum_calculation (string, default: ‘when_required’) - Control when boto3 calculates request checksums - Options: ‘when_required’, ‘always’, ‘never’
s3_store_response_checksum_validation (string, default: ‘when_required’) - Control when boto3 validates response checksums - Options: ‘when_required’, ‘always’, ‘never’

We will change the _create_s3_client method in S3 store driver to use these config options when creating boto3 S3 client.

Alternatives

Environment Variables: Current workaround with environment variables is not good for production because it affects all boto3 usage globally.
Boto3 Version Pinning: Downgrade boto3 version will lose other improvements and security fixes in newer versions.
Storage Backend Detection: Automatically detect S3-compatible vs AWS S3 is not reliable and has many errors.

Config-based approach gives most flexibility and control. It also keeps backward compatibility.

Data model impact

None

REST API impact

None

Security impact

This change is good for security. Operators can enable better data integrity protection for AWS S3 deployments. At same time, they can keep compatibility with S3-compatible storage backends. Default behavior keeps current security for existing deployments.

Notifications impact

None

Other end user impact

None

Performance Impact

Change has very small performance impact:

When data integrity protection is disabled (default), performance is same as current behavior
When enabled, there is small overhead from checksum calculation and validation, but this gives better data integrity

Other deployer impact

This change works immediately after merge. For existing deployments:

No changes required - Default behavior works with existing S3-compatible storage backends
For AWS S3 deployments wanting better data integrity: Set s3_store_enable_data_integrity_protection = true and configure checksum behavior
For mixed environments: Use different backends with different configs

Developer impact

This change adds new config options to S3 store driver and changes client creation logic. Other developers working on Glance store drivers should know about this new config pattern for handling boto3 compatibility problems.

Implementation

Assignee(s)

Primary assignee:: abhishekk
Other contributors:: None

Work Items

Add new config options to S3 store driver and use them in client creation logic
Add unit tests for all configuration scenarios
Add multistore tests for different backend configurations
Update s3 documentation

Dependencies

This change needs boto3 1.36.x or later for the new checksum config options
No new library dependencies needed

Testing

Add unit and tempest tests to cover all configuration scenarios.

Documentation Impact

Update configuration reference documentation to explain the new S3 store config options. Add configuration examples and migration guidance to help operators understand how to configure these options for their use cases.

References

Launchpad Bug #2121144: Image uploads to S3-compatible storage fail with Epoxy version
boto3 Issue #4392: S3 data integrity protections enabled by default
boto3 Issue #4398: Guidance on checksum behavior configuration

Add bash completion for command line

Sun, 11 Jan 2026 00:00:00

https://blueprints.launchpad.net/python-glanceclient/+spec/add-bash-completion

Currently glance client does not support command completion. The intention is to add this functionality to the client.

Problem description

Nowadays glance client does not have bash completion for the command line. This feature will improve the client usability.

Proposed change

Incorporation of bash completion feature. This feature uses the linux script called bash_completion.d to request all the commands and parameters to Glance through bash_completion command. After obtaining them, they are filtered and shown.

Example of use:

glance <tab><tab> —>shows all the commands

glance image- <tab><tab> —>shows all the commands starting with the word ‘image’: image-list, image-show, image-create…

If there is only one, it will be completed.

glance image-create <tab><tab> —> shows all optional arguments

To complete this feature, it is needed to modify devstack and the packaging tool in order to include this script in the installer (deb, rpm).

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

This change needs copying of the glance.bash_completion to specific system path:

This will affect devstack
This will affect specific packaging (deb, rpm)

Developer impact

Commands and Parameters are extracted directly from argparse. Developers won’t need to maintain a separate list when they add new commands or parameters.

Implementation

Assignee(s)

Primary assignee:: <juan-m-olle>

Work Items

This feature needs:

To add glance.bash_completion script
To add bash_completion command to glance client. This new command returns all available commands the client has and it is used by glance.bash_completion.

Dependencies

None

Testing

Unit test will check new shell command functionality.

Documentation Impact

New feature needs to be mentioned.

References

None

Metadata Definitions Catalog

Sun, 11 Jan 2026 00:00:00

https://blueprints.launchpad.net/glance/+spec/metadata-schema-catalog

A common API hosted by the Glance service for vendors, admins, services, and users to meaningfully define available key / value pair and tag metadata. The intent is to enable better metadata collaboration across artifacts, services, and projects for OpenStack users.

This is about the definition of the available metadata that can be used on different types of resources (images, artifacts, volumes, flavors, aggregates, etc). A definition includes the properties type, its key, it’s description, and it’s constraints. This catalog will not store the values for specific instance properties.

For example, a definition of a virtual CPU topology property for number of cores will include the key to use, a description, and value constraints like requiring it to be an integer. So, a user, potentially through Horizon, would be able to search this catalog to list the available properties they can add to a flavor or image. They will see the virtual CPU topology property in the list and know that it must be an integer. In the Horizon example, when the user adds the property, its key and value will be stored in the service that owns that resource (Nova for flavors and in Glance for images).

Diagram: https://wiki.openstack.org/w/images/b/bb/Glance-Metadata-API.png

Problem description

A challenge we’ve experienced with using OpenStack is discovering, sharing, and correlating metadata across services and different types of resources. We believe this affects both end users and administrators.

Various OpenStack services provide techniques to abstract low level resource selection to one level higher, such as flavors, volume types, or artifact types. These resource abstractions often allow “metadata” in terms of tags or key-value pairs to further specialize and describe instances of each resource type. However, collaborating and understanding what metadata to use on each type of resource can be a disconnected and difficult process. This often involves searching wikis and opening the source code. There is no common way for vendors or operators to publish their metadata definitions. It becomes more difficult as a cloud’s scale grows and the number of resources being managed increases.

In addition, cloud operators should be able to selectively control the property metadata that they want to be visible in the CLI or horizon UI. Just because an enabled driver or scheduler filter supports a certain property doesn’t mean that the cloud operator wants that property to be readily visible for selection in the UI.

Background and Examples

At the Juno Atlanta summit, the Graffiti team demonstrated the concepts running under a POC. The following video is a recap of what was demo’d at the summit and helps to set context:

https://www.youtube.com/watch?v=Dhrthnq1bnw

We received very positive feedback from members of multiple dev projects as well as numerous operators. We were specifically asked multiple times about getting the metadata definition catalog concepts into Glance so that we can start to officially support the ideas we demonstrated in Horizon.

Additional examples are at the end of this document.

Terminology

The term metadata can become very overloaded and confusing. This proposed catalog is about the additional metadata that is passed as either arbitrary key / value pairs or assigned as “tags” (name only) across various artifacts and OpenStack services.

Different APIs may use tags and key / value pairs differently. Tags often are not used to drive runtime behavior. However, key / value pairs are often used by the system to potentially drive runtime behavior, such as scheduling, quality of service, or driver behavior. Other times, key / value pairs are only intended for end user use and are not directly used to drive runtime behavior. The metadata may also be used by an external entity such as a 3rd party policy engine. Some service APIs mix the metadata into a single bucket and don’t differentiate between the two uses of the metadata.

The terminology for key / value pairs varies across services.

A few examples of metadata today:

Nova	Cinder	Glance
Flavor extra specs Host Aggregate metadata Instances metadata tags	Volume & Snapshot image metadata metadata VolumeType extra specs qos specs	Image & Snapshot properties tags

In this proposal, we use the term “object” to describe a group of 1…* key / value pairs that may be applied to different kinds of resources for different purposes.

Relationship to proposed artifacts API

This is not about artifact storage, such as Heat templates or various application packages. This API is about the additional metadata that is beyond the syntax of the base artifact. Once defined, the metadata can often be applied to different types of artifacts and resources. For example, application category tags like “Big Data” may apply to an image or volume that provides Hadoop or may apply to a Host Aggregate or Flavor that is suitable for “Big Data” applications.

Proposed change

We are proposing a new Metadata Definitions Catalog. The following subsections detail the concepts managed by the catalog.

Properties (Juno release)

A property describes a single property and its primitive constraints. Each property can ONLY be a primitive type:

string, integer, number, boolean, array

Each primitive type is described using simple JSON schema notation. This means NO nested objects and no definition referencing.

Objects (Juno release)

An object describes a group of one to many properties and their primitive constraints. Each property in the group can ONLY be a primitive type:

string, integer, number, boolean, array

Each primitive type is described using simple JSON schema notation. This means NO nested objects.

The object may optionally define required properties under the semantic understanding that a user who uses the object should provide all required properties.

Object derivation with property inheritance was considered, but we will defer this complexity until it is proven to be necessary.

Namespaces (Juno release)

Metadata definitions are contained in namespaces.

Specify the access controls (CRUD) for everything defined in it. Allows for admin only, different projects, or the entire cloud to define and use the definitions in the namespace
Associates the contained definitions to different types of resources

Tags (Future release)

A catalog of possible tags that can be used to help ensure tag name consistency across users, resource types, and services. So, when a user goes to apply a tag on a resource, they will be able to either create new tags or choose from tags that have been used elsewhere in the system on different types of resources. For example, the same tag could be used for Images, Volumes, and Instances. Tags are case insensitive (BigData is equivalent to bigdata but is different from Big-Data).

Tag Hierarchy (Future release)

The notion of hierarchy has come up in various application related discussions. It is very simple to support hierarchy. For example, tags of “MySQL” and “Postgres” could be created and set as having a parent of the “Database” tag. A user could then tag just “MySQL” on something like an image or software template. Subsequent searches for resources could be performed for all “Databases” by simply retrieving the list of tags that are children of the “Database” tag.

Diagram:

https://wiki.openstack.org/w/images/6/61/Glance-Metadata-Namespace.png

If needed in the future, it may make sense for there to be a default public namespace visible to all cloud users for all resource types. Having a default public namespace will make it very easy to manage general metadata without the overhead of a full multi-tenant environment.

Resource Type Association (Juno release)

Resource type association specifies the relationship between resource types and the namespaces that are applicable to them. This information can be used to drive UI and CLI views. For example, the same namespace of objects, properties, and tags may be used for images, snapshots, volumes, and flavors. Or a namespace may only apply to images.

Resource types should be aligned with Heat resource types. http://docs.openstack.org/developer/heat/template_guide/openstack.html

It is important to note that the same base property key can require different prefixes depending on the target resource type. Below are a few examples:

The desired virtual CPU topology can be set on both images and flavors via metadata. The keys have different prefixes on images than on flavors. On flavors keys are prefixed with hw:, but on images the keys are prefixed with hw_.

For more: http://git.openstack.org/cgit/openstack/nova-specs/tree/specs/juno/virt-driver-vcpu-topology.rst

Another example is the AggregateInstanceExtraSpecsFilter and scoped properties (e.g. properties with something:something=value). For scoped / namespaced properties, the AggregateInstanceExtraSpecsFilter requires a prefix of “aggregate_instance_extra_specs:” to be used on flavors but not on the aggregate itself. Otherwise, the filter will not evaluate the property during scheduling.

So, on a host aggregate, you may see:

companyx:fastio=true

But then when used on the flavor, the AggregateInstanceExtraSpecsFilter needs:

aggregate_instance_extra_specs:companyx:fastio=true

In some cases, there may be multiple different filters that may use the same property with different prefixes. In this case, the correct prefix needs to be set based on which filter is enabled.

This spec handles the above cases.

Alternatives

This could be done as a completely separate service. However, it was suggested by numerous community members at the Juno summit that it made sense to be part of the expanded Glance mission, since many of the primary targets of the metadata would be artifacts hosted in Glance. In addition, this also allows the associated UI components to be built out in Horizon natively rather than as plug-ins.

We also discussed a Horizon only solution with the Horizon PTL and found some technical reasons why that wouldn’t make sense:

Horizon is a stateless server by design at this point. The only place any persistent data can exist is if you choose to store session information on the server in a database. The default setup for Horizon now uses signed cookies to maintain session data and avoids a DB requirement.

There is no privileged account running on the Horizon server and thus no way to build a persistent datastore only the admin can obtain. A persistent privileged session as this creates many security issues.

Horizon can be set up in an HA manner, which would require either duplicate DB on multiple Horizon servers or another server dedicated to the DB backend for Horizon.

Services may always support a way to discover their available metadata. This API does not prevent that from occurring. However, this does provide a single central API to publish and discover metadata without every service having to implement such a facility. In addition, cloud operators should be able to selectively control the property metadata that they want to be visible in the CLI or horizon UI. Just because an enabled driver or scheduler filter supports a certain property doesn’t mean that the cloud operator wants that property to be readily visible for selection in the UI. This API allows for cloud operators to have full control over what is made visible.

A key use case is the collaboration on metadata using a common catalog. This is complementary to tags and key / value pairs being added ad-hoc across all services. We think in the future the metadata API could also be backed by a search indexer across services to include ad-hoc metadata as well as defined metadata. However, that is not the focus of this blueprint.

Data model impact

This will use a relational database and exist in the same database as the existing Glance relational data, but there is not anticipated impact to existing Glance data models. This is all new functionality.

This is about the definition of the available metadata that can be used on different types of resources (images, artifacts, volumes, flavors, aggregates, etc). A definition is not just a key and value, so we will not be using a key / value store database. The definition includes the properties type, its key, it’s description, and it’s constraints. When metadata is used on a resource, a key with user supplied value will be stored on whatever service owns that resource and is out of the scope of this spec. For example, an instance of a key / value pair would be in the Cinder database or the Glance registry.

Support will be added to: * glance/db/sqlalchemy/api.py * registry/api.py * simple/api.py

A new package will be added at glance/db/sqlalchemy/metadata_defs_api

The table classes will be in glance/db/sqlalchemy/models_metadata_defs.py

The following DB schema is the initial suggested schema. We will improve and take comments during code review. Constraints not shown for readability.

Suggested Basic Schema:

Table: metadef_namespaces

+--------------+--------------+------+-----+----------------+
| Field        | Type         | Null | Key | Extra          |
+--------------+--------------+------+-----+----------------+
| id           | int(11)      | NO   | PRI | auto_increment |
| namespace    | varchar(80)  | NO   | UNI |                |
| display_name | varchar(80)  | YES  |     |                |
| description  | text         | YES  |     |                |
| visibility   | varchar(32)  | YES  |     |                |
| protected    | tinyint(1)   | YES  |     |                |
| owner        | varchar(255) | NO   |     |                |
| created_at   | datetime     | NO   |     |                |
| updated_at   | datetime     | YES  |     |                |
+--------------+--------------+------+-----+----------------+

Table: metadef_objects

+--------------+-------------+------+-----+----------------+
| Field        | Type        | Null | Key | Extra          |
+--------------+-------------+------+-----+----------------+
| id           | int(11)     | NO   | PRI | auto_increment |
| namespace_id | int(11)     | NO   | MUL |                |
| name         | varchar(80) | NO   |     |                |
| description  | text        | YES  |     |                |
| required     | text        | YES  |     |                |
| json_schema  | text        | YES  |     |                |
| created_at   | datetime    | NO   |     |                |
| updated_at   | datetime    | YES  |     |                |
+--------------+-------------+------+-----+----------------+

Table: metadef_properties

+--------------+-------------+------+-----+----------------+
| Field        | Type        | Null | Key | Extra          |
+--------------+-------------+------+-----+----------------+
| id           | int(11)     | NO   | PRI | auto_increment |
| namespace_id | int(11)     | NO   | MUL |                |
| name         | varchar(80) | NO   |     |                |
| json_schema  | text        | YES  |     |                |
| created_at   | datetime    | NO   |     |                |
| updated_at   | datetime    | YES  |     |                |
+--------------+-------------+------+-----+----------------+

Table: metadef_resource_types

+------------+-------------+------+-----+----------------+
| Field      | Type        | Null | Key | Extra          |
+------------+-------------+------+-----+----------------+
| id         | int(11)     | NO   | PRI | auto_increment |
| name       | varchar(80) | NO   | UNI |                |
| protected  | tinyint(1)  | NO   |     |                |
| created_at | datetime    | NO   |     |                |
| updated_at | datetime    | YES  |     |                |
+------------+-------------+------+-----+----------------+

Table: metadef_namespace_resource_types

+-------------------+-------------+------+-----+-------+
| Field             | Type        | Null | Key | Extra |
+-------------------+-------------+------+-----+-------+
| resource_type_id  | int(11)     | NO   | PRI |       |
| namespace_id      | int(11)     | NO   | PRI |       |
| properties_target | varchar(80) | YES  |     |       |
| prefix            | varchar(80) | YES  |     |       |
| created_at        | datetime    | NO   |     |       |
| updated_at        | datetime    | YES  |     |       |
+-------------------+-------------+------+-----+-------+

REST API impact

In the REST API everything is referred by namespace and name rather than synthetic IDs. This helps to achieve portability (import / export using JSON).

APIs should allow coarse grain and fine grain access to information in order to control data transfer bandwidth requirements.

Working with Namespaces Basic interaction is:

Get list of namespaces with overview info based on the desired filters. (e.g. key / values for images).

Get objects

Common Response Codes

Create Success: 201 Created
Modify Success: 200 OK
Delete Success: 204 No Content
Failure: 400 Bad Request with details.
Forbidden: 403 Forbidden
Not found: 404 Not found e.g. if specific entity not found
Not found: 405 Not allowed e.g. if trying to delete on a list resource
Not found: 501 Not Implemented e.g. HEAD not implemented

API Version

All URLS will be under the v2 Glance API. If it is not explicitly specified assume /v2/<url>

Create a namespace:: POST /metadefs/namespaces/

Namespace may optionally contain the following in addition to basic fields.

resource_type_associations
properties
objects
tags (future release)

Example Body (with no resource types, properties, objects, or tags):

{
  "namespace": "MyNamespace",
  "display_name": "My User Friendly Namespace",
  "description": "My description",
  "visibility": "public",
  "protected": true
}

Replace a namespace definition (not including properties, objects, or tags):

PUT /metadefs/namespaces/{namespace}

List Namespaces: Returns just the list of namespaces without any objects

properties, or tags.: GET /metadefs/namespaces/

Example Body:

{
    "namespaces": [
        {namespace1Here},
        {namespace2Here}
    ],
    "first": "/v2/metadefs/namespaces?limit=2",
    "next": "/v2/metadefs/namespaces?marker=namespace2Here&limit=2",
    "schema": "/v2/schemas/metadefs/namespaces"
}

With example namespace:

{
    "first": "/v2/metadefs/namespaces?sort_key=created_at&sort_dir=asc",
    "namespaces": [
        {
            "namespace": "OS::Compute::Quota",
            "display_name": "Flavor Quota",
            "description": "Compute drivers may enable quotas on...",
            "visibility": "public",
            "protected": true,
            "owner": "admin",
            "resource_type_associations": [
                {
                    "name": "OS::Nova::Flavor",
                    "created_at": "2014-08-28T17:13:06Z",
                    "updated_at": "2014-08-28T17:13:06Z"
                }
            ],
            "created_at": "2014-08-28T17:13:06Z",
            "updated_at": "2014-08-28T17:13:06Z",
            "self": "/v2/metadefs/namespaces/OS::Compute::Quota",
            "schema": "/v2/schemas/metadefs/namespace"
        },
        {
            "namespace": "OS::Compute::VirtCPUTopology",
            "display_name": "Virtual CPU Topology",
            "description": "This provides the preferred...",
            "visibility": "public",
            "protected": true,
            "owner": "admin",
            "resource_type_associations": [
                {
                    "name": "OS::Glance::Image",
                    "prefix": "hw_",
                    "created_at": "2014-08-28T17:13:06Z",
                    "updated_at": "2014-08-28T17:13:06Z"
                },
                {
                    "name": "OS::Cinder::Volume",
                    "prefix": "hw_",
                    "properties_target": "image",
                    "created_at": "2014-08-28T17:13:06Z",
                    "updated_at": "2014-08-28T17:13:06Z"
                },
                {
                    "name": "OS::Nova::Flavor",
                    "prefix": "hw:",
                    "created_at": "2014-08-28T17:13:06Z",
                    "updated_at": "2014-08-28T17:13:06Z"
                }
            ],
            "created_at": "2014-08-28T17:13:06Z",
            "updated_at": "2014-08-28T17:13:06Z",
            "self": "/v2/metadefs/namespaces/OS::Compute::VirtCPUTopology",
            "schema": "/v2/schemas/metadefs/namespace"
        }
    ],
    "schema": "/v2/schemas/metadefs/namespaces"
}

Filter by adding query parameters:

resource_types = <comma separated list> e.g. OS::Glance::Image
visibility     = Valid values are public, private.
                 Default is to return both public namespaces and private
                 namespaces visible to the user making the request.
limit          = Use to request a specific page size. Expect a response
                 to a limited request to return between zero and limit items.
marker         = Specifies the namespace of the last-seen namespace.
                 The typical pattern of limit and marker is to make an initial
                 limited request and then to use the last namespace from the
                 response as the marker parameter in a subsequent limited
                 request.

Returns specific namespace including metadata definitions (properties,

objects or tags).: GET /metadefs/namespaces/{namespace}

Query parameters:

resource_type  = When specified, the API will look up the prefix associated
                 with the specified resource type and will apply the prefix
                 to all properties (including object properties) prior to
                 returning the namespace. For example, if a
                 resource_type_association in the namespace for
                 OS::Nova::Flavor specifies a prefix of hw:, then all
                 properties in the namespace will be returned with
                 hw:<prop_name>. However if, OS::Glance::Image is specified
                 and the prefix is set to hw_, then the property will be
                 returned as hw_<prop_name>.

Example Body:

{
    "namespace": "MyNamespace",
    "display_name": "My User Friendly Namespace",
    "description": "My description",
    "resource_type_associations" : [
        {
           "name" :"OS::Nova::Aggregate",
           "created_at": "2014-08-28T17:13:06Z",
           "updated_at": "2014-08-28T17:13:06Z"
        },
        {
           "name" : "OS::Nova::Flavor",
           "prefix" : "aggregate_instance_extra_specs:",
           "created_at": "2014-08-28T17:13:06Z",
           "updated_at": "2014-08-28T17:13:06Z"
        }
    ],
    "properties": {
        "nsprop1": {
            "title": "My namespace property1",
            "description": "More info here",
            "type": "boolean",
            "default": true
        }
    },
    "visibility": "public",
    "protected": true,
    "owner": "The Test Owner"
}

Delete a namespace including all content (properties, objects and tags(future release)): DELETE /v2/metadefs/namespaces/{namespace}
List resource types associated with a namespace: GET /v2/metadefs/namespaces/{namespace}/resource_types

Example:

{
  "resource_type_associations": [
      {
          "name": "OS::Glance::Image",
          "prefix": "hw_",
          "created_at": "2014-08-28T17:13:06Z",
          "updated_at": "2014-08-28T17:13:06Z"
      },
      {
          "name": "OS::Cinder::Volume",
          "prefix": "hw_",
          "properties_target": "image_metadata",
          "created_at": "2014-08-28T17:13:06Z",
          "updated_at": "2014-08-28T17:13:06Z"
      },
      {
          "name": "OS::Nova::Flavor",
          "prefix": "hw:",
          "created_at": "2014-08-28T17:13:06Z",
          "updated_at": "2014-08-28T17:13:06Z"
      }
  ]
}

Field descriptions:

name               - (required) Resource type names should be aligned with
                                Heat resource types whenever possible:
                                http://docs.openstack.org/developer/heat/template_guide/openstack.html
prefix             - (optional) Specifies the prefix to use for the given
                                resource type. Any properties in the
                                namespace should be prefixed with this
                                prefix when being applied to the specified
                                resource type. Must include prefix separator
                                (e.g. a colon :).
                                Must include prefix separator (e.g. a colon :).
properties_target  - (optional) Some resource types allow more than one
                                key / value pair per instance.  For example,
                                Cinder allows user and image metadata on
                                volumes. Only the image properties metadata
                                is evaluated by Nova (scheduling or drivers).
                                This property allows a namespace target
                                to remove the ambiguity.

Associate Namespace to resource type: POST /metadefs/namespaces/{namespace}/resource_types

Example:

{
  "name" :"OS::Cinder::Volume",
  "properties_target" : "image_metadata",
  "prefix" : "hw_",
  "created_at": "2014-08-28T17:13:06Z",
  "updated_at": "2014-08-28T17:13:06Z"
}

De-associate Namespace from resource type: DELETE /metadefs/namespaces/{namespace}/resource_types/{resource_type}
Get list of all possible resource types: GET /metadefs/resource_types

Objects

Add Object in a specific namespace:: POST /metadefs/namespaces/{namespace}/objects

Example:

POST /metadefs/namespaces/CompanyXNamespace/objects

{
  "name": "StorageQOS",
  "description": "Our available storage QOS.",
  "required": [
      "minIOPS"
  ],
  "properties": {
      "minIOPS": {
          "type": "integer",
          "description": "The minimum IOPs required",
          "default": 100,
          "minimum": 100,
          "maximum": 30000
      },
      "burstIOPS": {
          "type": "integer",
          "description": "The expected burst IOPs",
          "default": 1000,
          "minimum": 100,
          "maximum": 30000
      }
  }
}

Replace an object definition in a namespace:: PUT /metadefs/namespaces/{namespace}/objects/{object_name}
Delete all objects in specific namespace:: DELETE /metadefs/namespaces/{namespace}/objects
Delete specific object in specific namespace:: DELETE /metadefs/namespaces/{namespace}/objects/{object_name}
Get a specific object in a namespace:: GET /metadefs/namespaces/{namespace}/objects/{object_name}

List objects in a specific namespace:

Return all objects including its schema properties: GET /metadefs/namespaces/{namespace}/objects

Filters by adding query parameters:

limit          = Use to request a specific page size. Expect a response
                 to a limited request to return between zero and limit items.
marker         = Specifies the namespace of the last-seen namespace.
                 The typical pattern of limit and marker is to make an initial
                 limited request and then to use the last namespace from the
                 response as the marker parameter in a subsequent limited
                 request.

Example Body:

{
      "objects": [
      {
          "name": "object1",
          "namespace": "my-namespace",
          "description": "my-description",
          "properties": {
              "prop1": {
                  "title": "My Property",
                  "description": "More info here",
                  "type": "boolean",
                  "default": true
              }
          }
      }
  ],
  "first": "/v2/metadefs/objects?limit=1",
  "next": "/v2/metadefs/objects?marker=object1&limit=1",
  "schema": "/v2/schema/metadefs/objects"
}

Properties (not in an object)

Add Property in a specific namespace:: POST /metadefs/namespaces/{namespace}/properties

Example:

POST /metadefs/namespaces/OS::Compute::Hypervisor/properties

{
      "name": "hypervisor_type",
      "type": "array",
      "description": "The type of hypervisor required",
      "items": {
          "type": "string",
          "enum": ["hyperv", "qemu", "kvm"]
       }
}

Replace a property definition in a namespace:: PUT /metadefs/namespaces/{namespace}/properties/{property_name}
Delete all properties in specific namespace:: DELETE /metadefs/namespaces/{namespace}/properties
Delete Property in specific namespace:: DELETE /metadefs/namespaces/{namespace}/properties/{property_name}
Get a specific property in a namespace:: GET /metadefs/namespaces/{namespace}/properties/{property_name}

List properties in a specific namespace:

Returns details of all properties in a namespace including property schema:: GET /metadefs/namespaces/{namespace}/properties

Filters by adding query parameters:

limit          = Use to request a specific page size. Expect a response
                 to a limited request to return between zero and limit items.
marker         = Specifies the namespace of the last-seen namespace.
                 The typical pattern of limit and marker is to make an initial
                 limited request and then to use the last namespace from the
                 response as the marker parameter in a subsequent limited
                 request.

Namespace membership management (Deferred to Future Release)

Allows different projects to have visibility to a non-public namespace.

For example, a cloud operator may have special hardware that is capable of running cloud and weather simulations. Images that have a certain property on it will get scheduled for that hardware and the operator only wants certain projects to see that property and hide it from other projects so that the cloud hardware isn’t misused.

Schema

JSON Schema for Namespace:

{
  "required": [
      "namespace"
  ],
  "properties": {
      "namespace": {
          "type": "string",
          "description": "The unique namespace text.",
          "maxLength": 80
      },
      "description": {
          "type": "string",
          "description": "Provides a user friendly description of the namespace.",
          "maxLength": 500
      },
      "display_name": {
          "type": "string",
          "description": "The user friendly name for the namespace. Used by UI if available.",
          "maxLength": 80
      },
      "owner": {
          "type": "string",
          "description": "Owner of the namespace.",
          "maxLength": 255
      },
      "visibility": {
          "enum": [
              "public",
              "private"
          ],
          "type": "string",
          "description": "Scope of namespace accessibility."
      },
      "protected": {
          "type": "boolean",
          "description": "If true, namespace will not be deletable."
      },
      "created_at": {
          "type": "string",
          "description": "Date and time of namespace creation (READ-ONLY)",
          "format": "date-time"
      },
      "updated_at": {
          "type": "string",
          "description": "Date and time of the last namespace modification (READ-ONLY)",
          "format": "date-time"
      },
      "properties": {
          "$ref": "#/definitions/property"
      },
      "objects": {
          "items": {
              "type": "object",
              "properties": {
                  "properties": {
                      "$ref": "#/definitions/property"
                  },
                  "required": {
                      "$ref": "#/definitions/stringArray"
                  },
                  "name": {
                      "type": "string"
                  },
                  "description": {
                      "type": "string"
                  }
              }
          },
          "type": "array"
      },
      "resource_type_associations": {
          "items": {
              "type": "object",
              "properties": {
                  "prefix": {
                      "type": "string"
                  },
                  "properties_target": {
                      "type": "string"
                  },
                  "name": {
                      "type": "string"
                  }
              }
          },
          "type": "array"
      },
      "schema": {
          "type": "string"
      },
      "self": {
          "type": "string"
      },
      "additionalProperties": false
  }
}

Note

See Schema Definitions below for $ref.

Variations on Namespace schema:

Namespace can also contain the following:

resource_type_associations
properties
objects
tags (future release)

JSON Schema for Resource Types:

{
  "name": "resource_type_associations",
  "links": [
      {
          "href": "{first}",
          "rel": "first"
      },
      {
          "href": "{next}",
          "rel": "next"
      },
      {
          "href": "{schema}",
          "rel": "describedby"
      }
  ],
  "properties": {
      "schema": {
          "type": "string"
      },
      "next": {
          "type": "string"
      },
      "resource_type_associations": {
          "items": {
              "additionalProperties": false,
              "required": [
                  "name"
              ],
              "name": "resource_type_association",
              "properties": {
                  "name": {
                      "type": "string",
                      "description": "Resource type names should be aligned with Heat resource types whenever possible: http://docs.openstack.org/developer/heat/template_guide/openstack.html",
                      "maxLength": 80
                  },
                  "prefix": {
                      "type": "string",
                      "description": "Specifies the prefix to use for the given resource type. Any properties in the namespace should be prefixed with this prefix when being applied to the specified resource type. Must include prefix separator (e.g. a colon :).",
                      "maxLength": 80
                  },
                  "properties_target": {
                      "type": "string",
                      "description": "Some resource types allow more than one key / value pair per instance.  For example, Cinder allows user and image metadata on volumes. Only the image properties metadata is evaluated by Nova (scheduling or drivers). This property allows a namespace target to remove the ambiguity.",
                      "maxLength": 80
                  },
                  "created_at": {
                      "type": "string",
                      "description": "Date and time of resource type association (READ-ONLY)",
                      "format": "date-time"
                  },
                  "updated_at": {
                      "type": "string",
                      "description": "Date and time of the last resource type association modification (READ-ONLY)",
                      "format": "date-time"
                  }
              }
          },
          "type": "array"
      },
      "first": {
          "type": "string"
      }
  }
}

JSON Schema for Properties

Namespaces and Objects also contain “properties”. Properties conform to JSON schema v4 syntax. But are limited to the following types: * string * integer * number * boolean * array

Each primitive type is described using simple JSON schema notation. This means NO nested objects and no definition referencing. .. note:: See Schema Definitions below for property definition.

JSON Schema for Objects:

{
  "required": [
      "name"
  ],
  "name": "object",
  "additionalProperties": false,
  "properties": {
      "name": {
          "type": "string"
      },
      "required": {
          "$ref": "#/definitions/stringArray"
      },
      "properties": {
          "$ref": "#/definitions/property"
      },
      "description": {
          "type": "string"
      },
      "created_at": {
          "type": "string",
          "description": "Date and time of object creation (READ-ONLY)",
          "format": "date-time"
      },
      "updated_at": {
          "type": "string",
          "description": "Date and time of the last object modification (READ-ONLY)",
          "format": "date-time"
      },
      "schema": {
          "type": "string"
      },
      "self": {
          "type": "string"
      }
  }
}

Objects also contain “properties” as mentioned above. .. note:: See Schema Definitions below for $ref.

JSON Schema common definitions:

{
  "definitions": {
      "property": {
          "additionalProperties": {
              "required": [
                  "title",
                  "type"
              ],
              "type": "object",
              "properties": {
                  "additionalItems": {
                      "type": "boolean"
                  },
                  "enum": {
                      "type": "array"
                  },
                  "name": {
                      "type": "string"
                  },
                  "title": {
                      "type": "string"
                  },
                  "default": {},
                  "minLength": {
                      "$ref": "#/definitions/positiveIntegerDefault0"
                  },
                  "required": {
                      "$ref": "#/definitions/stringArray"
                  },
                  "maximum": {
                      "type": "number"
                  },
                  "minItems": {
                      "$ref": "#/definitions/positiveIntegerDefault0"
                  },
                  "readonly": {
                      "type": "boolean"
                  },
                  "minimum": {
                      "type": "number"
                  },
                  "maxItems": {
                      "$ref": "#/definitions/positiveInteger"
                  },
                  "maxLength": {
                      "$ref": "#/definitions/positiveInteger"
                  },
                  "uniqueItems": {
                      "default": false,
                      "type": "boolean"
                  },
                  "pattern": {
                      "type": "string",
                      "format": "regex"
                  },
                  "items": {
                      "type": "object",
                      "properties": {
                          "enum": {
                              "type": "array"
                          },
                          "type": {
                              "enum": [
                                  "array",
                                  "boolean",
                                  "integer",
                                  "number",
                                  "object",
                                  "string",
                                  null
                              ],
                              "type": "string"
                          }
                      }
                  },
                  "type": {
                      "enum": [
                          "array",
                          "boolean",
                          "integer",
                          "number",
                          "object",
                          "string",
                          null
                      ],
                      "type": "string"
                  },
                  "description": {
                      "type": "string"
                  }
              }
          },
          "type": "object"
      },
      "positiveIntegerDefault0": {
          "allOf": [
              {
                  "$ref": "#/definitions/positiveInteger"
              },
              {
                  "default": 0
              }
          ]
      },
      "stringArray": {
          "uniqueItems": true,
          "items": {
              "type": "string"
          },
          "type": "array"
      },
      "positiveInteger": {
          "minimum": 0,
          "type": "integer"
      }
  }
}

Security impact

None

Notifications impact

None

Other end user impact

We intend to expose this via Horizon and are working on related blueprints.

Update python-glanceclient as needed.

Performance Impact

No changes to existing APIs or code.

This is expected to be called from Horizon when an admin wants to annotate tags (future) or key / value pairs onto things likes images and volumes. This API would be hit for them to get available metadata.

Other deployer impact

DB Schema Creation for new API will now be a newer version

Default resource types will be hardcoded: * OS::Glance::Image * OS::Cinder::Volume * OS::Nova::Flavor * OS::Nova::Aggregate * OS::Nova::Instance

glance-manage will have new commands for loading, unloading, and exporting metadata definitions: * db_load_metadefs - Loads metadata definitions from a specified directory * db_unload_metadefs - Unloads all metadata definitions in the database * db_export_metadefs - Exports metadata definitions to s a specified directory

The python-glanceclient will also support a new set of API and CLI commands for fine grained management of the metadata definitions in the catalog.

Default definition files will be checked into glance under etc/metadefs

Please note, the default definitions are only suggestions based on potential metadata in a given OpenStack deployment. The configuration of the actual deployment environment will likely require the cloud operator to limit what metadata should be made available in this catalog. They may limit it based on enabled drivers and filters or may choose to only offer a subset of the options offered by those drivers and filters. Just because an enabled driver or scheduler filter supports certain properties doesn’t mean that the cloud operator wants all the properties to be readily visible for selection in the UI.

Deployers can customize the definitions to be suitable to their cloud deployment by deleting namespaces, modifying namespaces, creating new namespaces, and changing namespace to resource type associations.

devstack will be modified to call this command to load in all the default metadata definitions.

Developer impact

None (New API)

Implementation

Assignee(s)

Primary assignee:: lakshmi-sampath
Other contributors:: wayne-okuma michal-dulko-f pawel-skowron pawel-koniszewski facundo-n-maldonado santiago-b-baldassin travis-tripp

Work Items

Investigate Pecan / WSME (Pecan ruled out, WSME chosen)

Changes would be made to:

The database API layer to add support for CRUD operations on namespaces

The database API layer to add support for CRUD operations on properties

The database API layer to add support for CRUD operations on objects

The database API layer to add support for CRUD operations on resource type associations

The REST API for CRUD operations on the namespaces

The REST API for CRUD operations on the objects

The REST API for CRUD operations on the properties

The REST API for CRUD operations on the resource type associations

The python-glanceClient to support operations

Dependencies

Same dependencies as Glance, except for WSME.

The implementation will be adding WSME object marshalling.

Testing

Unit tests will be added for all possible code with a goal of being able to isolate functionality as much as possible.

Tempest tests will be added wherever possible.

Documentation Impact

Docs needed for new API extension and usage

References

Youtube summit recap of Graffiti POC demo.

Mailing list thread after Juno summit.

Meeting log where markwash discussed graffiti after summit.

Current glance metadata properties in documentation:

Current documented Glance metadata properties.

Hierarchical tagging concepts were partially inspired by AWS marketplace. In the marketplace, you can filter by a hierarchy of categories. It made sense to us that this would be easy to achieve across various kinds of artifacts and resources through tags (future release).

AWS Categories

Additional Examples

Libvirt Driver Options

Images / Snapshots / Volumes (image metadata)

Today you can provide options to various drivers by putting metadata on images, snapshots, and volumes. The drivers read this information and use them. Currently this is only documented on the wiki.

Driving it from the metadata catalog using the common format we can expose them for easy use in the UI or CLI. This kind of metadata ideally could be published to the catalog programmatically. It would be associated with images, snapshots, and volumes.

UI Concept:

https://wiki.openstack.org/w/images/f/f7/Libvirtdriveroptions-objects.PNG

Example Data (subset):

{
  "objects": {
      "name": "LibVirtDriverOptions",
      "properties": {
          "hw_video_model": {
              "type": "array",
              "description": "The video image driver used.",
              "items": {
                  "type": "string",
                  "enum": [
                      "vga",
                      "cirrus",
                      "vmvga",
                      "xen",
                      "gxl"
                  ]
              }
          },
          "hw_machine_type": {
              "type": "string",
              "description": "Enables booting an ARM system using the
                              specified machine type. etc"
          },
          "hw_rng_model": {
              "type": "string",
              "description": "Adds a random-number generator device to
                              the image's instances. etc",
              "defaultValue": "virtio"
          }
      }
  }
}

Basic Host Aggregate / Flavor pairing of properties

Today you can ensure that flavors are launched on specific hosts using host aggregates. The basic way to do that is to put the same key / value pair on both the flavor and the host aggregate. With a metadata catalog, an admin could easily describe in detail the key / value pairs and their meaning in an exportable format for use in a single cloud / region or to import for reuse in another cloud deployment. As a very trivial example, you could use a property to collaborate on hosts and flavors that provide SSD. A more advanced object would be one that has different properties for things like min IOPS, burst IOPS, etc.

/metadefs/namespace/MyHostGroups/detail

Diagram:

+------------------------+
|  MyHostGroups          |    +-----------------+
|  +------------------+  +--> | Flavor          |
|  |SSD               |  +    +-----------------+
|  +------------------+  |    +-----------------+
|                        +--> | Host Aggregate  |
+------------------------+    +-----------------+

Example:

{
  "namespace": "MyHostGroups",
  "title": "My Host Groups",
  "description": "Different ways that we like to group our private cloud",
  "resource_type_associations" : [
      {
         "name" :"OS::Nova::Aggregate"
      },
      {
         "name" : "OS::Nova::Flavor",
         "prefix" : "aggregate_instance_extra_specs:"
      }
  ],
  "objects": {
      "name": "SSD",
      "properties": {
          "MyHostGroups:SSD": {
              "title": "SSD",
              "description": "Describe instances with SSD storage.",
              "type": "boolean",
              "default": true
          }
      }
  },
  "visibility": "public",
  "protected": true,
  "owner": "The Test Owner"
}

Sample Namespace with properties and objects

/metadefs/namespace/MyNamespace/detail

Example:

{
  "namespace": "MyNamespace",
  "display_name": "My User Friendly Namespace",
  "description": "My description",
  "resource_type_associations": [
      {
          "name": "OS::Glance::Image",
          "prefix": "hw_",
          "created_at": "2014-08-28T17:13:06Z",
          "updated_at": "2014-08-28T17:13:06Z"
      },
      {
          "name": "OS::Cinder::Volume",
          "prefix": "hw_",
          "properties_target": "image_metadata",
          "created_at": "2014-08-28T17:13:06Z",
          "updated_at": "2014-08-28T17:13:06Z"
      },
      {
          "name": "OS::Nova::Flavor",
          "prefix": "filter1:",
          "created_at": "2014-08-28T17:13:06Z",
          "updated_at": "2014-08-28T17:13:06Z"
      }
  ],
  "properties": {
      "nsprop1": {
          "title": "My namespace property1",
          "description": "More info here",
          "type": "boolean",
          "default": true
      },
      "nsprop2": {
          "title": "My namespace property2",
          "description": "More info here",
          "type": "string",
          "default": "value1"
      }
  },
  "objects": [
      {
          "name": "object1",
          "namespace": "MyNamespace",
          "description": "My object1 description",
          "properties": {
              "prop1": {
                  "title": "My object1 property1",
                  "description": "More info here",
                  "type": "array",
                  "items": {
                      "type": "string"
                  }
              }
          }
      },
      {
          "name": "object2",
          "namespace": "MyNamespace",
          "description": "My object2 description",
          "properties": {
              "prop1": {
                  "title": "My object2 property1",
                  "description": "More info here",
                  "type": "integer",
                  "default": 20
              }
          }
      }
  ],
  "visibility": "public",
  "protected": true,
  "owner": "The Test Owner"
}

Provide the ability to temporarily deactivate an image

Sun, 11 Jan 2026 00:00:00

https://blueprints.launchpad.net/glance/+spec/deactivate-image

This blueprint provides for a method to deactivate an image temporarily to prevent download of image data (including booting a new instance from the image). A method for reactivation is also described.

Problem description

As the ability for users to import images from outside the cloud becomes commonly available, a concern is the possibility for a user to import a malicious or otherwise problematic image (e.g. an image containing a trojan horse). If an image is deemed suspicious, an admin should be able to put the image “on hold” preventing instances from being built with it until it has be properly examined and determined to be safe or found dangerous and deleted. All image properties will remain accessible and editable, but the image will not be downloadable by non-admin users.

Proposed change

To expose this ability, a new image state ‘deactivated’ will be introduced. In order to transition into and out of this state, two new functional API calls will be added to the REST interface (detailed below).

When an image is placed in the deactivated state, image data may not be downloaded by a non-admin user. This will include all other operations (such as export and cloning) which may need access to image data. Admins may continue to download the data to facilitate testing and examination of the image. Operations which do not access image data (e.g. update, delete, or image-list) continue to operate as usual for deactivated images.

Only ‘active’ images may be deactivated. Images in the ‘deactivated’ state may only transition to ‘active’ or ‘deleted’. Deactivating a ‘deactivated’ image or reactivating an ‘active’ image will succeed as no-ops.

Restrictions to the deactivate and reactivate functions can be handled through the current policy mechanisms and thus the responsibility falls to the deployer to implement / update the applicable policies per their specific requirements. By default, only administrators will have permissions sufficient to deactivate and reactivate an image.

Alternatives

No alternatives seem to be readily available. Policy changes were considered but this is an action which needs to function on an image-by-image basis regardless of role (other than admin) and across accounts (in the case of shared images).

Data model impact

None

REST API impact

The design for the deactivate and reactivate calls specified below follows the controller pattern from Subbu Allamaraju, RESTful Web Services Cookbook (O’Reilly, 2010), section 2.6. It was discussed by the OpenStack API Working Group on the mailing list and at a series of meetings in February 2015. The consensus was that this design is an appropriate pragmatic RESTful interface.

Using Glance tasks for deactivate/reactivate was discussed by the API WG, but the consensus was that since Glance tasks are specifically designed for asynchronous operations, their use was not appropriate here. The recommendation is that any new asynchronous operations should be implemented in the tasks API, but the controller pattern is appropriate for synchronous actions like deactivate and reactivate.

The API WG concluded that this API design proposal was acceptable at its meeting on 19 February 2015.

http://eavesdrop.openstack.org/meetings/api_wg/2015/api_wg.2015-02-19-00.00.log.html#l-146

POST:/v2/images/{image_id}/actions/deactivate * Description: Deactivate an image * Method: POST * Normal response code(s): 204 * Expected error http response code(s): 403
- When calling deactivate on an image which is not currently in the ‘active’ or ‘deactivated’ state.
- URL for the resource: /v2/images/{image_id}/actions/deactivate
- Parameters which can be passed via the url {image_id}, String, The ID for the image.
POST:/v2/images/{image_id}/actions/reactivate * Description: Reactivate an image * Method: POST * Normal response code(s): 204 * Expected error http response code(s): 403
- When calling reactivate on an image which is not currently in the ‘deactivated’ or ‘active’ state.
- URL for the resource: /v2/images/{image_id}/actions/reactivate
- Parameters which can be passed via the url {image_id}, String, The ID for the image.
GET:/v2/images/{image_id}/file * Description: Downloads binary image data. * Method: GET * Normal response code(s): 200, 204 * Expected error http response code(s): 403
- When attempting to download a deactivated image
- URL for the resource: /v2/images/{image_id}/file
- Parameters which can be passed via the url {image_id}, String, The ID for the image.
GET:/v2/images/{image_id} * Description: Retrieve the image metadata * Method: GET * Normal response code(s): 200
- Retrieving image metadata for a deactivated image will continue to function normally
- Expected error http response code(s): None
- URL for the resource: /v2/images/{image_id}
- Parameters which can be passed via the url {image_id}, String, The ID for the image.
GET:/v1/images/{image_id} * Description: Returns the image details as headers and the image binary in the body of the response. * Method: GET * Normal response code(s): 200 * Expected error http response code(s): 403
- When attempting to download a deactivated image
- URL for the resource: /v1/images/{image_id}
- Parameters which can be passed via the url {image_id}, String, The ID for the image.
HEAD:/v1/images/{image_id} * Description: Retrieve the image metadata * Method: HEAD * Normal response code(s): 204
- Retrieving image metadata for a deactivated image will continue to function normally
- Expected error http response code(s): None
- URL for the resource: /v1/images/{image_id}
- Parameters which can be passed via the url {image_id}, String, The ID for the image.

Security impact

This change enhances security of the overall system by giving administrators the ability to suspend usage of potentially malicious images while they are being audited.

There are no negative security impacts.

Notifications impact

None

Other end user impact

Support for the new API operations should be added to python-glanceclient.

End users will be unable to perform any operations on a deactivated image which requires access to the image data. This would include downloading, booting, and exporting the image.

Performance Impact

None

Other deployer impact

In order to restrict access to these operations, deployers will need to configure the ‘deactivate’ and ‘reactivate’ policies accordingly.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: eddie-sheffield
Other contributors:: None

Reviewers

Core reviewer(s):: nikhil-komawar
Other reviewer(s):: hemanth-makkapati

Work Items

Add allowed state transistions
Add image actions controller to v2 api
Add ‘deactivate’ action to controller and router
Add ‘activate’ action to controller and router
Add policy checks for ‘deactivate’ and ‘reactivate’
Add check for deactivated image on download to v2 api
Add check for deactivated image on download to v1 api

Dependencies

None

Testing

Tempest tests for the new operations and verifying download restrictions on deactivated images need to be added.

Documentation Impact

Documentation is required for:

The new API functions
The new policies (as described in Other Deployer Impact)

References

Earlier version of this spec from the wiki: * https://wiki.openstack.org/wiki/Glance-deactivate-image

Discussions concerning the “Function API” approach used here: * https://etherpad.openstack.org/p/glance-adding-functional-operations-to-api * http://lists.openstack.org/pipermail/openstack-dev/2014-May/036416.html * http://osdir.com/ml/openstack-dev/2015-02/msg01563.html * “RESTful Web Services Cookbook, Section 2.6” - http://it-ebooks.info/book/392/

Software Metadata Definitions

Sun, 11 Jan 2026 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/glance/+spec/software-metadefs

The spec will provide a base library of metadata definitions for various common software products, components, and libraries that may exist on particular image (or volume or instance). These metadata definitions will make it easier for end users and admins to easily describe the software and its properties. This information will enable an improved and faster user experience for applying software metadata, searching based on software metadata, and viewing the software information about an image.

Various improvements in horizon are underway to take advantage of this metadata. For example, a user launching an instance will be able to expand the image row to see addition information about the image. This will include providing the metadata definition for properties on the image. This same information will also be visible from the image and instance details page.

The following blueprint is for launch instance:

https://blueprints.launchpad.net/horizon/+spec/launch-instance-redesign

The following screenshot mockup provides an example from the instance details:

https://wiki.openstack.org/w/images/3/37/Instance-details-mock-january-2015.png

The following blueprint is for instance details (see source tab):

https://blueprints.launchpad.net/horizon/+spec/instance-details-redesign

Additional horizon blueprints are being prepared that will display metadata, but are still undergoing collaborative UX design first.

Problem description

In Juno, the metadata definitions catalog was introduced with a primary intent to improve collaboration on the metadata that can be applied to different resources such as images, flavors, host aggregates, and volumes. The primary development focus in Juno was on what is currently system metadata in OpenStack (properties that affect scheduling and driver behavior).

The metadata fields on these same resources can also be used to provide additional, rich information that describes the resource for user understanding and search. This information can be leveraged to improve instance launching, application catalogs, and a variety of other user interaction points.

Current methodologies present and proposed in OpenStack require repetitive descriptions, categorization, imagery, and attributes to be applied on each instance of a resource.

For example, if multiple images contain Apache then each image will require the user to type in information about Apache, will require them to provide some sort of categorization tag, and will not have any common “template” like fields for the software which can be used for faceted search.

This leads to redundancy, spelling mistakes, inconsistency as well as a general lack of ability to present a rich UI experience. In addition, there is no common set of base definitions for software that can be shared across deployments.

Proposed change

The implementation of this spec will provide a base library of metadata definitions for various common software products, components, and libraries that may exist on particular image (or volume or instance).

The following are examples of the type of commonly used software that will be included: * Databases such as MySQL, PostgreSQL, Oracle, MongoDB, etc. * Web servers such as Apache, nginx, IIS * Runtimes like php, java, python, etc.

The review process of the actual proposed definitions will allow for the community to suggest additional software to include or to suggest removing certain definitions.

These metadata definitions will make it easier for end users and admins to easily describe the software and its properties. This information will enable an improved and faster user experience for applying software metadata, searching based on software metadata, and viewing the software information about an image.

Users of the glance client / cli will be able to lookup the available definitions just like they do today. So the below methodology that can be done to look up something like virtual CPU topology properties will work for the software definitions:

glance --os-image-api-version 2 md-namespace-list

glance --os-image-api-version 2 md-namespace-show OS::Compute::VirtCPUTopology

glance --os-image-api-version 2 md-property-show OS::Compute::VirtCPUTopology cpu_maxsockets

The metadata definitions will be included as additional json files in the etc/metadefs directory just like the existing example system metadata definitions.

Alternatives

As every single image, artifact, or other type of resource is added to the system, the admin and users could be responsible for putting a good description that is consistent with other descriptions, could try to remember all the right tags to add, or could try to remember all the same properties to upload to it. This leads to redundancy, spelling mistakes, inconsistency as well as a general lack of ability to present a rich UI experience. It also doesn’t enable predictable search facets for rich search experience.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

No impacts glance client.

Work is being done in horizon to take better advantage of these definitions.

Performance Impact

None.

Other deployer impact

This spec will provide deployers with a base set of metadata definitions for software similar to what they are provided for system metadata definitions. They can choose to either use or not use the base definitions provided in this spec.

In Juno, example software definitions were not provided. To upgrade, they will need to deploy these files to etc/glance/metadefs and then call glance-manage db_load_metadefs. This call automatically loads new namespaces that aren’t already loaded.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: travis-tripp
Other contributors:: lakshmi-sampath murali-sundar

Reviewers

Core reviewer(s):: nikhil-komawar <launchpad-id or None>
Other reviewer(s):: wayne-okuma murali-sundar

Work Items

Provide definition files for different categories of software.

Databases
Web servers
Runtime environments
More

Dependencies

None

Testing

Existing code ensures that metadata definitions are loaded properly.

Documentation Impact

Possibly update docs to talk about the base software definitions provided.

References

http://docs.openstack.org/developer/glance/metadefs-concepts.html

Glance Image Signing and Verification

Sun, 11 Jan 2026 00:00:00

https://blueprints.launchpad.net/glance/+spec/image-signing-and-verification-support

Before Liberty, OpenStack did not support the following feature:

Signature validation of uploaded signed images

Deploying authentication protects image integrity by verifying that an image has not been modified after the upload by the user. This feature improves the enterprise-ready posture of OpenStack.

Although an initial implementation was merged into Glance for the Liberty release, adding functionality for this feature to Nova requires modifications to this implementation [1].

Problem description

Before Liberty, there was no method for users to verify that a previously uploaded image had not been modified. An image could potentially be modified in transit (such as when it is uploaded to Glance or transferred to Nova) or Glance itself could be untrusted and modify images without a user’s knowledge. An image that is modified could include malicious code. Providing support for image signatures and signature verification would allow the user to verify that an image has not been modified prior to booting the image.

There are several use cases that this feature supports:

An image is signed by an End User, using the user’s private key. The user then uploads the image to Glance, along with the signature created and a reference to the user’s public key certificate. Glance uses this information to verify that the signature is valid, and notifies the user if the signature is invalid.
An image is created in Nova, and Nova signs the image at the request of the End User. When the image is uploaded to Glance, the signature and public key certificate reference are also provided. Glance verifies the signature before storing the image, and notifies Nova if the signature verification fails.
A signed image is requested by Nova, and Glance provides the signature and a reference to the public key certificate to Nova along with the image so that Nova can verify the signature before booting the image.

Proposed change

For the initial implementation in Liberty, this change used the property feature of Glance to store the metadata items needed for image signing and verification. These include a public key certificate reference, and the signature. These are provided when the image is created, and are accessible when the image is uploaded. Note that the feature only supports image uploads with the Glance API v2 (and does not support using the Glance API v1). Also note that multiple formats for the key (such as SubjectPublicKeyInfo) and for the signature (such as PSS) are supported. The format of the signature is stored as one of the properties.

The certificate reference is used to access the certificate from a key manager, where the certificate is stored. This certificate is added to the key manager by the end user before uploading the image. Note that the signature is done offline.

Glance supports computing a checksum of an image when an image is uploaded, and this checksum is stored with the image. In the Liberty implementation, this same hash (which is hardcoded to be MD5) was used for the signature verification. The checksum hash is computed in glance_store (when the image data is uploaded), and is then used to verify the signature in Liberty. The Glance frontend uses the reference to the public key certificate to retrieve the certificate from the key manager, and then uses this public key along with the signature, the computed checksum, and the rest of the signature metadata to verify the signature. If the signature verification fails, the image is transition to a killed state, and the user is notified that the upload failed and given a reason why.

However, in Mitaka, this approach will be modified to instead validate a signature of the image data, rather than validating an image signature of a hash of the image data. To explain this further, consider the following two options using ‘RSA-PSS’ as the signature type, and ‘SHA-256’ as the signature hash method:

A signature of the Glance checksum of the image data (hardcoded to MD5)

signature = RSA-PSS(SHA-256(MD5(IMAGE-CONTENT)))

A signature of the image data directly

signature = RSA-PSS(SHA-256(IMAGE-CONTENT))

The first approach (‘sign-the-hash’) was implemented in Liberty, but the second approach (‘sign-the-data’) will be used instead in Mitaka. Although the sign-the-hash approach will still be supported in Mitaka, it will be deprecated, and removed in a later release. Note that, as seen above, the ‘sign-the-data’ approach still involves creating a hash of the image. However, this is done as part of the signature verification process, and is internal to the signature verifier. This is in contrast to the ‘sign-the-hash’ approach, where the verifier does a hash of the hash of the image data as part of the verification process.

To support the ‘sign-the-data’ approach, a few modifications will have to be made. Currently, the ‘checksum’ is computed in each of the glance_store/_drivers/*.py ‘add’ methods. When ‘checksum.update()’ is called for the image data chunk, this data chunk will also be passed to a signature_utils module (through an optional callback method), provided that the necessary signature properties are present. These data chunks will then be used to update the signature verifier, and when the image data is done being read the verifier will be finalized and the verification will occur. As in Liberty, if the signature verification fails, the image is transition to a killed state, and the user is notified that the upload failed and given a reason why.

Alternatives

An alternative to replacing the sign-the-hash approach with the sign-the-data approach would be to leave the Liberty implementation as-is. However, there has been pushback from the Nova community with this approach [2], since it requires initially using MD5 (which is not cryptographically secure) as the basis, and then querying Glance for the hash method used, assuming the hash was made configurable. In the interest of using an implementation that is accepted by both Glance and Nova, as well as removing any attachment to MD5, it is necessary to modify the initial approach.

An alternative to the sign-the-data approach is to create a separate configurable hash for use with verifying/creating the signature. However, creating a separate hash is no different performance-wise to signing the image data directly, since part of the signature verification process is computing a hash of the image for use with verification. Also, the use of this separate hash, though stronger than MD5, would be limited, since having a signature makes the need for a hash obsolete.

An alternative to storing a reference to the public key certificate in Glance would be to store the actual public key certificate in Glance. However, this approach would be insecure, since Glance, unlike a dedicated key manager, has not been created with storing keys or certificates in mind.

An alternative to using asymmetric keys for integrity and confidentiality is to use symmetric keys. However, in order for Glance to be able to verify the image, it would need to have access to the key used to create the signature. This access would enable Glance to modify the image and create a new signature without the user’s knowledge. Using asymmetric keys enables Glance to verify the signature without giving Glance the power to modify the image and signature.

An alternative to using the Glance properties to store and retrieve the signature metadata would be to create an API extension that support signatures. Then, instead of the user setting the metadata using the property key value pairs, the API extension would be used. Currently, if a user were to use the metadata keys (for the certificate and signature) for other purposes, the image uploads would fail. Another item of note is that an API extension would allow for the management of multiple signatures per image in a clean manner, which is not possible with the properties approach. However, the Images API does not support extensions, so this is not a valid approach.

Another alternative to using the Glance properties to store and retrieve the signature metadata would be to use the CMS (cryptographic message syntax) format as defined in RFC 5652 Section 5. However, the size for this would be variable, and could not use the existing Glance properties, which would require API modifications. For the initial implementation, Glance properties will be used, with the plan to migrate to using CMS in a future implementation as the need for increased flexibility arises.

An alternative to requiring the user to provide the signature separate from the image is to support images that already have an embedded signature. Although this could be included as a future improvement, the initial implementation will not provide embedded signature support, since it is advantageous to keep the initial effort focused and small.

An alternative to focusing on a single-cloud implementation would be to include support for multi-clouds in the initial implementation. If images are exchanged between different clouds, signature verification could be used to confirm that images have not be modified. However, in the interest of a more simplistic initial implementation, explicit support for multi-clouds will be saved for future iterations.

Data model impact

None.

REST API impact

No API changes will be needed for the initial implementation, provided that other services are able to retrieve all of the properties of a given image.

Note that the existing API allows for providing the signature metadata as Glance properties, and returning an error message if verification fails.

Security impact

This change improves the enterprise-ready posture of OpenStack by enabling signature signing and verification.

Although keys are used in this change, the keys themselves are assumed to be stored in a key manager, and only a reference to the certificate is stored in Glance.

This change involves hashing the image data for use in verifying and creating signatures for the image.

Note that the signature length is currently limited to 255 bytes, since this is the maximum size supported for Glance properties. In turn, this limits the size of the keys that can be used for signature creation.

Notifications impact

This change will involve adding log messages to indicate the success or failure of signature verification and creation.

Other end user impact

The user will be required to provide the appropriate information needed for the signing and verification in order to use this feature.

There are no changes that need to be made to python-glanceclient.

Performance Impact

The feature will only be used if a user has provided the appropriate properties during the image upload. Otherwise, no signature verification or creation will occur.

When signature verification and creation do occur, there will be some latency associated with retrieving the certificate from the key manager. Also, as a part of the signature verification or creation, a hash of the image data is computed by the ‘verifier’ or ‘signer’ which will have a small impact on performance.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: brianna-poulos
Other contributors:: dane-fichter

Reviewers

Core reviewer(s):: flaper87 nikhil_k
Other reviewer(s):: joel-coffman

Work Items

The feature will be tackled in the following stages:

Add signature verification on upload to verify a signature of the image data directly, rather than a signature of the MD5 hash of the image data, when the new signature metadata property names are present.
Add a log entry to mark the sign-the-hash verification path as deprecated in Mitaka, when the old signature metadata property names are present.
Remove the sign-the-hash verification steps in the release after Mitaka.

Dependencies

In order to take advantage of the signatures in Glance, Nova will need to be updated to retrieve the signatures from Glance and verify them. However, Glance does not depend on Nova to have this support in order to have the feature added. The spec for this in Nova [2] has been approved.

Testing

Before Nova support for this feature is added, unit tests will be sufficient. Once Nova support is added, Tempest tests should ensure that the interaction between Nova and Glance works as expected.

Documentation Impact

Instructions for how to use the change will need to be documented. These include instructions for the user on how to create keys and signatures offline before providing this information during the creation of an image.

This documentation will also include descriptions for each of the following signature metadata properties (note that “img_” has been included at Nova’s request):

img_signature: the signature of the “checksum hash” encoded in base64 format
img_signature_hash_method: the hash method used to create the signature
img_signature_key_type: the key type used in creating the signature
- valid values are: “RSA-PSS”
img_signature_certificate_uuid: the uuid used to retrieve the certificate from castellan

References

cryptography: https://cryptography.io/en/latest/

[1] http://bit.ly/1Q0M0C7

[2] https://review.openstack.org/#/c/188874/

[3] http://git.openstack.org/cgit/openstack/castellan

Deprecate Glance Registry

Sun, 11 Jan 2026 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/glance/+spec/deprecate-registry

Glance registry has, historically, helped to scale Glance by allowing for centralizing the database access (glance-registry) while distributing the data access (glance-api). In addition, it’s helped as a safe-guard for the database access credentials as it shouldn’t be deployed as a public endpoint.

With the release and support of Glance’s V2, the Glance Registry service has become redundant, hence this proposal to deprecate it.

Problem description

Glance Registry serves a very specific use case, which is to serve as a proxy for the database operations in a Glance deployment. This use case came as part of older Glance API versions and a certainly older architecture.

Unfortunately, it’s becoming more of a burden to maintain it - development, operations and documentation wise - than a benefit for the project itself.

From a development perspective, the team has to maintain the API updated for every change in the database API. From an operational perspective, there’s another set of API nodes that need to be deployed, monitored, and upgraded. From a documentation stand point, the team needs to make sure the service docs are up-to-date, the best practices are spelled out and configuration files updated.

The benefits of this service have been discussed in a recent thread [0] on both developers and operators, mailing lists. The output of this thread is not really conclusive, although it suggests there’s no real use case for this service anymore and that OPs would be better off by not having it.

Rolling upgrades was brought as a possible blocker for this deprecation. As it’s been explained in the thread [1], upgrading Glance (or even the planned work on rolling upgrades) should not depend on the presence of this service. Anything needed from Glance Registry should be possible to obtain from Glance API itself.

Glance Glare is not going to use Glance registry, which would also leave us with an inconsistent deployment and a bad user experience.

Proposed change

This spec proposes to deprecate the glance-registry service. Mark the service as deprecated and ready for removal in the Q release.

Alternatives

Keep maintaining the Glance Registry service.

Data model impact

None

REST API impact

The public facing API won’t be changed. The registry API will be deprecated and not required anymore.

Security impact

Deployers will have to put the database credentials in the glance-api config files. This might be seen as a security issue for some deployments as this means an attacker that gains access to the glance-api server could potentially access glance’s database. However, this is not any different than what other services.

Notifications impact

None

Other end user impact

None

Performance Impact

This change should actually improve the overall performance as it’d get rid of an extra step glance-api needs to go through to access the image metadata.

Other deployer impact

Glance deployments will eventually have to move away from using the registry. This can’t be done for glance-api nodes using v1 but it can certainly be done for v2-only nodes.

Developer impact

Less code to maintain, happier developers.

Implementation

Assignee(s)

Primary assignee:: flaper87
Other contributors:: <launchpad-id or None>

Reviewers

Core reviewer(s):: rosmaita jokke nikhil
Other reviewer(s):: <launchpad-id or None> <launchpad-id or None>

Work Items

Add a glance-api only job
Mark the registry service as deprecated

Dependencies

https://review.openstack.org/#/c/315190/

Testing

Test a registry-less deployment in the OpenStack CI.

Documentation Impact

Document the motivations behind this deprecation and a recommended upgrade path from Mitaka to Newton and on.

References

Spec Lite: Introduce db sync check feature

Sun, 11 Jan 2026 00:00:00

problem:: It is very hard for automation of deploy and upgrade operations to know if there are db migrations pending. It requires the automation to know what the latest version is, and compare that to the output of a command to check the current version, then interpret the potential difference somehow.
solution:: Similar to the linked feature added to Keystone’s manage command, Glance should support an operation which enumerates any outstanding db upgrade operations and provide user friendly message based on that status. Each expand, migrate, and contract operation required to upgrade the db should be listed in the proper order of execution in the response. This may be implemented by using a glance-manage db check option. When this option is present no db upgrades would be performed but potential operations would be reported, acting similar to the pattern of a dry-run.
impacts:: Introduces new option to the db sync operation in glance-manage.
timeline:: Queens RC-1
link:: https://bugs.launchpad.net/keystone/+bug/1642212
reviewers:: rosmaita, abhishekk, jokke
assignee:: bhagyashris

Image Import Refactor

Sun, 11 Jan 2026 00:00:00

https://blueprints.launchpad.net/glance/+spec/image-import-refactor

Note

This is a very long spec, so we’ve added a FAQ to cover some common questions so that people can make better informed comments on the implementation patches.

In this spec we propose a refactoring of the current Glance image import process to meet the criteria of being discoverable, interoperable, and flexible. The goal is to present a uniform interface for image import that will satisfy the requirements of public and private clouds of various sizes.

Note

This spec is based on ideas expressed in mailing list discussions (see [OSM1], [OSM2]), a meeting in #openstack-glance (see [OSL1], [OSE2]), a video meeting of interested people (summarized in [OSW3]), and a session at the Mitaka design summit [OSE3]. It does not, however, reproduce all the discussion that took place, so the interested reader may wish to glance through those documents to gain wider context.

As a basis for the discussion to follow, image import is described by the following use case:

A cloud end-user has a bunch of bits that they want to give to Glance in the expectation that (in the absence of error conditions) Glance will produce an Image (record, file) tuple that can subsequently be used by other OpenStack services that consume Images.

Among the motivations for the above use case are:

An end user creates a specialized custom image offline and wants to use it in various OpenStack clouds.
A particular cloud may not offer a public image of some Excellent But Obscure Operating System (EBOOS). The EBOOS User Group could make a VM image available on its website, and EBOOS enthusiasts could import it into the OpenStack cloud of their choice.
An end user finds an interesting image in the OpenStack App Catalog and wants to boot instances from it in an OpenStack cloud.
An end user creates a snapshot of an instance in one OpenStack cloud and wants to boot instances from it in another OpenStack cloud. (Obviously, this would require image export as well.)

Note

Image Creation in OpenStack Clouds

It’s worth distinguishing three distinct use cases around image creation:

A deployer wishes to create public images that end users may use to boot instances.
Another OpenStack service creates an image from some other resource it manages (for example, Nova creates an image of a server, or Cinder creates an image from a volume) at the behest of an end user.
An end user wishes to import an image.

Glance should support all three scenarios.

Background

Glance contains a “tasks” API that is a result of discussions during and after the Havana design summit (see [OSD1], [OSW2], and [OSW1]). This API was designed to present a uniform interface to end-users that allowed a large degree of customization by individual cloud providers, and currently defines an ‘import’ task. Since the Havana design summit, however, the DefCore movement in OpenStack has developed as a means of ensuring interoperability among OpenStack branded clouds. The current Glance tasks API is too customizable to be suitable for DefCore purposes, and in fact, does not fare well when assessed on the dimensions of interoperability and discoverability.

The primary problem with tasks as defined in the current API is that they have an “input” element, defined in the task schema as a JSON blob, whose exact content is left up to the cloud deployer. This allows for flexibility on the part of a cloud deployer, but introduces a discoverability problem, as the only mechanism currently available for determining acceptable content of the “input” element is the deployer’s documentation. While some flexibility is good, this much flexibility makes it impossible for a competent end-user of one OpenStack cloud to be sure this competence extends to a different OpenStack cloud.

One goal of this spec is to implement image import in such a way that it will be a suitable candidate for inclusion in DefCore. (See [OSR1] for the list of 12 criteria for being included in DefCore Guidelines.)

Note

Currently, Glance is included in two DefCore programs: “OpenStack Powered Compute” and “OpenStack Powered Platform”. (See [OSO1] for a definition of these terms.)

Current Upload Workflow

Here’s a quick reminder of the current image upload workflow.

POST v2/images

This creates an image record and returns an Image response containing (among other things) an ‘id’ field. (The image record can be modified by PATCH calls, but we’ll ignore that here.) The key thing is that a record must be created so that the user has an image_id to work with for the purposes of uploading the actual image bits.
PUT v2/images/{image_id}/file

This call instructs Glance to accept the incoming image data and place it into the storage backend. The associated image record must have the container_format and disk_format properties set or the call will not succeed. This call returns no content.

This current image upload workflow will still exist for backward compatibility and for use by Glance administrators and trusted OpenStack services.

Problem description

For reasons set out in [OSW3] and [OSE1], it is desirable for deployers to have the ability easily to separate untrusted end-user image import from the simple upload facility used by trusted sources (typically, Nova or another OpenStack service, or Glance administrators supplying public images for use in a cloud). What we aim to do in this spec is to define a suitable end-user image import mechanism that will satisfy the requirements of all OpenStack clouds, whether small or large, public or private.

Summary of the Constraints Around This Project

Here are, to the best of my recollection, what was agreed upon between the Glance community, DefCore (mostly Doug Hellman), infra (mostly Monty), and various interested parties who showed up at the design session on image import at the Tokyo summit.

First, background, so you can see what problems needed to be addressed:

(At least some) Public cloud operators do not want to expose the current glance v1/v2 image upload as it is too fragile.
The TC passed a resolution in December 2015 [NEW1] saying that end user image upload (what we – following industry parlance – are calling “image import”) must be available in OpenStack clouds.
1. The TC resolution says that an OpenStack cloud should support import of a vanilla linux image; no mandate about image format, size, etc.
2. Part of the goal of this spec is design a discoverable and interoperable API for image import so that the TC resolution can be satisfied and image import can be included as a DefCore requirement. (See [NEW2] for more about this point.)
The “Tasks” API is a disaster from the interoperability and discoverability standpoint. (We know this because at least one large public cloud has exposed image import via Glance Tasks, and the OpenStack infra team has a lot to say about how bad it is. Just ask them.)
1. interoperability failures: The Task object, as defined by v2/schemas/task contains an “input” and “result” element which are defined to be JSON blobs; anything could go in there, so possibly radically different stuff for each OpenStack cloud
2. discoverability failures: You don’t have to support a particular disk/container format, but there must be a way to find out what a particular cloud supports (and this “way” should be the same for all OpenStack clouds, and no, documentation doesn’t count)
There are three cases for “image upload” that Glance should support.
1. Admin upload of “base” or “public” images
2. Image upload from OpenStack services (for example, Nova or Cinder)
3. End user image import

Note

My view is that we are working on the image import use case, and what we come up with there could, but doesn’t have to, be used/usable for the other two use cases. The key point to keep in mind here is that the discovery of various vulnerabilities may cause operators to halt import (temporarily), and they will want to do that while still keeping the other 2 use cases operational.

As Doug pointed out on a previous patch: “It would be nice to have only one API, but the hole we have right now is the public-facing use case and so that’s where the focus of this work should be. If we can make the results work for the other cases, that’s a bonus, but not required.”

OK, without further ado, here’s what was agreed upon:

The constraints that an adequate image import solution must meet

There must be a well-defined image import structure/framework that should be supportable by all OpenStack clouds.
1. “well-defined”
  1. calls have request/response schemas that are discoverable
  2. the values that will enable a client to have a successful image import (e.g., supported formats) must be discoverable
  3. “discoverable” == via API call (in what Flavio calls a “follow-your-nose” fashion)
    1. specific API request: this would be the GET v2/info/import call
    2. available in headers: the headers would be returned with the POST v2/images response. The idea is that the content of that response is the JSON representation of the Image record (so the import methods available and other associated import information don’t really belong in a particular image resource), and having the info come back in the headers could allow a client to determine the import method to use without having to make the discovery call.
2. “supportable by all OpenStack clouds”
  1. it’s acceptable for there to be multiple import methods as long as each is well defined. (See the “Proposed Change” section below for details. An “import method” has to do with how the image data is delivered to Glance. The API calls won’t change, and their body and structure will remain the same for the various methods.)
  2. no cloud has to support all import methods, but it’s expected that to achieve certification as “OpenStack Powered Compute” (and hence, to even have a shot at certification as an “OpenStack Powered Platform”), a cloud must expose at least one of these.
3. Since Swift is not part of the “OpenStack Powered Compute” program, Glance must expose at least one import method that does not rely upon the presence of an end-user-accessible object store.
The “three step dance” import style was deemed acceptable
1. one: create image record, two: upload data, three: import call
2. steps one and two can be independent. For example, in the ‘swift-local’ method sketched out below, an end user could upload the image data to swift first (accomplishing step two), then do step one, followed by step three.
The import workflow should allow for server-side operator customization, but no operator is required to perform such customization.
1. We’re talking about customization in processing the uploaded data. The API request/response structure is not customizable.

Proposed change

Import Workflow

The import workflow will respect the basic structure of the current upload workflow described above.

End user creates an image record. (We’ll refer to this as image-create, just keep in mind that what’s created is only the image record.)
End user makes the data available to Glance. (This could be via direct upload, or via some other well-defined import method that is discoverable by an end user.)
End user instructs Glance to process the data and create a bootable image.

A key distinction between image upload and image import is that imported images are not immediately available for use, that is, they are not ‘active’ at the completion of the data PUT call. This allows deployers optionally to process the image data (for example, by performing a validation process) before the image becomes ‘active’. Additionally, deployers may need to put protected properties on the image record at this point. Thus the import call needs to be asynchronous.

Discovery

An end user needs to be able to do the following:

value discovery

The end user needs to determine what container format and disk format are accepted by this cloud, the maximum allowed image size (both actual and virtual), what import methods are available.
method discovery

The end user wants to know what import methods this site supports. Although this information will be returned in the value discovery call described above, we also propose to return it in a response header from the call used to create the image record. This way, a client is not required to make the value discovery call as part of the image import workflow.
format discovery

The end user wants to know what an import request body looks like. This information will be provided by a JSON schema.

We assume that the user already knows how to discover the image schema for the purposes of creating an image and reading an image response.

Import methods

We define one initial import method, glance-direct, but we envision more methods such as swift-local. (We include swift-local in the discussion throughout so that it’s clear how the import scheme described in this spec can be extended in an interoperable way to include other ways of getting the image data into Glance).

Note

We might want to use different terms here that would expose less internals to the users. For instance, we could just use direct and indirect as recommended by Steve Lewis in PS8. We can discuss the name during development and amend this spec appropriately.

glance-direct: The end-user does a PUT of image data directly to Glance using a URL included in a response header to the image-create request. (The URL will also be known by convention, namely, v2/images/{image_id}/stage. After the data has been uploaded, the end-user follows with a call to Glance to process the data and complete the import.
swift-local: The end-user places the image data in the user’s object store account. Data placement may occur before or after the image record is created. After the data has been uploaded and an image record is created, the end-user makes a call to Glance to process the data and complete the import.

A particular Glance installation does not have to support all methods, but it’s expected that it will expose at least one.

Note

I’m trying to get by without giving the end-user any visibility into the staging area (formerly, the “bikeshed”). But see the discussion initiated by Stuart at line 207 in Patch Set 7 about separation of concerns and the example of Amazon’s s3 “multipart upload”.

API changes

value discovery

GET v2/info/import

The response is an object in JSON notation. This document should provide an end-user with sufficient information to perform an image import.

{
    "max_upload_bytes": {
        "description": "You may not upload more than this number of bytes when importing a virtual disk.",
        "type": "integer",
        "value": 10737418240
    },
    "max_virtual_bytes": {
        "description": "You may not upload a virtual disk whose virtual size exceeds this number of bytes.",
        "type": "integer",
        "value": 26843545600
    },
    "max_upload_time": {
        "description": "You only have this much time to complete your data upload.  Expressed in seconds.  (Does not apply to the 'swift-local' import method.)",
        "type": "integer",
        "value": 600
    },
    "data_TTL_after_import_error": {
        "description": "If an error occurs when you issue the command to import this data, your uploaded data is subject to deletion after this amount of time.  Expressed in hours.  (Does not apply to the 'swift-local' import method.)",
        "type": "integer",
        "value": 6
    },
    "source_container_format": {
        "description": "Your image data must be in one of these container formats.",
        "type": "array",
        "value": [ "bare" ]
    },
    "source_disk_format": {
        "description": "Your image data must be in one of these disk formats.",
        "type": "array",
        "value": [ "vhd", "vmdk", "raw" ]
    },
    "target_container_format": {
        "description": "Your image will be saved in Glance in one of these container formats.",
        "type": "array",
        "value": [ "ova" ]
    },
    "target_disk_format": {
        "description": "Your image will be saved in Glance in one of these disk formats.",
        "type": "array",
        "value": [ "vhd" ]
    },
    "os_type" : {
        "description": "Operating systems allowed for import.  Some operating systems may be non-importable due to licensing constraints.",
        "type": "array",
        "value": [ "linux", "windows" ]
    },
    "import-methods": {
        "description": "Import methods available at this site.",
        "type": "array",
        "value": [ "glance-direct", "swift-local" ]
    },
    "import-schema-location": {
        "description": "Location of the JSON schema for the image import request/response.",
        "type": "string",
        "value": "v2/schemas/import"
    }
}

Note

Should we allow users for sending target_* fields?

Suppose the end-user needs the image to be in a different format to use a particular flavor or availability zone. We could handle this by introducing a conversion task (out of scope for this spec, however), where end-user specifies an existing image and requests that an new image be created in a specific supported format (where “supported” format depends on the cloud). This would allow decoupling among:

what image formats are actually in use in the cloud
what image formats are supported for import
what image formats are supported for conversion

It would be good to have such decoupling because:

the in-use formats depend on what kind of hypervisors you have and what format they prefer
the import formats depend on what formats you are confident your screening processes are adequate for
the conversion formats depend on what you’re confident will convert correctly

summary

Method type: GET
Normal http response code(s): 200 (OK)
Expected error http response code(s): 400, 401, 405
- 400: request body passed
- 401: unauthorized
- 405: only GET supported for this call
URL for the resource: v2/info/import
- alternative: v2/info (I think we want the sub-resource, however, as it will keep the response simpler when export, image conversion, and possibly other functions are added.)
Parameters which can be passed via the URL: none
JSON schema definition for the body data: not allowed
JSON schema definition for the response data: none

format discovery

GET v2/schemas/import

The response is the following JSON schema.

{
    "id": "http://openstack.org/glance/import/request/schema#",
    "$schema": "http://json-schema.org/draft-04/schema#",
    "description": "OpenStack Glance image import request schema.",
    "type": "object",
    "additionalProperties": false,
    "required": [
        "method",
        "source_disk_format",
        "source_container_format",
    ],
    "properties": {
        "method": {
            "type": "object",
            "oneOf": [
                { "$ref": "#/definitions/glance-direct" },
                { "$ref": "#/definitions/swift-local" }
            ]
        },
        "source_disk_format": {
            "enum": [ "vhd", "vmdk", "raw" ],
            "description": "The disk format of the data to be imported."
        },
        "source_container_format": {
            "enum": [ "bare", "ovf" ],
            "description": "The container format of the data to be imported.  If no container, use 'bare'."
        },
        "os_type": {
            "enum": [ "linux", "windows" ],
            "description": "The type of the operating system contained in the image."
        }
    },
    "definitions": {
        "glance-direct": {
            "properties": {
                "name": {
                    "enum": [ "glance-direct" ],
                    "description": "Identifier for this import-method."
                }
            },
            "required": [
                "name"
            ],
            "additionalProperties": false
        },
        "swift-local": {
            "properties": {
                "name": {
                    "enum": [ "swift-local" ],
                    "description": "Identifier for this import-method."
                },
                "swift-location": {
                    "type": "string",
                    "maxLength": 255,
                    "description": "Name of the Swift object to be imported in container/name format."
                }
            },
            "required": [
                "name",
                "swift-location"
            ],
            "additionalProperties": false
        }
    }
}

Note that the values for source_disk_format and source_container_format will be pulled from configuration options used to supply values for the value discovery call. This will allow an end-user to do accurate schema-validation on the request.

Note

I haven’t addressed Stuart’s question “Do we need to have both ‘global’ and disk format specific parameters?” The question appears on the schema in Patch Set 5. Depending on how it’s answered, a usable schema may be more complicated than the one proposed here.

Note

The schema needs to allow for some provider-specific properties. An example is os_type, which affects how the Xen hypervisor creates the guest filesystem. A cloud may protect this property because otherwise changing its value from linux to windows will allow end users to run unlicensed Windows servers in a cloud, leading to violation of licensing terms, lawsuits, and general unhappiness.

I’ve included os_type in the schema, but there may be some other such properties, so I’ve added a work item to contact the operators group and the product working group so that we can get a (hopefully) definitive list before the Mitaka release and amend this spec and the schema appropriately.

summary

Method type: GET
Normal http response code(s): 200 (OK)
Expected error http response code(s): 400, 401, 405
- 400: request body passed
- 401: unauthorized
- 405: only GET supported for this call
URL for the resource: v2/schemas/import
Parameters which can be passed via the URL: none
JSON schema definition for the body data: not allowed
JSON schema definition for the response data: response is a JSON schema

image-create

This call already exists. We propose adding additional response headers to facilitate discovery. (We also discuss it here because the call is integral to the image import workflow.)

POST v2/images

The current request body and response body remain unchanged. The image is created in queued status (just as it is now for the v2 API).

Note that disk_format and container_format are currently optional in this request, which works well for image import since it’s possible that the formats of the imported data will be modified during the import process (for example, an OVA might be unpacked, or a bare disk might be packaged). The disk_format and container_format will be required in the image-import call, as will the common image property os_type. Values specified in the image-import call will overwrite any existing values in the image object.

New response headers

OpenStack-image-import-methods

The value of this header will be a comma-separated list of import method keywords. For example, OpenStack-image-import-methods: glance-direct,swift-local

OpenStack-image-glance-direct-url

The value of the header will be the URL to which the image data could be PUT by a subsequent call. This header will be included only if the site supports the glance-direct import method. (If the glance-direct name is changed, this header will be renamed as well. The method name is included here so that if there are multiple import methods that specify URLs to whihch data can be uploaded, the client will have a way to distinguish them.)

data-put

PUT v2/images/{image_id}/stage

This call will follow the specification for the current PUT v2/images/{image_id}/file call, with the following changes:

The call to /file is accepted only when the disk and container format fields have been set on an image. That will not be required for the call to /stage because the user will be supplying these in the subsequent image-import call.
The call will fail when the number of uploaded bytes exceeds the max_upload_bytes value published in the value discovery call.
The call will fail when the upload time exceeds the max_upload_time value published in the value discovery call.
The image will be set in status uploading rather than saving. Calls to /file shouldn’t be accepted if the image status is uploading. Likewise, calls to /stage shouldn’t be accepted if the image status is saving.

The user indicates that the data has been staged by issuing the image-import call (see below). Thus multiple data-put calls are allowed. This could allow two users in the same tenant to issue competing data-put calls, but for this implementation we will consider it the responsibility of the tenant to ensure that users cooperate appropriately.

If the subsequent processing on the image concludes that the image should be rejected, the image data will be deleted and the image will go to killed status.

In order to propagate information to the end user, a new message field will be added to the Image object. See [OSE4] for examples of the type of content the message field will contain.

summary

Method type: PUT
Normal http response code(s): 204 (No Content)
- Alternative is 202, but that’s not quite right because nothing’s going to happen with respect to the image data until the user makes a subsequent import call. In other words, I think 202 implies “your image is on the way”, and that’s not the case here. It made sense on the previous patch set, where a successful PUT did trigger the import processing, but that’s not the case anymore.
Expected error http response code(s): 401, 405, 409, 415
- 401: unauthorized
- 405: only PUT supported for this call
- 405: the glance-direct import method is not supported at this site
- 409: associated image is not in appropriate status
- 415: unsupported media type (must be application/octet-stream)
URL for the resource: v2/images/{image_id}/stage
Parameters which can be passed via the URL: none
JSON schema definition for the body data: none
JSON schema definition for the response data: no content in response

implementation issues

The exact nature of the stage (formerly, “bikeshed”) needs to be determined.

image-detail

This call already exists, but we discuss it anyway because it’s integral to the image import workflow.

GET v2/images/{image_id}

This returns an Image object formatted as defined by the JSON schema available at v2/schemas/image.

Two new image status values will be added to the Image object:

uploading: Why needed: This status conveys to the user that an import data-put call has been made. While in this status, a call to PUT /file is disallowed. (Note that a call to PUT /file on a queued image puts the image into saving status. Calls to PUT /stage are disallowed while an image is in saving status. Thus it’s not possible to use both upload methods on the same image.)
importing: Why needed: This status conveys to the user that an import call has been made but that the image is not yet ready for use. Data-put calls are not accepted when an image is in this state.

A new property will be added to the Image object:

message

Why needed: If an error occurs during the import process, the image will go to status killed, but that doesn’t tell the user what happened or provide any clue about the appropriate action to take.

Additionally, the message element can be used for non-error communication between Glance and the end user. See [OSE4] for examples of such usage.

image-import

POST v2/images/{image_id}/import

The request body must conform to the JSON schema retrievable from the format discovery call, otherwise the call will fail. There is no response body.

The status of {image_id} must be uploading or queued or the call will fail with a 409 (Conflict). (We need to allow image-import from queued for non-upload import methods, for example, swift-local or some type of copy-from functionality.) If Image {image_id} does not exist or is not owned by the caller, a 404 will be returned.

summary

Method type: POST
Normal http response code(s): 202 (Accepted)
Expected error http response code(s): 400, 401, 404, 405, 409, 415
- 400: malformed request
- 401: unauthorized
- 404: image record doesn’t exist or is not owned by the caller
- 405: only POST supported for this call
- 409: associated image is not in appropriate status
- 415: unsupported media type (must be application/json)
URL for the resource: v2/images/{image_id}/import
Parameters which can be passed via the URL: none
JSON schema definition for the body data: v2/schemas/import
JSON schema definition for the response data: no response

example (`glance-direct` method)

{
    "method": {
        "name": "glance-direct"
    },
    "source_disk_format": "raw",
    "source_container_format": "bare",
    "os_type": "linux"
}

example (`swift-local` method)

{
    "method": {
        "name": "swift-local",
        "swift-location": "import-data/my_image.raw"
    },
    "source_disk_format": "raw",
    "source_container_format": "bare",
    "os_type": "linux"
}

Alternatives

Here we consider some alternatives for ‘native’ (non-Swift) upload.

Use existing native v2 upload

We could use the existing synchronous v2 upload call, namely:

PUT v2/images/{image_id}/file

Advantages:

No impact on existing API users.

Disadvantages:

Rules out long lived validation/processing.

Change existing native v2 upload to be asynchronous

(This would include adding some simple discoverability call.)

Advantages:

Small change from existing behaviour. May not impact all use cases.

Disadvantages:

Changes existing API behaviour. Not considered backwards compatible.
Users/libraries must change to expect non-synchronous behaviour.

Asynchronous upload call

This would be a tweak to the existing v2 upload call, to make it asynchronous – but in a backwards compatible way. Backwards compatibility could be achieved by either using a different resource path, adding a header, or using a query parameter. (This would include adding some simple discoverability call.)

Advantages:

Very similar to existing upload mechanism.
Still just two API calls (POST, PUT)
No new image states are required.

Disadvantages:

Initially uploaded data is not cached for retry. (Same as existing upload.)
Users/libraries are expected to switch to the new method.
Slightly different work-flow than ‘non-native’ imports (ie imports from external sources)

Independent fileIds

Here, the uploaded bytes are not initially part of a particular image. This would work the same way as the import-from-Swift case, in that fileId operations are kept separate from image operations in a similar way that Swift object operations are kept seperate from image operations.

First you upload the bytes with a POST. This stores the bytes and generates a unique fileId. At this point the bytes are not ‘part’ of any image. You then make a call for an image to consume that fileId (analogous to consuming a Swift object). You then (optionally) delete the original fileId.

This could be implemented in slightly different ways.

A fileId could be completely unassociated with any image, in which case it could be reused as source data for more than one image.
Alternatively the fileId could be restricted to a particular image by requiring the image uuid to be specified when creating the fileId. In both cases, the only time a fileId would impact image state is when the image is consuming the fileId and goes through the usual queued, saving, active stages.

Advantages:

Swift and Glance work in the same way. (Initial data is not considerd ‘part’ of the image.)
Can be extended to uploading several fileIds sequentially or in parallel.
fileIds are cached for retry if the import fails
fileIds can each have their own size and checksum (for quota/integrity)
seperation of concerns (less impact on existing code areas, eg image delete would never have to handle deleting more than one data blob.)
No new image states are required.

Disadvantages:

More code, eg to list fileIds
More API calls than the simple async upload case
Users/libraries are expected to switch to the new method.

Asynchronous upload call and Independent fileIds

These two options could be combined. ‘Asynchronous upload’ would provide a ‘simple’ call for the common simple upload case, ‘independent fileIds’ would provide a more involved set of calls for parallel/segmented upload. (There are examples elsewhere of of having a simple call for the basic case and more advanced calls for more advanced functionality, eg regular Swift upload versus Swift Large Object upload.)

Advantages:

Simple API available for standard uploads
More advanced APIs available to those who need them
Could implement the simple case first

Disadvantages:

More code
Users/libraries are expected to switch to the new method.

URL nomination

Glance could nominate an opaque URL for the data to be PUT to. For example, the POST request could return:

put-data-to: https://example.com/xxx

If Swift is available the URL would be a Swift TempURL, if Swift is not available the URL would point to an appropriate Glance URL. As far as I know Swift’s TempURL allows range offset so parallel uploads/partial retries should work.

Advantages:

Nice and RESTFul
Almost equivalent behaviour whether Swift is in use or not

Disadvantages:

Swift TempURLs limit upload sizes (5GB by default). (Changes to Swift would be required.)
A user may have pre-existing image data in Swift. That case would need to be handled anyway.
In the Swift TempURL case no token header is required to be sent with the data. In the Glance case a token header would be required (unless we added TempURL type functionality to Glance). So the two things would not be completely equivalent. You’d have to know whether you’re sending to Glance or Swift – or at least whether to send a token or not. You could potentially just send the token in both cases, but that’s a little untidy (eg principle of least privilege).
Would require a special account for Swift imports (this would be the upload target), even in multi-tenant store mode because it may not be safe to nominate a URL in the user’s own account.
While (I think) parallel uploads/partial retries are possible with Swift’s TempURL they could only be attempted if it was known that the target was Swift, ie if the URL isn’t opaque. So we’d be back to having to know which service we’re sending the data to. (Unless we forfeited some of Swift’s advantages.)

Use existing tasks API

This works when Swift is present, but doesn’t provide a way to do an asynchronous ‘native’ upload.

Out of scope

As discussion on this spec has advanced, we’ve agreed that it does not encompass the following features. At the same time, we should keep in mind that at a later point, we may want to include such features. Thus, whatever design we come up with for image import should make it easy to accommodate them.

Out of scope: Image conversion during the import process
Out of scope: Specification of a common disk/container format that all sites must support
Out of scope: Equivalence of disk/container format(s) used in a particular cloud and the disk/container format(s) supported for image import. (In other words, no requirement that a site must allow import of all formats in use at that site. See the “note” in value-discovery section, above, for more about this.)
Out of scope: Image export (though obviously we want a solution that lends itself easily to export as well as import)

Data model impact

Two new image status values will be added: importing and uploading.

A new image property will be added: message.

REST API impact

Given that at least one new call is being added to the API, a minor version bump should occur.

Not exactly in scope for this spec, but the general consensus is that the current “tasks” API should be deprecated in this cycle and made an admin-only API.

Security impact

As its intent is to consume user-provided data, this feature will introduce new security risks. Historically, the Images v1 API was not meant to be exposed directly to end users, as it was expected that end users would use the Compute API for image related calls. The Compute API defines an “image-create” action, but this is an action on the servers resource, that is, it creates a snapshot of an existing instance, not the upload of an end-user supplied image. Further, discussion at the Havana summit (see [OSE1]) indicated that Images v2 upload shouldn’t be exposed to end users either, but should be reserved for trusted users. So even though it’s previously been possible for a particular deployment to expose image upload directly to end users, it hasn’t been a recommended practice.

Image import must be designed so that a deployer can halt image imports during an emergency.

We anticipate the following security risks:

It consumes user-provided data. Depending upon the container_format and disk_format allowed, this may expose Glance (and other projects that consume images from Glance) to decompression bombs or various tar-based attacks. (Consumers of images may already have mitigation strategies in place, but Glance itself currently does not.)
It enables a resource exhaustion attack. Vectors include: uploading a arbitrary number of bytes (can be mitigated by a max size restriction), uploading an allowable number of bytes over an extremely slow connection (can be mitigated by a Glance-side timeout), concurrent imports (can be mitigated by user quota, user limit, task queuing, or some combination).

(The remainder of the questions in this section will be addressed as the discussion advances.)

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

The image import process should emit notifications. (Will specify further after the basic design is worked out.)

We should implement notifications for at least the following parts of the import workflow:

The import operation has been accepted
Processing has begun
Processing status periodically (percentage)
Processing has been completed (success/failure)

Policy impact

Image import, as described herein, is an end user operation, not an admin operation. It is, however, necessary that it be governed appropriately by Glance policies. While the default policy setting will allow end users to import images, a small public deployer with limited resources might want to restrict image import to specific users.

Additionally, as mentioned above, it should be possible for a deployer to “turn off” image import if a vulnerability is discovered.

Other end user impact

We need to articulate clearly how the traditional Glance image immutability guarantee applies to image import, namely, an image is immutable once it goes active.

The python-glanceclient will be modified to support image import. (Will describe the interface after the basic design is worked out.)

Performance Impact

Describe any potential performance impact on the system. (Will fill in after the basic design is worked out.)

Other deployer impact

This change may affect how people deploy and configure OpenStack in the following ways:

Nova depends upon Glance to supply images and accept VM snapshots. Those operations are unaffected by this proposal.
Processing image imports may require dedicated resources, for example:
- dedicated nodes if the image validation is CPU-intensive

Developer impact

It is not anticipated that this feature will affect other developers working on OpenStack.

Implementation

Assignee(s)

Primary assignee:: brian-rosmaita
Other contributors:: mclaren

Reviewers

Core reviewer(s):: All current glance cores.

Work Items

API version bump: https://bugs.launchpad.net/glance/+bug/1523934
Policy rules and new configuration options: https://bugs.launchpad.net/glance/+bug/1523937

These depend on the policy change.
- Start working on the header changes: https://bugs.launchpad.net/glance/+bug/1523941
- Work on the API schema changes (discoverability): https://bugs.launchpad.net/glance/+bug/1523944
- Import call (task triggering): https://bugs.launchpad.net/glance/+bug/1523955
Make task api admin only: https://bugs.launchpad.net/glance/+bug/1527716
Document that /file is not recommended for public nodes.
Introduce policies so that the /file endpoint can be easily disabled for operators following the recommendation above. The default setting for these policies will not change the current behavior.
Contact the operators and product working groups about properties similar to os_type that should be included in the import request schema.
(internal) task configuration
python-glanceclient support
tempest tests
documentation

Dependencies

It is not anticipated that this feature will require dependencies not already used by the current Glance tasks.

Testing

It should be possible to add a tempest test that imports a small image.

Documentation Impact

The following will need to be added to the glance developer docs.

any new API calls
changes to any existing API calls
tasks info for glance developers who want to work on tasks
tasks info for deployers who want to use tasks (I suggest starting with this in the dev docs and then move it to one of the OpenStack operator manuals eventually)

FAQ

Everything you always wanted to know about Glance Image Import but were afraid to ask.

What is the use case addressed?

A cloud end-user has a bunch of bits that they want to give to Glance in the expectation that (in the absence of error conditions) Glance will produce an Image (record, file) tuple that can subsequently be used by other OpenStack services that consume Images.
Why is the “regular” upload insufficient?

Unlike taking a “snapshot” of a instance using the Compute API, where Nova handles the image creation, packaging, basic cataloging, and uploading into Glance, exposing image upload opens a cloud to human error (for example, not a VM image, incorrect type of image for the hypervisors in that cloud, maliciously packaged/structured image to attack the hypervisor) that exposes a cloud to extra support load, denial of service, or security concerns.

Plus, it’s not just security issues. Another common use case would be an end user uploading an OVA that could be automatically unpacked and the manifest introspected to set appropriate metadata on the image. Or if OpenStack were to agree to a common image interchange format, conversion of the imported image might be required.
How are the above problems addressed by the proposed import process?

In contrast to “regular” upload, which puts the image data directly into the storage backend, the image import process allows an operator to examine the putative image and perform validation/conversion/packaging (or nothing) before it’s stored in the backend.
Doesn’t this introduce unwanted variability into the import process?

No, it does not. The API is fixed so that the API request and response formats are identical in all OpenStack clouds. The variability is behind the scenes, after the user has made the data available to Glance but before the image status goes to ‘active’.

Note that while there are multiple import methods, each method is precisely defined and schematized so that the API calls/responses are standard.
Why multiple methods?

This is a situation in which a one-size-fits-all approach doesn’t really work. There are several issues that come into play that affect the user experience of both end users and operators, for example, the size of the images to be imported, the speed of available connections, the ability to deploy extra upload nodes, the presence/absence of an object store, then volume of end users doing image imports, the size of a cloud’s operations team, etc. Some usage patterns can be accommodated by the method we’re calling ‘glance-direct’ whereby an end-user directly streams the object into Glance; others might be better accommodated by import from an object store whereby end users can make use of appropriate tooling to ensure a good user experience when uploading large binary objects. Since OpenStack provides free software for an object store solution, and since Swift is deployed in roughly half of all OpenStack clouds, we propose implementing a ‘swift-local’ import method in which the image data would be uploaded to Swift out of band.

Additionally, in anticipation of the deprecation of the Images v1 API, some operators have pointed out that the “copy-from” capability in the older API is missing from the Images v2 API. While it’s not a driver for this work, a well-defined copy-from import method can be accommodated by this design.

In short, the multiple import methods allow us to have a consistent and discoverable API across clouds that will empower operators and end users to offer/use the import methods that work best for them.
What if I don’t like there being multiple import methods?

Believe me, we have thought about this carefully. The alternative would be to have different API calls for each of the different import methods. That, however, would mitigate against the goal of having stable API calls supported in all OpenStack-branded clouds, since not every cloud operator will want to expose all import methods. That would make it difficult for image import to be in DefCore.

Let’s be really careful in describing what we’re talking about here. The approved design for image import allows operator choice, but in a very constrained way. As far as the end user is concerned, each import method will operate identically in each OpenStack cloud, and we envision there being a small number of these methods as each must make it through the specs process and be implemented in-tree. Thus it will be possible to code a client to handle image import seamlessly from the end user point of view.
If there are multiple methods, must a cloud support them all in order to achieve the “OpenStack” brand via DefCore?

This is ultimately up to the DefCore project. Our idea is that since all OpenStack clouds would support at least one of the import methods, all OpenStack clouds would thereby have a working Images v2 API image import facility, and could pass an image import requirement.
If there are multiple methods, how can I determine what’s available in a particular cloud?

This can be done programmatically. (a) The GET v2/info/import returns a well-defined document that contains the list of import methods supported at a particular site. (b) A header that comes back from the POST v2/images call also contains the list of import methods supported at the site. See the value discovery section of this spec.
If there are multiple methods, how do I know what the request for the method I want to use is supposed to look like?

There’s a JSON schema with this information. See format discovery.
If there are multiple import methods, what’s to stop someone from adding more?

Additional methods would have to be proposed in a Glance spec and would have to meet the discoverability and interoperability constraints we’re implementing for image import. So it’s not the kind of thing that can be done lightly.
What about the old image upload call?

The PUT to v2/images/{image_id}/file makes sense as a call used by operators and trusted services, so we are not proposing that it be removed. We do, however, recommend that operators not expose it directly to end users.
I have a question you haven’t covered here.

Well, you could read through this entire spec. But here are some pointers:

For details about the import workflow and API calls, start at the Proposed change section.

If you want more background about why that workflow was chosen, you need to read the beginning of this document.

If you want a quick summary of the design constraints, start at the Summary of the Constraints Around This Project section.
Wait, wasn’t this spec approved for Mitaka?

Yes, it was. It didn’t get implemented because we wanted to make sure there was a thorough exploration of Alternatives before implementation began. The ultimate the consensus was that we should stick with the original proposal. The key point here is that the implementation of this spec has not been rushed through by any means.
The image import spec is really, really long, but I can’t get enough of it. Where can I read even more?

The spec has a list of References you may find interesting.

There are also some recent etherpads:
- Newton contributors’ meeting: image import
- Image Import MVP
Additionally, you may find the discussion on the following patches entertaining:

References

[OSD1]

http://developer.openstack.org/api-ref-image-v2.html#os-tasks-v2

[OSE1] (1,2)

https://etherpad.openstack.org/p/havana-getting-glance-ready-for-public-clouds

[OSE2]

https://etherpad.openstack.org/p/glance-upload-mechanism-reloaded

[OSE3]

https://etherpad.openstack.org/p/Mitaka-glance-image-import-reloaded

[OSE4] (1,2)

https://etherpad.openstack.org/p/glance-image-import-example

[OSL1]

http://eavesdrop.openstack.org/irclogs/%23openstack-glance/%23openstack-glance.2015-09-22.log.html#t2015-09-22T14:31:00

[OSM1]

http://lists.openstack.org/pipermail/openstack-dev/2015-September/thread.html#74360

[OSM2]

http://lists.openstack.org/pipermail/openstack-dev/2015-September/thread.html#74383

[OSO1]

http://www.openstack.org/brand/interop/

[OSR1]

http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/process/CoreCriteria.rst

[OSW1]

https://wiki.openstack.org/wiki/Glance-tasks-api

[OSW2]

https://wiki.openstack.org/wiki/Glance-tasks-api-product

[OSW3] (1,2)

https://wiki.openstack.org/wiki/Glance-tasks-import

[NEW1]

https://governance.openstack.org/resolutions/20151211-bring-your-own-kernel.html

[NEW2]

https://github.com/openstack/defcore/commit/10562c245a6332f52cb5c5d15739dfab15b2baa6

Other Relevant Supporting Information

[OSB1] https://blueprints.launchpad.net/glance/+spec/upload-download-workflow

[OSG1] https://review.openstack.org/#/c/220166/

[OSG2] https://review.openstack.org/#/c/220166/4/doc/source/tasks.rst

[OSW4] https://wiki.openstack.org/wiki/Glance-upload-mechanism-reloaded

Operator maintained images lifecycle

Sun, 11 Jan 2026 00:00:00

https://blueprints.launchpad.net/glance/+spec/hidden-images

This spec addresses the problem that cloud operators have into keep a public image list with only the latest images versions available.

Problem description

Cloud operators supply public images that can be used by end users to boot servers. An example is an image containing the CentOS 7 operating system. Such images must be updated as security concerns, etc., are addressed. In Glance, however, image data is immutable, so each update results in a new public image. Further, operators do not want to delete the “old” public images, as end users may require them for different use cases like server rebuilds. As a result, the default image-list for end users becomes very large. Further, the default image-list may contain multiple CentOS 7 images, for example, making it difficult for end users to determine which image to use.

Note

Example

An operator provides an image for CentOS 7 with a standard set of packages as image 1. Some minor security problem is discovered in OpenSSL, so the operator provides image 2 of CentOS 7 with updated OpenSSL. Then a kernel vulnerability is discovered and the operator issues image 3 of CentOS 7 with updated OpenSSL and a patched kernel. Each of these is a “version” of the image, but the same version of CentOS 7. The operator wants a new end user to start with image 3, but a user who’s been around a while longer may want to continue to use image 1 and patch/upgrade himself (for example, the OpenSSL update brings in a dependency that conflicts with some key software the user is running). If all three images have public visibility, then all three of them will appear in an end user’s default image-list.

A current practice is to address this by adding a custom property on an image, for example, "is_current": "yes", but this is operator-specific and not interoperable. This only solves part of the problem, however, because end users must be educated to look for the "is_current" image property. It would be better if only those images with "is_current": "yes" were included in the end user’s default image-list in the first place.

Proposed change

This spec proposes adding a new boolean column "os_hidden" in images table. Images where "os_hidden" = True will be omitted from the image list presented to the user. This will apply to all image visibilities. However, the images will continue to be discoverable.

Note

Example

An user wants a CentOS 7 provider image, so he uses: "?visibility=public" on the GET v2/images call. He sees a CentOS 7 image, but notices that it was created_at today, so he realizes that it’s not the same image that he’s searching for. So now he uses "?visibility=public&os_hidden=true" to get the list of all available images.

If the image has "os_hidden" = False the image is not omitted from the image list. It preserves the current behaviour.

At image creation, if not specified, it’s used "os_hidden" = False.

Changing the property “os_hidden” will be considered an image update. Because, the policy is already defined for this operation no other changes are required.

In the response of create/show image the new property will be displayed as os_hidden. If a pre-Rocky image already has a custom property named as os_hidden then that property will no longer be visible in the response from Rocky release.

All operations in the image will continue to be available considering the policy defined.

Alternatives

Instead using a new image property we can have a new visibility = “hidden”. Images with this new visibility state will not be in the default image list. To list images with visibility “hidden” it will be required to explicitly request it. Ex: "property --visibility=hide" Images with the visibility “hidden” will always be discoverable by the user.

This solution is less flexible because visibility “hidden” has potentially the same scope as “public”. The user roles that can use this visibility need to be controlled by policy.

Another approach is to use the proposed new image property “hidden” but not make these images discoverable with the API. However, there is the use case where a project may require a particular image version (for example: different OS releases like CentOS7.4 to CentOS7.5; appliance vendors that support their software on particular images). If “hidden” images are not discoverable cloud admins will need implement their own solution to expose these images.

Data model impact

Add the “os_hidden” boolean column in images table.

For the E-M-C migration strategy is proposed:

Triggers: not required. A pre-Rocky glance release will reject an image-update call setting ‘os_hidden’ with a 400 because it doesn’t recognize the field.
Expand: will add a boolean “os_hidden” column to the images table.
Contract: not required
Data Migration: Not required.

REST API impact

A new property “os_hidden” will be accepted for the GET call. GET v2/images … os_hidden=true/false By default the API will consider os_hidden=false.

Security impact

None

Notification impact

None

Other end user impact

The end user needs to be aware that the Cloud provider may “hide” old images. This is specific to each Cloud provider.

Performance impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: - Abhishek Kekane

Work Items

Add support in GET call for the property “os_hidden”. Consider the default “os_hidden=false”.
Change the image table schema adding a new field.
Change the glance-client to support the new property.

Dependencies

None

Testing

TBD

References

Handle sparse images

Sun, 11 Jan 2026 00:00:00

https://blueprints.launchpad.net/glance-store/+spec/handle-sparse-image

Some drivers like rbd and filesystem support sparse image, meaning not really write null byte sequences but only the data itself at a given offset, the “holes” who can appear will automatically interpreted by the storage backend as null bytes, and do not really consume your storage.

Problem description

As glance deal with instance image, it appear that they are majorly composed of null bytes sequence to represent the whole disk size of the instances, by exemple the 8GB base CentOS 7 cloud image contain 1GB of data for 7GB of holes, so it will significantly optimize storage usage and upload time.

Current implementation of rbd and filesystem driver rely on the utils.chunkreadable function, which will basically split the file to import into block of CHUNK_SIZE, then these blocks will be directly written to the backend whatever the content, and the offset will be incremented by the size of the chunk.

Here is an example for a ceph backend with a standard CentOS 7 cloud image using Glance:

$ rbd du 9b86961e-6bf3-4d0d-99dc-7c762fe6881d
NAME                                      PROVISIONED USED
9b86961e-6bf3-4d0d-99dc-7c762fe6881d@snap       8 GiB 8 GiB
9b86961e-6bf3-4d0d-99dc-7c762fe6881d            8 GiB   0 B
<TOTAL>                                         8 GiB 8 Gi
$ rbd export 9b86961e-6bf3-4d0d-99dc-7c762fe6881d /tmp/Centos7full.raw
$ md5sum /tmp/Centos7full.raw
aae49f6f57aecb9774f399149a0b7f35 /tmp/Centos7full.raw

And the same result when uploading the same image with qemu-img convert or rbd import:

$ rbd du 437e8de0-b897-4846-96aa-aff70cd8794c
NAME                                      PROVISIONED USED
437e8de0-b897-4846-96aa-aff70cd8794c@snap       8 GiB 1008 MiB
437e8de0-b897-4846-96aa-aff70cd8794c            8 GiB      0 B
<TOTAL>                                         8 GiB 1008 MiB
$ rbd export 437e8de0-b897-4846-96aa-aff70cd8794c /tmp/Centos7sparse.raw
$ md5sum /tmp/Centos7sparse.raw
aae49f6f57aecb9774f399149a0b7f35 /tmp/Centos7sparse.raw

We can see here that the checksum of the downloaded file, either sparse or not stay the same, so it should not have impact on the file integrity. In both case, the glance image-download command will produce a non sparse file because download process just read the file in the backend chunk after chunk, so null byte sequence will be read, sparse file or not.

Proposed change

There is two successive optimization we can make to achieve the same result as other import tool like qemu-img:

Do not write null bytes sequences inside chunk (Write optimization)
Rely on filesystem instruction to skip holes (Read optimization)

A new configuration option enable_thin_provisioning will be added to rbd and filesystem backend in order to make it switchable by operator. Enable it will enable both read and write optimization.

Do not write null bytes sequences inside chunk

This first optimization will work in all case, wether or not the image file is sparse or not, it is the behaviour implemented in qemu-img. It consist on checking if the chunk readed is only composed of null bytes, if it’s the case, just increase the offset without writing any data to the store.

Rely on filesystem instruction to skip holes

This second optimization will rely on the syscall SEEK_HOLE and SEEK_DATA, available since kernel 3.8 and python 3.3. It consist on directly skipping holes, without even reading the null bytes sequences, which can be very long in case of a large image like an appliance (hundred of GB). As it rely on linux kernel syscall, older linux kernel or Windows node will just skip the optimization and work like before.

This second optimization can only work when the image file is actually considered as sparse by the filesystem, so it require to be converted “in-place” on staging store to raw file by the convert plugin of import workflow. If not, by exemple by sending directly a raw file trough Glance REST API, filesystem of the staging store won’t be aware of the hole.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Write optimization

These tests have been done against 2 rbd backend sent through web-download image-import workflow, with raw conversion enabled.

For a 8GO Centos qcow2:

Chunk size	8MB	32MB	64MB
Time without sparse upload	3min31	3min26	3min28
Time with sparse upload	1min59	1min58	2min04
	-44%	-43%	-40%
Storage used without sparse upload	8 GiB/8 GiB	8 GiB/8 GiB	8 GiB/8 GiB
Storage used with sparse upload	1.0 GiB/8 GiB	1.0 GiB/8 GiB	1.0 GiB/8 GiB
	-88%	-88%	-88%

For a 200GO Centos qcow2:

Chunk size	8MB
Time without sparse upload	4h
Time with sparse upload	41min11
	-83%
Storage used without sparse upload	200 GiB/200 GiB
Storage used with sparse upload	5.8 GiB/200 GiB
	-88%

Read optimization

The following tests have been done by reading data of a Centos 7 image file

	Centos 8GB Qcow2	Centos 8GB RAW	Centos 100GB Qcow2	Centos 100GB RAW
Read all file (including holes)	0m3.964s	0m16.746s	0m4.666s	3m4.003s
Read only data (skip holes)	0m2.662s	0m4.686s	0m3.916s	0m4.425s
	-32,8%	-72,0%	-16,1%	-97,6%

The optimization for the Qcow2 image tends to be negligible, as Qcow2 images does not have holes, so it should be very fast in all case. The point here is to show that there is no negative impact for Qcow2 images, and huge positive one for raw images, so we can apply this behaviour in all case.

Other deployer impact

Addition of a new enable_thin_provisioning configuration option for rbd and filesystem store will require operator to enable it. Without this option, behaviour will stay the same as before.

As this configuration option is per store, it is possible in a multi-store environment to choose on which store it will be enabled.

Developer impact

None, as these optimizations are handled inside drivers itself and should not change their interfaces.

Implementation

Assignee(s)

Primary assignee:: alistarle
Other contributors:: yebinama

Work Items

Update drivers who can handle sparse images: filesystem and rbd.

Dependencies

None

Testing

Testing that there is no functional regression for the modified drivers.
Testing that it does not have a negative impact on system where SEEK_DATA/SEEK_HOLE instruction are not available.

Documentation Impact

Document the new configuration option enable_thin_provisioning for rbd and filesystem driver.

References

Original ceph.io article who expose these optimizations: https://ceph.io/planet/importing-an-existing-ceph-rbd-image-into-glance/

Initial abandonned patch in glance_store: https://review.opendev.org/#/c/430641/

Python implementation of SEEK_HOLE/SEEK_DATA syscall: https://bugs.python.org/issue10142

Add new cinder’s attachment API and multiattach support

Sun, 11 Jan 2026 00:00:00

https://blueprints.launchpad.net/glance-store/+spec/attachment-api-and-multiattach-support

Make glance cinder store use cinder’s new attachment APIs for image operations.

Problem description

Currently when we make simultaneous requests to glance for using the image to create multiple volumes or launch multiple instances, the same image volume needs to be attached multiple times to serve the requests and since the support doesn’t exist in glance cinder store, some of the requests fails.

Proposed change

Cinder’s new attachment API was introduced in microversion 3.27 which should be supported by cinder and is required to pass while making calls through cinderclient. There were 2 new methods introduced, attachment_complete in MV 3.44 and passing attach mode during attachment_create in MV 3.54 which are also required and hence we will be using MV 3.54 for attachment related operations. All these changes are required prior to adding the multiattach handling.

This should be a 2 stage change:

1) Replace existing API calls to cinder for attachment with new attachment API calls

Add multiattach handling for glance cinder usecase

Alternatives

Use cinder’s image volume cache that will clone the first bootable volume created from image and subsequent requests will result in cinder’s clone operation instead of downloading the image from glance.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

The current number of calls to cinder will be reduced resulting in performance improvement. Example: initialize_connection and attach calls will be replaced by attachment_update and similarly terminate_connection and detach calls will be replaced by attachment_delete.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: * whoami-rajat

Other contributors:: None

Reviewers

Core reviewer(s):

abhishek-kekane
jokke
rosmaita
smcginnis

Work Items

Replace existing calls to old attachment methods with the new attachment API calls
Modify cinderclient calls to include microversion
Add code for multiattach handling
Add appropriate unit tests

Dependencies

None

Testing

Unit tests to have the code coverage and the new attachment code path will be tested with the glance cinder tempest job running on glance gate.

Documentation Impact

None

References

None

Eliminate Redundant Downloads of Uncached Images

Sun, 11 Jan 2026 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/glance/+spec/duplicate-downloads

Multiple requests for an image that is not yet cached on the Glance API node handling the request currently results in multiple download requests for the same image from the backend store. For example, 1000 concurrent build requests based off an uncached image can result in 1000 download requests from the backend store.

Problem description

Feature: Elasticity

    In order to briefly leverage the power of the Cloud to do some work,
    As an OpenStack Powered Cloud customer leveraging Glance caching
    I want to quickly provision a large number of servers, perform some
    work, and then destroy them.

    Scenario Outline: Concurrent Requests for Uncached Image
        Given a single Glance API node
        And the requested image exists in the backend store
        And the requested image is uncached on the Glance API node
        When <n> concurrent request(s) for the image is/are made
        Then the image will be downloaded from the backend store <m> times
        And the image will be cached on the Glance API node
        And every request for the image will succeed

        Examples: Concurrent Requests
            | n | m |
            | 1 | 1 |
            | 2 | 1 |

    Scenario: Concurrent Requests for Uncached Image Fails
        Given a single Glance API node
        And the requested image exists in the backend store
        And the requested image is uncached on the Glance API node
        When 2 concurrent requests for the image are made
        And mid-download the client closes the first connection
        Then only the first download request will fail
        And the image will be cached on the Glance API node

    Scenario: Stream to all requests while caching
        Given a Glance API node (1) with this feature deployed
        And a Glance API node (2) without this feature deployed
        And the requested image exists in the backend store
        And the requested image is uncached on Glance API node 1
        And the requested image is uncached on Glance API node 2
        When 2 concurrent requests are made to API node 1
        And 2 concurrent requests are made to API node 2
        Then the 2 requests to API node 1 will succeed
        And the 2 requests to API node 2 will succeed
        And the image will be cached on Glance API node 1
        And the image will be cached on Glance API node 2
        And the request completion time between the 2 requests to node 1
            will be statistically less than or equal to
            the request completion time between the 2 requests to node 2

Proposed change

Currently, the Glance caching middleware returns an iterator that downloads from the cache only if the image is already cached. If the image is uncached, the request is passed onto the API to obtain an iterator that will download directly from the store. The response from the API containing this direct iterator is returned back through the caching middleware. If the image is completely uncached when the middleware processes the response, it will wrap the direct download iterator from the API in an another iterator that will tee to the cache (i.e. read from the store and write to both the client and the cache via a split pipe).

Therefore, depending on the state of the cache, one of three iterators can be returned: an iterator to the cache (if the image is completely cached), an iterator to the store (if the image is partially cached), or a teeing iterator that streams from the store and writes to the cache (if the image is completely uncached). This approach is racey and can result in many responses downloading directly from the store and a subset of those teeing data to the same location on the filesystem.

The proposed solution is for the first download request to instead of writing the whole file to the cache, we write the file to cache in chunks. Then, the subsequent download requests read from the chunks that have been written. Once the subsequent request finishes reading all the available chunks in the cache, it will wait for the next available chunk written to the cache by the first request. It will keep doing this until the first request finishes all the chunks.

For the first request:

if the cache entry does not exist:
    mark the image "caching"
    create a new folder in the cache directory with the image id
    take the iterator from the download (like we are doing now)
    write the data in 1GB chunks to cache
    upon finish, mark the image "cached"

For the subsequent request:

if the image is marked "caching" or "cached":
    read the chunk from the cache until we get all the expected chunks
    if a chunk is not available:
    wait for it to be written by the first request

Note

Note: The hit count of cached image should not be increased for each chunk read, instead it should be increased once per actual request to read the image from cache.

Alternatives

Add a configuration option, eliminate_duplicate_downloads, to enable this feature. The addition of a configuration option to control how the caching middleware behaves puts unnecessary burden on the operator. The caching middleware should meet the expected behaviors as outlined in the problem description without introducing a new configuration option. The only value of such option is to allow a phased roll-out of the feature. If the consensus is to introduce such an option, being defaulted to disabled, it should then be deprecated and defaulted to enabled in the next release.
To avoid streaming partial image to multiple clients in case of the initial caching request failing we could block all the subsequent requests until the image is fully in cache and serve those only from cache.

This approach would cause significant delay serving the rest of the clients with a benefit of saved bandwidth in those rare cases where the caching gets interrupted by the image or store going unavailable. Due to possible very long delays on large images this would complicate the download process as some kind of keepalive for the client connection would be needed to avoid timeouts.
Create a lock within the middleware request handler: This prevents requests from reaching the root app and establishing a download iterator in a race to be the first to initiate the download in the cache middleware response handler. However, it comes at a reliability and complexity cost. Logic would have to be implemented in the request handler to recover from failures between the request and response. That’s a lot of squeeze for not a lot of juice.
Move the cache out of the middleware into the root app and provide a locking mechanism around caching and downloading. There are architectural benefits to this. However, it is a serious undertaking, and I believe that any conversations around this should be had completely outside the context of this change.
Move cache out of Glance API: This requires client side logic and new / external caching code.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

See Other deployer impact

Performance Impact

Image request time for concurrent requests will decrease.
Bandwidth consumed between Glance API nodes and backend store will decrease.

Other deployer impact

Every request being served from the cache will impact the reliability and performance profile. The bottleneck between the backend store and Glance will be removed for the thundering herd problem. However, there could still be a bottleneck between the hypervisors and the Glance API nodes.

Developer impact

None

Implementation

Assignee(s)

Primary assignee: Mridula Joshi

Reviewers

Core reviewer(s): Erno Kuvaja

Work Items

Add tests
Update the cache methods in the drivers
Update the cache request handler
Update the cache response handler
Update the docs

Dependencies

None

Testing

Unit Tests
Functional Tests

Documentation Impact

Document any new configuration options, if any.

References

https://review.opendev.org/c/openstack/glance-specs/+/206120

Spec Lite: Add Application Credential Support to Cinder Backend

Tue, 16 Dec 2025 00:00:00

project:

glance_store

problem:

The Cinder backend driver in glance_store currently uses V3Password authentication for backend storage operations. This prevents Zero Downtime Password Rotation (ZDPR) from working for Glance deployments using Cinder as the image backend.

When the service user password is rotated during ZDPR, Cinder backend operations fail because the driver cannot authenticate using the rotated password. This creates a critical gap in ZDPR functionality, as image uploads and downloads to/from Cinder volumes will fail during password rotation, defeating the zero-downtime goal.

The main Glance API service already supports application credentials via keystonemiddleware, but backend storage operations (Cinder and Swift) do not support application credentials, creating an incomplete ZDPR implementation.

solution:

Add application credential support to the Cinder backend driver by:

Adding configuration options for application credentials: - application_credential_id (registered in both [cinder] and [backend_defaults] groups) - application_credential_secret (registered in both [cinder] and [backend_defaults] groups, marked as secret)
Modifying the Cinder driver’s authentication logic to: - Use BackendGroupConfiguration to read configuration (supports fallback to [backend_defaults] section) - Check for application credential configuration first - Use V3ApplicationCredential authentication if AC credentials are available - Fall back to V3Password authentication if AC credentials are not configured (backward compatible)
Update get_cinder_session() function to support application credentials for Cinder volume operations.

This unified approach allows Cinder backend to use the same AC credentials from [backend_defaults] section (when same service user is used) or per-backend overrides (when different service users are configured), aligning with existing glance_store architecture.

how:

Add application_credential_id and application_credential_secret options to Cinder driver configuration options
Check for AC credentials and use V3ApplicationCredential when available
Maintain backward compatibility by falling back to V3Password if AC credentials are not provided

alternatives:

None

impacts:

DocImpact, ConfigImpact

timeline:

Include in 2026.1 release (or next appropriate release cycle)

link:

None

reviewers:

croelandt, rosmaita

assignee:

abhishekk

Spec Lite: Add Application Credential Support to Swift Backend

Tue, 16 Dec 2025 00:00:00

project:

glance_store

problem:

The Swift backend driver in glance_store currently uses V3Password authentication for trustee authentication when using service user credentials. This prevents Zero Downtime Password Rotation (ZDPR) from working for Glance deployments using Swift as the image backend.

When the service user password is rotated during ZDPR, Swift backend operations fail because the driver cannot authenticate using the rotated password. This creates a critical gap in ZDPR functionality, as image uploads and downloads to/from Swift containers will fail during password rotation, defeating the zero-downtime goal.

The main Glance API service already supports application credentials via keystonemiddleware, but backend storage operations (Swift and Cinder) do not support application credentials, creating an incomplete ZDPR implementation.

solution:

Add application credential support to the Swift backend driver by:

Adding configuration options for application credentials: - application_credential_id (registered in both [swift] and [backend_defaults] groups) - application_credential_secret (registered in both [swift] and [backend_defaults] groups, marked as secret)
Modifying the Swift driver’s authentication logic to: - Use BackendGroupConfiguration to read configuration (supports fallback to [backend_defaults] section) - Check for application credential configuration first - Use V3ApplicationCredential authentication if AC credentials are available - Fall back to V3Password authentication if AC credentials are not configured (backward compatible)
Update both SingleTenantStore and MultiTenantStore init_client() functions to support application credentials for trustee authentication.

This unified approach allows Swift backend to use the same AC credentials from [backend_defaults] section (when same service user is used) or per-backend overrides (when different service users are configured), aligning with existing glance_store architecture.

how:

Add application_credential_id and application_credential_secret options to Swift driver configuration options
Update SingleTenantStore.init_client() and MultiTenantStore.init_client() to check for AC credentials and use V3ApplicationCredential when available
Maintain backward compatibility by falling back to V3Password if AC credentials are not provided

alternatives:

None

impacts:

DocImpact, ConfigImpact

timeline:

Include in 2026.1 release (or next appropriate release cycle)

link:

None

reviewers:

croelandt, rosmaita

assignee:

abhishekk

Download Image from Suggested Stores

Tue, 07 Oct 2025 00:00:00

https://blueprints.launchpad.net/glance/+spec/download-from-specific-store

Right now when you download an image using GET /v2/images/{image_id}/file, Glance tries the default store first. If the image is not there, it tries other stores until it finds the image. This spec adds a way to tell Glance what ordering to use to locate the image data for a particular request. You can give a list of stores you want to try. This helps compute nodes and services like Nova and Cinder download from stores that are close to them.

Problem description

In deployments with multiple stores, images can be in different places like Swift, Cinder, filesystem, or S3. Right now Glance downloads from the default store first. If not there, it tries other stores one by one until it finds the image.

This has problems:

Services cannot say which store to use
Services cannot pick a faster store or one that is close to them
Operators cannot test if a specific store is working
Cannot spread downloads across different stores

The store weight feature from 2023.2 lets operators set store priorities. But this is only for operators. Services like Nova and Cinder cannot choose stores based on where they are running.

Use cases:

Services can find available stores using existing APIs like GET /v2/info/stores and GET /v2/info/stores/detail. These show store names, descriptions, types, and properties that help decide which store to use.

Performance: A compute node can download from a local store instead of a remote one
Proximity: Edge compute nodes can download from stores close to them (like “us-east” or “eu-west”)
Testing: Operators can test if a specific store works
Load distribution: Spread downloads across stores so one store does not get too much load

Cost concerns:

If users can pick any store, they might pick expensive ones. For example, if there is a backup store on S3, downloading from it costs money for data egress. Users may not know this. To fix this, operators can use policy to control who can use this feature. Operators can also use good store names and descriptions to guide users to cheaper stores.

Proposed change

Add new query parameters to the existing download endpoint. This lets services suggest the ordering of stores to try.

API call: GET /v2/images/{image_id}/file?prefer=store1,store2

How it works:

If caching is enabled, and the image is cached, the ‘prefer’ parameter is ignored and the data is returned.
Check if the stores in the list exist and are configured. Return errors if store does not exist.
Try stores from the list in order. Download from the first store that has the image.
If image is not found in any of the listed stores, fall back to default behavior (try all stores)

This approach is better than a new endpoint because:

Does not add another download API
Easy for Nova and Cinder to add preferred stores without big changes
Still backward compatible - if no parameters given, works like before

Note

Important cache behavior: Store preference parameters only work when fetching images from backend stores, not when serving from cache. When an image is cached, the cache middleware serves it directly and bypasses the store preference logic entirely. This means cached images will always return HTTP 200 regardless of store parameters, and store preference validation (400 errors for invalid stores) will not occur for cached images.

Policy for store preference:

A new policy rule “download_from_store” will be added to control who can use store preference parameters. The default policy will allow anyone who can download images to also use store preference. This matches the current behavior where store preference is available to all users who can download. Deployers can restrict this policy later if needed. For example, they can make it admin or service role only to prevent regular users from picking expensive stores.

Alternatives

New endpoint approach: Create GET /v2/stores/{store_id}/{image_id}/file endpoint. This follows the same pattern as delete-from-store API. But it only allows one store at a time. Services would need to make multiple calls to try different stores. This makes things slower and more complex.
Header-based approach: Use HTTP headers to say which store. But headers are harder to discover and use.
Store weight only: Use the existing store weight from 2023.2. But this only works at operator level. Services cannot choose stores based on where they run.
No change: Keep current behavior. But this does not let services pick stores close to them.

Data model impact

None

REST API impact

Modified endpoint: GET /v2/images/{image_id}/file

New query parameters:

prefer: Comma-separated list of store names (optional)

Response codes:

200 OK: Image downloaded
400 Bad Request: Invalid store name or bad parameters
403 Forbidden: User does not have permission to use this feature
404 Not Found: Image not found

Examples:

Download from preferred stores:

curl -H “X-Auth-Token: $TOKEN”
“http://glance-api:9292/v2/images/{image_id}/file?prefer=local-store,backup-store”

Download normal way (no change):

curl -H “X-Auth-Token: $TOKEN”
“http://glance-api:9292/v2/images/{image_id}/file”

Store discovery:

Services can find available stores using GET /v2/info/stores:

curl -H “X-Auth-Token: $TOKEN”
“http://glance-api:9292/v2/info/stores”

Response example:

{
  "stores": [
    {"id": "local-store", "description": "Fast local store"},
    {"id": "backup-store", "description": "S3 backup store"}
  ]
}

Backward compatibility:

If no query parameters are given, the endpoint works exactly like before. This means existing code keeps working without any changes.

Security impact

Access control:

The normal image download policy still applies. Users must have permission to download the image. A new policy rule “download_from_store” controls who can use store preference parameters. By default, anyone who can download images can also use store preference. Deployers can restrict this policy later if needed.

Store validation:

Glance will check that store names are valid and configured before trying to use them.

No new data exposure:

This only changes which store to download from. Users can only download images they already have access to.

Notifications impact

None

Other end user impact

User docs and API reference need updates to explain the new query parameters.

Performance Impact

Positive impact:

Services can pick faster stores or stores close to them. This can make downloads faster. uncached requests.

Other deployer impact

Store management:

Deployers should use good store names and descriptions. This helps services pick the right stores. For example, “local-fast” vs “backup-s3” makes it clear which is local and which is expensive.

Cost considerations:

Services might pick expensive stores like S3 backup stores without knowing the cost. To handle this:

Use clear store names to show which stores are expensive
Add policy rules to limit who can use store suggestion
Consider making this admin or service role only initially

Policy controls:

Operators can add optional policy rules to control who can use this feature. This can be based on user roles or projects.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: abhishek-kekane
Other contributors:: None

Work Items

Add query parameter parsing to download endpoint
Implement store list validation
Add policy support for store suggestion control
Add logic to try stores from the list, fallback to default if not found
Add unit tests for new parameters
Add functional tests for different scenarios
Update API documentation

Dependencies

None

Testing

Add required unit and functional tests
Add tempest tests

Documentation Impact

Update API documentation
Update user docs
Update operator docs

References

Store weight specification: https://specs.openstack.org/openstack/glance-specs/specs/2023.2/approved/glance_store/store-weight.html
Store weight implementation: https://review.opendev.org/c/openstack/glance_store/+/885595

Enhanced filesystem driver with configurable timeout protection

Tue, 30 Sep 2025 00:00:00

https://blueprints.launchpad.net/glance-store/+spec/filesystem-timeout-protection

Add timeout protection to filesystem operations. This prevents operations from hanging forever when storage is not available. This works for all filesystem types (local or network) based on configuration.

Problem description

The filesystem store driver in glance_store does not have any timeouts for filesystem operations. When storage becomes unavailable, operations like os.path.exists(), os.unlink(), and os.statvfs() can hang forever. This causes Glance API timeouts and bad user experience.

Common scenarios where these problems happen:

NFS server outages or network problems
Storage backend failures (PowerFlex, PowerStore, etc.)
Network connectivity issues between Glance and storage
Storage maintenance windows
Local disk failures that cause IO operations to hang

When these problems happen, Glance operations like image deletion, retrieval, and storage capacity checks will hang forever. This causes:

API timeouts and bad user experience
Resource exhaustion on Glance API servers
Cannot perform maintenance operations
Cascading failures in OpenStack deployments

Proposed change

Add timeout protection to the filesystem driver. This prevents operations from hanging forever. This keeps backward compatibility by default. Default is blocking IO (timeout = 0 means wait forever).

Three new configuration options will be added:

[filesystem]
# Timeout for all filesystem operations (seconds)
# Set to 0 to disable timeout protection (blocking IO, normal behavior)
# Set > 0 to enable timeout protection with thread pool
# Recommended: 30 seconds for network storage, higher for slow networks
filesystem_store_timeout = 0

# Thread pool size for timeout-protected operations
# Only meaningful when filesystem_store_timeout > 0
# Ignored when filesystem_store_timeout = 0 (no thread pool is created)
# Each store instance gets its own pool to avoid starvation
# Set based on expected concurrency and WSGI worker count
filesystem_store_thread_pool_size = 10

# Thread pool usage threshold for warning logs (percentage)
# Only meaningful when filesystem_store_timeout > 0
# Ignored when filesystem_store_timeout = 0 (no thread pool is created)
# When thread pool usage exceeds this threshold, a warning is logged
# indicating that the pool is getting busy and may start blocking
filesystem_store_threadpool_threshold = 75

The options filesystem_store_thread_pool_size and filesystem_store_threadpool_threshold are only used when filesystem_store_timeout > 0. When filesystem_store_timeout = 0, these options are ignored since no thread pool is created.

We use ThreadPoolExecutor to wrap filesystem operations when filesystem_store_timeout > 0. When filesystem_store_timeout = 0 (or not set), operations run directly without thread pool overhead (blocking IO, current behavior).

Each store instance will have its own thread pool.

When an operation times out:
- The calling thread stops waiting and raises TimeoutError
- The worker thread continues running (cannot be killed) but we move on
- Error message: “Operation timed out after {timeout}s. Check storage health and connectivity.”

When filesystem_store_timeout = 0 (or not set):

Operations run directly without thread pool overhead
No timeout protection (current behavior, backward compatible)
The options filesystem_store_thread_pool_size and filesystem_store_threadpool_threshold are ignored (no thread pool is created).

Operations that will be wrapped (when timeout > 0):

For the first implementation, timeout protection will be limited to metadata operations. Data pipeline operations (get(), add()) are more complex and will be considered for a future enhancement.

delete() - wraps os.unlink()

Note

When delete() times out, we return an error to the user. But the worker thread may still complete the deletion later if storage becomes available again. This can leave the DB record while the file is gone. We will consider deleting from the DB first, then the store. That way, if the store delete times out, the DB is already cleaned up. Operators can handle orphaned files later. This potential inconsistency will be documented. We may enhance this later to handle it better. For example, if a subsequent delete() finds the file already gone, treat it as success (idempotent delete).
get_size() - wraps os.path.getsize()
_get_capacity_info() - wraps os.statvfs()

Data pipeline operations (future enhancement):

add() - wraps os.path.exists(), file open(), and file writes
get() - wraps file location resolution (os.path.exists, os.path.getsize) and ChunkedFile creation (which opens the file via open())

Note

Protecting get() is complex because it only protects the initial file open, not the streaming reads that happen later when the iterator is consumed. If the mount goes away during streaming, those reads will still hang. Full protection would require a wrapper iterator that delegates all reads to the thread pool, which adds significant complexity and overhead.

Alternatives

None.

The following alternatives were considered but none are useful:

Filesystem Detection Approach

We could detect filesystem types (NFS, CIFS, etc.) and only apply timeout protection to network filesystems. This adds complexity without clear benefit. Local filesystems can also hang (failing disks). Operators can configure timeout protection if needed.

Dedicated NFS Driver

We could create a separate NFS driver alongside the filesystem driver. This would require operators to change their driver configuration. This would break backward compatibility.

Application-Level Timeouts

We could implement timeouts only at the API level. This does not work because the hanging happens at the filesystem driver level. We need protection at the storage layer.

No Configuration Options

We could skip adding configuration options and just hardcode timeouts. This was rejected because different storage backends have different latency characteristics. Operators need the flexibility to tune timeouts.

Data model impact

None.

REST API impact

None.

Security impact

This prevents resource exhaustion attacks. An attacker could exploit hanging filesystem operations to consume server resources. Since each store instance has its own thread pool, an attack on one filesystem cannot exhaust all server resources. The damage is limited to that specific store.

Notifications impact

None.

Other end user impact

Users will get faster failure responses when storage is unavailable. This happens instead of indefinite hanging (when timeout protection is enabled). Operators will also get warning logs when thread pool usage exceeds the configured threshold, helping them identify potential issues before operations start failing.

Performance Impact

When timeout protection is enabled (timeout > 0), there is some overhead from thread pool operations. When disabled (timeout = 0), there is no performance impact (normal behavior). This prevents resource exhaustion from hanging operations when enabled.

Other deployer impact

Deployers can configure the timeout value based on their storage characteristics. The default of 0 (blocking IO) keeps backward compatibility. For network storage or unreliable storage, operators can set filesystem_store_timeout > 0 to enable protection.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: abhishek-kekane

Work Items

Implement thread pool wrapper for filesystem operations with timeout
Update delete(), get_size(), and _get_capacity_info() to use thread pool when timeout > 0
Implement thread pool usage monitoring and warning logs when threshold is exceeded
Add generic timeout error messages
Document timeout edge cases (e.g., delete operations that complete after timeout)
Write unit and functional tests for timeout behavior

Dependencies

None

Testing

Tempest tests for timeout behavior

Documentation Impact

Document new configuration options
Provide guidance on timeout values for different storage types
Document potential inconsistency when delete() times out (file may be deleted later, leaving DB record)

References

None.

RBD Erasure-Coded Pools Support

Tue, 30 Sep 2025 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/glance/+spec/rbd-erasure-coded-pools

This specification proposes adding support for erasure-coded pools in the Glance RBD store driver for cold storage and archival use cases. Erasure-coded pools reduce storage overhead by 50-75% compared to traditional 3x replication, but have significant performance costs (CPU overhead, slower operations). This makes them suitable for infrequently-accessed images where storage efficiency is more important than performance.

Problem description

The current RBD store driver in Glance only supports replicated pools, which typically use 3x replication and require 200% storage overhead. For large-scale deployments with terabytes of images, particularly cold storage or archival images that are infrequently accessed, this overhead is expensive.

Erasure-coded pools can provide similar data protection with 50-75% less storage overhead, but they have significant performance trade-offs (higher CPU usage on the Ceph cluster, slower write and read operations, slower recovery). These trade-offs make them unsuitable as a general replacement for replication, but valuable for specific use cases where storage cost is more important than performance.

Proposed change

This specification proposes extending the RBD store driver to support erasure-coded pools using Ceph’s two-pool model.

Use librbd’s native data_pool parameter to store image metadata in a replicated pool and image data in an erasure-coded pool.

Before enabling this feature, deployers must create and configure the required pools on their Ceph cluster. The following commands are expected to be run:

$ ceph osd pool create images_data erasure
$ ceph osd pool create images replicated
$ ceph osd pool set images_data allow_ec_overwrites true

The allow_ec_overwrites true setting is required on the erasure-coded pool. Without this setting, image creation will fail when using the two-pool model, as librbd needs to be able to overwrite objects in the erasure-coded pool for metadata operations.

Add rbd_store_data_pool configuration option to specify the erasure-coded pool for data storage. If not configured, the driver behaves exactly as it currently does with single replicated pools.

Implementation:

When creating new images, pass the data_pool parameter to librbd’s create() method. librbd handles all the complexity of managing the two pools.
Existing images remain in their original single-pool location (all data and metadata in the replicated pool). librbd transparently handles reading them from their current location.
New images created after enabling rbd_store_data_pool will use the two-pool model (metadata in the replicated pool, data in the erasure-coded pool).

When rbd_store_data_pool is enabled, the replicated pool (specified by rbd_store_pool) will contain both: * Existing images: All image data and metadata (single-pool storage) * New images: Only image metadata (two-pool storage)

While this mixing is technically supported by librbd, it is not recommended for production deployments. Operators should consider one of the following approaches:

Use separate pools (recommended): Create a new replicated pool specifically for metadata when enabling erasure-coded pools, keeping the existing pool for legacy images. For example: - Keep existing images pool for legacy images - Create new images_meta pool for metadata of new images - Create images_data pool for data of new images
Migration path: Use Glance multistore to migrate existing images to a separate backend before enabling erasure-coded pools, or use external tools to migrate images between pools.
Accept mixing: Allow mixing in the same pool if the deployment can tolerate the operational complexity.

Migration of existing images to use the two-pool model is not part of this specification. If needed, this would be a future enhancement that could use Glance multistore capabilities or external migration tools.

Note

Performance Considerations:

Erasure-coded pools have significant performance trade-offs. The Ceph OSD daemons (storage nodes) need substantial CPU power to encode and decode data. This overhead is not on Glance itself, but on the Ceph cluster. Higher k+m values (e.g., 8+3 vs 4+2) increase CPU usage.

Writes are slower due to encoding work. Reads can be slower, especially if data reconstruction is needed when nodes are down. Rebuilding data after hardware failures is much more CPU-intensive and slower than with replicated pools.

For large image uploads, operators may need to increase timeout settings (particularly for image imports using the stage->import workflow) if write performance to EC pools is significantly slower than to replicated pools.

Best use cases are cold storage or archival images that are infrequently accessed, where storage cost savings are more important than performance.

This feature is disabled by default. Deployers should test performance with their specific Ceph hardware and erasure coding profile before enabling rbd_store_data_pool in production.

Alternatives

Implement a system that automatically selects pools based on image characteristics. This adds complexity and isn’t needed for most deployments.

Use external tools to move images between pools. This lacks integration with Glance and requires separate management tools.

Don’t add this feature. Deployments that need cost-efficient cold storage would have to accept the storage overhead or use external solutions.

The proposed solution is simple: just pass the data pool configuration through to librbd, which already supports this natively.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

When rbd_store_data_pool is configured, write and read operations will be slower than with replicated pools due to the CPU overhead of erasure coding on the Ceph cluster. The exact impact depends on the Ceph hardware and the erasure coding profile used (k+m values).

When rbd_store_data_pool is not configured (the default), there is no performance impact.

Other deployer impact

New option rbd_store_data_pool to specify the erasure-coded pool for image data. When not configured, behavior is unchanged.

Deployers need to create and configure erasure-coded pools on their Ceph cluster before enabling this feature. Specifically:

Create an erasure-coded pool for image data (e.g., images_data)
Create a replicated pool for image metadata (e.g., images)
Set allow_ec_overwrites true on the erasure-coded pool (required for librbd to function correctly with the two-pool model)

Ensure the Ceph cluster has sufficient CPU resources to handle erasure coding overhead.

Can be enabled without service interruption. Existing images continue to work from their current pools.

Note that when rbd_store_data_pool is enabled, the replicated pool will contain both legacy images (with all data) and new images (metadata only). While this mixing is supported, it is recommended to use separate pools for clean separation (see “Proposed change” section for details).

Warning

Do not enable erasure-coded pools in Glance if Nova or Cinder share the same RBD pool as Glance. Erasure-coded pools use a two-pool model where image metadata is stored in the replicated pool (rbd_store_pool) and image data is stored in the erasure-coded pool (rbd_store_data_pool). If Nova or Cinder are configured to use the same pool as Glance’s metadata pool, they will not be able to properly access or create resources because they do not support the two-pool model. This will cause failures when Nova tries to boot instances from images stored in erasure-coded pools, or when Cinder tries to create volumes from such images.

May want to monitor pool usage and Ceph cluster CPU utilization.

Test write/read performance and timeout behavior before enabling in production.

Developer impact

The RBD store driver needs to be modified to pass the data_pool parameter to librbd when configured.

Implementation

Assignee(s)

Primary assignee:: cyril-roelandt or abhishekk
Other contributors:: pranali-deore (Tempest testing) whoami-rajat (Cinder changes)

Work Items

Add the rbd_store_data_pool configuration option to glance_store
Modify the RBD driver to pass the data_pool parameter to librbd’s create() method when creating images
Update devstack-ceph-plugin to create erasure-coded pools for testing
Coordinate with Cinder and Nova teams - they may need similar changes to their RBD configuration to work with images stored in erasure-coded pools
Add tests for the two-pool scenario
Update documentation with configuration examples and performance guidance
Note: Migration tools are not part of this spec

Dependencies

Need a Ceph cluster with erasure-coded pools configured.

Cinder writes volumes and snapshots directly to Ceph. It may need similar data_pool support in its rbd_pool configuration to create volumes/snapshots in erasure-coded pools.

Nova may write instance snapshots directly to Ceph pools. It may need similar data_pool support in its libvirt.images_rbd_pool configuration.

The devstack-ceph-plugin needs updates to create erasure-coded pools for testing.

Testing

Test the two-pool configuration and error scenarios with unit tests.

Add tempest tests after devstack-ceph plugin supports creating erasure-coded pools.

The devstack-ceph-plugin needs to be updated to:

Create the erasure-coded pool (e.g., images_data) with the erasure pool type
Create the replicated pool (e.g., images) for metadata
Set allow_ec_overwrites true on the erasure-coded pool

Without the allow_ec_overwrites setting, image creation operations will fail during testing, as verified in testing with Ceph Tentacle release.

Documentation Impact

Document the rbd_store_data_pool option, performance considerations, and when to use erasure-coded pools.

Include how to set up Ceph erasure-coded pools and enable the feature in Glance.

References

2026.1 Project Priorities

Thu, 28 Aug 2025 00:00:00

TODO(glance-ptl): fill this in after the PTG

Make provision to set image size on upload

Wed, 16 Apr 2025 00:00:00

https://blueprints.launchpad.net/glance/+spec/set-size-on-upload

Historically, the python-glanceclient included the option to specify a --size parameter for the image-upload and image-stage commands. However, this feature is currently unused and is retained in the codebase solely for backward compatibility. Even if a user provides this option when executing the aforementioned commands, it is not forwarded to the Glance API.

Problem description

As the python-glanceclient uploads image data without requiring a prior specification of the image size, it functions as intended. However, this can lead to complications when Glance utilizes Cinder, RBD, or S3 as its backend. Since Glance does not have the data size beforehand, it conducts an iterative “resize-before-write” process, gradually expanding the Cinder volume by 1 GB at a time until all image data is received. This approach poses a challenge because the Cinder volume needs to be detached and reattached during each iteration, which can significantly slow down the process.

The same issue arises when using Ceph (RBD) as the backend, where a similar “resize-before-write” operation occurs, resulting in time-consuming operations.

If Glance is configured to use S3 as a backend, the multipart upload feature cannot be utilized because the image size is unknown in advance.

Proposed change

We propose to enable the use of the --size option for the image-upload and image-stage commands, as well as to add this option to the image-create command. The image size will be calculated automatically if a file-like object is detected during the upload or staging operations. Otherwise, users can specify the image size via the --size command-line option. If the --size option is provided, we will not calculate the image size internally, and the user-specified size will be sent directly to the Glance API. We will introduce a new request header x-openstack-image-size which will be passed to API.

On the API side, we will read the size passed from glanceclient or any request source via header x-openstack-image-size and pass it to the designated storage backend. This approach will help prevent resize-on-write operations for Cinder and RBD backends and enable the use of the multipart upload feature for the S3 backend. Each backend of glance_store will verify the size during the data pipeline whether the total data size exceeds the user-specified limit. If the size exceeds the allowed limit, the upload will be rejected, the data will be removed from the backend, and an appropriate error message will be returned to the user. Additionally, at the end of the process, we will compare the actual image size to the size provided by the user. If there is any mismatch in actual and expected size, the data will be deleted from the backend, the upload rejected, and a warning will be logged, consistent with existing checks for image disk format and hash.

If the image size is not included in the request header, we will have no option but to allow the storage backends to perform resize-on-write operations or to utilize the single-part upload feature for the S3 backend.

For asynchronous operations (import APIs), such as the glance-direct and web-download import methods, we already gather data in a local staging area before transferring it to the backend. We will ensure that the image size is set before starting the import operation to avoid these resize operations.

Alternatives

The Python Requests library also facilitates streaming uploads by sending data in chunks with a “Content-Length: <file size>” header, rather than using “Transfer-Encoding: chunked.” This approach avoids chunked transfer, which we should steer clear of, as it is a more stable method for long transfers. Since the development of the glanceclient library is currently at a standstill, we should avoid introducing changes that could lead to instability.

Data model impact

None

REST API impact

The PUT /v2/images/{image_id}/file and PUT /v2/images/{image_id}/stage APIs will be modified to recognize the x-openstack-image-size header.

HTTP 400 response will be returned if user provided size does not matches with actual image size.

Security impact

None

Notifications impact

None

Other end user impact

The end user can provide the image size in advance if they are aware of it.

Performance Impact

This will improve the upload/import process for cinder, ceph (rbd) and s3 backend of glance.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: abhishekk
Other contributors:: abhishekk

Work Items

Add command line option --size to image-create command
Calculate file size if file like object is provided
Make sure upload and stage commands passes size to API as request header
Make similar provision in openstack client and sdk
At Glance side read the size form request header and pass it to backend
Then compare provided size with actual size
Make sure image size is set before import operation starts
Required functional, unit and tempest tests
Document new behavior

Dependencies

https://bugs.launchpad.net/glance-store/+bug/2110185

Testing

Required unit, functional and tempest tests

Documentation Impact

Document the use of --size command line option and new request header x-openstack-image-size at glance side

References

2025.2 Project Priorities

Fri, 21 Mar 2025 00:00:00

TODO(glance-ptl): fill this in after the PTG

Add new import method to support downloading image from another glance/region

Tue, 14 Jan 2025 00:00:00

https://blueprints.launchpad.net/glance/+spec/glance-download-import

This spec describe a new import method called glance-download that implements a glance to glance download in a multi-region cloud with a federated Keystone.

Problem description

When dealing with a multi-region cloud it often appears that operators or customers need to copy images from a region to another, for example:

Copy all your public images between your regions (operator)
Copy instance snapshot in another region to have a backup (user)
Build your base application image from a factory in one region, then propagate it to multiple regions (user)

We can’t rely on the “copy-image” import method to copy an image from a backend to another because it requires the same glance endpoint in the same region which is not our use-case here.

The only way we have to do it now is to locally download the image data and upload it elsewhere, which requires some orchestration, and is a huge disk space and bandwidth loss.

Proposed change

Implement an internal plugin called glance-download based on the existing internal plugin web-download that will import an image stored on a remote glance. The web-download workflow will remain unchanged, the only difference is to retrieve the downloadable data from an other glance endpoint instead of from an arbitrary URL.

We should note several things:

To authenticate on the remote glance we propose to use the context token of the call, so it will require a federated Keystone environment between the two Glance.
The creation of the image must be handled by the end user as for the web-download plugin meaning that it is the responsibility of the user to take care of disk format, container format and metadata of the newly created image. If necessary, the plugin will update the container_format and disk_format to match what is set on source glance.
The plugin will come with an extra task in the taskflow that will be in charge of setting the container_format and disk_format to be the same as it is on the source glance. It will also copy some extra properties defined in the extra_properties_prefixes option of glance_download_properties section. The default extra_properties_prefixes values are ‘hw_’, ‘trait:’, ‘os_distro’, ‘os_secure_boot’ and ‘os_type’ which are needed to ensure an instance can boot on the image. An operator will be able to remove or add other extra properties by modifying this configuration variable. This extra_properties_prefixes is a list of prefixes, meaning that all the metadata that are starting with a prefix belonging to that list will be copied.

[glance_download_properties]
extra_properties_prefixes = [
    hw_,
    trait:,
    os_distro,
    os_secure_boot,
    os_type
  ]

If metadata injection is configured on the target glance it will override the metadata as the injection is run after the import.

Alternatives

We could imagine a take out alternative where the owner of the image in the source cloud generates a limited-use tokenized URL that allows access to the image without any keystone auth. Such solution is more risky as we do not have any authentication mechanism to access the remote image. It will also require rewriting the code as there is no existing source.

This would also require developing a mechanism to manage creation and expiration of the temporary urls which would result in a more complex solution that requires more time to develop, document and test.

Data model impact

None

REST API impact

Modification of existing API resource

Resource /v2/images/<image id>/import
Method: POST
Common response code:
- 201: import job queued
- 400: bad request with details
- 401: Unauthorized
- 403: Forbidden
JSON body definition

"method": {
    "name": {
        "description": "Name of the method used, here is glance-download",
        "type": "string"
    },
    "glance_image_id": {
        "description": "The image id to download on remote glance",
        "type": "string"
    },
    "glance_region": {
        "description": "The region name of remote glance",
        "type": "string"
    },
    "glance_service_interface": {
        "decription": "The interface of remote glance, default to 'public'",
        "type": "string"
    }
}

Example:

 "method": {
     "name": "glance-download",
     "glance_image_id": "02ea04ba-72b3-4687-810d-8ba10c991a97",
     "glance_region": "REGION1",
     "glance_service_interface": "admin"
}

Security impact

We use the token of the request to authenticate on remote glance. As we are in multi-region context with a federated keystone, there is no security impact.

Notifications impact

None

Other end user impact

Users will have a new import mechanism open to them, after updating their client

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: pslestang
Other contributors:: alistarle

Work Items

glance:
- Create a base download class that will be inherited by web-download and glance-download
- Patch the web-download class to inherit from base download class
- Write the glance-download class
- Patch the api image import to support the glance-download method
- Add in task flow a class in charge of:
  setting the correct container_format and disk_format
  
  copying the metadatas defined in extra_properties option of the glance_download_properties section. Default list must be [‘hw_’, ‘trait:’, ‘os_distro’, ‘os_secure_boot’, ‘os_type’]
- The class has to be added to taskflow as a normal task to be reusable if needed. We only have to check for input parameters to know if it can be run or not.
- Add the glance-download internal plugin in setup
- write unit/functional tests
- update documentation
- glance and openstack client
  add support for glance-download method
- update documentation

Dependencies

None

Testing

Unit and functional tests in Glance
Tempest tests. Testing glance-download plugin with the g-api-r separate endpoint looks good even if it shares the same database to validate the workflow.

Documentation Impact

The documentation needs to be updated to identify this new import method

References

https://review.opendev.org/c/openstack/glance/+/840318

Implement Calculate Hash API

Mon, 09 Dec 2024 00:00:00

https://blueprints.launchpad.net/glance/+spec/calculate-hash-api

Problem description

There will be 2 cases when image will be in active state without os_hash_value and checksum,

During the implementation of New Add Location API [1] it’s been noticed that some images will remain in ‘active’ state without checksum and os_hash_value if the hash calculation is failed after adding the location to the image.
Some legacy images which are created by consumers like instance snapshot or volume snapshot(image created by upload volume operation when glance backend is cinder) may also not have checksum, os_hash_value and os_hash_algo set.

Currently there is no other way to calculate hash for such images.

Proposed change

We are planning to add a separate new admin-only api to calculate the os_hash_value and checksum for the image which are in active state.

This API will be handled as follows:

Since hash calculation will be a long running operation, it will be executed in the background in the async task by setting the value of os_hash_algo before caclulation process starts. If hash calculation fails, retry mechanism for hash calculation will be added by using existing configuration option http_retries for maximum retries. If after all the retries, the hash calculation still fails we will not update the hash and checksum values and image will stay in active state and os_hash_algo will be reverted back to None.
Incase delete image has been attempted during hash calculation of that image, there are different responses from stores while reading the data,
- RBD throws ImageNotFound during data read but deletes the data from backend and image remains in active state even though delete call fails with InUseByStore error. There is a bug reported for this issue [1]. In this case as a workaround hash calculation will be marked as failed with proper log message and the image will be marked as deleted if it’s not marked as deleted by delete api.
- Filesystem backend throws NotFound since delete operation is successful.
- Swift allows image deletion during data reading or image-download.
- Cinder backend does not allow to delete the image since while reading data or downloading the image from volume, volume will be in-use state.
- Since this is async API call, admin can use API /v2/images/{id}/tasks to check the progress of the task.

We will introduce a new admin-only policy calculate_hash.

Alternatives

We can have a separate new command under glance-manage to run from cron.

Data model impact

None

REST API impact

New API

Calculate Hash

This spec proposes the following new endpoint:

POST /v2/images/{image_id}/hash
- JSON request body
```
{
    "os_hash_algo": "sha512"
}
```
- Response - Accepted - 202
  - Error - 409 (if image is not in ACTIVE state),
    403 (Forbidden for normal user) 400 (BadRequest if invalid os_hash_algo passed) 404 (Image ID does not exist)

Security impact

None

Notifications impact

None

Other end user impact

Changes in glanceclient and openstackclient will be required to expose for admin users only.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: pranali-deore
Other contributors:: None

Work Items

Implement the API with unit/functional tests
Document the API in api-ref
Write a tempest test to check these API
Implement support in OSC/SDK
Implement support in glanceclient
Add documentation for behaviour of new API

Dependencies

None

Testing

Unit and functional tests in Glance. Tempest tests against the same.

Documentation Impact

The documentation needs to be updated with the new API extension and usage.

References

API for cleaning and pruning cache

Tue, 05 Nov 2024 00:00:00

https://blueprints.launchpad.net/glance/+spec/clean-prune-cache-api

New API calls for cleaning and pruning image cache directories.

Problem description

As of today if image caching is enabled in the deployment, then deployer need to configure glance-cache-cleaner and glance-cache-pruner as cron jobs so that invalid image cache files gets cleaned periodically and if cache directory oversized then old cache files gets deleted automatically. This is an overhead for the deployer to make sure cron jobs are configured and running effectively.

Proposed change

We are propsing to deprecate glance-cache-cleaner, glance-cache-pruner and glance-cache-prefetcher command line tools in this cycle and remove them in G developemnt cycle. As we already have an API call /v2/cache/{image_id} to cache an image, similarly we will add two new API POST calls /v2/cache/clean and /v2/cache/prune which will do the job of pruning and cleaning for us. These two APIs will be admin only and non-admin user will be restricted to use it.

We will introduce two new policies cache_clean and cache_prune default to used by admin for restricting the use of these new APIs.

Alternatives

Instead of adding new API calls we can move existing code to reuse it as a periodic call under glance API service. This will need to introduce two additional configuration parameters to introduce interval of periodic calls.

Data model impact

None

REST API impact

This spec propose the following new APIs:

Clean invalid cached images

[New API] Clean invalid cached images

Clean invalid cached images:

POST /v2/cache/clean
{}

Response codes: * 200 – Upon authorization and successful request. * 403 – Permission denied

Prune image cache directory

[New API] Prune image cache directory

Prune image cache directory:

POST /v2/cache/prune
{}

* JSON response body

.. code-block:: json

    {
        "total_files_pruned": <total_files_pruned>,
        "total_bytes_pruned": <total_bytes_pruned>
    }

Response codes: * 200 – Upon authorization and successful request. * 403 – Permission denied

Security impact

As described in proposed change section new policies will be enforced to avoid security breach.

Notifications impact

None

Other end user impact

The glance client and openstack client should be updated, with new commands:

glance cache-clean
glance cache-prune
openstack cache clean
openstack cache-prune

Performance Impact

None

Other deployer impact

Deployer needs to stop using glance-cache-cleaner, glance-cache-pruner and glance-cache-prefetcher command line tools in the environment.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: abhishekk

Work Items

Introduce new API calls
Enforce new policy rules
Documentation
Testing

Dependencies

None

Testing

New tempest test to cover this scenario

Documentation Impact

The API documentation needs to be updated
Need to update Cache documentation as well with new commands

References

https://docs.openstack.org/glance/victoria/admin/cache.html

New API to list nodes where Image is cached

Wed, 30 Oct 2024 00:00:00

https://blueprints.launchpad.net/glance/+spec/list-nodes-image-cached

Deployer would like to see on which glance worker nodes an image is cached in one API call.

Problem description

As of today if you have configured multiple glance nodes and you need to see whether image is cached on particular node then you need to query to that particular glance node or query each glance node for the same.

Use case, In a DCN environment you want to see whether that particular edge node has image cached or not.

Proposed change

Glance now by default uses a centralized database for cache, which allows us to have access to all cached images across glance nodes. We are adding a new admin only API which will accept image_id as an input parameter and will return us the direct URL of all glance nodes where that particular image is cached.

We will also introduce a new policy list_cached_nodes so that users with appropriate permissions can perform this operation.

If caching is not enabled in deployment or deployment is not using centralized database for caching then this API will return HTTP 409 Conflict response to the user.

In the future we can use this API from consumers like nova or cinder in such a way that they can directly send the API request to particular glance node where the image is cached.

Alternatives

None

Data model impact

None

REST API impact

This spec propose the following new API:

New API

List nodes where image is cached

Common Response Codes

Accepted: 200 Accepted
Forbidden: 403 Forbidden
Conflict: 409 Conflict
Not Found: 404 Not Found

[New API] List nodes where image is cached

List nodes:

GET /v2/cache/nodes/{image_id}
{
    "cached_nodes": [
        http://node_1:60999,
        http://node_2:60999,
        http://node_5:60999,
     ],
}

Security impact

As described in proposed change section new policy will be enforced to avoid security breach.

Notifications impact

None

Other end user impact

The glance client and openstack client should be updated, with new commands:

glance cache-nodes-list <image_id>
openstack cache nodes list <image_id>

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: abhishekk

Work Items

Introduce new API call
Enforce new policy rule
Documentation
Testing

Dependencies

None

Testing

New tempest test to cover this scenario

Documentation Impact

The API documentation will need to be updated
Need to update Cache documentation as well with new commands

References

https://docs.openstack.org/glance/victoria/admin/cache.html

Sharing Images using the Image API v2

Wed, 09 Oct 2024 00:00:00

The OpenStack Image Service API v2 allows users to share images with each other in the following ways:

An image can be shared with specific other users of the cloud. This mode of sharing has been available since version 2.1 of the API. Thus when we speak about “shared” images in this document, we’re talking about this kind of sharing. It’s described in the section Sharing Images with Particular Users, below.
An image can be shared with all users of the cloud. This mode of sharing became available in version 2.5 of the API. Images shared in this way are referred to as “community” images because they’re available to the entire community of users in a cloud (and because we couldn’t think of a better name). Community images are discussed below in the section Sharing Images with All Users.

To keep the discussion to follow clear, here’s some terminology that will be used.

producer: A user who owns an image that’s going to be shared with other users.
consumer: A user who wants to use an image.
shared image: An image that’s been made accessible to specific other users, as described in Sharing Images with Particular Users. Since version 2.5 of the API, the visibility property of such an image is shared. (If you are using a prior version of the API, the visibility is private.)
community image: An image that’s been made accessible to all users in a cloud, as described in Sharing Images with All Users. Since version 2.5 of the API, the visibility property of such an image is community. (This type of image is not available prior to version 2.5.)

Sharing Images with All Users

Since version 2.5, the Image Service API v2 offers another kind of image sharing, Community Images. A community image is made available to all users in a cloud without the requirement of creating members on the image.

A community image is an image whose visibility value is community. The ability to communitize an image may be prohibited or restricted to specific users at the discretion of the cloud operator. To make an image a community image, use the image update call to change the image’s visibility appropriately. (See Update an image for more information.)

Community images do not appear in the default image list of any user other than the image owner. In order to discover community images, make the image-list call with a ‘visibility’ filter:

GET v2/images?visibility=community

As with the standard image-list call, other filters may be applied to the request. For example, to see the community images supplied by the image producer identified by 931efe8a-0ad7-4610-9116-c199f8807cda, the following call would be made:

GET v2/images?visibility=community&owner=931efe8a-0ad7-4610-9116-c199f8807cda

Note

See How do you identify a producer or consumer? for information about how to determine an image producer’s identifier.

Keep in mind that the name property of an image is not required to be unique, so filtering by name may result in multiple matches. For example, there may be multiple image producers promoting something named “Fred’s Excellent OS”, and thus the following call (remembering to URL encode the apostrophe and spaces):

GET v2/images?visibility=community&name=Fred%27s%20Excellent%20OS

may result in several image records for different image producers. If you want to find only images supplied by some particular producer, filtering by owner will give you more accurate results. Additionally, if “Fred’s Excellent OS” image is ever deleted by Fred, who you trust, some other producer George, who you don’t even know, could create a “Fred’s Excellent OS” image. If you are searching by name only, you’ll find George’s image—which you probably don’t want to use until you’ve figured out who George is, whether he’s trustworthy, and what exactly he’s put on the image.

No provision is made in this API for producer-consumer communication. All such communication must be done independently of the API.

An example workflow for community images is:

The producer posts information about community images on a public website. This information would be a description of the image including the image’s UUID.
A consumer uses the UUID in Images v2 API calls to access the image.

An alternative workflow is:

The producer posts information about community images on a public website. This post would include the producer’s identifier.
A potential consumer uses the producer’s identifier in an image-list call as in the example above to discover the available images and to determine the UUID of an image to be consumed.
The consumer uses the UUID in Images v2 API calls to access the image.

Standardize Image Encryption and Decryption

Tue, 03 Sep 2024 00:00:00

OpenStack already has the ability to create encrypted volumes and ephemeral storage to ensure the confidentiality of block data. Even though it is also already possible to store encrypted images, there is only one service (Cinder) that utilizes this option, but it is only indirectly usable by Nova (a user must create a volume from the image first), and thus users don’t have an intuitive way to create and upload encrypted images. In addition, all metadata needed to detect and use encrypted images is either not present or specifically scoped for Cinder right now. In conclusion, support for encrypted images does exist to some extent but only in a non-explicit and non-standardized way. To establish a consistent approach to image encryption for all OpenStack services as well as users, several adjustments need to be implemented in Glance, Cinder and OSC.

Problem description

An image, when uploaded to Glance or being created through Nova from an existing server (a VM snapshot), may contain sensitive information. The already provided signature functionality only protects images against alteration. Images may be stored on several hosts over long periods of time. First and foremost this includes the image storage hosts of Glance itself. Furthermore it might also involve caches on systems like compute hosts. In conclusion they are exposed to a multitude of potential scenarios involving different hosts with different access patterns and attack surfaces. The OpenStack components involved in those scenarios do not protect the confidentiality of image data.

As stated in the introduction above, some disk image encryption implementations for ephemeral disks in Nova and volumes in Cinder already touch on this topic but not always in a standardized and interoperable way. For example, the way of handling image metadata and encryption keys can differ. Furthermore, users are not easily able to make use of these implementations when supplying their own images in a way that encryption can work the same across services.

That’s why we propose the introduction of a streamlined encrypted image format along with well-defined metadata specifications which will be supported across OpenStack services for the existing encryption implementations and increase interoperability as well as usability for users.

Use Cases

A user wants to upload an image, which includes sensitive information. To ensure the integrity of the image, a signature can be generated and used for verification. Additionally, the user wants to protect the confidentiality of the image data through encryption. The user generates or uploads a key in the key manager (e.g. Barbican) and uses it to encrypt the image locally before uploading it. A mechanism to let the OpenStack Client (OSC) do the encryption could be added in a later version. Consequently, the image stored on the Glance host is encrypted.
A user wants to create a new server or volume based on a) an encrypted image created externally or b) an image created as a backup from already encrypted storage objects in components like Nova and Cinder. The corresponding compute or volume host has to be able to directly use the encrypted image or (if incompatible) transfer its encryption from e.g. qcow2-LUKS to raw LUKS-encrypted blocks to be used for volumes. For this the OpenStack services need access to the key in the key manager and a few image properties about the encrypted image.
A user wants to download and directly decrypt an encrypted image to be used privately or in another deployment. If possible, the download mechanism could be adjusted on client side to directly decrypt such an image.

Proposed change

There are two ways encryption in images can be handled. The first is having a qcow2 formatted disk with an internal encryption. The ‘disk_format’ for such an image will consequently be ‘qcow2’ and the encryption can be detected through the presence of the proposed additional metadata. For other images the ‘disk_format’ will be used to indicate the encryption. It should state the main encryption mechanism used, which for now is ‘luks’. We additionally assume that the format of a decrypted LUKS-image will always be ‘raw’. The format can only be checked after a decryption of at least the first few bytes this is currently out of scope for Glance. Every service, that uses such encrypted LUKS-images needs to be aware of it.

Furthermore, we propose the following additional metadata properties carried by images of this format:

‘os_encrypt_format’ - the specific mechanism used, e.g. ‘LUKSv1’
‘os_encrypt_key_id’ - reference to key in the key manager
‘os_encrypt_key_deletion_policy’ - on image deletion indicates whether the key should be deleted too
‘os_decrypt_container_format’ - format change, e.g. from ‘compressed’ to ‘bare’
‘os_decrypt_size’ - size after payload decryption

To upload an encrypted image to Glance we want to extend the OpenStack Client to allow the specification of the necessary metadata properties as the key ID and the encryption and optionally metadata properties as for example the specification of the key deletion policy. Later on there might be support added for encrypting images using the specified key ID directly in the OpenStack Client.

In other words: the user has to encrypt an image before the upload. While uploading the encrypted image to Glance, the metadata properties above have to be specified.

We propose to align the encryption with Nova and Cinder and use LUKS, which will be allowed in combination with qcow and raw images. We use this two versions for the following reasons:

Nova can directly use qcow-LUKS encrypted when creating a server. This is the standard procedure of Nova. Nova can also handle LUKS-encrypted raw images.
Cinder allows the creation of Images from encrypted volumes. These will always result in LUKS-encrypted raw images. Those images can be converted directly to volumes again. Cinder currently expects encrypted images to be raw images.

In the latter case it is already possible to upload such an encrypted image to another OpenStack infrastructure, upload the key as well and set the corresponding metadata. After doing so the image can be used in the second infrastructure to create an encrypted volume.

We want to align the existing implementations between Nova and Cinder by standardizing the used metadata parameters and adding interoperability where applicable. Furthermore, we want to provide users with the means to encrypt images outside of the infrastructure for upload in Glance which will later be handled in similar ways by both Cinder and Nova.

The key management is handled differently than with encrypted volumes or encrypted ephemeral storage. The reason for this is, that the encryption and decryption of an image should never happen in Glance but only on client side. Therefore the service which needs to create a key for a newly created encrypted image may not be the same service which then has to delete the key (in most cases Glance). To delete a key, which has not been created by the same entity, is bad behavior. To avoid this, we choose to do the following:

if a user uploads an image the user is responsible for creation and deletion of the key.
if Cinder or Nova are uploading an image, they are responsible for creating a key (e.g. as it is handled in Cinder currently).

Optionally the deletion of the secret can be delegated to Glance through setting the special metadata parameter “os_encrypt_key_deletion_policy” to true. This behavior is already implemented for encrypted images from Cinder, we will only rename the property so it is not solely be usable by Cinder.

To not accidentally delete a key, which is used to encrypt an image, we will let Glance register as a consumer of that key (secret in Barbican [1]) when the corresponding encrypted image is uploaded and unregister as a consumer when the image is deleted in Glance. When the parameter “os_encrypt_key_deletion_policy” is set to “True”, we will try to delete the key. If that fails, because there was still a consumer, we let Glance log that as a warning and proceed with the image deletion process. In this case the key might still be used for another image or some other ressource and we do not want to delete it, we rather assume that the “os_encrypt_key_deletion_policy” was mistakenly set to “True”.

Image conversion will not be encryption-aware as part of this spec and as such, conversion of encrypted images will not be supported. The vmdk format is not supported by this spec and the conversion itself would need decryption and encryption to be handled by Glance. This would be more than the scope of this spec will be. So if image conversion is enabled and an encrypted images that needs conversion is uploaded the API will return a 400 Error and the image will be put in the queued state as a result.

Alternatives

We could introduce individual container types in Glance for each combination of data format and cipher algorithm instead of a single container type with metadata. This decision affects the implementation in nova and cinder. Regarding the image encryption, we also explored the possibility of using more elaborated and dynamic approaches like PKCS#7 (CMS) but ultimately failed to find a free open-source implementation (e.g. OpenSSL) that supports streamable decryption of CMS-wrapped encrypted data. More precisely, no implementation we tested was able to decrypt a symmetrically encrypted, CMS-wrapped container without trying to completely load it into memory or suffering from other limitations regarding big files.

We also evaluated an image encryption implementation based on GPG. The downside with such an implementation is, that everytime such an image is used to create a server or a volume the image has to be decrypted and maybe re-encrypted for another encryption format as both Nova and Cinder use LUKS as an encryption mechanism. This would not only have impact on the performance of the operation but it also would need free space for the encrypted image file, the decrypted parts and the encrypted volume or server that is created.

We evaluated to use a single container format for all encrypted images, but as Cinder already stores Images within different containers (e.g. ‘compressed’) we decided to use the usual container format and check for the presence of encryption parameters instead to detect an encrypted image.

Data model impact

The impact depends on whether the implementation will make actual changes to the image data model or simply use the generic properties field in the metadata. In the latter case the encryption properties would be added to metadefs.

REST API impact

While uploading an image, which should be encrypted, additional properties in the request body will need to be introduced to specify the desired encryption format and key id. Both to be used while encrypting the image locally before uploading it.

Example request: ` REQ: curl -g -i -X POST http://a.b.c.d/image/v2/images -H "Content-Type: application/json" .... -d ' {"disk_format": "LUKS", "name": "cirros", "container_format": "compressed", "os_encrypt_format": "LUKSv1", "os_encrypt_key_id": "...", "os_encrypt_key_deletion_policy": "True", "os_decrypt_format": "raw", "os_decrypt_container_format": "bare", "os_decrypt_size": "...", ...}' `

Additionally the GET image API call will display all set properties.

Security impact

There are impacts on the security of OpenStack:

confidentiality of data in images will be addressed in this spec
image encryption is introduced formally, thus cryptographic algorithms will be used in all involved components (Nova, Cinder, OSC)
Glance may lose the ability to provide a first-layer defense against image policy violations (such as rejecting invalid/disallowed formats), because inspection of encrypted data is not possible.

Notifications impact

None

Other end user impact

Users should be able to optionally, but knowingly upload an encrypted image.
If an administrator has configured Glance to reject unencrypted images, such images will not be accepted when attempted to be uploaded to Glance.

Performance Impact

The proposed encryption/decryption mechanisms in the OpenStack components will only be utilized on the client side and skipped entirely for images that aren’t encrypted.

When creating a volume or server from an encrypted image the only operation that may be triggered is the conversion between qcow-LUKS and raw LUKS blocks.

Thus, any performance impact is only applicable to the newly introduced encrypted image type where the processing of the image will have increased computational costs and longer processing times than regular images. Impact will vary depending on the individual host performance and supported CPU extensions for cipher algorithms.

Other deployer impact

A key manager - like Barbican - is required, if encrypted images are to be used.

Developer impact

None

Upgrade impact

We can assume, that all images that are encrypted and already present in an OpenStack deployment were created from encrypted Cinder volumes. They need to be adjusted in the following way:

all images that have ‘cinder_encryption_key_id’ set, need to convert it to ‘os_encrypt_key_id’
all images that have ‘cinder_encryption_key_deletion_policy’ set, need to convert it to ‘os_encrypt_key_deletion_policy’

Implementation

Assignee(s)

Primary assignee: Markus Hentsch (IRC: mhen)

Other contributors: Josephine Seifert (IRC: Luzi)

Work Items

Add standardized parameters with encryption support to Glance
Add registering as consumer for a Barbican secret when uploading an encrypted image
Add unregistering as consumer for a Barbican secret when deleting an encrypted image
Add support for providing the new image properties to the python-openstackclient and openstacksdk, so that an encrypted image can be uploaded
Change the usages of ‘cinder_encryption_key_deletion_policy’ and ‘cinder_encryption_key_id’ throughout the Glance codebase to the new parameters
Add unit test and functional test for uploading encrypted images
Add a migration script for the transformation of legacy properties of the volume based encrypted images
Adjust the documentation to show the new and changed parameters
Add the image encryption as documentation in the security guide

Dependencies

The secret consumer API in Barbican is required for Glance to be able to register and unregister as a consumer of a secret

Testing

Tempest tests would require access to encrypted images for testing. This means that Tempest either needs to be provided with an image file that is already encrypted and its corresponding key or needs to be able to encrypt images itself. This point is still open for discussion.

Documentation Impact

It should be documented for deployers, how to enable this feature in the OpenStack configuration. An end user should have documentation on how to create and use encrypted images.

References

[1] Barbican Secret Consumer Spec: https://review.opendev.org/#/c/662013/

History

Revisions
Release Name	Description
Dalmatian	Introduced

2025.1 Project Priorities

Tue, 30 Jul 2024 00:00:00

TODO(glance-ptl): fill this in after the PTG

User-configurable hash algorithms

Fri, 14 Jun 2024 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/glance/+spec/configurable-hash-algorithms

Allow end-users to specify a hash algorithm to be used when creating a new image, allowing them to compare existing hashes provided by the creator of the image with hashes returned by Glance, rather than having to generate the hashes themselves on the client side.

Problem description

As described in the docs, the Secure Hash Algorithm feature adds image properties that may be used to verify image integrity based on its hash. Two properties are added, os_hash_algo which contains the name of the hash algorithm used to generate the hash, and os_hash_value which contains the hash itself.

We would like to be able to use web-download to upload an image to Glance - be that an OpenShift release images or a stock CentOS Stream or Ubuntu image - and then verify the hash of the uploaded image against a signature given by the image provider. However, currently the hash algorithm represented by os_hash_algo is not user-configurable: it can only be configured by the [DEFAULT] hashing_algorithm configuration option. This defaults to sha512, which was chosen as it was more performant than SHA-256 (see Secure Hash Algorithm Support), but SHA-512 signatures are not published by all image providers: for example, while Debian publishes SHA-512 signatures (source), Ubuntu only publishes SHA-256 signatures (source) as does Fedora (source). When signatures are not provided in a suitable format, it is necessary to either ask an admin to change the hash algorithm (which affects the entire deployment and might not be possible, particularly in public cloud environments) or generate the hash on the client-side before uploading the image (which limits the usefulness of the web-download image import mechanism). We would like to address this gap.

Proposed change

To resolve this, we propose allowing users to select the hash algorithm to be used when creating the image. We will rename the [DEFAULT] hashing_algorithm config option to [DEFAULT] default_hashing_algorithm (providing an alias for the older name) and add a new config option, [DEFAULT] allowed_hashing_algorithms, which will be a list of algorithms that are permitted to specified by the user and will initially default to ['sha512', 'sha256', 'sha1', 'md5']. In this way, users can select a hashing algorithm that suits their use case while operators can still restrict certain algorithms to e.g. maintain regulatory compliance.

Alternatives

Instead of allowing users to specify the hash algorithm to be used, we could start storing multiple hash algorithms. This would require replacing the two image properties, os_hash_algo and os_hash_value with either a new map-like property or flat algorithm-specific properties (e.g. os_hash_value.sha512).

This was rejected because it has a far larger API impact, would require database changes, and would increase CPU utilisation due to the need to generate multiple hashes for every image.
Instead on introducing a new configuration option, we could modify the behavior of the existing [DEFAULT] hashing_algorithm option such that it now accepts a list of allowed hashing algorithms, with the first item (index 0) being the default used.

This was rejected because it overloads the option to have two purposes - both configuring a default and configuring allowed options - which could be confusing for operators. In addition, any deployments that have non-default values (e.g. sha256) will have those values persisted and the value will now affect both default and allowed hashing algorithms, which may be undesirable from an operator perspective yet easy to miss.

Data model impact

None.

REST API impact

The POST /images API will now allow users to specify the os_hash_algo property during image creation. If a user specifies an unsupported algorithm, the request will be rejected with a HTTP 400 (Bad Request) error and appropriate error message.

Security impact

There are a number of potential issues with these but we believe none of them are actual issues.

A malicious user could rely on a hash collision with a (significantly) weaker or insecure algorithm to trick users into believing they are downloading e.g. an official Ubuntu image when in fact they are downloading a weaker image.

Using this would require that the malicious user has the ability to create public or community images, or they would require the potential victim to accept a share request for a shared image. This is unlikely in a public cloud environment. In addition, and perhaps more importantly, a hash collision attack using SHA-256 or SHA-512 has not been publicly demonstrated. This is obviously not the case for SHA1 and MD5 but as both algorithms’ lack of security is well documented and well known, there should be no expectation of security from end-users.
A malicious user could conduct a denial-of-service attack on Glance by uploading images using an expensive hashing algorithm.

The set of hashing algorithms that a user can specify will be operator-configurable, meaning only well-known, well-understood algorithms will be permitted by default. In addition, such a hashing algorithm would have to be astoundingly expensive to make a dent in the existing overhead costs of downloading and storing a glance image. We are not aware of any such hash algorithms in practical use.
Use of a particular algorithm could cause the operator to run afoul of regulatory requirements.

The supported hashing algorithms will be operator-configurable. The operator can merely disable those that are not permitted due to e.g. FIPS compliance requirements.

Notifications impact

None.

Other end user impact

This field is already supported by openstacksdk but is currently silently ignored. This will no longer be the case. Put another way, this will now work:

import openstack

conn = openstack.connect()
openstack.enable_logging(debug=True)

image = conn.image.create_image(
    'foobar',
    os_hash_algo='sha256',
)
print(image)

Or, using OpenStackClient (OSC):

openstack image create --property os_hash_algo=sha256 foobar

We may wish to add a new helper alias to the image create command in OpenStackClient, to allow users to specify this well-known alias easily but this is a nice-to-have.

Performance Impact

None.

Other deployer impact

Deployers will now have the ability to configure the hashing algorithms that users can use when creating image. While this will default to a sensible set of default algorithms, they may wish to tweak this further to meet regulatory or organisational requirements.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: stephen.finucane
Other contributors:: None

Work Items

Add the new configuration option
Add necessary API logic to allow the user to specify this option during image creation and respect it during image upload.
Update documentation

Dependencies

None.

Testing

Unit test and manual testing should suffice here, though we can also test this via a new Tempest test.

Documentation Impact

We will need to update the API documentation along with the Secure Hash Algorithm feature documentation.

References

None.

Improve filesystem store driver to utilize NFS capabilities

Mon, 29 Apr 2024 00:00:00

https://blueprints.launchpad.net/glance/+spec/improve-filesystem-driver

Problem description

The filesystem backend of glance can be used to mount NFS share as local filesystem, so it is not required to store any special configs at glance side. Glance does not care about NFS server address or NFS share path at all, it just assumes that each image is stored in the local filesystem. The downside of this assumption is that glance is not aware whether NFS server is connected/available or not, NFS share is mounted or not and just keeps performing add/delete operations on local filesystem directory which later might causes problem in synchronization when NFS is back online.

Use case: In a k8s environment where OpenStack Glance is installed on top of OpenShift and NFS share is mounted via the Volume/VolumeMount interface, the Glance pod won’t start if NFS share isn’t ready. Whereas if NFS share is not available after Glance pod is available then upload operation will fail with following error:

sh-5.1$ openstack image create --container-format bare --disk-format raw --file /tmp/cirros-0.5.2-x86_64-disk.img cirros
ConflictException: 409: Client Error for url: https://glance-default-public-openstack.apps-crc.testing/v2/images/0ce1f894-5af7-44fa-987d-f4c47c77d0cf/file, Conflict

Even though the Glance Pod is still up, liveness and readiness probes starts failing and as a result the Glance Pods are marked as Unhealthy:

Normal   Started         12m                    kubelet            Started container glance-api
  Warning  Unhealthy       5m24s (x2 over 9m24s)  kubelet            Liveness probe failed: Get "https://10.217.0.247:9292/healthcheck": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy       5m24s (x3 over 9m24s)  kubelet            Liveness probe failed: Get "https://10.217.0.247:9292/healthcheck": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy       5m24s                  kubelet            Readiness probe failed: Get "https://10.217.0.247:9292/healthcheck": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy       4m54s (x2 over 9m24s)  kubelet            Readiness probe failed: Get "https://10.217.0.247:9292/healthcheck": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy       4m54s                  kubelet            Readiness probe failed: Get "https://10.217.0.247:9292/healthcheck": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Later in time, according to the failure threshold set for the Pod, the kubelet marks the Pod as Failed, and we can see a failure, and given that the policy is supposed to recreate it:

glance-default-single-0                                         0/3     CreateContainerError   4 (3m39s ago)   28m

$ oc describe pod glance-default-single-0 | tail
Normal   Started    29m                    kubelet   Started container glance-api
Warning  Unhealthy  10m (x3 over 26m)      kubelet   Readiness probe failed: Get "https://10.217.0.247:9292/healthcheck": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Warning  Unhealthy  10m                    kubelet   Liveness probe failed: Get "https://10.217.0.247:9292/healthcheck": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Warning  Unhealthy  10m                    kubelet   Readiness probe failed: Get "https://10.217.0.247:9292/healthcheck": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning  Unhealthy  9m30s (x4 over 26m)    kubelet   Liveness probe failed: Get "https://10.217.0.247:9292/healthcheck": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Warning  Unhealthy  9m30s (x5 over 26m)    kubelet   Liveness probe failed: Get "https://10.217.0.247:9292/healthcheck": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Warning  Unhealthy  9m30s (x2 over 22m)    kubelet   Readiness probe failed: Get "https://10.217.0.247:9292/healthcheck": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Warning  Unhealthy  9m30s (x3 over 22m)    kubelet   Readiness probe failed: Get "https://10.217.0.247:9292/healthcheck": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Warning  Unhealthy  9m30s                  kubelet   Liveness probe failed: Get "https://10.217.0.247:9292/healthcheck": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning  Failed     4m47s (x2 over 6m48s)  kubelet   Error: context deadline exceeded

Unlike other deployments (deployment != k8s) where even if NFS share is not available the glance service keeps running and uploads or deletes the data from local filesystem. In this case we can definitely say that NFS share is not available, the Glance won’t be able to upload any image in the filesystem local to the container and the Pod will be marked as failed and it fails to be recreated.

Proposed change

We are planning to add new plugin enable_by_files to healthcheck wsgi middleware in oslo.middleware which can be used by all openstack components to check if desired path is not present then report 503 <REASON> error or 200 OK if everything is OK.

In glance we can configure this healthcheck middleware as an application in glance-api-paste.ini as an application:

[app:healthcheck]
paste.app_factory = oslo_middleware:Healthcheck.app_factory
backends = enable_by_files (optional, default: empty)
# used by the 'enable_by_files' backend
enable_by_file_paths = /var/lib/glance/images/filename,/var/lib/glance/cache/filename (optional, default: empty)

# Use this composite for keystone auth with caching and cache management
[composite:glance-api-keystone+cachemanagement]
paste.composite_factory = glance.api:root_app_factory
/: api-keystone+cachemanagement
/healthcheck: healthcheck

The middleware will return “200 OK” if everything is OK, or “503 <REASON>” if not with the reason of why this API should not be used.

“backends” will the name of a stevedore extentions in the namespace “oslo.middleware.healthcheck”.

In glance, if local filesystem path is mounted on NFS share then we propose to add one marker file named .glance to NFS share and then use that file path to configure enable_by_files healthcheck middleware plugin as shown below:

[app:healthcheck]
paste.app_factory = oslo_middleware:Healthcheck.app_factory
backends = enable_by_files
enable_by_file_paths = /var/lib/glance/images/.glance

If NFS goes down or somehow the /healthcheck starts reporting 503 <REASON> admin can take appropriate actions to make NFS share available again.

Alternatives

Introduce few configuration options for filesystem driver which will help to detect if the NFS share is unmounted from underneath the Glance service. We proposed to introduce below new configuration options for the same:

filesystem_is_nfs_configured - boolean, verify if NFS is configured or not
filesystem_nfs_host - IP address of NFS server
filesystem_nfs_share_path - Mount path of NFS mapped with local filesystem
filesystem_nfs_mount_options - Mount options to be passed to NFS client
rootwrap_config - To run commands as root user

If filesystem_is_nfs_configured is set, i.e. if NFS is configured then deployer must specify filesystem_nfs_host and filesystem_nfs_share_path config options in glance-api.conf otherwise the respective glance store will be disabled and will not be used for any operation.

We are planning to use existing os-brick library (already used by cinder driver of glance_store) to create the NFS client with the help of above configuration options and check if NFS share is available or not during service initialization as well as before each image upload/import/delete operation. If NFS share is not available during service initialization then add and delete operations will be disabled but if NFS goes down afterwards we will raise HTTP 410 (HTTP GONE) response to the user.

Glance still doesn’t have capability to check whether particular NFS store has storage capability to store any particular image beforehand. Also it does not have capability to verify if network failure occurs during upload/import operation.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Need to configure healthcheck middleware for glance.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: abhishekk
Other contributors:: None

Work Items

Add enable_by_files healthcheck backend in oslo.middleware
Document how to configure enable_by_files healthcheck middleware
Unit/Functional tests for coverage

Dependencies

None

Testing

Unit Tests
Functional Tests
Tempest Tests

Documentation Impact

Need to document new behavior of filesystem driver if NFS and healthcheck middleware is configured.

References

Oslo.Middleware Implementation - https://review.opendev.org/920055

2024.2 Project Priorities

Thu, 18 Apr 2024 00:00:00

The Dalmatian Cycle priorities were discussed during Dalmatian Virtual PTG. The preliminary list was maintained on the Dalmatian PTG etherpad.

The following indicates milestonewise priorities.

Priority Item	Owner(s)	Spec(s)	Target release milestone
Deprecate metadata_encryption_key	Pranali Deore	deprecate metadata encryption key	M1
New Location APIs	Pranali Deore	new location apis	M2
New sub-store NFS for FS store	Abhishek Kekane	improve_filesystem_driver	M2
Add CLI utility for migrating images	Abhishek Kekane	None	M3
Removed all unused config options	Cyril Roelandt	None	M3
All image/metadef APIs OSC & SDK Support	Mridula Joshi	None	M3
Image Encryption	Josephine Seifert	image encryption	M3

Important dates

This is an abbreviated list focused on dates relevant to Glance. See Dalmatian Release Schedule for the complete list for OpenStack.

Milestone	Week of	What
R-20	May 17	Dalmatian-1 Milestone
R-13	Jul 05	Dalmatian-2 Milestone Glance Spec freeze
R-6	Aug 23	glance_store Dalmatian release (final release for non-client libraries)
R-5	Aug 30	Dalmatian-3 milestone Glance feature freeze python-glanceclient dalmatian release (final release for client libraries);
R-3	Sept 13	RC-1 release and string freeze
R-1	Sept 27	final RCs
R-0	Oct 04	Dalmatian release

Spec Lite: Deprecate metadata_encryption_key

Thu, 18 Apr 2024 00:00:00

project:: glance
problem:: metadata_encryption_key config option and it’s related functionality was added quite a long back ago. Even though as per the description it encrypts the location metadata, it actually encrypts location url. It encrypts the location url only for image upload/import/download/show APIs, doesn’t encrypt url on location APIs. If it’s enabled during upgrade then it will break the existing deployment since existing image url is not been encrypted. It doesn’t even work for location url encryption as expected since it does not encrypts the legacy images url on start up, download of that image fails with InvalidLocation error.
solution:: We decided to deprecate metadata_encryption_key config option in this cycle and remove it in F (2025.2) cycle.
impacts:: None
how:: In this cycle deprecate metadata_encryption_key configuration options. Remove it along with it’s related functionality in F cycle.
alternatives:: None
timeline:: Milestone 2
link:: None
reviewers:: abhishekk, croelandt, mrjoshi
assignee:: pdeore

2024.1 Project Priorities

Wed, 01 Nov 2023 00:00:00

The Caracal Cycle priorities were discussed during Caracal Virtual PTG. The preliminary list was maintained on the Caracal PTG etherpad.

This list is an estimate of what the Glance project team can accomplish during the 2024.1 Cycle based on our developer’s estimates of how much time they can commit to Glance.

The following indicates milestonewise priorities.

Priority Item	Owner(s)	Spec(s)	Target release milestone
New Location APIs	Pranali Deore	new location apis	M2
Centralize DB for cache	Abhishek Kekane	centralize DB for cache	M2
Deprecate Location Strategy	Abhishek Kekane	deprecate location strategy	M2
Deprecate Cachemanage middleware	Abhishek Kekane	deprecate cachemanage middleware	M2
Remove/Migrate Single Store Tests	Pranali Deore	None	M3
All image/metadef APIs OSC & SDK Support	Mridula Joshi	None	M3
Image Encryption	Josephine Seifert	image encryption	M3

Important dates

This is an abbreviated list focused on dates relevant to Glance. See Caracal Release Schedule for the complete list for OpenStack.

Milestone	Week of	What
R-20	Nov 17	Caracal-1 Milestone
R-12	Jan 12	Caracal-2 Milestone Glance Spec freeze
R-6	Feb 23	glance_store Caracal release (final release for non-client libraries)
R-5	Mar 01	Caracal-3 milestone Glance feature freeze python-glanceclient caracal release (final release for client libraries);
R-3	Mar 15	RC-1 release and string freeze
R-1	Mar 29	final RCs
R-0	Apr 05	Caracal release

Spec Lite: Deprecate cachemanage middleware

Wed, 01 Nov 2023 00:00:00

project:: glance
problem:: Deprecate cachemanage middleware
solution:: Cache operations (CR(u)D) is now part of glance API. Even though we need to depend on cache middleware to enable it. There are two middlewares related to caching, one is cache and other is cachemanage and we can now get rid of cachemanage middleware and glance-cache-manage command line utility.
impacts:: None
how:: In this cycle we are going to deprecate cachemanage middleware and api pipeline keystone+cachemanage (defined in glance-api-paste.ini) and glance-cache-manage command line utility. A deprecation warning message will be added to each command of glance-cache-manage utility as well as during initialization of cachemanage middleware.
alternatives:: None
timeline:: Milestone 2
link:: None
reviewers:: pdeore, mrjoshi, croelandt
assignee:: abhishekk

Spec Lite: Deprecate location strategy

Wed, 01 Nov 2023 00:00:00

project:: glance
problem:: Deprecate location strategy
solution:: In Bobcat we added support for weighing mechanism to glance store. To use this new feature we decided to deprecate location strategy in this cycle and remove it in ‘D’ (2024.2) development cycle.
impacts:: None
how:: In this cycle we are going to deprecate following configuration options related to location strategy: * location_strategy * store_type_preference Also a warning message regarding deprecation will be added to location_order and store_type strategy module during initiation phase.
alternatives:: None
timeline:: Milestone 2
link:: None
reviewers:: pdeore, mrjoshi, croelandt
assignee:: abhishekk

Image Encryption and Decryption

Wed, 01 Nov 2023 00:00:00

OpenStack already has the ability to create encrypted volumes and ephemeral storage to ensure the confidentiality of block data. In contrast to that, images are currently handled without protection towards confidentiality, only providing the possibility to ensure integrity using image signatures. For further protection of user data - e.g. when a user uploads an image containing private data or confidential information - the image data should not be accessible for unauthorized entities. For this purpose, an encrypted image format is to be introduced in OpenStack. In conclusion, several adjustments to support image encryption/decryption in various projects, e.g. Nova, Glance, Cinder and OSC, need to be implemented.

Problem description

An image, when uploaded to Glance or being created through Nova from an existing server (VM), may contain sensitive information. The already provided signature functionality only protects images against alteration. Images may be stored on several hosts over long periods of time. First and foremost this includes the image storage hosts of Glance itself. Furthermore it might also involve caches on systems like compute hosts. In conclusion they are exposed to a multitude of potential scenarios involving different hosts with different access patterns and attack surfaces. The OpenStack components involved in those scenarios do not protect the confidentiality of image data.

Using encrypted storage backends for volume and compute hosts in conjunction with direct data transfer from/to encrypted images can enable workflows that never expose an image’s data on a host’s filesystem. Storage of encryption keys on a dedicated key manager host ensures isolation and access control for the keys as well. With such a set of configuration recommendations for security focused environments, we are able to reach a good foundation. Future enhancements can build upon that and extend the provided security enhancements to a broader set of variants.

That’s why we propose the introduction of an encrypted image format.

Use Cases

A user wants to upload an image, which includes sensitive information. To ensure the integrity of the image, a signature can be generated and used for verification. Additionally, the user wants to protect the confidentiality of the image data through encryption. The user generates or uploads a key in the key manager (e.g. Barbican) and uses it to encrypt the image locally using the OpenStack client (osc) when uploading it. Consequently, the image stored on the Glance host is encrypted.
A user wants to create an image from an existing server with ephemeral storage. This server may contain sensitive user data. The corresponding compute host then generates the image based on the data of the ephemeral storage disk. To protect the confidentiality of the data within the image, the user wants Nova to also encrypt the image using a key from the key manager, specified by its ID. Consequently, the image stored on the Glance host is encrypted.
A user wants to create an image from an existing volume. This volume may contain sensitive user data. The corresponding volume host then generates the image based on the data of the volume. To protect the confidentiality of the data within the image, the user wants Cinder to also encrypt the image using a key from the key manager, specified by its ID. Consequently, the image stored on the Glance host is encrypted.
A user wants to create a new server or volume based on an encrypted image created by any of the use cases described above. The corresponding compute or volume host has to be able to decrypt the image using the symmetric key stored in the key manager and transform it into the requested resource (server disk or volume). For this, the user needs access to the key in the key manager which is controlled via their project role assignment.

Proposed change

For Glance we propose to add a new container_format called ‘encrypted’. Furthermore, we propose the following additional metadata properties carried by images of this format:

‘os_glance_encrypt_format’ - the main mechanism used, e.g. ‘GPG’
‘os_glance_encrypt_type’ - encryption type, e.g. ‘symmetric’
‘os_glance_encrypt_cipher’ - the cipher algorithm, e.g. ‘AES256’
‘os_glance_encrypt_key_id’ - reference to key in the key manager
‘os_glance_decrypt_container_format’ - format after payload decryption
‘os_glance_decrypt_size’ - size after payload decryption

To upload an encrypted image to Glance we want to add support for encrypting images using a key ID which references the symmetric key in the key manager (e.g. Barbican) in the OpenStack Client. This also involves new CLI arguments to specify the key ID and encryption method and this implementation should make use of a centralized encryption implementation provided by a global library, shared between all involved OpenStack components to eliminate the need of individual implementations of the encryption mechanism.

In other words: the user or openstack service as cinder for example has to encrypt an image before the upload. While uploading the encrypted image to glance, the metadata properties above have to be specified and the container format has to be set to ‘encrypted’.

We propose to use an implementation of symmetric encryption provided by GnuPG as a basic mechanism supported by this draft. It is a well established implementation of PGP and supports streamable encryption/decryption processes.

We require the streamability of the encryption/decryption mechanism for two reasons:

Loading entire images into the memory of compute hosts or a users system is unacceptable.
We propose direct decryption-streaming into the target storage (e.g. encrypted volume) to prevent the creation of temporary unencrypted files.

There is already one existing case in Cinder’s current implementation where encrypted images are created. This is when an image is created directly from an encrypted volume. Since the encrypted block data is simply copied into the image, the encryption (usually LUKS) is automatically inherited - as is the encryption key, which is simply cloned in Barbican. We will not change this behavior as a part of this spec. Our changes will only apply, when the user actively intends to create an encrypted image from any volume using the new image encryption extensions.

The key management is handled differently than with encrypted volumes or encrypted ephemeral storage. The reason for this is, that the encryption and decryption of an image will never happen in Glance but in all other services, which consume images. Therefore the service which needs to create a key for a newly created encrypted image may not be the same service which then has to delete the key (in most cases Glance). To delete a key, which has not been created by the same entity, is bad behavior. To avoid this, we choose to let the user create and delete the key. To not accidently delete a key, which is used to encrypt an image, we will let Glance register as a consumer of that key (secret in Barbican [1]) when the corresponding encrypted image is uploaded and unregister as a consumer when the image is deleted in Glance.

Alternatives

Regarding the image encryption, we also explored the possibility of using more elaborated and dynamic approaches like PKCS#7 (CMS) but ultimately failed to find a free open-source implementation (e.g. OpenSSL) that supports streamable decryption of CMS-wrapped encrypted data. More precisely, no implementation we tested was able to decrypt a symmetrically encrypted, CMS-wrapped container without trying to completely load it into memory or suffering from other limitations regarding big files.

We also evaluated an image encryption implementation based on LUKS which is already used in Cinder and Nova as an encryption mechanism for volumes and ephemeral disks respectively. However, we were unable to find a suitable solution to directly handle file-based LUKS encryption in user space. Firstly, the handling of LUKS devices (even when file-based) via cryptsetup always requires the dm-crypt kernel module and corresponding root privileges. Secondly, in contrast to native LUKS used by LibVirt, the LUKS handling available via cryptsetup creates temporary device mapper endpoints where data is read from or written to. There is no direct reading/writing from/to an encrypted LUKS file and LUKS opening/closing needs to be handled accordingly. Lastly, LUKS is a format primarily designed for disk encryption. Although it may be used for files as well (by formatting files as LUKS devices), the handling is rather inconvenient; for example, the size of the LUKS container file needs to be calculated and allocated beforehand since it acts like a disk with a fixed size.

Data model impact

REST API impact

Security impact

There are impacts on the security of OpenStack:

confidentiality of data in images will be addressed in this spec
image encryption is introduced, thus cryptographic algorithms will be used in all involved components (Nova, Cinder, OSC)

Notifications impact

None

Other end user impact

Users should be able to optionally, but knowingly upload an encrypted image.
If an administrator has configured Glance to reject unencrypted images, such images will not be accepted when attempted to be uploaded to Glance.

Performance Impact

The proposed encryption/decryption mechanisms in the OpenStack components will only be utilized on-demand and skipped entirely for image container types that aren’t encrypted.

Other deployer impact

Deployers can toggle the acceptance or enforce the usage of encrypted images by adding/omitting ‘encrypted’ in ‘container_formats’ accordingly.
Deployers enforcing the usage of encrypted images by omitting all other image types in ‘container_formats’ will make public images unavailable due to the lack of a public secrets functionality in Barbican.
A key manager - like Barbican - is required.

Developer impact

None

Upgrade impact

none

Implementation

Assignee(s)

Primary assignee: Markus Hentsch (IRC: mhen)

Other contributors: Josephine Seifert (IRC: Luzi)

Work Items

Add container type(s) with encryption support to Glance
Add registering as consumer for a Barbican secret when uploading an encrypted image
Add unregistering as consumer for a Barbican secret when deleting an encrypted image
Provide compatibility to the image_conversion plugin for Interoperable Image Import (skip conversion attempt for encrypted payload)
Add support for providing the new image properties to the python-glanceclient, so that an image with the container_format: encrypted can be uploaded

Dependencies

GPG is required to be installed on all systems that are required to perform encryption/decryption operations in order to support the proposed base encryption mechanism.
This spec requires the implementation of appropriate encryption/decryption functionality in a global library shared between the components involved in image encryption workflows (Nova, Cinder, OSC). We determined to use os-brick.
The secret consumer API in Barbican is required for glance to be able to register and unregister as a consumer of a secret

Testing

Documentation Impact

It should be documented for deployers, how to enable this feature in the OpenStack configuration. An end user should have documentation on, how to use encrypted images.

References

[1] Barbican Secret Consumer Spec: https://review.opendev.org/#/c/662013/

Nova-Spec: https://review.openstack.org/#/c/608696/

Cinder Spec: https://review.openstack.org/#/c/608663/

History

Revisions
Release Name	Description
Victoria	Introduced

Use Centralized database for caching

Thu, 26 Oct 2023 00:00:00

https://blueprints.launchpad.net/glance/+spec/centralized-cache-db

Problem description

We use two different database(s) for glance. One is a centralized database managed by oslo.db (default is MySQL) for storing information related to regular operations; the other one is a sqlite database for storing information about caching related operations. Even though the sqlite database is only created if we enable cache middleware we still need to maintain it along with centralized database. Also glance caches are local to each controller node (glance api service), so in case of multiple glance services running we have multiple sqlite database(s) operational.

Proposed change

We propose to use centralized database (default MySql) for caching related operations and stop using sqlite database from this release onwards. In this case we can get rid of creating sqlite database per glance service and have everything under central database.

At present users can choose between sqlite (Default) and xattr for how caching can be controlled using image_cache_driver configuration option. We propose to introduce new driver centralized_db and required configuration options, make it default and mark sqlite driver and related configuration options as deprecated in this cycle and remove it in ‘D’ development cycle.

In order to use central database for caching we recommend to have almost similar schema (columns) as sqlite database with additional reference to local node (glance service). New deployments will directly start using this new central database if cache is enabled.

In case of upgrades/updates we need to deal with migrating existing records from sqlite database to central database. Since we might have multiple glance(s) running this will be difficult to handle using alembic migrations. To avoid this using migration we can perform this operation one time during service startup. If sqlite database is present at service start and image_cache_driver is not set to centralized_db then we will read records from it and insert those in newly created cached_images table in central database. Once all records are migrated we will clear the sqlite database table and keep the sqlite database file as it is (to be deleted by administrator/operator later if required). Important point here is once deployer chooses to use centralized_db and we migrate their records out of sqlite to centralized database, then we will not migrate them back if deployer wants to revert back to sqlite driver.

As deployment can have multiple glance services running, to record cache details separately for multiple glance services, we can use existing configuration option worker_self_reference_url which will differentiate cache records for each glance service running in the deployment. worker_self_reference_url is used to identify nodes in the case of new import workflow earlier, and now the same can be used for this purpose as well. Since worker_self_reference_url is required for glance-direct import method as well, if we found that it was not set by the deployer for any of the glance setup, then at the service startup we will log appropriate error message and prevent the service from starting. Also, since in the case of multiple nodes, worker_self_reference_url needs to be set to a unique value per node, and if, by mistake, two or more nodes have the same value for it, then we will not migrate the cache data of that node to a centralized database with logging an appropriate warning message.

Once we migrate existing cache records to central database we need to ensure that existing cache related command line utilities like cache-cleaner, cache-pruner are working as expected. Since these utilities use dedicated configuration file (glance-cache.conf), we need to make provision to introduce required configuration options in this file so that these tools can connect to central database.

Alternatives

None

Data model impact

Create two new tables in mysql, one to record cache related operations and other to record glance api references as shown below.

Schema for cached_images:

CREATE TABLE cached_images (
  id bigint auto increment PRIMARY KEY,
  image_id varchar(36),
  last_accessed DateTime,
  last_modified DateTime,
  size bigint,
  hits int,
  checksum varchar(32),
  node_reference_id int,
  UNIQUE KEY `uq_cache_unique_id` (`image_id`,`node_reference_id`)
  FOREIGN KEY (node_reference_id) REFERENCES cache_node_reference(node_reference_id)

)

Schema for cache_node_reference:

CREATE TABLE cache_node_reference (
  node_reference_id int PRIMARY KEY auto increment,
  node_reference_url varchar(255)
  UNIQUE KEY `uq_node_reference_url` (`node_reference_url`)
)

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: abhishekk
Other contributors:: None

Work Items

Deprecate existing sqlite driver and related configuration options
Create alembic scripts for two new tables
Add model/queries to sqlalchemy for new tables
Introduce new centralized_db driver and related configuration options
Change cache code to use new mysql database
Add logic at service startup to migrate data from sqlite to mysql
Modify related commandline tools like cache-cleaner, cache-pruner to use centralized driver
Tempest tests to ensure multiple glance nodes uses centralized database.
Grenade test(s) to verify upgrading scenario(s)

Dependencies

None

Testing

Unit Tests
Functional Tests
Tempest Tests
grenade tests to verify upgrade scenario

Documentation Impact

Need to document use of worker_self_reference_url in case of import workflow and cache related operations.

References

None

New Location APIs

Thu, 18 May 2023 00:00:00

https://blueprints.launchpad.net/glance/+spec/new-location-apis

Problem description

Currently we have two security vulnerabilities with show_multiple_locations config option, OSSN-0065 [1] and OSSN-0090 [2]. If we enable show_multiple_locations and the policies for add/update (set_image_location), get (get_image_location) and remove (delete_image_location) locations are set for non-admins then non-admin users can modify location data to corrupt an image that they own. Note that the policies for add, get and remove locations are set for non-admins by default else a non-admin user cannot associate data with an image record, or retrieve image data, or delete image data.

When show_multiple_locations is False, users cannot modify image locations via the image-update API call, even if they have the {get,set,delete}_image_location permissions. However, there are some popular use cases where other services can bypass Glance and store or access image data directly in the backend by writing or reading image locations, using the image owner’s credentials, and this is why operators want to set show_multiple_locations to True. What operators want to do, however, is to enable optimized image data access; exposing image locations to non-admin users is a side-effect, not the goal. We currently recommend that operators who want to use optimized data access use a specialized Glance instance for services, and only expose glance-api to end users with show_multiple_locations set False. This is inconvinient for certain users.

Proposed change

There will be 3 phases in which the work will be done as follows:

Introduce 2 new API calls that allow operations on image locations which are described in detail in the REST API impact section. These calls will replace the image-update mechanism for consumers like cinder and nova.
Modify the consumer (cinder/nova) code to use the new location APIs. Also modify HTTP store to use new location APIs.
Remove show_multiple_locations config option when it is no longer required by other services (cinder/nova) to perform operations on locations. This will mostly be done 1 or 2 cycles after the consumers have adapted the new location APIs to handle the upgrade cases.

The config option show_multiple_locations has been deprecated since Newton but we will keep the config option until the consumers of glance locations (nova, cinder, http store etc) start using the new location APIs. Since this is a major effort spanning across multiple services (nova, cinder, glance), we will implement the work items in different cycles to provide enough time for developers (to implement this) and operators (to move away from the config option).

We will introduce 2 new policies, for each API performing different operations like add and get, as follows:

The add policy can default to the project member or service role (when it is implemented).
The get policy will default to the service role for authorization.

Along with the new add policy, we will add a check in the location add API code to check the status of image and only add location if it is in QUEUED state and adding location when the image is in other states will be disallowed. This is done in order to prevent malicious users from modifying the image location again and again since the location added for the first time is the correct one as far as Glance is concerned.

We will also introduce a new configuration parameter do_secure_hash on the glance side which will tell the API if we want to do the hash calculation or not. This will be useful in cases when nova, cinder etc, adds a location in glance since glance does not calculate the hash and checksum automatically in these cases. The value of do_secure_hash will be True by default.

After nova or cinder send a request for adding a location for the VM snapshot or upload volume case respectively and do_secure_hash is True, glance will start a background process that will calculate the hash of the image. Unless we have validation_data (in the request body) to be verified, image will be set to active state after registering the location even if the hash calculation is ongoing in the background. This is done so that the image can be used to create instances and bootable volumes instantly after we’ve registered the location and not wait for the hash calculation since it is a long running task. After the hash calculation completes, image properties will be updated with the checksum, os_hash_algo and os_hash_value values.

Following are the cases of image transition with different values of do_secure_hash and validation_data:

do_secure_hash is True and validation_data is not None:

Image transition: (queued, importing, active)

In this case the consumer provides the hash values for validation and hash is calculated by glance. An example of this case will be providing validation_data for HTTP store. Here image hash will be calculated and verified before setting image to active state.
do_secure_hash is True and validation_data is None:

Image transition: (queued, active)

In this case validation data will not be provided by the consumer but hash is calculated by glance. Examples of this case will be when nova snapshots an instance or cinder uploads a volume to image. Here image hash calculation will be done and updated after setting image to active state. This is a tricky case because the consumer will have no idea if the active image will ever have a hash value or not and if it should wait for the hash to be populated in the image or not. To handle this, we will set the os_hash_algo value in the image properties so the consumer will know that hash calculation is ongoing for this image and the hash will be populated here. Here are the following cases:
- active image and no os_hash_algo: This image will not have hash value populated.
- active image and has os_hash_algo: Poll for active image status and os_hash_algo until you get os_hash_value. Polling for active image status is optional since the image gets active when validation_data is not provided and hash calculation is ongoing in the background i.e. this case. The os_hash_algo value will be popped if hash calculation fails.
do_secure_hash is False and validation_data is not None:

Image transition: (queued, active)

In this case validation data will be provided by the consumer and hash is not calculated by glance. An example of this case will be providing validation_data for HTTP store. Here image hash will not be calculated and verified but directly set to image with values provided by the user.
do_secure_hash is False and validation_data is None:

Image transition: (queued, active)

In this case validation data will not be provided by the consumer and hash is not calculated by glance. This can happen for all cases. Here hash value won’t be set in the image.

If the hash calculation fails, we will add a retry mechanism that will reinitiate the task. We will add a new configuration option http_retries with a default value of 3 i.e. the hash calculation will be executed maximum 3 times by default if the first and second tries fail. If after all the retries, the hash calculation still fails, we will not update the hash and checksum values and image will stay in active state.

End-user access to image locations via the Image API is no longer necessary. Since Train, Glance has multiple stores support, and we have added API calls that allow users to manipulate data locality with respect to store. Further, a store is an opaque identifier, whereas an image location exposes backend details that users don’t need to know.

Here are the current use cases for the direct manipulation of image locations along with an explanation of how they can be handled by the new Location API.

When using a copy-on-write (COW) backend shared by Nova and Glance, Nova can create an image record in Glance, snapshot a server image directly in the backend, and set the location on the image record. This use case is covered by the new add-location call, and having its default policy be project member (image owner) or service.
A user wants to have a single image record, but have image data stored in multiple locations for locality (i.e., to have image data as close as possible to where it’s consumed). This use case is handled by the glance multiple stores feature plus image import, which since API v2.8, allows a ‘stores’ parameter specifying where the image data should be stored. This applies to both newly created images and existing images (via the copy-image import method). In this workflow, Glance itself manipulates the image locations; there is no need for the user to interact with locations directly.
An operator wants to introduce a new storage backend and decommission the current backend while keeping the same image catalog. Similar to #2, this can be handled by using the copy-image import method and the delete-image-from-store API call introduced in v2.10. Note that there are some exceptions to this like:
1. HTTP store is read-only, so we can’t use copy-image in this case.
2. For RBD store, we will create a dependency chain if we launch a VM or create a bootable volume from it hence we can’t delete the source image until all of it’s children are flattened.
3. For cinder store, if the cinder backend uses COW cloning, it is similar to the RBD case mentioned in b) else the image delete will succeed.

Following APIs are not being implemented:

Update: For service to service interaction, there is no value in updating the metadata of a location. This would be beneficial if we plan to remove the existing location code from image-update call and support the usecase of operators/end-users doing location operations.

Delete: We already have Delete Image From Store API for this purpose. We don’t require the Delete Image From Store API call for the current usecase but if we plan to extend the location APIs in future, we can do this by updating the policies enforced by Delete Image From Store operation from the default role:admin to role:admin or role:service.

Alternatives

We can remove the show_multiple_locations config option and filter the images with the admin_or_service role. This will require the consumers to provide admin credentials during add or get of an image to get the location. This was the original proposal but due to the disagreement here [3], we changed the design to the current proposal.
Another alternative is to add this functionality in the import workflow. We can add a new import method direct-location which will allow end users to specify the location and metadata parameters and create a new image based on the given parameters. We can also update an existing image with location and metadata values but will require the image to be in queued state.

For this, we will need to add a new import method direct-location and also add --metadata and --location parameters to the following commands:
- glance image-create-via-import --import-method direct-location --location <location> --metadata <key1=value1, key2=value2 ...>
- glance image-import --import-method direct-location --location <location> --metadata <key1=value1, key2=value2 ...>

Data model impact

None

REST API impact

We are going to add 2 new location APIs:

Add Location

This will add a new location to an existing image. The request body will contain the location URL and validation_data [4] (optional). The purpose of including validation_data in the request body is when the consumer wants to validate the image hash or just directly wants to add the hash values to the image. The cases of validation_data with do_secure_hash are described in the Proposed change section. An example where validation_data will be provided is the HTTP store case, where the user will provide hash value for the HTTP image.

Unlike old location API, we will not provide support of adding a location on a particular index. If we want to get the benefit of indexes, we can use the old location APIs or set location strategy as store_type [5]. A new location strategy store_identifier is proposed [6] and should be useful to download image from a specific store in case multiple stores are configured.

POST /v2/images/{image_id}/locations
- JSON request body
```
{
    "url": "cinder://lvmdriver-1/1a304872-b0ca-4992-b2c2-6874c6d5d5f9",
    "validation_data": {
        "os_hash_algo": "sha512",
        "os_hash_value": "6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869",
    }
}
```
- JSON response body
  - Success - 200
```
{
    "url": "cinder://lvmdriver-1/1a304872-b0ca-4992-b2c2-6874c6d5d5f9",
    "metadata": "{'store': 'lvmdriver-1'}"
    "validation_data": {
        "os_hash_algo": "sha512",
        "os_hash_value": "6b813aa46bb90b4da216a4d19376593fa3f4fc7e617f03a92b7fe11e9a3981cbe8f0959dbebe36225e5f53dc4492341a4863cac4ed1ee0909f3fc78ef9c3e869",
    }
}
```
  - Error - 409 (Location already exists or if image is not in QUEUED state), 403 (Forbidden for users that are not owner), 400 (BadRequest if hash validation fails)

Get Location(s)

This will show all the locations associated to an existing image. Returns an empty list if an image contains no locations.

GET /v2/images/{image_id}/locations

JSON response body

[
    {
        "url": "cinder://lvmdriver-1/0f031ed1-5872-43d5-a638-4b0d07c10ab5",
        "metadata": "{'store': 'lvmdriver-1'}"
    },
    {
        "url": "cinder://cephdriver-1/11b4fa9f-a44b-46c9-950c-0026c467252c",
        "metadata": "{'store': 'cephdriver-1'}"
    }
]

Error - 404 (Image ID does not exist), 403 (Forbidden for normal users)

The transition of image state during the image create operation will be as follows. Image upload (PUT), image stage (PUT) and location add (POST), will transition the image from queued to the next state that could be either of the following:

saving
uploading
importing
active

Below are the valid transitions for image from queued state.

‘queued’: (‘saving’, ‘uploading’, ‘importing’, ‘active’, ‘deleted’)

Security impact

No worse than it is now, and possibly better.

The get-locations policy is restricted to the ‘service’ role, so users will not be able to see image locations. Thus with ‘show_multiple_locations’ and ‘show_direct_url’ set to False, the new get-locations API will not expose location information to users.
The add-location policy is restricted by default to image-owner. This will allow end users to add a location to an image to address current uses of this functionality that we aren’t aware of. Even allowing this, the data-substitution attack is blocked because the API call will only be allowed for an image in ‘queued’ status. The add-location API cannot be used to add a location to an image in other states and then delete the original location, so the OSSN-0065 attack is not possible under this scenario. Further, the add-locations call (unlike the current method of updating locations via PATCH), does not require the locations to be visible to succeed. Thus operators will be able to configure Glance with ‘show_multiple_locations’ and ‘show_direct_url’ set to False, even when other services are sharing a COW backend with Glance and the operator wants an optimized workflow.

Notifications impact

None

Other end user impact

Since the new APIs are mainly for service to service interaction (except the HTTP store case), we will only expose the location add API via CLI. However, we will need to add methods for all APIs in openstacksdk (that will call the new location APIs) that will be used by other consumer services like cinder and nova. End users can still use the existing commands (that internally calls the image-update API) to perform operations on locations:

glance location-add: Add a location (and related metadata) to an image.
glance location-delete: Remove locations (and related metadata) from an image.
glance location-update: Update metadata of an image’s location.

We will also add a new command to glanceclient and OSC that will allow end users to add the location url and validation-data for HTTP store case.

glance add-location-properties --url <location> --validation-data <os_hash_algo=value1, os_hash_value=value2>
openstack image add location properties --url <location> --validation-data <os_hash_algo=value1, os_hash_value=value2>

Performance Impact

In the old location API, the consumers (nova, cinder) registered the location in glance and the checksum, hash etc values weren’t calculated. After the consumers adapt to the new location API, and the do_secure_hash config parameter is True (default), glance will read the image and calculate the hash in the background. The hash calculation will be a long running task so it will consume resources, however, this won’t affect the operation requested by nova or cinder as the image will transition to active state even when the hash calculation is ongoing.

The performance downside will result in creation of more secure images and the impact needs to be conveyed to the operators/end users with documentation and releasenotes. Since do_secure_hash will be a configurable parameter on glance side, we will add suitable help text to convey the performance and security impact of enabling/disabling this option.

Other deployer impact

None

Developer impact

Consumers like Cinder, Nova and HTTP store need to modify code to call the new client functions to access the API. Some of the key things to consider while implementing consumer side changes are:

We will use SDK to make the API calls. The changes to call new location APIs will be in SDK and also in OSC/glanceclient for location ADD in case of HTTP store.
Keep backward compatibility with old behavior. Glance should support the legacy behavior as well as the new way to add/get locations. This is useful in upgrade cases where one compute node is running 2023.1 (Antelope) code and the other compute node has been upgraded to 2024.1 (CC) release.
Testing should be done to see if the existing functionalities supported with the legacy location APIs works as expected with the new APIs.

Implementation

Assignee(s)

Primary assignee:: pdeore
Other contributors:: whoami-rajat

Work Items

Add 2 new Location APIs for add and get operations.
Modify consumers like cinder and nova and http store to use the new location APIs.
Add a new configuration parameter do_secure_hash in glance and document it’s impact.
Add a new configuration parameter http_retries in glance and document it’s usage.
Add SDK support to call the new APIs.
Add a releasenote mentioning that we will remove the config option show_multiple_locations when the consumers (nova/cinder/http store) shift to using new location APIs.
Tempest tests for the new add-location and get-location APIs.

Dependencies

None

Testing

Unit Tests
Functional Tests
Integration Tests
Tempest Tests

Documentation Impact

Need to document new location APIs.

References

Deprecate show_multiple_locations option | https://review.opendev.org/c/openstack/glance/+/313936
Update deprecated show_multiple_locations helptext | https://review.opendev.org/c/openstack/glance/+/426283
Update show_multiple_locations deprecation note | https://review.opendev.org/c/openstack/glance/+/625702
Original security bug | https://bugs.launchpad.net/ossn/+bug/1549483
New security bug | https://bugs.launchpad.net/ossn/+bug/1990157

2023.2 Project Priorities

Mon, 17 Apr 2023 00:00:00

The Bobcat Cycle priorities were discussed during Bobcat Virtual PTG. The preliminary list was maintained on the Bobcat PTG etherpad.

This list is an estimate of what the Glance project team can accomplish during the 2023.2 Cycle based on our developer’s estimates of how much time they can commit to Glance.

The following list is roughly indicates milestonewise priorities.

Priority Item	Owner(s)	Spec(s)	Target release milestone
New Location APIs	Pranali Deore	new location apis	M1
RBD Trash	Eric Harney	No Spec yet	M2
Image Encryption	Josephine Seifert	image encryption	M2
All image APIs SDK Support	Mridula Joshi	None	M1
Tempest coverage for New Location APIs	Pranali Deore	new location apis	M2
All Metadef APIs SDK Support	Mridula Joshi	None	M2
Remove/Migrate Single Store Tests	Pranali Deore	None	M3
Calculation multihash,checksum for existing location calls	Mridula Joshi	No Spec Yet	M3

Important dates

This is an abbreviated list focused on dates relevant to Glance. See Bobcat Release Schedule for the complete list for OpenStack.

Milestone	Week of	What
R-21	May 12	Bobcat-1 Milestone
R-13	Jul 07	Bobcat-2 Milestone Glance Spec freeze
R-6	Aug 25	glance_store Bobcat release (final release for non-client libraries)
R-5	Sept 07	Bobcat-3 milestone Glance feature freeze python-glanceclient Bobcat release (final release for client libraries);
R-3	Sept 15	RC-1 release and string freeze
R-1	Sept 29	final RCs
R-0	Oct 06	Bobcat release

Spec Lite: Add support for extending attached (in-use) volumes

Thu, 29 Dec 2022 00:00:00

project:

glance_store

problem:

When creating an image in cinder store, we perform a series of API calls to cinder creating, attaching, detaching, extending etc the volume. The sequence of operations performed to copy image into the volume are, attach the volume, copy image contents into the volume (until we’ve space left in volume), detach the volume, see if extend is required (if image is bigger than current volume size) and perform it. We repeat the operations until the whole image is copied into the volume. The part where we detach the volume, extend and attach it again is very inefficient as we’ve to do it for every 1GB of image. For some backends, cinder supports extending attached (in-use) volumes and we should use that to optimize the image create operation. Cinder backends that support extend in-use volumes: https://docs.openstack.org/cinder/latest/reference/support-matrix.html#operation_online_extend_support

Another issue noticed related to this is, since cinder only supports extend and not shrink, we end up with extending the volume 1 GB at a time and eventually taking more time to create large images.

solution:

We will introduce a new config option, cinder_do_extend_attached which will be a boolean option. Operators can set it to true if the cinder backend they are using supports extending attached volumes. The default value will be false. If we have cinder_do_extend_attached set to true, we will call the cinder os-extend API with microversion 3.42 that will allow us to extend the attached volume. Finally, we will call the extend_volume method of os-brick to instruct the kernel to resize the volume on the host.

Additional efforts:

To address the concern related to 1 GB resize, cinder store already has handling of size, if correctly passed by glance, we initialize the volume size with the image size avoiding unnecessary extend operations. So glance needs to be updated to pass correct image size to cinder store.

impacts:

None

alternatives:

None

timeline:

2023.1

link:

reviewers:

Abhishek Kekane, Brian Rosmaita, Dan Smith, Erno Kuvaja

assignee:

Rajat Dhasmana (whoami-rajat)

2023.1 Project Priorities

Thu, 01 Sep 2022 00:00:00

TODO(pdeore): fill this in after the PTG

Spec Lite: Add region_name parameter to s3 store

Mon, 22 Aug 2022 00:00:00

project:: glance_store
problem:: The current s3 driver determine region_name from amazon endpoints, but some operators are using an s3 compatible API which is not from amazon. In such situation, the region_name parameter cannot be guessed from endpoints and result beeing set to None. When region_name=None is given to instanciate the s3 client, the internal boto3 code is then using a default region_name which is wrong (like us-west-1).
solution:: We need to add a new parameter s3_store_region_name in glance config so that we can use this value instead of guessing it from s3_store_host.
impacts:: None
assignee:: Arnaud Morin <arnaud.morin@ovhcloud.com>

Provision for immediate caching of an image

Wed, 20 Apr 2022 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/glance/+spec/instant_caching

Provision for immediate caching of a given image on a particular glance node.

Problem description

In Yoga we have added support for caching related operations under V2 API, but still we need to depend on existing periodic job to cache the actual image. This means user/deployer/admin needs to wait until next periodic job to run for caching of an actual image. Even though default time for periodic job is 300 seconds (5 minutes) it may vary deployment wise and the user has to wait until the actual image is cached.

Proposed change

We are proposing to remove existing periodic job from glance-api and modify existing PUT /v2/cache/{image_id} API which will queue an image for caching and start caching it immediately. If any existing instant caching operation is in progress then the latest image will be added to the queue so that it will be picked up for caching as soon as previous operation completes. User can use GET /v2/cache API call to see the size of the queue.

Alternatives

Instead of modifying existing API, we can add a new POST API which will start immediate caching of specified image. We can use existing policy cache_image so that user with appropriate permissions can perform this operation. To make the policy compatible with secure RBAC we need to pass required parameters like ImageTarget while enforcing the policy. It is recommended to keep cache_image operation limited to admin use only.

Another alternative solution could be, instead of removing the old periodic job, the new API should trigger the periodic job instantly to cache all the images rather than just caching a specified image at a time. This solution need to be thread safe as there might be a chance of existing periodic job is already running or new periodic job will start running while we trigger the same via API.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Caching is and will remain local to each glance node and as this command will be executed remotely, operator needs to know the direct URL of each glance node which are behind the load balancer. Operator needs to provide this direct URL to glanceclient so that client should hit particular node to trigger immediate caching on that node.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: dansmith or abhishekk
Other contributors:: None
Reviewers:: cyril-roelandt/rosmaita/jokke/pdeore/mrjoshi

Work Items

Modify existing API (PUT /v2/cache/{image_id})
Remove the ‘cache_prefetcher_interval’ config option and related tests.
Modify documentation, update API reference
Tempest coverage for Caching operations

Dependencies

None

Testing

All new cache API should be tested using tempest tests

Documentation Impact

The API documentation will need to be updated
Need to update Cache documentation as well with new information

References

https://docs.openstack.org/glance/victoria/admin/cache.html

Expanding stores-info detail for other stores

Tue, 29 Mar 2022 00:00:00

https://blueprints.launchpad.net/glance/+spec/expanding-stores-detail

Problem description

In Yoga release we added a new API (GET /v2/info/stores/detail) which exposes the store specific details of a store. At the moment the Discovery API only exposes the store details and properties of the RBD backend. We want to make the API more generic and expose details of other stores also.

Proposed change

We will be extending the functionality of Discovery (GET /v2/info/stores/detail) API which will expose the store specific details about the store like store type and other specific store properties of other stores like cinder, swift, filesystem etc. Options that can be beneficial to be exposed for different stores :

Cinder: cinder_volume_type
Filesystem: filesystem_store_dir, chunk_size, thin_provisioning, filesystem_store_dirs
Swift: container, obj_size and chunk_size
S3: s3_store_large_object_size, s3_store_large_object_chunk_size, s3_store_thread_pools

We will use the existing method get_store_from_store_identifier which returns the store class instance and will utilize it to fetch the store specific information to return it via API.

Alternatives

None

Data model impact

None

REST API impact

With this new implementation, we will now be returning the “properties” value for other stores(apart from RBD).

GET /v2/info/stores/detail

The output will be as follows:

{
    "stores": [
        {
            "id":"reliable",
            "type": "rbd",
            "description": "Reliable RBD store",
            "default": true,
            "properties": {
                "pool": "pool1"
                "chunk_size": 65553
                "thin_provisioning": false
            }
        },
        {
            "id":"cheap",
            "type": "file",
            "description": "Cheap file store",
            "properties": {
                "datadir": "fdir"
                "chunk_size": 65553
                "thin_provisioning": false
            }
        },
        {
            "id":"fast",
            "type": "cinder",
            "description": "Fast Cinder Store",
            "properties": {
                "volume_type": "volume1"
                "use_multipath": false
            }
        },
        {
            "id":"slow",
            "type": "swift",
            "description": "Slow Swift store",
            "properties": {
                "container": "container1"
                "obj_size": 52428
                "chunk_size": 204800
            }
        },


    ]
}

Security impact

This API does expose some additional sensitive information, but only to admins, consistent with other things we already expose.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: mrjoshi, whoami-rajat
Other contributors:: None

Work Items

Fetch details from the specific store/backed and return it in the API response

Dependencies

None

Testing

Unit Tests
Functional Tests

Documentation Impact

Add documentation providing details about the properties exposed for each store

References

https://review.opendev.org/c/openstack/glance-specs/+/817391 https://review.opendev.org/c/openstack/glance/+/824438

Zed Project Priorities

Tue, 15 Feb 2022 00:00:00

TODO(abhishekk): fill this in after the PTG

Spec Lite: Add ability to purge all needed rows by glance-manage script.

Mon, 29 Nov 2021 00:00:00

project:: glance
problem:: When a user wants to purge all deleted rows they use “glance-manage db purge” and “glance-manage purge_images_table”. A user never knows how many deleted rows are still in a table. So they need to launch the script many times until the script reports that 0 rows were deleted in every table. It’s inconvenient.
solution:: use the same notation as with our other limits on “–max-rows” when set to -1 it would purge them all rather than the default 100 lines or what ever specified on the command line.
impacts:: None
how:: if “–max-rows” equals -1 the script purges deleted rows without number limit.
alternatives:: None
timeline:: Zed
link:: https://review.opendev.org/c/openstack/glance/+/813691
reviewers:: Abhishek Kekane, Cyril Roelandt
assignee:: mitya-eremeev-2

Expose store specific information

Wed, 10 Nov 2021 00:00:00

https://blueprints.launchpad.net/glance/+spec/expose-store-specific-info

Problem description

When we upload a volume from cinder’s RBD backend as an image to glance’s RBD backend, the generic code path to copy data chunk by chunk is executed, which makes the operation very slow. This can be optimized by using COW cloning for rbd backends and for this Cinder requires store type and rbd specific store info from glance side.

Proposed change

At the moment the Discovery (GET /v2/info/stores) API provides the list of multiple-backend present and the default store but not the type of the stores.

We will be extending the functionality of Discovery (GET /v2/info/stores) API by adding a new API (GET /v2/info/stores/detail) which will expose the store specific details about the store like store type and other specific store properties.

Since the store specific information is mostly intended for other services (like cinder) or operators to consume and not for end users, this operation will be admin only and we will introduce a new policy stores_info_detail, that will default to admin only rule, to restrict this for non-admin users. In future, when the service role will be in place in keystone to facilitate service to service interaction, this policy rule will be adjusted accordingly.

We will use the existing method get_store_from_store_identifier which returns the store class instance and will utilize it to fetch the store specific information to return it via API in a defined format. For more details see REST API impact section.

Alternatives

None

Data model impact

None

REST API impact

We are going to add a new API GET /v2/info/stores/detail which will return the store details like store type and store specific properties. It will be validated by the new policy rule stores_info_detail which defaults to admin only and then the detailed info related to the stores will be returned.

GET /v2/info/stores/detail

The output will be as follows:

{
    "stores": [
        {
            "id":"reliable",
            "type": "rbd",
            "description": "More expensive store with data redundancy",
            "default": true,
            "properties": {
                "pool": "pool1"
            }
        },
        {
            "id":"cheap",
            "type": "file",
            "description": "Less expensive store for seldom-used images",
            "properties": {}
        },
    ]
}

We are going to add a field type to specify the type of store. Also we will add a field properties which will be a JSON object type and contain the store specific properties. For the current usecase we are only going to add RBD store info and leave the properties for other stores as empty JSON objects {}.

Security impact

Since this optimization skips the writing of image that happens on the glance side, it will also skip the checksum and hash value calculated in that scenario. Due to the above case, we will add a new config option on the cinder side to enable/disable this optimization. By default, it will be disabled.

Another case is we require direct-url and image locations in the image detail response to take benefit of any optimization in nova-glance or cinder-glance interactions which is a security concern described in OSSN-0065.

Notifications impact

None

Other end user impact

A new optional parameter --detail will be added to the stores info command on glanceclient side.

The parameter will accept boolean values. If True, it will expose store specific information else will show the non-detailed information about stores as it works currently.

Stores Detail: glance stores-info --detail

--detail

With sufficient permissions, display additional information about the stores.

Performance Impact

Uploading volume to image incase of cinder RBD to glance RBD will be significantly improved.

Image size	2GB	3GB	5GB
Time without COW clone	1min17Sec	1min15Sec	2min49Sec
Time with COW clone	1.29Sec	2.32Sec	1.63Sec
	-98%	-97%	-99%

Other deployer impact

None

Developer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: mrjoshi, whoami-rajat
Other contributors:: None

Work Items

Add an optional parameter --detail in the stores-info command on the glanceclient side.
Add a new API to glance GET /v2/info/stores/detail.
Create a new policy stores_info_detail that will default to admin only rule and enforce it if detail flag is passed.

Dependencies

None

Testing

Unit Tests
Functional Tests
Tempest Tests

Documentation Impact

A new section for the new API GET /v2/info/stores/detail needs to be added in the api-ref.

References

https://review.opendev.org/c/openstack/cinder-specs/+/810363

Glance Quota API

Tue, 09 Nov 2021 00:00:00

https://blueprints.launchpad.net/glance/+spec/quota-api

In the Xena cycle, Glance gained quota support. Through the use of unified limits, it was possible to implement this without introducing an API. However, for full functionality and a better user experience, Glance should expose an API so that users can see their current usage (and limit) instead of just receiving an error when they go over. This spec outlines the addition of that API.

Problem description

Glance users can now be limited by quotas set by the operator, but are unable to see their current usage (and limits) in any way other than the error message they receive when they go over. An API is required to expose the limit and usage to the user.

Proposed change

Glance already has an information discovery API, which will be used as a base for exposing the information: a new route for /info/quota will be added. Information exposed will include the limit and current usage for each quota type for the calling user. If a quota is disabled, then the limit for that element will be exposed as -1 in accordance with Keystones “unlimited” sentinel notation.

Alternatives

We could continue to have no API.

Data model impact

None

REST API impact

This spec includes a single new endpoint:

New API

List quotas

Common Response Codes

Success: 200 OK
Failure: 400 Bad Request with details
Forbidden: 403 Forbidden

[New API] List quotas

List all quotas available for the user:

GET /v2/info/quotas
{
  "quotas": {
    "image_size_total": {
      "usage": 32,
      "limit": 1024
    },
    "image_staging_total": {
      "usage": 0,
      "limit": 1024
    },
    "image_count_total": {
      "usage": 4,
      "limit": 100
    },
    "image_count_uploading": {
      "usage": 0,
      "limit": 10
    }
  }
}

Response codes:

200 – Upon authorization and successful request. The response body contains the JSON payload with quota information.
400 – Quota information is not available (likely due to configuration or backend error)
403 – Permission denied

JSON schema for the response:

 'usage': {
    'type': 'array',
    'items': {
        'type': 'object',
        'additionalProperties': True,
        'validation_data': {
            'type': 'object',
            'additonalProperties': False,
            'properties': {
                'usage': {'type': 'integer'},
                'limit': {'type': 'integer'},
            },
        },
    },
},

Security impact

The other information discovery APIs do not check any policy, and none will be added for this new quota API. It is assumed that there is no reason to disable a user’s view of their own usage and limit information. The former is already visible by listing current resources and doing the math.

Notifications impact

None.

Other end user impact

A glanceclient and openstackclient change will be required to expose this to interactive users.

Performance Impact

None.

Other deployer impact

No impact beyond the requirements for enabling quotas in general.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: danms

Work Items

Implement the API with unit/functional tests
Document the API in api-ref
Write a tempest test to check the API
Implement support in OpenstackClient
Implement support in glanceclient
Client docs

Dependencies

This will require a bump to an (already-released) new version of oslo.limit in order to query the limits without active enforcement.

Testing

Unit and functional tests in Glance. Tempest tests against the existing quota-enabled jobs.

Documentation Impact

The api-ref will need updating, as well as usage information for the interactive clients.

References

https://specs.openstack.org/openstack/glance-specs/specs/xena/approved/glance/glance-unified-quotas.html

Provision to append the new metadef-tags with the existing metadef-tags

Mon, 11 Oct 2021 00:00:00

https://blueprints.launchpad.net/glance/+spec/append-tags

In the Glance API we are not having the provision to append the new metadef-tags with the existing metadef-tags. Introducing a new optional parameter would help the user to select whether to append the tags or to go with current behaviour i.e. overwrite the existing tags.

Problem description

Our md-tag-create-multiple (POST /v2/metadefs/namespaces/{namespace_name}/tags) API overwrites existing tags for specified namespace rather than creating new one in addition to the existing tags. Whereas if you try to create different tags using md-tag-create (POST /v2/metadefs/namespaces/{namespace_name}/tags/{tag_name}) it is adding new tag in addition to existing ones. So we should have consistency between the two APIs i.e. the ability to append the tags in md-tag-create-multiple API.

Proposed change

At the moment the glance API only overwrites the existing tags with the newly created tags i.e. the original/default behaviour. The new behaviour suggests to append the new tags with the existing ones. The goal is to provide an optional header X-Openstack-Append that takes boolean values, which will default to the original behaviour. If not present, the behavior is the same as passing X-Openstack-Append: False.

If header is present then we are going to append new tags else keep the old behavior.

In addition to this we will add an optional parameter at glanceclient side which will default to the original behaviour and if the user wants to append the new tags it can be by changing the value of parameter to True.

Alternatives

None

Data model impact

None

REST API impact

We are going to change the API and will be adding a X-Openstack-Append in the header. The rest API look like this:

POST /v2/metadefs/namespaces/{namespace_name}/tags

Header fields:

X-Openstack-Append

The <boolean> ‘X-Openstack-Append’ is an optional header, refers to appending the tags to the existing tags. It takes boolean values if True then it will append the tags ,if False it will overwrite the tags, default value is False.

Example curl usage:

curl -i -X POST -H "X-Auth-Token: $token" -H "X-Openstack-Append: False"
-H "Content-Type: application/json"
-d '{"tags": [{"name": "sample1"}, {"name": "sample2"}]}'
 $image_url/v2/metadefs/namespaces/{namespace_name}/tags

Security impact

None

Notifications impact

None

Other end user impact

We will add an optional parameter --append to the glanceclient command md-tag-create-multiple to provide the facility of appending the tags. If the parameter is present then it will the append the tags to existing ones else will overwrite the existing tags.

Create multiple tags: glance md-tag-create-multiple <NAMESPACE> --names <NAMES> [--delim <DELIM>] --append

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: mrjoshi
Other contributors:: None

Work Items

Add a boolean parameter in the header and change API functionality as per parameter’s value
Add an optional parameter --append at the client side
Add unit test coverage for checking the functionality.
Add tempest test

Dependencies

None

Testing

We will provide unit tests coverage for testing the functionality based on the header.

Documentation Impact

The documentation needs to be updated with the new API behaviour.

References

https://bugs.launchpad.net/glance/+bug/1939169

https://review.opendev.org/c/openstack/glance/+/804966 https://review.opendev.org/c/openstack/python-glanceclient/+/813591

Yoga Project Priorities

Thu, 02 Sep 2021 00:00:00

TODO(abhishekk): fill this in after the PTG

Spec Lite: Remove sqlalchemy-migrate

Tue, 13 Jul 2021 00:00:00

project:: glance
problem:: The sqlalchemy-migrate library is unmaintained and has been deprecated in favour of alembic. It is not compatible with sqlalchemy 2.0 and is issuing many warnings suggesting Python 3.10 compatibility may be in doubt. Glance switched to alembic for new migrations during the Ocata cycle and no longer needs to carry the old sqlalchemy-migrate-based migrations.
solution:: Remove the sqlalchemy-migrate-based migrations and related tooling.
impacts:: None. Users coming from a pre-Ocata deployment will need to deploy a pre-Xena glance before deploying Xena, but that’s standard practice anyway.

timeline:: Xena milestone 3
link:: https://review.opendev.org/c/openstack/glance/+/760411
assignee:: stephenfin

Spec Lite: Policy tests refactroing

Wed, 23 Jun 2021 00:00:00

project:: Glance
problem:: At the moment our unit and functional tests are referring to policy.yaml located in glance/tests/etc of the project repository. As now we are putting efforts into refactoring the policy layer in glance we should be testing our defaults except where we want to test something specifically different.
solution:: Instead of using policies from policy.yaml file we should be testing our defaults except where we want to test something specifically different.
impacts:: None
alternatives:: None
timeline:: Xena Milestone-2
reviewers:: Steap, rosmaita, pdeore
assignee:: abhishek-kekane, dansmith

Policy Layer Refactoring

Wed, 16 Jun 2021 00:00:00

https://blueprints.launchpad.net/glance/+spec/policy-refactor

During implementation of V2 Image API in glance an Onion layered architecture was introduced. The reason behind implementing the layered architecture was to avoid regression in any layer if either of the layer is modified. Glance has a separate layer for Policy injections which is closer to database rather than API. This spec will act as a Master specification for policy refactoring and have several spec-lite’s to explain respective implementation details.

Problem description

The current policy enforcement occurs in Policy layer. As such, it is conceptually tied to the objects implemented in the Glance architecture. A problem with this design, which has only revealed itself as the v2 API has matured, is that operators want to use policies to control who can make API calls (as they can with most other OpenStack services). In Glance, however, policies directly affect the objects dealt with internally by Glance, and only indirectly affect who can make API calls. This makes it difficult for operators to configure Glance.

In addition, while implementing Secure RBAC in glance we also noticed that certain API calls enforce multiple policies down the layer. For example in case of modify_image policy it enforces get_image policy which enforces get_image_location policy. This can be confusing for operators modifying the policy for modify_image and wondering why it hasn’t taken effect if the get_image policy or get_image_policy short-circuits the operation.

Proposed change

We need a better way of handling policies:

One of the major proposals is to move the actual policy enforcement up to the API layer so that an operator can, for example, easily restrict access to a particular call. Most of the OpenStack projects have policy enforcements closer to API layer, so these efforts will also put us more in-line with the current thinking of policy enforcement.
Make get_* policies be enforced only while showing particular resource rather than enforcing it for each API call. For example get_image policy should be enforced only for showing particular image to end user and not for other API calls such as update, delete, upload or download image.
Backward compatibility will be maintained while moving policies closer to API layer. (NOTE: RBAC related changes are not considered here.)
At the moment our unit and functional tests are referring to policy.yaml file from the test repo, instead our default policies should be used and overridden as and when required.
In order to test new policy changes with RBAC we need two different CI jobs which will run our tests with old policies as well as with the new RBAC flag enabled.

Note: Some of the above changes will have its own spec lite to further discuss the implementation details.

Alternatives

Keep it as it is and use hacks while implementing other scopes of Secure RBAC.

Data model impact

None

REST API impact

No changes to the REST API, but see “Other Deployer Impact” section, below.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Considering experimental support added for project scope realted to Secure RBAC, operators need to understand which policies to govern and how to configure them properly. Also, there’s likely to be some tweaking and testing of any custom policies during upgrade to Xena (or beyond).

Developer impact

Developers will have to be more aware of policies and where policy enforcement must happen.

Implementation

Assignee(s)

Primary assignee:: dansmith abhishek-kekane
Other contributors:: pdeore

Work Items

Move API specific policy checks up to the API layer
Enforce get_* policies at API layer only
Enforce resource specific policies close to database layer
Policy enforcement should be compatible with secure RBAC structure
Tests should run using default policies and not policy.yaml
New CI jobs to run tempest tests with and without Secure RBAC enabled

Dependencies

None

Testing

As explained in Work Items section our unit and functional tests need to use our default polices in code rather than policy.yaml file. We also need new CI jobs to run tempest tests with and without Secure RBAC enabled.

Documentation Impact

Policies are documented in code, so the documentation will be updated as the refactoring occurs.

References

Spec Lite: CLI support for new cache APIs

Fri, 04 Jun 2021 00:00:00

project:: python-glanceclient
problem:: In Xena we are deprecating glance-cache-manage CLI utility as we are moving caching operations under glance-api by introducing new API endpoints for cache related operations.
solution:: We need to add new CLI commands; cache-queue, cache-list, cache-delete, cache-delete-all to support new cache APIs.
impacts:: APIImpact, DocImpact
how:: We will add new CLI interface for cache-queue, cache-list, cache-delete, cache-delete-all commands. As caching is local to each glance node and most of the deployments configures glance nodes behind the load-balancers, operator/user need to pass actual endpoint of the glance node. Existing optional command line option –os-image-url will be used to provide the actual endpoint to the client. The default value for –os-image-url can also be set to using OS_IMAGE_URL environment variable. If this optional parameter is not provided while executing above new commands or is using default value set using OS_IMAGE_URL environment variable all of the above commands should exit with appropriate error message.
alternatives:: None
timeline:: Xena Milestone 2
link:: https://review.opendev.org/c/openstack/glance-specs/+/665258
reviewers:: dansmith, abhishek-kekane, cyril-roelandt
assignee:: jokke

Implement Unified Keystone Quotas in Glance

Mon, 26 Apr 2021 00:00:00

https://blueprints.launchpad.net/glance/+spec/glance-unified-quotas

Glance currently lacks per-tenant quota capabilities, which is something that other OpenStack projects have had for quite some time. Especially since Glance is a key project for a working cloud, and because it involves consumption of expensive resources, quotas are necessary for operators to properly restrict usage. As noted in the recent PTG, public clouds charge for every resource, and thus per-tenant limits may be less important. However, private clouds generally do not bill for usage, and instead rely on pre-paid quotas for resource sharing. This work aims to implement quotas for Glance.

Problem description

Glance does not have per-tenant quotas.

As an operator of a private cloud, I need to be able to define a quota for a tenant to avoid them consuming unlimited resources since I do not bill them for each one.
As an operator of a cloud, I have customers of differing sizes, and wish to use per-tenant quotas to provide “safety valve” limits on runaway usage. Having just one global value that applies to all users is not powerful enough to provide reasonable limits for tenants of varying size.

Proposed change

Keystone has implemented something called “unified limits,” which allows an operator to define (in Keystone) default and per-tenant quota limits. By default the “flat” model provides a very basic one-tenant-one-limit structure, but more complex hierarchies are possible. These can be set using Keystone’s client (openstackclient) utility to register, set, and update limits per tenant.

By utilizing this Keystone functionality, Glance does not need to define new APIs and client support to allow operators to set the limits, nor is it required to build a persistence model to store them, clean them up, etc. Instead, Glance can query the limit values from Keystone and compare them against current/proposed usage in order to determine if a particular resource consumption should be allowed. Effectively, Glance needs only enforce the quotas, but it does not need to store or manage them. Limits are fetched from Keystone automatically and per-request by the oslo_limit library, and thus quota changes take effect immediately.

Adding new quota limits is relatively easy using this model. While more can be added in the future, this spec covers the work required to add the following limits:

image_size_total: A limit on the storage consumed by all the non-deleted images for the tenant. Images with multiple locations will count as multiple usages. Specified in Mebibytes (MiB). This will be enforced in the image upload path, as well as the import path which are the APIs in which a user actually consumes more storage.
image_stage_total: A limit on the storage consumed by all the images in uploading, importing, and any other transient import-related states for the tenant. In order to capture space used by copying images, we will also include active images with a non-zero list of os_glance_importing_to_stores items. Specified in Mebibytes (MiB). This will be enforced in the image stage path.
image_count_total: A limit on the total number of non-deleted images owned by the tenant. Specified as a count of images. This will be enforced in the image create path.
image_count_uploading: A limit on the total number of images currently in either the uploading, saving, importing and any other transient states related to upload or import by the tenant. In order to capture images being copied, we will include active images with a non-zero list of os_glance_importing_to_stores items. Specified as a count of images. This will be enforced in the image upload and stage paths, and will provide a limit on the number of images that can be adding external data to the system at any given point.

Since the staging area for Glance API workers is likely to be far more constrained than the general-purpose image storage, those quotas are separated. An operator likely wants to give a user a large amount of image storage, which is backed by a sophisticated and distributed backend. However, the staging area is really “temporary space” which is used in the process of importing images, and is likely to be far more constrained (local disk on an API worker). By allowing a separate quota for the staging area, an operator gains the ability to provide many TiB of space for general image storage, while restricting users to a small number of import operations at a time. It helps to avoid a user staging a large amount of image data and then leaving it there for a long period of time. If staging quota is not desired, then setting it to -1 in Keystone signals to oslo_limit that it should be “unlimited.” By allowing for a separate count and size staging quota, the operator can either restrict the amount of image data, or the number of operations in progress.

For the storage-related limits, the initial revision of this will simply enforce the quota at the beginning of the relevant long-running storage-using operation. This has the benefit of being simple and predictable, but does make the quota enforcement “soft” in that a user must go over the limit before they are stopped. Existing global limits are enforced as “hard” thresholds in cases where the user provided a size ahead of time, and can interrupt a long-running network operation if breached. The benefit of this is that friendly users are not allowed to exceed their limit, although users that do not provide the size (such as when streaming an image) still require checking the limit after the operation is complete. The drawback of the current behavior is that if a user exceeds the 1TiB limit at 900GiB, they will have wasted time and network bandwidth, only to have the entire stream discarded. Since the global limits are intended to stop gross over usage and be large enough for any tenant, they are unlikely to be hit normally. The per-tenant ones described here are more likely to be hit regularly, and thus the soft behavior is chosen. Enforcing them as hard limits during or after a transfer could be done as a follow-on effort.

Honoring these limits will be conditional based on a configuration option. By default limit quotas will be disabled, which will most easily facilitate upgrades. When disabled, nothing will be different from how it is today. When enabled, Glance will fetch limits from Keystone, expecting the operator has registered them as prescribed, and enforce them against a user’s usage. I think ultimately it probably makes the most sense to make that enabled by default, and potentially eventually ultimately remove the knob entirely. Obviously, we can keep the knob indefinitely if that is more desirable.

Alternatives

One alternative is always to do nothing. Glance has not had quotas for this long, so we could just not add them and rely on the global resource limits, as they are today.

Another alternative is to implement quotas in Glance itself and not utilize the Keystone unified limits functionality. This would require new APIs, client support, and database models. It would also run contrary to the efforts in the wider community to move towards defining all quota limits in Keystone.

We could also decide that soft quotas are not good enough and take one of the following paths to get hard quotas:

Start requiring API clients to declare image sizes at upload time so that we can do the quota check accurately up front.
Refactor oslo-limit to expose the details of the limits, and use that in a refactored Image.set_data() wrapper to halt data transfer mid-stream when the limit is exceeded.

Either of these are candidates for future work that build on top of the work proposed in this spec.

Data model impact

There are no additional values that need to be stored if we do this. We already store the image size, which is what we need to count at enforcement time for the storage-based limits. Since Keystone stores the actual per-tenant quota limit values, we do not need to store those.

REST API impact

This will introduce some more ways to get HTTP 413 limit errors, but since they are already possible due to the existing limits, there is effectively no change.

A future effort should add a quota usage API to allow clients to check consumption against their limits.

Security impact

This should improve the overall security of the system, because resource limits can be custom-tailored to each tenant instead of relying only on global large-enough-for-anyone limits.

Notifications impact

None.

Other end user impact

No client changes are required. Actual end users may notice a behavioral change related to their operator being able to enforce a smaller quota on them than previously was possible (which is the goal of this effort).

Performance Impact

Querying Keystone for the limits for a tenant is not free, and will introduce some dependency and latency. However, this interaction (specifically with Keystone) is in the critical path for all API usage anyway. At least initially, a configuration option will be provided to enable this behavior, defaulting to disabled.

Other deployer impact

Operators will need to define the registered and actual limits for the Glance values in Keystone prior to enabling enforcement. This is done by registering them in keystone with default values using the openstack client. Quotas for individual users can be then set using a similar mechanism. All of this can be done prior to and after enabling enforcement on the glance side.

Operators upgrading from Wallaby or earlier need not do anything specific during the upgrade, as enforcement is off by default. If the operator wishes to utilize quotas after upgrading, they can register the quotas and enable glance enforcement in the config at any time.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: danms

Work Items

Introduce the base infrastructure for querying and checking limits from Keystone.
Add the image_size_total quota enforcement to upload and import.
Add the image_stage_total quota enforcement to stage.
Add the image_count_total quota enforcement to create.
Add the image_count_uploading quota enforcement to stage and upload.
Figure out what to do about ceph snapshots which show up as zero size.
Add tempest support for limits and tests for these quotas.
Implement configuring these quotas in devstack and integrate testing into a job.
Add operator docs to explain how to enable, configure and use these quotas.

Dependencies

This will require us to add oslo_limit as a dependency.
This will require devstack support to test
Tempest testing will be considered required for this, as the interaction across Keystone and Glance is key to ensuring correctness.

Testing

Unit and functional testing will be straightforward in the tree. Tempest testing will be provided, by setting a small limit on a tenant and then uploading a few images to ensure we run across the each limit.

Documentation Impact

Since per-tenant quotas in Glance do not exist yet, the docs will need to be updated to add coverage. Operators will need to know which limits to configure in Keystone, how to do that, as well as how to enable enforcing in Glance. Operators will also need to understand the soft-limit nature of the size-based quotas, and how Glance’s willingness to accept unbounded uploads impact quota enforcement.

References

Keystone unified limit docs: https://docs.openstack.org/keystone/latest/admin/unified-limits.html
Glance Xena PTG etherpad where this was raised again recently: https://etherpad.opendev.org/p/xena-glance-ptg
Gerrit topic for implementation and tests: https://review.opendev.org/q/topic:%22bp%252Fglance-unified-quotas%22+(status:open%20OR%20status:merged)

Xena Project Priorities

Thu, 04 Mar 2021 00:00:00

TODO(abhishekk): fill this in after the PTG

Distributed Image Import Support

Thu, 04 Feb 2021 00:00:00

https://blueprints.launchpad.net/glance/+spec/distributed-image-import

Glance is moving towards supporting rich operations on images, mostly during create time, via the import mechanism. This opens the door to things like metadata injection, format conversion, and copying between stores. Currently in order for this to work for what the user would consider the closest analog to image-upload (which is the glance-direct import method), the API nodes require access to shared storage which is a real blocker to adoption by deployers, and is the subject of this spec.

Problem description

Currently, when images are uploaded via the import mechanism, they are stored in a special area called “staging.” This is implemented under the covers as a glance_store but it must be a locally-accessible directory on the host filesystem. When using multiple API worker nodes (as any real deployment would), the staging directories of all worker nodes must be shared (i.e. mounted on a common NFS server) in order to support the glance-direct import method. This is obviously a problem for HA, performance, and a non-starter for any arrangement where some glance API workers are located in remote sites.

In order to get an image from zero to usable with a glance-direct import, there are multiple API requests that are required. One of these is the “staging” of the image data, which is followed by an “import” operation which moves the data from the staging area to its final destination(s). In a multi-node load-balanced scenario, the “stage” operation will almost definitely hit a different worker than the “import” operation, which will result in the latter not having access to the staged image data in its staging store, and thus a failure.

Proposed change

The goal of the work outlined by this spec, is to allow the API workers to keep their staging store directories local and un-shared, but still enabling the import operation to work. In order to do this, we will:

Record the URL by which the staging worker can be reached from the other workers in the database, and
Proxy the import request to the host that has it staged via that URL if the image is not local.
Any delete request while the image is staged also needs to be proxied, to ensure that the temporary file is deleted from the staging directory on the appropriate node.

With the above change, we can eliminate the need for shared storage between the API worker nodes, allowing them to be isolated from an HA point of view, as well as distributed geographically. It requires very little actual change, as the non-local recipient node simply proxies the request it receives to the node that has it staged and returns the result. Both the import and delete operations are quick and do not require a chained client -> proxy -> destination arrangement to persist for long periods of time.

Alternatives

One alternative is always to do nothing. We could continue to require shared storage for the staging area between the API nodes to support the import feature. We could also direct users to use image uploading instead of importing in cases where a shared directory is not feasible.

Another alternative would be to do effectively the same thing as described here, but over RabbitMQ or some other RPC mechanism. That has the disadvantage of needing additional supporting infrastructure that glance does not currently require today, as well as new code to handle sending and receiving those RPC calls and directing them to the appropriate internal actions.

Data model impact

In order to do this, we only need to store one new piece of information, and only for a short period of time. That is the direct URL of the API worker node that has staged an image. When the image is finally imported (which usually happens immediately after staging), that URL is no longer needed (nor relevant).

Initially, this implementation will use the reserved and quota-independent os_glance namespace to store the URL in the image’s extra_properties.

Later, when work is done to complete the usage of the staging directory as a proper glance store, we may be able to store the URL in the location metadata when the staging image data is registered there. When this happens and assuming there is an appropriate interface to use that location metadata, the plan will be to make this implementation use that metadata store instead.

REST API impact

None.

Security impact

The proxy behavior will be done with the user’s token, as presented to the worker that the load balancer selects. No additional authorization is added, and that token is used to make the request to the appropriate worker on the user’s behalf. Thus, this operation is entirely transparent from a security perspective.

Notifications impact

None.

Other end user impact

More users will be able to use the image import functionality after this is implemented as operators unwilling or unable to provide shared storage between their workers will no longer need to disable glance-direct import for their users.

Performance Impact

Eliminating the use of a shared NFS (or similar) storage location for the staging store should improve performance of upload and import, since the staging directory can be local. It also vastly reduces the need to move a potentially very large image back and forth over the network multiple times in the process of doing a single image import (reduces from a minimum of four round-trips of the image data to two).

Other deployer impact

Deployers may wish to enable image import after upgrading to a release that supports this, where previously they needed to disable the feature (or just glance-direct. They will need to configure each API worker with an additional element indicating the direct URL by which they can be reached, and ensure that API nodes are able to communicate with each other in this way.

Deployers that currently support import via shared storage may want to quiesce image activity while they split the workers from the shared storage location to local directories.

Deployers wishing to keep the shared storage for image staging may choose to do so with no impact or action required.

Deployers wishing to keep the import feature (or just the glance-direct method) disabled, may also do so with no impact or action required.

Developer impact

When we move to the location-based metadata approach detailed above, we will need to change the API from using the image extra_properties dict to passing that information through to the store routines. It is expected that this will be less than ten lines of code.

Implementation

Assignee(s)

Primary assignee:: danms

Work Items

Build a mechanism by which we can use the user’s authorization token to make an outbound call to another service
Add a configuration element allowing the operators to teach the API workers what their externally-visible URL is.
Make the API workers record their own URL on the image during the image stage operation.
Make the import and delete operations proxy to the appropriate URL when it is determined appropriate to do so.

Dependencies

Devstack needs support for starting additional glance workers in order to properly test this.
Tempest needs support for looking up alternative image services in the service catalog.

Testing

Unit tests for the API behaviors and import tasks are sufficient, as the changes are minimal.

Functional tests for the image proxying.

A set of tempest tests that stage and import/delete images on different glance workers with separate staging directories will be written to ensure CI coverage for this behavior in a realistic sense.

Documentation Impact

Since this just makes something work that did not before, no large amount of documentation will need to be written. As mentioned above, deployers will have one new config option to set on API nodes as well as network and firewall considerations to address in order for this to work, which will be covered in the documentation.

References

Much discussion on this was done on another spec:

https://review.opendev.org/c/openstack/glance-specs/+/763574

The code implementation for this also has discussion relevant to the topic:

https://review.opendev.org/c/openstack/glance/+/769976

This was discussed at the Wallaby PTG in the glance sessions, under the topic of “Cluster Awareness”:

https://etherpad.opendev.org/p/glance-wallaby-ptg

This has been discussed in multiple glance meetings:

New Image API /v2/images/{id}/tasks for task information

Mon, 23 Nov 2020 00:00:00

https://blueprints.launchpad.net/glance/+spec/messages-api

The image service now supports importing images or copying existing images in multiple glance stores using glance tasks, but there is restriction that only one import operation or copy operation is permitted simultaneously on an image to avoid race condition.

Problem description

Task APIs are not available to normal users. In Victoria we have fixed race condition while copying existing images in multiple stores which uses “task-id” to obtain lock on the image to prevent other import operations on the same image. The additional image property “os_glance_import_task” will be used to store “task id”. We are updating ‘message’ property of the task which helps calculate time based on last updated time of task to burst the lock as well as show how much data has been copied of that image. Copying operation may take a long time and normal user can be clueless about when lock will be busted or why his (other) requests are rejected without contacting to the administrator.

Proposed chage

Add new fields image_id, request_id and user fields to tasks database and new API /v2/images/{id}/tasks which will fetch the tasks associated with image_id of that image and returns it to the user. This new API will return all tasks associated with image which are not expired. This expiration time is calculated when task reaches a final state of success or failure using task_time_to_live configuration parameter defined in glance-api.conf file. If active task is not present for given image then it will return empty list to user.

The request-id will effectively help user to find out what happened with his request, why his request has been denied and which task is currently being performed on the image.

The user field is an alternate to request_id field. In general client tools can access/generate request-ids but possibility normal users don’t have access to the request-ids, in this case the user field will help them to identify their particular task and its status. So from now on the tasks will have either request-id or user or both associated with it.

More details on this API can be found in the REST API section of this spec.

Alternatives

Add new API endpoint /v2/messages/{task_id} which will return related task information to the user. In this case user have to know task_id in advance. To know the task_id in advance he needs to call GET API of image to figure out whether os_glance_import_task property is set on the image or not.

Another alternative is to expose task show API to all the users. At the moment task API’s are managed with two different policies; “tasks_api_access” and then crud level policies such as “add_task”, “get_tasks” etc. So in this case we first need to expose tasks_api_access to all users (than admin) and then need to expose individual level policies to end user. This might be confusing and need to document carefully otherwise default access might be provided to all task API’s by mistake.

Data model impact

This spec proposes to add image_id, request-id and user fields to tasks database table. Those will be null and does not require any migration script to add this information to existing records.

REST API impact

New API

Show tasks associated with the given image, for example, information about all active (not expired) tasks associated with the image.

Common Response Codes

Not Found: 404 Not Found with details.

API Version

All URLS will be under the v2 Glance API. If it is not explicitly specified assume /v2/<url>

[New API] Get tasks associated with image

Show tasks associated with given image:

GET /v2/images/{image_id}/tasks

This API takes no query parameters and when authorized returns tasks associated with given image. If it does not found any active task associated with the image then it will return empty list to the user. Example of the valid response:

{
    "tasks": [
        {
            "task": {
                "id": "ee22890e-8948-4ea6-9668-831f973c84f5",
                "image_id": "dddddddd-dddd-dddd-dddd-dddddddddddd",
                "request-id": "rrrrrrr-rrrr-rrrr-rrrr-rrrrrrrrrrrr",
                "user": "uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu",
                "type": "api_image_import",
                "status": "processing",
                "owner": "64f0efc9955145aeb06f297a8a6fe402",
                "expires_at": null,
                "created_at": "2020-12-18T05:20:38.000000",
                "updated_at": "2020-12-18T05:25:39.000000",
                "deleted_at": null,
                "deleted": false,
                "input": {
                    "image_id": "829c729b-ebc4-4cc7-a164-6f43f1149b17",
                    "import_req": {
                        "method": {"name": "copy-image"},
                        "all_stores": true,
                        "all_stores_must_succeed": false
                    }
                    "backend": [
                        "fast",
                        "cheap",
                        "slow",
                        "reliable",
                        "common"
                    ]
                },
                "result": null,
                "message": "Copied 15 MiB"
            },
        }
    ]
}

Response codes:

200 – Upon authorization and successful request. The response body contains the JSON payload with the known stores.

Example curl usage:

curl -g -i -X GET -H "X-Auth-Token: $token"
    -H "Content-Type: application/octet-stream"
    $image_url/v2/images/{image_id}/tasks

Security impact

None

Notifications impact

None

Other end user impact

This proposal introduces a few other user impacts worth noting.

Glance client Ideally the glance client (CLI + REST client) should be updated in accordance with this spec. Notably:

CLI / API support for get task information from image.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: abhishek-kekane
Other contributors:: None

Work Items

Implementation tasks may consist of:

Add expand script for adding new fields to tasks database
Modify tasks data layer to use newly added fields
Modify tasks CRUD operations to use newly added fields
Add support for new API.
Add python-glanceclient support
Add API documentation for new API

Dependencies

None

Testing

Need to add new unit tests for coverage

Documentation Impact

As mentioned in the ‘work items’ section, we’ll need to ensure the glance docs are updated for:

The new get tasks from image REST API.
Overall glance multi-store documentation to educate deployers on the feature and how/when it’s used.

References

PoC - https://review.opendev.org/c/openstack/glance/+/763739

Wallaby Project Priorities

Mon, 14 Sep 2020 00:00:00

TODO(abhishekk): fill this in after the PTG

Calculate virtual size of image

Wed, 15 Jul 2020 00:00:00

https://blueprints.launchpad.net/glance/+spec/calculate-virtual-size

‘virtual_size’ should be set on image to avoid running qemu_img operations on consumer nodes.

Problem description

Image has virtual_size field in images table but it is never set unless we use taskflow introspect to create image. Consumers of glance like nova or cinder never consume image before calculating virtual size, which means every time they call for image, they need to perform qemu_img info call to calculate the virtual size of the image.

Use Case:

‘virtual_size’ should be set on image to avoid running qemu_img operations on consumer nodes.

Proposed change

All supported sparse disk image formats (i.e. what glance calls disk_format) simulate a larger virtual disk than the actual data that is stored, and record that virtual size in metadata. We propose to add a handler for each of the formats that can examine the chunks while streaming the image to extract the relevant metadata to determine the virtual disk size.

In glance, users can create images using two ways. 1) Create image API 2) Import image API

With this change virtual size will be set to image even user tries to use any of above two methods to create the image. Also images created by nova (nova-snapshot except ‘direct-to-backend snapshot’) or cinder (volume-upload-to-image) will be able to set the virtual size to image as well.

NOTE: To start with we are going to calculate virtual size for images which will have container-format bare or other uncompressed formats only. Later as and when required or expected we will enhance it for other container-formats.

We will be performing this operation during uploading image data to glance store, so for image import this will happen during the actual import phase and not the stage phase.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: danms
Other contributors:: None

Reviewers

Core reviewer(s):: abhishek-kekane jokke rosmaita
Other reviewer(s):: whoami-rajat

Work Items

Add handler for each of the disk-format
Add wrapper around chunked reader to calculate virtual size
Add unit tests for coverage
Add functional tests

Dependencies

None

Testing

Tempest test to verify that virtual size is set on image using glance, nova snapshot and cinder upload-to-image operations.

Documentation Impact

Please refer to ‘Other deployer impact’

References

https://review.opendev.org/#/c/744234

Allow copying unonwned images by policy

Thu, 02 Jul 2020 00:00:00

https://blueprints.launchpad.net/glance/+spec/copy-unowned-image

Currently Glance has a mechanism which allows a user to copy images between two (or more) backend stores. That only works for images that they themselves own, which covers many of the use-cases for the feature. However, considering different types of clouds, admins may want users to be able to copy images between stores that they do not own for the purposes of performance optimization and storage efficiency.

Problem description

Currently a user may only copy images that they own, and there is no way to allow otherwise. This may lead to anti-pattern behavior, such as downloading an image and re-uploading it so that it can be copied. Further, it may prevent optimizations, such as a user pre-copying a public image to a store for an edge site to improve boot time, and CoW re-use.

Specifically, Nova is working on a feature to automate storage efficiency improvements, which rely on this ability in many cases. Glance has supported multiple backend stores for some time, but Nova (specifically the ceph/rbd image backend) can only cooperate with Glance on a single ceph cluster. If multiple clusters are involved, the worst-case scenario is that Nova ends up downloading the image from Glance via HTTP and then uploads it to the ceph backing store as a private-to-the-instance base image. This results in a loss of fast-snapshot capability as well as massive storage inefficiency. See Nova spec for glance multistore capability.

Proposed change

Allow the copy-image import method to check policy for permission instead of hard-coding “is admin or owner” behavior. If granted by policy, use an admin context to kick off the actual import task on behalf of the requesting user.

Because the “is admin or owner” behavior is baked deep into many layers in Glance, enabling this requires some substantial work. As a part of this process, the import task will coalesce all image operations that may be delegated to a non-owner into a single location to aid in auditing. If and only if the policy grants this permission (which is only possible for copy-image) then an admin-capable ImageRepo will be provided to the task for use in image property manipulation.

When the copy-image operation is performed by a user other than the owner through this mechanism, it should be clear that the ownership of the image (and any new resources created in the process) remains unchanged.

Alternatives

One alternative is to just say “sorry, only owned images may be copied.”
Another alternative is to have Nova store Glance admin credentials, and effectively implement the policy check on their side, using an admin user to copy the image if and when necessary.

Data model impact

None.

REST API impact

No new APIs or return codes are added. There is a small implied behavioral change in the form of users potentially being able to copy images they could not before.

Security impact

As with any policy change, there is a potential for unintentional granting of elevated permission to some users. Assuming the code is correct, this is no different from other policy knobs. The default policy for this operation will remain as “is admin or owner” so no actual change will happen unless the operator so elects.

There is also a potential impact with the use of an admin context for the operation which has more power to inflict damange or leak information than there would be otherwise. This effect is mitigated by the following design points:

The admin context is not passed to the import task, but rather used only to grab an admin-capable ImageRepo which is given to the task.
All of the related actions done as admin on behalf of the user are confined to a single “actions” object for easy auditing.
This is only (currently) applied to the copy-image import method, which means that (ideally) the maximum privilege escalation would be copying an image that the user can already see to a new backend store.

Notifications impact

None.

Other end user impact

None.

Performance Impact

No impact on Glance, but there is a net positive impact to the overall system in the case where Nova is able to use this feature to co-locate an image in the backend of a given remote site.

Other deployer impact

None.

Developer impact

There are a number of changes to layers in the onion between the API and the task itself. This is mostly around the optional admin_repo argument that may be passed through gateway, proxy, and down to the task.

Implementation

Assignee(s)

Primary assignee:: danms

Work Items

Add tests that validate the current behavior which provide a measurement of the change as the feature takes shape.
Refactor the task creation code to allow passing an optional admin_repo from the API to the task.
Add a context.elevated() helper, like other projects have, which keeps the user’s context mostly whole but with admin privileges.
Refactor the api_image_import module to consolidate all image operations that will need to be delegated to the admin_repo if it is passed from the API.
Add a policy action for copy_image which is the primary knob by which an admin can enable this feature.
Update documentation for the copy-image method of image_import to explain the expectations of the admin, owner, and user delegate when non-owners are allowed to do this.

Dependencies

None.

Testing

New functional and unit tests will be added before the actual changes are made to validate current behavior. Each patch along the way will modify or augment those to test the new behavior and ensure there are no regressions.
Nova will ultimately gain a test job that deploys multiple stores, including one rbd-backed store, and will initiate a copy-image import on behalf of a tempest ephemeral tenant that does not own the public cirros image.

Documentation Impact

Documentation is likely needed around the policy knob, explaining what it does, how to use it, and why an admin may want to use it.

References

Make cinder driver compatible with multiple stores

Thu, 28 May 2020 00:00:00

https://blueprints.launchpad.net/glance_store/+spec/multiple-cinder-backend-support

From Train onwards Glance is fully supporting configuring multiple glance stores. Glance can configure to use cinder as a store using available cinder driver of glance_store. Cinder makes available volume-types, which describe the characteristics of volumes. Currently, the cinder glance_store can only use one volume-type. The point of glance multi-store implemented in Train is to give operators the ability to expose glance stores of differing characteristics to end users. Even though all images will wind up in a single cinder installation, it is possible for operators to expose different categories of storage in cinder by creating different volume-types. So when a cinder installation exposes multiple volume-types of differing characteristics, what we want to do here is to be able to map different glance ‘stores’ to cinder volume-types.

Problem description

1. As of now cinder can configure different volume-types but glance will be able to use only one of those available volume-types. If glance store is set to use cinder then every time new image is created it will always be uploaded to default volume-type unless operator has configured cinder_volume_type in glance-api.conf file.

2. If cinder store is configured to use in glance then while creating the image cinder store of glance creates location URL with cinder:// prefix only. When cinder has configured multiple backends and glance has also configured multiple cinder stores then it is difficult to analyse the image is stored in which of the cinder store as the location url for the image as cinder://<image_id>.

Proposed change

We propose that Glance be able to expose multiple cinder stores that differ by what volume-type each store uses. These would be defined in the normal way in the glance configuration file using the enabled_backends option (see example below). Further, when using multiple glance stores, each cinder store must have the cinder_volume_type option set.

While initializing the store object, glance will validate that volume type set using cinder_volume_type is exist in cinder. If it’s not then that store will be excluded by disabling ‘add’ and ‘delete’ operations. To connect to cinder from glance operator needs to specify cinder_store_auth_address, cinder_store_user_name, cinder_store_password and cinder_catalog_info for each of the store in glance-api.conf file.

Below are some multiple cinder store configuration examples.

Example 1: Fresh deployment

For example, if cinder has configured 2 volume types glance-fast and glance-slow then glance configuration should look like;:

[DEFAULT]
# list of enabled stores identified by their property group name
enabled_backends = fast:cinder,slow:cinder

# the default store, if not set glance-api service will not start
[glance_store]
default_backend = fast

# conf props for fast store instance
[fast]
rootwrap_config = /etc/glance/rootwrap.conf
cinder_volume_type = glance-fast
description = Really fast and expensive storage
cinder_catalog_info = volumev2::publicURL
cinder_store_auth_address = http://localhost/identity/v3
cinder_store_user_name = glance
cinder_store_password = admin
cinder_store_project_name = service
# etc..

# conf props for slow store instance
[slow]
rootwrap_config = /etc/glance/rootwrap.conf
cinder_volume_type = glance-slow
description = Slower but less expensive storage
cinder_catalog_info = volumev2::publicURL
cinder_store_auth_address = http://localhost/identity/v3
cinder_store_user_name = glance
cinder_store_password = admin
cinder_store_project_name = service
# etc..

Example 2: Upgrade from single cinder store to multiple cinder stores, if default_volume_type is set in cinder.conf and cinder_volume_type is also set in glance-api.conf then administrator needs to create one store in glance where cinder_volume_type same as old glance configuration:

# cinder.conf
The glance administrator has to find out what the default volume-type is
in the cinder installation, so he/she needs to discuss with either cinder
admin or cloud admin to identify default volume-type from cinder and then
explicitly configure that as the value of ``cinder_volume_type``.

Example config before upgrade:

[glance_store]
stores = cinder, file, http
default_store = cinder
cinder_state_transition_timeout = 300
rootwrap_config = /etc/glance/rootwrap.conf
cinder_catalog_info = volumev2::publicURL
cinder_volume_type = glance-old

Example config after upgrade:

[DEFAULT]
enabled_backends = old:cinder, new:cinder

[glance_store]
default_backend = new

[new]
rootwrap_config = /etc/glance/rootwrap.conf
cinder_volume_type = glance-new
description = Newly defined second (cinder) store
cinder_catalog_info = volumev2::publicURL
cinder_store_auth_address = http://localhost/identity/v3
cinder_store_user_name = glance
cinder_store_password = admin
cinder_store_project_name = service
# etc..

[old]
rootwrap_config = /etc/glance/rootwrap.conf
cinder_volume_type = glance-old # as per old cinder.conf
description = Previously existing (cinder) store
cinder_catalog_info = volumev2::publicURL
cinder_store_auth_address = http://localhost/identity/v3
cinder_store_user_name = glance
cinder_store_password = admin
cinder_store_project_name = service
# etc..

Operator can decide on the basis of deployment strategy which volume type they wants to use by coordinating with cinder admin or cloud operator.

We also propose to modify location url for cinder and use store identifier in location url so that user or operator will identify in which cinder store of glance image is stored. The new location URL should looked like cinder://store-name/image-id.

For legacy images stored in cinder backend we will modify the lazy loading mechanism of glance which will update the location URL of existing images as per new format. The lazy loading operation is a check before GET API call which traverse through image location and based on location URI it identifies in which glance store image data is stored and updates that information in location metadata. This mechanism is also useful in a way that if in future operator decides to change the name of the glance store or retire one of the configured store by migrating the images to new stores.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

The security impact is same as it was with single store but we’re just pointing it out here; The image-volume is stored in the configured project cinder_store_project_name and can be accessed with configured user cinder_store_user_name.

There could be a potential risk if someone was able to get a hold of these credentials and access the image-volumes. Worst case is someone could alter the image-volumes if they had permission to perform any cinder operation on it such as retype, attach etc.

Care will have to be taken to ensure it isn’t accessible by normal users.

Notifications impact

None

Other end user impact

None

Performance Impact

After upgrade from single cinder store to use multiple cinder stores first image-list or first get call for image will take additional time as we are performing the lazy loading operation to update legacy image location url to use new image location urls. Subsequent get or list calls will perform as they were performing earlier.

Other deployer impact

Operators should be aware of different volume types available in cinder. They can either use type-list command of cinder client or coordinate with cinder admin and decide which volume-type of cinder should be configured in glance-api.conf.

Developer impact

None

Implementation

Assignee(s)

Primary assignee: * whoami-rajat * abhishek-kekane

Other contributors:: None

Reviewers

Core reviewer(s):

jokke
rosmaita
smcginnis

Work Items

Modify cinder driver initialization to set new cinder location url
Modify usage of cinder location url
Modify lazy loading mechanism to update legacy image location URLs
Unit tests

Dependencies

None

Testing

Appropriate unit and functional tests to ensure the changes to glance function correctly. Aslo we could add a job which will run tests using cinder stores in glance.

Documentation Impact

We’ll need to ensure that glance/glance_store docs are updated for:

Usage of cinder volume types as cinder stores of glance.
We should also document that, if cinder store is used as glance backend, Only the Image Service API should be used to manipulate images. Manipulating image data directly via the Block Storage Service API is not supported and may lead to adverse consequences, including data loss.
How to upgrade from single cinder store to multiple cinder stores.

References

None

Victoria Project Priorities

Mon, 06 Apr 2020 00:00:00

TODO(abhishekk): fill this in after the PTG

Spec Lite: Deprecate admin_role

Tue, 24 Mar 2020 00:00:00

project:: glance
problem:: Glance has a configuration option that grants complete admin access to anyone with a particular role. This is confusing because it overrides any settings in the policy configuration file. Further, the default value is ‘admin’, which is likely to be an actual role defined in any OpenStack cloud.
solution:: Deprecate the ‘admin_role’ configuration option in Ussuri and remove it during the Victoria development cycle. Additionally, change the default setting to something that would never match any actual role, for example, ‘__NOT_A_ROLE_07697c71e6174332989d3d5f2a7d2e7c_NOT_A_ROLE__’. That way, the ‘admin_role’ would only be effective if an operator configured it on purpose, and this “backdoor” will be effectively closed immediately.
impacts:: Possibly documentation (though our policy docs are woefully out of date).

Spec Lite: Deprecate Checksum Computation

Wed, 19 Feb 2020 00:00:00

project:: glance
problem:: The glance ‘checksum’ image property contains an MD5 hash of image data. MD5 has not been considered secure for some time, and in order to comply with various security standards, an implementation of the MD5 algorithm may not be available on glance nodes.
solution:: Announce that Glance will no longer populate the ‘checksum’ on new images beginning with the Victoria release. Instead, operators should rely on the secure “multihash” feature that was introduced in Rocky. The ‘checksum’ property will remain on legacy images.
impacts:: None.

how:

In Ussuri: release note. In Victoria: Remove the code that uses MD5. (This will affect primarily the glance_store drivers.)

alternatives:

We could check to see if the algorithm is available, and if it is, compute the MD5. But this seems pointless as the secure multihash is already being computed for all new images.

We could remove the ‘checksum’ entirely, but this would require a migration to the multihash. For at least some backends, this would mean downloading the image data for each legacy image to do the computation, which could take a very long time.

timeline:

Deprecation and release note in Ussuri; removal in Victoria.

Spec Lite: Deprecate allow_additional_image_properties

Mon, 16 Dec 2019 00:00:00

project:: glance
problem:: The allow_additional_image_properties configuration option was originally introduced to prevent users from flooding the image_properties table with junk. Given that we now have the image_property_quota option, allow_additional_image_properties is unnecessary. Further, if an operator were to set it False, all sorts of stuff would break as multiple OpenStack services (for example, Cinder, Nova) currently write/rely on custom image properties. Of course, in such a case, an operator would have to change the option to True to get everything working again; but the point is that if this setting is always True, then it is unnecessary and polluting the code with unnecessary branch points.
solution:: Deprecate the allow_additional_image_properties in Ussuri scheduled for removal in the V development cycle, consistent with the standard OpenStack deprecation policy.
impacts:: None. Operators who really want to use this option can instead set image_property_quota to 0. (‘0’ means zero; a negative value means ‘unlimited’ for this option.)
assignee:: rosmaita (or anyone who would like to address this low-hanging fruit)

Deleting image from single store

Mon, 09 Dec 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/glance/+spec/delete-from-store

New API feature to remove image from single store instead of whacking the whole image.

Problem description

Currently only way to remove image from single store is by exposing known problematic locations API and utilize that to remove the location. With multiple-stores support there is definitely more user oriented use-cases for removing image from specific store.

Proposed change

Introduce new “/v2/stores” endpoint to Images API v2 to provide safe way to delete images from the specific store.

New API call: ‘DELETE /v2/stores/<StoreID>/<ImageID>’

Alternatives

We could consider utilizing the current “v2/images/<ImageID>” endpoint and append the store ID at the end of that. The risk with this approach is that it’s way too easy for the API user to make a mistake dropping the StoreID and accidentally delete the whole image instead of just removing it from single store.

Data model impact

None

REST API impact

New API endpoint “v2/stores/<StoreID>/<ImageID>” that accepts only DELETE http method.

The request will fail if this is the only location indicating that the user should delete the image instead.

Security impact

This change does not have any know security impacts.

Notifications impact

Notification of image being removed from the store can be considered.

Other end user impact

python-glanceclient will have feature to support this API call.

Performance Impact

This feature has no know performance impacts.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: jokke

Work Items

API change for glance-api service
Testing for the new feature
python-glanceclient support
Documentation needs to be updated including the new workflow

Dependencies

None

Testing

The change will need unit and functional tests.

Documentation Impact

None

References

None

Ussuri Project Priorities

Tue, 26 Nov 2019 00:00:00

The Ussuri review priorities were discussed at the Shanghai PTG. The preliminary list was maintained on the Shanghai PTG etherpad.

This list is an estimate of what the Glance project team can accomplish during the Ussuri cycle based on our developers’ estimates of how much time they can commit to Glance.

The following list is roughly indicates milestonewise priorities.

Priority Item	Owner(s)	Spec(s)	Target release milestone
Remove native ssl support	Erno Kuvaja	None	M1
Import image in multiple stores	Grégoire Unbekandt	multiple image import	M2
Copy existing image in multiple stores	Abhishek Kekane	copy image in multiple stores	M2
S3 driver for glance_store	Naohiro Sameshima	s3 driver	M2
remove sheepdog driver from glance_store	Abhishek Kekane	None	M2
Permanent solution for Subunit parser error	Abhishek Kekane	None	M2
Nova - snapshot/backup upload to multiple stores	Abhishek Kekane	nova snapshot	M2
Cinder - volume upload to multiple stores	Abhishek Kekane	cinder uploadtoimage	M2
Cluster awareness of glance API nodes	Erno Kuvaja	cluster awareness	M2
remove registry code from glance	Abhishek Kekane	None	M2
Delete image from single store	Erno Kuvaja	delete store from image	M2
image-import.conf parsing issue with uwsgi	Unassigned	None	M2
Multiple cinder store support in glance_store	Abhishek Kekane	cinder glance_store	M3
Image encryption	Josephine Seifert	image encryption	M3
Tempest work	Abhishek Kekane	None	M3

Important dates

This is an abbreviated list focused on dates relevant to Glance. See Ussuri Release Schedule for the complete list for OpenStack.

R-22	Dec 13	U-1 milestone
R-13	Feb 14	U-2 milestone
R-6	April 03	glance_store Ussuri release (final release for non-client libraries)
R-5	April 22	U-3 milestone Glance feature freeze python-glanceclient Ussuri release (final release for client libraries);
R-3	April 24	RC-1 release and string freeze
R-1	May 08	final RCs
R-0	May 15	Ussuri release

Copy existing image in multiple stores

Mon, 18 Nov 2019 00:00:00

https://blueprints.launchpad.net/glance/+spec/copy-existing-image

Despite the Image service supports several back ends for storing virtual machine image, there is no way at the moment to copy existing image bits into multiple stores and avoid the operator to manually copy data and update image locations.

Problem description

At the moment, if cloud provider decided to upgrade their cloud to Train release to use the ability of glance of configuring multiple stores, then there is no way to replicate/copy existing image bits to newly added stores. As a result, operators today need to perform a number of manual steps in order to replicate/copy image bits on glance stores despite using the ‘enabled_backends’ configuration option.

Note

Example

An operator upgrades his existing setup of OpenStack cloud to configure different sites, each with their local glance stores which nova hosts are accessing directly (Ceph).

This operator use multiple stores support and want its images to be available in each store to prevent the images to be downloaded through glance each time a new virtual machine is created and let Nova use COW.

For this purpose, he need to manually copy existing images data in store2 to storeN and register these others locations URL using the glance API.

Since glance supports multi stores, it should propose a feature to copy existing image data into these stores at once to facilitate operators’ work.

Proposed change

This spec depends on Ability to import an image in multiple stores [1] to copy the image in multiple stores. In addition to above changes and existing import methods glance-direct and web-download, this spec proposes to introduce additional import method copy-image.

If the image from which data needs to be copied in multiple stores is not in active state then the API should reject the request. If image exists then it will be copied to staging area. Once copying to staging task is completed, import task will import the data to all the specified stores. In case of any failure during copying existing data to staging area or successful completion of copying/importing data to specified store, image data from the staging area will be removed.

Introduce one additional Task which will be internal plugin and allow us to copy the existing image to staging area. To copy the existing image into staging area, we will first give preference to default_backend of that specific glance-api node. If image is not associated with default_backend then we will iterate through all the available backends of that specific glance-api node to copy(download) image from that location to staging area. Failure mechanism will be same as dependent spec Ability to import an image in multiple stores [1].

If ‘all_stores’ boolean field is specified, the API should reject the request.
If an unavailable image is specified, the Api should reject the request.
If an unavailable store is specified, the Api should reject the request.
If image is already present at specified location, the Api should reject the request.
If base image is deleted while copying the data to new stores, then the copying process will be terminated and copied data will be removed as well.
If ‘all_stores_must_succeed’ is set to ‘true’ (default behavior) and an error occurs during the upload in at least one store, the request should be rejected, the data will be deleted from stores where copying is done (not staging), and the state of the image remains the same.
If ‘all_stores_must_succeed’ is set to ‘false’, the request will fail (data deleted from stores, …) only if the upload fails on all stores specified by the user. In case of a partial success, the locations added to the image will be the stores where the data has been correctly uploaded.
In case of failures or successful completion of copying/importing data to specified stores, image data from staging will be removed.

Users will be able to follow copying/importing operation progress by looking at 2 reserved image custom properties:

os_glance_importing_to_stores: This property contains the list of stores that has not yet been processed. At the beginning of the copy/import flow, it will be filled with the stores provided in the request. Each time a store is fully handled, it will be removed from the list. This property will be emptied at the end of the process even if an error occurs and there are still remaining stores.
os_glance_failed_import: Each time an import in a store fails, it is added to this list. This property is emptied at the beginning of the copy/import flow.

Image locations will be updated each time an upload/copy in a store succeed (with information from this store).

Current location strategies modules should not be affected by theses changes as we don’t change the behavior of the image locations. It is currently possible to do the same by patching an image and specify a list of locations.

In the same vein, as it is already an option to choose a store when using import workflow, there is no need to add a new policy to restrict the copying of image in multiple stores.

Alternatives

Continue copying images manually to different stores and update the locations using Locations API.

Data model impact

None

REST API impact

This spec proposes the following API changes:

Modified APIs

Import an image.

Common Response Codes

Normal http response code: 202
- 202: Accepted
Expected error http response codes: 400, 401, 404, 409, 410
- 400: Bad Request with details.
- 401: Unauthorized
- 404: Not Found (image doesn’t exist or is not owned by the caller)
- 409: Conflict (image is not in appropriate status)
- 410: Gone (Image deleted while operation in progress)

API Version

This change will require minor version bump. All URLS will be under the v2 Glance API. If it is not explicitly specified assume /v2/<url>

[Modified API] Import image to the store

Import image to the store:

  POST /v2/images/{image_id}/import

If present, copy this image in multiple stores specified using `stores`
option.

Example curl usage:

curl -i -X POST -H "X-Auth-Token: $token"
     -H "Content-Type: application/json"
     -d '{"method":{"name":"copy-image"},
          "stores": ["ceph1", "ceph2"],
          "allow_failure": false}'
     $image_url/v2/images/{image_id}/import

Security impact

None

Notifications impact

Notification will be sent for each of the successful copy of image.

Other end user impact

None

Performance Impact

As we’ll write data in multiple stores, this will increase the IO from the glance nodes in accordance of the number of stores specified. From the user point of view, the import workflow will also take more time depending on the stores where the copying is done.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

abhishekk

Reviewers

Core reviewer(s):

jokke
rosmaita

Work Items

Implementation tasks may consist of:

Write new internal plugin to copy image data in staging area

Dependencies

None

Testing

Appropriate unit and functional tests to ensure the changes to glance function correctly. The major testing item is to ensure that if the copy taskflow fails, data will be deleted only from the new stores, not from the stores where image is already in and image status does not change.

Documentation Impact

We’ll need to ensure the glance docs are updated for:

New body field for image import.
New import method for image import.

References

Re-support S3 Driver as Glance Store Backend

Wed, 09 Oct 2019 00:00:00

https://blueprints.launchpad.net/glance/+spec/re-support-s3-driver

Glance store had supported S3 backend until version Mitaka, and it has already been removed (in Newton [1]) due to lack of maintainers. However, there seems to exist a certain number of operators who want to use S3 for the glance backend because S3 is a major storage and there is a lot of S3 compatible storage. So, I propose revive the S3 driver and support again as a member of glance store drivers [2].

Problem description

There seems to exist a certain number of operators who want to use S3 or S3 compatible storage for the glance backend, but glance doesn’t support S3 store anymore so there is no way to use the S3 backend for them.

Proposed change

Revive the S3 Store Driver that was used until Mitaka, and support multiple store configuration [3].

The following configurations would be added to glance_store section of glance-api.conf::

* s3_store_host
  The host where the S3 server is listening.
  This item accepts a string value.
  (e.g. s3-region.amazonaws.com, http://s3-region.amazonaws.com, https://s3-region.amazonaws.com or
  my-object-storage.com, http://my-object-storage.com, https://my-object-storage.com)

* s3_store_access_key
  The S3 query token access key.
  This item accepts a string value.

* s3_store_secret_key
  The S3 query token secret key.
  This item accepts a string value.

* s3_store_bucket
  The S3 bucket to be used to store the Glance data.
  This item accepts a string value, and if s3_store_create_bucket_on_put is
  set to true, it will be created automatically even if the bucket does not
  exist.
  It is desirable to have a DNS-compliant naming convention.

* s3_store_create_bucket_on_put
  A boolean to determine if the S3 bucket should be created on upload if it
  does not exist or if an error should be returned to the user.
  This item accepts True or False.

* s3_store_bucket_url_format
  The S3 calling format used to determine the bucket. Either 'auto' or 'path'
  or 'virtual' can be used.
  In 'path'-style, the endpoint for the object looks like 'https://s3.amazonaws.com/bucket/example.img' or 'https://my-object-storage.com/bucket/example.img'.
  And in 'virtual'-style, the endpoint for the object looks like 'https://bucket.s3.amazonaws.com/example.img' or 'https://bucket.my-object-storage.com/example.img'.
  If you do not follow the DNS naming convention (e.g. that includes '.' in
  the bucket name), you can get objects in the path style, but not in the
  virtual style.

* s3_store_large_object_size
  What size, in MB, should S3 start chunking image files and do a multipart
  upload in S3.
  This item accepts a positive integer value.

* s3_store_large_object_chunk_size
  What multipart upload part size, in MB, should S3 use when uploading parts.
  The size must be greater than or equal to 5M.
  Note that the maximum possible number of image divisions is 10,000.

* s3_store_thread_pools
  The number of thread pools to perform a multipart upload in S3.
  This item accepts a positive integer value.

As a result, operator can configure multiple stores including S3 as shown below::

[DEFAULT]
# list of enabled stores identified by their property group name
enabled_backends = fast:s3, cheap:s3, reliable:file

# the default store, if not set glance-api service will not start
default_backend = fast

# conf props for file system store instance
[reliable]
filesystem_store_datadir = /var/lib/images/data/
description = Reliable filesystem store
# etc..

# conf props for s3 store instance
[fast]
s3_store_host = https://s3.amazonaws.com
s3_store_access_key = access-key-for-fast
s3_store_secret_key = secret-key-for-fast
s3_store_bucket = bucket-for-fast
# etc..

# conf props for s3 store instance
[cheap]
s3_store_host = https://my-object-storage.com
s3_store_access_key = access-key-for-cheap
s3_store_secret_key = secret-key-for-cheap
s3_store_bucket = bucket-for-cheap
# etc..

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

This change will have to be explicitly configured in the store options.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: nao-shark

Work Items

Add configurations to glance-api.conf (s3_store_host, s3_store_access_key, s3_store_secret_key, s3_store_bucket, etc).
Revive the S3 driver and unit test that were used until Mitaka, and support multiple store configuration.
Documentation on how to use and configure S3 backend.

Dependencies

None

Testing

Test uploading of an image to the S3 backend.
Then test downloading of the image again.
Test delete the image.
Test configure multiple s3 drivers.

Documentation Impact

The documentation should be expanded to describe how to enable and use the S3 store.

References

Spec Lite: Add new container format

Fri, 12 Jul 2019 00:00:00

project:: glance
problem:: Compression method using hardware to accelerate has already been implemented in cinder project. A new container format ‘compressed’ needs to be added to container format list in glance to support this.
solution:: A new container format ‘compressed’ option needs to be added to the default container_format configuration list.
impacts:: New container format option

how:

If users chose to upload volume using compression method, cinder will do compression and then upload to glance. The uploaded image’s metadata will be ‘compressed’ in container_format so that it can be correctly processed after downloaded. Volume disk_format will not be changed by this compression process.

Glance will not handle the compression and decompression progress, which means glance will not check if it is a vaild compressed image. Whoever is to upload or download the image is responsible for compression or decompression.
Uploaded image will not be automatically compressed if simply changing the container_format as ‘compressed’. Compression or decompression happens when image is uploading from volume or downloading to a volume.
Glance will not identify what format the compressed image really is. In glance’s view, it is just a blob. Image consumer (such as cinder) will identify the real format such as gzip, rar or other format.

other impact:

Nova cannot currently handle images in a compressed container format. We propose patching Nova to reject such images: https://review.opendev.org/#/c/673407/

timeline:

Include in train release

link:

https://review.opendev.org/#/c/652275/

assignee:

ZhengMa

Ability to import an image in multiple stores

Thu, 04 Jul 2019 00:00:00

https://blueprints.launchpad.net/glance/+spec/import-multi-stores

Despite the Image service supports several back ends for storing virtual machine image, the import workflow only allow to push image data in one store. The goal of this spec is to allow this workflow to import image data into multiple stores and avoid the operator to manually copy data and update image locations.

Problem description

At the moment, the import workflow only allow to push image data in one store. As a result, operators today need to perform a number of manual steps in order to replicate image bits on backend glance stores despite using the ‘enabled_backends’ configuration option.

Note

Example

An operator provides an Openstack cloud with different sites, each with their local stores which nova hosts are accessing directly (Ceph).

For this purpose, when creating a new image, he needs to use the import workflow to store image data in store. Since the image is now in status “ACTIVE”, he can’t use the import workflow anymore and need to manually upload data in store2 to storeN and register these others locations URL using the glance API.

Since glance supports multi stores, it should propose a feature to upload image data into these stores at once to facilitate operators’ work.

Proposed change

This spec proposes the following high level feature to support multiple stores import:

Enhance the image import API to support targeting a list of stores for the image bits.

The idea is to provide a new ‘stores’ array field in the json payload where the user will list all stores he wants to import its data in (example: [‘ceph_fast’, ‘ceph_cheap’]).

If an unavailable store is submitted, the Api should reject the request.

If user or operator wants to import the image to all the enabled stores then it might be difficult or overhead to specify all the stores in the list. For that purpose a boolean field ‘all_stores’ will be provided.

If set to false (default behavior), we use the logic previously described.
If set to true, the data will be imported to the set of stores you may consume from this particular deployment of Glance (ie: the same set of stores returned to a call to /v2/info/stores on the glance-api the request hits). This can’t be used simultaneously with the ‘stores’ parameter. If user submits both, the Api should reject the request.

Another boolean field, ‘all_stores_must_succeed’, will be added in the payload to specify which error behavior the user wants to be applied:

If this field is set to ‘true’ (default behavior) and an error occurs during the upload in at least one store, the request should be rejected, the data deleted from stores (not staging), and the state of the image remains the same. The status of the image will be set to ‘ACTIVE’ only when the request is fully executed and successful.
If this field is set to ‘false’, the request will fail (data deleted from stores, …) only if the upload fails on all stores specified by the user. In case of a partial success, the locations added to the image will be the stores where the data has been correctly uploaded. The status of the image will be set to ‘ACTIVE’ as soon as an upload in one store is fully executed and successful.

Image locations will be updated each time an upload in a store succeed (with informations from this store).

Users will be able to follow task progress by looking at 2 reserved image properties:

os_glance_importing_to_stores: This property contains a list of stores that has not yet been processed. At the beginning of the import flow, it will be filled with the stores provided in the request. Each time a store is fully handled, it will be removed from the list.
os_glance_failed_import: Each time an import in a store fails, it is added to this list. This property is emptied at the beginning of the import flow.

If the image is deleted during the process of the request, import to remaining stores will not be processed and already uploaded data should be deleted.

Current location strategies modules shouldn’t be affected by theses changes as we don’t change the behavior of the image locations. It is currently possible to do the same by patching an image and specify a list of locations.

In the same vein, as it is already an option to choose a store when using import workflow, there is no need to add a new policy to restrict the import in multiple stores.

Alternatives

An alternative to this solution would be to allow to import data on an image with status ‘ACTIVE’. This will be the subject of another spec.

Data model impact

None

REST API impact

This spec proposes the following API changes:

Modified APIs

Import an image.

Common Response Codes

Normal http response code: 202
- 202: Accepted
Expected error http response codes: 400, 401, 403, 404, 409, 410
- 400: Bad Request with details.
- 401: Unauthorized
- 404: Not Found (image doesn’t exist or is not owned by the caller)
- 409: Conflict (image is not in appropriate status)
- 410: Gone (image deleted while operation in progress)

API Version

This change will require minor version bump. All URLS will be under the v2 Glance API. If it is not explicitly specified assume /v2/<url>

[Modified API] Import image to the store

Import image to the store:

POST /v2/images/{image_id}/import

This modifies the existing REST API to add three new optional body fields. For backwards compatibility, if the ‘stores’ parameter is not specified, the header ‘X-Image-Meta-Store’ is evaluated. If neither parameter i.e. ‘X-Image-Meta-Store’ header and ‘stores’ are specified then the store configured as default (e.g. default_backend) is used to upload the image to. If both parameters are supplied, or ‘all_stores’ parameter is set to true and ‘X-Image-Meta-Store’ header or ‘stores’ are set, the request will be rejected as Bad Request (ie: http 400).

New body fields:

stores – (String Array) If present contains the list of store id to import the image binary data to.
all_stores – (Boolean, default to false) If set to true, the data will be imported in all configured stores. If set to false, ‘stores’ and/or ‘X-Image-Meta-Store’ are evaluated.
all_stores_must_succeed – (Boolean, default to true) If set to false, the task will fail only if import fails in all specified stores. If set to true, the task will fail if import fails in one of the mentioned store.

Changed response codes:

400 – If the ‘stores’ field is present, but specifies a list of store id with at least one id that doesn’t exist or is read-only (like http). Or, if any two or more of the three ‘all_stores’:’true’, ‘stores’, ‘X-Image-Meta-Store’ are specified.

Example curl usage:

curl -i -X POST -H "X-Auth-Token: $token"
     -H "Content-Type: application/json"
     -d '{"method":{"name":"glance-direct"},
          "stores": ["ceph1", "ceph2"],
          "all_stores_must_succeed": false}'
     $image_url/v2/images/{image_id}/import

Security impact

None

Notifications impact

When going through the image import workflow, the payload sent during notification stages already contains a field “backend” which contains the store specified by the user when using multiple backend support. Notifications should be sent for each store asked by the user containing the status of the upload to that particular store. The new properties will be added to the notification payload.

Note

Example

An operator calls the import image api with the following parameters:

curl -i -X POST -H "X-Auth-Token: $token"
     -H "Content-Type: application/json"
     -d '{"method": {"name":"glance-direct"},
          "stores": ["ceph1", "ceph2"],
          "all_stores_must_succeed": false}'
    $image_url/v2/images/{image_id}/import

The upload fails for ‘ceph2’ but succeed on ‘ceph1’. Since the parameter ‘all_stores_must_succeed’ has been set to ‘false’, the task ends successfully and the image is now active.

Notifications sent by glance should look like (payload is truncated for clarity):

{
    "priority": "INFO",
    "event_type": "image.prepare",
    "timestamp": "2019-08-27 16:10:30.066867",
    "payload": {"status": "importing",
                "name": "example",
                "backend": "ceph1",
                "os_glance_importing_to_stores": ["ceph1", "ceph2"],
                "os_glance_failed_import": [],
                ...},
    "message_id": "1c8993ad-e47c-4af7-9f75-fa49596eeb10",
    ...
}

{
    "priority": "INFO",
    "event_type": "image.upload",
    "timestamp": "2019-08-27 16:11:30.058812",
    "payload": {"status": "active",
                "name": "example",
                "backend": "ceph1",
                "os_glance_importing_to_stores": ["ceph2"],
                "os_glance_failed_import": [],
                ...},
    "message_id": "8b8993ad-e47c-4af7-9f75-fa49596eeb11",
    ...
}

{
    "priority": "INFO",
    "event_type": "image.prepare",
    "timestamp": "2019-08-27 16:10:30.066867",
    "payload": {"status": "importing",
                "name": "example",
                "backend": "ceph2",
                "os_glance_importing_to_stores": ["ceph2"],
                "os_glance_failed_import": [],
                ...},
    "message_id": "1c8993ad-e47c-4af7-9f75-fa49596eeb10",
    ...
}

{
    "priority": "ERROR",
    "event_type": "image.upload",
    "timestamp": "2019-08-27 16:11:30.058812",
    "payload": {"status": "active",
                "name": "example",
                "backend": "ceph2",
                "os_glance_importing_to_stores": [],
                "os_glance_failed_import": ["ceph2"],
                ...},
    "message_id": "8b8993ad-e47c-4af7-9f75-fa49596eeb11",
    ...
}

Other end user impact

Glance client

The glance client (CLI + REST client) must be updated in accordance with this spec. Notably:

CLI / API support for specifying a list of store id on import.
CLI / API support for specifying all_stores_must_succeed option on import.

Performance Impact

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

yebinama

Reviewers

Core reviewer(s):

jokke

Work Items

Implementation tasks may consist of:

Image import with a list of stores supplied.
Add python-glanceclient support

Dependencies

None

Testing

Appropriate unit and functional tests to ensure the changes to glance function correctly.

Documentation Impact

We’ll need to ensure the glance docs are updated for:

New body fields for image import.

References

https://review.opendev.org/#/c/667132/

Cache API

Wed, 26 Jun 2019 00:00:00

https://blueprints.launchpad.net/glance/+spec/cache-api

Managing split deployments is currently a bit difficult and sometimes require operators to run commands on multiple nodes. Introducing a cache API would make managing such deployments easier.

Problem description

Adding new API endpoints related to caching would prove beneficial in two ways:

Along with the planned work on cluster awareness, this would make it easier to manage split deployments (multiple glance-api nodes on different locations)
Users would have to deal with fewer commands since the glance-cache-prefetcher is now part of glance-api as a periodic job and glance-cache-manage will now be part of python-glanceclient.

Proposed change

The main change this specification describes is the extension of the Glance API to handle cache-related operations. The goal is to have an API that can replace the “glance-cache-manage” tool. These are the operations we would like to support:

List cached,queued images
Delete cached, queued images
Delete all cached and queued images using single API call
Queue image for prefetching

At the moment to enable the glance caching in the deployment operator needs to configure two different middleware, cache and cachemanage. cache middleware is responsible for configuring cache resources such as cache directory, initializing cache driver etc. whereas cachemanage middleware used to deal with actually caching resources via glance-api. As caching now will be part of glance V2 API, we would like to deprecate cachemanage middleware and remove it as per deprecation and removal policy. The newly added cache endpoints will be EXPERIMENTAL in this cycle and will be marked as stable in following cycle.

In addition to above proposed change this will introduce three new policies cache_image, cache_list and delete_cache so that user with appropriate permissions can perform these operations. We will also deprecate the existing manage_image_cache’ policy in favor of new policies. To make the new policies compatible with secure RBAC we need to ensure to pass required parameters like ImageTarget while enforcing the new policies. It is recommended to keep `cache_image, cache_list and delete_cache operations limited to admin use only.

Alternatives

If we do nothing about cluster awareness and we do not provide an easy-to-use API for cache management, operators could probably write homemade solutions to aggregate data from multiple nodes, but this would be error-prone and not user-friendly.

Data model impact

None

REST API impact

This spec proposes the following new endpoints:

New API

List cached and queued images
Delete a cached or queued image
Delete all cached or queued images
Queue image for caching

Common Response Codes

Create Success: 201 Created
Delete Success: 204 No Content
Failure: 400 Bad Request with details.
Forbidden: 403 Forbidden

[New API] List cached or queued images

List all images cached or queued for caching on this node:

GET /v2/cache
{
   "cached_images": [
      {
         "id": "d75b9181-e1ce-45b4-8147-fb66fc4ea82f",
         "last_accessed": 1560451015.977297,
         "last_modified": 1560451015.977297,
         "size": 12345,
         "hits": 42,
         "checksum": "6788146b9d3fddc8dd03d86bfe9239b0"
      },
   ],
   "queued_images": [
       "id": "d75b9181-e1ce-45b4-8147-fb66fc4ea82f",
   ]
}

Response codes:

200 – Upon authorization and successful request. The response body contains the JSON payload with the cached and queued images.

[New API] Delete a queued or cached image

Delete a specific image from the cache:

DELETE /v2/cache/{image_id}

Response codes:

204 – The image was deleted
403 – Permission denied

[New API] Delete all queued or cached images

Delete all queued or cached images:

DELETE /v2/cache

Response codes:

204 – The cache was purged
403 – Permission denied

[New API] Queue an image for caching

Pre-cache an image:

PUT /v2/cache/{image_id}

Response codes:

202 – The request has been accepted, the image will be queued for caching
403 – Permission denied
404 – Image not found

Security impact

As described in proposed change section either existing policies or new policies will be enforced to avoid security breach.

Notifications impact

None

Other end user impact

The glance client should be updated, with new commands:

glance cache-list
glance cache-image <IMAGE-ID>
glance cache-delete <IMAGE-ID>
glance cache-delete-all

Provision will be made to pass direct URL (host:port) to these commands which will direct the call to particular glance node. Implementation details for the same will be described in specific glance-client spec.

Performance Impact

None

Other deployer impact

Caching will be local to each glance node and as these commands will be executed remotely, operator needs to know the direct URL of each glance node which are behind the load balancer. Operator need to provide this direct URL to glanceclient so that client should hit particular node to retrieve the cache information of that node.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: jokke
Other contributors:: cyril-roelandt

Work Items

Deprecate cachemanage middleware
Add new cache endpoint (/v2/cache)
Make new policies compatible with secure RBAC
Add python-glanceclient support
Deprecate glance-cache-manage
Deprecate glance-cache-prefetcher
Modify documentation, update API reference
Devstack support to enable cache on remote node

Dependencies

None

Testing

The new API endpoints should be tested using Tempest tests.

Documentation Impact

The API documentation will need to be updated
Need to update Cache documentation as well with new commands

References

https://docs.openstack.org/glance/victoria/admin/cache.html

Spec Lite: Lazy update stores information

Wed, 22 May 2019 00:00:00

project:

glance

problem:

In Rocky multiple backend support is added as experimental feature. The location object will have the ‘store_id’ (Store name) stored as a metadata attribute which helps to identify in which store the image is located. Glance API service has information about all configured stores in a global map.

After enabling/upgrading glance-api to use multiple stores feature, existing images will not have a stores information associated with them. As a result if existing image is requested for download then it will search that image in all configured stores which will cause a performance overhead.

All configured stores are stored in memory, if operator change the store name in configuration file and restart the glance-api service there is no way to update the new name to existing images which has store information associated with them.

solution:

Add a decorator to get image call which will retrieve the location object from image and then from location URL it will fetch the store information which is stored in the location map. Once location url is matched with the url from the global map, it will check if location metadata has existing store associated with it and then store name will be added/updated to the location metadata.

If user/operator uses GET call to show a single image or list all the images the particular decorator will update the store information to single or all images available depending upon the request. There will be slight impact on performance on list call as it will revisit the location call for each of the image in the list.

alternatives:

None, carry on using current mechanism.

impacts: DocImpact

timeline:: Include in Train release.
link:: None
assignee:: abhishekk

Spec Lite: Add location url prefix attribute to each store

Tue, 21 May 2019 00:00:00

project:

glance_store

problem:

In multiple store implementation, glance has added new metadata ‘store’ to location object, so that it will be easy to identify the store that particular image is uploaded. When operator upgrades glance node to use multiple stores, then existing images does not have store information associated with it. So if user wants to download any such particular image, then as it does not have ‘store’ associated with it, the said image will be searched in all the configured stores. This may cause the performance overhead.

solution:

To overcome this, we propose to add new attribute ‘_url_prefix’ to each of the store object. When glance-api service starts it stores a global map in the memory which includes store object, scheme and location_class for each of the store.

Sample of location map stored in the memory:

{
    'file_2': {
        'store': <glance_store._drivers.filesystem.Store object>,
        'store_entry': 'file',
        'location_class': <class 'glance_store._drivers.filesystem.StoreLocation'>
    },
    'file_1': {
        'store': <glance_store._drivers.filesystem.Store>,
        'store_entry': 'file',
        'location_class': <class 'glance_store._drivers.filesystem.StoreLocation'>
    }
}

At the time of initialization of each store, a location url will be retrieved using each stores configuration and assigned to ‘_url_prefix’ attribute of each store object. Whenever any GET call to image (before upgrading to multiple stores) is made, the location url of that image will be matched with ‘_url_prefix’ and equivalent store information will be updated to that image’s location metadata.

alternatives:

None, carry on using current mechanism.

impacts: DocImpact

timeline:: Include in Train release.
link:: None
assignee:: abhishekk

Train Project Priorities

Fri, 17 May 2019 00:00:00

Top priorities for Glance in Train cycle are:

Release glance_store 1.0.0
Getting multi-store moved away from Experimental
Removing dependencies to registry from cache tooling
Cleaning up deprecations
Cluster awareness:
- Image import call rerouting to the node that has access to image in staging
- Centralized cache management via API endpoint:
  - Listing cached images across cluster
  - Precaching images
- Copy/Remove images cross multi-store
- Distributed Store Discovery

On top of these housekeeping and feature items the community will prioritize bugfixes and reviews as appropriate.

Spec Lite: ‘all’ visibility image filter

Thu, 16 May 2019 00:00:00

project:: glance
problem:: When deployments use community images these community images are not returned when doing an image list. Currently murano and other projects use a property to determine if the image should be used. It is not possible to filter on all images with the specific property. Sahara also shares this issue. Horizon currently also needs to show all images that a user can boot from in the boot intance view and this cannot be done without 2 requests to glance.
solution:: We need to add a new visibility called ‘all’ which will return all images that are available to the user.
impacts:: None
how:: We will add the ability to list images with visibility=’all’ to return all images. This will require a bump in the api version.
alternatives:: The other option is to make 2 requests to glance each time you want to list all images or filter by all images which is inefficiant
assignee:: sorrison

Spec Lite: Add description field to image

Thu, 16 May 2019 00:00:00

project:

glance

problem:

Currently there is not a dedicated image property to record the description of an image. In most of our customer environments, users typically upload multiple images. Because of the large number of users, it is difficult to distinguish the purpose of an image through the name field (possibly with duplicate names).

For example: Nova and Cinder have the description field, and the description field allows you to record the specific purpose of the object.

solution:

Add description field as a “common image property” to image. This way it will appear in the image schema (good for interoperability purposes) but will be stored as a user-defined image property (so will not require any database changes). You can set --property to add description properties when executing the CLI.

This is an exception to the Glance policy that new image properties should be prefixed with os_. It’s not necessary for this spec for the following reasons.

Under this proposal, description will be stored as a user specified (“custom”) image property in the image_properties database table. Because it is not stored as a new column in the images table, it will not block the display of any description property on currently existing images.
It’s to be expected that a property named description will in fact contain some kind of description of the image it’s a property of. Thus we do not expect that there will be an inconsistency between any existing description image properties and the description of description that will appear in the image schema.
Using the name description is consistent with the other services (for example, Nova and Cinder) that recognize description metadata on resources.

CLI execution example:

$ openstack image create \
    --property description='This is a test image file.' \
    --file cirros-0.3.4-x86_64-disk.img \
    --disk-format qcow2 \
    --container-format bare \
    test_image

alternatives:

Do nothing, given that users can already create such a property. This alternative, however, has the disadvantage that it does not provide a guideline for local practices to conform with, which in turn makes interoperability problematic.

timeline:

Include in Stein release.

link:

https://review.openstack.org/620433

reviewers:

Brian Rosmaita

assignee:

Brin Zhang

Spec Lite: Embed validation data with image locations

Thu, 16 May 2019 00:00:00

project:

glance

problem:

A new image using the HTTP store may have its locations initialised using the add or replace operation in an HTTP PATCH request, but there is currently no way to provide values for accompanying checksum and multihash values.

solution:

Allow embedding of values for checksum, os_hash_algo and os_hash_value in a new write-only JSON object named validation_data, along with the url and metadata for an image location. These values will be used to populate the corresponding image properties.

New values for any of these items will only be accepted if the image status is queued and the corresponding image property is not already populated. To allow idempotency, this object may be included when adding or replacing locations for an image which is in active status or/and already has the corresponding properties populated, but the supplied values must exactly match the existing ones.

The object may be included in one or more of the items in a locations list, but values must be consistent across all instances.

Although the validation_data object will be optional, if it is present, the os_hash_algo and os_hash_value items will be required, to force adoption of multihash. Since multihash will be the default mechanism for clients in the Stein release, checksum will be optional, but included to accommodate legacy consumers that have not yet implemented multihash. The consumer is expected know to to populate checksum only if their deployment requires it.

os_hash_algo must match the Glance server’s DEFAULT.hashing_algorithm configuration option. Whilst it seems redundant to require an input with only one acceptable value, this is required to ensure that the user knows which algorithm is required. The checksum and os_hash_value cannot be verified (since the Glance server does not have a copy of the image data), but they will be validated as hexadecimal values of the correct size for the respective algorithms.

Any violations of the above rules will result in a HTTPConflict exception (HTTP status 409).

The following will be added to the properties for locations items in the images schema:

'validation_data': {
    'description': _(
        'Values to be used to populate the corresponding '
        'image properties. If the image status is not '
        '"queued" or/and the image properties are already '
        'populated, any supplied values must exactly match '
        'existing ones.'
    ),
    'type': 'object',
    'writeOnly': True,
    'properties': {
        'checksum': {
            'type': 'string',
            'minLength': 32,
            'maxLength': 32,
        },
        'os_hash_algo': {
            'type': 'string',
            'maxLength': 64,
        },
        'os_hash_value': {
            'type': 'string',
            'maxLength': 128,
        },
    },
    'required': [
        'os_hash_algo',
        'os_hash_value',
    ],
},

Support will also be added to the add_location() method and the location-add shell command in python-glanceclient.

alternatives:

Implement an import method to directly register images for use with the HTTP store (without requiring use of HTTP PATCH).

timeline:

Include in Stein release. Need approval ASAP, so I can proceed with a private backport for my Rocky upgrades (v1 API removed).

link:

https://review.openstack.org/597368

assignee:

imacdonn

Spec Lite: Add glance-cache-manage utility using v2 API

Thu, 16 May 2019 00:00:00

project:

glance

problem:

In Rocky, the v1 dependant glance-cache-manage command was removed while removing Images API v1 entry points. As a part of Edge computing glance cache needs to be enabled on far-edge nodes via Split Control plane where glance-cache-manage utility will be essential to queue image for prefetching, to list & delete images from Image Cache.

solution:

In Stein, as per Edge computing architecture, glance cache will be enabled on far-edge nodes. Hence it will be good to add glance-cache-manage utility using v2 API in glance. This utility will have the following commands and the same interface as Queens glance-cache-manage utility [0] insofar as possible, [0] https://docs.openstack.org/glance/queens/cli/glancecachemanage.html

1. Queue the image with identifier <IMAGE_ID> for caching,

   $ glance-cache-manage --host=<HOST> queue-image <IMAGE_ID>

2. List all images currently cached

   $ glance-cache-manage --host=<HOST> list-cached

3. List all images currently queued for caching.

   $ glance-cache-manage --host=<HOST> list-queued

4. Delete an image from the cache

   $ glance-cache-manage --host=<HOST> delete-cached-image <IMAGE_ID>

5. Remove all images from the cache

   $ glance-cache-manage --host=<HOST> delete-all-cached-images

6. Deletes an queued image from the cache

   $ glance-cache-manage --host=<HOST> delete-queued-image <IMAGE_ID>

7. Remove all images from the cache queue

   $ glance-cache-manage --host=<HOST> delete-all-queued-images

alternatives:

None

impacts:

DocImpact

timeline:

Include in Stein release.

link:

None

assignee:

pdeore

Running Glance on Windows

Thu, 16 May 2019 00:00:00

https://blueprints.launchpad.net/glance/+spec/windows-support

The goal is to allow Glance services to run on Windows, which enables Hyper-V OpenStack deployments to take full advantage of Microsoft storage solutions.

Problem description

At the moment, Glance services cannot be run on Windows. To bring some context, there are OpenStack deployments that rely on Hyper-V compute nodes along with other Microsoft technologies.

In terms of storage, the most commonly used SDS solution is Storage Spaces Direct (S2D) along with highly available Scale-Out File Server (SoFS) SMB shares.

One issue is that the Linux cifs driver currently provides limited HA SMB share support (most versions can’t even connect to HA shares, there’s no Witness service support, automatic failover was recently added).

That being considered, most deployers have to use a separate storage backend for their Glance images (e.g. Swift, Ceph, etc).

Allowing Glance to run on Windows will allow deployers to benefit more from their S2D backends, commonly used for storing VM disks (currently handled by Nova as root images or Cinder as attachable volumes). In particular, this will be useful for hyper-converged deployments, where having to deploy yet another storage backend may not be desired.

Proposed change

Getting Glance services to run on Windows is not that difficult. There are just a few things that we need to change:

avoid forking, not available on Windows
avoid unavailable signals
avoid renaming/deleting in-use files
avoid using binaries that aren’t available on Windows
avoid having eventlet monkey-patch the os module as this will cause subprocess.Popen to fail
avoid missing features or libraires (e.g. xattr)
some small differences in path handling (‘/’ will be handled as a relative path)
avoid connecting to ‘0.0.0.0’, use ‘127.0.0.1’ instead. This mostly applies to the functional tests

All the above points are covered in less than 150 LOC (a few hundred more if we count the tests as well), without affecting the Linux behavior.

The os-win library will be used for most of the Windows low-level operations, which is an official OpenStack project.

The main goal is to use the filesystem driver along with SMB shares or Cluster Shared Volumes (CSV).

IIS will not be supported initially, using eventlet wsgi instead (which happens to be the recommended way of deploying Glance at the moment).

Alternatives

For Hyper-V deployments, the alternative is to keep running Glance on separate Linux hosts, having to deploy an additional storage backend or go with the limited HA SMB share support that’s currently available on Linux.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Performance wise, there shouldn’t be any big differences. Spinning up new processes may be a bit slower on Windows, for which reason the functional tests may take longer to complete.

Other deployer impact

Glance will now be able to run on Windows. Not much changes config wise.

Keep in mind that we’re mainly targeting the file store for the beginning (which is the main reason we want Glance to run on Windows).

Glance will prefferably run as a Windows service. In general, we provide MSI packages that simplify installing and configuring OpenStack services (e.g. Nova, Neutron agent, Cinder, etc), which can also be run unattendedly.

We intend to provide Juju charms as well that will allow deploying Glance on Windows.

One thing to note is that os-win currently supports Windows Sever 2012 or above.

Developer impact

Developers should keep in mind that Glance is supposed to be portable.

Judging from past experience, this hasn’t been a problem with other OpenStack projects. Glance doesn’t use too many platform specific libraries or binaries, so it should be fine.

Implementation

Assignee(s)

Primary assignee:: lpetrut
Other contributors:: None

Work Items

This work has been divided in 3 patches:

a small patch that allows Glance services to run on Windows
an additional one that allows multiple processes to be used by API services
a patch that allows all Glance unit and functional tests to run on Windows

A few small changes were needed on os-win as well, exposing some low-level Windows primitives.

Dependencies

We rely on os-win, which is an official OpenStack library that exposes low-level Windows functionality. It’s currently used by a few other OpenStack projects, such as Nova, Cinder, Ceilometer, networking-hyperv, etc.

Testing

The already existing tests provide enough coverage. Still, the unit and functional tests will need some small changes in order to be able to run on Windows (currently relying on some Linux specific functionality).

We intend to provide 3rd party CI testing. For the record, we’re currently voting on Nova, Cinder and Neutron patches, running tests against Hyper-V.

Documentation Impact

The documentation should be updated to point out the fact that Glance is now Windows compatible, along with some installing and configuration guide.

References

Spec Lite: Show Store Info

Thu, 16 May 2019 00:00:00

project:: python-glanceclient
problem:: In multi-store environment, user may want to know which backend store images are stored in. But currently store info (store id) can only be showed in command “glance image-show” which can only show one image. It’s not convenient when want to know the store info of a list of images.
solution:: This spec aims to add –include-stores option to image-list, command like “glance image-list –include-stores”. As store id is the same in most of production environment, so it’s not good to list this info in -v, otherwise the store id column will be the same value and it’s a bit redundant.
impacts:: None
timeline:: S-1
assignee:: LiangFang

Barbican secret deletion support

Thu, 02 May 2019 00:00:00

https://blueprints.launchpad.net/glance/+spec/barbican-secret-deletion-support

The Block Storage Service (Cinder) offers end users the ability to create images from encrypted volume types. When it does this, Cinder stores a secret in Barbican in a 1-1 relation to the image and puts the Barbican ID of the secret as an image property, cinder_encryption_key_id. When a user deletes such an image, the Barbican secret is no longer applicable to any resource, but it persists in Barbican. As a result, deployers have a situation where useless secrets are piling up in Barbican. This would be mitigated if Glance deleted the unique Barbican secret of an image at the time of image deletion.

Problem description

Cinder would like Glance to delete a resource (the Barbican secret) that Glance has not created. Given that most deployments allow users to set custom image properties, it’s possible for a user to put the cinder_encryption_key_id metadata on any image, which could result in the premature deletion of an in-use key since the 1-1 relation between Barbican secret ID and Glance image has been broken. Hence Glance is reluctant to implement functionality that could result in end user data loss.

Proposed change

Add the Glance common image property cinder_encryption_key_deletion_policy. At this time, cinder_encryption_key_id will also be added as a common image property. Since the common image properties are stored in the image_properties table, they must have a JSON type of string. As common image properties, they will be added to the default schema-image.json file as follows:

{
    "cinder_encryption_key_id": {
       "description": "Identifier in the OpenStack Key Management Service for the encryption key for the Block Storage Service to use when mounting a volume created from this image",
       "type": "string"
    },
    "cinder_encryption_key_deletion_policy": {
        "description": "States the condition under which the Image Service will delete the object associated with the 'cinder_encryption_key_id' image property.  If this property is missing, the Image Service will take no action",
        "type": "string",
        "enum": [
            "on_image_deletion",
            "do_not_delete"
        ]
    }
}

Note

Note: The common image properties are interoperability suggestions but are not mandatory features of Glance. As such, there is no guarantee that a particular Glance installation will allow them to be listed in the image-schema response. Thus, we cannot rely on schema-validation of the value for this property. Note that this does not affect Cinder writing these properties to images (as long as the Glance configuration option allow_additional_image_properties is True, which is its default value).

Note

We should seriously consider deprecating the allow_additional_image_properties configuration option during this cycle. I believe it was originally introduced to prevent users from flooding the image_properties table with junk and it predated the image_property_quota option. Now that we have the latter option, it’s really unnecessary, and if an operator were to set it False, all sorts of stuff would break as multiple OpenStack services currently write/rely on custom image properties.

Note

There’s a bug someone asked me about in IRC once (I can’t find it in Launchpad, it may not have been filed) that an operator can modify /etc/schema-image.json to include arbitrary properties (which was that file’s original purpose) and assign them JSON types other than ‘string’. The type is enforced by image create/update but you get a 500 because it has to be a string in the database (but the API won’t accept a string if the schema says it’s boolean or something). We should document that these things must be strings.

The presence of this property with the appropriate value gives Glance explicit permission to delete the key, even though Glance has not created it. When this property is present on an image with the value ‘on_image_deletion’, Glance will search the image’s properties for the property cinder_encryption_key_id. If it exists, Glance will make a request to Barbican to delete the secret whose ID is the value of the image property.

Because cinder_encryption_key_deletion_policy is a custom image property, it can be added/deleted/updated by the image owner. Hence, if the image owner has a reason to preserve a key, permission to delete can be revoked by changing the value to ‘do_not_delete’ or by simply deleting the cinder_encryption_key_deletion_policy property.

Error conditions:

If cinder_encryption_key_id is not present on the image, image deletion will not be affected by the presence of delete_encryption_key_on_image_deletion: on_image_deletion.
If cinder_encryption_key_id indicates a nonexistent Barbican ID, this should be logged, but will not affect image deletion.
If cinder_encryption_key_deletion_policy has a value other than on_image_deletion or do_not_delete, Glance will do the following:
- take no action with respect to the cinder_encryption_key_id
- delete the image

Alternatives

Do nothing and leave it up to the end user to explicitly delete the secret from Barbican. This is problematic for the Cinder workflow, which has kept secret management hidden from the user, hence the user may not be aware until sometime after the image has been deleted (and its metadata gone) that the Barbican secret can be deleted, and the secret ID is no longer available. Since it’s impossible to know for any secret whether anything might be holding a reference to it, the user can’t simply go to Barbican and determine what secrets are no longer in use. So this is not a really a viable alternative.
Leave the cinder_encryption_key_id as the only metadatum, and have its presence be sufficient for Glance to delete the secret upon image deletion. This has the drawback that an end user who’s using the Cinder feature in a nonstandard way has no way to prevent Glance from deleting a Barbican secret that may still be in use.
Do not make cinder_encryption_key_deletion_policy a common image property, but instead let it be a regular custom image property, similar to the custom image properties that are recognized by the Nova scheduler. Because of the data loss possibility posed by automatic secret deletion, it would be better to have its function clearly documented in the image schema, rather than leaving this to documentation only.
Make cinder_encryption_key_deletion_policy a boolean instead of an enumerated string type. Unfortunately, it cannot actually be a boolean type because it must be stored as a string, which is confusing for tooling developers because its value would be "true", not the JSON true value. Additionally, an image-show response containing:
```
cinder_encryption_key_deletion_policy: on_image_deletion
```
is self-documenting with respect to what this image property means.

Data model impact

None. A common image property appears in the image schema, but is stored in the image_properties table with the custom image properties.

REST API impact

None.

Security impact

No impact on the security of Glance. This arguably makes the Cinder-Glance volume-to-image functionality more secure. It no longer leaves a user in the position of having to clear out a bunch of excess secrets from Barbican, one of which may actually be in use. Additionally, it doesn’t require a user to manually delete a key (the user might delete the wrong one).

Notifications impact

None.

Other end user impact

The Cinder workflow is designed to hide key handling from the user; this is consistent with that design.

Performance Impact

Glance will need to make a call to Barbican as part of the image delete process.

Other deployer impact

This will be a bonus to deployers who will no longer have to worry about useless Barbican secrets piling up.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: brian-rosmaita or abhishek-kekane

Work Items

Modify the image schema to include the new common image property and to make the current property Cinder uses, cinder_encryption_key_id, an official common image property.
Implement the code to delete the secret in Barbican upon image deletion.

Dependencies

None.

Testing

Since the scenario we’re interested in is whether Glance can delete a secret from Barbican under certain conditions, a tempest test could be implemented that creates a secret, puts the appropriate metadata on an image, deletes the image and verifies that the Barbican secret is/is not deleted.

If there’s an existing tempest test that actually creates an image of an encrypted volume in Glance, then we could piggyback on that to verify that key deletion occurs. This, of course, would have to wait until Cinder has implemented putting the new cinder_encryption_key_deletion_policy flag on images created from encrypted volumes.

Documentation Impact

Document the new properties along with the rest of the common image properties.

References

Train PTG discussion: https://etherpad.openstack.org/p/cinder-train-ptg-planning

Replace Snet Config with Endpoint Config

Thu, 27 Dec 2018 00:00:00

https://blueprints.launchpad.net/glance/+spec/replace-snet-config-with-endpoint-config

The snet option forces the deployer to name the desired endpoint after the public endpoint.

Problem description

The snet option forces the deployer to name the desired endpoint after the public endpoint. In order to switch between multiple internal networks, names have to be changed.

Proposed change

Instead of constructing a URL with a prefix from what is returned by auth, specify the full URL via configuration (e.g. https://www.example.com/v1/not_a_container). The location of an object is obtained by appending the container and object to the configured URL.

Alternatives

Only use auth v2. Store and retrieve multiple internal endpoints from the catalog. This is the preferred approach and requires migrating code that is using v1 to v2 which is a much larger and completely separate effort. Furthermore, a configurable endpoint could still be useful for overriding catalog.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Configuration options will change:

Removed config option: “swift_enable_snet”. The default value of “swift_enable_snet” was False [1]. The comments indicated not to change this default value unless you are Rackspace [2].
Added config option “swift_store_endpoint”. The default value of “swift_store_endpoint” is None, in which case the storage url from the auth response will be used. If set, the configured endpoint will be used. Example values: “swift_store_endpoint” = “https://www.example.com/v1/not_a_container”

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: jesse-j-cook

Reviewers

Core reviewer(s):: nikhil-komawar
Other reviewer(s):: johngarbutt ben-roble

Work Items

Modify code to use endpoint config
Remove snet prefix code

Dependencies

None

Testing

Verify auth works
Verify configured endpoint is reached

Documentation Impact

Document new endpoint configuration option.
Remove documentation for snet configuration option.

References

https://review.openstack.org/#/c/139726/

Spec Lite: Add periodic job to prefetch images into cache

Thu, 27 Dec 2018 00:00:00

project:: glance
problem:: During Queens release glance registry is marked as deprecated and will be removed during Train cycle. Glance cache-prefetcher utility uses glance registry to cache the images.
solution:: In Train, as per Edge computing architecture, glance cache will be enabled on far-edge nodes. As of now glance-cache-prefetcher is dependent on registry which is deprecated and due for removal. In order to remove the dependency on registry, we are proposing to add a new periodic job to glance-api service which will run as per interval set using ‘cache_prefetcher_interval’ configuration option and fetch images which are queued for caching in cache directory. This new periodic job will only run if cache is enabled by the operator.
alternatives:: None
impacts:: DocImpact
timeline:: Include in Train release.
link:: None
assignee:: abhishekk

Ocata Review Priorities

Mon, 12 Nov 2018 00:00:00

Here are the Ocata review priorities.

Priority Item	Owner(s)	Spec(s)
OpenStack “Community” Goal	Abhishek Kekane Alexander Bashmakov	eliminate oslo incubated code
Community Images	Timothy Symanczyk Dharini Chandrasekar	community images
Image Import Refactor	Erno Kuvaja Cyril Roelandt	image import refactor
Rolling Upgrades	Hemanth Makkapati Alexander Bashmakov	rolling upgrades database strategy alembic migrations

Important dates

This is an abbreviated list focused on dates relevant to Glance. See Ocata Release Schedule for the complete list for OpenStack.

Milestone	Week of	What
R-15	November 7	Glance spec proposal freeze
R-14	November 14	O-1 milestone
R-13	November 21	Glance spec freeze
R-10	December 12	O-2 milestone
R-9 through R-7	December 19 through January 6	expect team to be half-strength (at best) due to holidays
R-5	January 16	glance_store Ocata release (final release for non-client libraries)
R-4	January 23	O-3 milestone, Glance feature freeze, python-glanceclient Ocata release (final release for client libraries)
R-3	January 30	RC-1 release and string freeze
R-1	February 13	final RCs
R-0	February 20	Ocata release

Pike Review Priorities

Mon, 12 Nov 2018 00:00:00

Here are the Pike review priorities. Due to the Pike cycle Glance personnel situation, this list is reduced from what was published on the PTG etherpad. Thus in this cycle we’re introducing a category of untargeted specs. These specs have been reviewed, approved, and are sufficiently detailed that they could be implemented by new contributors, but given the current personnel situation, we cannot realistically include them as priorities for the Pike release.

Priority Item	Owner(s)	Spec(s)
Image Import Refactor	Erno Kuvaja	image import refactor
OpenStack Pike “Community” Goal 1	was Alexander Bashmakov, seeking new owner	support python 3.5
OpenStack Pike “Community” Goal 2	Matthew Treinish	control plane API endpoints deployment via WSGI

Important dates

This is an abbreviated list focused on dates relevant to Glance. See Pike Release Schedule for the complete list for OpenStack.

Milestone	Week of	What
R-22	March 27	Glance spec proposal freeze
R-20	April 10	P-1 milestone
R-19	April 17	virtual midcycle meeting; Glance spec freeze
R-16	May 8	OpenStack Summit/Forum
R-12	June 5	P-2 milestone
R-6	July 17	glance_store Pike release (final release for non-client libraries)
R-5	July 24	P-3 milestone; Glance feature freeze; python-glanceclient Pike release (final release for client libraries); Pike community goals completion
R-3	August 7	RC-1 release and string freeze
R-1	August 21	final RCs
R-0	August 28	Pike release

Queens Review Priorities

Mon, 12 Nov 2018 00:00:00

The Queens review priorities were discussed at the Denver PTG and refined during the spec review process early in the Queens cycle. The preliminary list was maintained on the Denver PTG etherpad.

This list is an estimate of what the Glance project team can accomplish during the Queens cycle based on our developers’ estimates of how much time they can commit to Glance. If additional resources become available, it’s possible that some of the Untargeted Approved Specifications could be added to Queens.

The following list is roughly in priority order (highest to lowest).

Priority Item	Owner(s)	Spec(s)
Image Import Refactor	Erno Kuvaja	image import refactor
Inject Metadata Automatically	Abhishek Kekane	inject metadata automatically
Secure Hash Algorithm Support	Scott McClymont	multihash
Fix OSSN-0075	Brian Rosmaita	fix ossn-0075
Glanceclient: Switch to Turn Off Schema Validation	unassigned	validation switch
Refactor Glance Scrubber	Xiyuan Wang	refactor glance scrubber
Remove the Images API v1	Brian Rosmaita	remove v1
Actually Deprecate the Glance Registry	Erno Kuvaja	deprecate glance registry
Glanceclient: Deprecate Images API v1 Support	Brian Rosmaita	deprecate v1 support
OpenStack Queens “Community” Goal	Lance Bragstad	policy in code

Important dates

This is an abbreviated list focused on dates relevant to Glance. See Queens Release Schedule for the complete list for OpenStack.

Milestone	Week of	What
R-22	Sept 25	Glance spec proposal freeze (Thursday 28 Sept 13:00 UTC)
R-21	Oct 2	Glance spec freeze (Friday 6 Oct 23:59 UTC)
R-19	Oct 16	Q-1 milestone
R-16	Nov 6	Sydney Summit/Forum (expect low activity)
R-12	Dec 4	Q-2 milestone
R-9	Dec 25	Holidays (expect low activity)
R-8	Jan 1	Holidays (expect low activity)
R-6	Jan 8	glance_store Queens release (final release for non-client libraries)
R-5	Jan 22	Q-3 milestone Glance feature freeze python-glanceclient Queens release (final release for client libraries); remove Images API v1 remove Registry API v1 Queens community goal completion
R-3	Feb 5	RC-1 release and string freeze
R-1	Feb 19	final RCs
R-0	Feb 26	Queens release

Spec Lite: Rethinking our filesystem access

Thu, 27 Sep 2018 00:00:00

project:

glance

problem:

Glance use filesystem in quite a few places with various mechanisms. Tasks and staging operation introduced as a part of import image workflow are consuming glance_store by overriding the configs and initializing the store via internal functions.

solution:

In Rocky multiple backend support is added as experimental feature. We should use this to reserve certain stores for these operations. As a part of this we will deprecate work_dir and node_staging_uri configuration options and reserve two filesystem stores ‘os_glance_tasks_store’ and ‘os_glance_staging_store’, which can be used to get rid of initializing store via internal functions. These stores will be hard-coded in glance for the time being and injected to enabled_backends config option to load when glance_store will be initialized at the time of glance-api service starts. Operator needs to insure that these stores will not be included in enabled_backends config option in glance-api.conf.

These reserved stores will be injected only if multiple backends are enabled i.e. enabled_backends config option is defined in glance-api.conf. If multiple backend is not enabled then node_staging_uri and work_dir config options will work as it is.

Sample code to show how these config options will be injected and used:

reserved_stores = {
    'os_glance_staging_store': 'file',
    'os_glance_tasks_store': 'file'
}

enabled_backends = CONF.enabled_backends
if enabled_backends:
    enabled_backends.update(reserved_stores)

Then operators need to ensure to have below sections defined in glance-api.conf:

[os_glance_tasks_store]
filesystem_store_datadir = /var/lib/glance/tasks_work_dir

[os_glance_staging_store]
filesystem_store_datadir = /var/lib/glance/staging

NOTE: The path for filesystem_store_datadir for ‘os_glance_tasks_store’ and ‘os_glance_staging_store’ should be different from actual path if file backend is used. The os_glance_* prefix is reserved for glance and cannot be used by operators to name their stores.

‘os_glance_tasks_store’ and ‘os_glance_staging_store’ will be excluded from ‘stores-info’ call and will not be accepted as a ‘backend’ option in create image calls.

alternatives:

None, carry on using current mechanism.

impacts: DocImpact

timeline:: Include in Train release.
link:: None
assignee:: abhishekk

Stein Project Priorities

Thu, 16 Aug 2018 00:00:00

TODO: fill this in after the PTG

Mitigate OSSN-0075

Tue, 07 Aug 2018 00:00:00

https://blueprints.launchpad.net/glance/+spec/mitigate-ossn-0075

OpenStack Security Note OSSN-0075, “Deleted Glance image IDs may be reassigned”, was made public on 13 September 2016. The current situation is that due to a lack of agreement of how to fix it, we’ve left operators in a bad state: our advice is that soft-deleted rows in the ‘images’ table in the Glance database should not be purged from the database, yet at the same time, the glance-manage tool deletes such rows without warning.

Problem description

Briefly, the problem is that Glance has always allowed a user with permission to make the image-create call the option of specifying an image_id. If the specified image_id clashed with an existing image_id, the image-create operation would fail; otherwise, the specified image_id would be applied to the new image. Consistency is enforced by a uniqueness constraint on the ‘id’ column in the ‘images’ table in the database. Since Glance database entries are soft-deleted, a proposed image_id will be checked against all image_ids that were assigned since the last purge of the ‘images’ table.

As described in OSSN-0075, this problem becomes a security exploit when (a) a popular public or community image is deleted, (b) the database is purged, and (c) a user creates a new image with that same image_id. Users consuming an image by image_id, which is the way Nova and Cinder consume images, may then wind up booting virtual machines using an image different from the one they intend to use.

Note that the new image would have its own data and checksum that would be different from the original data and checksum, but there would be no way for Nova, for instance, to know that these had changed. Were someone to boot a server using the image_id, Nova would receive image data and then verify the checksum against whatever checksum Glance has recorded as associated with the image, which would be the new checksum.

The idea that once an image goes to ‘active’ status, the (image_id, image data, checksum) will not change is called image immutability. It’s important to note that image immutability is required for Glance or else it cannot function as an image catalog. If each consumer had to keep track of the image_id and checksum and other essential properties in order to verify the downloaded data, then there’d be no point in having Glance maintain this information.

Note

The primary use case for allowing end-users to specify an image_id at the time of image creation is to make it easy to find the “same” image data (that is, the data is bit-for-bit identical although it’s stored in different locations) in different regions of a cloud. It’s important to note that the “sameness” of images in different regions is not guaranteed by Glance. (A Glance installation can guarantee the immutability of images within its own region, but it has no way of knowing what’s happening in other regions.) Thus, under the current situation, when an end user relies on the image_id as the guarantor that they’re getting the “same” data in different cloud regions, the end user is actually relying upon the trustworthiness of the image owner.

This is a separate issue from OSSN-0075 and is independent of whether or not the Glance database is ever purged. We point it out as something for operators to keep in mind. To be clear about the issue, here’s an example. Suppose that a cloud operator puts an image with image_id A in regions R, S, T, though for some reason the operator does not put that image in region U. Any cloud user in region U could create an image with image_id A in region U. The image could then be made available to some target user by image sharing, or with the entire cloud by giving it ‘community’ visibility.

An operator can avoid this scenario by creating an image record with image_id A in region U and not uploading any data to it. The image will remain in ‘queued’ status, and if the visibility is not changed to ‘public’ or ‘community’, the image will not appear in any end user’s image-list response.

There is also room for end user education here, namely, that image consumers should not rely solely upon image_id to guarantee that they are receiving the same image data in cross-region scenarios.

Through discussions with operators, it’s clear that the ability to set the image_id on image creation is being used out in the field, so we can’t simply block this ability. At the same time, we must allow the database to be occasionally purged, as there is evidence that for large deployments, having a large number of soft-deleted rows in the ‘images’ table affects the response time of the image-list API call.

Proposed change

Modify the current glance-manage db purge command so that it will not purge the images table.

Introduce a new command, glance-manage db purge-images-table to purge the images table. The new command will take the same options as the current purge, namely, --age-in-days and --max-rows. The rationale for this being a new command (rather than a --force option to the current command) is twofold: (1) it’s likely that the age-in-days used will be different for the images table, and (2) given that purging the images table has a security impact, having it as a completely separate command emphasizes this.

Alternatives

Introduce a policy governing whether or not a user is allowed to specify the image_id at the time of image creation. The downside of this proposal is twofold:
- it breaks backward compatibility given that this ability has been allowed up to now in both the v1 and v2 versions of the Image API
- it breaks interoperability in that end uses will have the ability in some clouds but not in others
A further problem with this proposal is that if the cross-region use of a particular image_id is denied to end users, they will have to use some other piece of image metadata for this purpose. Since cinder and nova both use the image_id when services are requested, user workflows will have to change to introduce an extra call to the image service to find the image record before the image_id to pass to cinder or nova is determined.
Instead of introducing a new column in the images table, introduce a new single-column table with a uniqueness constraint to record “used” UUIDs. The image-create operation would try to insert a proposed UUID into this table instead of the ‘images’ table and fail as it currently does if the uniqueness constraint were violated. This “used” UUID table would never be purged, but the glance-manage tool could continue to purge all other tables.

This alternative has the advantage of not impacting the image-list call. It would eventually introduce a small delay into the image-create operation, but that’s probably acceptable.

The downside is that this proposal introduces an unpurgable table that is unbounded in size.
A variation on alternative #2: instead of a single-column table, have at least a deleted_at column in addition to the image_id. This table would not be touched by the “normal” glance-manage database purge operation. Rather, an additional purge operation could be introduced for this table that would purge rows that were, say, 5 years old from the table.

A problem with this suggestion is that a determined attacker could nonetheless flood the “used” image_ids table. This is possible because while it might make sense to limit the number of existing images a user owns, it doesn’t make sense to limit the number of deleted images a user owns. For example, an end user who creates an image of some important server every day, but only keeps around a week’s worth, will accumulate many deleted images (multiplied by the number of servers this is being done for), but this is perfectly legitimate behavior. So I’m not sure how flooding the “used” image_id table could be prevented, except by something like rate-limiting, though that would have to be set in such a way as not to impact legitimate use cases.
Introduce a new field, preserve_id, for use in the images table. This field will be for internal Glance use only and will not be exposed through the API. This field will be null by default and will be set true whenever the ‘visibility’ field of an image is set to ‘public’ or ‘community’. There will be no way to unset the value of the field. In addition to this, modify the glance-manage tool so that it will never delete an entry from the images table that has preserve_id == True.

As with alternatives 2 and 3, the database table will continue to grow, but this growth is constrained by keeping only rows relevant to the OSSN-0075 exploit. On the other hand, all an attacker has to do is read this spec to realize that by creating image records with community visibilty, the images table can still be flooded with spurious image records. Thus this strategy is too easily defeated to be worth implementing, especially as it might give operators a false sense of security.

Data model impact

None

REST API impact

None

Security impact

This change will enhance security by providing operators with a means of mitigating the exploit described in OSSN-0075.

Notifications impact

None

Other end user impact

None

Performance Impact

The images table will grow indefinitely, though the associated tables (image_properties, image_tags, image_members, image_locations) can be purged by the glance-manage tool.

The images table can be partially purged at appropriate intervals.

Other deployer impact

Operators will have to monitor Glance for abnormal usage patterns and take appropriate action.

Additionally, operators should be made aware of the cross-region version of the OSSN-0075 exploit (as discussed in the Note in the Problem Description section).

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

brian-rosmaita

Other contributors:

undetermined

Work Items

Modify the glance-manage tool:
- The current behavior is that it purges all tables of soft-deleted rows. Change the behavior so that the images table is not purged by default.
- Add a new command to purge the images table. It should take the --age-in-days and --max-rows options just like the current purge command.
update operator documentation
release note

Dependencies

No new dependencies.

Testing

Appropriate unit tests to ensure the changes to glance and the glance-manage tool function correctly.

Documentation Impact

The Glance Administrator Guide will need to be updated.

References

OSSN-0075: Deleted Glance image IDs may be reassigned.

multi-store backend support

Tue, 07 Aug 2018 00:00:00

https://blueprints.launchpad.net/glance/+spec/multi-store

The Image service supports several back ends for storing virtual machine image namely Block Storage service (cinder), Filesystem (a directory on a local file system), HTTP, Ceph RBD, Sheepdog, Object Storage service (swift) and VMware ESX. As of now operator can configure single backend on a per scheme basis but it’s not possible to configure multiple backends for same or different stores. For example if a cloud deployment has multiple ceph deployed glance will not be able to use all those backends at once.

Consider the following use cases for providing multi-store backend support:

Deployer might want to provide different level of costing for different tiers of storage, i.e. One backend for SSDs and another for spindles. Customer may choose one of those based on his need.
Old storage is retired and deployer wants to have all new images being added to new storage and old storage will be operational until data is migrated.
Operator wants to differentiate the images from images added by user.
Different hypervisors provided from different backends (For example, Ceph, Cinder, VMware etc.).
Each site with their local backends which nova hosts are accessing directly (Ceph) and users can select the site where image will be stored.

Problem description

At the moment glance only supports a single store per scheme. So for example, if an operator wanted to configure the Ceph store (RBD) driver for 2 backend Ceph servers (1 per store), this is not possible today without substantial changes to the store driver code itself. Even if the store driver code was changed, the operator today would still have no means to upload or download image bits from a targeted store without using direct image URLs.

As a result, operators today needs to perform a number of manual steps in order to replicate or target image bits on backend glance stores. For example, in order to replicate a existing image’s bits to secondary storage of the same type / scheme as the primary:

It’s a manual out-of-band task to copy image bits to secondary storage.
The operator must manage store locations manually; there is no way to query the available stores to back an image’s bits in glance.
The operator must remember to register secondary location URL using the glance API.
Constructing the location URL by hand is error prone as some URLs are lengthy and complex. Moreover they require knowledge of the backing store in order to construct properly.

Also consider the case when a glance API consumer wants to download the image bits from a secondary backend location which was added out-of-band. Today the consumer must use the direct location URL which implies the consumer needs the logic necessary to translate that direct URL into a connection to the backend.

Proposed change

This spec proposes the following high level features to support multiple stores of the same scheme/type:

Support in the glance-api.conf and supporting logic to configure, manage and use multiple stores of the same or different type/scheme.
Enhance the image upload API to (optionally) support targeting a backend store for the image bits.
Enhance image download code to download image from any of the enabled backends.
Enhance the image import API to (optionally) support targeting a backend store to download the image bits from.
Additional metadata attribute(s) added to an image locations’ metadata properties related to backing stores.
A new REST API to list all configured stores known to the glance service.

** Config/glance-store changes**

In order to have multi-store support of same or different type/scheme we propose to deprecate ‘stores’, ‘default_store’ config option and add a new config option enabled_backends under DEFAULT section, ‘default_backend’ option under ‘glance_store’ section and ‘description’ option under each section of desired store of glance-api.conf. Operator can use enabled_backends to specify comma separated key:value of storage backends and its store type and then each backends will have a different section to specify related configuration options. If ‘default_backend’ is not explicitly set then appropriate exception will be raised which will prevent the service from starting.

The reason for deprecating ‘default_store’ config option is that glance verifies the default store at the time of service start and if it is not one of the listed store (file, http, rbd, swift, cinder, vmware or sheepdog) then it raises the error and prevents the service from start. This validation has been done using ‘choices’ attribute of ConfigOption object itself. After enabling support of multi-store it’s difficult to have this kind of validation check using ‘choices’ attribute will be difficult as default_store name will be store identifier and not actual store.

Consider the following example snippet of glance-api.conf which configures 3 stores for use:

[DEFAULT]
# list of enabled stores identified by their property group name
enabled_backends = fast:rbd, cheap:rbd, reliable:file

# the default store, if not set glance-api service will not start
default_backend = fast

# conf props for file system store instance
[reliable]
filesystem_store_datadir = /var/lib/images/data/
description = Reliable filesystem store
# etc..

# conf props for ceph store instance
[fast]
rbd_store_ceph_conf = /opt/ceph1/ceph.conf
description = Fast access to rbd store
# etc..

# conf props for ceph store instance
[cheap]
rbd_store_ceph_conf = /opt/ceph2/ceph.conf
description = Less expensive rbd store
# etc..

The example above is for Ceph, but it could apply to any storage backend. As shown in the example above the ‘enabled_backends’ property is a comma delimited list of store identifiers which are setup for glance. Each of the store identifiers defines a property group name which itself contains the store specific properties used to configure the store instance and a store type which will be used to register configuration options related to that store under the new configuration section defined using store identifiers. Under the covers the store identifier (aka id) maps to the store class used for the instance.

While ‘stores’ and ‘default_store’ are available an operator can leave the ‘enabled_backends’ property unset in which case multi-store support is not “enabled” and glance will work as-is. Once these config options are removed operator at least need to set one store in the ‘enabled_backends’ or else glance-api service will fail to start.

In order to accommodate multiple stores of the same scheme, the scheme_map used to index stores must also index by store identifier per scheme. The store identifier is (implicitly) the name of the conf group given in the glance-api.conf (e.g. reliable, fast and cheap from the example conf given above).

Consider the following indexing approach:

scheme_map[scheme][store_id]

Where ‘scheme’ is the scheme(s) supported by the store implementation and ‘store_id’ is the identifier for the store.

To maintain backward compatibility with old store API’s while they are not deprecated we also propose to add new store api’s like add, get, delete etc. to glance_store. The old store api’s will be removed when the deprecation period for ‘stores’ and ‘default_store’ config option is over.

API to list stores

To accommodate the management and discovery of known glance stores, glance should also provide a REST API which permits users to list all stores configured for glance. This is a read-only list of glance stores which includes the store identifier, description for each store and a flag which will tell whether a store is default or not.

NOTE: The list of glance stores is based on the glance-api.conf with respect to store configuration. Operators will not be able to configure new stores using the API. So in order to show the description in the response as mentioned earlier we propose to add new config option ‘description’ to each store with some default description and operator can set it as per their need.

As discussed herein, the store identifier can be used to address a particular instance of a store for applicable operations such as image upload and image import.

We propose the new REST resource be located at the following URI:

/v2/info/stores

More details on this API can be found in the REST API section of this spec.

Image upload API The existing image upload API permits an operator to upload image bits to the glance service. However today this API does not provide a means for the operator to target a specific store to back the image bits on (technically the v1 API allows you to specify a store scheme to target). This spec proposes the API be enhanced to permit an operator to specify which store will back the image bits being uploaded.

We propose a header field be used as a means to transport the identifier of the store to back the image bits. Again the identifier under the covers is the property group name of the respective glance store driver. When the image upload API is used to upload image bits, the glance logic will determine if the target store is specified, and if so the image bits will be added to the target store.

If no store is targeted in the upload request, the ‘default_backend’ is used to back the image bits.

Using this scheme operators will be able to specify which store they want an image bits to be backed on during an upload operation. We propose the ‘X-Image-Meta-Store’ header be used as the means to transport a target store identifier.

More details on this API can be found in the REST API section of this spec.

Image download If cloud get upgraded to use multi-store support from the single store then glance need to deal with the locations from old stores to new stores. For example, if operator is using rbd (say ceph) backend and now he has upgraded the environment and introduced two additional rbd stores as ceph1 and ceph2 with default store as ‘ceph1’ then somehow glance must be able to download the images from old stores (ceph).

With multiple backends available, Glance needs some enhancements so it can fulfill the current image download API contract. The download request will traverse through all the configured backends to look for the image. As we are adding store information in location metadata, at first it will look the image in the store which is set in the location metadata. If location metadata is not set then as per example from the config section above, if the location is rbd://<something>, then it will search the image in all available ceph stores i.e. ‘fast’ and ‘cheap’ in this case. This way user will be able to download his image from the old store even after cloud is upgraded to use multiple stores.

Image import API The existing image import API allows end users to import image from the staging area into glance backend. However today this API does not provide a means for the end user to target a specific store to import the image. This spec proposes the API be enhanced to permit an end user to specify which store will back the image being imported.

We propose a header field be used as a means to transport the identifier of the store to import the image. Again the identifier under the covers is the property group name of the respective glance store driver. When the image import API is used to import the image, the glance logic will determine if the target store header is specified, and if so the image will be imported to the target store.

If no store is targeted in the import request, the ‘default_backend’ is used to import the image.

Using this scheme end users will be able to specify which store they want an image to be imported during an import operation. We propose the ‘X-Image-Meta-Store’ header be used as the means to transport a target store identifier.

More details on this API can be found in the REST API section of this spec.

Store locations metadata attributes In the current glance API, a consumer of the glance API has no way to correlate an image location with its respective store other than by inspecting the image’s location URL. While this may work fine for many use cases, a more user friendly relation is needed in a multi-store environment; user’s need an explicit relation between an image location and its respective store (e.g. what store is this location backed on).

This spec proposes that when image bits are added with the image upload API, the core glance logic is responsible for adding a metadata attribute to the image location URL to reflect the backing store’s identifier. For example, if a user uploads an image to the ‘ceph1’ store in our example above, once the image bits are uploaded, the image location URL is added and in the URL’s metadata a property with the store’s identifier is added to the location metadata object.

With the store identifier in the image location URL metadata, we can expose it to the end user with a new attribute as ‘store’ in the image response so that it can be used for subsequent operations or within the API consumers logic.

The new image response with store attribute will be something like:

"size": 1234,
"store": ["reliable"],
"checksum": 1234567890,
"name": "Import image",
"status": active

Alternatives

Two major alternatives come to mind with respect to a multi-store approach in glance; multi-store using a service per store and per driver multi-store support.

Per store service In the service per store approach, each of the configured store instances would run as a separate process/service. As a result each service would have its own AMQP/RPC interface, own PID, etc.. To route requests to store services, we’d use a scheduler which itself is another process / service. This is the same approach used in cinder multi-backend support.

Although the service per store approach may be a longer term goal for glance, today we haven’t seen enough justification from our consumer base to justify the major refactoring/changes which are required to move glance to this model.

Therefore this spec proposes the multi-store approach outlined within this spec - let’s get an initial approximation multi-store working and gage our next steps based on consumer feedback in the community.

Per driver support Another potential approach would be to push the multi-store logic down within each glance store driver’s implementation. For example the store driver itself could contain logic to multiplex with multiple backends.

While this approach would work, it would require each store driver’s implementation to change and would result in suboptimal reuse of code.

This spec proposes a multi-store approach which has little or no impact to each store driver; they can be used as-is in a multi-store implementation. By pushing the logic to support multiple stores of the same scheme up into the core of glance, we get maximal reuse and store driver implementations needn’t be concerned with such logic.

Data model impact

This spec does not propose any changes to the data model. Rather the approach herein can maintain all new stateful data either in memory or within the existing schema used by glance. However store identifier will be stored as a metadata in the location object.

REST API impact

This spec proposes the following API changes:

New API

List all stores known to the glance service.

Modified APIs

Import an image.
Upload an image file.
Get image details.

Common Response Codes

Create Success: 201 Created
Modify Success: 200 OK
Delete Success: 204 No Content
Failure: 400 Bad Request with details.
Forbidden: 403 Forbidden

API Version

All URLS will be under the v2 Glance API. If it is not explicitly specified assume /v2/<url>

[New API] List stores

List all stores known to the glance service:

GET /v2/info/stores

This API takes no query parameters and when authorized returns a listing of all stores known to the glance service. The stores known to glance are those which have been configured in the glance-api.conf and have been loaded during glance startup. The response body payload is JSON and contains a JSON object per store. Each store JSON object contains the store’s identifier (id), description and if a particular store is a default store the it will have a flag telling store is default. For example:

{
   "stores":[
      {
         "id":"reliable",
         "description": "Reliable filesystem store"
      },
      {
         "id":"fast",
         "description": "Fast access to rbd store",
         "default": true
      },
      {
         "id":"cheap",
         "description": "Less expensive rbd store"
      }
   ]
}

Response codes:

200 – Upon authorization and successful request. The response body contains the JSON payload with the known stores.

[Modified API] Create image We propose to add an ‘OpenStack-image-store-ids’ header to the image-create response which would have the available stores. Using this user can decide which store he needs to upload/import his image and wouldn’t have to make a separate get-stores call.

New response headers

OpenStack-image-store-ids

The value of this header will be a comma-separated list of stores available. For example,

OpenStack-image-store-ids: fast, cheap, reliable

[Modified API] Upload image binary data

Get image binary data:

PUT /v2/images/{image_id}/file

This modifies the existing REST API to support a new header field which is optional and if present specifies the store id to upload the image data to. For backwards compatibility, if the header is not specified the store specified as the default (e.g. default_store) is used to upload the image to.

New header fields:

X-Image-Meta-Store – If present contains the store id to upload the image binary data to.

New / changed response codes:

400 – If the X-Image-Meta-Store header is present, but specifies a store id for a store that doesn’t exist by that id.

Example curl usage:

curl -i -X PUT -H "X-Auth-Token: $token" -H "X-Image-Meta-Store:
    ceph1" -H "Content-Type: application/octet-stream"
    -d @/home/glance/ubuntu-12.10.qcow2
    $image_url/v2/images/{image_id}/file

[Modified API] Get image details

Get the details for a specified image:

GET /v2/images/{image_id}

Although this spec does not impose any changes on the glance API layer, this call will now shows the location’s store id. In case if there are multiple locations, then all locations will be displayed as comma separated list. The new image response with store attribute will be something like:

"size": 1234,
"store": ["reliable"],
"checksum": 1234567890,
"name": "Import image",
"status": active

[Modified API] Import image to the backend

Import image to the backend:

POST /v2/images/{image_id}/import

This modifies the existing REST API to support a new header field which is optional and if present specifies the store id to import the image data to. For backwards compatibility, if the header is not specified the store specified as the default (e.g. default_store) is used to import the image to.

New header fields:

X-Image-Meta-Store – If present contains the store id to upload the image binary data to.

New / changed response codes:

400 – If the X-Image-Meta-Store header is present, but specifies a store id for a store that doesn’t exist by that id.

Example curl usage:

curl -i -X PUT -H "X-Auth-Token: $token" -H "X-Image-Meta-Store:
    ceph1" -H "Content-Type: application/json"
    -d '{"method":{"name":"glance-direct"}}'
    $image_url/v2/images/{image_id}/import

Security impact

None

Notifications impact

Need to add ‘stores’ field in the notification response as one of the use case of this proposal is offering different price tiers for storage which will help systems which consumes notifications to perform the billing.

Other end user impact

This proposal introduces a few other user impacts worth noting.

Glance client Ideally the glance client (CLI + REST client) should be updated in accordance with this spec. Notably:

CLI / API support for listing glance stores.
CLI / API support for specifying a store id on upload/import.

Configuration Deployers will need to be aware of the configuration aspects for glance multi-store. From a conf point of view, configuring multi-store for glance will look very much (from a high level) like configuring cinder for multi-backend. The conf file specifics will need to be documented.

Performance Impact

A very little hit on the performance of downloading the image from the old stores. For example, if existing user is using single rbd (say ceph) backend and now he has upgraded the environment and introduced two additional rbd stores as ceph1 and ceph2 with default store as ‘ceph1’ then if image which needs to be download from old store (ceph) will take some time as it needs to be looked in all the enabled backends.

Other deployer impact

Once merged, glance multi-store is not enabled unless the deployer configures the enabled_backends property in glance-api.conf and thus is backwards compatible out-of-the-box. When multi-store is disabled, v2 API users can use the list stores API and will retrieve a list of the current stores configured (of course only 1 store per scheme).

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: abhishek-kekane
Other contributors:: None

Work Items

Implementation tasks may consist of:

Conf support for multi-store.
Multi-store loading and indexing.
List stores API.
Location URL metadata store id on upload.
Add new ‘store’ attribute in image response.
Image upload with store targeting.
Image import with store targeting.
Multi-store delete and other store access codepaths.
Add python-glanceclient support

Dependencies

None

Testing

Need to add new tempest tests to verify multi-store support

Documentation Impact

As mentioned in the ‘work items’ section, we’ll need to ensure the glance docs are update for:

The new list stores REST API.
New header field for image upload.
New header field for image import.
New store id in image response.
Overall glance multi-store documentation to educate deployers on the feature and how it’s used.

References

https://review.openstack.org/#/c/150967

Secure Hash Algorithm Support

Tue, 07 Aug 2018 00:00:00

https://blueprints.launchpad.net/glance/+spec/multihash

This spec provides a future-proof mechanism for providing a secure SHA512 hash for each image.

Problem description

The hash of an image can be obtained from the checksum field in the image metadata. Currently, this checksum field holds a md5 hexdigest value. While md5 can still be used to verify data integrity against unintentional corruption, it should not be used to verify data integrity against tampering. A more collision-resistant hashing algorithm should be used for broader data integrity coverage.

Proposed change

This spec proposes adding two new fields to the image metadata. The proposed keys for this change would be os_hash_algo and os_hash_value. The proposed value is a hash value that is formatted as a sha512 hexdigest. The hexdigest will be obtained using the current python hashlib library. os_hash_algo will be sha3_512 and os_hash_value will be the hex. At this point in time, the config will default to sha3_512, but this change will allow us to alter allowed hashing algorithms at any time. The os_hash_algo field will also allow for users to dynamically pass this value to their python code. We will leave MD5 calculations and checksums for backwards compatibility.

Adding a new image will set all of checksum, os_hash_algo and os_hash_value fields.

Requesting the image metadata of an image without the os_hash_algo/os_hash_value value set, will result in a null for the os_hash_algo/os_hash_value field in the metadata response.

Alternatives

Update the value of the checksum field with the hexdigest of a different hashing algorithm. This will likely break client-side checksum verification code.
Add only the new os_hash_value field. While it is a simpler change, it is a less future proof approach because it ties the sha512 algorithm to the os_hash_value property in the same way that the md5 algorithm is tied to the current checksum property. When a sha512 collision is produced, there will have to be a new spec in to add yet another checksum field.

The approach described in this spec of using an additional os_hash_algo field will allow changing the hash algorithm without adding a new field or breaking the API contract. All we’ll have to do is update the algorithm used and put its name in the os_hash_algo property, and then any image consumer will know how to interpret what’s in the os_hash_value property.

It is worth noting that the Glance implementation of image signature validation gives us a precedent for having a value in one property (img_signature) and the name of the algorithm used in another property (img_signature_hash_method) [1]. Thus the proposal in this spec is completely consistent with a related Glance workflow.
Use the hexdigest of a different algorithm. Tests using hashlib’s sha256 and sha512 algorithms consistently yielded faster times for sha512 (SEE: Performance impact). The implementation of the sha512 algorithm within hashlib demonstrates reasonable performance and should be considered collision-resistant for many years.
Implement a single multihash field in the image metadata in which we calculate the SHA512 value. The single field will contain the coded algorithm name and hash [0]. The advantage is that it is only one field being added to the data model and schema. However, having only one field would make it difficult to use the hash quickly as the end user would be required to decode what algorithm is being used before verifying the hash.

Data model impact

Triggers: None
Expand: Two new columns (os_hash_algo & os_hash_value) with type string/varchar (defaulting to null) will be added to the images table. We will create an index similar to the one on checksum as well.
Migrate: None
Contract: None
Conflicts: None

REST API impact

Two new fields (os_hash_algo & os_hash_value) will exist in requests for the image metadata.

Security impact

A new more collision-resistant hash will be returned in addition to the current checksum.

Notification impact

None

Other end user impact

None

Performance impact

New image uploads (and the first download of previously uploaded images) will take an additional amount of time to calculate the sha512 checksum:

5G binary blob of random data: * md5: ~9s * sha256: ~22s * sha512: ~14s * 1Gbps line speed upload: 42s

1.5G Ubuntu 16.04 cloud image: * md5: ~2.9s * sha256: ~7.2s * sha512: ~4.6s * 1Gbps line speed upload: 12s

555M Debian 8 cloud image: * md5: ~1.0 * sha256: ~2.5 * sha512: ~1.6 * 1Gbps line speed upload: 4.5s

Note: SHA512 has been selected and should have minimal impact on overall upload time with regards to the entire process.

Test Code:

#!/usr/bin/env python3

import hashlib
import time


def runtime(f):
    def wrapper(*args, **kwargs):
        start = time.time()
        f(*args, **kwargs)
        print("Time elapsed: %s" % (time.time() - start))
    return wrapper


@runtime
def checksum(filename, algorithm):
    algorithms = {"md5": hashlib.md5,
                  "256": hashlib.sha256,
                  "512": hashlib.sha512,
                  }
    with open(filename, "rb") as f:
        m = algorithms[algorithm]()
        for chunk in iter(lambda: f.read(65536), ''):
            m.update(chunk)
    print("%s: %s" % (algorithm, m.hexdigest()))

checksum("fake.img", "512")
checksum("fake.img", "256")
checksum("fake.img", "md5")
checksum("fake.img", "256")
checksum("fake.img", "md5")
checksum("fake.img", "512")

Developer impact

Any future checksum verification code should use the os_hash_algo & os_hash_value fields. Fallback to the checksum field if not properly populated.

Implementation

Assignee(s)

Primary assignee: Scott McClymont

Other contributors:

Work Items

Add tests
Update the db to add os_hash_algo & os_hash_value columns to the images table (including expand, migrate, contract, and monolith code)
Update the sections of code that calculate the checksum to also calculate os_hash_algo & os_hash_value (includes calculation on upload)
Discuss updating on download os_hash_algo & os_hash_value when value is null
Update internal checksum verification code to use the os_hash_value field and fallback to checksum field when not present
Update glance client
Update docs
Add the os_hash_algo value to the discovery API if the API is ready

Dependencies

None

Testing

Update the tests to verify proper population of image properties

References

Spec Lite: Remove the Images API v1

Tue, 07 Aug 2018 00:00:00

problem:: The Images API verison 1 was DEPRECATED in the Newton release: https://docs.openstack.org/releasenotes/glance/newton.html It needs to be removed from the codebase.
dependencies:: The copy_from import-method must be implemented and the Images API version 2.6 must be CURRENT. (That’s because the key requested feature missing from Images v2 is a copy-from functionality.)
solution:: This will require some preliminary work before the code is removed, for example, removing any Tempest tests that use the v1 API. These will be noted as “Work Items” on the Blueprint.
how:: Remove all v1 tests, remove v1 endpoints from Images API, remove glance-cache-manage command that is still relying fully on v1 API. Gradually clean out the v1 code base that’s left behind.
impacts:: DocImpact (All v1 docs will have to be removed)
timeline:: target for the Q-3 milestone (week of 22 January 2017)
link:: https://blueprints.launchpad.net/glance/+spec/remove-v1
reviewers:: all core reviewers
assignee:: jokke

Spec Lite: Deprecate store_capabilities_update_min_interval

Tue, 07 Aug 2018 00:00:00

project:

glance_store

problem:

The configuration option store_capabilities_update_min_interval is confusing because no existing stores implement the StoreCapability.update_capabilities() method. This has come up in the context of nfs being used for the filesystem backend. If nfs is not ready for writing when the glance api starts, glance will mark the filesystem as not writeable. Operators have tried to get around this problem by setting a non-zero positive value for this option only to find that it doesn’t work.

solution:

Use oslo.config to mark the option as ‘deprecated’ with an appropriate note. Option will be deprecated in Rocky for removal in ‘S’.

alternatives:

An alternative would be to rewrite the option help text to make it clear that there is no current store for which the option is actionable, but that a framework is in place through which dynamic capability determination could be implemented. Currently a debug level message to this effect is logged on store startup although it is not obvious that the message is related to the ineffectiveness of setting the store_capabilities_update_min_interval option. The message is: “Store %s doesn’t support updating dynamic storage capabilities. Please overwrite ‘update_capabilities’ method of the store to implement updating logics if needed.” (This message is logged independently of setting the option.)

The advantage to this approach is that the framework would be available to someone who wanted to implement dynamic updates for a store, and the option would not have to be re-introduced.

impacts:

None

timeline:

Rocky milestone 2

assignee:

rosmaita

Support revert pending delete image

Wed, 20 Jun 2018 00:00:00

https://blueprints.launchpad.net/glance/+spec/pending-delete-rollback

Glance support soft delete images. If this feature is enabled, when users delete an image, the image and its locations will first be in a special pending_delete status that is not displayed in the API response. Then the image will be deleted by glance-scrubber process in period. But now, there is no way to revert/rollback the pending_delete images to active.

Problem description

Delayed_delete feature is usually used when the image is too large to delete at once. With this feature, then the image data will not be deleted at once and will be cleaned by glance-scrubber process. The problem is that there is no way to revert the delete action if the image is deleted by mistake. The only way admin operator can do is to wait until the image data is deleted and then reupload image data again.

Proposed change

This proposal aims to recover an image which is in pending_delete state so as to provide the revert capability for the purposes of allowing emergency operational action to recover an accidental delete. It is important to keep in mind, however, that whether the recovery of a particular image will be possible or not depends upon Glance configuration option settings and quick operator action.

Since the pending_delete image will be only deleted by glance-scrubber and it’s an admin action, there is no need to expose a new API. A better way is to enhance glance-scrubber to support restoring the image from pending_delete status to active.

A new parameter called –restore will be added to glance-scrubber command. The usage is like: glance-scrubber –restore <image_id>. glance-scrubber first checks to see if the scrubber process is running, if so, an error message that there is a scrubber currently running and you must kill it first & scrubber terminates will be raised to admin. If not, scrubber will switch image status from pending_delete to active.

Please be sure that the glance-scrubber daemon is killed before restore the pending_delete image to avoid image data inconsistency. After restoring the image, glance-scrubber daemon can be restarted.

Limitations

This is intended as an emergency operation for the use case where an operator inadvertently deletes an important image and immediately realizes the mistake and takes action within the scrub_time seconds set in the glance-api.conf file. The pending-delete status is a purely internal Glance status and the image still shows as being in deleted status in API responses. Thus there is no way to tell via the API whether an image may be restorable or not.

Further, when the image is restored, some of its metadata is irrecoverable. Any additional properties, tags, or members will not be restored. In other words, this is purely a possible data recovery operation, not a full image restore.

Alternatives

The alternative way which is not recommend is to create a new API to revert the pending_delete images::

POST /v2/images/{images_id}/actions/revert

The response body could be like::

Response: 200 OK
{
    "status":"active",
    "name":"cirros-0.3.1-x86_64-uec",
    "tags":[
    ],
    "kernel_id":"be50418b-a03c-4947-9122-b80a57f47ac4",
    "container_format":"ami",
    "created_at":"2017-09-11T08:42:14Z",
    "ramdisk_id":"e1256074-9f7b-4067-8356-4a5759c1db11",
    "disk_format":"ami",
    "updated_at":"2017-09-11T08:42:16Z",
    "visibility":"public",
    "self":"/v2/images/26c16e07-24ca-4abc-a523-bec068012363",
    "protected":false,
    "id":"26c16e07-24ca-4abc-a523-bec068012363",
    "file":"/v2/images/26c16e07-24ca-4abc-a523-bec068012363/file",
    "checksum":"f8a2eeee2dc65b3d9b6e63678955bd83",
    "min_disk":0,
    "size":25165824,
    "min_ram":0,
    "schema":"/v2/schemas/image"
}

Data model impact

Allow image status changing from pending_delete to active.

REST API impact

None.

Security impact

This is an administrator action. No security impact at all.

Notifications impact

None.

Other end user impact

There is no impact for non-admin users. For administrators, they’ll have the ability to rollback the image’s status from pending_delete to active by glance-scrubber tool.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

wangxiyuan(wangxiyuan@huawei.com)

Work Items

change glance-scrubber to include the –restore <image_id> option.
change the image status transition to allow: pending_delete -> active
Update the related documentation and test.
Release note should be added.

Dependencies

None

Testing

Related unit test should be added.

Documentation Impact

Related doc should be updated.

References

None.

Spec Lite: Deprecate owner_is_tenant

Wed, 20 Jun 2018 00:00:00

project:

glance

problem:

The owner_is_tenant option, which is True by default, allows an operator to run Glance in a nonstandard configuration where the image owner is the user who created the image. In all other OpenStack services, resources are owned by the project (as they are in Glance when the default setting is used).

A survey of operators conducted in March 2017 indicated that no operators who responded (14) are using this option. As it is little used, results in a nonstandard OpenStack experience, and complicates the Glance code, the option should be deprecated in Rocky. Following the standard OpenStack deprecation policy, it should be removed early in the ‘S’ cycle.

solution:

Use the oslo.config facilities to mark the option as deprecated. As it appears that this option is used only in its default setting of True, no migration path is proposed.

impacts:

None

timeline:

R-1

assignee:

rosmaita

Spec Lite: Glance cluster awareness

Wed, 06 Jun 2018 00:00:00

project:: glance
problem:: Individual glance nodes are fully un-aware that there might be other nodes operating in the same space. This lack of communication between the nodes causes issues separating workers and operating in different locations.
solution:: We have message queues already utilized in notifications and other projects. Lets expand that for communications between different glance-api nodes to gain better operational efficiency.
impacts:: DocImpact

alternatives:: We could use something else than the message queues but effectively that would end up recreating something like registry but just as a message broker.
timeline:: Possible initial functionality in Rocky, more use cases covered S and beyond. This can be expanded fairly easily so we can proceed as time permits.
assignee:: jokke

Semver Utility for DB storage

Sun, 13 May 2018 00:00:00

https://blueprints.launchpad.net/glance/+spec/semver-support

Working with versions of various objects is a common problem, which already has a number of market-adopted solutions. One of these solutions is Semantic Versioning - a system of rules and requirements for assigning version numbers to software components and other objects. One can find the specification for SemVer freely at semver.org <http://semver.org>

It is proposed to add support of the semantic versioning concept into Glance, according to the version 2.0.0 of the specification [1], so Glance objects (starting from Artifacts, but probably including Images in future) may be properly versioned.

Problem description

Semantic versioning provides the ability to compare two or more objects based on the version with which they are associated. According to the specification [1] objects are compared first by their major versions, then minor versions, then patch versions, also there is a concept of “pre-release” versions (alphas, betas, release candidates (RC) etc) which should always be considered “lower” then the “released” version with the same values of numeric versions.

For example, 1.2.2 < 1.2.3-beta < 1.2.3

If we want to store versioned objects in the catalog (this may be applied to images, artifacts and other entities), then we need to be able to execute this kind of semantic comparison for large amounts of entities.

So, the comparison should be made not only in memory, but at the database as well, and there is no generic datatype in modern RDBMs to store this kind of versioning information. So, a method for storing easily-sortable version identifiers should be introduced.

Proposed change

First of all, Glance has to adopt some utility to parse strings which contain version information, verify their compliance with the specification and properly process the version objects in memory. There is a number of mature libraries which have this functionality and there is no need to re-implement them. After some research it has been suggested to use “semantic_version” library which is available at pypi [2]. This library is not present in OpenStack global requirements, so a patchset [3] has been submitted to add it there.

To be able to sort these version objects in the database it is required to convert them into some generic comparable data type. Due to the nature of version information (fixed numeric components for major-minor-patch part) and arbitrary sequences of alphanumeric strings for pre-release and metadata labels it is suggested to store them separately as three database fields: one for the numeric part, another for the pre-release label and the last one for build metadata.

Three numeric components (major, minor and patch) may be converted into a single unsigned 64-bit integer number: first 16 bits of this number will be allocated to store major revision, next 16 bits - for minor revision, next 16 bits - for patch revision. Remaining 16 bits will be used to store the release type flag (to make sure that the final release had higher precedence then pre-releases) and may be reserved for future improvement and storing additional information which is irrelevant for semantic versioning but may have some other meanings (see Alternatives section below for more details).

The labels of pre-release version should be stored independently from the numeric part as a regular string, because - according to the semver spec - they are to be compared according to regular alphanumeric comparison only if the numeric parts of the versions are identical.

So, these two values - long number and a string - may be combined into a single composite index in the database, which will provide efficient capabilities to sort and filter objects with the versions assigned.

However, there is one important difference between the semver requirement and simple comparison of alphanumeric strings: semver requires that the labels are compared “per component” (where “component” is a dot-separated part of the label), and the components which consists only of digits are to be treated as integers rather then ASCII strings. For example, version “1.0.0-alpha.4.foo” should have lower precedence then “1.0.0-alpha.10.bar”, because their numeric components are equal, and the labels have identical first component (‘alpha’) but differs in the second (“4” vs “10”), and 4 is less then 10. But the labels are compared as string database fields, the precedence will be wrong as “alpha.4.foo” is lexically greater then “alpha.10.bar” (due to “4” being greater then “1”).

To solve this problem it is suggested to add one constraint to this semver implementation: to limit the maximum length of numeric components in the pre- release label to a reasonably low value (say, 6 characters) and add extra leading zeros to these components when saving them to database.

In this case the “alpha.4.foo” label from the example above will become “alpha.000004.foo”, and “alpha.10.foo” - “alpha.000010.foo”. ASCII-based comparison of these strings will give the results which are consistent with the requirements of semver. Later, when these values are read from the database the leading zeros may be removed so the labels look fine again.

This applies only to the pre-release label part. Build metadata (the part which is separated by the ‘+’ character) does not take part in the precedence resolution, so it neither has to be part of the database index nor has to be pre-processed in anyway.

It is suggested to create custom composite field for SQLAlchemy which will encapsulate the above described logic (converting from semantic version into 3 database-friendly values and back) and will be usable for building version-aware model classes.

Alternatives

Semantic Versioning is not the only specification which defines the format for version string. There is another standard - PEP440 - which describes a scheme for identifying versions of Python software distributions [4]. It shares some common features with Semantic Versioning but has different and a bit more complicated notation.

Besides slightly different syntax (it just concatenates pre-release segments to the right of release number, while semver separates them with a dash), it puts extra constraints on what the pre-release label may contain. In semver, pre-release label may contain arbitrary alphanumeric characters, while in PEP440 they may be only be ‘a’, ‘b’ or ‘rc’ followed by a number. This could theoretically allow to store the pre-release component as part of the same 64-bit long database field which is used to store the release number (e.g. the release type flag takes 2 bits, and remaining 14 bits are left for the number of the pre-release build) - however this significantly decreases the flexibility of the pre-release version field.

Also, PEP440 adds more additional entities: it has a concept of development builds (being one additional special segment which goes after the pre-release segments), Epochs (which precedes the build number), local version (which is actually similar to build metadata of semantic versioning but has different purpose and also takes part in precedence resolution by following about the same rules as arbitrary pre-release label of semantic versioning) etc. Also, unlike semantic versioning PEP440 does not have any limits on the amount of numeric components in the build number: so, it may be anything from simple “1” to “1.2.3.4.5.6.7.8.9.10” and beyond. This, of course, gives more flexibility and power, but may not be easily mapped to efficient database storage.

Which is more important, PEP440 is a standard which is native to Python world, but is not known outside, while the purpose of Glance Artifacts is to be as generic as possible in terms of the nature of its objects. This means that the users of the artifacts are not restricted to be Python developers only: they may not be the developers at all. So, following easier and more generic standard seems preferable.

There is one more standard which stands between semver and pep440. It is called “Linux Compatible Semantic Versioning 3.0.0”, is a fork of regular semver (its 2.0 version) and is developed within OpenStack community [5]. It tries to blend regular semver with versions of Linux Distribution packages and uses some concepts of pep440 for it.

This notation is easier to map to the database type, however it is still local to relatively small community of developers (OpenStack developers in this case), so more generic and widely adopted standard as semver seems more preferable.

However we are not limited to having only a single versioning notation. In future we may add support for extra schemas, including some subset of pep440 or Linux Compatible Semantic Versioning. This may be implemented as part of further Artifact Repository roadmap or other activities. This particular spec leaves this out of scope and focuses only on semver implementation.

Data model impact

None: this spec does not cover any actual database changes, it just describes the utility which will allow to operate with semver objects and convert them to data which may be usable for DB storage - and back.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

The proposed change does not affect existing code in any sense.

Other deployer impact

This spec assumes that [3] is merged, i.e. the semantic_version library is added to the global requirements.

Developer impact

The usage of the lib should be documented for developers, so they may efficiently use it in their code.

Implementation

Assignee(s)

Primary assignee:: ativelkov

Reviewers

Core reviewer(s):: jokke
Other reviewer(s):: ivasilevskaya mfedosin travis-tripp icordasc

Work Items

Initial implementation of the feature may be done in a single changeset. However it seems preferable to add this support to semantic_version library [2] and remove it from glance codebase aftwerwards. If the maintainer of the library does not accept this functionality (or if we decide to add support for more versioning notations later) then this code may be transferred to some common OpenStack library, such as Oslo.

After this feature is implemented we should continue the work to add support for other versioning schemas, such as pep440, Linux Compatible Semantic Versioning and others. These should be added as independent features covered by separate specs.

Dependencies

None

Testing

A unit test should be added demonstrating the data structure usage, comparison, string parsing and conversion operation to DB type (long)

Documentation Impact

Developers’ guide has to be updated to hint the developers on how to properly use the library in their code.

References

[1] http://semver.org [2] https://pypi.org/project/semantic_version/ [3] https://review.openstack.org/#/c/151466/ [4] https://www.python.org/dev/peps/pep-0440/

Taskflow Integration

Sun, 13 May 2018 00:00:00

https://blueprints.launchpad.net/glance/+spec/taskflow-integration

Add a new task executor using the taskflow library.

“TaskFlow is a Python library for OpenStack (and other projects) that helps make task execution easy, consistent, scalable and reliable. It allows the creation of lightweight task objects and/or functions that are combined together into flows (aka: workflows) in a declarative manner.”: Taskflow Wiki

Problem description

Glance currently comes with an eventlet executor which is not easily extensible by nature and doesn’t support many features that taskflow support out of the box such as execution on remote workers.

Proposed change

We propose here a new executor (self-contained) implementing the interfaces of the executor API. This executor will route the tasks to the eventlet executor of taskflow. We will be using the Taskflow Green Thread Pool Executor which ensures that eventlet green threads are used when using the taskflow engine. The initial implementation should provide the same result as the eventlet executor already contained in Glance. However, subsequent blueprints will come to leverage more advanced functionalities.

Alternatives

Use the existing eventlet executor. This approach is likely to become rewriting taskflow.

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

None with this spec. However, in the future, it will be possible to plug-in to the taskflow notification engine and potentially drop its messages onto a notification bus.

Other end user impact

The end user should be able to transparently execute tasks with all the executors.

Performance Impact

For serial execution of tasks, the performance of the evenlet executor and the taskflow executor should be close to similar. However, for more complex workflows, we should be able to achieve performance improvements by parallelizing the work, and also distributing it with taskflow.

Other deployer impact

The deployer will have to update glance-api.conf and specify ‘taskflow’ as the executor. Also, it will be possible to choose the engine mode ‘serial’ or ‘parallel’ and the maximum number of workers. Remote workers (not supported with this spec) will require more infrastructure.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: arnaud
Other contributors:: harlowja

Reviewers

Core reviewer(s):: nikhil, zhiyan
Other reviewer(s):: harlowja

Work Items

Implementation of the taskflow executor with unit tests.

Dependencies

None.

Testing

It will be possible to add a specifc configuration in DevStack leverage this new executor. Ultimately, this is the executor that should be used at the gate.

Documentation Impact

Initially, the documentation should explain how to configure glance-api.conf and what is taskflow. Later on, it should be explained how to achieve more complex scenario.

References

https://wiki.openstack.org/wiki/TaskFlow
https://github.com/openstack/taskflow
https://pypi.org/project/taskflow/
https://review.openstack.org/#/c/85211/14 (needs to be rebased once the spec is approved)
Discussions at the OpenStack Summit in Paris

Rocky Review Priorities

Mon, 26 Mar 2018 00:00:00

The Rocky review priorities were discussed at the Dublin PTG and refined during the spec review process early in the Rocky cycle. The preliminary list was maintained on the Rocky PTG priorities etherpad.

This list is an estimate of what the Glance project team can accomplish during the Rocky cycle based on our developers’ estimates of how much time they can commit to Glance. If additional resources become available, it’s possible that some of the Untargeted Approved Specifications could be added to Rocky.

The following list is roughly in priority order (highest to lowest).

TODO: add priorities

Important dates

This is an abbreviated list focused on dates relevant to Glance. See Rocky Release Schedule for the complete list for OpenStack.

Countdown	Week of	What
R-19	Apr 16	Rocky-1 milestone Friday, April 20: Bug scrub
R-14	May 21	OpenStack Summit
R-12	Jun 04	Rocky-2 milestone Spec Freeze Friday, June 8: Bug scrub
R-6	Jul 16	glance_store Rocky release
R-5	Jul 23	Rocky-3 milestone glanceclient Rocky release Feature Freeze Rocky community goals completed Friday, July 27: Bug scrub
R-3	Aug 06	Glance Release Candidate 1 String freeze
R-1	Aug 20	Final Glance Release Candidates
R-0	Aug 27	Glance Rocky Release

Artifact Repository

Thu, 08 Mar 2018 00:00:00

https://blueprints.launchpad.net/glance/+spec/artifact-repository

Extend Glance’s functionality to store not only the VM images but any other artifacts, i.e binary objects accompanied with composite metadata.

Glance should become a catalog of such artifacts, providing capabilities to store, search and retrieve their artifacts, their metadata and associated binary objects.

Problem description

Various OpenStack services often need to catalog different objects they use to operate. Such objects may contain various data as well as metadata needed for identification and description.

Images used by Nova to run the VMs are just the best known examples of such objects. Other examples include Heat templates, Solum Plan Files, Mistral Workbooks, Murano Application Packages etc.

“To catalog” means to have the following functionality:

To store the object reliably
To guarantee its immutabity once it is stored
To provide search capabilities to find the objects in catalog
To return the detailed info about the requested object
To allow fetching the object for usage by a Service
To control the access to the object: enforce usage and modification policies, publication scenarios etc
To manage the object lifecyle

Obviously, this set of functionality is common for different kinds of objects, and is usually unrelated to the primary mission of respective projects using these objects. That is why it is suggested to have a dedicated service which will provide the catalog functionality for other OpenStack services. As Glance already serves as a catalog of Images for Nova, it is suggested to extend its mission so it may serve as a catalog for other services as well.

The objects stored in the catalog are called Artifacts, and the catalog itself – Artifact Repository.

Proposed change

It is proposed to add Artifact Repository functionality to Glance, i.e. create a set of APIs, data models and services which will allow to store artifacts of different types in Glance.

It is required to introduce the following concepts and behaviors:

Artifact

Artifact is a block of binary data stored with a description – metadata.

All artifacts should have the following properties:

Artifacts shall be consumable by OpenStack services. Like Nova uses Images to run the VMs, so will other existing or upcoming OpenStack services use different kinds of Artifacts. Ability to be consumed by the services is important: data chunks which are not consumable by other services (e.g. user data) are NOT artifacts. If one is looking to store such unrelated data then Swift (or any other generic object storage) should be a proper choice.
Artifacts description (metadata) should have a well-defined, discoverable structure. Artifacts’ metadata should contain only the fixed set of fields. Different types of artifacts may have different sets of allowed fields, but attaching unexpected values should be prohibited.
Artifacts are immutable. I.e. once the artifact is made available, there should be no way to change its binary content and at least some meaningful parts of its metadata. So once the consumer has discovered an artifact, its content is guaranteed to remain the same during the whole life of the object, and no other object may get the same ID. The artifact may be deleted, but any modifications introduced will result in creating a new object with a new ID.

Note: To properly handle modification scenarios it is advised not to delete the existing artifact, but to create the new version of it. See the “Artifact Versions” section below

Artifact types

Type of the Artifact defines its purpose and contents. It is defined by the OpenStack service which is supposed to use the artifact, however the mapping between an Artifact type and a Service is not one-to-one: a single service may utilise a number of different artifact types, and in some cases the same types of artifact may be consumed by different services.

Examples of Artifact types include Images (consumed by Nova), Templates (consumed by Heat and other services working on top of the Heat, such as Solum or Murano), Solum plan files (consumed by Solum), Application Packages (consumed by Murano Application Catalog) etc. The type of artifact defines the structure of both its binary data and metadata. Some of the repository operations may also be defined per artifact type and may have different behaviour for different types.

The new types of artifacts may be added. This will be implemented using a plugin system. Each plugin defines one or more of the artifact types and completely describes their data structure, metadata-scheme and actions.

Artifact types supported by the particular Glance repository, as well as the structure and available actions of a particular type should be discoverable via a set of special API calls.

Artifact plugin

A plugin is a python module which defines one or more Artifact Types, with their custom metadata fields, BLOB kinds, custom validation logic etc (see respective sections below).

Glance uses stevedore library to discover the artifact types defined in the plugins. Each endpoint defined in the plugin identifies the Artifact Type being exported. The name of the each endpoint should be equal to the name of the artifact type. All the endpoints should belong to glance.artifacts.types namespace.

Each Artifact Type has a version assigned to it. From code point of view each version of the same Artifact Type is a different python class with different value of version-identifying attribute. A single plugin may contain definitions of several versions of the same Artifact Type. Also, different versions of the same Artifact Type may be defined in different plugins: it is up to the plugin developers to decide which composition suits their needs better.

If the plugin contains a single version of some artifact type then the corresponding endpoint should export the class definition of this particular type with type name matching the endpoint name.

If the plugin contains multiple versions of the artifact type then the corresponding endpoint should export a list of class definitions, and each of these classes should have distinct type version values and identical type name, all equal to the name of endpoint.

A given version of specific artifact type may be defined only in a single plugin, i.e. if there may be no second plugin defining the same artifact type with the same version.

At startup Glance will inspect all the enabled plugins and ensure that no artifact-type conflict occurs. If several active plugins define the same artifact type and version, the critical exception occures and artifact’s server does not start. It is up to the operator to resolve the conflict and uninstall the unnecessary plugin(s).

To enable a plugin the cloud operator has to install the python module into the python environment of the node which runs Glance and make appropriate changes into the configuration files.

Only the types from explicitly enabled plugins are available for usage.

Metadata Structure

The metadata describing the artifacts has to have a predefined and discoverable structure. Some of the metadata fields are common for all the types of Artifacts, while the others are defined for particular types only.

Some of the fields are immutable: i.e. their value can be changed only while the artifact is being interactively created (see below) and is fixed since the moment once the artifact is published in the repository. Other fields may be modified at any time. Also there are system fields which cannot be modified at any time - their value is set automatically by the repository.

**Common metadata fields**
Name	Access	Required	Data type	Description
ID	System	N/A	UUID String	Identifies the artifact in the repository. Is assigned automatically once the artifact’s draft is registered. Is not supposed to be explicitly set by the user, however administrators may override this value if needed.
Type	Immutable	Yes	Alphanumeric string, valid artifact type name	Identifies the type of artifact, and so its data and metadata structure should be one of the types supported by the repository (handled by the active plugins)
TypeVersion	System	N/A	String following SemVer notation	A system field defining the version of artifact type, i.e. the version of the plugin which contains the artifact type definition. The value is set automatically when the artifact record is created. In future, this may be used to do automatic or manual migrations of artifacts data and metadata.
Name	Immutable	Yes	Alphanumeric string, 255 characters max	Defines the name of the artifact. In the future certain uniquiness of (name, version) will be guaranteed, so that it will be capable of identifying the artifact within a catalog.
Version	Immutable	Yes	String following the SemVer notation	Defines the version of the artifact. See “Artifact versioning” section below for the details.
Description	Mutable	No	String	Arbitrary text describing the artifact, 255 characters max
Tags	Mutable	No	Set of alphanumeric strings, each 255 characters max	A set of arbitrary string labels associated with the artifact for search and filter capabilities
Visibility	Mutable	Yes	Enum, any of PRIVATE, PUBLIC	A field indicating if the artifact is accessible for all the tenants in the cloud (PUBLIC), or for its owner only (PRIVATE).
State	System	N/A	Enum, any of: CREATING, ACTIVE, DEACTIVATED, DELETED	A system field set by the repository, indicating the current phase of the artifact’s lifecycle. See “Artifact Lifecyle” below for details
Owner	System	N/A	String, OpenStack tenant (project) identifier	An identifier of the tenant (project) to which the artifact has been uploaded
CreatedAt	System	N/A	Timestamp	Time when the artifact record was created in the repository. Never changes once set
UpdatedAt	System	N/A	Timestamp	Time when the artifact was last modified. Is updated automatically by repository on every operation which updates the artifact record or its relatives.
PublishedAt	System	N/A	Timestamp	Time when the artifact was published, i.e. moved from CREATING to ACTIVE state. Empty for artifacts in CREATING state
DeletedAt	System	N/A	Timestamp	Time when the artifact was deleted, i.e. moved from CREATING, ACTIVE or DEACTIVATED state to DELETED. Is empty for artifacts not in DELETED state

Type-specific metadata fields

For each type of artifact, additional metadata fields may be defined. This set of fields and their properties are specified by the artifact types. The plugin which defines the artifact defines all its type-specific fields. Each type-specific metadata field is represented by a data structure having the following properties:

Field name - alphanumeric value identifying the field. Should be unique per artifact type
Field type - a type of the data stored in the field. Identifies the types of queries which may be made to search and filter the artifacts in catalog. The following types are supported:
- String - simple string value (255 characters max). In listing queries may use this field to filter artifacts by value equality (e.g. “filter all the artifacts of type ‘Image’ having the field ‘OS_TYPE’ equal to ‘Linux’”)
- Integer. - integer field. In listing queries may use this field to filter artifacts by both value equality and value range (e.g. “filter all the artifacts of type ‘Image’ having the field ‘REQUIRED_RAM’ less or equal to 16”)
- Numeric - same as Integer, but stored as Numeric
- Boolean* - boolean values, having either ‘True’ or ‘False’ as the value. In listing queries may use this field to filter artifacts by value equality.
- Text - Long text field. May not be used for filtering, just stores arbitrary large text descriptions, JSONs etc. The loading of these fields from the Database should be deferred when the artifacts are listed, taken for modification etc - they should be loaded only when the client request the contents of the artifact. This reduces the load on Database. Additionally the plugin developers may put custom constraints to limit the length of the text.
- Array - a list of values of some specific type (String, Text, Integer, Numeric, Boolean) or their combination. Such field may be used for tag-like filtering semantics (i.e. the artifacts of some type may have a set field called ‘Category’, each such artifact may have a number of categories assigned, and the filtering query may be made to list only the artifacts having the specific category)
- Dict - a set of key-value pairs, where key are string values (255 chars max) while values may be any of String, Text, Integer, Numeric or Boolean. May be used for filtering the artifacts by having specific value of some key or by just having some specific key with any value.
Required - boolean value indicating if the field value should be specified for the artifact to be valid.
Mutable - boolean value indicating if the field value may be changed after the artifact is published.
Internal - a boolean value indicating that the value of this field is not shown to the end-user (cannot be fetched using API or used in queries), but may be accessed internally by the plugin logic.
Default - a default value for the field may be specified by the Artifact Type.
Allowed values - a list of possible values for the field to store
Constraints - a number of constraints may be defined for each metadata field. Available constraints depend on the field type:
- String - minLength, maxLength, pattern
- Integer and Numeric - minimum, maximum values
- Array - types of the elements (either a single type for all the elements or specific type for each element), minimum and maximum numbers of elements in the array, type-specific constrains for elements.
- Dict - types of the values (either a single type for all the values of specific type for each value), minimum and maximum numbers of values in the dict, type-specific constrains for values.
Also, custom, code-driven constrains may be defined for each field by the plugin developer.

Artifact Versions

The Artifact repository is able to store different versions of the same artifact. From repository’s point of view these are different objects (having different IDs, stored independently and having independent lifecycle), but having the same type and name, but different version fields.

The version field is one of common immutable metadata fields (see above). It follows the SemVer notation, i.e. its value is composed out of 3 numeric parts (major version, minor version and patch, with optional alphanumeric prerelease suffix), e.g. 5.1.3 or 1.1.4-alpha. An order is defined between version values according to the SemVer spec, e.g. 1.2.3 > 1.2.2, 1.0.1 < 1.1.0, 1.0.0 > 1.0.0-some-suffix etc, according to spec. When the version is specified, some of its parts may be omitted: e.g. 10 will stand for 10.0.0, 5.1 will stand for 5.1.0. The repository will accept such an input and will convert it to the fill semver notation. The major version part is mandatory. Either major or minor parts should be non-zero: e.g. 0.1 is possible while 0.0 is not.

Binary Data

The data is the heart and soul of artifacts and the primary reason of their existence. Sometimes it is just a simple text (or XML, YAML or JSON object represented as text) which may be stored in the artifact’s additional metadata fields, however in many cases it is a large binary object, such as VM image or other large block of bytes. And even textual data is often completely unrelated to artifact’s metadata - and thus has to be treated separately.

Be the data really binary or textual it should be stored independently from the artifact record in some dedicated storage system - in a same manner as VM images are stored independently from their definitions in the current version of Glance.

When the artifact is initially created in the repository (the record is inserted into the database), it is in the CREATING state (see the “Artifact Lifecycle” section below) and has no binary data associated. Then - before publishing the artifact - its owner may upload the data by using the appropriate API calls or specify the location/uri of the existing binary blob pre-uploaded to some back-end storage. If the new blobs are uploaded, they will be placed into the storage system (the one which is being used for the particular instance of Glance) and associated with the artifact record by using its id. If the existing location/uri is passed, only the association will be made.

When the artifact repository is browsed, only the artifact metadata is returned: the data objects for each specific artifact have to be downloaded separately, by sending an appropriate API calls either to Glance or directly to the appropriate underlying storage system.

Unlike the current Glance Images, a single artifact may have more than one binary object associated.

For example, a Heat template, being stored as an artifact, may require a Heat template itself, a number of provider templates, plus, probably, an Icon to be displayed in the catalog UI. All of these are considered to be “artifact’s data” (although the templates are not actually binary), and there is no sense in merging them into a single block of data: different templates may be required at different moments of time, while the icon is needed in UI only - so they are stored as different “binary objects” and are retrieved one-by-one when really needed

This means that each artifact may have a number of different BLOBs associated.

The amount of blobs and their types (“Heat template”, “Provider Templates” and “Icon” in the example above) are defined per each artifact type. When the artifact is in CREATING state its owner will have to upload all the required parts one-by-one, specifying their types every time. If some required blob is not uploaded, the “publishing check” will not pass and the artifact will not be accepted into the repository. So, in general, this is similar to specifying the values of additional type-specific metadata fields.

For each artifact type its plugin can define several BLOB properties that correspond to the BLOBs supported by this particular type. BLOB properties can be defined in 2 forms - either as a single BinaryObject, or as BinaryObjectList. The latter can be constrained with maximum and minimum length. Each BLOB property is represented by a data structure having the following properties:

Name (alphanumeric string) - name of the blob property. Should be unique per artifact type.
Required (boolean) - a flag indicating if the blob(s) of this property is required for the artifact to be valid.
Minimum Length (for BinaryObjectList only) (a positive integer value or None) - specifies the minimal required amount of blobs per artifact. If set to None, then the number of blobs in BLOBs list per artifact has no lower bound.
Maximum Length (for BinaryObjectList only) (a positive integer value or None) - specifies the maximum amount of blobs per artifact. If set to None, then the number of blobs in BLOBs list per artifact has no upper bound.

For example, the above-mentioned artifact type “Heat template” may define an optional BinaryObjectList property with the Name set to “ProviderTemplate”, and a required BinaryObjectList property with the name “Icon” and minimum and maximum length constraints equal to 1. This means that the artifacts of this type have to contain exactly one icon blob and zero or more provider template blobs.

The repository provides an API call to inspect the blob contents of each artifact.

In the example above the API returns a dictionary containing two keys: “ProviderTemplates” and “Icon”. The first key defined a list of records, each one corresponding to a different blob with a provider template, the second one defines a single record corresponding to the blob with the icon. Each of the blob records contains an id of the particular blob, which may be used to download the blob.

Binary data is immutable, and no new blobs may be added as well as no existing blobs may be deleted once the artifact is moved to the ACTIVE state. When the artifact is deleted, all its blobs are deleted as well. The blob data is accessible to a user only if they have permission to access the artifact containing that blob.

Artifact Dependency Relations

In OpenStack ecosystems the entities rarely go alone: usually they interact with each other. Artifacts aren’t different: quite often an artifact may need to work with other artifacts in one way or another. In this case it may be reasonable to introduce a concept of Artifact Dependency: an artifact may depend on any number of other artifacts of any type.

For example there may be an artifact containing a reusable pack of Puppet manifests which configure some common software components on the VMs (artifact A). Another artifact containing some Heat Software Config template (artifact B) may utilise the puppet manifests from the artifact A for some post-deploy configuration actions on the VMs created by the template. In this case artifact B may specify artifact A as its dependency, so the service which is going to use B may detect, that it may need A as well and download it in advance.

This eventually creates chains of transitively-dependent artifacts. An API method provided by Artifact Repository should allow to retrieve all the dependencies of a given artifact. Subsequent calls to the same API will allow to retrieve all the transitive dependencies.

These dependency relations are actually a special kind of artifact metadata: they are stored in repository database together with artifact records. They are immutable, i.e. can be defined only while the artifact is in the CREATING state and cannot be modified after it is published.

The operations to create, view or modify a dependency for a particular artifact should be implemented as a set of APIs similar to other metadata editing.

The dependencies cannot be circular (e.g if A depends on B, B depends on C and C depends on D, then D cannot depend on A), and the published artifact may not depend on an artifact being in a “CREATING” state. For example, if A depends on B and both are in “CREATING” state, then B should be published before A, otherwise the publishing check of A will fail.

The dependencies may be created only on the artifacts which belong to the same tenant as the dependent artifact. Once the dependency is established, a target artifact cannot be deleted before its dependent artifact is deleted. If both are deleted and should be restored, they restoreation should happen in the same order as creation: the target artifact first, the dependent - second.

Artifacts Lifecycle

An Artifact may be in one of the three states, indicating different phases of its lifecycle:

CREATING state indicates that the artifact record has been created in repository (ID was assigned), but the artifact is still being constructed (some fields have to be set, the data uploaded etc). Mutable metadata fields may be modified while the artifact is in this state. The binary data may be uploaded or location uri for pre-uploaded blobs may be specified, the dependencies may be set or modified.

Artifacts in the CREATING state are not visible to tenants other than the owner’s even if the ‘Visibility’ field is set to ‘PUBLIC’. When artifact is in the CREATING state its owner may call a ‘Publish’ API method to publish the artifact. This will validate the artifact and change its state to ACTIVE. When the artifact is in CREATING state its owner may initiate its deletion. This will change the state of the artifact to ‘DELETED’.
ACTIVE state indicates that the artifact is available for use. When the artifact is in this state the immutable metadata fields may not be updated, no new binary data may be uploaded, existing data may not be deleted.

Artifacts in ACTIVE state are visible to other tenants if the ‘Visibility’ field is set to PUBLIC. When the artifact is in this state it is assumed that it may be used by the third-party services. When the artifact is in ACTIVE state its owner may initiate its deletion. This will change the state of the artifact to ‘DELETED’.
DEACTIVATED state indicates that the artifact which had been previously published has been temporary deactivated (i.e. prevented from being used) by the cloud administrator. Artifact is visible to all the users which may access it regularly (i.e. to its owner and to other tenants if it is made public) and its metadata can be fetched as well, but the binary data is not accessible and the artifact should not be used by third-party services until the deactivation is removed. Usually this is used for investigations of reported problems with artifacts. When the investigation is completed the artifact may be reactivated again (by changing the state back to ACTIVE) or deleted.
DELETED state indicates that the artifact has been deleted and its data and metadata objects are not accessible anymore.

This state is available only if a “delayed_delete” option is enabled in the configuration file, which makes the deletion to be non-immediate: the artifacts are just marked as deleted, however the actual deletion is delayed for the configurable amount of time.

If delayed_delete is disabled, then artifacts do not enter the DELETED state, their records are permanently removed from the repository when the deletion operation is executed.

A state transition diagram can be found here.

Composing an Artifact

Artifacts may be created using an iterative process, which starts with creating an artifact record in the database. Then a number of APIs may be called to set the values of various metadata properties, upload the binary data, specify the dependency links etc.

While the aftifact is being composed it remains in CREATING state.

Publishing an Artifact

Publishing is an operation which makes an artifact available for usage: when all the metadata fields are set and the binary data parts are uploaded (which may be a lengthly multi-step process), it may be made available by calling a special API method to change its state to ACTIVE.

During the execution of this call Glance will verify that all the dependency relations are set to the existing and already published artifacts. If at least one of the dependencies either does not exist (i.e. the artifact has been deleted after the dependency was created) or is not in the ACTIVE state, then the publishing action will fail.

Deleting an Artifact

An artifact may be deleted only by its owner or a cloud administrator. When the delete action is called, the following actions are executed:

All the artifacts of the current tenant are inspected. If any of them have a Dependency Relation to the artifact being deleted, the deletion operation is aborted
If the delayed delete is enabled, artifact’s state is set to DELETED and the deletion operation is scheduled to in the configurable time interval
If the soft delete is not enabled, then a background deletion operation is scheduled to run immediately

The deletion operation (be it delayed or immediate) does the following:

Delete all the binary data objects of the artifact from the underlying storage system.

Delete the artifact’s record from the database, including all the custom metadata property values, associated tags and outgoing dependency relations

Alternatives

The need of a catalog service seems obvious, however sometimes questions are asked if Glance is appropriate project for this. Some people were suggesting to use a completely separate service for this, some suggested to use Swift instead of Glance.

Why not a separate service?

Because Glance’s Images are perfect examples of Artifacts: they are binary objects stored with arbitrary metadata describing them. Glance already provides search and filter capabilities, they just have to be extended to support artifact types other then images.

Why not to use Swift?

For a number of reasons.

First, Swift is not a catalog, it is just an object storeage. There are no search or filter capabilities, object immutability is not guranteed.

Then, Swift has a different level of abstraction: it does not care about the data which is being stored within it, while Glance should be aware of the artifact types specifics.

Last but not least, many production-grade OpenStack deployments do not have Swift deployed, as they do not need it. Meanwhile, Glance is present on each and every OpenStack cloud.

Meanwhile, Swift still may be used as an underlying storage for artifacts - in the same way as Glance uses it to store Images.

Data model impact

Support will be added to: * glance/db/sqlalchemy/api.py * registry/api.py * simple/api.py

A new script glance/db/sqlalchemy/artifacts.py will be added

The table classes will be in glance/db/sqlalchemy/models_artifacts.py

The following DB schema is the initial suggested schema. Constraints are not shown for readability.

**Artifacts Table**
Field	Type	Null	Key
id	varchar(36)	NO	PRI
name	varchar(255)	NO	MUL
type_name	varchar(255)	NO	MUL
type_version_prefix	bigint(20)	NO
type_version_suffix	varchar(255)	YES
type_version_meta	varchar(255)	YES
version_prefix	bigint(20)	NO
version_suffix	varchar(255)	YES
version_meta	varchar(255)	YES
description	text	YES
visibility	varchar(32)	NO	MUL
state	varchar(32)	NO	MUL
owner	varchar(255)	NO	MUL
created_at	datetime	NO
updated_at	datetime	NO
published_at	datetime	YES
deleted_at	datetime	YES

**Artifact Blobs Table**
Field	Type	Null	Key
id	varchar(36)	NO	PRI
name	varchar(36)	NO	MUL
artifact_id	varchar(36)	NO	MUL
size	bigint(20)	NO
position	int(11)	YES
item_key	varchar(329)	YES	MUL
checksum	varchar(329)	YES	MUL
created_at	datetime	NO
updated_at	datetime	NO

**Artifact Blob Locations Table**
Field	Type	Null	Key
id	varchar(36)	NO	PRI
value	text	NO
blob_id	varchar(36)	NO	MUL
status	varchar(36)	YES
position	int(11)	YES
created_at	datetime	NO
updated_at	datetime	NO

**Artifact Dependencies Table**
Field	Type	Null	Key
id	varchar(36)	NO	PRI
artifact_source	varchar(36)	NO	MUL
artifact_dest	varchar(36)	NO	MUL
artifact_origin	varchar(36)	NO	MUL
is_direct	tinyint(1)	NO
position	int(11)	YES
name	varchar(36)	YES
created_at	datetime	NO
updated_at	datetime	NO

**Artifact Properties Table**
Field	Type	Null	Key
id	varchar(36)	NO	PRI
artifact_id	varchar(36)	NO	MUL
string_value	varchar(255)	YES
int_value	int(11)	YES
numeric_value	decimal(10,0)	YES
bool_value	tinyint(1)	YES
text_value	text	YES
position	int(11)	YES
name	varchar(36)	NO	MUL
created_at	datetime	NO
updated_at	datetime	NO

**Artifact Tags Table**
Field	Type	Null	Key
id	varchar(36)	NO	PRI
artifact_id	varchar(36)	NO	MUL
value	varchar(255)	YES
created_at	datetime	NO
updated_at	datetime	NO

REST API impact

All the new APIs to be placed under the /v2/artifacts API branch

All the APIs which are specific to the particular artifact type should be placed to /v2/artifacts/{artifact_type}, where artifact_type is a constant defined by the artifact type definition (i.e. by the plugin), which usually should be plural of the artifact type name. For example, for artifacts of type “template” this constant may be called ‘templates’, so the API endpoints will start with /v2/artifacts/templates.

The artifact_type constant should unambiguously identify the artifact type, so the values of this constants should be unique among all the artifact types defined by the active plugins.

The artifact_type constant should be followed by type_version identifier containing a SemVer-compliant string prefixed with v, e.g. v1.1.5. This will identify the particular version of the artifact type if there are many of them available. In the “List Artifacts” and “Get an Artifact” API calls the type_version is optional and may be omitted.

List artifacts
- GET /v2/artifacts/{artifact_type}/[{type_version}/]creating - list
  artifact drafts
  
  Returns the list of artifacts in CREATING state having the specified type and owned by the current tenant. If the user is administrator returns the artifacts in CREATING state owned by all the tenants.
- GET /v2/artifacts/{artifact_type}/[{type_version}/] - list artifacts
  which are ready for usage
  
  Returns the list of artifacts in ACTIVE state, which are either owned by the current tenant or are made available to everyone with setting Visibility metadata field to PUBLIC.
- GET /v2/artifacts/{artifact_type}/[{type_version}/]deactivated - list
  artifacts in DEACTIVATED state.
  
  Returns the list of artifacts in DEACTIVATE state which are temporary suspended from usage.
- URL parameters:
  
  artifact_type identifier of the artifact type, should be equal to a
  valid constant defined in one of the active artifact plugins.
  
  type_version optional identifier defining the version of artifact type. If omitted all versions of an artifact type are assumed, but sorting and filtering capabilities are limited to generic properties only.
- Query parameters:
  Query may contain parameters intended for filtering and soring by most of common and type-specific metadata fields. Type-specific fields may be used only if type_version parameter is set to specific version of the artifact type. The set of parameters and their values should be compliant to the schema defined by the artifact type and its version.
  
  Filtering:
  
  Filter keys may be any generic and type-specific metadata fields of primitive type, like ‘string’, ‘numeric’, ‘int’ and ‘bool’. But filtering by type-specific properties is allowed only when artifact version is provided.
  
  Direct comparison requires a property name to be specified as query parameter and the filtering value as its value, e.g. ?name=some_name
  
  Parameter names and values are case sensitive.
  
  Artifact API supports filtering operations in format ?name=<op>:some_name, where op is one of the following:
  
  eq: equal;
  
  ne: not equal;
  
  gt: greater than;
  
  ge: greater or equal than;
  
  lt: lesser than;
  
  le: lesser or equal than.
  
  Set comparison filtering is available for all the set-valued type-specific metadata fields as well as common field “tags”.
  
  Set comparison requires a property name to be specified as query parameter. The property may be repeated several times, e.g. the query ?tags=abc&tags=cde&tags=qwerty will filter artifacts having either ‘abc, ‘cde’ or ‘qwerty’ tags associated.
  
  Checking for entry into the array property is performed by in operation. It’s done the same way as other filters, e.g. ?items_array=in:array_element.
  
  Sorting
  
  In order to retrieve data in any sort order and direction, artifacts REST API accepts multiple sort keys and directions.
  
  Artifacts API will align with the API Working group sorting guidelines and support the following parameter on the request:
  
  sort: Comma-separated list of sort keys, each key is optionally appended with <:dir>, where ‘dir’ is the direction for the corresponding sort key (supported values are ‘asc’ for ascending and ‘desc’ for descending)
  
  Sort keys may be any generic and type-specific metadata fields of primitive type, like ‘string’, ‘numeric’, ‘int’ and ‘bool’. But sorting by type-specific properties is allowed only when artifact version is provided.
  
  Default value for sort direction is ‘desc’, default value for sort key is ‘created_at’.
  
  Pagination
  
  limit and marker query parameters may be used to paginate through the artifacts collection in the same way as it is done in the current version of Glance “List Images” API.
  
  Maximum limit number is 1000. It’s done for security reasons to protect the system from intruders to prevent them from sending requests that can pull the entire database at a time.
- HTTP Responses:
  
  200 if artifact_type is valid
  
  404 if no Artifact Type is defined to handle specified value of artifact_type
- Response schema: [JSON list with artifacts’ metadata]
Create a new Artifact draft
- POST /v2/artifacts/{artifact_type}/{type_version}/creating
- Creates a new artifact record in database, the status of artifact is set to CREATING. Request body may contain initial metadata of the artifact.
- URL parameters:
  
  artifact_type identifier of the artifact type, should be equal to a valid constant defined in one of the active artifact plugins.
  
  type_version identifier defining the version of artifact type.
- HTTP Responses:
  
  201 if everything went fine. Location header is set to the artifact location
  
  404 if no Artifact Type is defined to handle specified value of artifact_type and/or type_version
  
  400 if an artifact of this type with the same name and version already exists.
- Response schema: [JSON with created artifact]
Publish an Artifact
- POST /v2/artifacts/{artifact_type}/{type_version}/{id}/publish
- Publishes an artifact, i.e. moves it to ACTIVE state.
- URL parameters:
  
  artifact_type identifier of the artifact type, should be equal to a valid constant defined in one of the active artifact plugins.
  
  id identifier of the artifact
  
  type_version identifier defining the version of artifact type.
- HTTP Responses:
  
  200 if everything went fine
  
  404 if no artifact with the given ID was found or if the type of the found artifact differs from type specified by artifact_type parameter (if it is not equal to generic value ‘artifacts’) or if the found artifact is not owned by the current tenant.
  
  400 if the artifact draft has dependencies on missing or non-published artifacts.
  
  403 if the artifact is not in the CREATING state
- Request body: None
- Response schema: [JSON with published artifact]
Get an artifact
- GET /v2/artifacts/{artifact_type}/[{type_version}]/{id}
- Returns an artifact record with all the common and type-specific metadata
- URL parameters:
  
  artifact_type identifier of the artifact type, should be equal to a valid constant defined in one of the active artifact plugins.
  
  id identifier of the artifact
  
  type_version optional identifier defining the version of artifact type. Unlike in other (modifying) calls in this one it can be omitted to provide possibility to get the artifact by its type and id regardless of type version.
- HTTP Responses:
  
  200 if everything went fine
  
  404 if no artifact with the given ID was found or if the type of the found artifact differs from type specified by artifact_type parameter (if it is not equal to generic value ‘artifacts’) or if the found artifact is not accessible by the current tenant.
- Response schema: [JSON artifact definition, TBD]
Update an Artifact
- PATCH /v2/artifacts/{artifact_type}/{type_version}/{id}
- Updates artifact’s metadata fields. If the artifact is in state other than CREATING then only mutable fields may be updated.
- URL parameters:
  
  artifact_type identifier of the artifact type, should be equal to a valid constant defined in one of the active artifact plugins.
  
  id identifier of the artifact
- HTTP Responses:
  
  200 if everything went fine
  
  404 if no artifact with the given ID was found or if the type of the found artifact differs from type specified by artifact_type parameter (if it is not equal to generic value ‘artifacts’) or if the found artifact is not owned by the current tenant.
  
  403 if the PATCH attempts to modify the immutable property while the artifact’s state is other than CREATING
- Request schema: [JSON patch, TBD]
- Response schema: [JSON artifact definition, TBD]
Delete an Artifact
- DELETE /v2/artifacts/{artifact_type}/{type_version}/{id}
- Deletes an artifact. See Deleting an Artifact for details
- URL parameters:
  
  artifact_type identifier of the artifact type, should be equal to a valid constant defined in one of the active artifact plugins. May also contain value equal to “artifacts”
  
  id identifier of the artifact
  
  type_version identifier defining the version of artifact type.
- HTTP Responses:
  200 if everything went fine
  
  404 if no artifact with the given ID was found or if the type of the
  found artifact differs from type specified by artifact_type parameter (if it is not equal to generic value ‘artifacts’) or if the found artifact is not owned by the current tenant.
  
  400 if there are other artifacts which depend on the given one.

A detailed example:

Let’s assume that the artifact type has several type-specific properties, defined as follows:

tags = Array(item_type=String()) dependencies = ArtifactReferenceList() blob = BinaryObject()

Sample HTTP-requests for the given artifact will be given below.

GET /v2/artifacts/{artifact_type}/{id}

Retrieves an artifact with type artifact_type and id id.

PATCH /v2/artifacts/{artifact_type}/{type_version}/{id} body = [{‘op’: ‘add’, ‘path’: ‘/tags/-’, ‘value’: ‘new’}]

This request appends string ‘new’ to a tags property. Here tags is not a reserved name, like in images, it is just the name chosen for an artifact property representing an array of strings. As this is a data-modifying request, it should include the type_version part of the URI to ensure that caller knows exactly which version of the artifact schema is being targeted.

PATCH /v2/artifacts/{artifact_type}/{type_version}/{id} body = [{‘op’: ‘remove’, ‘path’: ‘/dependencies/0’}]

This request removes first element from the artifact’s dependencies property. Mind that dependencies is not a reserved word, but a custom property name for a list of ArtifactReferences.

PUT /v2/artifacts/{artifact_type}/{type_version}/{id}/blob or POST /v2/artifacts/{artifact_type}/{type_version}/{id}/blob

content-type ‘application/octet-stream’ body = SOMEDATA

This request uploads data specified in request’s body to BinaryObject property blob.

GET /v2/artifacts/{artifact_type}/{id}/blob/download

This request retrieves the value of BinaryObject property blob.

DELETE /v2/artifacts/{artifact_type}/{type_version}/{id}/files/{file_id}

Security impact

The artifact types (their type-specific metadata fields and BLOB kinds) are defined in independent python modules (plugins). This code may be provided by third-party developers, and thus has to be inspected for potential security problems. However, an explicit cloud operators action is required to enable this modules (i.e. they are not user-supplied), so while the operator enables only the trusted plugins there are no security threats.
The artifact types may define custom logic to validate the values of some of their type-specific metadata fields. This Custom logic is part of the plugin (i.e. cloud operator’s action is required to enable it). Thus there is no security issue here unless the operator enables untrusted plugin.
As this change adds a set of new APIs, a set of new policies should be added as well to provide a role-based access to these APIs. Plugin-specific policies may be added later. However, this spec does not include any actions for these policies, so they have to be specified by later specs.

Notifications impact

To be added later with different specifications / blueprints

Other end user impact

As new APIs are being added, appropriate changes to python-glanceclient and glance CLI should be made to support this interactions.

Performance Impact

The new APIs and data models should follow the same architecture as the existing ones. Thus there should be now impact on performance when calling the APIs.

Long running tasks (such as artifact deletion) should be made asynchronous and should be executed by dedicated background workers, so they will not interfere with the API performance. The implementation of the asynchronous task should reuse the existing delayed delete functionality.

Other deployer impact

The database schema changes should be made with migrations. This migrations have to be executed prior the usage of the new functionality.

Glance deployment should include adding and activating the plugins which define artifact types.

Developer impact

Existing APIs are not modified.

All the artifact-related actions will be available as new API endpoints under the /v2/ branch

The existing images API will not be migrated to artifacts in v2 branch. After artifacts are implemented and stabilized there will be an alternative implementation of Images which will use artifacts’ semantic instead of the current one, however this implementation will not substitute the existing one

Deprecation of the current images API is beyond the scope of the current spec and should not happen unless a separate blueprint is filed and a proper deprecation period is announced.

Implementation

Assignee(s)

Primary assignee:: ativelkov
Other contributors:: gokrokve mfedosin ivasilevskaya

Reviewers

mfedosin jokke hemanth-makkapati nikhil-komawar

Work Items

The database layer and data migrations for Artifacts and their common metadata properties
The database layer and data migrations to store and manage the values of type-specific metadata fields and blob references
The plugin interfaces which will allow to specify artifact types and their metadata structures.
The REST API for artifact composition, publishing, deletion and search of artifacts, as well as retrival of their data and metadata
Modifications to plugin interface to allow custom logic definition for importing and exporting artifacts as a single operation
The REST API to support import/export operations for artifacts
The database layer and data migrations to add artifact dependency relations
The REST API to support defining dependency relations during artifact composition
delayed_delete feature: modifying the current delayed delete to support artifact deletion.
Modifications to python-glancecleint to make use of all the new APIs.

Dependencies

No new dependencies required.

Testing

All the new APIs should be covered by functional and integration tests with Tempest

Data migrations and database API should be covered by Unit tests.

Documentation Impact

All the new APIs, configuration options and policies should be documented.

A new document - “Plugin developers guide” has to be added.

References

Glare API

Thu, 08 Mar 2018 00:00:00

https://blueprints.launchpad.net/glance/+spec/glare-api

This specification describes stable API of Glance Artifact Repository service (aka Glare) and covers all most popular use cases.

Problem description

Glare needs a stable API that satisfy requirements of API-WG and DefCore.

Proposed change

This specification aims to solve the described problem by complete refactoring of Glance Artifact Repository API and changing calls to Glare service.

The behavior of the servers and API calls for Glare will be changed. To describe service behavior after refactoring in details we prepared some glossary and list of Use Cases.

User list

Developer - person who is contributing to Glare.
Glare - represents Glare service deployed within OpenStack.
User - Glare user who wants to consume Glare artifacts. User should be created in Keystone under some project and domain.
Cloud Admin - person who is responsible for Glare administration and support within OpenStack.

Glossary

Glare (from GLance Artifact REpository) - a service that provides access to a unified catalog of immutable objects with structured meta-information as well as related binary data (these structures are also called ‘artifacts’). Glare controls artifact consistency and guaranties that binary data and properties won’t change during artifact lifetime.

Note

Artifact type developer can declare properties which values may be changed, but he has to do it explicitly, because by default all properties are considered immutable.
Artifact - in terms of Glare, an Artifact is a structured immutable object with some properties, related binary data and metadata.
Artifact Version - property of artifact that defines its version in SemVer format.
Artifact type - defines artifact structure of both its binary data and properties. Examples of OpenStack artifact types that will be supported in Glare are: Heat templates, Murano Packages, Nova Images, Tacker VNFs and so on.
Artifact status - specifies the state of the artifact and the possible actions that can be done with it. List of possible artifact statuses:
- queued - Artifact created but not activated, so it can be changed by Artifact owner.
- active - Artifact activated, immutable properties cannot be changed. Artifact available for download to other Users.
- deactivated - Artifact is not available to other users except administrators, used when Cloud Admin need to check the artifact.
- deleted - Artifact deleted.

**Artifact status transition table**
Artifacts Status	queued	active	deactivated	deleted
queued	X	activate Artifact	N/A	delete Artifact
active	N/A	X	de-activate Artifact	delete Artifact
deactivated	N/A	reactivate Artifact	X	delete Artifact
deleted	N/A	N/A	N/A	X

Artifact Property - property of artifact that defines some information about Artifact. Artifact Properties always have name, type, value and optional additional parameters, described bellow.

Types are based on types from oslo.versionedobjects library. All types are inherited from FieldType class and define custom behavior.

Glare uses several primitive types from oslo.versionedobjects directly:
- String;
- Integer;
- Float;
- Boolean;
And also Glare expands this list with custom types:
- Blob;
- Dependency;
- Structured generic types Dict or List.
Each property has additional parameters:
- required_on_activate - boolean value indicating if the property value should be specified for the artifact before activation. (Default: True)
- mutable - boolean value indicating if the property value may be changed after the artifact is activated. (Default: False)
- system - boolean value indicating if the property value cannot be edited by User. (Default: False)
- sortable - boolean value indicating if there is a possibility to sort by this property’s values. (Default: False)
Note

Only properties of 4 primitive types may be sortable: integer, string, float and boolean.
- default - a default value for the property may be specified by the Artifact Type. (Default: None)
- validators - a list of functions or lambdas like f(self, v) -> Bool. When user sets a value to the property with additional validators Glare checks them before setting the value and raises ValueError if at least one of the requirements is not satisfied.
- filter_ops - a list of available filter operators for the property. There are seven available operators: ‘eq’, ‘neq’, ‘lt’, ‘lte’, ‘gt’, ‘gte’, ‘in’. All of them cam be applied to primitive properties only. Dict values can be sorted using the following syntax “?<dict_name>.<key_name>=<op_name>:<value>.”
Artifact Dependency - property type that defines soft dependency of the Artifact from another Artifact. It is an url that allows user to obtain some Artifact data. For external dependency the format is the following: http(s)://<netloc>/<path> For internal dependencies dependency contains only <path>. Example of <path>: /artifacts/<artifact_type>/<artifact identifier>. User can use filters and page_limits in path to define dynamic dependencies. All that Glare will do is request GET with dependency to receive Artifact info.
Artifact Blob - property type that defines binary data for Artifact. User can download Artifact blob from Glare. Each blob property has a flag external, that indicates if the property was created during file upload (False) or by direct user request (True). In other words, “external” means that blob property url is just a reference to some external file and Glare does not manage the blob operations in that case. Json schema that defines blob format:
```
{
    "type": "object",
    "properties": {
        "size": {
            "type": ["number", "null"]
        },
        "checksum": {
            "type": ["string", "null"]
        },
        "external": {
            "type": "boolean"
        },
        "status": {
            "type": "string",
            "enum": ["saving", "active", "pending_delete"]
        }
    },
    "required": ["size", "checksum", "external", "status"]
}
```
Artifact blob properties may have the following statuses:
- saving - Artifact blob record created in table, blob upload started.
- active - blob upload successfully finished.
- pending_delete - indicates that blob will be deleted soon by Scrubber (if delayed delete is enabled) or by Glare itself.

**Blob status transition table**
Blob Status	saving	active	pending delete
saving	X	finish blob upload	request for artifact delete
active	N/A	X	request for artifact delete
pending_delete	N/A	N/A	X

Artifact Dict and List - compound generic property types that implement Dict or List interfaces respectively, and contain values of some primitive type, defined by element_type attribute.
Artifact visibility - defines who may have an access to active artifact. Initially there are 2 options: ‘private’ artifact is accessible by its owner and admin only and ‘public’, when all users have an access to the artifact by default. When artifact is ‘queued’ its visibility is ‘private’. It’s allowed to change visibility only when artifact has ‘active’ status.
Artifact locking - when artifact is queued all its properties are editable, but when it becomes active it is “locked” and cannot be modified (except for those properties explicitly declared as mutable).

Use Case list

Use Case 1. Add Artifact type.

Pre-condition: None.

Success condition: New Artifact type has been added to Glare and it can be used by other OS users.

Steps:
1. Operator adds Artifact type class to ‘glare/objects’ folder.
Note

Artifact types are pre-published in Glance repository. All of them will be reviewed and committed by Glance developers and reviewers.
1. Operator implements all abstract methods and defines data API.
Note

We are using image data API for image artifact type and common unified data API for other artifacts.
1. Operator regenerates (with oslo config generator) or updates Glare configuration file to enable new Artifact type.
2. Operator restarts Glare service, Glare validates new Artifact type.
Alternative scenarios:

4A. Artifact type validation failed - Glare won’t start.
Use Case 2. Update Artifact type.

Pre-condition: Artifact type supported by Glare (Artifact type class has been implemented).

Success condition: Artifact type definition has been updated in Glare
and it can be used by other OpenStack users. Old artifacts (that existed before update) are available to OpenStack users.

Steps:
1. Operator adds new field or update field in Artifact type class.
2. Operator updates artifact type class (bump version) to cover support compatibility between database and new changes.
3. Operator re-generates or updates Glare configuration file (if needed).
4. Operator restarts Glare service, Glare validates updated Artifact type.
Alternative scenarios:

4A. Artifact type validation failed - Glare won’t start.
Use Case 3. Disable artifact type.

Pre-condition: Artifact type supported by Glare (Artifact type class has
been implemented).

Success condition: Artifact type definition has been updated in Glare and
it can be used by other .

Steps:
1. Operator deletes artifact type name from Glare configuration file.
2. Operator restarts Glare.
Alternative scenarios:
- None.
Note

Artifacts of disabled artifact type will be presented in database and all data will be in storage, but users won’t be able to access those artifacts with Glare API.
Use Case 4. List artifact types.

Pre-condition: Running Glare API.

Success condition: List of Artifact type definitions provided in that
deployment to User.

Steps:
1. User requests list of supported Artifact types (GET /schemas).
2. Glare checks Glare configuration, artifact type classes and generates JSON representation of artifact type.
3. Glare returns the representation to the user (200 OK).
Alternative scenarios:
- None.
Use Case 5. Get artifact type info.

Pre-condition: Running Glare API.

Success condition: Artifact type definition provided to User

Steps:
1. User requests Artifact type info (GET /schemas/{artifact_type}).
2. Glare checks glare configuration, artifact type class and generates JSON schema
3. Glare returns JSON schema to the user (200 OK).
Alternative scenarios:

2A. No Artifact type found with requested name (404 Not Found)
Use Case 6. Create artifact.

Note

No blob upload here. Only creating an Artifact entity in DB.
Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
- User has a permission to create artifact.
Success condition: Artifact entity has been created in Glare DB.

Steps:
1. User defines Artifact allowed properties and requests for Artifact creation (POST /artifacts/{artifact_type}).
2. Glare checks Artifact the possibility of artifact creation.
3. Glare creates Artifact entity in DB (only record in DB).
4. Glare returns metadata of created artifact (201 Created).
Alternative scenarios:

2A. Property value is incorrect (400 Bad Request)

2B. Property with name doesn’t exist (400 Bad Request)

2C. Dependency format is incorrect (400 Bad Request)

2D. Property is system (403 Forbidden)

2E. Artifact type is not found (404 Not Found)

2F. Artifact with required name and version already exists for this user (409 Conflict)
Use Case 7. Upload Artifact binary data.

Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
- Artifact instance has been successfully created.
- User has enough quota space.
- Artifact has ‘queued’ status.
Success condition: Data is uploaded in storage, Artifact blob has been
successfully crated and has status ‘active’.

Steps:
1. User specifies Artifact identifier, blob property name, blob binary data, content-type as application/octet-stream and request upload (PUT /artifacts/{artifact_type}/{artifacts_id}/{blob_name}).
2. Glare checks if Artifact identifier and blob property is valid.
3. Glare creates a blob record in db, changes its status to ‘saving’ and associates the record with related Artifact blob property.
4. Glare uploads blob binary to store back end.
5. Glare sets all required metadata (‘size’, ‘checksum’ and so on) to the blob and changes its status to ‘active’.
Note

Glare doesn’t use registry service, so do not need trusts here.
1. Glare responds with 200 OK.
Alternative scenarios:

2A. No artifact found (404 Not Found).

2B. No blob property found (400 Bad Request).

2C. Metadata is not valid (400 Bad Request).

3C. Blob is already uploaded and has status ‘active’ (409 Conflict)

3D. Blob is saving (409 Conflict).

4A. Artifact quota per tenant exceed (413 HTTPRequestEntityTooLarge).

4B. Blob upload failed. Glare initiates Blob killing and responds with appropriate error (depends on backend error).
Use Case 8. Add a custom location for artifact.

Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
- Artifact instance has been successfully created.
- Artifact has ‘queued’ status.
Success condition: blob location has been successfully added to Artifact.

Steps:
1. User specifies Artifact identifier, blob property name, location, content-type as application/json and sends the request (PUT /artifacts/{artifact_type}/{artifacts_id}/{blob_name}).
  - Body:
    {"url": "<some_url>"}
2. Glare checks if Artifact identifier and blob property name is valid.
3. Glare checks given location, creates blob record in db, associate it with Artifact and adds location to blob.
4. Glare changes blob property status to active.
5. Glare responds with 200 OK.
Alternative scenarios:

2A. No artifact found (404 Not Found).

2B. No property found (400 Bad Request).

3A. Location url is not valid (400 Bad Request).

3B. Location url is not downloadable (400 Bad Request).

3C. Blob record is already exists (409 Conflict).
Use Case 9. Activate Artifact.

Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
- Artifact instance has been successfully created.
- Success condition:* Artifact is activated in Glare (status is ‘active’).
Steps:
1. User defines Artifact id and requests Artifact activation (PATCH /artifacts/{artifact_type}/{artifacts_id}).
  - Body:
    [{ "op": "replace", "path": "/status", "value": "active" }]
2. Glare checks if all required Artifact properties specified according to Artifact definition.
3. Glare checks if all required blob properties are active according to Artifact definition.
4. Glare checks if all Artifact dependencies are correct.
5. Glare activates Artifact so it becomes visible to other Users and returns response (200 OK).
Alternative scenarios:

2-4A. some obligatory properties or blobs were not specified, dependencies are not correct (no dependency found by url or there are multiple artifacts per dependency) (400 Bad Request).

2B. Artifact doesn’t exist or deleted (404 Not Found).
Use Case 10A. Update Artifact non-blob property, Artifact is queued.

Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
- Artifact instance has been successfully created.
Success condition: Artifact property has been updated in Glare.

Steps:
1. User requests for Artifact property, blob property or dependency update (PATCH /artifacts/{artifact_type}/{artifacts_id}).
2. Glare updates required properties and returns updated artifact (200 OK).
Alternative scenarios:

2A. Dependency url is not correct or downloadable (400 Bad Request).

2B. Artifact doesn’t exist or deleted (404 Not Found).

2C. Parameter is incorrect (400 Bad Request).

2D. Property is system (403 Forbidden).

2E. Artifact with updated name and version already exists for this user (409 Conflict).
Use Case 10B. Update Artifact non-blob property, Artifact is active.

Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
- Artifact instance has been successfully created.
Success condition: Artifact has been updated in Glare.

Steps:
1. User requests for Artifact property or dependency update (PATCH /artifacts/{artifact_type}/{artifacts_id}).
2. Glare checks if dependency is mutable and can be updated.
3. Glare checks if property is mutable and can be updated.
4. Glare updates required properties (200 OK).
Alternative scenarios:

2A. Dependency property is immutable (403 Forbidden).

2B. Dependency url is not correct or downloadable (400 Bad Request).

3A. Property is immutable (403 Forbidden).

3B. Parameter is incorrect (400 Bad Request).
Use Case 11. Download blob.

Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
- Artifact instance has been successfully created.
Success condition: Blob is downloaded.

Steps:
1. User provides blob property name and Artifact identifier (GET /artifacts/{artifact_type}/{artifacts_id}/{blob_name}).
2. Glare provides Artifact binary stream to the User (200 OK).
Alternative scenarios:

2A. Artifact blob contains no data (204 No Content).

2B. Artifact blob has status ‘saving’ (204 No content).

2C. Download is not successful (depends on backend error).

2D. Artifact is deactivated and user doesn’t have a permission
to download (403 Forbidden)
Use Case 12. Get Artifact info

Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
- Artifact instance has been successfully created.
Success condition: Artifact definition returned to User.

Steps:
1. User requests for Artifact info with artifact identifier and artifact type (GET /artifacts/{artifact_type}/{artifacts_id}).
2. Glare returns Artifact properties, blobs information and dependency information (200 OK).
Alternative scenarios:

2A. No grants to view artifacts (403 Forbidden).

2B. No artifact or artifact type found (404 Not Found).
Use Case 13. Deactivate Artifact.

Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
- Artifact instance has been successfully created.
Success condition: Artifact status is deactivated.

Steps:
1. User sends a request to deactivate artifact (PATCH /artifacts/{artifact_type}/{artifacts_id}).
  - Body:
    [{ "op": "replace", "path": "/status", "value": "deactivated" }]
Note

access to the deactivation is managed by oslo policy.
1. Glare changes status of artifact from active to deactivated (200 OK).
Note

current dependencies are soft dependencies. So no integrity check between artifacts here.

Alternative scenarios:

2A. Artifact status is not active (400 Bad Request).

2B. User doesn’t have permission to deactivate artifact (403 Forbidden).
Use Case 14. Reactivate Artifact.

Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
- Artifact instance has been successfully created.
Success condition: Artifact is active.

Steps:
1. User sends a request to re-activate Artifact (PATCH /artifacts/{artifact_type}/{artifacts_id}).
  - Body:
    [{ "op": "replace", "path": "/status", "value": "active" }]
Note

access to the deactivation is managed by oslo policy.
1. Glare changes status of Artifact from deactivated to active (200 OK).
Alternative scenarios:

2A. Artifact status is not deactivated (400 Bad Request).

2B. User doesn’t have permission to reactivate artifact (403 Forbidden).
Use Case 15. Publish Artifact.

Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
- Artifact instance has been successfully created.
- Artifact status is ‘active’.
Success condition: Artifact is published (i.e. artifact visibility is public).

Steps:
1. User sends a request to publish Artifact (PATCH /artifacts/{artifact_type}/{artifacts_id})
  - Body:
    [{ "op": "replace", "path": "/visibility", "value": "public" }]
2. Glare changes visibility of Artifact from ‘private’ to ‘public’ (200 OK).
Alternative scenarios:

2A. Artifact status is not ‘active’ (400 Bad Request).

2B. User doesn’t have permission to publish artifact (403 Forbidden).

2C. Public artifact with required name and version already exists (409 Conflict).
Use Case 16. List Artifacts.

Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
Success condition: List of artifacts returned to User.

Steps:
1. User request for Artifact list with specified artifact type, limit, marker, filters (GET /artifacts/{artifact_type}).
2. Glare returns all Artifacts to the User (200 OK).
Alternative scenarios:

2A. Number of Artifacts is too large (400 Bad Request) (use limit to restrict number of Artifacts in response).

2B. Query params is invalid (sorting or filtering by non-existing key) (400 Bad Request).
Use Case 17. Delete artifact.

Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
- Artifact instance has been successfully created.
Success condition: Artifact has been successfully deleted from Glare.

Steps:
1. User requests Artifact delete with Artifact type and Artifact identifier (DELETE /artifacts/{artifact_type}/{artifacts_id}).
2. Glare updates artifact status to ‘deleted’ and set all artifact’s blobs statuses to ‘pending_delete’.

Note

no dependency consistency check here.
1. Glare deletes all artifact’s custom properties and tags.
4. Glare sequentially deletes blob data from store and removes blob instances from db.
1. Glare responds 204 No Content.
Alternative scenarios:

2A. Some blob is uploading. Glare finishes uploading of the blob and deletes it immediately.

2B. Some blob is downloading. It depends on the store backend, in common case downloading will be canceled.

4A. There was a storage error during blob deleting. Glare stops blob deleting, user have to remove them from Store with Scrubber.

4B. ‘delayed_delete’ config option is enabled. Glare does nothing and returns 204 No Content. Blob data should be removed later with Scrubber.
Use Case 18. Add Artifact tag.

Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
- Artifact has been successfully created.
Success condition: Artifact tag has been successfully added to Artifact.

Steps:
1. User specifies tag name and Artifact identifier and sends request. (PUT /artifacts/{artifact_type}/{artifacts_id}/tags/{tag_value})
2. Glare adds tag to the Artifact (200 OK).
Alternative scenarios:

2A. Tag is already exist (200 OK).

2B. Artifact does not exist (404 Not Found).
Use Case 19. Delete Artifact tag.

Pre-conditions:
- Artifact type definition has been added to Glare.
- Artifact type is active in Glare.
- Artifact has been successfully created.
Success condition: Artifact tag has been successfully removed from Artifact.

Steps:
1. User specifies tag name and Artifact identifier and sends request. (DELETE /artifacts/{artifact_type}/{artifacts_id}/tags/{tag_value}).
2. Glare removes tag from Artifact (204 No Content).
Alternative scenario:

2A. Tag or Artifact does not exist (404 Not Found).

Alternatives

We need to address API-WG and DefCore comments anyway, so there are no alternatives here.

Data model impact

No data model impact.

REST API impact

All API calls are placed under the /artifacts/{artifact_type} branch, where artifact_type is a constant defined by the Artifact Type, which usually should be plural of the artifact type name. For example, for artifacts of type “template” this constant should be called ‘templates’, so the API endpoints will start with /artifacts/templates.

Glare API complies with OpenStack API-WG guidelines:

Filtering, sorting and pagination

Tags

Errors

For updating artifact properties Glare API uses json-patch

Glare supports microversions to define what API version it should use: API-WG microversion guidelines

Info

list of available API versions

GET /

Returns json representation of the minimum and maximum API versions. Code 200 OK

list of available artifact types schemas

GET /schemas

Returns json representation of all active artifact types. Code 200 OK

description of artifact type

GET /schemas/{artifact_type}

Returns json schema of given artifact type. Code 200 OK

Artifacts

list of artifacts with given type name

GET /artifacts/{artifact_type}

Returns json representation of list of artifacts. Code 200 OK.

Note

These requests support query parameters that comply with API-WG filtering, sorting and pagination guidelines

create an artifact of the given type name

POST /artifacts/{artifact_type}

Returns info about created instance in json format. Code 201 Created.

show artifact

GET /artifacts/{artifact_type}/{artifacts_id}

Returns info about artifact with given ID in json format. Code 200 OK.

update artifact

PATCH /artifacts/{artifact_type}/{artifacts_id}

Returns updated artifact info in json format. Code 200 OK.

delete artifact

DELETE /artifacts/{artifact_type}/{artifacts_id}

Code 204 No Content.

Blobs

upload file to artifact

PUT /artifacts/{artifact_type}/{artifacts_id}/{blob_name}

Code 200 OK.

download file from artifact

GET /artifacts/{artifact_type}/{artifacts_id}/{blob_name}

Returns binary stream of requested blob. Code 200 OK

Tags

Note

These requests comply with API-WG tags guidelines

add a tag to artifact

PUT /artifacts/{artifact_type}/{artifacts_id}/tags/{tag_name}

Code 200 OK.

remove a tag from artifact

DELETE /artifacts/{artifact_type}/{artifacts_id}/tags/{tag_name}

Code 200 OK.

delete all tags from artifact

DELETE /artifacts/{artifact_type}/{artifacts_id}/tags

Code 204 No Content.

replace list of tags for artifact

PUT /artifacts/{artifact_type}/{artifacts_id}/tags

Code 200 OK.

get list of tags for artifact

GET /artifacts/{artifact_type}/{artifacts_id}/tags

Code 200 OK.

Examples of API requests

For example, we have an artifact type ‘example_type’ with properties:

id: StringField
name: StringField
visibility: StringField
status: StringField
blob_file: BlobField
metadata: DictOfStringsField
version: VersionField

Create artifact

Request:

Method: POST

URL: http://host:port/artifacts/example_type
Body:
{
   "name": "new_art"
}

Response:

201 Created

{
     "status": "queued",
     "name": "new_art",
     "id": "art_id1",
     "version": null,
     "blob_file": null,
     "metadata": {},
     "visibility": "private"
 }

Get artifact

Request:

Method: GET

URL: http://host:port/artifacts/example_type/art_id1

Response:

200 OK

{
    "status": "queued",
    "name": "new_art",
    "id": "art_id1",
    "version": null,
    "blob_file": null,
    "metadata": {},
    "visibility": "private"
}

List artifacts

Request:

Method: GET

URL: http://host:port/artifacts/example_type

Response:

200 OK

{
    "example_type": [{
        "status": "queued",
        "name": "new_art",
        "id": "art_id1",
        "version": null,
        "blob_file": null,
        "metadata": {},
        "visibility": "private"
    }, {
        "status": "queued",
        "name": "old_art",
        "id": "art_id2",
        "version": null,
        "blob_file": null,
        "metadata": {},
        "visibility": "private"
    }, {
        "status": "queued",
        "name": "old_art",
        "id": "art_id3",
        "version": null,
        "blob_file": null,
        "metadata": {},
        "visibility": "private"
    }],
    "first": "/artifacts/example_type",
    "schema": "/schemas/example_type"
}

Request:

Method: GET

URL: http://host:port/artifacts/example_type?name=eq:old_art

Response:

200 OK

{
    "example_type": [{
        "status": "queued",
        "name": "old_art",
        "id": "art_id2",
        "version": null,
        "blob_file": null,
        "metadata": {},
        "visibility": "private"
    }, {
        "status": "queued",
        "name": "old_art",
        "id": "art_id3",
        "version": null,
        "blob_file": null,
        "metadata": {},
        "visibility": "private"
    }],
    "first": "/artifacts/example_type?name=ne%3Anew_name",
    "schema": "/schemas/example_type"
}

Update artifact

Request:

Method: PATCH
URL: http://host:port/artifacts/example_type/art_id1

Body:

[{
    "op": "replace",
    "path": "/name",
    "value": "stark"
}, {
    "op": "add",
    "path": "/metadata/slogan",
    "value": "winter is coming"
}]

Response:

200 OK

{
    "status": "queued",
    "name": "stark",
    "id": "art_id1",
    "version": null,
    "blob_file": null,
    "metadata": {
        "slogan": "winter is coming"
    },
    "visibility": "private"
}

Upload blob

Request:

Method: PUT

URL: http://host:port/artifacts/example_type/art_id1/blob_file

Body:

What Is Dead May Never Die

Response:

200 OK

{
    "status": "queued",
    "name": "stark",
    "id": "art_id1",
    "version": null,
    "metadata": {
        "slogan": "winter is coming"
    },
    "blob_file": {
        "status": "active",
        "checksum": "8452e47f27b9618152a2b172357a547d",
        "external": false,
        "size": 16
    },
    "visibility": "private"
}

Download blob

Request:

Method: GET

URL: http://host:port/artifacts/example_type/art_id1/blob_file

Response:

200 OK

What Is Dead May Never Die

Activate artifact

Request:

Method: PATCH

URL: http://host:port/artifacts/example_type/art_id1
Body:
[{
    "op": "replace",
    "path": "/status",
    "value": "active"
}]

Response:

200 OK

{
    "status": "active",
    "name": "stark",
    "id": "art_id1",
    "version": null,
    "metadata": {
        "slogan": "winter is coming"
    },
    "blob_file": {
        "status": "active",
        "checksum": "8452e47f27b9618152a2b172357a547d",
        "external": false,
        "size": 16
    },
    "visibility": "private"
}

Deactivate artifact

Request:

Method: PATCH

URL: http://host:port/artifacts/example_type/art_id1
Body:
[{
    "op": "replace",
    "path": "/status",
    "value": "deactivated"
}]

Response:

200 OK

{
    "status": "deactivated",
    "name": "stark",
    "id": "art_id1",
    "version": null,
    "metadata": {
        "slogan": "winter is coming"
    },
    "blob_file": {
        "status": "active",
        "checksum": "8452e47f27b9618152a2b172357a547d",
        "external": false,
        "size": 16
    },
    "visibility": "private"
}

Reactivate artifact

Request:

Method: PATCH

URL: http://host:port/artifacts/example_type/art_id1
Body:
[{
    "op": "replace",
    "path": "/status",
    "value": "active"
}]

Response:

200 OK

{
    "status": "active",
    "name": "stark",
    "id": "art_id1",
    "version": null,
    "metadata": {
        "slogan": "winter is coming"
    },
    "blob_file": {
        "status": "active",
        "checksum": "8452e47f27b9618152a2b172357a547d",
        "external": false,
        "size": 16
    },
    "visibility": "private"
}

Publish artifact

Request:

Method: PATCH

URL: http://host:port/artifacts/example_type/art_id1
Body:
[{
    "op": "replace",
    "path": "/visibility",
    "value": "public"
}]

Response:

200 OK

{
    "status": "active",
    "name": "stark",
    "id": "art_id1",
    "version": null,
    "metadata": {
        "slogan": "winter is coming"
    },
    "blob_file": {
        "status": "active",
        "checksum": "8452e47f27b9618152a2b172357a547d",
        "external": false,
        "size": 16
    },
    "visibility": "public"
}

Delete artifact

Request:

Method: DELETE

URL: http://host:port/artifacts/example_type/art_id1

Response:

204 No Content

Security impact

No security impact

Notifications impact

No notification impact

Other end user impact

The new API will need the client support. The support of the Glare API v0.1 may be removed at the same moment when v1 support is added or at any moment afterwards.

Performance Impact

No performance impact

Other deployer impact

The Glare service will follow the Glance release cycle. It’s expected that artifact type updating, however, will proceed more quickly than a 6 month cycle. Thus we want to make it possible to update artifact types more frequently, but in a controlled manner that will be easy for deployers.

We propose to address this situation by creating a single new repository where all new glare-objects (Artifact types) will go. It will either be a oslo-inc style library, but preferably a non-client OpenStack library which can be released to pip and the Glare objects can be then pulled into Glance repository using the right upper constraints per branch and at the same time adhere to the OpenStack packaging fundamentals.

The motivation, intent and purpose behind creating this new separate repo for library is to form a organized way for OpenStack wide cross-project developers to contribute to custom artifact object types and provide a consistent feedback. At the same time, common code can be shared by the glare-objects and best practices can be documented in the same place.

Release of all the glare objects will be precise with that of glance requirements and testing can be done comprehensively for all custom objects in the same release of the library (irrespective if subset of objects have had changes or not). This should ease the packaging pain and avoid operator/upgrade mess.

Developer impact

As the library would be a OpenStack non-client library, it would require privileged access to the source code from non-regular glance developers who are primarily working on projects like Murano or Heat, and so on.

We would need to establish some common code and best practices for developers in this repository.

Since the Glare v 0.1 API is EXPERIMENTAL, developers should be prepared for its deprecation.

Implementation

Assignee(s)

Primary assignee: mfedosin

Other contributors:: kkushaev dshakhray nikhil-komawar

Reviewers

Glance core team because we need to spread the knowledge about Glare to whole team.

Work Items

Add new API router and controllers. Mark v1 as EXPERIMENTAL
Write helper methods to implement query parameter parsing (e.g. for filtering, sorting, tags etc, according to API-WG guidelines)
Cover stable API with unit and functional tests.
Set v0.1 API as DEPRECATED and v1 as STABLE.
Implement Glare v1 client

Dependencies

The work on separating Glare API from Glance API (i.e. the transition from Glance v3 to Glare v0.1), defined in spec [6] should be completed first.

Testing

Glare api should be covered by functional and unit tests. Integration tests with Tempest should be implemented as well.

Documentation Impact

Glare API, configuration options and policies should be documented.

A new document - “Artifact type developers guide” has to be added.

References

Glance Client Spec Lite

Thu, 08 Mar 2018 00:00:00

Please keep this template section in place and add your own copy of it between the markers. Please fill only one Spec Lite per commit.

<Title of your Spec Lite>

problem:: <What is the driver to make the change.>
solution:: <High level description what needs to get done. For example: “We need to add client function X.Y.Z to interact with new server functionality Z”.>
impacts:: <All possible *Impact flags (same as in commit messages) or ‘None’.>

Optionals (please remove this line and fill or remove the rest until End of Template):

how:: <More technical details than the high level overview of solution if needed.>
alternatives:: <Any alternative approaches that might be worth of bringing to discussion.>
timeline:: <Estimation of the time needed to complete the work.>
link:: <Link to the change in gerrit that already would provide the solution. After committing the Spec Lite depend the change to the Spec Lite commit.>
reviewers:: <If reviewers has been agreed for the functionality, list them here.>
assignee:: <If known, list who is going to work on the feature implementation here>

End of Template

Add your Spec Lite before this line

Rolling upgrades

Thu, 08 Mar 2018 00:00:00

https://blueprints.launchpad.net/glance/+spec/rolling-upgrades

This spec provides a gap analysis of what is required for the Glance project to assert the various rolling upgrade tags and specifies the actions necessary to close the gaps.

The goal for Ocata is to assert the assert:supports-zero-downtime-upgrade tag, with a stretch goal of asserting the assert:supports-zero-impact-upgrade tag. Given that asserting these tags depends on completion of another feature (see spec proposal [GSP1]), the backup goal is to assert the assert:supports-rolling-upgrade tag. In any case, Glance will make progress on the upgrade front in this cycle.

Problem description

There are currently four upgrade tags:

assert:supports-upgrade [GOV3]
assert:supports-rolling-upgrade [GOV1]
assert:supports-zero-downtime-upgrade [GOV4]
assert:supports-zero-impact-upgrade [GOV5]

supports-upgrade

The assert:supports-upgrade tag [GOV3] has been asserted for Glance [GOV0].

supports-rolling-upgrade

The requirements for asserting the assert:supports-rolling-upgrade tag are listed in [GOV1]. They are:

The project is already tagged as type:service [GOV2].
- Glance status: done
The project has already successfully asserted the assert:supports-upgrade tag [GOV3].
- Glance staus: done
The project has a defined plan that allows operators to roll out new code to subsets of services, eliminating the need to restart all services on new code simultaneously.
- Glance status: need to define a plan.
  
  More detail about the required plan, as described in [GOV1]:
  
  This plan should clearly call out the supported configuration(s) that are expected to work, unless there are no such caveats. This does not require complete elimination of downtime during upgrades, but rather reducing the scope from “all services” to “some services at a time.” In other words, “restarting all API services together” is a reasonable restriction.
  
  The Glance services consist of the Glance API server and the optional Glance Registry API server. The Glance Registry API server is not intended to be exposed to end users or other OpenStack services; it is expressly designed for internal Glance use only.
  
  (Note that it’s OK for there to be specific configurations under which a rolling upgrade is expected to work. In particular, it’s likely that we will require deployments using the optional Glance registry to run it on dedicated nodes.)
  
  Glance has had healthcheck middleware that can be used to signal to a load balancer that an API node is out of service since the Liberty release [GLA2]. This can be leveraged to take Glance nodes running “old” code out of rotation while nodes running “new” code are brought in.
  
  Note that while this proposal will allow a mixed deployment of API versions to run simultaneously, it does not envision that this will include multiple versions of the API running on the same node simultaneously. In other words, we do not intend to support a scenario in which “new” API code is deployed to a node while “old” Glance processes are running on that node. Instead, we expect operators to allow the “old” nodes to drain completely and all processes running the “old” code to be stopped before the “new” code is deployed to that node. (If the Glance nodes are VMs, a completely drained node could simply be deleted and be replaced by a fresh VM containing the “new” code.)
Full stack integration testing with services arranged in a mid-upgrade manner is performed on every proposed commit to validate that mixed-version services work together properly.
- Glance status: needs to be implemented (but note that the tests required for the next tag would more than satisfy this requirement).

supports-zero-downtime-upgrade

The assert:supports-zero-downtime-upgrade tag indicates that a project supports minimal rolling upgrade capabilities in such a way that no disruptions to API availability occur during the upgrade.

The requirements for asserting this tag are listed in [GOV4]. They are:

The project is already tagged as type:service [GOV2].
- Glance status: done
The project has already successfully asserted both the assert:supports-upgrade and assert:supports-rolling-upgrade tags.
- Glance status: the assert:supports-rolling-upgrade tag has not yet been asserted. See the previous section for what’s required.
Services must completely eliminate API downtime of the control plane during the upgrade.
- Glance status: The key issue is how to handle database changes required for release N while release N-1 code is still running. This is addressed by another spec, “Database strategy for rolling upgrades” [GSP1].
Services must be capable of receiving and handling requests throughout the upgrade process with a normal success rate. Services must prevent regression by implementing a zero-downtime gate job wherein both a new version of the service and an old version of the service are run concurrently.
- Glance status: needs to be implemented.

supports-zero-impact-upgrade

The assert:supports-zero-impact-upgrade tag indicates that a project supports both rolling upgrade capabilities and a zero-downtime upgrade (as described above) in such a way that no perceivable API performance penalty occurs during the upgrade.

The requirements for asserting this tag are listed in [GOV5]. They are:

The project is already tagged as type:service.
- Glance status: done
The project has already successfully asserted the assert:supports-upgrade, assert:supports-rolling-upgrade, and assert:supports-zero-downtime-upgrade tags.
- Glance status: see the previous sections.
Services must completely eliminate any perceivable performance penalty during the upgrade process. Operators should not expect any portion of the upgrade or migration process to place abnormally high load on any part of the cloud, or to cause delays in the handling of API requests, even intermittently.
- Glance status: Given that we’re talking about Glance services only (not the DBMS and not the storage backend), this should be achieved when the zero-downtime upgrade is implemented.
Services must prevent regression by implementing a zero-impact gate job wherein both a new version of the service and an old version of the service are run concurrently under load. A measurement of API response times must show that there are no statistically significant outliers during the upgrade process when compared to normal operations.
- Glance status: needs to be implemented.

Proposed change

There are two major changes:

Process Documentation

What we need to establish is that the Glance project has “a defined plan that allows operators to roll out new code to subsets of services, eliminating the need to restart all services on new code simultaneously.”

The “Gaps” section of the Product Working Group’s “Rolling Updates and Upgrades” user story [PWG1] provides a useful list of the phases an operator would go through in performing a rolling upgrade of an OpenStack cloud. We propose to document the relevant phases clearly for Glance so that operators can understand the Glance rolling upgrade story.

The phases identified by the Product Working Group are:
1. Maintenance Mode
2. Live Migration
3. Upgrade Orchestration - Deploy
4. Multi-version Interoperability
5. Online Schema Migration
6. Graceful Shutdown
7. Upgrade Orchestration - Remove
8. Upgrade Orchestration - Tooling
9. Upgrade Gating
10. Project Tagging
For Glance, upgrading from release N-1 to release N, we can compress these into:
1. Upgrade Orchestration - Deploy
  - stage the code for release N to new Glance nodes
2. Online Schema Migration - Part 1
  - initial database schema migration (the “expand” phase as described in [GSP1])
  - background data migration (as described in [GSP1])
3. Multi-version interoperabilty
  - start the release N nodes
  - take the release N-1 nodes out of rotation, allowing them to drain
4. Upgrade Orchestration - Remove
  - take each release N-1 node offline once it has completed processing its current requests
5. Online Schema Migration - Part 2
  - final database schema migration (the “contract” phase as described in [GSP1])
Testing

Full stack integration testing with services arranged in a mid-upgrade manner is performed on every proposed commit to validate that mixed-version services work together properly.
- This testing must be performed on configurations that the project considers to be its reference implementations.
- The arrangement(s) tested will depend on the project (i.e. should be representative of a meaningful-to-operators rolling upgrade scenario) and available testing resources.
- At least one representative arrangement must be tested full-stack in the gate.

We propose using Grenade [GRN1] for the full stack integration tests.

Alternatives

One alternative would be to choose not to support rolling upgrades in Glance. Such a choice, however, would impact other services that depend upon Glance (for example, Nova). Such services would experience disruptions during the Glance upgrade. So this doesn’t seem to be a serious alternative.
The proposal in this spec is to use the “disable by file” feature of the oslo healthcheck middleware to take the Glance nodes running “old” code out of rotation and allow them to drain. Stuart McLaren has suggested an alternative, namely to piggyback on the zero downtime configuration reload feature of Glance (available since the Kilo release [GLA1]) and create a “graceful stop” function that would accept a signal to shut down child processes as they complete. (See [GSP2] for details.)

Since we’ve got the “disable by file” functionality available, this alternative isn’t necessary to achieve the upgrade tags. It would, however, be an operator-friendly enhancement that we could pick up at some point.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

It is anticipated that a rolling upgrade will require operator intervention.

Developer impact

Developers will need to be aware of Glance features that enable rolling upgrades and make sure they aren’t removed. (Developers will also need to work within the constraints of the database strategy for rolling upgrades, but that developer impact is covered by another spec.)

Implementation

Assignee(s)

Primary assignee:: rosmaita hemanthm
Other contributors:: nikhil

Work Items

Verify the accuracy of current Glance upgrade documentation.
Write documentation for rolling upgrade (developer docs).
Write documentation for rolling upgrade (operator docs).
Grenade tests.
Assert the tag and notify the OpenStack Technical Committee.

Dependencies

To achieve the assert::supports-zero-downtime-upgrade tag, this spec depends upon implementation of the spec “Database strategy for rolling upgrades” [GSP1].

Testing

We’ll need to implement gate tests (see above).

Documentation Impact

Documentation of general information for Glance rolling upgrades, in particular:
- The supported configuration(s) for rolling upgrades
- The operator workflow for performing a rolling upgrade

References

[GLA1]

https://review.openstack.org/#/c/122181/

[GLA2]

https://review.openstack.org/#/c/148595/

[GOV0]

https://review.openstack.org/#/c/245897/

[GOV1] (1,2,3)

https://governance.openstack.org/reference/tags/assert_supports-rolling-upgrade.html

[GOV2] (1,2)

https://governance.openstack.org/reference/tags/type_service.html

[GOV3] (1,2,3)

https://governance.openstack.org/reference/tags/assert_supports-upgrade.html

[GOV4] (1,2)

https://governance.openstack.org/reference/tags/assert_supports-zero-downtime-upgrade.html

[GOV5] (1,2)

https://governance.openstack.org/reference/tags/assert_supports-zero-impact-upgrade.html

[GRN1]

https://github.com/openstack-dev/grenade

[GSP1] (1,2,3,4,5,6)

https://review.openstack.org/#/c/331740/

[GSP2]

https://review.openstack.org/#/c/331489/8/specs/ocata/approved/glance/rolling-upgrades.rst@75

[PWG1]

http://specs.openstack.org/openstack/openstack-user-stories/user-stories/proposed/rolling-upgrades.html

Glance Store Spec Lite

Thu, 08 Mar 2018 00:00:00

Please keep this template section in place and add your own copy of it between the markers. Please fill only one Spec Lite per commit.

<Title of your Spec Lite>

problem:: <What is the driver to make the change.>
solution:: <High level description what needs to get done. For example: “We need to add client function X.Y.Z to interact with new server functionality Z”.>
impacts:: <All possible *Impact flags (same as in commit messages) or ‘None’.>

Optionals (please remove this line and fill or remove the rest until End of Template):

how:: <More technical details than the high level overview of solution if needed.>
alternatives:: <Any alternative approaches that might be worth of bringing to discussion.>
timeline:: <Estimation of the time needed to complete the work.>
link:: <Link to the change in gerrit that already would provide the solution. After committing the Spec Lite depend the change to the Spec Lite commit.>
reviewers:: <If reviewers has been agreed for the functionality, list them here.>
assignee:: <If known, list who is going to work on the feature implementation here>

End of Template

Add your Spec Lite before this line

Glance Client Spec Lite

Thu, 08 Mar 2018 00:00:00

Please keep this template section in place and add your own copy of it between the markers. Please fill only one Spec Lite per commit.

<Title of your Spec Lite>

problem:: <What is the driver to make the change.>
solution:: <High level description what needs to get done. For example: “We need to add client function X.Y.Z to interact with new server functionality Z”.>
impacts:: <All possible *Impact flags (same as in commit messages) or ‘None’.>

Optionals (please remove this line and fill or remove the rest until End of Template):

how:: <More technical details than the high level overview of solution if needed.>
alternatives:: <Any alternative approaches that might be worth of bringing to discussion.>
timeline:: <Estimation of the time needed to complete the work.>
link:: <Link to the change in gerrit that already would provide the solution. After committing the Spec Lite depend the change to the Spec Lite commit.>
reviewers:: <If reviewers has been agreed for the functionality, list them here.>
assignee:: <If known, list who is going to work on the feature implementation here>

End of Template

Add your Spec Lite before this line

Glance Spec Lite

Thu, 08 Mar 2018 00:00:00

Please keep this template section in place and add your own copy of it between the markers. Please fill only one Spec Lite per commit.

<Title of your Spec Lite>

problem:: <What is the driver to make the change.>
solution:: <High level description what needs to get done. For example: “We need to add client function X.Y.Z to interact with new server functionality Z”.>
impacts:: <All possible *Impact flags (same as in commit messages) or ‘None’.>

Optionals (please remove this line and fill or remove the rest until End of Template):

how:: <More technical details than the high level overview of solution if needed.>
alternatives:: <Any alternative approaches that might be worth of bringing to discussion.>
timeline:: <Estimation of the time needed to complete the work.>
link:: <Link to the change in gerrit that already would provide the solution. After committing the Spec Lite depend the change to the Spec Lite commit.>
reviewers:: <If reviewers has been agreed for the functionality, list them here.>
assignee:: <If known, list who is going to work on the feature implementation here>

End of Template

Community Goal: Support Python 3.5

problem:: Satisfy the Pike community goal Support Python 3.5
solution:: Ensure that Glance meets the criteria of the Goal.
impacts:: None.
assignee:: Open

End of Community Goal: Support Python 3.5

Add your Spec Lite before this line

Glance Store Spec Lite

Thu, 08 Mar 2018 00:00:00

Please keep this template section in place and add your own copy of it between the markers. Please fill only one Spec Lite per commit.

<Title of your Spec Lite>

problem:: <What is the driver to make the change.>
solution:: <High level description what needs to get done. For example: “We need to add client function X.Y.Z to interact with new server functionality Z”.>
impacts:: <All possible *Impact flags (same as in commit messages) or ‘None’.>

Optionals (please remove this line and fill or remove the rest until End of Template):

how:: <More technical details than the high level overview of solution if needed.>
alternatives:: <Any alternative approaches that might be worth of bringing to discussion.>
timeline:: <Estimation of the time needed to complete the work.>
link:: <Link to the change in gerrit that already would provide the solution. After committing the Spec Lite depend the change to the Spec Lite commit.>
reviewers:: <If reviewers has been agreed for the functionality, list them here.>
assignee:: <If known, list who is going to work on the feature implementation here>

End of Template

Community Goal: Support Python 3.5

problem:: Satisfy the Pike community goal Support Python 3.5
solution:: Ensure that glance_store meets the criteria of the Goal.
impacts:: None.
assignee:: Alex Bashmakov

End of Community Goal: Support Python 3.5

Add your Spec Lite before this line

Glance Client Spec Lite

Thu, 08 Mar 2018 00:00:00

Please keep this template section in place and add your own copy of it between the markers. Please fill only one Spec Lite per commit.

<Title of your Spec Lite>

problem:: <What is the driver to make the change.>
solution:: <High level description what needs to get done. For example: “We need to add client function X.Y.Z to interact with new server functionality Z”.>
impacts:: <All possible *Impact flags (same as in commit messages) or ‘None’.>

Optionals (please remove this line and fill or remove the rest until End of Template):

how:: <More technical details than the high level overview of solution if needed.>
alternatives:: <Any alternative approaches that might be worth of bringing to discussion.>
timeline:: <Estimation of the time needed to complete the work.>
link:: <Link to the change in gerrit that already would provide the solution. After committing the Spec Lite depend the change to the Spec Lite commit.>
reviewers:: <If reviewers has been agreed for the functionality, list them here.>
assignee:: <If known, list who is going to work on the feature implementation here>

End of Template

Community Goal: Support Python 3.5

problem:: Satisfy the Pike community goal Support Python 3.5
solution:: Ensure that python-glanceclient meets the criteria of the Goal.
impacts:: None.
assignee:: Alex Bashmakov

End of Community Goal: Support Python 3.5

Add your Spec Lite before this line

Spec Lite: Remove the Registry API v1

Thu, 01 Mar 2018 00:00:00

problem:

During the Queens cycle, the two Glance services that depend upon the Glance Registry API v1 will be removed (API v1) or refactored (the scrubber). Hence, the Registry API will be redundant and should be removed.

dependencies:

The Images API v1 must be removed
The scrubber must be refactored so it doesn’t use the Registry API v1

solution:

Remove the Registry API v1 from the codebase.

impacts:

DocImpact

timeline:

target for the Q-3 milestone (week of 22 January 2017)

link:

https://blueprints.launchpad.net/glance/+spec/remove-registry-v1

reviewers:

all core reviewers

Spec Lite: Deprecate Images API v1 Support

Thu, 01 Mar 2018 00:00:00

problem:: The Images API v1 is targeted to be removed in Rocky, hence there is no reason to continue carrying v1 support in the glanceclient.
solution:: Announce that the Rocky release of the python-glanceclient will be the last release that will support the Images v1 API and that support will be removed in the first major release of the S cycle.
impacts:: DocImpact
alternatives:: Continue to support the Images API version 1. This is not necessary for interoperability, however, as the Images API v1 was never included in “DefCore” tests. Continued support would not be a good use of the team’s time.
timeline:: Early Rocky cycle
link:: https://blueprints.launchpad.net/python-glanceclient/+spec/deprecate-v1-support
reviewers:: all core reviewers
assignee:: jokke

Spec Lite: Switch to Turn Off Schema Validation

Thu, 01 Mar 2018 00:00:00

problem:: While an API is in EXPERIMENTAL status, the schemas may not accurately reflect the content of requests and responses, causing the client to error out.
solution:: Introduce a new option controlled by an environment variable that will turn off schema validation.
impacts:: None
timeline:: Q-1 milestone (week of 16 October 2017)
link:: https://blueprints.launchpad.net/python-glanceclient/+spec/no-schema-validation
reviewers:: all core reviewers

Inject metadata properties automatically to non-admin images

Mon, 26 Feb 2018 00:00:00

https://blueprints.launchpad.net/glance/+spec/inject-automatic-metadata

Cloud operator wants to launch a VM based on certain image metadata properties on different compute nodes.

Problem description

The operator provides public images to the users and users can also add their own images in glance and use these images to launch VMs. As an operator, all images provided by an operator should be launched on a specific set of compute nodes whereas images that are created by non-admin users should be launched on other set of compute nodes. The decision to launch VMs on certain compute nodes will be decided based on image metadata. When an operator will create images, they can specify certain image metadata which will be used by placement api or scheduler service to decide where the vm should be launched but if user creates image, there is no way user will know what image metadata properties to set and hence in the present placement api and scheduler logic, it is possible to launch a VM on any compute nodes. This is a big problem.

Use Case:

The operator wants user based images to be launched on certain set of compute nodes (host aggregate groups) based on image metadata properties.

Proposed change

Make a provision to inject image metadata to non-admin images while adding image to glance backend with the help of property protection. We are planning to introduce two new config options to achieve the same. 1) ‘ignore_user_roles’ is a comma-separated list of roles. Defaults to admin. 2) ‘inject’ defaults to None.

‘inject’ config option will consist of json format (key: value pair) of image metadata which an operator wants to add to the newly created images if user role is not as per mentioned in ‘ignore_user_roles’ config option. If user role is not same as mentioned in ‘ignore_user_roles’ and ‘inject’ doesn’t contain any metadata properties, then no metadata will be injected to the non-admin images.

We propose to create new config file ‘glance-image-import.conf’ which will have all config options related to image import tasks stuff. Each pluggable task can have own section which define its configuration options in this file.

For example, for injecting metadata properties ‘glance-image-import.conf’ will have ‘property_injections’ section described below;

[inject_metadata_properties] ignore_user_roles = admin,… inject = { “property1”: “value”, “property2”: “value,another value” }

Note: When user creates a volume from image, all image metadata will be copied to volume and made available as volume_image_metadata. Now, when user will copy back a volume to image, it will attempt to add ‘property1’ metadata and it will fail as it is only allowed to be created by admin. To address this issue need to make provision like ignore all properties specified in ‘inject’ config option if the request comes from a user whose roles is not specified in ‘ignore_user_roles config option. For example when user performs copy back volume to image then if properties mentioned in inject option will be silently ignored i.e. newly created image from volume does not contains special metadata properties mentioned in ‘inject’ config option.

[some_other_yet_undefined_task] will_have = “it’s own configs”

We will use property protection so that non-admin user won’t be able to add, update or delete the injected metadata properties automatically. Non-admin users however will be able to view these metadata properties. For example if ‘inject’ is set to ‘property1=value’ then in property_protections.conf file, an operator needs to restrict write and delete access for ‘property1’ to non-admin users as described below:

[property1] create = admin read = admin,member,_member_ update = admin delete = admin

In glance, users can create images using two ways. 1) create image API 2) Import image API

In case of import image API, as it uses taskflow, an additional task ‘InjectMetadataProperties’ will be created to inject the metadata. If ‘inject’ property is not None, then only ‘InjectMetadataProperties’ will be part of the tasks, otherwise it will be simply ignored.

No changes will be made to create image API. The intent is that the old upload workflow will only be used by services, it won’t be exposed to end users in most deployments.

Note: There is a restriction on how many image metadata properties an user can set to a image which is configurable using ‘image_property_quota’ config option. If it’s value is 128 and say, if ‘inject’ config option count is 3, then user would be able to add only 125 metadata items by themselves and remaining 3 will be injected automatically. If user tries to add more than 125 metadata items in case of import image api call then the task will marked as failed with appropriate failure message.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

There is a limit on having additional image metadata properties which defaults to 128. This is configurable using ‘image_property_quota’ option. If an operator sets this limit to higher value and more metadata properties are injected, then it will impact performance during image-list or image-show calls.

This impact is not specifically a result of the proposal in this spec, it’s also the case under the current situation.

Other deployer impact

Administrator need to set ‘ignore_user_roles’ and ‘inject’ config options in ‘inject_metadata_properties’ section of ‘glance-image-import.conf’ config file if metadata properties to be injected for new images for non-admin users.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <bhagyashri-shewale>
Other contributors:: None

Reviewers

Core reviewer(s):: Erno Kuvaja, Brian Rosmaita
Other reviewer(s):: None

Work Items

Make provision to read config options from new glance-image-import.conf
Add two new config options
Add ‘InjectMetadataProperties’ task for import call
Add unit tests for coverage
Add functional tests

Dependencies

None

Testing

Functional tests will be added to verify that metadata will be injected for non-admin images only.

Documentation Impact

Please refer to ‘Other deployer impact’

References

None

Spec Lite: Refactor Glance Scrubber

Mon, 26 Feb 2018 00:00:00

problem:: The scrubber code uses the registry v1 client. This is bad because it doesn’t support Keystone v3 auth, which is widely used by default. Additionally, when the Images API v1 is removed, nothing else will be using the registry v1 client.
solution:: Refactor the scrubber so that it doesn’t use the registry at all. Planning for registry deprecation began in Newton and the deprecation is being officially announced in Queens. This refactoring will allow us to remove the registry on schedule in the S release.
impacts:: None
timeline:: Q-2 milestone or thereabouts (early December)
link:: https://blueprints.launchpad.net/glance/+spec/scrubber-refactor
reviewers:: rosmaita, abhishekk, jokke
assignee:: wangxiyuan

Fake Keystone Server for Functional Tests

Tue, 23 Jan 2018 00:00:00

The URL of the launchpad blueprint:

https://blueprints.launchpad.net/glance/+spec/fake-keystone-for-testing

As keystone is only supported auth method for glance we should be doing our functional tests against keystone pipeline as well.

Problem description

Currently we are defaulting the Glance authentication pipeline to noauth. This is not supported configuration with glance and causes some issues.

Default config cannot be changed to keystone pipeline unless we have Keystone functionality available during our tests as the servers are spun up with default configuration as well as with the test specific one.
Functional tests does not reflect to real world as they uses unsupported authentication pipeline. Also certain functionalities (like using registry with v2) cannot be tested as noauth does not provide the needed user information.

Proposed change

Fake Keystone server with minimal functionality would solve this issue without increasing the overhead for our testing too much:

Needed functionality would be:

Support for auth token verification (this can be done with pre determined set of tokens and token management would not be needed)
Support for user - password authentication (this can also be done with pre determined set of user/password combinations and user management would not be needed).

Alternatives

Monkey patching the code we are actually testing to act against the design agreed when implemented. This would allow us to pass the test issues, but leave a hole for unwanted behaviour to pass tests.

Data model impact

None

REST API impact

None

Security impact

This would potentially harden the security as the functional tests would be ran against more real life like configuration.

Notifications impact

None

Other end user impact

None

Performance Impact

There is impact on test performance as more processes are needed to run the tests.

This can be circumvented by more coarse functional testing as agreed on weekly Glance meeting (14:39): http://eavesdrop.openstack.org/meetings/glance/2014/ glance.2014-06-05-14.05.log.html

Other deployer impact

None

Developer impact

Better visibility of how changes affects due to better testing experience.

Implementation

Assignee(s)

Erno Kuvaja <jokke>

Work Items

Implementation of the Fake Keystone server
Changing the functional tests using it
Refactoring the functional tests to circumvent the performance hit / improve the performance

Dependencies

None

Testing

As this is testing functionality/change testing of it would not be needed.

Documentation Impact

None

References

None

Glance Spec Lite

Tue, 23 Jan 2018 00:00:00

Please keep this template section in place and add your own copy of it between the markers. Please fill only one Spec Lite per commit.

<Title of your Spec Lite>

problem:: <What is the driver to make the change.>
solution:: <High level description what needs to get done. For example: “We need to add client function X.Y.Z to interact with new server functionality Z”.>
impacts:: <All possible *Impact flags (same as in commit messages) or ‘None’.>

Optionals (please remove this line and fill or remove the rest until End of Template):

how:: <More technical details than the high level overview of solution if needed.>
alternatives:: <Any alternative approaches that might be worth of bringing to discussion.>
timeline:: <Estimation of the time needed to complete the work.>
link:: <Link to the change in gerrit that already would provide the solution. After committing the Spec Lite depend the change to the Spec Lite commit.>
reviewers:: <If reviewers has been agreed for the functionality, list them here.>
assignee:: <If known, list who is going to work on the feature implementation here>

End of Template

Add your Spec Lite before this line

Glance Spec Lite

Tue, 23 Jan 2018 00:00:00

Please keep this template section in place and add your own copy of it between the markers. Please fill only one Spec Lite per commit.

<Title of your Spec Lite>

problem:: <What is the driver to make the change.>
solution:: <High level description what needs to get done. For example: “We need to add client function X.Y.Z to interact with new server functionality Z”.>
impacts:: <All possible *Impact flags (same as in commit messages) or ‘None’.>

Optionals (please remove this line and fill or remove the rest until End of Template):

how:: <More technical details than the high level overview of solution if needed.>
alternatives:: <Any alternative approaches that might be worth of bringing to discussion.>
timeline:: <Estimation of the time needed to complete the work.>
link:: <Link to the change in gerrit that already would provide the solution. After committing the Spec Lite depend the change to the Spec Lite commit.>
reviewers:: <If reviewers has been agreed for the functionality, list them here.>
assignee:: <If known, list who is going to work on the feature implementation here>

End of Template

Return 409 if setting location to saving or deactivated image

problem:: Currently, if ‘show_multiple_locations’ is activated, user can set custom location to an image, even if it has ‘saving’ or ‘deactivated’ status. Example: http://paste.openstack.org/show/506998/
solution:: Add a check, that looks at the image status and if it’s different from ‘queued’ or ‘active’ then returns Conflict error (409 response code).
impacts:: users will get Conflict error, when they try to set location to image in ‘saving’ or ‘deactivated’ state.
timeline:: Expected to be merged within the N-2 time frame.
link:: https://review.openstack.org/#/c/324012/
assignee:: Mike Fedosin

Return 409 if setting location to saving or deactivated image

Use policy to control deleting deactivated images

problem:: Currently, a user is permitted to delete a ‘deactivated’ image. Thus, if an image is suspected dangerous and deactivated by an administrator, a user could nonetheless remove the data before a security team has time to review it, thereby removing evidence of any wrongdoing. This presents a problem for an administrator who would like an investigation to take place.
solution:: Introduce a policy check named delete_deactivated_image to govern whether a user is permitted to delete a ‘deactivated’ image.
impacts:: Adds a new policy. The default configuration will preserve current behaviour, that is, all users will be permitted to perform the action.
timeline:: Expected to be merged within the N-2 time frame.
link:: https://review.openstack.org/#/c/256381
assignee:: Niall Bunting

End of policy to control deleting deactivated images

Add functionality to soft delete the tasks

problem:: Currently there is no mechanism for deleting tasks on regular basis. Thus, if a task is expired; it still comes up on calling task list or show. This can hamper the performance as the number of tasks returned will be more than the number of tasks that are active. Consequently, it will be tedious for the user to manage them.
solution:: Introduce a method which soft deletes the tasks by marking the deleted status as true in the database; so that, on calling task show or list, the expires tasks are not returned.
impacts:: Adds a new method. Users will now get only the active tasks.
timeline:: Expected to be merged within the N-2 time frame.
link:: https://review.openstack.org/#/c/209255/
assignee:: Geetika Batra

End of Add functionality to soft delete the tasks

Add vhdx to list of supported disk formats

problem:: Glance currently support vhd disk formats and is being used in known deployments. vhdx disks can have much larger storage capacity than the older vhd format. More info can be found at offcial site https://technet.microsoft.com/en-us/library/hh831446(v=ws.11).aspx
solution:: disk_formats configuration option needs to be updated to indicate that this is a acceptable format. Some documentation updates are required showing the same and providing info about the official documentation.
impacts:: This will have documentation and upgrade impact. Release note should be added to indicate interest parties about this addition.
timeline:: Expected to be merged within the N-3 time frame.
link:: https://review.openstack.org/#/c/347352
assignee:: Stuart McLaren

End of Add vhdx to list of supported disk formats

Deprecate `show_multiple_locations` configuration option

problem:: Glance currently has two ways in which locations can be configured to be exposed to users :- 1) using show_multiple_locations configuration option 2) Role Based Access Control (RBAC) rules for delete_image_location, get_image_location, set_image_location. If a cloud operator chooses to disable exposing multiple locations using 1), the approach in 2) becomes redundant. Also, it can be somewhat confusing for operators to have multiple ways to tune a particular feature. If proper defaults are set and documentation for setting appropriate policies exists, configuration option show_multiple_locations doesn’t hold much value.
solution:: We choose to deprecate the configuration option show_multiple_locations. Moving forward the right way to control the multiple locations feature will be using the Role Based Access Control (RBAC) rules for Create, Read, Update and Delete (CRUD) on locations. The policy.json file (example: http://git.openstack.org/cgit/openstack/glance/tree/etc/policy.json) should be used to govern the access a particular role has to the location(s). This also ensures appropriate security measures are taken into consideration by operators when using locations feature. This encourages being aware of the kind of users this sensitive feature will be exposed to rather than simple switching on/off a single configuration option. However, for Newton release we will only be deprecating the configuration option, no default values or setting will be changed. Currently we disallow using multiple locations by default, so the value of show_multiple_locations option will be kept as False. We would like to give some time to the operators to understand how to control multiple locations using this new RBAC method. In Ocata, we will be changing the default values of delete_image_location, get_image_location and set_image_location policies and will be removing show_multiple_locations option from the tree.
impacts:: This will have documentation and upgrade impact. Release note should be added to notify interested parties about this addition.
timeline:: Expected to be merged within the N-3 time frame.
link:: https://review.openstack.org/#/c/313936/
assignee:: Flavio Percoco

End of Deprecate `show_multiple_locations` configuration option

Add your Spec Lite before this line

Glance Store Spec Lite

Tue, 23 Jan 2018 00:00:00

Please keep this template section in place and add your own copy of it between the markers. Please fill only one Spec Lite per commit.

<Title of your Spec Lite>

problem:: <What is the driver to make the change.>
solution:: <High level description what needs to get done. For example: “We need to add client function X.Y.Z to interact with new server functionality Z”.>
impacts:: <All possible *Impact flags (same as in commit messages) or ‘None’.>

Optionals (please remove this line and fill or remove the rest until End of Template):

how:: <More technical details than the high level overview of solution if needed.>
alternatives:: <Any alternative approaches that might be worth of bringing to discussion.>
timeline:: <Estimation of the time needed to complete the work.>
link:: <Link to the change in gerrit that already would provide the solution. After committing the Spec Lite depend the change to the Spec Lite commit.>
reviewers:: <If reviewers has been agreed for the functionality, list them here.>
assignee:: <If known, list who is going to work on the feature implementation here>

End of Template

Add your Spec Lite before this line

Database strategy for rolling upgrades

Mon, 27 Nov 2017 00:00:00

https://blueprints.launchpad.net/glance/+spec/database-strategy-for-rolling-upgrades

This spec outlines a database modification strategy for Glance that will facilitate zero-downtime rolling upgrades and make it possible for Glance to assert the assert:supports-zero-downtime-upgrade tag.

Problem description

In order to apply the assert:supports-zero-downtime-upgrade tag [GVV2] to Glance, the assert:supports-rolling-upgrade tag [GVV1] must first be asserted. It states that

The project has a defined plan that allows operators to roll out new code to subsets of services, eliminating the need to restart all services on new code simultaneously.

In order to assert the assert:supports-zero-downtime-upgrade tag, Glance must completely eliminate API downtime of the control plane during the upgrade. A key issue for Glance in this regard is how to handle database changes required for release N while release N-1 code is still running. We outline a strategy by which this may be accomplished below.

Note

In what follows, we make the assumption that an operator interested in rolling upgrades will meet us halfway, that is, will be using an up-to-date version of the underlying DBMS that supports online schema changes that allow as much concurrency as possible.

Proposed change

We propose an expand and contract strategy to implement database changes in such a way that a rolling upgrade may be accomplished within the confines of a single release. Some OpenStack services, such as Cinder, that have addressed this problem have chosen to make database changes over a sequence of releases, but we believe that given the structure of Glance and typical usage patterns, database changes can be made and finalized within a single release. Such an approach is preferable for an open source project in which the cast of characters may change considerably from cycle to cycle.

We first present an overview of the upgrade strategy and then provide a detailed example of how this will work for a change that will be occurring in the Ocata release.

Overview

The following diagram depicts a typical upgrade of an OpenStack service. The older services are shutdown completely (mostly during a maintenance window), the new code is deployed and finally the new services are started. Obviously, this involves some downtime for the users. To minimize/eliminate downtime, the services can be upgraded in a rolling fashion, that is, upgrading few services at a time. This results in a situation where both old (say N-1) and new (say N) services must co-exist for a certain period of time. In a straightforward upgrade where the new services have no database changes associated with them, the services can co-exist right from the onset as both rely on the same schema.

                              |        |         |
                              |        |         |
                              |        |         |
     -----------------------+ |        |         | +---------------------
                            | |     Deploy N     | |
                  N-1       | |  (upgrade code   | |     N
                            | |  and/or config   | |
     -----------------------+ |  and migrate db) | +---------------------
                              |        |         |
                            Stop N-1   |       Start N
                            Services   |       Services
                              |        |         |
                              |        |         |
                              |        |         |
                              |        |         |
                              |        |         |
                              |        |         |

                               <---------------->
                                     Downtime
                               <---------------->

However, in the presence of database changes it isn’t yet possible for the services to co-exist. The primary reason is the way we do database changes/migrations currently. A typical migration in Glance is an atomic change that includes both schema and corresponding data migrations together. While the schema migrations perform necessary additions/removals/modifications to the schema, data migrations perform corresponding changes to the data. This approach at times, depending on the nature of schema changes, is backwards-incompatible. That is, older services may not be able to run with new schema. This essentially limits the ability for old and new services to co-exist and consequently prohibits rolling upgrades.

To achieve rolling upgrades, database migrations need to be done in such a way that both old and new services can co-exist over a period of time. A well known strategy is to re-imagine database changes in expand and contract style instead of one atomic change. With the expand and contract style, we achieve the desired schema changes in two distinct steps:

Expand: In the expand step, we make only additive changes that are required by the new services. This keeps the schema intact for older services to run along with the new services. The typical schema changes that fall into this category are adding columns and tables.

An exception to this additive-only change strategy is that it may be necessary to remove some constraints in order to allow database triggers (discussed below) to work.

Contract: All the other changes, that is non-additive changes, are grouped into contract step. Changes like removing a column, table and/or constraints are made in this step.

Additionally, if any constraints were removed during the expand step, they are restored during the contract phase. Any database triggers installed during the expand phase are also removed at this point.

This breakup gives us the ability to perform the minimum required changes first (while keeping schema compatibility with old services) and delay the other changes until a later point in time. Therefore, we always first expand the database in order to start a rolling upgrade while the old services are still running. Once the database is expanded, the new columns and tables are created. However, they would be empty. At this point, we should start migrating the data over to the new column. But, at the same time, it is important to keep the new and old columns in sync. Any writes to old column must be sync-ed to the new column. And, vice-versa (although we don’t write to the new column yet, we have to keep the old column in sync when the new services come up and start writing to the new column while the services co-exist). We use database triggers to keep the columns in sync.

We add the triggers to the database along with the additive changes during the database expand. At this point, we start migrating the data over to the new column. However, because the old services are live at this point, we migrate the data in small batches to avoid excessive load on the database and thereby any interruption to the old services. These migrations can be scheduled to run during low-traffic hours to minimize impact on older services. Once the data migrations finish, the new column is populated and ready for use, we start deploying the new services.

We deploy services in small batches by taking some nodes out of rotation, wait for them to drain connections, upgrade the services and put the nodes back into rotation. It is during this period that old and new services co-exist. When the new services come up, they start reading from and writing to the new column. Any data written to the new column is synced over to the old column (by the triggers added during database expand) and available for older services to consume. Once all the older services are upgraded, it is now safe to contract the database. This ensures that we reach the desired state of database schema. We also drop the database triggers during the database contract because the old column would cease to exist and only the new column would be in use.

         ---------------------------------------+
                                                |
                         N-1                    |
                                                |
         ---------------------------------------+

                 |    |           |    |         |     |
                 |    |        Finish  |         |     |
                 |    |         Data   |
              Expand  |      Migrations| +----------------------------
             Database |           |    | |
                 &    |           |    | |     N
                Add   |           |    | |
              Triggers|           |    | +----------------------------
                 |    |           |    |
                 |    |           | Start N      |     |
                 |  Start         |  Deploy      | Contract
                 |  Data          |    |         | Database
                 | Migrations     |    |         |     &
                 |    |           |    |         |   Drop
                 |    |           |    |         |  Triggers
                 |    |           |    |     Finish N  |
                 |    |           |    |      Deploy   |
                 |    |           |    |         |     |
                 |    |           |    |         |     |

To summarize, as shown in the above diagram, we split the database migrations into schema and data migrations. The schema migrations can be additive or contractive in nature, or a combination of both. Additive schema migrations are run before the upgrade begins to prepare the database for new services while it is still usable by old services. (This phase is also called “database expand”.) During database expand, we also add triggers on old and new columns to keep them in sync. Once the database is expanded, we start migrating the data over to the new column in small batches. When the data migrations are complete, we upgrade the old services in a rolling fashion. Once all the old services are upgraded, we run the contractive migrations on the database. (This phase is also called “database contract”.) The triggers are also dropped during database contract.

In addition to the description of the process given above, here are a few constraints on how upgrades will work:

A typical upgrade is complete only when the entire expand-migrate-contract cycle for a release is performed. We do not propose to support an N-1 to N upgrade while an N-2 to N-1 upgrade is in progress.
“Leapfrogging” releases (that is, allowing a direct N-2 to N upgrade, skipping N-1) is not supported.
It’s possible that in a single release there may be multiple features, worked on independently by different developers, that will require some kind of database modification. What we are proposing in this spec is that for each release, there will be a single expand-migrate-contract operation from the operator’s point of view. In other words, all feature teams will have to coordinate so that all expands are performed, followed by all migrations, and concluding with all contractions. This will be easy for features whose changes are completely independent, but may be more difficult for others. However, preserving zero-downtime database changes will be a Glance project priority once this spec has been approved, so such interactions will be addressed on the specs for features.

Note

The current Glance spec template asks this question in the “Data model impact” section:

What database migrations will accompany this change?

This should be modified along the following lines. (Note: this is only a suggestion, we can argue about the best wording on the patch that modifies the spec template.)

Glance is committed to zero-downtime database migrations. Explain what database migrations will accompany this change. Do they have the potential to interfere with the database migrations for other specs that have been approved for this cycle?

Keep in mind that our goal here is to achieve the upgrade tags. While it’s not prohibited to exceed them, they do specify a baseline for achievement that’s been adopted by the OpenStack community. Hence simply meeting the requirements for the tags is a worthwhile goal.

Steps

Let’s look at a rolling-upgrade strategy for Glance in more detail. Consider the case where a database change is made such that data stored in “the old column” in release N-1 will be stored in “the new column” in release N. The following are steps that we take to achieve rolling-upgrade.

Expand Database

Goal: Prepare database for N by expanding the database

As shown in the below diagram, initially, we have N-1 reading from and writing to the old column. We then expand the schema for N which introduces the new column while N-1 is still running. This should have minimal to no impact on N-1 services.

Note

It is important to note that while database expand operations are required to be strictly additive in nature, adding constraints can sometimes be disruptive as they are known to lock the table. This is alleviated by online DDL capabilities in MySQL 5.6 for InnoDB. So, simple changes may not be a concern. (In any case, the plan proposed in this spec adds constraints during the contract phase only.)

     -----------------------------------------------------------

                               N-1

     -------+-----------------------+---------------------------
            |                       |
            |           |           |                          |
         Read/Write     |        Read/Write                    |
            |       Expand N        |                          |
            |           &           |                        Start
       +----v-----+    Add     +----v-----+----------+        Data
       |   Old    |  Triggers  |   Old    |   New    |     Migrations
       +----------+     |      +---------------------+         |
       |          |     |      |          |          |         |
       |          |     |      |          |          |         |
       |          |     |      |          |          |         |
       |          |     |      |          |          |         |
       |          |     |      |          |          |         |
       |          |     |      |          |          |         |
       |          |     |      |          |          |         |
       |          |     |      |          |          |         |
       |          |     |      |          |          |         |
       |          |     |      |          |          |         |
       +----------+     |      +----------+----------+         |
                        |          ^             ^             |
                        |          |  Triggers   |             |
                        |          +-------------+             |
                        |                                      |
                        | <--------------------------------- > |
                        |           Expand Database            |
                        | <--------------------------------- > |

While expanding the database, we also add triggers that keep the old and new columns in sync.

Deliverable: We propose to make this available by extending the glance-manage utility with an expand command. The database could be expanded by running glance-manage db expand.

Migrate Data

Goal: Populate the new column(s) for N to use

At this point, only release N code is running and it continues to read from and write to the old column as shown in the below diagram. All writes made by N-1 to the old column are synced with the new column. While the triggers slowly start populating the new column, we commence the background data migrations to populate data into the new column in a non-intrusive manner.

       -------------------------------------------------

                           N-1

       -----------------+-------------------------------
                        |
            |           |                          |
            |        Read/Write                    |
            |           |                          |
          Start         |                        Finish
          Data     +----v-----+----------+        Data
       Migrations  |   Old    |   New    |     Migrations
            |      +---------------------+         |
            |      |          |          |         |
            |      |          |          |         |
            |      |          |          |         |
            |      |          |          |         |
            |      |          |          |         |
            |      |          |          |         |
            |      |          |          |         |
            |      |          |          |         |
            |      |          |          |         |
            |      |          |          |         |
            |      +----------+----------+         |
            |          ^             ^             |
            |          |  Triggers   |             |
            |          +-------------+             |
            |                                      |
            | <--------------------------------- > |
            |            Migrate Data              |
            | <--------------------------------- > |

Deliverable: We propose to extend the glance-manage utility to migrate the data in batches. The batch size could be controlled with an optional parameter, for example, max_rows. The parameter would allow operators to schedule migrations of no more than N rows at a time, in case they have a large database and want to run the migration only during off-peak times. Without the optional parameter, all rows would be migrated. The utility will return an appropriate response if it’s run and it finds that there are no rows that need to be migrated.

For example: glance-manage db migrate --max_rows=10.

Deploy

Goal: Deploy N by upgrading N-1 in a rolling fashion and have both versions: co-exist during the deploy

As the new column is now ready to use, we start deploying N in small batches. Release N-1 has no idea that an upgrade is occurring, but the release N code is co-existing with N-1 services as shown in the below diagram. While N-1 and N services are using the old and new column respectively, the triggers are keeping the two columns in sync whenever there is a database write. This enables N to see the updates made by N-1 and vice-versa.

                ------------------------------------+
                                                    |
                                    N-1             |
                                                    |
                --------------+---------------------+    |
                              |                          |
                   |          |
                   |          |   +------------------------------
                   |          |   |
                   |          |   |   N
                   |          |   |
                   |          |   +-------+----------------------
                   |          |           |
                   |        Read/       Read/            |
                   |        Write       Write            |
                   |          |           |              |
                   |          |           |              |
                   |      +---v------+----v-----+     Finish N
                   |      |  Old     |   New    |      Deploy
                Start N   +---------------------+        |
                Deploy    |          |          |        |
                   |      |          |          |        |
                   |      |          |          |        |
                   |      |          |          |        |
                   |      |          |          |        |
                   |      |          |          |        |
                   |      |          |          |        |
                   |      |          |          |        |
                   |      |          |          |        |
                   |      |          |          |        |
                   |      +----------+----------+        |
                   |          ^             ^            |
                   |          |  Triggers   |            |
                   |          +-------------+            |
                   |                                     |
                   |                                     |
                   |    <---------------------------->   |
                   |               Deploy                |
                   |    <---------------------------->   |
                   |                                     |

Note

As the N-1 and N services co-exist, users may notice inconsistent behavior in certain situations. Typically, a new release is backwards-compatible with the previous release. As such, all requests should exhibit similar behavior across both versions. However, some changes to the API (for example: bug fixes) may result in different behavior across two releases. So, a user may witness different responses to similar requests depending upon which service processes the request.

Similarly, user requests for new features introduced with the new version, may fail when they are processed by an older service. While this inconsistency is not desirable, it could be seen as a decent compromise over incurring downtime during the upgrade.

Contract Database

Goal: Complete the schema migrations desired in N

When all the services are upgraded, the old column is unused. The new column would be the source of truth henceforth. The old column is ready for removal. At this point, we contract the database, which drops the old column and triggers added during database expand.

              |
              |

         -------------------------------------------

                                N

         -----------------------+-------------------
                                |
              |               Read/
              |               Write
              |                 |
              |                 |
          Contract          +---v----+
           Database         |   New  |
              &             +--------+
            Drop            |        |
           Triggers         |        |
              |             |        |
              |             |        |
              |             |        |
              |             |        |
              |             |        |
              |             |        |
              |             |        |
              |             |        |
              |             ---------+
              |
              |
              |     <----------------------->
              |         Contract Database
              |     <----------------------->
              |

Note

In addition to removing the unused columns and tables, SQL constraints such as nullability, unique and default must be set here.

Deliverable: We propose to make this available by extending the glance-manage utility with a contract command. The database could be contracted by running glance-manage db contract.

Rolling Upgrade Process for Operators

Following is the process to upgrade Glance with zero downtime:

Backup Glance database.
Choose an arbitrary Glance node or provision a new node to install the new release. If an existing Glance node is chosen, gracefully stop the Glance services.
Upgrade the above chosen node with new release and update the configuration accordingly. However, Glance services MUST NOT be started just yet.
Using the upgraded node, expand the database using the command glance-manage db expand.
Then, schedule the data migrations using the command glance-manage db migrate --max_rows=<max. row count>.

Data migrations must be scheduled to run until no more rows are left to migrate.
Start the Glance processes on the first node.
Taking one node at a time from the remaining nodes, stop the Glance processes, upgrade to the new release (and corresponding configuration) and start the Glance processes.

Note

Before stopping the Glance processes on a node, one may choose to wait until all the existing connections drain out. This could be achieved by taking the node out of rotation. This way all the requests that are currently being processed will get a chance to finish processing. However, some Glance requests like uploading and downloading images may last a long time. This increases the wait time to drain out all connections and consequently the time to upgrade Glance completely. On the other hand, stopping the Glance services before the connections drain out will present the user with errors. This can at times be seen as downtime as well. Hence, an operator must be judicious when stopping the services.

Contract the database by running the command glance manage db contract from any one of the nodes.

Example

To understand how this would work in action, consider the following example of a Glance database change proposed for Ocata.

Note

This does not prescribe the actual Ocata database change. It is included here as a realistic example for a sanity check of this proposal.

The “old column”: Newton (release N-1): boolean is_public column in images table. This column has nullable=False and default=False.

The “new column”: Ocata (release N) : enum (or string … key point is it’s a different data type) visibility column in images table. This column can have one of the values ‘public’, ‘private’, ‘shared’, ‘community’. After the database contraction has completed, this column would have nullable=False, and default=’private’. (During the migrate and deploy phases, this column will probably have nullable=true with no default.)

Using the proposed strategy, the database upgrade would proceed as follows.

Pre-upgrade: Version N-1 code read/write to is_public.
Expand Database: Add the visibility column and the appropriate triggers to keep the old and new values in sync.
Migrate Data: Crawl the ‘images’ table. For any row where visibility is null, set the value for visibility as follows:
- If is_public is ‘1’: set visibility to public
- If is_public is ‘0’: if the image has any members, set visibility to shared; otherwise set visibility to private
Migrate any data (using triggers) written by N-1 code to the old column using the above criteria.
Deploy: Deploy N code in a rolling fashion. N code will start using the visibility column.

Here’s an analysis of database activity.
- Write operations
  - The v1 API
    - nothing to worry about, has no concept of visibility
  - The v2 API
    1. visibility set to public
      - N-1: will put ‘1’ in is_public * Triggers will put ‘public’ in visibility
      - N: will put ‘public’ in visibility * Triggers will put ‘1’ in is_public
    2. visibility set to private
      - N-1: will put ‘0’ in is_public * Triggers will put ‘private’ in visibility
      - N: will put ‘private’ in visibility * Triggers will put ‘0’ in is_public
    3. visibility set to community
      - N-1: call will fail at API level, will never hit the database
      - N: will put ‘community’ in visibility * Triggers will put ‘0’ in is_public
        
        Note
        
        This essentially means that a community image will be considered as private image by N-1. Thus, barring the owner, a community image won’t be visible to any one. Since N-1 has no notion of community images, this behavior can be seen as consistent with respect to N-1. However, it may be confusing the owner of the community image for whom the image will appear as community with N and private with N-1. Thus, the owner may try to change the visibility again. To discourage this, we may prohibit any writes to the is_public column when it has ‘0’ and visibility column has community. This could be done again by using the same triggers that we added during database expand. The first alternative mentioned in the alternatives section avoids this situation.
    4. visibility set to shared
      - N-1: call will fail at API level, will never hit the database
      - N: will write ‘shared’ in visibility
        
        Triggers will write ‘0’ in is_public; this will allow image sharing to continue to work properly on the release N-1 nodes as well as the v1 API on all nodes.
- Read operations
  - Read operations across API versions and releases should remain unaffected as the triggers keep both old and new columns in sync by translating the data appropriately.
Contract Database: The only API nodes running are version N. The is_public column is no longer in use. Drop is_public and add nullable=True and default=private on visibility column.

Alternatives

This is a small variant of the above described strategy. The fundamental idea behind the above described strategy is: When both versions co-exist, sync the writes made by one set of services to be available for the others to consume. We achieve this using triggers. On the other hand, what if we eliminate the need to sync? That is, what if we disallow any writes to both old and the new columns while the services co-exist? This can be achieved by using triggers again. Essentially, the triggers we add during the database expand step will intercept and disallow writes to the old and new columns.

For the above given example, all requests attempting to change the visibility of image will fail for the duration of deploy step where the services co-exist. Reads would be permitted, however. Once the deploy is finished and we contract the database (the triggers are dropped here), writes to new column would be permitted as usual. This gives us a way to eliminate the need for syncing data across columns. Consequently, there is much less complexity in the triggers and the upgrade is less error-prone. However, it is important to note that one may see an increased error rate during the deploy due to disallowed writes. Although the API will be responsive throughout this entire period (and hence “up”), the increased rate of 5xx responses will make it impossible to assert the assert:zero-downtime-upgrade tag [GVV2]. Since being able to assert this tag is the aim of this spec, this alternative is not acceptable.
A well known alternative replaces the use of triggers by migrating the data online from within the application. While the triggers approach migrates the data online on a database write operation, the other approach attempts to migrate the data on an on-demand basis in the event of a database read operation.

Approaches taken by other Openstack Projects:

Nova: See [NOV1].

Cinder: See [CIN1].

Keystone: See [KEY1].

Neutron: See [NEU1], [NEU2].

Data model impact

None

REST API impact

There would be no impact to REST API contracts as such.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

The background data migration will consume extra database resources, but this can be managed if the migration script is carefully written.

Other deployer impact

Deployers who intend to deploy Glance the old way, that is with downtime, remain unaffected.
Each step of the migration requires operator intervention.

Developer impact

Any developer working on a feature that requires database changes must write additional code to support the rolling upgrade strategy outlined in this document. By confining the database changes to a single release, however, developers of release N+1 do not have to worry about completing procedures begun during the migration of release N-1 to N.

Implementation

Assignee(s)

Primary assignee:: alex_bash hemanthm
Other contributors:: nikhil

Work Items

Write documentation for rolling upgrade (developer docs).
Write documentation for rolling upgrade (operator docs).
Introduce expand/contract migration streams and the corresponding glance-manage CLI.
Work with the developers of Ocata features that require database changes to implement code to follow the rolling upgrade strategy. These include:
- community images and enhanced image sharing
- image import

Dependencies

None

Testing

In order to assert the rolling upgrades tag, Glance must have full stack integration testing with a service arrangement that is representative of a meaningful-to-operators rolling upgrade scenario.

Ideally these tests will be able to simulate Glance running at scale, since, as discussed above, some DBMS problems may not be revealed in a small test database.

Documentation Impact

Developer documentation: the upgrade strategy.
Operator documentation:
- configuration options for putting the code into the various modes
- running the database scripts

References

[GVV1]

https://governance.openstack.org/reference/tags/assert_supports-rolling-upgrade.html

[GVV2] (1,2)

https://governance.openstack.org/reference/tags/assert_supports-zero-downtime-upgrade.html

[NOV1]

http://www.danplanet.com/blog/2015/10/07/upgrades-in-nova-database-migrations/

[CIN1]

https://specs.openstack.org/openstack/cinder-specs/specs/mitaka/online-schema-upgrades.html

[KEY1]

https://specs.openstack.org/openstack/keystone-specs/specs/mitaka/online-schema-migration.html

[NEU1]

https://specs.openstack.org/openstack/neutron-specs/specs/liberty/online-schema-migrations.html

[NEU2]

http://docs.openstack.org/developer/neutron/devref/upgrade.html

Archived Priority Lists

Mon, 30 Oct 2017 00:00:00

During each Project Team Gathering (or “design summit”), we agree on what the whole community wants to focus on for the upcoming release. This is the historical archive of the output of those discussions.

Spec Lite: Actually Deprecate the Glance Registry

Mon, 30 Oct 2017 00:00:00

problem:: A spec to Deprecate the Glance Registry Service was accepted in Newton, but it contained the ambiguous statement, “Mark the service as deprecated and ready for removal in the Q release.” It’s now the Q release, so we need to actually deprecate it by announcing officially that the Registry Service is deprecated in Queens and subject to removal in the S release.
solution:: On startup, the Registry Service should output an appropriate message to the log.
timeline:: Q-1 milestone (week of 16 October 2017)
link:: https://blueprints.launchpad.net/glance/+spec/deprecate-registry
reviewers:: all core reviewers
assignee:: rosmaita

Spec Lite: Neutralize Language

Mon, 30 Oct 2017 00:00:00

problem:

The glance_store library is being consumed by another project (Glare) but much of the language in Exception messages, log messages, and configuration option help text talks about “images”.

Additionally, some of the names of configuration options are image-centric. (Actually, I could only find one of these):

vmware_store_image_dir (and its default value is ‘openstack_glance’)

solution:

Make the language in Exception messages, log messages, and the configuration option help text neutral, for example, replace ‘image’ by ‘object’ and any other appropriate language changes.
Introduce new configuration options for any options with image-centric names and deprecate the old options using the facilities oslo.config supplies for this purpose.

impacts:

The deprecated options will be listed in a release note.

Spec Lite: Rolling Upgrade Testing

Sat, 28 Oct 2017 00:00:00

problem:: Zero-downtime database upgrades were introduced on an EXPERIMENTAL basis in the Ocata release to facilitate rolling upgrades. In order for rolling upgrades to be considered officially supported, and to allow Glance to assert the associated TC tags, we need to have in-gate testing of upgrades.
solution:: Add in-gate testing of upgrades using Grenade or some other appropriate framework.
impacts:: DocImpact: update the docs to reflect the non-experimental status of zero-downtime database upgrades.
link:: https://blueprints.launchpad.net/glance/+spec/rolling-upgrade-tests

Lite Spec: Community Goal: Control Plane API endpoints deployment via WSGI

Thu, 24 Aug 2017 00:00:00

problem:: Implement the Pike community goal Control Plane API endpoints deployment via WSGI.
solution:: Implement a devstack plugin to run the Images API v2 supplied by Glance in mod_wsgi.
impacts:: None.
link:: Patches
assignee:: Matt Treinish

Add ‘protected’ filter to image-list call

Thu, 10 Aug 2017 00:00:00

https://blueprints.launchpad.net/glance/+spec/add-protected-filter

Images contain a boolean ‘protected’ field. Filtering on this field in the image-list request is not currently supported. Operators and users who make use of the ‘protected’ field to prevent accidental deletion of images, however, would find such filtering useful. Not supporting filtering on this field is counterintuitive as Glance supports filtering on all other image fields.

Problem description

Use cases:

An End User wants to list protected images that are accessible to that end user.
An End User wants to list non-protected images that are accessible to that end user.

Proposed change

Add code so that when ‘protected=<value>’ is included in a query string on the image-list call, the value is converted to a boolean, making it suitable for database filtering (the ‘protected’ field is a boolean in the database backend).

For a user experience consistent with the image-create and image-update calls, the only accepted values for this parameter will be ‘true’ or ‘false’ in that exact case combination. Any other value for this parameter should return a 400 to indicate a bad request.

To summarize:

Introduce a new query parameter to the GET v2/images call, namely, protected.
The parameter will act as a filter on the ‘protected’ field of an Image.
The set of values for the parameter will be strict, that is, the call will accept only those strings that satisfy the JSON boolean data type (in other words, only ‘true’ or ‘false’ in that exact case combination).
Unrecognized values will result in a 400 (Bad Request) response with an appropriate message.
When the protected query parameter is not present, no filtering will be done on the ‘protected’ field of Images. (In other words, there’s no default value.)

Alternatives

Do nothing. This isn’t really an option, because even though ‘protected’ is not currently supported as a filter, image-list calls passing a value for ‘protected’ in the query string are accepted and do have an effect (the API behaves as if protected=False has been specified). This behavior appears weird to end users.
Implement the filter, but accept a liberal set of values, recognizing, for example, ‘true’ or ‘false’ in any case combination; ‘yes’ or ‘no’ in any case combination; and 1 or 0. This can be accomplished by using the oslo string utilities, which contains a function bool_from_string that converts strings to an appropriate boolean value [APF-1]. Using the oslo library has an additional bonus in that the usage will be consistent across other OpenStack projects.

The downside to this alternative is that it is inconsistent with the image-create and image-update calls in the Images v2 API. The latter calls are governed by the Image json-schema, which clearly specifies a JSON boolean type as the value of the protected field, and hence only ‘true’ or ‘false’ in that exact case combination is what these calls accept. Thus, liberal acceptance of commonly regarded boolean values in the query string may cause API users to be surprised when these liberal values don’t work for image-create or image-update, leading to user dissatisfaction.
Implement the filter, but be extremely lax, that is, any value that isn’t mapped to the boolean value True by the oslo bool_from_string function would be considered False. This actually isn’t very user friendly. For example, a French language speaker applying the query filter ‘protected=oui’ will receive the complement of what was requested.

Data model impact

None

REST API impact

This is a very minor modification to the GET v2/images call. The only changes are:

Parameters which can be passed via the URL now include protected.

Security impact

None.

Notifications impact

None.

Other end user impact

The only impact on python-glanceclient is that the command

glance image-list --property protected=true

will now work correctly.

Performance Impact

None.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:

fengzhr

Other contributors:

rosmaita
dharinic

Work Items

Implement the change roughly along the lines of https://review.openstack.org/#/c/449108/
Update the api-ref
Write a release note

Dependencies

None.

Testing

Functional tests consistent with the current image-list filtering tests.

Documentation Impact

Small addition to the v2 image-list documentation in the api-ref.

References

[APF-1]

http://git.openstack.org/cgit/openstack/oslo.utils/tree/oslo_utils/strutils.py

Lite Spec: Introduce Glance Taskflow stopfile feature

Wed, 31 May 2017 00:00:00

problem:: When preparing to take a Glance node down for maintenance or upgrade it is necessary to allow long-running operations to complete without allowing new operations to begin. The Taskflow engine does not have the capability to prevent individual executors from starting new jobs, and so attempting to take a Glance node out of the Taskflow processing pool risks a race condition with that executor starting a job just before the service is terminated which could cause the Task processing to fail.
solution:: Introduce a disable_by_file_path feature to Glance Taskflow which will prevent the node from picking up new jobs. This allows an operator or automation engine to touch the appropriate file before terminating the service. This feature should depend on the oslo healthcheck middleware configuration to disable the taskflow engine. As an additional impact, this work will require Glance to instantiate a single taskflow engine as a singleton and reuse that engine instance on all subsequent taskflow operations.
impacts:: None
timeline:: Expected to have a fix merged in the Pike cycle before milestone 2.
link:: https://docs.openstack.org/developer/oslo.middleware/healthcheck_plugins.html
assignee:: Open

HealthCheck Middleware

Thu, 25 May 2017 00:00:00

https://blueprints.launchpad.net/glance/+spec/healthcheck-middleware

HealthCheck Middleware provides <SERVER>:<PORT>/healthcheck endpoint. This endpoint behaves similar way as it does in Swift[1]. Normal operations returns 200 OK and if config option disable_path is enabled and that path exists it will return 503 DISABLED BY FILE.

Problem description

Currently Glance does not have any reasonable means to provide health check information for example to HAProxy. Any such deployment causes extensive logging and causes unnecessary operations in the server end (FE. GET request to / which leads to version check).

Proposed change

Take advantage of oslo.middleware‘s healthcheck middleware by:

Adding a new paste filter - healthcheck
Adding the new filter to the default Glance API and Registry pipelines

Disabled functionality would allow using a file in the filesystem to return 503 DISABLED BY FILE response dropping the node from the HAProxy. This would allow the nodes disabled to finish their current operations and improve maintenance experience in HA deployments.

Alternatives

Current operations model.

Take Swift’s slightly smaller implementation and copy it directly into Glance’s source tree.

Data model impact

None

REST API impact

GET ‘/healthcheck’ 200 OK, 503 DISABLED BY FILE

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Potential performance improvement as healthcheck is done quite frequently and the middleware is lightweight filter at the start of the pipeline.

Other deployer impact

healthcheck would be added to the start of the used pipeline to be used. Optional disable_path=PATH option would be needed to the config files to enable the discreet disable functionality.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: jokke kamil-rykowski
Core Reviewers:: kragniz flaper87 icordasc

Work Items

Code Change Tests Config Change Documentation Change

Dependencies

None

Testing

Simple functional tests to verify the responses form the <SRV>:<P>/healthcheck

Documentation Impact

Documentation changes are quite minimal explaining the new config option and functionality.

References

[1] _https://github.com/openstack/swift/blob/master/swift/common/middleware/healthcheck.py

HTTP Proxy Support for Glance S3 Driver

Thu, 25 May 2017 00:00:00

https://blueprints.launchpad.net/glance/+spec/http-proxy-support-for-s3

Currently the S3 store does not allow operators to connect to an S3 backend through a proxy. This can create limitations on the ability to connect to the S3 backend securely from a different network. I propose to add the option to use a proxy to connect to an S3 backend.

Problem description

If glance store is configured to use the S3 backend and the backend is behind a private network and needs to be accessed remotely, there is no secure way to access the S3 backend securely.

Proposed change

Boto, the library that is used to make the connection to the S3 backend, already supports proxy configurations. I propose that we enable the connection to accept additional config options to give users the option to connect through a proxy.

The following configurations would be added:

s3_store_enable_proxy: Enables the use of a proxy
s3_store_proxy_host: The proxy server (required when proxy is enabled)
s3_store_proxy_port: The port to connect to the proxy
s3_store_proxy_user: The username of the proxy connection.
s3_store_proxy_password: The password to be used to connect through the proxy.

Alternatives

The user can use system wide proxy parameters, but would limit the ability to connect from an outside network.

Data model impact

None

REST API impact

None

Security impact

This would introduce security settings to be modified by user. The ability to connect through a proxy will provide a good way to secure connections.

Notifications impact

None

Other end user impact

This introduces proxy configuration options in the store configuration.

Performance Impact

None

Other deployer impact

This change will have to be explicitly configured in the store options.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: cpallares

Reviewers

Core reviewer(s):: flaper87 sigmavirus24
Other reviewer(s):: rosmaita

Work Items

Add configurations (proxy name, port, user, password, default number of retries to S3, etc).
Modify connections made to S3 to optionally accept proxy parameters.
Create additional unit tests for connections made to the S3 backend using a proxy.

Dependencies

None

Testing

Unit testing will be needed for testing proxy connection.

Documentation Impact

Documentation for the S3 store will need to be updated to include proxy opts.

References

Glance Image Signing and Verification

Thu, 25 May 2017 00:00:00

https://blueprints.launchpad.net/glance/+spec/image-signing-and-verification-support

OpenStack currently does not support the following feature:

Signature validation of uploaded signed images

Deploying authentication will protect image integrity by verifying that an image has not been modified after the upload by the user. This feature improves the enterprise-ready posture of OpenStack.

Problem description

There is no method for users to verify that a previously uploaded image has not been modified. An image could potentially be modified in transit (such as when it is uploaded to Glance or transferred to Nova) or Glance itself could be untrusted and modify images without a user’s knowledge. An image that is modified could include malicious code. Providing support for image signatures and signature verification would allow the user to verify that an image has not been modified prior to booting the image.

There are several use cases that this feature will support:

An image is signed by an End User, using the user’s private key. The user then uploads the image to Glance, along with the signature created and a reference to the user’s public key certificate. Glance uses this information to verify that the signature is valid, and notifies the user if the signature is invalid.
An image is created in Nova, and Nova signs the image at the request of the End User. When the image is uploaded to Glance, the signature and public key certificate reference are also provided. Glance verifies the signature before storing the image, and notifies Nova if the signature verification fails.
A signed image is requested by Nova, and Glance provides the signature and a reference to the public key certificate to Nova along with the image so that Nova can verify the signature before booting the image.

Proposed change

For the initial implementation, this change will use the property feature of Glance to store the metadata items needed for image signing and verification. These include a public key certificate reference, and the signature. These are provided when the image is created, and are accessible when the image is uploaded. Note that this proposed change will only support image uploads with the Glance API v2 (and will not support using the Glance API v1). Also note that multiple formats for the key (such as SubjectPublicKeyInfo) and for the signature (such as PSS) will be supported. The format of the signature will be stored as one of the properties.

The certificate reference will be used to access the certificate from a key manager, where the certificate will be stored. This certificate is added to the key manager by the end user before uploading the image. Note that the signature is done offline.

Glance already supports computing checksums of images when an image is uploaded, and this checksum is stored with the image. This same hash (which by default is MD5) will be used for the signature verification.

The checksum hash is computed in glance_store (when the image data is uploaded), and is then used to verify the signature. The Glance frontend should use the reference to the public key certificate to retrieve the certificate from the key manager, and then use this public key along with the signature, the computed checksum, and the rest of the signature metadata to verify the signature. If the signature verification fails, the image will transition to a killed state, and the user will be notified that the upload failed and given a reason why.

Alternatives

An alternative to using the hash already created in the store backend for the signature verification/creation is to compute a hash in the store frontend. However, eventlet.wsgi.Input file-like object that represents the image data can only be read once, and needs to be read in the store backend in order to upload the image. In order to read the image data in the Glance frontend, Glance could copy the data into a file, use the file to verify/create the signature, and then give this file to the store backend to upload. This would be similar to what is done with the S3 backend [1]. However, this approach would take significantly more time (during image upload), and there would not be much gained.

An alternative to using the existing MD5 hash algorithm is to create a separate configurable hash for use with verifying/creating the signature. However, creating a separate hash negatively affects the performance, without providing much benefit. Note that since there are preferable hash algorithms to MD5 that are more secure, a separate change is being proposed to allow for the configuring of this hash algorithm [2]. This will not be included as a part of this change, in the interest of having a straightforward initial implementation.

Data model impact

None.

REST API impact

No API changes will be needed for the initial implementation, provided that other services are able to retrieve all of the properties of a given image.

Note that the existing API allows for providing the signature metadata as Glance properties, and returning an error message if verification fails.

Security impact

This change improves the enterprise-ready posture of OpenStack by enabling signature signing and verification.

Although keys are used in this change, the keys themselves are assumed to be stored in a key manager, and only a reference to the certificate is stored in Glance.

This change involves hashing the image data for use in verifying and creating signatures for the image.

Notifications impact

This change will involve adding log messages to indicate the success or failure of signature verification and creation.

Other end user impact

The user will be required to provide the appropriate information needed for the signing and verification in order to use this feature.

There are no changes that need to be made to python-glanceclient.

Performance Impact

The feature will only be used if a user has provided the appropriate properties during the image upload. Otherwise, no signature verification or creation will occur.

When signature verification and creation do occur, there will be some latency associated with retrieving the certificate from the key manager. Since the hash is already being created for images, the hash creation has no impact to performance.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: brianna-poulos
Other contributors:: dane-fichter

Reviewers

Core reviewer(s):: flaper87 sigmavirus24 nikhil_k
Other reviewer(s):: joel-coffman

Work Items

The feature will be tackled in the following stages:

Enable Glance to verify signatures provided by the user during an image upload initiated by the user.
Enable Glance to verify signatures provided by Nova during an image upload of a snapshot taken by Nova.

Dependencies

The cryptography library, which will be used for hash creation and signature verification and creation, is already a part of the global-requirements of OpenStack. However, it is not a part of Glance, and will need to be added there.

Glance currently does not interact with any key managers. Since a key manager is needed to manage the keys, changes will need to be made to allow Glance to retrieve the public key certificate using a key manager. Specifically, Castellan [3] will be used to interface with the key manager chosen. The initial key manager will be Barbican, but Castellan can be configured to use a different backend.

Testing

Before Nova support for this feature is added, unit tests will be sufficient. Once Nova support is added, Tempest tests should ensure that the interaction between Nova and Glance works as expected.

Documentation Impact

This documentation will also include descriptions for each of the following signature metadata properties:

signature: the signature of the “checksum hash” encoded in base64 format
signature_hash_method: the hash method used to create the signature
signature_key_type: the key type used in creating the signature
- valid values are: “RSA-PSS”
signature_certificate_uuid: the uuid used to retrieve the certificate from castellan
mask_gen_algorithm: only used for RSA-PSS, defines the mask generation algorithm used in the signature generation, optional and defaults to “MGF1”
- valid values are: “MGF1”
pss_salt_length: only used for RSA-PSS, defines the salt length used in the signature generation, optional and defaults to PSS.MAX_LENGTH

References

cryptography: https://cryptography.io/en/latest/

[1] http://goo.gl/Y3u3lK

[2] https://review.openstack.org/191542

[3] http://git.openstack.org/cgit/openstack/castellan

[4] https://review.openstack.org/#/c/188874/

Tag Metadata Definitions - python-glanceclient Changes

Thu, 25 May 2017 00:00:00

https://blueprints.launchpad.net/glance/+spec/metadefs-tags-cli

This blueprint adds in supporting code for Metadata Definition (metadef) tags. The metadef tag catalog was approved and added into the Glance server in the Kilo release.

The implemented Kilo spec is here for reference: https://github.com/openstack/glance-specs/blob/master/specs/kilo/metadefs-tags.rst

Problem description

This blueprint specifies the CRUD details for the metadef tags.

The bulk of the work for this blueprint was committed within the Kilo time frame, but, did not make it out of the review process in time for the release.

Proposed change

We are proposing enhancements to the python-glanceclient to allow metadef tags to be created, retrieve, updated and deleted.

The following sub-commands will be added to python-glanceclient:

o md-tag-create –name <NAME> <NAMESPACE>: Adds tag <NAME> to namespace <NAMESPACE>. Retains all other tag names.
o md-tag-create-multiple –names <NAMES> [–delim <DELIM>] <NAMESPACE>: Replaces all tags in the <NAMESPACE> with those listed in <NAMES>. The delimeter used in the list can be overidden with the single char <DELIM> Comma is the default delimter.
o md-tag-delete <NAMESPACE> <TAG>: Deletes the tag <TAG> from the namespace.
o md-tag-list <NAMESPACE>: List all tags associated with the namespace.
o md-tag-show <NAMESPACE> <TAG>: Shows the details of the specified tag.
o md-tag-update –name <NAME> <NAMESPACE> <TAG>: Renames the tag associated with <NAMESPACE> <TAG> to the new <NAME>.
o md-namespace-tags-delete <NAMESPACE>: Deletes all tags associated with the namespace.

The sub-commands will be allowed when specified with –os-image-api-version of 2. Example: glance –os-image-api-version 2 md-tag-list <NAMESPACE>

This is consistent with how the other metadef commands have been implemented.

Alternatives

An alternative to the “help” listing of metadef related sub-commands has been suggested to lessen the amount of “md” related subcommands. This should be handled in a separate bug if done.

Data model impact

None. The following DB schema is the implemented schema in Kilo. Constraints not shown for readability.

Basic Schema:

CREATE TABLE `metadef_tags` (
  `id`                     int(11) NOT NULL AUTO_INCREMENT,
  `namespace_id`           int(11) NOT NULL,
  `name`                   varchar(80) NOT NULL,
  `created_at`             timestamp NOT NULL,
  `updated_at`             timestamp
)

REST API impact

None. The python-glanceclient will use the metadef Tag REST API created in Kilo.

API Version

All URLS will be under the v2 Glance API. If it is not explicitly specified assume /v2/<url>

Security impact

None

Notifications impact

None

Other end user impact

We intend to expose this via Horizon and are working on related blueprints.

Performance Impact

None anticipated.

This is expected to be called from Horizon when an admin wants to annotate tags onto things likes images and instances. This API would be hit for them to get available tags or create new ones.

Other deployer impact

None

Developer impact

None (New API)

Implementation

Assignee(s)

Primary assignee:: wayne-okuma
Other contributors:: None

Reviewers

Core reviewer(s):: zhiyan Ian Cordasco (sigmavirus24) Stuart McLaren
Other reviewer(s):: lakshmi-sampath travis-tripp

Work Items

Changes would be made to:

The python-glanceclient to support the new sub-commands

Dependencies

Same dependencies as Glance.

Testing

Unit tests will be added for all possible code with a goal of being able to isolate functionality as much as possible.

Documentation Impact

Docs needed for new API extension and usage

References

Youtube summit recap of Graffiti Juno POC demo that included tags.

Current glance metadata definition catalog documentation.

Simple application category tags (no hierarchy)

Images, volumes, software applications can be assigned to a category. Similarly, a flavor or host aggregate could be “tagged” with supporting a category of application, such as “BigData” or “Encryption”. Using the matching of categories, flavors or host aggregates that support that category of application can be easily paired up.

Note: If a resource type doesn’t provide a “Tag” mechanism (only key value pairs), a blueprint should be added to support tags on that type of resource. In lieu of that, a key of “tags” with a comma separated list of tags as the value be set on the resource

Migrate glance-replicator to requests for HTTPS Support

Thu, 25 May 2017 00:00:00

https://blueprints.launchpad.net/glance/+spec/migrate-replicator-to-requests

As operators and users become more security conscious, it is important to support deployments of Glance served only over HTTPS. In its current state, glance-replicator uses httplib and thus does not properly verify HTTPS connections. This allows for various and very serious attacks to be performed while the user of glance-replicator attempts to communicate with Glance.

Problem description

Many deployments currently support both HTTP and HTTPS connections to Glance’s API. As best practices evolve, it will become more common that Glance and other OpenStack services are served only over HTTPS with valid X.509 certificates. Currently, if an operator were to deploy Glance and serve it using only HTTPS, glance-replicator would still allow for a large range of attacks by an observer since it does not verify the certificate that the server provides.

Among other things, the user’s connection to Glance could easily be intercepted by a man-in-the-middle serving a phony certificate who would then proxy or even alter the data sent over the connection. Since the typical user of glance-replicator is an administrator, any service token they have could then be intercepted and used, which is dangerous given the privileges associated with an administrator.

Proposed change

This specification proposes that the code using httplib in glance-replicator be rewritten to use requests. requests supports automatic certificate verification on all HTTPS connections and allows users to provide custom certificate bundles for self-signed certificates.

Given that an operator may choose to sign their own ceritificates for their deployment of Glance, this specification also proposes the addition of a command-line option to glance-replicator to allow the operator to specify a custom certificate bundle to use when verifying the certificate.

Alternatives

One alternative to requests that’s already used in other OpenStack projects is httplib2. This library provides a nearly identical API to httplib and performs certificate verifcation. The library, however, is being actively replaced by many of these same projects by requests. Reducing the number of dependencies that an operator needs to install is also very favorable.

An alternative to making the user specify their custom certificate bundle is to provide a glance-replicator.conf file. This would be an entirely new file. Adding yet another configuration file may add to confusion as to which files are necessary when Glance is deployed as a whole.

Data model impact

None

REST API impact

None

Security impact

For deployments of Glance being served over HTTPS, this will improve the security of the user’s connection.

Notifications impact

None

Other end user impact

Users who have not properly configured HTTPS may receive errors. Since glance-replicator previously did not generate errors, this may be an unpleasant experience for the user. It is the position of the author of this specification that an option to insecurely connect to Glance is a poor choice since the errors will encourage the operators to properly configure Glance to be served over HTTPS.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: icordasc
Other contributors:: junhongl

Reviewers

Core reviewer(s):: flaper87 flwang
Other reviewer(s):: nikhil-komawar kragniz

Work Items

Refactor glance-replicator to drop a some of its conventions surrounding httplib
Replace httplib with requests
Add option to specify a custom certificate bundle
Add documentation to glance-replicator surrounding the new option and features

Dependencies

None

Testing

requests-mock will be used to write unit tests for glance-replicator to ensure that proper coverage is achieved.

Documentation Impact

glance-replicator’s man page will need to be updated regarding the new configuration options. We should note the two current ways of setting a custom certificate:

requests will look for REQUESTS_CA_BUNDLE and CURL_CA_BUNDLE environment variables
The new glance-replicator option.

References

Bugs:

https://bugs.launchpad.net/glance/+bug/1408940

Remove custom client SSL handling

Thu, 25 May 2017 00:00:00

https://blueprints.launchpad.net/python-glanceclient/+spec/remove-custom-client-ssl-handling

The Glance client currently supports disabling SSL compression via the –no-ssl-compression argument. This spec proposes deprecating this special handling of SSL.

Note: This is transport layer compression, not application layer (http) compression.

Problem description

Custom SSL handling was introduced because disabling SSL layer compression provided an approximately five fold performance increase in some cases. Without SSL layer compression disabled the image transfer would be CPU bound – with the CPU performing the DEFLATE algorithm. This would typically limit image transfers to < 20 MB/s. When –no-ssl-compression was specified the client would not negotiate any compression algorithm during the SSL handshake with the server which would remove the CPU bottleneck and transfers could approach wire speed.

In order to support ‘–no-ssl-compression’ two totally separate code paths exist depending on whether this is True or False. When SSL compression is disabled, rather than using the standard ‘requests’ library, we enter some custom code based on pyopenssl and httplib in order to disable compression.

This spec proposes removing the custom code because:

It is a burden to maintain

Eg adding new code such as keystone session support is more complicated

It can introduce additional failure modes

We have seen some bugs related to the ‘custom’ certificate checking

Newer Operating Systems disable SSL for us.

Eg. While Debian 7 defaulted to compression ‘on’, Debian 8 has compression ‘off’. This makes both servers and client less likely to have compression enabled.

Newer combinations of ‘requests’ and ‘python’ do this for us

Requests disables compression when backed by a version of python which supports it (>= 2.7.9). This makes clients more likely to disable compression out-of-the-box.

It is (in principle) possible to do this on older versions too

If pyopenssl, ndg-httpsclient and pyasn1 are installed on older operating system/python combinations, the requests library should disable SSL compression on the client side.

Proposed change

Deprecate the ‘–no-ssl-compression’ option. Remove the custom http handling code and print a warning when ‘–no-ssl-compression’ is specified.

Alternatives

Do not deprecate

The cost/benefit of not deprecating would mean that custom code paths would have to be maintained for a small number of corner cases (that can be addressed by other means).

Add dependencies on ndg-httpsclient and pyasn1.

This is a possibility for legacy installations, but this should not be needed for the vast majority of cases.

Data model impact

None

REST API impact

None

Security impact

Certificate checking will no longer be done by custom glance client code, but by the ‘requests’ library. I verified that for older python installs (2.7) certificate checking is performed correctly by the requests library.

Systems that have SSL compression enabled may be vulnerable to the CRIME (https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-4929) attack. Installations which are security conscious should be running the Glance server with SSL disabled.

Notifications impact

None

Other end user impact

SSL potentially not being disabled. A new deprecation warning.

Performance Impact

If SSL is not disabled user’s will experience a performance hit – until they use one of the alternative methods to disable it.

Other deployer impact

Deprecation warnings. Will need to use an alternative method to disable SSL if appropriate.

Developer impact

Should simplify things.

Implementation

Assignee(s)

Stuart McLaren

Reviewers

Ian Cordasco

Work Items

Client change
(small) nova/cinder changes

Dependencies

None

Testing

There is limited https testing in the gate by default. Some manual functional testing will be done, and devstack will be spun up with https enabled.

Documentation Impact

The cli help will be updated. Any relevant .rst docs will be updated also.

References

Previous effort:

https://review.openstack.org/#/c/23424

Reuse the deleted image-member before create a new one

Thu, 25 May 2017 00:00:00

https://blueprints.launchpad.net/glance/+spec/reuse-the-deleted-image-member

Check the deleted image-member before create a new one, then update it if it exists, otherwise, create a new one.

Problem description

If glance backend database is not MySQL or PostgreSQL,the unique constraint of image-member only includes image-id and member. In this case, if an image-member is deleted, then create it again with the same parameters, glance initiates a query to see if there is already an existing one, but the result does not include the record which was marked as deleted, glance will try to create a new one with the same parameters, then it will fail with duplicate error.

Proposed change

Use only one record to maintain the member-ship between a pair of image and tenant. When create a new image-member, at first check all existing image-member records including the deleted image-member, then update it if it exists, otherwise, create a new one.

Alternatives

Unify the unique constraint for image-member like we did for MySQL and PostgreSQL in 022_image_member_index.py, it has migrated unique constraint of image-member for MySQL and PostgreSQL, now its unique constraint includes image-id, member and deleted_at. Currently the column “deleted_at” is nullable. For other databases like DB2, its unique constraint is more restricted than MySQL. The columns under unique constrains should be “NOT NULL”, otherwise, an error occurs. Thus, we can not create the same unique constraint for this kinds of database.

We would alter “deleted_at” column to “not nullable” in migration. That means we have to insert a default timestamp value for the new created image-member, an active member with a no-blank timestamp for “deleted_at” would confuse user.

Another solution is migrating the unique constraint from (image-id, member, deleted_at) to (image-id, member,created_at) for MySQL and PostgreSQL, from (image-id, member) to (image-id, member,created_at) for other databases. created_at is not nullable, so the new constraint will be applicable to all databases. This solution needs data migration for different database.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

When create a new image-member, it will search all image-member records including the deleted items. But if we enable this patch, it will very rare to see many deleted image-member, as the membership will be kept by only one record. The only case is, multiple deleted image-members were created before applying this patch, we could make a migration script in another patch to remove them.

Other deployer impact

Configuration options will change:

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Long Quan Sha

Reviewers

Core reviewer(s):: nikhil-komawar Ian Cordasco
Other reviewer(s):: Kamil Rykowski Abhishek Kekane

Work Items

Reuse the deleted image-member when create a new one

Dependencies

None

Testing

Verify the deleted image-member is updated when create a new one

Documentation Impact

None

References

Proposed patch:

https://review.openstack.org/190895

unique constraint for PostgreSQL, the same as MySQL:

http://www.postgresql.org/docs/8.1/static/ddl-constraints.html

unique constraint for DB2:

https://www-01.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.admin.dbobj.doc/doc/c0020151.html

Scrub Images in Parallel

Thu, 25 May 2017 00:00:00

https://blueprints.launchpad.net/glance/+spec/scrub-images-in-parallel

This change proposal introduces a way for the scrubber to scrub images in parallel when delayed delete is enabled.

Problem description

As of today, when delayed delete is enabled, images are being scrubbed serially while the image locations, if multiple, are being scrubbed in parallel. For the general case, this may not achieve much performance gain as the number of images is likely to be more than the number of image locations per image. Consequently, the scrubber can fall behind when the number of pending_delete images increase.

Proposed change

This change will attempt to parallelize image scrubbing while leaving image locations to be scrubbed serially.

A new config option would be introduced to offer the flexibility to choose between serial or parallel scrubbing. Also, using this config option, one can regulate the degree of parallelism to a desired level.

Alternatives

One can run multiple scrubbers to keep up with the increase in pending_delete images. However, in that case, scrubbers may race as they all get the same set of images from the registry. This may not be a bad option if one can ignore all the errors in the logs. Nevertheless, scrubbing images serially is still inefficient.

Data model impact

None

REST API impact

None

Security impact

As mentioned in the performance impact section, while setting the degree of parallelism one should take into account any rate-limits enforced by the backend store. If the degree of parallelism is set to go beyond the rate-limits, an attacker may be able to force the scrubber hit the rate-limits by creating and deleting several images in a short span of time.

Notifications impact

None

Other end user impact

None

Performance Impact

When using parallel scrubbing one must take into account any rate-limits on the backend store. Depending on extent of parallelism desired, scrubber may hit the rate-limits of the backend store and may eventually slowdown or fail.

Other deployer impact

Deployers would have to set the new configuration option introduced with this change proposal to be able to scrub images in parallel.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: hemanth-makkapati
Other contributors:: jesse-j-cook

Reviewers

Core reviewer(s):: nikhil-komawar
Other reviewer(s):: flaper87 brian-rosmaita

Work Items

Use eventlet to parallelize image scrubbing
Monkey-patch required modules for eventlet
Test on devstack

Dependencies

None

Testing

Documentation Impact

The new configuration option would require documentation.

References

None

Add created and updated time filtering to v2 API

Thu, 25 May 2017 00:00:00

https://blueprints.launchpad.net/glance/+spec/v2-additional-filtering

This spec introduces a feature to the Glance v2 API for filtering image lists based on when they were created or last updated using a comparative operators.

The introduction of this feature is important because it provides a migration path for consumers of the Glance v1 API who depend on changes-since filtering to begin consuming the v2 API instead.

Problem description

From the v2 API list images spec protected properties such as created_at and updated_at are not identified as available for comparison operator filtering the way size_min and size_max effectively are. [1]

The created_at and updated_at fields are not indexed resulting in full-table scans in the database when the default sort [2] on image lists is used or the changes-since filter from v1 is used.

Further, v2 does not support the changes-since filter which was available in v1 [3]. This creates a situation where functionality available in the v1 API is not available in the v2 API and this limits operators and users ability to usefully filter images.

To promote the adoption of the Glance v2 API providing feature parity would be helpful. As a complete replacement for the v1 API becomes available so too does the option to begin deprecation of the v1 API. To advance both of these ends, feature parity is important.

Proposed change

Two new filters are proposed to be added for the image list endpoint of the v2 API: the created_at and updated_at times for images.

With the proposed feature we enable consumers of the v2 API to quickly identify old instance snapshots for clean-up and other potential use cases.

Alternatives

Something must be done to address the missing changes-since filter to address priorities of both the Nova and Glance projects as expressed through the Liberty summit.

We could implement changes-since as it exists in v1 API instead of making the functionality more rich by allowing many comparative operators.

We could exclude the created_at filter from the feature at this time, but the additional effort to include it is minimal, and it is possible that other potential use cases may benefit as well as taking the opportunity to raise an index on the column to benefit the default sort.

Data model impact

Modify the Image domain class to raise indexes for the created_at and updated_at columns on the images DB table.

The additional indexes raised will impose a longer upgrade window for larger Glance installs while the indexes are built.

REST API impact

Following the pattern of existing filters, the new filters may be specified as query parameters, using the field to filter as the key and the filter criteria as the value in the parameter.

Changes apply exclusively to Image API v2 Image entity listings:: GET /v2/images/{image_id}

Filter by adding optional query parameters:

created_at     = Specifies to limit returned images based on when the image
                 was created, with value expressed as an ISO 8601 datetime.
updated_at     = Specifies to limit returned images based on when the image
                 was last updated, with value expressed as an ISO 8601
                 datetime.

These filters will be added using syntax that conforms to the latest guidelines from the OpenStack API Working Group, and any applicable draft guidelines [4]. This implies support for optional comparison operators with the implied default comparison operator being ‘equals’ for exact matches.

An example of an acceptable criteria using an optional comparison operator:

gte:1985-04-12T23:20:50.52Z

This represents any time at or after 20 minutes and 50.52 seconds after the 23rd hour of April 12th, 1985 in UTC.

Note

In all cases, literal expressions in any of these filters will be treated as case-insensitive.

Both new filters will support access control through oslo.policy enforcement to allow a deployer to restrict usage of these filters. Because these filters support this access control, no guarantee is given for availability of these filters for all API consumers. Performance or security concerns around these filters in any given deployment can be addressed through the use of custom policy rules. Standard response codes will apply when access to the filter is forbidden.

Security impact

None

Notifications impact

None

Other end user impact

Update python-glanceclient as needed.

Performance Impact

The appropriate indexes will also be updated on each row create and/or update to the associated image row. This is considered a minimal impact.

Performance for sort operations on the indexed columns would be improved.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: steve-lewis

Reviewers

Core reviewer(s):: flwang flaper87
Other reviewer(s):: None

Work Items

Registry changes to support new filters
API changes to expose new filters
Add Policy enforcement hooks

Dependencies

None

Testing

Unit and functional tests will be added as appropriate.

Documentation Impact

Docs needed for new API filters and usage as well as the additional policy options.

References

Nova blueprint to use Images V2 API https://blueprints.launchpad.net/nova/+spec/use-glance-v2-api
Nova change request supporting the blueprint https://review.openstack.org/#/c/144875/

Use keystoneclient Sessions

Thu, 25 May 2017 00:00:00

https://blueprints.launchpad.net/python-glanceclient/+spec/session-objects

Keystoneclient sessions are being used as a common base library to standardize handling of tokens and authentication credentials going beyond a basic username and password.

This improves security as a single point of update for these issues and means that additional authentication mechanisms may be added transparently to individual clients.

Glanceclient should adopt sessions as other clients have.

Problem description

The OpenStack clients all grew rather organically. Each handled their own authentication and this was copied and pasted between every service. This results in:

inconsistencies with what parameters are accepted between clients.

A security issue when bugs are found and need to be patched in each client.

Inconsistent use of the service catalog, regions etc.

Problems adding new features as they become available in keystone.

Glanceclient is structured differently from the other clients. It is one of the few that nicely separates the responsibilities of the CLI and the library. However it does do custom HTTPS handling and other security related fixes that are not reused amongst other components.

Because of offloading to the CLI it does not handle things like token refreshing, and puts the onus of operating the service catalog onto the user - generally other services.

Proposed change

I want to bring glanceclient more inline with other clients. Now that there is the facilities that things like the service catalog can be managed consistently by other libraries there is less of an excuse for glanceclient to avoid using this information.

This will involve creating a different type of HTTPClient in the event that session and other options are detected. This will allow glanceclient to continue to operate as it does today unless users opt-in by using new parameters. These new parameters are managed by the keystoneclient Adapter object and so will simply involve passing kwargs through.

The CRUD layers of glanceclient and the requests path will be unaffected.

Alternatives

Glanceclient can continue to operate as it does now. This is not as bad as other clients as it pushes this configuration back on the service, rather than incorrectly handling options internally. However this makes it a special case and glanceclient will not benefit from the efforts to standardize authentication flows and options that the other services gain.

Data model impact

None

REST API impact

None

Security impact

This change will reimplement how glanceclient handles authentication, token management and how you select endpoints. It handles this though by offloading these concepts to keystoneclient so that any security issues can be handled there.

It will deprecate the custom HTTPs handling that glanceclient does. It is my understanding that this custom handling was largely to disable BEAST style attacks by preventing the client from advertising SSL compression capabilities. Whilst compatibility will be maintained for the current client it may not be possible to maintain this functionality when using a common handling logic.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

The flow on effect here is that other services that have credentials and parameters in config files that are used to talk to glance will be standardized with other services.

Developer impact

This will change the parameters that are provided when establishing a glanceclient. The new parameters are standardized across all the other OpenStack client libraries. Existing parameters will obviously be maintained and deprecated in time.

This will greatly assist integration with service to service communication as auth_token middleware is providing an auth plugin that can be used directly, as well as standardizing the options that are used for all the clients.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: jamielennox

Other contributors:

Reviewers

I would appreciate anyone who wants to be a point of contact for review.

Core reviewer(s):: None at this time

Work Items

There are some initial changes to testing that are required to fit the new model. These are considered generally useful and not necessarily specific to this review.
Add session handling and handling of existing parameters to the glanceclient.
Convert the glanceclient CLI to use the standard parameters and option handling. In other projects I have done this for we have not always completed this step. Most clients are moving towards the OpenStackClient project for CLI and are not worried about significant refactoring of their CLI applications.

Dependencies

None

Testing

We can unit test these changes. It should also be possible to use a testscenarios approach such that existing CRUD tests are run with both a traditionally created client and a client created with a session.

Documentation Impact

Update documentation on how to instantiate a Client.

References

None

Lite Spec: Remove Glare code from the Glance repository

Thu, 25 May 2017 00:00:00

problem:

Glare became a separate project with its own code repository during Newton. The code was copied out of the Glance tree, but remained in the Glance repository. It is no longer being maintained within Glance, and that has begun to cause some problems, for example, blocking a recent stevedore upper constraints change; see Change-Id: I141b17f9dd2acebe2b23f8fc93206e23bc70b568.

solution:

Remove all Glare code from the Glance repository and drop all artifacts tables from the Glance database.

impacts:

No API Impact as the Glare API was EXPERIMENTAL in both versions that ran on the code being removed (‘/v3’ on the Glance endpoint in Liberty, ‘/v0.1’ on its own endpoint in Mitaka).

As a courtesy to projects/packagers/deployers that may have consumed Glare from the Glance code repository, an openstack-dev announcement and an openstack-operators announcement were sent out on 16 February 2017. There has been no response so far.

A detailed release note will be included in the patch.

timeline:

Pike-1

link:

Change-Id: I3026ca6287a65ab5287bf3843f2a9d756ce15139

assignee:

rosmaita

Glance Spec Lite

Fri, 03 Feb 2017 00:00:00

Replace mox with mock for unit test

problem:: Mox does not support python 3. We have a shim module ‘mox3’ that was built a couple of years ago, is unmaintained, and as it gets tested more heavily is showing race conditions under python3.
solution:: Replace mox with mock.
impacts:: This change will use mock instead of mox.
timeline:: Expected to be merged within the Ocata time frame.
link:: https://review.openstack.org/#/c/407959/
assignee:: Howard Lee

End of Replace mox with mock for unit test

Glance Expand/Contract migrations with Alembic

Fri, 03 Feb 2017 00:00:00

https://blueprints.launchpad.net/glance/+spec/alembic-migrations

This spec outlines the motivation and implementation details for porting Glance database migrations from SQLAlchemy-based scripts to Alembic.

Problem description

Existing database migrations are implemented in Glance using SQLAlchemy- migrate scripts. This approach makes designing migrations for rolling upgrades ([1] & [2]) extremely uncomfortable. The expand and contract migrations would have to be created in strict sequential order with all of the latter following all of the former. This places unnecessary restrictions on the order of migrations and becomes very error prone as multiple schema changes are introduced in a single cycle. Essentially, using SQLAlchemy-migrate constrains expand and contract migrations to be run atomically and does not easily facilitate separating them into phases as required during rolling upgrades.

Proposed change

We are proposing to port the migration flows to be handled by Alembic [3]. There are numerous benefits to this change. Alembic allows labeling the various migration scripts with designated branch names (e.g. expand and contract), which makes it easier to group the two types of changes and run them separately.

Another benefit is that Alembic migrations are chained in the manner of linked list. This is accomplished by explicitly specifying revision dependencies in each migration script. This will allow developers to get away from dependence on numeric ordering of migration scripts and obviate the need for placeholder migrations for retroactive backports, of the like: 023_placeholder. To facilitate this feature, a new naming convention is proposed for migration scripts:

<cycle>_<branch-label><##>_<short-desc>.py

e.g. ocata_expand01_add_visibility_column.py

Additionally, Alembic supports automatic generation of migration scripts using SQLAlchemy’s ORM. In the future, developers are encouraged to utilize this feature to minimize their interaction with the underlying DBMS and rely on the Python code to define the new schema.

Alternatives

The best aleternative available is to continue using SQLAlchemy-migrate scripts. The downsides of this appropach are outlined in the Problem description section above.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Performance of existing offline migration flows is not expected to be impacted in a signifficant way.

Other deployer impact

The API of the glance-manage command-line utility remains backwards compatible. However, the operators will notice that the db_version command returns a named head revision rather than a number. This change will be documented. Additionally, the operators will need to be made aware of the new db_expand, db_contract, and db_data_migrate APIs in the glance-manage command-line utility. Refer to the Documentation Impact section below.

Developer impact

Going forward, any time a database schema change is implemented, developers will need to create Alembic migration scripts conforming to the established naming convention and making sure to link each new revision to the appropriate down revision.

Implementation

A new alembic_migrations folder is added under glance/db/sqlalchemy This folder will contain all the necessary code to implement Alembic based migrations, including:

Alembic configuration and environment files. For details see: Migration Environment
helper scripts for creating initial Glance DB schema
versions directory, containing migration scripts added in different release cycles

Assignee(s)

Primary assignee:: hemanthm abashmak

Work Items

Implement Alembic migration infrastructure and scripts for existing database migrations (up to Newton).
Introduce expand/contract migrations for new Ocata features [4].
Update the glance-manage utility to utilize Alembic for db_sync and related commands. Add db_expand and db_contract commands to prepare for rolling upgrades.

Dependencies

A new Python module will be added to Glance’s requirements list:
- alembic - version 0.8.7 or hihger

Testing

Existing and new database migration tests (unit and functional) will be ported and/or added to exercise the updated glance-manage utility and test the integrity of Alembic-based migration scripts.

Documentation Impact

The new database version names resulting from the move to Alembic will need to be documented to make developers aware of the change to db_version output.

Also, as part of the drive to enable the assert:supports-zero-downtime -upgrade tag [5], this change will introduce and implement new API methods for the glance-manage utility: db_expand, db_contract, and db_data_migrate. These will need to be thoroughly documented and their usage explained in the glance-manage reference as well as the upgrade guide. It is advised to wait until Glance is able to assert the zero-downtime upgrade tag to add documentation for the new commands.

References

Add community-level image sharing

Fri, 03 Feb 2017 00:00:00

This spec is mostly based on an earlier blueprint created by Iccha Sethi, and later picked up by Louis Taylor. We at Symantec have already implemented a similar function within our local environment. We want to re-target this feature to Newton release.

The blueprint may be found at: https://blueprints.launchpad.net/glance/+spec/community-level-v2-image-sharing

This feature will allow image owners to share images across multiple tenants/projects without explicitly creating members individually through the glance API V2. “Community images” will not appear in users’ default image listings.

This new feature adds a new value for the visibility of an image to the underlying data model, named community.

Problem description

Currently, there is no provision for an image to be available for use by users other than the owner of the image, unless the image is either made public or explicitly shared by the owner of the image with other users. If the number of users who require access to the image is large, the overhead of explicit sharing can become a burden on the owner of the image. Also, if such images are not needed by the majority of users, then the image listing results will be spammed if they are all made public.

Public images appear in image-list for all users. This can be undesirable because:

An open source project wants to make a prepared VM image available so users simply boot an instance from that image to use the project’s software. The developers of the project are too busy writing software to worry about the hassle of maintaining a list of “customers” as they’d have to do with current v2 image sharing. At the same time, the cloud provider hosting the prepared image doesn’t want to make this image public, as that would imply a support relationship that doesn’t exist between image consumers and the cloud provider. Moreover, if too many such images get published, then image listing will be spammed as this type of image should not be seen by most of the users.
A cloud provider wants to discourage users from using a public image that is getting old. For example, an image which is missing some security patches or is no longer supported by the vendor. The cloud provider doesn’t want to delete the image because some users still require it. Users could need the old image to rebuild a server, or because they have custom patches for that particular image.

In both cases, the vendor can make the image a “community” image and alleviate the challenges presented above. This means that:

a) The community image won’t appear in users’ default image lists. This means they won’t know about it unless they are motivated to seek it out, for example by asking other users for the UUID.

b) Since the image is no longer “public,” it doesn’t imply the same level of support from the vendor as a provider-supplied public image.

Proposed change

An additional value for the visibility enum will be added in the JSON schema, named 'community'. This makes the possible values of visibility:

['public', 'private', 'shared', 'community']

Images with these visibility values have the following properties:

public:
- Who: all users:
  - have this image in the default image-list
  - can see image-detail for this image
  - can boot from this image
private:
- Who: users with tenant_id == owner_tenant_id only:
  - have this image in the default image-list
  - see image-detail for this image
  - can boot from this image
shared:
- Who: users with tenant_id == owner_tenant_id:
  - have this image in the default image-list
  - can see image-detail for this image
  - can boot from this image
  - can see full member list for this image
- Who: users with tenant_id in the member-list with member_status == 'accepted' of the image:
  - have this image in the default image-list
  - can see image-detail for this image
  - can boot from this image
  - can see a member list containing only themselves for this image
- Who: users with tenantId in the member-list with member_status == 'pending' or member_status == 'rejected':
  - do not have this image in their default image-list
  - can see image-detail for this image
  - can boot from this image
  - can see a member list containing only themselves for this image
community:
- Who: users with tenant_id == owner_tenant_id:
  - have this image in the default image-list
  - can see image-detail for this image
  - can boot from this image
- Who: all users:
  - do not have this image in their default image-list
  - can see image-detail for this image
  - can boot from this image

Membership Behaviour and Transitions

After much discussion, it has been decided that this topic is complex enough to break out into its own separate proposal. Between the significant number of state transitions, and the problem use cases brought up by several people, fully designing what the “correct” behaviour shall be is outside the scope of this spec.

The behaviour for the initial implementation for community images shall respect the current paradigm in glance - where changing visiblities does not affect an image’s internal member-list. To be precise, consistent with current behavior:

An image’s member-list persists even when the visibility of an image has changed to a value that renders the member-list inert.
Image member operations (that is, creating image members or changing the member_status of image members) are allowed only when an image has a visibility under which the member-list is not inert. The justification for this is that it is confusing to users to be able to perform member operations on an image when these operations have no effect on the accessibility of the image.

Alternatives

Adding image aliases

A completely different way of solving the usecase for cloud providers (discouraging users from using an older version of a public image) could be to create a mechanism to make an image alias, which could point at the newest version of the public image. There is an abandoned blueprint for this feature [1]. This, however, is much harder to implement and does not fit with the other use cases.

Data model impact

Schema changes

The visibility of the image will be stored in the database within the images table inside a new column named visibility. The visibility will be in the set of ['public', 'private', 'shared', 'community'].

The default value for visibility is shared. (See Other end user impact for a discussion.)

This change makes the is_public column redundant. If no v1 code actually uses is_public, the column will be removed.

Appropriate indexes will be added to facilitate quick responses.

Database migrations

(Note: this is a statement of expected outcomes, not an algorithm for performing the migration.)

All rows with is_public == 1:
- visibility = public
For all unique image_id in image_members where deleted != 1 and for which is_public == 0 for that image:
- visibility = shared
For all other images:
- visibility = private
(See Other end user impact for further discussion.)

REST API impact

The changes described in this document will require an API version bump.

Image discovery

If you want to list all community images, and only community images, then you would use:

GET /v2/images?visibility=community

All other appropriate filters will be respected. Of note is the use of an owner parameter. This, when supplied together with the visibility=community filter, allows a user to request only those community images owned by that particular tenant:

GET /v2/images?visibility=community&owner={owner_tenant_id}

Note that visiblity will be considered a core property of the image object, and as such included within image lists generated via the v2 interface.

Making an image a “community image”

As permitted by the new policy.json rule communitize_images, an admin or owner would use the existing image-update call to change an image’s visiblity to 'community':

PATCH /v2/images/{image_id}

Request body:

[{ "op": "replace", "path": "/visibility", "value": "community" }]

The response and other behaviour remains the same as was previously defined for this call.

Removing community level access from an image

An admin or owner of an image can remove community-level access from an image by using the image-update call. For example, instead of setting it to 'community' as before, we set it to 'private':

PATCH /v2/images/{image_id}

Request body:

[{ "op": "replace", "path": "/visibility", "value": "private" }]

An admin or user with permission to publicize an image could replace community visibility with public.

As in the above case, the response and other behaviour remains the same as was previously defined for this call.

Security impact

See “other deployer impact”.

Notifications impact

Current notifications contain the is_public attribute, which is true if and only if the image is a public image. This must be maintained for backwards compatibility.

An additional attribute of visibility will be added for each image to indicate its visiblity, with possible values of ['public', 'private', 'shared', 'community'].

Other end user impact

Consistent with current functionality, when a visibility is specified at the time of image creation, that visibility will be respected. If the user creating the image does not have appropriate permissions to set the specified visibility value on an image, the image-create call will fail as it does now.

Default visibility

In order to preserve backward compatibility with the current workflow for sharing an image, the default visibility of an image will be shared:

An image with shared visibility and no image members is not accessible to anyone other than the image owner, so this respects the current behavior.
Currently, when an image is created with default visibility, the owner can immediately add members to the image to share it. By making the default visibility value shared, we preserve this behavior. If the default visibility were private, then this behavior would be broken as the image owner would have to make an API call to change the visibility of the image to shared before members could be added.
Although the Image API v1 is deprecated in Newton, it still exists and will likely be deployed in clouds using the Ocata release of OpenStack. Since the v1 API has no concept of visibility, there’s no way for a user to change the visibility to shared so that an image can have members added to it. By making the default visibility shared, we achieve the following desirable results:
- A v1 user can create an image and add members using the current workflow, that is, sharing an image is simply a matter of putting a member on it.
- In deployments using both v1 and v2 of the Image API, if an image is created using v1 and then an image-show call is made in v2, the visibility of shared makes it clear to the v2 user that the image is in a state where it can both accept members and be accessed by any existing members.

If the default visibility value were not shared, then v1 would need to be modified so that private images could accept members and be accessed by members. This, however, would create a situation where an image would behave radically differently depending upon whether you tried to access it by the v1 or the v2 API. We don’t want to do that.

A key question here is whether changing the default visibility from private to shared introduces a backward incompatibility into the v2 API. Here are three arguments that it does not:

The current situation is that the visibility of an image created by a non-admin user has visibility != public. That situation is preserved.
If we don’t make the default visibility shared, then we are in fact introducing a backward incompatibility in that any current user scripts that create an image and then immediately add members to it will break. (This was in fact a problem with an earlier version of this spec. While the API working group agreed that it was a “minor” backward incompatibility that would be permissible, they pointed out that Glance users were likely to be annoyed by the workflow change. The change in visibility value is a lesser of two evils.)
What we want to be the case is that when a user creates an image without specifying a visibility, the image behaves the same way it does now. Images that behave in that way have the visibility that we now call shared. This is to be expected; previously, we had insufficient keywords to identify the various accessibility statuses of images; now we do, and we should use these keywords appropriately.

The conclusion is that setting the default visibility to private would have a greater end user impact than the proposal to make the default visibility value shared.

Note that if the v2 API is used to set the visibility of an image to private, if the v1 API is used to share the image, the request will fail with a 409 (as it will if a member-related call is made to the v2 API). This is appropriate, as someone has explicitly disabled sharing for that image.

One final point implicit in the foregoing is worth stating explicitly. If the v1 API is used to update the is_public property of an image, it will have the following effect upon the image’s visibility:

If the call updates is_public to True, then the corresponding v2 visibility of the image will be public.
If the call sets is_public to False, then the corresponding v2 visibility of the image will be set to the default value, that is, shared.

This will preserve backward-compatible behavior in the v1 API. The arguments advanced previously in favor of shared being the default visibility apply to this case as well.

Migration of the visibility of existing images

The issue of how the database migration should work was sufficiently controversial that an operators’ survey was taken to gather data [SUR1]. The results of the survey were reported in [SUR2] (along with a recommendation for the migration path that, as you’ll see if you read on, was ultimately rejected). Briefly, operators were evenly split on whether images with visibility private should all be migrated to shared or whether only those images with members should be migrated to shared.

We’ve had a lot of discussion on this (see the comments on [GER1] for a sample). In the end, the compromise we reached is reflected in this spec. The primary justification for this migration path is that it’s worth trading some minor incompatibilities between pre-Ocata and Ocata image sharing in order to avoid end user confusion when they first use Glance after a deployment has been upgraded to Ocata and suddenly see that all the images they own now have a visibility of ‘shared’.

Thus, the end user impact of an upgrade to the Glance Ocata release will be the following:

Any existing “private” image with a non-empty member-list will be found to have visibility ‘shared’.
Any existing “private” image that has no image members will be found to have visibility ‘private’.
- Incompatibility: v2: Unlike pre-Ocata, the visibility of the image must be changed to ‘shared’ before the member-create call will succeed. v1: Users must explicitly set is_public to False before the member-create call will succeed.
Any images created after the upgrade for which a visibility is not specified at the time of image creation will have visibility ‘shared’.
- Inconsistency: The member-create call will immediately work for newly created images with default visibility, whereas migrated images will have to be dealt with as described above.
Pre-Ocata, an end user had “private” images that might actually be shared. The user had to do an API call to see the member-list to determine whether other users could access that image. In Ocata, an end user will have images with visibility shared that may not actually be accessible to other users. The user must do an API call to see whether the image has a non-empty member list.
- Incompatibility: The workflow required to determine whether an image is accessible to other users is roughly the same, but it now depends upon two aspects: (1) the visibility must be ‘shared’, and (2) the image’s member-list must be non-empty.
  
  The upside to this incompatibility is that in the Glance Ocata release, a visibility value of private will actually mean “private” instead of “private or shared”.

[SUR1]

http://lists.openstack.org/pipermail/openstack-operators/2016-November/012107.html

[SUR2]

http://lists.openstack.org/pipermail/openstack-operators/2016-December/012235.html

[GER1]

https://review.openstack.org/#/c/396919/

Client changes

OpenStackClient, as well as the library portion of python-glancelient, will be updated to expose this feature. The CLI glanceclient will not be supported.

Users will be able to see all community images by using openstack image list --community.

An option to openstack image set will be added named --visibility <VISIBILITY_STATUS>, where VISIBILTY_STATUS may be one of {public, private, shared, community}.

For example, to make an image a community image:

$ openstack image set --visibility community <IMAGE>

To make the image private again:

$ openstack image set --visibility private <IMAGE>

Performance Impact

None

Other deployer impact

The ability to create community images is moderated using policy.json. As mentioned above, a new rule will be created called communitize_image, which will have the default configuration of [role:admin or rule:owner].

Also users from Horizon will be able to see community images through a separate tab. Details regarding the Horizon feature can be found at: https://blueprints.launchpad.net/horizon/+spec/glance-community-images

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: timothy-symanczyk

Reviewers

Core reviewer(s):: brian-rosmaita nikhil-komawar

Work Items

Refactor db API to use visibility rather than is_public
Add functionality for storing the community state in the interfaces to both db backends:
- sqlalchemy
- simple
Add functionality to enable this and accept the image using the API
Add unit tests to test various inputs to the API
Add functional tests for the lifecycle of community images
Update OpenStackClient to use the new API functionality
Bump the API version

Dependencies

None

Testing

A tempest test must be added to cover creating a community image and it transitioning between public and private states.

Documentation Impact

New features must be documented in both glance and OpenStackClient.

References

Glance Spec Lite

Fri, 03 Feb 2017 00:00:00

Return 409 if removing/replacing the location of an image that’s not `active`

problem:: When show_multiple_locations is set to True, users currently can remove or replace locations of an image irrespective of the image status. This can result in bad experiences for the users:- 1) If one tries to change or remove the location for an image while it is in saving state, Glance would be trying to write data to a previously saved location while the user updates the custom location. This results in a race condition. 2) For images that are in queued state and no image data has been uploaded yet, there is no need for an image location to be removed and permitting users to remove the image location can result in a bad experience. However users can be allowed to replace the image location to maintain backward compatibility and also because replacing could mean replacing an empty location by a non-empty image location. 3) For images in deactivated state, it is essential that image locations are not updated as it does not abide with the purpose of the image state being set to deactivated and may cause security concerns.
solution:: 1) Return Conflict Error (409 response code) if an attempt to remove the image location is made when the status of the image is anything but active. 2) Return Conflict Error (409 response code) if an attempt to replace the image location is made when the status of the image is anything but active or queued.
impacts:: A Conflict Error will be thrown preventing users from removing an image location when the image status is not active and replacing an image location when the image status is not active or queued.
timeline:: Expected to be merged within the ocata-1 time frame.
link:: https://review.openstack.org/#/c/366995/
assignee:: Nikhil Komawar, Dharini Chandrasekar

End of Return 409 if removing/replacing the location of an image that’s not `active`

Expand `hypervisor_type` metadata with Virtuozzo hypervisor

problem:: Currently it is not possible to require Virtuozzo hypervisor by specifying hypervisor_type metadata, though Nova has had Virtuozzo support since the Kilo release.
solution:: We need to expand etc/metadefs/compute-hypervisor.json hypervisor_type property with the appropriate identifier, ‘vz’, as defined in http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/hv_type.py.
impacts:: This will have documentation impact. Release note should be added to notify interested parties about this addition.
timeline:: Expected to be merged within the O-2 time frame.
link:: https://review.openstack.org/#/c/341623/
assignee:: Maxim Netratov

End of Expand `hypervisor_type` metadata with Virtuozzo hypervisor

Add `ploop` to the list of supported disk formats

problem:: Currently ‘ploop’ format is not among supported by default disk formats, even though it’s been supported by Nova since the Kilo release.
solution:: disk_formats configuration option needs to be updated to add ‘ploop’ as one of the supported format.
impacts:: This will have documentation impact. Release note should be added to notify interested parties about this addition.
timeline:: Expected to be merged within the O-2 time frame.
link:: https://review.openstack.org/#/c/341633/
assignee:: Maxim Netratov

End of Add `ploop` to list of supported disk formats

Use `Range` HTTP header instead of `Content-Range` for parsing requests

problem:: When a HTTP request for a partial image download is sent, currently the Content-Range header is parsed to get the byte range from the request. Per RFC 7233 specification, the desired byte range should be specified in HTTP requests using the Range header rather than the Content-Range header. The latter is reserved for responses to such requests. Current implementation requires users to send requests that are not compatible with this specification. For example, a user has to give “bytes 12-30/32” instead of “bytes=12-32”.
solution:: Parse the Range header from HTTP requests and send a Content-Range entity header with the server response. Deprecate and retain the current support for Content-Range header requests for backward compatibility reasons.
impacts:: Users will be able to send partial download requests using the Range header in the requests with the appropriate value formats. For developers, the Range webob parser does not have a length attribute. We will have to pass the image size explicitly and perform checks to identify an unsatisfiable byte range request from the parsed Range header. This change will also require an API version bump.
timeline:: Expected to be merged within the Ocata time frame.
link:: https://review.openstack.org/#/c/367528/
assignee:: Dharini Chandrasekar

End of Use `Range` HTTP header instead of `Content-Range` for parsing requests

Image Metadata API calls

Sun, 29 Jan 2017 00:00:00

The following calls allow you to create, modify, and delete image metadata records.

Create image

POST /v2/images

Request body must be JSON-encoded and conform to the image JSON schema. For example:

{
    "id": "e7db3b45-8db7-47ad-8109-3fb55c2c24fd",
    "name": "Ubuntu 12.10",
    "tags": ["ubuntu", "quantal"]
}

Successful HTTP response is 201 Created with a Location header containing the newly-created URI for the image. The response body shows the created image entity. For example:

{
    "id": "e7db3b45-8db7-47ad-8109-3fb55c2c24fd",
    "name": "Ubuntu 12.10",
    "status": "queued",
    "visibility": "public",
    "tags": ["ubuntu", "quantal"],
    "created_at": "2012-08-11T17:15:52Z",
    "updated_at": "2012-08-11T17:15:52Z",
    "self": "/v2/images/e7db3b45-8db7-47ad-8109-3fb55c2c24fd",
    "file": "/v2/images/e7db3b45-8db7-47ad-8109-3fb55c2c24fd/file",
    "schema": "/v2/schemas/image"
}

Update an image

PATCH /v2/images/{image_id}

Request body must conform to the ‘application/openstack-images-v2.1-json-patch’ media type, documented in Appendix B. Using PATCH /v2/images/e7db3b45-8db7-47ad-8109-3fb55c2c24fd as an example:

[
    {"op": "replace", "path": "/name", "value": "Fedora 17"},
    {"op": "replace", "path": "/tags", "value": ["fedora", "beefy"]}
]

The response body shows the updated image entity. For example:

{
    "id": "e7db3b45-8db7-47ad-8109-3fb55c2c24fd",
    "name": "Fedora 17",
    "status": "queued",
    "visibility": "public",
    "tags": ["fedora", "beefy"],
    "created_at": "2012-08-11T17:15:52Z",
    "updated_at": "2012-08-11T17:15:52Z",
    "self": "/v2/images/e7db3b45-8db7-47ad-8109-3fb55c2c24fd",
    "file": "/v2/images/e7db3b45-8db7-47ad-8109-3fb55c2c24fd/file",
    "schema": "/v2/schemas/image"
}

The PATCH method can also be used to add or remove image properties. To add a custom user-defined property such as “login-user” to an image, use the following example request.

[
    {"op": "add", "path": "/login-user", "value": "kvothe"}
]

Similarly, to remove a property such as “login-user” from an image, use the following example request.

[
    {"op": "remove", "path": "/login-user"}
]

See Appendix B for more details about the ‘application/openstack-images-v2.1-json-patch’ media type.

Property protections

Version 2.2 of the Images API acknowledges the ability of a cloud provider to employ property protections. Thus, there may be image properties that may not be updated or deleted by non-admin users.

Add an image tag

PUT /v2/images/{image_id}/tags/{tag}

The tag you want to add should be encoded into the request URI. For example, to tag image e7db3b45-8db7-47ad-8109-3fb55c2c24fd with ‘miracle’, you would PUT /v2/images/e7db3b45-8db7-47ad-8109-3fb55c2c24fd/tags/miracle. The request body is ignored.

An image tag can be up to 255 characters in length. See the ‘image’ json-schema to determine which characters are allowed.

An image can only be tagged once with a specific string. Multiple attempts to tag an image with the same string will result in a single instance of that string being added to the image’s tags list.

An HTTP status of 204 is returned.

Delete an image tag

DELETE /v2/images/{image_id}/tags/{tag}

The tag you want to delete should be encoded into the request URI. For example, to remove the tag ‘miracle’ from image e7db3b45-8db7-47ad-8109-3fb55c2c24fd, you would DELETE /v2/images/e7db3b45-8db7-47ad-8109-3fb55c2c24fd/tags/miracle. The request body is ignored.

An HTTP status of 204 is returned. Subsequent attempts to delete the tag will result in a 404.

List images

GET /v2/images

Request body ignored.

Response body will be a list of images available to the client. For example:

{
    "images": [
        {
            "id": "da3b75d9-3f4a-40e7-8a2c-bfab23927dea",
            "name": "cirros-0.3.0-x86_64-uec-ramdisk",
            "status": "active",
            "visibility": "public",
            "size": 2254249,
            "checksum": "2cec138d7dae2aa59038ef8c9aec2390",
            "tags": ["ping", "pong"],
            "created_at": "2012-08-10T19:23:50Z",
            "updated_at": "2012-08-10T19:23:50Z",
            "self": "/v2/images/da3b75d9-3f4a-40e7-8a2c-bfab23927dea",
            "file": "/v2/images/da3b75d9-3f4a-40e7-8a2c-bfab23927dea/file",
            "schema": "/v2/schemas/image"
        },
        {
            "id": "0d5bcbc7-b066-4217-83f4-7111a60a399a",
            "name": "cirros-0.3.0-x86_64-uec",
            "status": "active",
            "visibility": "public",
            "size": 25165824,
            "checksum": "2f81976cae15c16ef0010c51e3a6c163",
            "tags": [],
            "created_at": "2012-08-10T19:23:50Z",
            "updated_at": "2012-08-10T19:23:50Z",
            "self": "/v2/images/0d5bcbc7-b066-4217-83f4-7111a60a399a",
            "file": "/v2/images/0d5bcbc7-b066-4217-83f4-7111a60a399a/file",
            "schema": "/v2/schemas/image"
        },
        {
            "id": "e6421c88-b1ed-4407-8824-b57298249091",
            "name": "cirros-0.3.0-x86_64-uec-kernel",
            "status": "active",
            "visibility": "public",
            "size": 4731440,
            "checksum": "cfb203e7267a28e435dbcb05af5910a9",
            "tags": [],
            "created_at": "2012-08-10T19:23:49Z",
            "updated_at": "2012-08-10T19:23:49Z",
            "self": "/v2/images/e6421c88-b1ed-4407-8824-b57298249091",
            "file": "/v2/images/e6421c88-b1ed-4407-8824-b57298249091/file",
            "schema": "/v2/schemas/image"
        }
    ],
    "first": "/v2/images?limit=3",
    "next": "/v2/images?limit=3&marker=e6421c88-b1ed-4407-8824-b57298249091",
    "schema": "/v2/schemas/images"
}

Get images schema

GET /v2/schemas/images

Request body ignored.

The response body contains a json-schema document that shows an images entity (a container of image entities). For example:

{
    "name": "images",
    "properties": {
        "images": {
            "items": {
                "type": "array",
                "name": "image",
                "properties": {
                    "id": {"type": "string"},
                    "name": {"type": "string"},
                    "visibility": {"enum": ["public", "private"]},
                    "status": {"type": "string"},
                    "protected": {"type": "boolean"},
                    "tags": {
                        "type": "array",
                        "items": {"type": "string"}
                    },
                    "checksum": {"type": "string"},
                    "size": {"type": "integer"},
                    "created_at": {"type": "string"},
                    "updated_at": {"type": "string"},
                    "file": {"type": "string"},
                    "self": {"type": "string"},
                    "schema": {"type": "string"}
                },
                "additionalProperties": {"type": "string"},
                "links": [
                    {"href": "{self}", "rel": "self"},
                    {"href": "{file}", "rel": "enclosure"},
                    {"href": "{schema}", "rel": "describedby"}
                ]
            }
        },
        "schema": {"type": "string"},
        "next": {"type": "string"},
        "first": {"type": "string"}
    },
    "links": [
        {"href": "{first}", "rel": "first"},
        {"href": "{next}", "rel": "next"},
        {"href": "{schema}", "rel": "describedby"}
    ]
}

Get image schema

GET /v2/schemas/image

Request body ignored.

The response body contains a json-schema document that shows an image. For example:

{
    "name": "image",
    "properties": {
        "id": {"type": "string"},
        "name": {"type": "string"},
        "visibility": {"enum": ["public", "private"]},
        "status": {"type": "string"},
        "protected": {"type": "boolean"},
        "tags": {
            "type": "array",
            "items": {"type": "string"}
        },
        "checksum": {"type": "string"},
        "size": {"type": "integer"},
        "created_at": {"type": "string"},
        "updated_at": {"type": "string"},
        "file": {"type": "string"},
        "self": {"type": "string"},
        "schema": {"type": "string"}
    },
    "additionalProperties": {"type": "string"},
    "links": [
        {"href": "{self}", "rel": "self"},
        {"href": "{file}", "rel": "enclosure"},
        {"href": "{schema}", "rel": "describedby"}
    ]
}

CIM Namespace Metadata

Mon, 24 Oct 2016 00:00:00

https://blueprints.launchpad.net/glance/+spec/cim-namespace-metadata-definitions

The OpenStack Kilo release introduced support for defining metadata definitions and associating the same with different resource types. To make workloads interoperable across OpenStack clouds, it is necessary to adopt a common set of metadata, particularly for hardware architecture, instruction sets, and their extensions. The Common Information Model(CIM)[A] and the Distributed Management Task Force (DMTF)[2] have defined a pretty exhaustive list. We propose including the same into OpenStack.

Problem description

A standard well known set of metadata tags are useful in describing cloud resources such as images, flavors, compute nodes and more in the interest of making orchestration across clouds interoperable. This is yet another step in our cloud federation story.

I have a workload running on OpenStack cloud-A and would like it to run no differently on OpenStack cloud-B. Flavors that specify the properties using a common standard set of descriptors will enable this.
A Network Virtual Function vendor would like to provide an appliance with metadata that different clouds can be expected to provide the same level of performance, to meet its typically stringent latency needs. Several vendors use the Open Virtualization Format (OVF) to package their appliances, which uses the CIM format, a DMTF standard.

We would like to introduce the CIM namespace for metadata in OpenStack to foster cloud interoperability.

Proposed change

The CIM schema files for processor, resources, virtualization, and storage [C - F] shall be parsed and json files added to the Glance etc/metadefs directory. The tool to parse shall also be added in the glance code base to facilitate gathering updates to the schema files in future OpenStack releases should there be any, though this is expected to be few and far between.

The Instruction set extensions shall ignore the enabled/disabled item. The reasoning here is that other than virtualization related instructions, namely VT-d, VT-x, few instruction extensions are disabled after the cloud provider expended money on buying the latest platforms.

In this incarnation of the solution, we shall not introduce semantics such as relationships between the various metadata tags. For instance, while describing hardware, a compute host that belongs to a certain processor architecture can only manifest instructions available in the same.

In this release we shall ignore the version of the CIM schema files. The expectation is that they will add elements, not remove elements, nor change the type of element values. We already have support in Glance to upload, overwrite, and merge metadata definitions [K]. This shall come in handy when a new CIM version of tags is introduced. We could add an API call that indicates the CIM version number supported for each of the CIM metadata definition files.

Alternatives

Manually create in each cloud the CIM namespace metadata using the metadata add API, a tedious error prone task that must be replicated in each and every OpenStack cloud instance.

Data model impact

The introduction of additional json files in the Glance etc/metadefs directories generated from parsing the CIM schema files for [C], [D], [E], and [F].

Security impact

None

Notifications impact

None

Other end user impact

Ease of use. The Horizon metadata for CIM namespace would look like the images captured in [G], [H], [I], and [J] for processor related CIM namespace elements. Goodness of elastic search is available via Searchlight APIs.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Lin Yang (lin-a-yang)
Other contributors:: None

Reviewers

Core reviewer(s): * Flavio Percoco * Ian Cordasco * Nikhil Komawar

Other reviewer(s):

Zhi Yan Liu
Travis Tripp

Work Items

CIM schema to json file

Dependencies

None

Testing

Tests to confirm our parsing is valid
Tests to confirm the metadata has been uploaded correctly in OpenStack

Documentation Impact

Need to add documentation stating that the CIM namespace is being introduced.

References

Migrate the HTTP Store to Use Requests

Fri, 21 Oct 2016 00:00:00

https://blueprints.launchpad.net/glance/+spec/http-store-on-requests

Currently, the glance_store uses httplib to talk to the backing HTTP Store. In the case where the store is served over plain-text (http://) this isn’t an issue. In the event that the store is served over TLS (https://) then the connection was not verified by httplib. In order to provide verification of the connection on all versions of Python, glance_store is moving to use Requests.

Problem description

Currently, the glance_store uses httplib to talk to the backing HTTP Store. In the case where the store is served over plain-text (http://) this isn’t an issue. In the event that the store is served over TLS (https://) then the connection was not verified by httplib [1]. If an operator is serving their store over HTTPS, they may be expecting Glance to verify the connection when downloading the image which is not the case.

At the moment, when Glance downloads an image from the backing store, it does not verify the checksum. If an attacker can properly position themselves, they can intercept the connection by providing a fake (a.k.a., spoofed) certificate. This allows the attacker to essentially perform a denial of service attack by providing bad image data on the behalf of the store. This assumes that the service consuming Glance’s images validates the checksum provided by Glance in the Content-MD5 header. (This also assumes the attacker cannot change that value in the database or before the header reaches the service making the request.) If an attacker is properly positioned, they can also easily perform surveillance of the system, even if they choose not to poison the data.

Further, the attacker could monitor Glance long enough to generate a malicious image with the appropriate checksum (since it is currently MD5 which is no longer cryptographically secure and is increasingly easy to create a collision [2] [3] [4]).

Proposed change

In order to provide verification of the connection on all versions of Python, glance_store should use Requests. A refactor has already taken place, but in order to provide proper backwards compatibility the HTTP Store needs new configuration options.

Users will need:

A way to disable HTTPS Verification

This spec proposes naming that option disable_https_verification.
A way to provide a certificate bundle for verification

This spec proposes naming that option https_ca_bundle.
A way to provide proxy information

This spec proposes naming that option http_proxy_information.

In order to reduce the impact on upgrades, this spec proposes defaulting the new disable_https_verification option to True with logged warnings that it will be changing to False by default in the next cycle. There will be an accompanying OpenStack Security Note (OSSN) written for this case.

Alternatives

The Encrypted and Authenticated Image Support specification might seem to be an alternative but that merely secures the image data, it does not secure the transport.

Data model impact

None

REST API impact

None

Security impact

This will improve the security of the system.

Notifications impact

None

Other end user impact

If the HTTP Store’s certificate expires, users will be unable to download images.

Performance Impact

By using sessions in Requests, multiple requests will be faster due to Requests implementation of connection pooling.

Other deployer impact

Deployers using self-signed certificates for their HTTP Store will need to provide the certificate as part of a bundle to be used by glance_store for verification.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: icordasc
Other contributors:: None

Reviewers

Core reviewer(s):: nikhil-komawar flaper87
Other reviewer(s):: sabari

Work Items

Re-factor the HTTP Store to use Requests
Add configuration options and documentation described above
Write and publish an OSSN

Dependencies

None

Testing

Unit tests should be added to the glance_store library to ensure that operators can disable verification or provide their own bundle.

Documentation Impact

New configuration options will be added and explained.

References

Add distinction layer in domain model

Fri, 17 Jun 2016 00:00:00

https://blueprints.launchpad.net/glance/+spec/add-distinction-layer

To improve the stability and performance it is suggested to make changes in the domain by adding distinction layer, which will store all changes of the object and pass to DB only modified attributes.

Problem description

Current Glance Images v2 architectures have a defect associated with risks of race conditions arising under heavy load on the system. Here is an example of the race condition: Suppose an image has properties {"foo": "bar", "goo": "far"}. User1 wants to change {"foo": "baz"}. User2 wants to change {"goo": "yuck"}. If user1 and user2 make concurrent requests, they both get the same data to work with. Suppose user1’s change gets written first, resulting in: {"foo": "baz", "goo": "far"}. Then user2’s change is written, resulting in: {"foo": "bar", "goo": "yuck"}. User1’s change has been reverted, which is not desired behavior.

Besides this, there is an issue of unreasonably increased load on the database when updating. This problem occurs because all image fields are being updated each time whether they’ve been modified or not, so there is lots of unnecessary traffic on the database. Also there is an issue an issue with changing the status of the image.

These issues do not appear at low load, but in the future, with wider use, it can lead to serious problems, such as loss of information and unstable behaviour.

A more detailed description of the problems can be found on references [7] and [8].

Proposed change

When updating of image the original image is retrieved from the database. Distinction layer creates dict representation of modified image and compares it with the original one. Then it passes to DB layer dict of modified attributes.

Continuing the example from the previous section, consider the situation after user1’s update has been processed and {"foo": "baz"} in the database. User2’s distinction layer image record contains {"foo": "bar"}, since that was the state of the image when the fetch occurred. However, since user2’s change request doesn’t touch foo, only {"goo": "yuck"} will be written to the database. Thus the final image record will have {"foo": "baz", "goo": "yuck"} as both users intend. Of course, if user2 does actually change the value of foo, then user2’s change will be persisted in the database, but this is expected behavior.

Alternatives

There have been attempts to solve the problem using inheritance. References on commits [1], [2]. But commits were abandoned, because inheritance is contrary to the Glance design strategy of favoring flat hierarchies and preferring aggregation over inheritance.

Alternatively, it was suggested to forbid updates while an image is in “saving” state. But this idea was rejected, because images tend to be in status ‘saving’ for a relatively long period, and users expect to be able to modify the image record during this time. Also it fixes only a special case, when status is ‘saving’, but the whole problem with race conditions and unreasonably increased load on the database is still here.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Performance tests were conducted. We used the database: MySQL Ver 14.14 Distrib 5.5.47 We determined the time spent in update one parameter of image. Tests showed the dependence of the execution time of the number of requests. Test code is presented in [3].

We tested two versions of the code: 1) “master” branch; 2) commit “Add distinction layer” [6].

The results are presented in table [4] and graph [5].

Graph showed the dependence of the execution time of the number of requests: * Blue line - “master” branch; * Red line - commit “Add distinction layer” [6].

The testing concluded that the use of distinction layer accelerates update average of 34.54%.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

dshakhray

Reviewers

mfedosin nikhil rosmaita

Work Items

Implement distinction layer:
1. Create a distinction layer and add it to ‘gateway.py’
2. Add classes ImageProxy, ImageRepoProxy, ImageFactoryProxy.
3. Add to ImageRepoProxy class methods get, add, save.
4. Add the class ImageRepoProxy additional methods to implement the logic.
5. Make the appropriate changes in the ‘db/simple/api.py’, ‘db/simple/api.py’, ‘db/__init__.py’.
Add related functional and unit tests.
Write a documentation about this layer.

Dependencies

None

Testing

Unit and functional tests will be added as appropriate.

Documentation Impact

All changes have to be described in detail in the related documentation section.

References

Newton Review Priorities

Thu, 12 May 2016 00:00:00

The list of Newton review priorities:

Priority	Owner(s)	Specs
DefCore Updates	Nikhil Komawar
Image Import Refactor	Brian Rosmaita, Nikhil Komawar	image refactor
Nova V1 -> V2 Support	Flavio Percoco, Mike Fedosin, Sudipta Biswas, Nikhil Komawar	nova.image refactor
Image Sharing	Timothy Symanczyk, Brian Rosmaita, Nikhil Komawar	image sharing
In Tree API Docs	Brian Rosmaita	api-ref docs

Side Priorities

The following are important, but subject to reviewer bandwidth given the previously stated priorities:

Priority	Owner(s)	Specs
Improved Help Text For Config Options	Hemanth Makkapati	helptext

Public glance_store’s API refactor

Wed, 11 May 2016 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/glance_store/+spec/public-api-refactor

The glance_store library was pulled out of Glance some cycles ago. In the process, the code was not changed at all and the refactor of such code was put on-hold for future cycles as the main goal was to have a working library as soon as possible.

The above forced the Glance team to ship a library with an API that was never meant to be public, hence the adoption of the library in other projects has been advised against.

Problem description

The glance_store public API is based on functions with inconsistent signatures, which may or may not behave the same depending on the passed arguments.

These set of functions worked well when the library was part of Glance but they don’t anymore. Just to name a few examples:

There are multiple ways to store data: add_to_backend and store_add_to_backend
Each store needs to implement a StoreLocation class
It’s possible to get a store class from a scheme (get_store_from_scheme), an URI (get_store_from_uri) or well, from a location (shrugs) (get_store_from_location)

The above might not strike as critical issues but they do cause a bad experience for consumers of this library.

Proposed change

The proposal is to refactor this API and provide a more consistent, backwards compatible and clean API. The proposed change is to have a single class capable of doing what the current set of functions do following some of the principles in the current API:

It must be stateless
It must be capable of storing, reading and deleting data from the store.

Some functions are going to be entirely deprecated as part of this change:

verify_default_store Users should not need this function and it should’ve never been public to begin with.
get_store_from_(scheme|uri|location) Users shouldn’t need to access the store driver at all. These 2 functions should’ve never been public to begin with.

The existing functions are going to be marked as deprecated and they’ll be refactored to use the new class instead. This will help us testing the behavior of the new implementation and it’ll verify we’re not breaking backwards compatibility. For example, functions like get_from_backend will use the new class but the tests for this function won’t be changed during the Newton release.

It’s important to note that this spec doesn’t propose changing any of the driver’s code. It’ll focus on the public API. Future enhancements of glance_store will take care of the internal API’s like driver’s.

Alternatives

Rewrite the whole library at once. We’ve attempted to do this and we’ve never gotten past the spec step.
Do nothing. This would leave us with a library that exposes unnecessary functionality and provides a bad experience for users.

Security impact

N/A… I mean, none that I’m aware of besides the usual stuff (“don’t mess this up”)

Other deployer impact

N/A

Developer impact

It’ll make devs happy/happier.

Implementation

Assignee(s)

Primary assignee:: Flavio Percoco (flaper87)
Other contributors:: Anyone? Pretty please?

Reviewers

Core reviewer(s):: Anyone? Pretty please?

Other reviewer(s):

Work Items

Write the new class and tests for it
Rewrite the existing functions
Mark the functions to remove in the Q release as deprecated

Dependencies

N/A

Testing

Existing tests won’t be changed. New tests will be written and we’ll rely on Glance’s gate to ensure backwards compatibility until we improve our gate story in glance_store.

Documentation Impact

Docs will be written for this new class

References

N/A

Remove DB downgrade

Mon, 09 May 2016 00:00:00

https://blueprints.launchpad.net/glance/+spec/remove-db-downgrade

SQL Migrations are the accepted method for OpenStack projects to ensure that a given schema is consistent across deployments. Often these migrations include updating of the underlying data as well as modifications to the schema. It is inadvisable to perform a downwards migration in any environment. As the bp[1] has been approved through openstack-specs after Kilo, and many projects, such as Nova, have implemented it, Glance should do it as well.

This spec proposed the work and Glance team would like to give users a cycle to deprecate downgrade in their environments. So we will remove them during Newton. In Mitaka, we’ll add a deprecation warning to the DB downgrade.

Problem description

Many migrations in OpenStack include data-manipulation to ensure the data conforms to the new schema; often these data-migrations are difficult or impossible to reverse without significant overhead. Performing a downgrade of the schema with such data manipulation can lead to inconsistent or broken state. The possibility of bad-states, relatively minimal testing, and no demand for support renders a downgrade of the schema an unsafe action.

Proposed change

1.Add a deprecation warning in Mitaka[3]. 2.Remove the downgrade script in Newton.

Alternatives

Downgrade paths can continue to be supported.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

Users should make a full database backup of their production data before attempting any upgrade[2].

Performance Impact

None

Other deployer impact

Glance could not be downgrade any more after Newton and it’s still work before it.

Developer impact

As the downgrade will be remove in the future.There is no need to write new downgrade script any more.

Implementation

Assignee(s)

wangxiyuan(wxy)

Reviewers

Flavio Percoco(flaper87) Nikhil Komawar(nikhil)

Work Items

Add a deprecation warning for users.
Remove CLI script for downgrade.
Remove downgrade migration.
Document change.
Add tests code to check there’s no downgrade any more and avoid deployer to add new downgrade script.

Dependencies

None

Testing

Write a test to confirm there is no downgrade in Glance any more.

Documentation Impact

To show that downgrade is deprecated now and will not be supported after Newton. And to indicate how to roll back the DB after that.

References

[1]:http://specs.openstack.org/openstack/openstack-specs/specs/no-downward-sql-migration.html [2]:http://docs.openstack.org/openstack-ops/content/ops_upgrades-roll-back.html [3]:https://bugs.launchpad.net/glance/+bug/1501233

Buffered Reader for Swift Driver

Thu, 24 Mar 2016 00:00:00

Include the URL of your launchpad blueprint: https://blueprints.launchpad.net/glance/+spec/buffered-reader-for-swift-driver

This change proposal introduces a buffering wrapper around image data for use during image uploads. This wrapper helps improve the reliability of image uploads by providing the ability to recover from errors occurring during the upload process. The recovering ability comes from buffering the image segments before uploading to swift. When an error occurs, Swift client resumes the upload process from the last successful position determined using ‘seek’ and ‘tell’ operations on the buffered image segment.

Problem description

What is the problem? Nova uploads image data to glance via a gzip stream. Glance’s swift backend driver reads segments of data from the gzip stream and uploads it to Swift cluster using the swift client. During this process, if an error occurs while uploading data to Swift cluster, swift client aborts the upload as it is incapable of recovering from errors. This eventually triggers the Swift driver to delete all the data uploaded thus far.

Why is the problem a problem? Being unable to recover from errors - Has a direct and negative impact on the reliability of snapshots. - Leads to wastage of network bandwidth utilized to upload data leading up to the error

Why is swift client incapable of recovering from errors? Why not improve it instead? Swift client is actually equipped to recover from socket errors, 401s, 408s, rate-limit errors, 5XXs and others. It handles a decent range of errors and indeed tries to recover from each of the errors in an appropriate way. When an error occurs, swift client tries to resume the upload by repositioning the input source to last successful position by using the reset function [0]. The reset function itself utilizes the ‘seek’ and ‘tell’ operations [1] on the input source to determine the appropriate position from which the upload shall be resumed. This approach works really well when swiftclient is working with a file on disk or any other input source that supports ‘seek’ and ‘tell’ operations. However, in the case of image uploads from nova to glance, the input source is a data stream, which unfortunately doesn’t support the ‘seek’ and ‘tell’ operations. This leads to swift client aborting the upload. So, providing ‘seek’ and ‘tell’ operations on the image data source would help solve this issue automatically by virtue of swift client’s retry mechanism.

Proposed change

Introduce a wrapper around image data, in the shape of a reader class called BufferedReader, that supports ‘seek’ and ‘tell’ operations.

Currently, glance uploads image data to Swift as Dynamic Large Object (DLO), which holds the data in several smaller objects called segments. Glance’s swift driver uses the swift client to upload each segment individually to the backend storage. However, before passing on the image segments to swift client, the swift driver wraps [2] the segments with ChunkReader class [3], which enables the swift client to upload the image segment in smaller chunks, usally 64KB in size.

In the proposed approach, instead of wrapping every segment with ChunkReader, we wrap it with BufferedReader, which buffers the image segment by tee-ing image data to a temporary file as it’s being uploaded to Swift. As the image segment is available on disk, BufferedReader proxies the ‘read’, ‘seek’ and ‘tell’ operations to the underlying temporary file. The temporary file is deleted if the image segment is uploaded successfully or when the upload is aborted after exhausting retries. If the segment fails to upload successfully, swift client’s retry mechanism resumes the upload by repositioning the input using ‘seek’ and ‘tell’ operations that are now available.

We propose to introduce the BufferedReader as a configurable option, while leaving the default behavior as is. Using this configuration option, the amount of disk space used for buffering can be regulated as well. If the allotted space for buffering is full, BufferedReader reverts to default behavior, i.e. data is uploaded to Swift without being buffered.

Alternatives

An alternative is to use memory buffering instead of disk buffering. With memory buffering, there’ll be a substantial increase in the memory footprint, which may degrade the API performance. Also, disk is a cheaper resource than memory but not necessarily faster.
A solution is proposed in [4] to solve this problem for 401s specifically. This proposal basically works by ensuring the token is not nearing expiration. If it is expiring soon, it renews the token. Though this works well for 401 in particular, a similar approach may not be possible for other errors.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

Notifications won’t be added, modified or deleted.

Other end user impact

None

Performance Impact

With buffering to disk, there is an additional cost involved with writing to and reading from disk. This may potentially add a performance penalty to image uploads. However, the bulk of time elapsed for image upload is spent in transferring data over the network. In comparison, read and write operations to disk are relatively much quicker. With some thorough testing, the performance impact of this approach can be studied in further detail.

Other deployer impact

As the image data is buffered to disk, there will be more disk utilization on the API than before. The disk utilization will be equivalent to: (image segment size * number of simultaneous snapshots)

Deployers need to take the extra disk utilization into account while using this feature. Additional care should be taken so that this disk utilization doesn’t impact the disk space available for glance cache in any way.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: hemanth-makkapati
Other contributors:: belliott jesse-j-cook

Reviewers

Core reviewer(s):: nikhil-komawar stuart-mclaren
Other reviewer(s):: brian-rosmaita mfedosin

Work Items

Implement BufferedReader class
Make the reader class configurable and leave the default behavior as-is
Test on devstack

Dependencies

No dependencies as such but [4] is a related effort.

Testing

Documentation Impact

The new configuration option would require documentation.

References

[0] https://github.com/openstack/python-swiftclient/blob/stable/kilo/swiftclient/client.py#L1296-L1297 [1] https://github.com/openstack/python-swiftclient/blob/stable/kilo/swiftclient/client.py#L1376-L1381 [2] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L546-L550 [3] https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/swift/store.py#L928-L942 [4] https://review.openstack.org/#/c/199049/

Swift Multi-tenant Store Service Token Support

Thu, 24 Mar 2016 00:00:00

This is a proposal for how Composite Tokens (aka Service Tokens) can be used by Glance to improve the Swift Multi-Tenant store.

Specifically, to:

Guarantee consistency between the Glance database and the Swift store. (For example remove the ability for users to delete image objects without Glance knowing.)
Store images separately from users’ regular Swift data, preventing noise in their account, and more clearly abstracting the Glance API.
Ensure policy enforcement
Remove the potential for namespace collisions

Blueprint:: https://blueprints.launchpad.net/glance/+spec/swift-multi-tenant-store-service-token-support

Problem description

Glance uses Swift to store data on behalf of users.

There are two approaches to how the data is stored:

Single-tenant. Objects are stored in a single dedicated Swift account (i.e., all data belonging to all users is stored in the same account).
Multi-tenant. Objects are stored in the end-user’s Swift account (project). Typically, dedicated container(s) are created to hold the objects.

There are advantages and limitations with both approaches as described in the following table:

Item	Feature/Topic	Single- Tenant	Multi- Tenant
1	Fragile to password leak (CVE-2013-1840)	Yes	No
2	Fragile to token leak (*)	Yes	No
3	Fragile to container deletion	Yes	No
4	Fragile to service user deletion	Yes	No
5	“Noise” in Swift account	No	Yes
6	Namespace collisions (user and service picking same name)	No	Yes
7	Guarantee of consistency (Glance database vs swift account) ($), (+)	Yes	No
8	Policy enforcement (e.g., Image Download)	Yes	No
9	Fair Swift rate limiting (users unaffected by others’ Swift use)	No	Yes

(*) “Fragile” here means that a single leaked token could be used to: access all users’ image data.
($) Users can delete objects which underpin images in the multi-tenant: case.
(+) If delayed delete is enabled it’s not clear that the multi-tenant: case will ‘scrub’ properly. This is not being considered as part of this spec.

Proposed change

Optionally require a Service Token when accessing image data stored in a multi-tenant Swift store.

This proposal uses the “Service Token Composite Authorization” support in the Keystone middleware [1].

This proposal also uses the Swift support for multiple reseller prefixes which allow storing objects in project-specific accounts while retaining control over how those objects are accessed via Composite Tokens [2].

The changes will apply to all accesses to the Swift backend (upload, download etc).

Example: Existing Image Download:

+----------------+
|      User      |
+-------+--------+
        |
        | 1. Get Glance Image 123
        |
        | GET http://glance:9292/v2/images/123/file
        | X-AUTH-TOKEN: <token for user project 'abc'>
        |
+-------v---------+
|     Glance      |
+-------+---------+
        |
        | 2. Get Swift Object Data
        |
        | GET http://swift:8080/v1/AUTH_abc/glance_123/123
        | X-AUTH-TOKEN: <token for user project 'abc'>
        |
+-------v---------+
|      Swift      |
+-----------------+

The proposed access model has two changes:

Rather than using the standard AUTH_ Swift reseller prefix a specialized prefix, eg IMAGE_ will be used.
A service token will be generated and supplied with the Swift request

Example: Proposed Image Download:

+----------------+
|      User      |
+-------+--------+
        |
        | 1. Get Glance Image 456
        |
        | GET http://glance:9292/v2/images/456/file
        | X-AUTH-TOKEN: <token for user project 'abc'>
        |
+-------v---------+
|     Glance      |
+-------+---------+
        |
        | 2. Get Swift Object Data
        |
        | GET http://swift:8080/v1/IMAGE_abc/glance_456/456
        | X-AUTH-TOKEN: <token for user project 'abc'>
        | X-SERVICE-TOKEN: <token for Glance service project>
        |
+-------v---------+
|      Swift      |
+-----------------+

The service token will be generated much as a token for the single-tenant Swift store is today, ie credentials will be stored as part of Glance’s configuration. Unlike the single-tenant store credentials, if the multi-tenant service account credentials leak they will not give direct access to all images.

The combination of the user and service token will allow access for for project abc under the IMAGE_ reseller prefix. Specifically, Swift will verify that the service token contains the particular role required to access the relevant reseller prefix. (For more detailed information see the relevant Swift spec [2]).

The Swift reseller prefix can be operator defined, and will be part of both the Swift and Glance configuration.

Requests to Swift for the IMAGE_ prefix which do not contain a suitably scoped service token will return HTTP Forbidden (403).

Existing non-service token behaviour will continue to be supported.

Service token generation will not be tied to a particular project. There is no reliance on a particular project. If the project is deleted a new project with the same role can be created and used to generate the service token.

A rolling password change of the service project can be performed by using either two separate projects or two users in the same project.

If an operator modifies their configuration to take advantage of the new behaviour pre-existing images — images stored under the old reseller prefix AUTH_ — will continue to be accessible. The service token will still be supplied to Swift, but it will be ignored.

Example: Image Download, backwards compatibility:

+----------------+
|      User      |
+-------+--------+
        |
        | 1. Get Glance Image 123
        |
        | GET http://glance:9292/v2/images/123/file
        | X-AUTH-TOKEN: <token for user project 'abc'>
        |
+-------v---------+
|     Glance      |
+-------+---------+
        |
        | 2. Get Swift Object Data
        |
        | GET http://swift:8080/v1/AUTH_abc/glance_123/123
        | X-AUTH-TOKEN: <token for user project 'abc'>
        | X-SERVICE-TOKEN: <token for Glance service project>
        |
+-------v---------+
|      Swift      |
+-----------------+

Alternatives

Two Swift installations could be used to give similar behaviour by firewalling user access to one Swift. That would incur a lot of hardware and operator overhead.

Data model impact

There is no impact on the data model per se.

(New image ‘location’ entries will be slightly different, as they will contain a different Swift path.)

REST API impact

None.

Security impact

This change enhances security by preventing direct access to image data via Swift. This removes the ability to bypass, for example, the image download policy for public images, shared images, and user owned images.

Notifications impact

None

Other end user impact

New image objects will not be listed in users’ Swift accounts.

Performance Impact

A service token will need to be requested by the Glance API process when Swift data is accessed. This should have minimal impact. The token can be cached so will only impact a minority of requests which access Swift. Requests which do not access Swift (eg listing images) will not require a service token.

There may be more cases of tokens expiring (and hitting uploads/downloads) as both the user token and the service token can potentially expire. There are some current efforts around mitigating token expiration. It may be possible to re-use some of those efforts for the service token.

Other deployer impact

Operators will need to create a service project and modify their Swift and Glance configurations if they wish to take advantage of the new behaviour. (Unmodified configurations will work as before.)

Pre-existing images will continue to be accessible.

Developer impact

We may propose some changes to python-swiftclient.

Implementation

Assignee(s)

Primary assignee: Stuart McLaren

Reviewers

Core reviewer(s): Flavio Percoco, Nikhil Komawar

Other reviewer(s): TBD

Work Items

Handle new configuration (Service credentials, Swift reseller prefix)

Token generation/caching

Any swift client changes

Test rolling password change

Dependencies

Keystone changes to introduce the concept of Service Tokens have been implemented [1]

Swift changes to introduce support for Service Tokens/multiple reseller prefixes have been implemented [2]

Required Swift client changes have been implemented [3]

Testing

Ideally this would become the default configuration for Glance tests in Tempest, and also the default configuration for devstack.

Documentation Impact

Example policy files will need to be created that show how to use the new data provided from X-SERVICE-TOKEN when making policy enforcement decisions.
Update the Glance configuration docs

References

Buffered Reader for Swift Driver

Thu, 24 Mar 2016 00:00:00

Include the URL of your launchpad blueprint: https://blueprints.launchpad.net/glance/+spec/buffered-reader-for-swift-driver

Problem description

Proposed change

Introduce a wrapper around image data, in the shape of a reader class called BufferedReader, that supports ‘seek’ and ‘tell’ operations.

Alternatives

An alternative is to use memory buffering instead of disk buffering. With memory buffering, there’ll be a substantial increase in the memory footprint, which may degrade the API performance. Also, disk is a cheaper resource than memory but not necessarily faster.
A solution is proposed in [4] to solve this problem for 401s specifically. This proposal basically works by ensuring the token is not nearing expiration. If it is expiring soon, it renews the token. Though this works well for 401 in particular, a similar approach may not be possible for other errors.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

Notifications won’t be added, modified or deleted.

Other end user impact

None

Performance Impact

Other deployer impact

As the image data is buffered to disk, there will be more disk utilization on the API than before. The disk utilization will be equivalent to: (image segment size * number of simultaneous snapshots)

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: hemanth-makkapati
Other contributors:: belliott jesse-j-cook

Reviewers

Core reviewer(s):: nikhil-komawar stuart-mclaren
Other reviewer(s):: brian-rosmaita mfedosin

Work Items

Implement BufferedReader class
Make the reader class configurable and leave the default behavior as-is
Test on devstack

Dependencies

No dependencies as such but [4] is a related effort.

Testing

Documentation Impact

The new configuration option would require documentation.

References

Swift Multi-tenant Store Service Token Support

Thu, 24 Mar 2016 00:00:00

This is a proposal for how Composite Tokens (aka Service Tokens) can be used by Glance to improve the Swift Multi-Tenant store.

Specifically, to:

Guarantee consistency between the Glance database and the Swift store. (For example remove the ability for users to delete image objects without Glance knowing.)
Store images separately from users’ regular Swift data, preventing noise in their account, and more clearly abstracting the Glance API.
Ensure policy enforcement
Remove the potential for namespace collisions

Blueprint:: https://blueprints.launchpad.net/glance/+spec/swift-multi-tenant-store-service-token-support

Problem description

Glance uses Swift to store data on behalf of users.

There are two approaches to how the data is stored:

Single-tenant. Objects are stored in a single dedicated Swift account (i.e., all data belonging to all users is stored in the same account).
Multi-tenant. Objects are stored in the end-user’s Swift account (project). Typically, dedicated container(s) are created to hold the objects.

There are advantages and limitations with both approaches as described in the following table:

Item	Feature/Topic	Single- Tenant	Multi- Tenant
1	Fragile to password leak (CVE-2013-1840)	Yes	No
2	Fragile to token leak (*)	Yes	No
3	Fragile to container deletion	Yes	No
4	Fragile to service user deletion	Yes	No
5	“Noise” in Swift account	No	Yes
6	Namespace collisions (user and service picking same name)	No	Yes
7	Guarantee of consistency (Glance database vs swift account) ($), (+)	Yes	No
8	Policy enforcement (e.g., Image Download)	Yes	No
9	Fair Swift rate limiting (users unaffected by others’ Swift use)	No	Yes

(*) “Fragile” here means that a single leaked token could be used to: access all users’ image data.
($) Users can delete objects which underpin images in the multi-tenant: case.
(+) If delayed delete is enabled it’s not clear that the multi-tenant: case will ‘scrub’ properly. This is not being considered as part of this spec.

Proposed change

Optionally require a Service Token when accessing image data stored in a multi-tenant Swift store.

This proposal uses the “Service Token Composite Authorization” support in the Keystone middleware [1].

The changes will apply to all accesses to the Swift backend (upload, download etc).

Example: Existing Image Download:

+----------------+
|      User      |
+-------+--------+
        |
        | 1. Get Glance Image 123
        |
        | GET http://glance:9292/v2/images/123/file
        | X-AUTH-TOKEN: <token for user project 'abc'>
        |
+-------v---------+
|     Glance      |
+-------+---------+
        |
        | 2. Get Swift Object Data
        |
        | GET http://swift:8080/v1/AUTH_abc/glance_123/123
        | X-AUTH-TOKEN: <token for user project 'abc'>
        |
+-------v---------+
|      Swift      |
+-----------------+

The proposed access model has two changes:

Rather than using the standard AUTH_ Swift reseller prefix a specialized prefix, eg IMAGE_ will be used.
A service token will be generated and supplied with the Swift request

Example: Proposed Image Download:

+----------------+
|      User      |
+-------+--------+
        |
        | 1. Get Glance Image 456
        |
        | GET http://glance:9292/v2/images/456/file
        | X-AUTH-TOKEN: <token for user project 'abc'>
        |
+-------v---------+
|     Glance      |
+-------+---------+
        |
        | 2. Get Swift Object Data
        |
        | GET http://swift:8080/v1/IMAGE_abc/glance_456/456
        | X-AUTH-TOKEN: <token for user project 'abc'>
        | X-SERVICE-TOKEN: <token for Glance service project>
        |
+-------v---------+
|      Swift      |
+-----------------+

The Swift reseller prefix can be operator defined, and will be part of both the Swift and Glance configuration.

Requests to Swift for the IMAGE_ prefix which do not contain a suitably scoped service token will return HTTP Forbidden (403).

Existing non-service token behaviour will continue to be supported.

A rolling password change of the service project can be performed by using either two separate projects or two users in the same project.

Example: Image Download, backwards compatibility:

+----------------+
|      User      |
+-------+--------+
        |
        | 1. Get Glance Image 123
        |
        | GET http://glance:9292/v2/images/123/file
        | X-AUTH-TOKEN: <token for user project 'abc'>
        |
+-------v---------+
|     Glance      |
+-------+---------+
        |
        | 2. Get Swift Object Data
        |
        | GET http://swift:8080/v1/AUTH_abc/glance_123/123
        | X-AUTH-TOKEN: <token for user project 'abc'>
        | X-SERVICE-TOKEN: <token for Glance service project>
        |
+-------v---------+
|      Swift      |
+-----------------+

Alternatives

Two Swift installations could be used to give similar behaviour by firewalling user access to one Swift. That would incur a lot of hardware and operator overhead.

Data model impact

There is no impact on the data model per se.

(New image ‘location’ entries will be slightly different, as they will contain a different Swift path.)

REST API impact

None.

Security impact

Notifications impact

None

Other end user impact

New image objects will not be listed in users’ Swift accounts.

Performance Impact

Other deployer impact

Operators will need to create a service project and modify their Swift and Glance configurations if they wish to take advantage of the new behaviour. (Unmodified configurations will work as before.)

Pre-existing images will continue to be accessible.

Developer impact

We may propose some changes to python-swiftclient.

Implementation

Assignee(s)

Primary assignee: Stuart McLaren

Reviewers

Core reviewer(s): Flavio Percoco, Nikhil Komawar

Other reviewer(s): TBD

Work Items

Handle new configuration (Service credentials, Swift reseller prefix)

Token generation/caching

Any swift client changes

Test rolling password change

Dependencies

Keystone changes to introduce the concept of Service Tokens have been implemented [1]

Swift changes to introduce support for Service Tokens/multiple reseller prefixes have been implemented [2]

Required Swift client changes have been implemented [3]

Testing

Ideally this would become the default configuration for Glance tests in Tempest, and also the default configuration for devstack.

Documentation Impact

Example policy files will need to be created that show how to use the new data provided from X-SERVICE-TOKEN when making policy enforcement decisions.
Update the Glance configuration docs

References

Support download from and upload to Cinder volumes

Tue, 01 Mar 2016 00:00:00

https://blueprints.launchpad.net/glance/+spec/cinder-store-upload-download

Current Glance’s Cinder backend driver only can get the volume size but doesn’t support read, write and deletion of volumes.

As missing parts in Cinder side (listed in [1]: os-brick library, readonly-volumes, multi-attach) are now supported, now we can implement read (download), write (upload), and deletion for it.

When an raw image is backed on the cinder store, a new volume with the image can be created by cloning the image volume. This will be useful to launch boot-from-volume instances rapidly as some Cinder drivers can clone a volume efficiently using copy-on-write. It will also improves storage capacity with the drivers which support thin-provisioning. [3]

Note that the image volume itself is not intended to be used as a boot-volume for an instance. Image volumes are marked read-only, so they cannot be attached to instances in read-write mode. Instead, cloned volumes should be used for booting instances.

Problem description

Without upload, download and deletion support of Glance’s Cinder backend driver, it can’t be used as a default store.

Proposed change

This spec proposes implementing image uploading to and downloading from Cinder volumes, and deletion of the volume on the image deletion.

Uploading steps:
- Create a new volume with the size that can store the image. It will be placed in the tenant specified in glance-api.conf.
- Attach the new volume to the glance-api host using the os-brick library.
  - The os-brick library collects the connector (host) information, such as the initiator IQN for iSCSI or WWN for FibreChannel.
  - Call initialize_connection API of Cinder with the connector information and the volume as arguments to get the volume connection information, such as target portal IP address and IQN for iSCSI.
  - Pass the connection information to the os-brick’s connect_volume method to attach the volume to the host. The volume device information such as the device path (e.g. /dev/sdX) is returned. This step requires root privilege to execute the command for iSCSI, FC and so on, so the glance-rootwrap must be installed and it should allow the commands for os-brick. [2]
- Write the image data to the volume from the given device path.
- Detach the volume from the host.
  - Call the os-brick’s disconnect_volume method and Cinder’s terminate_connection API to remove the volume connection.
- Mark the volume as read-only.
- Add volume metadata that indicate the image size, image ID, image owner (to track the owner even when it is placed in the service tenant). e.g.:
```
{
 'image_id': 'e38f5596-4bd3-4b63-b4ef-88bcc814ed8e',
 'image_owner':     'b58fffa08b4242988c42950b5f24fcd5',
 'image_size':      '228599296',
}
```
Downloading steps:
- Attach the image volume to glance-api host as read-only.
- Read the image data from the volume.
- Detach the volume.
Deletion steps:
- Call the Cinder’s delete API to delete the volume when the image is deleted.

As cinder currently doesn’t support ACL or public volumes among tenants, in order to support public or sharable images, the image volume should be placed in the service tenant which is only accessible from the Glance and Cinder services to control the accessibility.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

To attach and detach volumes to the host, root privilege is required to operate block devices on the host. Glance needs to import oslo-rootwrap to allow only required commands for these operations to be executed as root.

The Glance node must be a part of storage network (iSCSI, FC etc.). In theory that might open new attack vectors each way.

Notifications impact

None

Other end user impact

None

Performance Impact

When creating a new Cinder volume from an image stored in the Glance Cinder store, or creating a new image from a volume, Cinder is able to offload the image copy using storage array features such as background copy or copy-on-write, if some conditions (e.g. image stored in Cinder backend is in raw format) are satisfied. This will improve the performance of image copy significantly.

Nova-compute will be able to bypass the image copy by attaching the image volumes to the host, if Cinder is deployed.

Note that the image upload/download requires sufficient bandwidth to/from cinder storage.

Other deployer impact

To use cinder backends, the Glance node must be able to access the backend storage and it may require additional hardware connectivity (iSCSI, FC, etc.). Operators have to configure cinder and glance-api.conf appropriately.

To enable cinder store, cinder must be added to the stores option in the glance-api.conf.
To place the image volume into the specific tenant, authentication information for the tenant must also be provided.
The glance-rootwrap must be installed. The rootwrap config path should also be configured in glance-api.conf.
To offload the image copy on volume creation to the storage array, the allowed_direct_url_schemes option should contain cinder in cinder.conf. [3] Also, the glance-api host must be able to attach the Cinder volumes onto itself.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: tsekiyam

Reviewers

Core reviewer(s):: flaper87

Work Items

Import os-brick and oslo-rootwrap into glance_store.
Extend current cinder backend driver to support upload, download and deletion.

Dependencies

None

Testing

Enable cinder backend in the devstack.
Test uploading of an image to the cinder backend.
Then test downloading of the image again.
Delete the image and check if the volume is deleted.
Test the other normal glance store operations such as owner changes and sharing.

Documentation Impact

The documentation should be expanded to describe how to enable and use cinder store. Especially, it should explain new options to configure cinder authentication (tenant id, username, password) to store the volume into the specific tenant.

It also needs to cover the requirement to Glance node being the part of storage network (iSCSI, FC etc.) and having sufficient bandwidth towards the storage.

References

Supporting OVF Single Disk Image Upload

Tue, 01 Mar 2016 00:00:00

https://blueprints.launchpad.net/glance/+spec/ovf-lite

The Open Virtualization Format (OVF) is an open standard for packaging and distributing virtual appliances that is hypervisor and processor architecture agnostic. Both enterprise and Telco Network Virtual Function (NVF) providers desire OpenStack OVF support, to distribute in a standard format appliance images and deployment requirements to ensure optimal performance. An OVF package, also called an OVA (file extension ova), is essentially a tar ball that contains disk images, a manifest and the ovf xml file. A common use case is a single disk image package with the ovf file containing deployment metadata. NFV workloads in particular tend to require enhanced platform features and fine-grained resource provisioning to meet their performance criteria. For example, special instructions, hardware accelerators, high speed IO, NUMA awareness, and more. We propose to extract the single root disk image and its metadata and save the same in Glance with the goal of easing provisioning, in particular eliminating the need for error prone manual editing of image metadata. The glance image metadata can be used by a custom filter (in the Mitaka release no nova image properties filter changes can be upstreamed), or by defining host aggregates using tags from the CIM namespace[H], that which is used in the OVF to identify appropriate compute hosts.

Problem description

The OVF image format is a format backed by industry (VMware, Ericsson, IBM, and Red Hat to name a few) and the European Telecommunications Standards Institute. A common use case is a single disk image OVF package. To handle the same, VMware even developed a special purpose Nova driver that launches a virtual machine using an uploaded OVF package with a single image. However, each time the OVF image is used, it needs to be parsed and the image and its metadata extracted because it is not a first class glance image, incurring a performance overhead.

Proposed change

We propose supporting OVF package by adding a new task capable of extracting the package data and metadata. Support will be limited to extracting the root disk image and saving it to the Glance image store, followed by parsing the .ovf file to extract desired metadata specified via a configuration file. Note that OVF files, which are in XML format, can be complex, and XML allows users the freedom of representing particularly leaf values as either an element or as an attribute, making parsing non-trivial. We shall restrict ourselves in this release to the items in CIM_ProcessorAllocationSettingData given they are of primary interest to us for VM orchestration purposes. The extracted metadata will be recorded as image properties on the Glance image record. Typically, the image provider will specify metadata pertaining to hardware requirements and deployment parameters that ensure optimal performance. This import functionality, for the present, will be restricted to admin users, and the feature itself can be disabled or enabled through configuration using functionality as proposed in [L]. The metadata may serve, among other things, to aid Nova in optimal placement/orchestration to ensure meeting vendor performance, compliance, and/or other requirements.

Alternatives

The alternative is to manually enter image metadata as it is done today, which is both tedious and error prone, even as we try to automate service delivery in the cloud.

A full fledged solution where an OVF package has multiple disk images is out of scope for this effort but is a logical evolution of the feature. We anticipate in the future the creation of OVF artifacts and transforming their contents into Heat artifacts. The richness of OVF would need to be captured in Heat, including its networking aspects.

Data model impact

None

REST API impact

None

Security impact

Unpacking an unknown OVA tar file can have security risks, namely gzip expansion and tar escalation of privilege attacks[J]. In this initial implementation we shall restrict access to OVA import to admin users. In a future extension we could address some of these vulnerabilities such as, examining the tar manifest first (tar tvf foo.ova) to confirm no system paths or high value files like /etc/passwd are mentioned. To protect from gzip bomb style attacks an anti-virus program would come in handy and should be threaded into the process flow. A signature verification functionality would enable opening up the import functionality to non-admin users, particularly to trusted image suppliers.

Notifications impact

While notifications for OVA processing start and finish can be helpful there shall be none at this point.

Other end user impact

Ease of use.

Performance Impact

On image upload, additional processing takes place for the extracting and parsing of the image data and metadata, but this only affects OVA files. Other formats are unaffected.

With regards to Nova, launching an OVF package will be faster because it is processed on import (once) rather than on each use.

Other deployer impact

Existing OVF packages in Glance can be deployed as before using the VMware driver. Once this functionality is delivered, disk images may be directly deployed, like qcow and other images. The Nova OVF driver cannot be deprecated until all OVF packages are processed using this new strategy. Please note that this feature requires disk space to support untarring of archives and that operations might fail due to inadequate disk resources. Further, the deployer should check the value of image_property_quota in glance-api.conf to ensure that the quota is set high enough to allow for all the desired OVF metadata to be stored as image properties.

Developer impact

None

Implementation

A global configuration parameter specifying the metadata of interest to extract. Per feedback, this shall be maintained in a separate file ovf-metadata.conf with a sample file provided. For example:

{
  ovf_metadata_properties:
      ProcessorArchitecture,
      InstructionSet,
      InstructionSetExtensionName,
      ..
}

Possible values:

ProcessorArchitecture: x86, x86-64, ARM, MIPS
InstructionSet: ARM A32, ARM A64, MIPS32 (32-bit), MIPS64 (64-bit), POWER
InstructionSetExtensionName:
    Advanced Vector Extensions (AVX),
    RDRAND (aka SecureKey),
    Advanced Encryption Standard New Instruction (Intel® AES-NI).

See [H] for an exhaustive list.

Download OVA
Process the embedded .ovf descriptor file to extract metadata and the root disk. Parsing will use the schema in [G]
Save the root disk image in Glance
Attach the extracted metadata as image properties of the saved disk image

In this initial implementation we will not handle encryption keys or OVF certificates. Doing so however would add to the security of the feature and possibly reduce threats as mentioned in [J]. The defcore compliant refactored image TaskFlow also provides for increased security by way of limiting size of import and time taken.

An initial implementation using the TaskFlow task executor from Liberty is ready. When the refactored image import API becomes available, by restricting the OVF lite import feature to admins only, migrating to the same should be smooth. Tests to establish the same required. Thus progress on both efforts can occur in parallel.

Should we drop the desire to support upload of compressed ova, we eliminate the threat of compression attacks. In conjunction with first examining the embedded ovf file, it is possible to determine the anticipated size of the disk image and ensure it is within quota before proceeding with the upload. Thanks to Mike Gerdt for this compression constraint relaxation suggestion to make the feature more generally available, that is not restricted to admin users. However, for Mitaka, we shall restrict this feature to admin users only, thus allowing compressed files, and in so doing better network bandwidth use, and more importantly conforming to the image upload refactor work [M] underway.

Assignee(s)

Primary assignee: * Deepti Ramakrishna (dramakri) * Lin Yang (ling-a-yang)

Other contributors: * Jakub Jasek * Kent Wang

Reviewers

Core reviewer(s): * nikhil-komawar * Erno Kuvaja * Ian Cordasco * Flavio Percoco * Sabari Murugesan

Other reviewer(s): * Brian Rosmaita

Work Items

Create workflows for tasks, to parse .ovf file, to create glance image with metadata. Our Liberty solution missed the deadline.
Update configuration file to indicate metadata of interest.
Update configuration and usage documentation

Dependencies

Image save task, shall confirm else enhance that it checks user quota.

Testing

Tests for upload covering valid and invalid input such as invalid tar bundle, directory with no .ovf file, zero size and excessively large input file sizes.
With OVF Lite we would upload both an OVF file in Glance and a regular image. We could configure OpenStack to skip saving the OVF file.
Integration tests would involve being able to launch an image so loaded using Nova boot commands without using the VMware OVF driver.

Documentation Impact

OVF API for CRUD operations need to be documented to indicate the additional image creation. Delete and update operations would require delete and update of the additional image in Glance. Will help to document.

Backward functionality is preserved in an upgraded system by maintaining the Nova OVF driver.

In the future it would be good to introduce a script/task to be invoked in an upgraded environment that provides OVF Lite support for existing OVF files in Glance. Further, should we want to enable/disable this feature it would be good to work on [L] which currently is abandoned.

References

https://en.wikipedia.org/wiki/Open_Virtualization_Format
https://en.wikipedia.org/?title=ETSI
http://specs.openstack.org/openstack/nova-specs/specs/juno/approved/vmware-driver-ova-support.html
http://docs.openstack.org/developer/glance/formats.html
Original blueprint: https://wiki.openstack.org/wiki/Enhanced-Platform-Awareness-OVF-Meta-Data-Import
https://blueprints.launchpad.net/glance/+spec/introspection-of-images
OVF Envelope XML Schema Document (XSD). http://schemas.dmtf.org/ovf/envelope/2/dsp8023_2.0.1.xsd
http://schemas.dmtf.org/wbem/cim-html/2/CIM_ProcessorAllocationSettingData.html
CIM V2.38.0 schema. http://dmtf.org/standards/cim/cim_schema_v2380
https://bugs.python.org/issue21109#msg215656
https://en.wikipedia.org/wiki/Zip_bomb
Config option for importing subflows: https://review.openstack.org/#/c/194898/
Image Import Refactor: https://review.openstack.org/#/c/232371
https://blueprints.launchpad.net/glance/+spec/image-signing-and-verification-support
https://review.openstack.org/#/c/214810/
DMTF Common Information Model (CIM). http://dmtf.org/standards/cim

Prevention of Unauthorized errors during upload/download in Swift driver

Tue, 01 Mar 2016 00:00:00

https://blueprints.launchpad.net/glance/+spec/prevention-of-401-in-swift-driver

This change proposal introduces a new mechanism for uploading/downloading chunked files to Swift store, that prevents possible 401 errors if user token expires during the image upload/download. It helps to improve the reliability of image uploads and downloads by providing the ability to re-authenticate the user during the process, thereby enhancing the stability of the entire system.

Problem description

There are several shortcomings in the current Swift driver implementation:

1. In case of upload. If an uploading into Swift file exceeds user-set size limit, then the file is split into chunks and each chunk is uploaded with single PUT request as an independent object. These requests require authorization and, unfortunately, it may happen that user token expires in interim chunk, which will lead to an Unauthorized error and fail of upload.

2. In case of download. Because Swift driver supports “retrying” mechanism, it may happen, that in case of unsuccessful operation there will be another attempt to download the file. However, if the token has expired after the previous attempt, the new one will fail, too, with Unauthorized error.

These issues are typical for both, SingleTenant and MultiTenant implementations, although the decisions in both cases will be different.

Proposed change

It’s proposed to change workflows of uploading and downloading files into Swift in the following manner:

Prerequisites:

New optional dependency from python_keystoneclient has been added for Swift backend in glance_store. The additional layer (later ‘ConnectionManager’) between swift store and swiftclient connections will be added. ConnectionManager will be responsible for initializing keystone client and requesting swift client connections (if needed). ConnectionManager executes re-authentication if connection token is going to expire soon and re-authentication has been enabled. If re-authentication is not enabled ConnectionManager always returns the same connection (no connection refresh). If re-authentication is enabled, ConnectionManager requests all pre-requisites that allow to request the new token:

for single-tenant store: glance service user credentials (auth_url, domain, username, password, project). All required credentials stored in StoreLocation object that passed to the Store so there is no need for the new parameters or config options.
for multi-tenant store: trust id, glance service user credentials ( username, project, password). Service user credentials will be extracted from swift_store_config_file or (swift_store_user, swift_store_key) config options. So there is no need for new configuration options. trust_id will be created as a result of the following steps:
1. initialize keystone client scoped to user token
2. request user roles using keystone client from 1)
3. initialize keystone client scoped to glance service user project (use username and password to create Password auth plugin) and request user_id of the service user.
4. create trust using the client from 1), user roles from 2) and user_id from 3). Store trust_id in ConnectionManager.

After that ConnectionManager initializes and stores keystone client. Keystone client provides a method to request new valid token from Identity service. ConnectionManager can use that token to build a new Swift Connection.

Note If client cannot be created because of some reason then glance_store raises an exception (for single-tenant) or uses token from user context(for multi-tenant).

Additionally the method called ‘init_client’ need to be added to store and it will be responsible for initializing of keystone client.

Upload case.

Note All these changes will apply only to the chunked upload. If the upload is performed as a single request, it will use the old workflow.

In general, the following scenario is proposed:

1. Before the upload glance_store identifies if image need to be chunked. In case of chunked image glance_store initializes ConnectionManager that support re-authentication.

2. Before uploading each chunk, ConnectionManager checks if re-authentication is enabled and the token will expire soon (it uses ‘will_expire_soon’ method of authentication plugin and swift_store_expire_soon_interval value to define when the token needs to be requested). If the previous condition is true then ConnectionManager requests new token and creates new swiftclient connection. That connection will be used for uploading the chunk. Note ‘will_expire_soon’ used because glance_store might lose the data if request to swiftclient fail with Unauthorized error. Unfortunately, requests library starts reading the data before checking the token so we cannot retry the requests after failure. More specifically, the requests library doesn’t support 100-continue.
Download case.
In general, the following scenario is proposed:

1. Before the download, glance_store identifies if swift_retry_get_count is positive. It the option value is positive glance_store initializes ConnectionManager that allows to re-authenticate and initialize a new connection. Otherwise, ConnectionManager always returns the same swift connection.

2. In the case of unsuccessful download of the file, before the retry, ConnectionManager checks if re-authentication is enabled and connection token is going to expire soon. If the previous condition is true ConnectionManager requests new token and returns new connection.
1. Glance executes download retry normally.

Alternatives

At least one workaround for the whole functionality is available: extend token expiration time to allow Glance upload the image. This solution affects all services and it does not look like long term solution.

Also there are several workarounds of this issue, but they work for SingleTenant implementation only, because for MultiTenant case valid user token is required:

1. Introduce a wrapper around image data, in the shape of a reader class called BufferedReader, that supports ‘seek’ and ‘tell’ operations, which buffers the image segment by tee-ing image data to a temporary file as it’s being uploaded to Swift. It’s stated as optional resource-consuming solution that is used when connection between Glance and Swift is unstable, because it helps not only Unauthorized errors.

2. An alternative is to use memory buffering instead of disk buffering. With memory buffering, there’ll be a substantial increase in the memory footprint, which may degrade the API performance. Also, disk is a cheaper resource than memory but not necessarily faster.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

Keystone V3 should be supported to properly use trusts mechanism.

Performance Impact

If image file is too big then glance_store need to create trust and initialize client for every upload to multi-tenant store. The same needs to be done if swift_retry_get_count is positive and user is downloading an image from swift store in multi-tenant mode.

Other deployer impact

None

Developer impact

The following blueprint defines buffered reader for swift driver: https://blueprints.launchpad.net/glance/+spec/buffered-reader-for-swift-driver. It describes buffered reader that allows to retry image upload in case of failure. It may seem that this blueprint overlaps with Upload case in this specification but it is different. The goal of this specification is to provide a swiftclient connection with valid token to the user. Buffered reader is responsible for retrying the upload with the original user token.

Implementation

Assignee(s)

mfedosin kkushaev dshakhray

Reviewers

flaper87 stuart-mclaren

Work Items

Implement ConnectionManager class
Use ConnectionManager to request connections in ‘add’ and ‘get’ method of Swift store

Dependencies

None.

Testing

None.

Documentation Impact

One configuration option: will_expire_soon interval needs to be described in documentation. It defines period of time before token expiration when ConnectionManager must request new token and initialize new connection.

References

Trusts blueprint: https://blueprints.launchpad.net/glance/+spec/trust-authentication

Buffered reader blueprint: https://blueprints.launchpad.net/glance/+spec/buffered-reader-for-swift-driver

Add filters using an ‘in’ operator

Tue, 01 Mar 2016 00:00:00

https://blueprints.launchpad.net/glance/+spec/in-filtering-operator

This specification introduces a feature to the Glance v2 API for filtering images based on lists of id, name, status, container_format or disk_format using the ‘in’ operator. This feature is important because it allows Horizon to make only one request to Glance instead of multiple single requests, which improves the system performance and reduces the load.

Problem description

Horizon sends separate requests to get the names of the images presented in the ID forms of the instances. It would be better to send one request instead of a lot of separate requests.

Proposed change

I propose adding the ability to filter images by the properties id, name, status,``container_format``, disk_format using the ‘in’ operator between the values.

Alternatives

Alternative is implement this type of query to the Searchlight service. But Searchlight is an optional service, and Horizon is always required the listing of running instances.

Also this particular functionality, the “in” operator, should still be added for the following reasons: * consistency with Glare, which has implemented the “in” operator; * there’s a minimum set of filters that should be allowed for effective service-to-service communication.

Thus while try searching for UIs should be offloaded to Searchlight, this particular functionality is basic and useful enough that it’s worth including in the API.

Data model impact

None

REST API impact

Following the pattern of existing filters, new filters may be specified as query parameters using the field to filter as the key and the filter criteria as the value in the parameter.

Filtering based on the principle of full compliance with the template, for example ‘name = in:deb’ does not match ‘debian’.

Changes apply exclusively to the API v2 Image entity listings:

GET /v2/images/{image\_id}

Filter by adding optional query parameters:

id, name, status, disk_format, container_format

An example of an acceptance criteria using the ‘in’ operator for name:

?name=in:name1,name2,name3

These filters will be added using syntax that conforms to the latest guidelines from the OpenStack API Working Group and any applicable draft guidelines [1].

Example:

GET /v2/images?name=in:name1,name2
{
    "first": "/v2/images?name=in:name1,name2",
    "images": [
        {
            "checksum": null,
            "container_format": "bare",
            "created_at": "2015-12-18T12:02:09Z",
            "disk_format": "raw",
            "file": "/v2/images/381b6dfb-48c2-4fcd-860d-9c7b10876730/file",
            "id": "381b6dfb-48c2-4fcd-860d-9c7b10876730",
            "min_disk": 0,
            "min_ram": 0,
            "name": "name1",
            "owner": "a03febe481094927a96fe367c15c347b",
            "protected": false,
            "schema": "/v2/schemas/image",
            "self": "/v2/images/381b6dfb-48c2-4fcd-860d-9c7b10876730",
            "size": null,
            "status": "queued",
            "tags": [],
            "updated_at": "2015-12-18T12:02:09Z",
            "virtual_size": null,
            "visibility": "private"
        },
        {
            "checksum": null,
            "container_format": "bare",
            "created_at": "2015-12-18T12:02:15Z",
            "disk_format": "raw",
            "file": "/v2/images/a3b9db48-5b6f-40e5-9cc1-d586f01281cc/file",
            "id": "a3b9db48-5b6f-40e5-9cc1-d586f01281cc",
            "min_disk": 0,
            "min_ram": 0,
            "name": "name2",
            "owner": "a03febe481094927a96fe367c15c347b",
            "protected": false,
            "schema": "/v2/schemas/image",
            "self": "/v2/images/a3b9db48-5b6f-40e5-9cc1-d586f01281cc",
            "size": null,
            "status": "queued",
            "tags": [],
            "updated_at": "2015-12-18T12:02:15Z",
            "virtual_size": null,
            "visibility": "private"
        }
    ],
    "schema": "/v2/schemas/images"
}

Max page size will still be enforced. For example if the max page size is 3 and I do request ‘id=in:1,2,3,4’ - I only get 3 images and a link to get the fourth.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Performance tests were conducted. We determined the time spent in obtaining the list of images, filtering and repeating the image-get query. The results are presented in [2]:

Blue line - used repeating the request.
Red line - used a filter.

Test code is presented in [3].

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: dshakhray
Other contributors:: None

Reviewers

mfedosin jokke

Work Items

None

Dependencies

None

Testing

Unit and functional tests will be added as appropriate.

Documentation Impact

Docs should be updated with a description of new API filters and usage, as well as of the additional policy options.

References

Glance db purge utility

Thu, 25 Feb 2016 00:00:00

https://blueprints.launchpad.net/glance/+spec/database-purge

This spec adds the ability to sanely and safely purge deleted rows from the glance database for all relevant tables. Presently, we keep all deleted rows. I believe this is unmaintainable as we move towards more upgradable releases. Today, most operators depend on manual DB queries to delete this data, but this exposes the DB to human errors.

The goal is to have this be an extension to the glance-manage db command. Similar specs are being submitted to all the various projects that touch a database.

Problem description

Very long lived OpenStack installations will carry around database rows for years and years. This brings following problems:

If deleted data is kept in the DB, the number of rows can grow very large taking up the disk space of nodes. Larger disk space means more worry for disaster recovery, long running non-differential backups, etc.
Large number of deleted rows also means, an admin or authorized owner querying for the corresponding rows will get 5xx responses timing out on the DB, eventually slowing down other queries and API performance.
DB upgradeability is a big challenge if the older data style are less or inconsistent with the latest formats. An example would be the image locations string where older location string styles are different from the latest.

To date, there is no “mechanism” to programmatically purge the deleted data. The archive rows feature doesn’t solve this.

Proposed change

The proposal is to add a “purge” method to DbCommands in glance/glance/cmd/manage.py. This will take a maximum count of rows to delete and number of days argument and use that for a data_sub match. Like:

DELETE FROM images
  WHERE deleted != 0 AND deleted_at > data_sub(NOW()...)
  LIMIT count;

Alternatives

Today, this can be accomplished manually with SQL commands, or via script. There is also the archive_deleted_rows method. However, this won’t satisfy certain data destruction policies that may exist at some companies.

Data model impact

None, all tables presently include a “deleted_at” column.

REST API impact

None, this would be run from glance-manage

Security impact

None, This only touches already deleted rows.

Notifications impact

None

Other end user impact

This affects the operator’s ability to identify and delete all those rows from the DB that have location stored for backend data entities, that are not deleted (or partially deleted) due to some error at the time of the DELETE api call or during scrubber’s data purge run.

This has another operator/user impact for the calls that require changes-since filter. It is required by the Nova proxy API and exists in v1 (and potentially in v2). Purging the deleted data results into information evaporation that the changes-since filter use case is designed for. One example, if it is needed to check an old snapshot that has been deleted but has some vital info for cross checking. Some operators give/want to give that functionality to users. Purging breaks that contract and operators need to be aware about it.

Performance Impact

This has the potential to improve performance for very large databases. Very long-lived installations can suffer from inefficient operations on large tables.

Another performance impact is on the negative side unless the operators are being careful about the count. A long running purging operation will potentially cause delay in the upgrade process. It is also likely to result into a different differential backup of the DB and that may delay the process.

Other deployer impact

Some developers who want to optimize calls like re-adding a deleted image-member to image while re-sharing that image, will have to consider the rows can be deleted from now on.

Developer impact

None

Implementation

Assignee(s)

Primary assignee: * mmagr

Reviewers

Core reviewer(s): * flaper87 * nikhil-komawar

Other reviewer(s):: None

Work Items

Add purge functionality to manage.py db/api.py db/sqlalchemy/api.py Add tests to confirm functionality Add documentation of feature

Dependencies

None

Testing

The test will be written as such. Three rows will be inserted into a test db. Two will be “deleted=1”, one will be “deleted=0” One of the deleted rows will have “deleted_at” be NOW(), the other will be “deleted_at” a few days ago, lets say 10. The test will call the new function with the argument of “7”, to verify that only the row that was deleted at 10 days ago will be purged. The two other rows should remain.

Documentation Impact

The documentation needs to emphasize that the image_locations table will be trimmed, which will destroy all information about where the image was stored in various backends. The operator should keep this in mind when selecting the number-of-days value for the purge function.

References

This was discussed on both the openstack-operators mailing list and the openstack-developers mailing lists with positive feedback from the group.

http://lists.openstack.org/pipermail/openstack-dev/2014-October/049616.html

Deprecate Glance v3 API

Thu, 25 Feb 2016 00:00:00

https://blueprints.launchpad.net/glance/+spec/move-v3-to-glare

On Mitaka summit it was decided [1] to make v3 (artifact) API a standalone service with its own endpoint. To do that we have to deprecate Glance v3 API and related things, and create new service with all required utilities.

Problem description

Initially it was planned to make artifacts a new API with incremented version, i.e. “v3”. But since Glance is a core project, it falls under the DefCore specifications, which require (among other things) uniqueness of public APIs. Unfortunately v3 api is pluggable and it can’t be unique by design.

Also there are issues with understanding what Glance API is, because v1 and v2 are Image APIs, and they work with images only. But v3 was considered to be unified (Artifact) API, which may work with objects of any nature. The general feeling in the broader OpenStack community is that rather than being a new version of the Images API, the Artifacts API should be considered an entirely different thing.

Proposed change

Because v3 API has experimental status, it’s proposed to deprecate v3 API and move all its code to the whole new standalone service with new endpoint. It will release the Artifacts project from being subject to DefCore requirements and will allow the project to move forward faster.

Having the independent service also removes misunderstanding, because there will be stable, DefCore-approved Glance Image API and pluggable independent Artifact API, which may include Glance API in the future.

Alternatives

It’s possible to leave everything as-is, but it doesn’t remove all of the above issues.

Data model impact

None. All created for artifacts tables with prefix ‘artifacts-’ will stay in DB, but will be used by another service.

REST API impact

All of experimental APIs, that start with ‘/v3’, are deprecated and will be removed.

Security impact

None.

Notifications impact

None.

Other end user impact

Experimental ‘feature/artifacts’ branch will be removed from ‘python-glanceclient’ repo.

Performance Impact

None.

Other deployer impact

‘apiv3app’ application has to be removed from ‘glance-api-paste.ini’

‘enable_v3_api’ parameter has to be removed from glance-api config

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:

dshakray

Other contributors:

mfedosin

Reviewers

flaper87
nikhil-komawar
ativelkov

Work Items

glance-api-paste.ini: remove ‘apiv3app’ application;
glance/common/config.py: remove ‘enable_v3_api’ parameter;
glance/api/middleware/version_negotiation.py: remove major version discoverability, if version is 3, raise ValueError instead;
move glance/api/v3/* code into glance/artifacts/api;
make related changes in the tests;

Dependencies

None.

Testing

None.

Documentation Impact

None.

References

[1] Mitaka glance artifacts review: https://etherpad.openstack.org/p/mitaka-glance-artifacts-review

Implement trusts for Glance

Wed, 27 Jan 2016 00:00:00

https://blueprints.launchpad.net/glance/+spec/trust-authentication

This change proposal introduces a way for Glance to use Keystone trust authorization.

Problem description

Keystone tokens have some restricted lifetime. After the user token has expired, any request initiated by Glance which needs a valid user token will fail. This causes the original user’s request to also fail, even though the token was originally valid when passed to Glance.

This this spec intends to address the specific case where a token expires during image upload causing the call to the registry to set the image state ‘active’ to fail:

User requests image-upload.
Keystone Middleware accepts the request and passes the request to Glance.
Glance passes all required data to glance_store.
glance_store uploads an image but it takes a lot of time (more than token expiration time)
Glance sends a request to registry to change image status.
Keystone Middleware rejects the request because user token has expired.

As a result the image never transitions to ‘active’ status and so isn’t usable.

Increasing the token expiration time doesn’t seem to be a good long-term solution.

Note

This implementation of trusts in glance_store is out of scope for this specification. The current spec is related to Glance and it defines trusts implementation for communication with Glance Registry.

Note

Nova snapshots also suffer from a token expiration issue. Nova first creates a queued image before saving the instance state to local disk. Then the image bytes are uploaded. Potentially the token can expire while the instance state is being saved to disk. This spec will not address this case.

Proposed change

The proposal changes the Glance behavior when uploading images in Glance v2 api with enabled Registry server, i.e. when data_api = glance.db.registry.api.

Step 1. Glance received request for image-upload.

Step 2. Before upload begins Glance tries to create a trust with the following parameters:

token: user token
project: user project
roles: all user roles
trustee_user: system user that specified in CONF.keystone_authtoken configuration group
trustor_user: user who initiated the request.

Glance keeps trust_id until request processing is finished. If trust cannot be created because of some reason then Glance uses user token for further steps.

Step 3. Glance initiates and completes the image upload (this part is executed by glance_store).

Before starting step 4 Glance has an image uploaded to store and it needs to update the image record in database. That requires the user token to be valid for V2 API if data_api is Glance Registry.

Step 4. If authentication is required (see the text above) then Glance requests the new token using the trust_id (see Step 2). Glance updates the request context with the new token.

Step 5. Glance updates image record in database. If Registry is used then it receives the new token.

Alternatives

There has been some discussion around updating how the keystone middleware interprets a combination of a valid service token and expired user token – but this is in the discussion/pre-design stage and is not guaranteed to be implemented. Therefore we cannot currently base our solution on it, and it’s recommended to use trusts. In the future this may be superseded by the use of service tokens but it’ll have to be discussed when that time comes.

Earlier there was a config option, called ‘use_user_token’. If it’s disabled glance extracted user token from the context and changed it to admin’s. Unfortunately, this option was considered as harmful and not acceptable for real deployments, because it allowed to perform any operation in registry with admin rights. That’s why this behaviour was deprecated in Mitaka.

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

Keystone V3 should be supported to properly use trusts mechanism.

Performance Impact

None.

Other deployer impact

To deploy Glance with trusts the following config should be defined: * trustee user grants should be specified in CONF.keystone_authtoken group: username, password, auth_uri, ssl options.

In devstack all parameters are defined by default.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: mfedosin
Other contributors:: kkushaev

Reviewers

flaper87 stuart-mclaren

Work Items

Add trust authorization module to Glance.
Implement trust authorization between Glance and Glance registry for V2 API.

Dependencies

To use trusts Glance needs to support Keystone V3. If V3 is not supported, Glance will use old user token to finish upload operation.

Testing

The unit test and functional tests should be implemented.

Manual testing on devstack:

Preparation: Use ‘file’ as ‘default_store’ for glance-api, set ‘expiration’ option in keystone.conf to ‘40’ (seconds), set ‘token_cache_time’ in glance-registry.conf to ‘-1’ to disable it, set ‘data_api’ in glance-api.conf to ‘registry’.
Try to upload big image with v2 API (when upload takes at least 1 minute). Make sure that upload fails with Unauthorized error.
Apply trusts patches.
Try to upload image again. Make sure that upload was finished successfully.

Documentation Impact

None

References

Mitaka Project Priorities

Fri, 02 Oct 2015 00:00:00

List of priorities (in the form of use cases) the glance development team is prioritizing in Liberty (in no particular order).

Priority	Owner(s)	Specs
DefCore Updates	Flavio Percoco
Image Import Refactor	Brian Rosmaita, Stuart Mclaren	image refactor
Nova V1 -> V2 Support	Flavio Percoco, Mike Fedosin	nova.image refactor

Priorities without a clear plan

Here are some things we would like to be a priority, but we are currently lacking either a clear plan or someone to lead that effort:

Revisit tempests tests
- Verify all the required tests for the API v2 exist
- Tempest’s tests are used as a reference by DefCore
- These tests will prove the interoperability of the API
Glance trusts
Pull V3 out of glance-api’s process into its own process/endpoint

DefCore Updates

Establish a communication channel with DefCore so that constant syncs can be had. Interactions with the DefCore team are important, especially when it comes down to our API evolution. We must make sure this becomes a standard practice.

Image Import Refactor

Define and implement a consistent, reliable, user-friendly and public capable image import workflow.

The work here is focused on evolving the existing workflows into something that could be standardized and used as a reference by other groups like DefCore.

Nova V1 -> V2 Support

Start and complete the support for Glance’s v2 in Nova. This will be a joint effort between both communities and it’ll be split into several parts. Our tasks for this consist in contributing to Nova and helping them move forward.

Basic conversion of Images

Wed, 11 Mar 2015 00:00:00

https://blueprints.launchpad.net/glance/+spec/basic-import-conversion

Some store engines work better when working with specific image formats. For example, in the case of ceph, having a raw image improves performance and allows ceph to do smarter things with the image data. Therefore, this spec proposes adding a basic support to image conversion that will give operators full control on the destination format. The proposed change is the starting point for more complex conversion operations.

Problem description

With the support of asynchronous operations and the new import workflow in Glance (see the dependency tree), it will be possible to have an image format conversion on the fly while importing an image.

The conversion will be provided by a plugin of the import workflow. This plugin can be activated or not based on the deployer configuration. This means that the deployer will need to specify the preferred format of images for the deployment.

This blueprint will handle the conversion of formats supported by qemu-img convert: raw, qcow2, vdi, vmdk and vpc.

Internally, Glance will receive the bits of the image in a XX format. These bits will be stored in a temporary location. The plugin will be triggered to convert the image to its target format YY and moved to its final destination. When the task is finished, the temporary location is deleted. This means that the format uploaded initially is not kept by Glance.

Proposed change

Using Glance’s Asynchronous Workers, we can execute a background task that converts images from the source format to a pre-configured format.

Alternatives

Let the user convert images themselves and upload the final, converted, file.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Based on the maximum number of workers and the size that the administrator has configured for the conversion workspace, performance could be negatively affected if the operator misconfigures their maximum number of workers and allocated workspace.

In addition to the above, it’s also important to consider the time required to download the image locally, convert it and then upload it to the final store. The implementation proposes using qemu-image, which executes random accesses to the image data.

Other deployer impact

When configuring this feature, operators will need to know the average size of the images that they are managing and appropriately configure the max_workers setting and an appropriate amount of space for the workers to use.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: flaper87
Other contributors:: None

Reviewers

Core reviewer(s):: jokke_ nikhil-komawar kragniz icordasc

Work Items

Create specific workers for different conversion tasks
Create workflows for tasks
Update configuration files with additions
Update configuration documentation

Dependencies

This depends on Glance’s Asynchronous Workers.

Testing

Unit testing will be needed for the conversion tasks and for the new task flows.

Documentation Impact

This may have an impact on the upgrade and installation parts of the documentation. For operators upgrading, they’ll need to understand how to properly configure a system for image conversion. For new users, they’ll need to be warned about appropriately allocating space for the workers to use and possibly choosing a more conservative maximum worker number than is default until they can determine the appropriate number.

References

None

Introspection of Images

Tue, 10 Mar 2015 00:00:00

https://blueprints.launchpad.net/glance/+spec/introspection-of-images

Several image formats include metadata inside the image data. This spec proposes that Glance expose that metadata through introspection of the image. For example, we can read the metadata from a vmdk formatted image to know that the disk type of the image is “streamOptimized”. Allowing Glance to perform the introspection removes the burden from the administrator. Exposing this metadata also helps the consumer of the image; Nova’s workflow is very different based on the disk type of the image.

Problem description

Several image formats include metadata inside the image data. A stream optimized vmdk image could be introspected like so:

$ head -20 so-disk.vmdk

# Disk DescriptorFile
version=1
CID=d5a0bce5
parentCID=ffffffff
createType="streamOptimized"

# Extent description
RDONLY 209714 SPARSE "generated-stream.vmdk"

# The Disk Data Base
#DDB

ddb.adapterType = "buslogic"
ddb.geometry.cylinders = "102"
ddb.geometry.heads = "64"
ddb.geometry.sectors = "32"
ddb.virtualHWVersion = "4"

When a user of the image looks at the metadata they will see important information like the required disk type, “streamOptimized”. Extracting the metadata in Glance and exposing it through Glance’s API, means that administrators and image users alike do not need to perform the introspection themselves. Consumers like Nova will have very different workflows based on disk type alone and could also optimize the rest of its workflow based on other image metadata.

Proposed change

Using Glance’s Asynchronous Workers, we can extract the image metadata without requiring a separate node and without suffering significant performance degradation.

Alternatives

A separate worker node could ostensibly be controlled to prevent degradation on the Glance API nodes but is an unnecessary addition to the architecture.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Based on the maximum number of workers and the size that the administrator has configured for the introspection workspace, performance could be negatively affected if the operator misconfigures their maximum number of workers and allocated workspace.

Other deployer impact

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: flaper87
Other contributors:: jokke_

Reviewers

Core reviewer(s):: nikhil-komawar kragniz icordasc

Work Items

Create specific workers for different introspection tasks
Create workflows for tasks
Update configuration files with additions
Update configuration documentation

Dependencies

This depends on Glance’s Asynchronous Workers.

Testing

Unit testing will be needed for the introspection tasks and for the new task flows.

Documentation Impact

This may have an impact on the upgrade and installation parts of the documentation. For operators upgrading, they’ll need to understand how to properly configure a system for image introspection. For new users, they’ll need to be warned about appropriately allocating space for the workers to use and possibly choosing a more conservative maximum worker number than is default until they can determine the appropriate number.

References

None

Support for multivalue operators in metadata catalog

Thu, 26 Feb 2015 00:00:00

https://blueprints.launchpad.net/glance/+spec/metadata-multivalue-operators-support

The metadata catalog provides data that user can pin to resources, e.g., flavors extra specs. These are used by scheduler to choose a host that satisfies the requirements provided in extra specs. Nova scheduler implements operators to combine multiple values under one key in extra specs: <or> and <all-in>. Unfortunately, the catalog does not provide which operator is suitable for a given property.

This blueprint aims at this problem by extending current jsons with suitable operators for each property.

Problem description

There is work being done in Horizon to provide support for multiple values for a single key in extra specs, however, Glance currently does not provide information about which operators are suitable for a given property. In compute-host-capabilities.json there is a property called cpu_info:features. Both operators - <all-in> and <or> can be used for this property. There are also properties like cpu_info:model where <or> is the only operator that should work.

Proposed change

To make end-user (e.g. Horizon) aware which operator can be used for a given property, this blueprint extends existing properties (and also properties inside objects) with a new section under key - “operators”:

Currently an example property would be structured like this:

"cpu_info:features": {
    "title": "Features",
    "description": "Specifies CPU flags/features.",
    "type": "array",
    "items": {
        "type": "string",
        "enum": [
            "aes",
            "vme",
            "de"
        ]
    }
}

And after extension:

"cpu_info:features": {
    "title": "Features",
    "description": "Specifies CPU flags/features.",
    "operators": [
        "<or>",
        "<all-in>"
    ],
    "type": "array",
    "items": {
        "type": "string",
        "enum": [
            "aes",
            "vme",
            "de"
        ]
    }
}

The added section is:

"operators": [
    "<or>",
    "<all-in>"
]

This section will be optional, e.g., property that is integer does not need operator.

Also glance will not do any check on “operators” field. API consumer needs to take care to provide valid operators for nova scheduler. Currently there are three operators that are valid for nova scheduler and are usable with metadata definitions: <or>, <in> and <all-in>

Alternatives

None

Data model impact

This will not impact data model, because “operators” will be part of the blob stored in the json_schema column in the database table.

REST API impact

Example extended GET object body:

{
    "objects": [
        {
            "name": "object1",
            "namespace": "my-namespace",
            "description": "my-description",
            "properties": {
                "prop1": {
                    "title": "My Property",
                    "description": "More info here",
                    "operators": ["<all-in>"],
                    "type": "string",
                    "readonly": true
                }
            }
        }
    ],
    "first": "/v2/metadefs/objects?limit=1",
    "next": "/v2/metadefs/objects?marker=object1&limit=1",
    "schema": "/v2/schema/metadefs/objects"
}

Example POST/PUT body on objects:

{
    "name": "StorageQOS",
    "description": "Our available storage QOS.",
    "required": [
        "MyProperty"
    ],
    "properties": {
        "MyProperty": {
            "type": "string",
            "readonly": false,
            "description": "The My Property",
            "operators": ["<or>"],
            "enum": ["type1", "type2"]
        }
    }
}

Because “operators” field is optional API consumer needs to handle default value as it may be missing.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Upgrade of jsons or database in existing OpenStack installation will not be needed. “operators” section will not be mandatory so the installation can keep running without upgrade.

Also there is possibility that property/object will be missing operators even when the value has been added in the latest built-in metadata definition json templates, the upgrade process does not update existing database. Deployer needs to manually upgrade metadata definitions to the newest set.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: pawel-koniszewski
Other contributors:: None

Reviewers

Core reviewer(s):: lzy-dev

Work Items

Extend existing metadata jsons with operators section.
Extend API json schema with new option

Dependencies

Horizon blueprint that depends on this blueprint: https://blueprints.launchpad.net/horizon/+spec/metadata-widget-multivalue-selection

Nova blueprint that this blueprint depends on: https://blueprints.launchpad.net/nova/+spec/add-all-in-list-operator-to-extra-spec-ops

Testing

Current unit tests and functional tests will be extended to make sure that the new section is returned and that it is correct. Tests will also ensure that the operators part is optional.

Documentation Impact

The new attribute needs to be documented.

References

None

Image API v2 listing

Fri, 13 Feb 2015 00:00:00

Pagination

This call is designed to return a subset of the larger collection of images while providing a link that can be used to retrieve the next. You should always check for the presence of a ‘next’ link and use it as the URI in a subsequent HTTP GET request. You should follow this pattern until there a ‘next’ link is no longer provided. The next link will preserve any query parameters you send in your initial request. The ‘first’ link can be used to jump back to the first page of the collection.

If you prefer to paginate through images manually, the API provides two query parameters: ‘limit’ and ‘marker’. The limit parameter is used to request a specific page size. Expect a response to a limited request to return between zero and limit items. The marker parameter is used to indicate the id of the last-seen image. The typical pattern of limit and marker is to make an initial limited request then to use the id of the last image from the response as the marker parameter in a subsequent limited request.

Filtering

The list operation accepts several types of query parameters intended to filter the results of the returned collection.

A client can provide direct comparison filters using most image attributes (i.e. name=Ubuntu, visibility=public, etc). A client cannot filter on tags or anything defined as a ‘link’ in the json-schema (i.e. self, file, schema).

The ‘size_min’ and ‘size_max’ query parameters can be used to do greater-than and less-than filtering of images based on their ‘size’ attribute (‘size’ is measured in bytes and refers to the size of an image when stored on disk). For example, sending a size_min filter of 1048576 and size_max of 4194304 would filter the container to include only images that are between one and four megabytes in size.

Sorting

The results of this operation can be ordered by using classic and new sorting syntaxes. Classic syntax uses multiple ‘sort_key’ and ‘sort_dir’ parameters, and new one accepts a single ‘sort’ string with comma separated sort keys with optional sort direction after ‘:’. Both syntaxes provide an ability to sort output with multiple keys and directions but with some differences.

The classic syntax takes a list of keys and either exactly the same number of directions for each key, or only one that specifies the default value for all keys. The new syntax applies a default direction to all keys where it’s missing.

The API uses the natural sorting of whatever image attribute is provided as the sort key. List of image attributes can be used as the sort key: ‘name’, ‘status’, ‘container_format’, ‘disk_format’, ‘size’, ‘id’, ‘created_at’, ‘updated_at’. The sort dir parameter indicates in which direction to sort. Acceptable values are ‘asc’ (ascending) and ‘desc’ (descending). Default values for sort key and sort direction are ‘created_at’ and ‘desc’.

Examples of sorting:

New syntax with specified direction for keys:
```
sort=name:asc,status:asc
```
Sort: by name with asc order, then by status with asc order.
New syntax with missing direction:
```
sort=name,status:asc
```
Sort: by name with desc order, then by status with asc order.
New syntax without directions:
```
sort=name,status
```
Sort: by name with desc order, then by status with desc order.
Classic syntax with specified default direction:
```
sort_key=name&sort_key=status&sort_dir=asc
```
Sort: by name with asc order, then by status with asc order.
Classic syntax with missing direction:
```
sort_key=name&sort_key=status
```
Sort: by name with desc order, then by status with desc order.
Classic syntax with missing key and specified default direction:
```
sort_dir=asc
```
Sort: by created_at with asc order.
Classic syntax with specified direction for keys:
```
sort_key=name&sort_dir=desc&sort_key=status&sort_dir=asc
```
Sort: by name with desc order, then by status with asc order.
Classic syntax with different number of keys and directions:
```
sort_key=name&sort_dir=asc&sort_key=status&sort_dir=asc&sort_key=id
```
Will be an error.

Glance sorting enhancements

Fri, 13 Feb 2015 00:00:00

https://blueprints.launchpad.net/glance/+spec/glance-sorting-enhancements

Currently, the sorting support for Glance allows the caller to specify multiple sort keys and one sort direction. This blueprint enhances the sorting support for the /images and /images/detail APIs so that multiple sort keys and sort directions can be supplied on the request.

Problem description

There is no support for retrieving image data based on multiple sort directions; multiple sort keys and one direction are currently supported, and they’re defaulted to descending sort order by the “created_at” key.

In order to retrieve data in any sort order and direction, the REST APIs need to accept multiple sort keys and directions.

Use Case: A UI that displays a table with only the page of data that it has retrieved from the server. The items in this table need to be sorted by status first and by display name second. In order to retrieve data in this order, the APIs must accept multiple sort keys/directions.

Proposed change

The /images and /images/detail APIs will align with the API working group guidelines [1] for sorting and support the following parameter on the request:

sort: Comma-separated list of sort keys, each key is optionally appended with <:dir>, where ‘dir’ is the direction for the corresponding sort key (supported values are ‘asc’ for ascending and ‘desc’ for descending).

For example:

/images?sort=status:asc,name:asc,created_at:desc

Note: The “created_at” and “id” sort keys are always appended at the end of the key list if they are not already specified in the request.

The database layer already supports multiple sort keys and directions. This blueprint will update the API layer to retrieve the sort information from the API request and pass that information down to the database layer.

All sorting is handled in the glance.db.sqlalchemy.api._paginate_query function. This function accepts an ORM model class as an argument and the only valid sort keys are attributes on the given model class. Therefore, the valid sort keys are: ‘name’, ‘status’, ‘container_format’, ‘disk_format’, ‘size’, ‘id’, ‘created_at’, ‘updated_at’.

Alternatives

Multiple sort keys and directions could be passed using repeated ‘sort_key’ and ‘sort_dir’ query parameters. For example:

/images?sort_key=status&sort_dir=asc&sort_key=name&sort_dir=asc&
sort_key=created_at&sort_dir=desc

To provide users with the ability to use the classic familiar syntax and to ensure a smoother transition to the new one, classic syntax should be implemented too.

Data model impact

None

REST API impact

The following existing v2 GET APIs will support the new sorting parameters:

/v2/images
/v2/images/detail

Note that the design described in this blueprint could be applied to other GET REST APIs, but this blueprint is scoped to only those listed above. Once this design is finalized, then the same approach could be applied to other APIs.

The existing API documentation needs to be updated to include the following new Request Parameters:

Parameter	Style	Type	Description
sort	query	string	Comma-separated list of sort keys and optional sort directions in the form of key<:dir>, where ‘dir’ is either ‘asc’ for ascending order or ‘desc’ for descending order. Defaults to the ‘created_at’ and ‘id’ keys in descending order.

Currently, the images query supports the ‘sort_key’ and ‘sort_dir’ parameters; these will be deprecated. The API will raise a “BadRequest” error response (code 400) if both the new ‘sort’ parameter and a deprecated ‘sort_key’ or ‘sort_dir’ parameter is specified.

Neither the API response format nor the return codes will be modified, only the order of the images that are returned.

In the event that an invalid sort key or sort direction is specified, then a “BadRequest” error response (code 400) will be returned with a message like “Invalid input received: Invalid sort key” or “Invalid input received: Invalid sort dir” respectively.

Security impact

None

Notifications impact

None

Other end user impact

The python-glanceclient should be updated to accept sort keys and sort directions, using the ‘sort’ parameter being proposed in the cross-project spec [2].

Performance Impact

None

Other deployer impact

The choice of sort keys has no impact on data retrieval performance. Therefore, the user should be allowed to retrieve data in whatever order they need to for creating their views (see use case in the Problem Description).

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: mfedosin
Other contributors:: None

Reviewers

ativelkov

icordasc

nikhil-komawar

Work Items

Ideally the logic for processing the sort parameters would be common to all components and would be done in oslo; a similar blueprint is also being proposed in nova: https://blueprints.launchpad.net/nova/+spec/nova-pagination

Therefore, I see the following work items:

Update the existing API to retrieve the sort information and pass down to the DB layer (https://review.openstack.org/#/c/148326/);
Extend API with new syntax to support multiple sorting keys and directions (https://review.openstack.org/#/c/148512/);
Update the python-glanceclient to accept and process multiple sort keys and sort directions with classic and new sorting syntax (https://review.openstack.org/#/c/120777/, https://review.openstack.org/#/c/148688/, https://review.openstack.org/#/c/148981/);
Implement the full test coverage for the new changes.

Dependencies

CLI Sorting Argument Guidelines cross project spec [2];
Related (but independent) change being proposed in nova [3].

Testing

Both unit and Tempest tests need to be created to ensure that the data is retrieved in the specified sort order. Tests should also verify that the default sort keys (“created_at” and “id”) are always appended to the user supplied keys (if the user did not already specify them).

Documentation Impact

The /images and /images/detail API documentation will need to be updated to:

reflect the new sorting parameters and explain that these parameters will affect the order in which the data is returned.
explain how the default sort keys will always be added at the end of the sort key list.

References

Notification Support for Metadata Definitions

Thu, 22 Jan 2015 00:00:00

https://blueprints.launchpad.net/glance/+spec/metadefs-notifications

This blueprint adds metadata definition notification support.

The implemented juno spec for metadata notifications is here for reference:

http://specs.openstack.org/openstack/glance-specs/specs/juno/metadata-schema-catalog.html

Problem description

Metadata definition resources - namespaces, objects, properties, tags, and resource types - don’t provide any notification events when certain operations are performed on them. This doesn’t allow for any intelligent caching of the data, meaning that the only opportunity for catalog users to be aware of changes is to continually poll. Polling is not performant for catalog users and puts extra load on the API and database.

Proposed change

We are proposing to support notifications for events on the Metadata Definitions Catalog.

This implementation will include the following events that will be triggered when necessary:

metadef_namespace.create - namespace has been created
metadef_namespace.update - namespace has been updated
metadef_namespace.delete - namespace has been deleted
metadef_object.create - object has been created
metadef_object.update - object has been updated
metadef_object.delete - object has been deleted
metadef_property.create - property has been created
metadef_property.update - property has been updated
metadef_property.delete - property has been deleted
metadef_tag.create - tag has been created
metadef_tag.update - tag has been updated
metadef_tag.delete - tag has been deleted
metadef_resource_type.create - resource type has been added to namespace
metadef_resource_type.delete - resource type has been removed from namespace

In addition we will add a new configuration option to allow for disabling individual notifications. It will be a comma separated list with the following syntax:

disabled_notifications = <type>.<action>

For example, the following would disable all tag notification and property deletions:

disabled_notifications = metadef_tag,metadef_property.delete

The notifier will read this configuration and ignore notifications that are in the disabled list.

The reason for this proposed addition is that we aren’t sure whether or not tag notifications could get too chatty for some reason. Therefore, we’d like to give an option for deployers to disable it.

Alternatives

No notification support could be added. In this event, constant polling for changes could be done. The constant polling would lead to both latency in getting updates and extra load on the API and database.

Data model impact

None

REST API impact

No external API impacts will be visible. The API itself will have a notifier added as part of the call.

Security impact

None

Notifications impact

A notifier will be added to the REST API calls. The notifications listed above will be added. A new configuration option will be added to disable specific notifications.

Other end user impact

None

Performance Impact

Additional notifications will be generated, but the amount of changes to metadata definitions are not anticipated to be enough to impact performance.

However we want to add the new disabled_notifications configuration option to allow operators to disable the notifications if they determine that are too chatty.

Other deployer impact

Deployers will have to setup listeners to received notification on metadefs if they desire them.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: kamil-rykowski
Other contributors:: none

Reviewers

Core reviewer(s):: jokke
Other reviewer(s):: lakshmi-sampath travis-tripp

Work Items

Changes would be made to:

Glance v2 metadef APIs to add notifiers
notifer
gateway

Dependencies

Need to sync openstack/common from oslo-incubator for service module.

https://bugs.launchpad.net/glance/+bug/1413861

Testing

Unit tests will be added for all possible code with a goal of being able to isolate functionality as much as possible.

Tempest tests will be added wherever possible.

Documentation Impact

Docs needed for new notification usage

References

Current glance metadata definition catalog documentation.

Pass Targets to Glance’s Policy Enforcer

Wed, 21 Jan 2015 00:00:00

https://blueprints.launchpad.net/glance/+spec/pass-targets-to-policy-enforcer

Currently it’s possible to define custom rules in Glance’s policy.json that rely on attributes other than a user’s roles. Unfortunately, if you attempt to apply one of those rules, it will always cause the user to be prevented from performing the associated action. This specification proposes that we pass the proper target objects to the enforcer so these rules can be used and properly enforced.

Problem description

Currently, Glance promises that permissions can be configured in the policy.json file but any rule other than a role check currently results in a 403 Forbidden response. As this is a promised feature, implementing this specification is merely fixing the already promised behaviour.

It is not possible to restrict access to actions in Glance’s policy.json based on anything other than a user’s roles. The policy enforcer that Glance extends from oslo expects a dictionary-like object to be passed as a target of the action. Every method that uses the policy enforcer and enforces the policy defined for the corresponding action currently passes an empty dictionary ({}) which provides absolutely no data about the actual target.

If we define a rule similar to the following:

"tenant_is_owner": "tenant%(owner)s"

And we apply it to an action, e.g.,

"delete_image": "rule:tenant_is_owner"

Then every request to delete any image will be denied with a 403 (Unauthorized) response from Glance’s API. The reason stems from how the rule is parsed and the target is passed in. The "rule:tenant_is_owner" rule will be parsed as a GenericCheck. These checks are split on the : into a kind and a match (roughly, "<kind>:<match>"). The match portion is then interpolated with the target, i.e.,

match = self.match % target

So using our example above, we would do

match = "%(owner)s" % {}

Except that this raises a KeyError which means the check immediately returns False and fails. In this particular instance (deleting an image), if we passed an instance of glance.api.policy.ImageTarget, then what would instead happen is that the interpolation would succeed.

Proposed change

The solution for image-based resources is simple. We have the image on the policy proxies that relate to images. We simply pass that to glance.api.policy.ImageTarget and pass the resulting instance to the policy enforcer so it can be accessed like a dictionary when interpolated. For members and tasks, there is no target class that we can use. These are very thin classes that could easily be written.

Once we have the appropriately defined target classes, we would then update the places where the policy is enforced to use instances of those target classes.

With proper targets in place, we can also implement safer default policy rules for operators who rely solely on the default policy.json file.

Alternatives

Custom rule creation based on attributes of the target object could be disabled. This would severely limit an operator’s ability to restrict actions based on a user’s tenant and other properties of the target of the action.

Data model impact

None

REST API impact

None

Security impact

This will give operators a significant amount of control over the security of their Glance installations. Currently they can only restrict actions based on roles which may be sufficient in some cases. If the operator, however, wishes to restrict access based on other factors (besides role) they cannot do this. If they try to do it, there is no indication that it will not work but they can essentially produce a Denial of Service to users who should be able to perform actions based on the policy defined.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

To leverage the fixes in this specification, operators need to update their versions of policy.json used in their deployments. To write a rule, operators need to know that there are three values provided by glance that can be used in a rule on the left side of the colon (:). Those values are the current user’s credentials in the form of:

role
tenant
owner

The left side of the colon can also contain any value that Python can understand, e.g.,:

True
False
"a string"
&c.

Role checks are going to continue to work exactly as they already do. If the role defined in the check is one that the user holds, then that will pass, e.g., role:admin.

Using tenant and owner will only work with Images or actions that interact with an image. Consider the following rule:

tenant:%(owner)s

This will use the tenant value of the currently authenticated user. It will also use owner from the image it is acting upon. If those two values are equivalent the check will pass. All attributes on an image (as well as extra image properties) are available for use on the right side of the colon. The most useful are the following:

owner
protected
is_public

An operator, therefore, could construct a set of rules like the following:

{
    "not_protected": "False:%(protected)s",
    "is_owner": "tenant:%(owner)s",
    "not_protected_and_is_owner": "rule:not_protected and rule:is_owner",
    "delete_image": "rule:not_protected_and_is_owner"
}

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: icordasc

Reviewers

Core reviewer(s):: nikhil-komawar jokke
Other reviewer(s):: kragniz

Work Items

Create appropriate target classes
Use target classes to proxy targets to the policy enforcer
Add tests demonstrating that generic checks now work
Add better documentation surrounding policy rules to the existing documentation

Dependencies

None

Testing

Functional tests will be added where specific policy.json files are loaded to test access control of different targets.

Documentation Impact

There is no direct impact, but the existing policy.json documentation is thin and only describes what each rule controls. It does not describe the available target information or how to write rules.

References

Related bugs:

Initial work at an implementation:

https://review.openstack.org/#/c/146651/

Glance meeting discussion on 2015 January 15:

http://eavesdrop.openstack.org/meetings/glance/2015/glance.2015-01-15-14.02.log.html

Glance VMware store to support multiple datastores

Mon, 12 Jan 2015 00:00:00

https://blueprints.launchpad.net/glance/+spec/vmware-store-multiple-datastores

Glance, when configured to use VMware store, stores the images in a virtual storage container called datastore, identified by the configuration option vmware_datastore_name. The size of the datastore can only be increased to a certain limit which also depends upon the underlying physical storage.

The capacity of the store is thus bound by the size of a single datastore and poses serious scalability issues.

Problem description

Currently the number of images stored by glance, when configured to use VMware store, is limited by the capacity of a single datastore. The size of a datastore has an upper bound [1].

e.g The maximum datastore size is 64TB when backed by VMFS-5 filesystem. While the same for a datastore with NFS may depend on the storage array vendor. While this is the theoretical maximum based on the supported filesystem, datastores are also sized based on the underlying physical storage, including internal and external devices and networked storage. With networked storage the choice of storage protocol (iSCSI, FC, FCoE) also play a vital role in deciding the optimal size. Thus, the capacity of a datastore used in a datacenter varies by deployment.

Apart from the capacity, there may be a performance bottleneck when many compute nodes concurrently access a single datastore to download images.

Proposed change

To allow adding more capacity and improving performance, we suggest the use of multiple datastores for storing images.

There is only one major aspect of the proposed change: - Datastore Selection.

1) Datastore Selection:

This change proposes to select a datastore based on the priority given to it by the operator and the capacity to accommodate the image. A new configuration option (say vmware_datastores) will be added that would enable the operator to specify multiple datastores along with their relative weights. In case of equal priority the selection will be based on the maximum freespace available on the datastore. This approach is very similar to filesystem_store_datadirs used by filesystem store.

Example: say vmware_datastores is configured with nfs_datastore, iscsi_datastore, ssd_datastore with weights 100, 100 and 200 respectively. Here, ssd_datastore will be the preferred datastore if it can accommodate the image data being added. Otherwise, nfs_datastore or iscsi_datastore will be chosen based on maximum freespace available.

Alternatives

1) The datastore selection can be done through round-robin/randomized strategy instead of priority-based. This would be ideal if all datastores are of same type and perform similarly. But since datastores are of different types, the strategy is best driven by operator input. e.g SSD backed datastores may be faster that networked datastores. While its possible to determine the type of datastore, implicit assumptions cannot be made.

Data model impact

Does not have any impact on glance data model.

REST API impact

None

Security impact

This spec only adds a selection logic to the existing functionality and hence does not have any security impact.

Notifications impact

None

Other end user impact

None

Performance Impact

The use of multiple datastores will reduce concurrent operations on a single datastore.

Other deployer impact

This proposal will allow administrators to configure multiple datastores to save the image data in VMware storage backend. If enabled, it will override the existing configuration option vmware_datastore_name that allows to specify a single datastore. If disabled, the store will fallback to vmware_datastore_name option. Thus, the change is backwards compatible. We will deprecate the vmware_datastore_name option going forward.

New Multi Strings configuration option in glance-api.conf

vmware_datastores Optional. Default: Not set.

A datastore along with its datacenter path and a weight. If the datastore is a member of a datastore cluster, then the name of the cluster should also be included. Otherwise, the cluster name can be omitted. This option can be specified multiple times to specify multiple datastores. Thus, the required format becomes:

vmware_datastores =: <datacenter_path>:<datastore1>:<weight>
vmware_datastores =: <datacenter_path>:<datastore_cluster_name>:<datastore2>:<weight>

The weights are used to establish the relative priorities of the specified datastores. Thus, they can be any arbitrary integer values.

Examples:

vmware_datastores =: datacenter1:nfs_datastore:2
vmware_datastores =: datacenter1:ssd_datastore:3
vmware_datastores =: datacenter1:backup_datastore:1

In the above example, ssd_datastore will be given highest priority, followed by nfs_datastore and backup_datastore.

vmware_datastores =
datacenter1:nfs1_datastore:100

vmware_datastores =
datacenter1:nfs2_datastore:100

vmware_datastores =
datacenter1:backup_datastore:50

In this example, the nfs datastores will be given equal priority, followed by the backup_datastore. With equal priority, the contention between nfs datastores is resolved by the maximum freespace.

Note:- If the datacenter path or datastore name contains a colon (:) symbol, it must be escaped with a backslash.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: smurugesan (sabari)
Other contributors:: None

Reviewers

Core reviewer(s):: nikhil-komawar arnaudleg
Other reviewer(s):: rgerganov

Work Items

Implement new config option in VMware store driver.
Implement datastore selection logic in the VMware store driver.
Implement unit tests.
Change glance-api sample conf in glance repository.
Add a deprecation warning for vmware_datastore_name.
Update the documentation.

Dependencies

oslo.vmware has introduced a Datastore object which will be used in this implementation.
oslo.vmware has a vim_util module that has some low-level utility methods to interact with the vmware api’s. This is required to parse api responses.

Testing

Tempest tests are not required.

Documentation Impact

Document new configuration options.

References

[1] http://www.vmware.com/pdf/vsphere5/r55/vsphere-55-configuration-maximums.pdf

Create and Use `glance-manage` config file

Thu, 18 Dec 2014 00:00:00

https://blueprints.launchpad.net/glance/+spec/create-glance-manage-conf

glance-manage currently uses the configuration files meant for glance-registry and glance-api. This was ostensibly done to reduce the number of places that an operator may need to add, update, or remove settings. I would like to create a glance-manage.conf file to allow glance-manage to be independently configured.

Problem description

If glance-api is started by a user (or service) but a different user tries to use glance-manage they can encounter permissions errors if /var/log/glance/api.log is not writable to by the user trying to use glance-manage. While this is one symptom of the dependence on the registry and api configuration files, more may soon appear. Further, having separate configuration files will allow end users and deployers to configure the tools separately as well as use glance-manage on a node without requiring the presence of both glance-api.conf and glance-registry.conf.

Proposed change

I am proposing that we add another configuration file, glance-manage.conf to side-step this and any other issues we have with depending on the registry and api’s configuration files.

For Kilo, we will add the glance-manage.conf file and continue to load the glance-registry.conf and glance-api.conf files in the glance-manage command setup step. Currently the load order of configuration files (which causes glance-manage to use /var/log/glance/api.log) is:

glance-registry.conf
glance-api.conf

We will preserve this order and then load glance-manage.conf. We will only default to setting log_file in glance-manage.conf to prevent overriding settings from the other two files. We will also issue a deprecation warning pointing to this specification so that operators and end users know to configure glance-manage.conf for a later cycle. In that cycle, we will stop depending on glance-registry.conf and glance-api.conf. The documentation should also immediately, starting in the K cycle, begin to instruct users to configure settings for glance-manage in glance-manage.conf.

Alternatives

One way we could address this would be to remove the default log_file values in Glance’s configuration files. If we did this, all log files would then be named /var/log/{{service}}/{{ command }}.log, e.g., /var/log/glance/glance-api.log would be the file used by glance-api. Changing this would not only break current documentation but also end user expectations. Due to the considerable difference in behaviour without prior warning, we decided to take the approach outlined in this specification instead of using the same conventions as other projects.

Data model impact

None

REST API impact

None

Security impact

This would introduce another file that would need to be edited by end users. Some settings configured in glance-manage.conf may also be present in glance-registry.conf and glance-api.conf. Any potential security problems caused by needing to copy and synchronize settings between three files are applicable here. Alternatively, since glance-manage can be configured separately now, there will be no need to have the full API and registry configuration files on a node in order to run glance-manage.

Notifications impact

None

Other end user impact

This introduces another file with configuration options. Common configuration options for will need to be copied and pasted from file-to-file and will need attention to keep synchronized. This will likely increase the complexity of maintaining an installation of Glance.

Performance Impact

None

Other deployer impact

Deployers will be able to run glance-manage from nodes without needing glance-registry.conf or glance-api.conf to be present. This specification does introduce another file that deployers need to be aware of and know how to configure.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: icordasc
Other contributors:: None

Reviewers

Core reviewer(s):: nikhil-komawar
Other reviewer(s):: kragniz jokke_

Work Items

Generate a default glance-manage.conf as described above
Begin loading it in glance-manage
Add deprecation messages regarding glance-registry.conf and glance-api.conf.
Update the documentation to describe how to configure glance-manage with glance-manage.conf.

Dependencies

None

Testing

We can test this by ensuring that a separate log file is generated for glance-manage, i.e., /var/log/glance/manage.log is present after running the command.

Documentation Impact

This will require the changes to describe how to configure glance-manage to continue working as it has in the past.

References

https://bugs.launchpad.net/glance/+bug/1391211

Add capabilities to storage driver

Tue, 02 Dec 2014 00:00:00

https://blueprints.launchpad.net/glance/+spec/store-capabilities

This features will enable the static and dynamic ability of a storage driver instance based on it’s implementation as well as the configuration. The runtime status of the backend storage will influence the capacity too.

Henceforth, glance_store library can perform proper operations on storage as requested by upper layer, e.g. Glance. For example, to enable or disable image adding function to glance based on current implementation. The store also recognized whether a driver can be reused for all requests it handles or if it has to be recreated for each request because it is not stateless.

Problem description

The capabilities of store driver can be affected by the following factors:

Status of a connected backend storage.

The deployment configuration.

State of the driver code.

Technically one of capabilities is static whereas the other is dynamic.

Currently glance_store and glance don’t use these capabilities. The driver doesn’t have a way to expose its capabilities. On the other hand in the store’s backend handler or client, e.g. glance-{api,registry} there is no common method to check these generic operations of the store drivers.

Currently we observe at least two kind of problems:

Some necessary applicability check on the operation related with store access are missing, which inhibits the usability of glance service and glance_store lib.

Hard to implement such check logic neatly, including common check routine on these generic operations of store driver, and the check for particular feature.

Proposed change

Existing drivers need to be updated to expose proper capabilities.

List of required store capabilities as needed currently.

Capability Name

Description

READ_ACCESS

Generic read access

WRITE_ACCESS

Generic write access

RW_ACCESS

READ_ACCESS and WRITE_ACCESS

READ_OFFSET

Read all bits from a offset (Included READ_ACCESS)

WRITE_OFFSET

Write all bits to a offset (Included WRITE_ACCESS)

RW_OFFSET

READ_OFFSET and WRITE_OFFSET

READ_CHUNK

Read required length of bits (Included READ_ACCESS)

WRITE_CHUNK

Write required length of bits (Included WRITE_ACCESS)

RW_CHUNK

READ_CHUNK and WRITE_CHUNK

READ_RANDOM

READ_OFFSET and READ_CHUNK

WRITE_RANDOM

WRITE_OFFSET and WRITE_CHUNK

RW_RANDOM

RW_OFFSET and RW_CHUNK

DRIVER_REUSABLE

driver is stateless and its instance can be reused safely

Add common check routine on these generic operations for the store drivers.
Refactoring existing drivers to leverage these capabilities.
Add the logic to recreate driver instance if the storage or driver isn’t stateless.

Capability Name	Description
READ_ACCESS	Generic read access
WRITE_ACCESS	Generic write access
RW_ACCESS	READ_ACCESS and WRITE_ACCESS
READ_OFFSET	Read all bits from a offset (Included READ_ACCESS)
WRITE_OFFSET	Write all bits to a offset (Included WRITE_ACCESS)
RW_OFFSET	READ_OFFSET and WRITE_OFFSET
READ_CHUNK	Read required length of bits (Included READ_ACCESS)
WRITE_CHUNK	Write required length of bits (Included WRITE_ACCESS)
RW_CHUNK	READ_CHUNK and WRITE_CHUNK
READ_RANDOM	READ_OFFSET and READ_CHUNK
WRITE_RANDOM	WRITE_OFFSET and WRITE_CHUNK
RW_RANDOM	RW_OFFSET and RW_CHUNK
DRIVER_REUSABLE	driver is stateless and its instance can be reused safely

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

There is no expectation for any obvious degradation in the performance.

The reason being that there is a simple addition of a checker function being triggered in the form of a hook for each store operation, with current inclusion of the (‘get’, ‘delete’, ‘add’) operations.

Other deployer impact

None.

Developer impact

Developers implementing new drivers to the glance_store library would need to be aware of this concept. The static and dynamic ability of the storage backend could influence their and their implementation design.

All changes maintain backward-compatibility.

Implementation

Assignee(s)

Primary assignee:: zhiyan (lzy-dev)

Reviewers

Core reviewer(s):: Nikhil Komawar (nikhil-komawar) Stuart McLaren (stuart-mcLaren)

Work Items

Dependencies

None

Testing

Necessary unit and functional test cases, will be added into glance_store as well as glance.

Documentation Impact

None

References

Corresponding changes:

Catalog Index Service

Mon, 01 Dec 2014 00:00:00

https://blueprints.launchpad.net/glance/+spec/catalog-index-service

This is intended to improve performance of Glance API services while dramatically improving search capabilities.

It will improve performance by offloading user search queries from existing API servers. In addition, we are working on numerous improvements in Horizon which will include improvements to image, snapshot, artifact details and searching. The desired user experience is greatly dependent upon a rich, dynamic, near real time faceted and aggregated search capability with a strong query language.

This will initially be considered “Experimental API”. The API is intended to be as close to final as possible, but we reserve the right to completely abandon this or change it making it backwards in-compatible.

Problem description

Glance has metadata for all images and provides listing of them. If you want to search for images based on criteria, the API is limited and somewhat inflexible. Search currently is limited to union (“AND” operations) searching of certain hard coded attributes. There is no support for intersection ( “OR” operations).

Searching does not include searching descriptions and has limited support for property based searches. Full text searching of descriptions is typically slow with a traditional relational database. Adding full text indexing to the database can degrade overall performance of the database (affects inserts).

With the addition of metadata definitions in Juno release, the new search API should allow the users to specify the search criteria using the tags and property type definition in addition to adhoc string search. They should ideally get auto-completion of possible properties and property values with near real time response.

Artifact definitions will support storing dynamic properties based on the artifact type. However, proper indexing and search is not possible with a traditional RDBMS. You end up with full table scans across tables that are not applicable for a given query. In addition, writing and understanding the query engine is difficult to understand and maintain, especially after the original authors move on.

Adding new properties with constraints for highly performant searching is difficult to achieve in a relational database.

A search engine needs to be easily customizable so that the data being collected can be changed dynamically and the way it is indexed and searched can be easily modified without requiring source data migration. (e.g. a new type of object to group namespaces together called “category”)

Typical search interfaces should also provide facilities like auto-completion and search suggestions with near real time performance. This is not possible with a traditional database.

Adding more attributes to a search query over time should be easy to extend and maintain without exponentially increasing the complexity of the search logic.

User should be able to combine the search query across resources (images, metadata, artifacts) easily.

A search engine should not put additional load on the normal functions of the primary service and should easily accommodate more users by distributing the load on separate server instances. For example, a search query from Horizon should not impact Nova.

Proposed change

We are proposing a new catalog search service for Glance that will improve performance of Glance API services while dramatically improving search capabilities. The following subsections detail the concepts.

Block Diagram:

https://wiki.openstack.org/w/images/7/74/Index-service-block-diagram.png

The service will be based on Elasticsearch. Elasticsearch is a search server based on Lucene. It provides a distributed, scalable, near real-time, faceted, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. Elasticsearch is developed and released as open source under the terms of the Apache License. Notable users of Elasticsearch include Wikimedia, StumbleUpon, Mozilla, Quora, Foursquare, Etsy, SoundCloud, GitHub, FDA, CERN, and Stack Exchange. (Source: http://en.wikipedia.org/wiki/Elasticsearch)

The elastic-recheck project also uses Elasticsearch (and kibana) to classify and track OpenStack gate failures. (Source: http://status.openstack.org/elastic-recheck)

Indexing

Index

This will serve as the cache for all search requests. It will be backed by Elasticsearch.

Index Loaders

Index loaders define the data mappings for indexing and load the data from the source. They are called during initialization of service and on-demand later when required to index everything. They ensure that all appropriate RBAC information is included in the index to facilitate appropriate authorization on search responses.

The index loaders will attempt to maintain native API format as best as possible with as much direct pass through as possible so that data manipulation and maintenance is kept to a minimum.

An example Glance Image data mapping would be:

{
  'dynamic': True,
  'properties': {
      'id': {'type': 'string'},
      'name': {'type': 'string'},
      'description': {'type': 'string'},
      'tags': {'type': 'string'},
      'disk_format': {'type': 'string'},
      'container_format': {'type': 'string'},
      'size': {'type': 'long'},
      'virtual_size': {'type': 'long'},
      'status': {'type': 'string'},
      'visibility': {'type': 'string'},
      'checksum': {'type': 'string'},
      'min_disk': {'type': 'long'},
      'min_ram': {'type': 'long'},
      'owner': {'type': 'string'},
      'protected': {'type': 'boolean'},
  },
}

Index Updates

Once the index is initialized it needs to be constantly updated to keep it in sync with the data source. Update clients will listen for notifications from data sources to re-index the data for specific resources (e.g. an image or artifact).

For glance, it would listen on message Topic for notifications like (image.create, image.update, etc) and reindex for the effected image metadata. (More info at http://docs.openstack.org/developer/glance/notifications.html)

Index Management API

Allows for CRUD management of loading, updating and deleting data in the index. Indexing is allowed only for admin users.

Default policy.json will be:

{
  "catalog_index": "role:admin",
  "catalog_search": ""
}

Searching

The search API allows users to execute a search query and get back search hits that match the query. The query can either be provided using a simple query string as a parameter, or using a request body.

Note

Search query is not parsed and passed “as-is” to elastic search engine except for adding filters. Response from search engine could be filtered based on the plugin implementation of document type.

All search APIs can be applied across multiple types within an index, and across multiple indices with support for the multi index syntax.

This will allow for search phrase completion as well as search suggestions( such as handling misspellings)

The search will have two levels of RBAC.

1. API level policy checks using policy.json files. This will allow coarse grained RBAC support for simple deny / allow on API usage.

2. RBAC query filters. These will be defined in conjunction with index loaders. When a request comes in, the type(s) of resource(s) being requested will map to an RBAC query filter.

The RBAC query filter will add any appropriate filters to the request being sent into the elastic search service, such that only specific results that the user is allowed to view will be returned.

For example, the image index loader will include indexing owner information and visibility information. The RBAC filter will examine the incoming request and adds filters to the request so that the results don’t include non-shared / non-public images from a different project than the user making the request.

Property protected fields will be read from the config file and will be added as “source filtering” field(s) in elasticsearch query which will keep/remove the protected fields from the search output based on the authorization of the user.

Alternatives

Searching data could also be achieved by writing SQL queries on the Glance database but there are several factors which do not make it an ideal solution:

Joins across multiple tables in real time will make the response time very slow
Full text searching of descriptions is typically slow with a traditional relational database. Adding full text indexing to the database can degrade overall performance of the database
Property types can be added dynamically using metadefs and proper indexing in relational databases is not possible
Search queries will be running against the same database used by Glance core functions and inadvertently effecting their response time.
Adding more attributes to search query over time should be easy to extend and maintain without exponentially increasing the complexity of the search logic

User should be able to combine the search query across resources (images, artifacts) etc. and the search engine should not be tightly integrated with any specific module.

Another alternative would be for clients to load the entire data set and search within the client. This means every user gets all the data every time the user loads the page and has to keep it in sync with server side data. This is increases the load and burden on the core OpenStack service providing the data and is slower since the client has to load the entire dataset across the network. In addition, the client has to recreate the logic for things like search suggestions and complex queries with AND / OR logic.

It should be noted that NONE of these options also include an ability to do things like get search request scoring of results returned with a configurable threshold for results (something elastic search provides).

Data model impact

The data being indexed will be stored outside the Glance SQL database and therefore we don’t expect any data model changes in Glance.

REST API impact

Common Response Codes

Create Success: 201 Created
Modify Success: 200 OK
Delete Success: 204 No Content
Failure: 400 Bad Request with details.
Forbidden: 403 Forbidden
Not found: 404 Not found e.g. if specific entity not found
Method Not Allowed: 405 Not allowed e.g. if trying to delete on a list resource
Not Implemented: 501 Not Implemented e.g. HEAD not implemented

This is an experimental API

API Version

Search images supports both GET and POST. Elasticsearch supports GET with query params but its a limited subset of query DSL. GET is implemented here with a request body to make use of all the available query options

Please refer to the following URI for the Query DSL http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html

Search images(GET):

GET /v2/search

Example Request Body:

{
  "index": ["glance"],
  "type": ["image"],
  "query": {
      "query_string": {
          "query": "cirros"
      }
  }
}

Example Response Body:

{
  "took": 5,
  "timed_out": false,
  "_shards": {
      "total": 10,
      "successful": 10,
      "failed": 0
  },
  "hits": {
      "total": 3,
      "max_score": 0.40409642,
      "hits": [
          {
              "_index": "search",
              "_type": "image",
              "_id": "75fbdd4c-3e5b-4552-8950-9bb5262babcd",
              "_score": 0.40409642,
              "_source": {
                  "status": "active",
                  "virtual_size": null,
                  "name": "cirros-0.3.2-x86_64-uec-ramdisk",
                  "property": [],
                  "container_format": "ari",
                  "min_ram": 0,
                  "disk_format": "ari",
                  "properties": [],
                  "owner": "f72690e85b2a4ff095f50b7fad99429a",
                  "protected": false,
                  "checksum": "68085af2609d03e51c7662395b5b6e4b",
                  "min_disk": 0,
                  "is_public": true,
                  "size": 3723817,
                  "id": "75fbdd4c-3e5b-4552-8950-9bb5262babcd",
                  "description": ""
              }
          },
          {
              "_index": "search",
              "_type": "image",
              "_id": "95467ea8-dd34-4bdd-8a6a-f52e47ee9bce",
              "_score": 0.23091224,
              "_source": {
                  "status": "active",
                  "virtual_size": null,
                  "name": "cirros-0.3.2-x86_64-uec",
                  "property": [
                      "kernel_id_d00ea383-a1fa-48d3-b56c-880093730b53",
                      "ramdisk_id_75fbdd4c-3e5b-4552-8950-9bb5262babcd",
                      "hypervisor_type_uml",
                      "hw_watchdog_action_poweroff"
                  ],
                  "container_format": "ami",
                  "min_ram": 0,
                  "disk_format": "ami",
                  "properties": [
                     {
                          "name": "kernel_id",
                          "value": "d00ea383-a1fa-48d3-b56c-880093730b53"
                      },
                      {
                          "name": "ramdisk_id",
                          "value": "75fbdd4c-3e5b-4552-8950-9bb5262babcd"
                      },
                      {
                          "name": "hypervisor_type",
                          "value": "uml"
                      },
                      {
                          "name": "hw_watchdog_action",
                          "value": "poweroff"
                      }
                  ],
                  "owner": "f72690e85b2a4ff095f50b7fad99429a",
                  "protected": false,
                  "checksum": "4eada48c2843d2a262c814ddc92ecf2c",
                  "min_disk": 0,
                  "is_public": true,
                  "size": 25165824,
                  "id": "95467ea8-dd34-4bdd-8a6a-f52e47ee9bce",
                  "description": ""
              }
          },
          {
              "_index": "search",
              "_type": "image",
              "_id": "d00ea383-a1fa-48d3-b56c-880093730b53",
              "_score": 0.067124054,
              "_source": {
                  "status": "active",
                  "virtual_size": null,
                  "name": "cirros-0.3.2-x86_64-uec-kernel",
                  "property": [],
                  "container_format": "aki",
                  "min_ram": 0,
                  "disk_format": "aki",
                  "properties": [],
                  "owner": "f72690e85b2a4ff095f50b7fad99429a",
                  "protected": false,
                  "checksum": "836c69cbcd1dc4f225daedbab6edc7c7",
                  "min_disk": 0,
                  "is_public": true,
                  "size": 4969360,
                  "id": "d00ea383-a1fa-48d3-b56c-880093730b53",
                  "description": ""
              }
          }
      ]
  }
}

Search images(POST):

POST /v2/search

Please refer to the following URI for the Query DSL http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html

Example Request Body:

{
  "index": ["glance"],
  "type": ["image"],
  "query": {
      "query_string": {
          "query": "cirros"
      }
  }
}

Example Response Body:

{
  "took": 5,
  "timed_out": false,
  "_shards": {
      "total": 10,
      "successful": 10,
      "failed": 0
  },
  "hits": {
      "total": 3,
      "max_score": 0.40409642,
      "hits": [
          {
              "_index": "search",
              "_type": "image",
              "_id": "75fbdd4c-3e5b-4552-8950-9bb5262babcd",
              "_score": 0.40409642,
              "_source": {
                  "status": "active",
                  "virtual_size": null,
                  "name": "cirros-0.3.2-x86_64-uec-ramdisk",
                  "property": [],
                  "container_format": "ari",
                  "min_ram": 0,
                  "disk_format": "ari",
                  "properties": [],
                  "owner": "f72690e85b2a4ff095f50b7fad99429a",
                  "protected": false,
                  "checksum": "68085af2609d03e51c7662395b5b6e4b",
                  "min_disk": 0,
                  "is_public": true,
                  "size": 3723817,
                  "id": "75fbdd4c-3e5b-4552-8950-9bb5262babcd",
                  "description": ""
              }
          },
          {
              "_index": "search",
              "_type": "image",
              "_id": "95467ea8-dd34-4bdd-8a6a-f52e47ee9bce",
              "_score": 0.23091224,
              "_source": {
                  "status": "active",
                  "virtual_size": null,
                  "name": "cirros-0.3.2-x86_64-uec",
                  "property": [
                      "kernel_id_d00ea383-a1fa-48d3-b56c-880093730b53",
                      "ramdisk_id_75fbdd4c-3e5b-4552-8950-9bb5262babcd",
                      "hypervisor_type_uml",
                      "hw_watchdog_action_poweroff"
                  ],
                  "container_format": "ami",
                  "min_ram": 0,
                  "disk_format": "ami",
                  "properties": [
                     {
                          "name": "kernel_id",
                          "value": "d00ea383-a1fa-48d3-b56c-880093730b53"
                      },
                      {
                          "name": "ramdisk_id",
                          "value": "75fbdd4c-3e5b-4552-8950-9bb5262babcd"
                      },
                      {
                          "name": "hypervisor_type",
                          "value": "uml"
                      },
                      {
                          "name": "hw_watchdog_action",
                          "value": "poweroff"
                      }
                  ],
                  "owner": "f72690e85b2a4ff095f50b7fad99429a",
                  "protected": false,
                  "checksum": "4eada48c2843d2a262c814ddc92ecf2c",
                  "min_disk": 0,
                  "is_public": true,
                  "size": 25165824,
                  "id": "95467ea8-dd34-4bdd-8a6a-f52e47ee9bce",
                  "description": ""
              }
          },
          {
              "_index": "search",
              "_type": "image",
              "_id": "d00ea383-a1fa-48d3-b56c-880093730b53",
              "_score": 0.067124054,
              "_source": {
                  "status": "active",
                  "virtual_size": null,
                  "name": "cirros-0.3.2-x86_64-uec-kernel",
                  "property": [],
                  "container_format": "aki",
                  "min_ram": 0,
                  "disk_format": "aki",
                  "properties": [],
                  "owner": "f72690e85b2a4ff095f50b7fad99429a",
                  "protected": false,
                  "checksum": "836c69cbcd1dc4f225daedbab6edc7c7",
                  "min_disk": 0,
                  "is_public": true,
                  "size": 4969360,
                  "id": "d00ea383-a1fa-48d3-b56c-880093730b53",
                  "description": ""
              }
          }
      ]
  }
}

Index images: index, create, update and delete data:

POST /v2/index

Indexing is allowed only for admin users. Supported actions are index, create, update and delete

Example Request Body:

{
  "default_index": "search",
  "default_type": "image",
  "actions": [
      {
          "action": "create",
          "index": "search",
          "type": "image",
          "id": "d00ea383-a1fa-48d3-b56c-880093730b54",
          "data": {
              "status": "active",
              "virtual_size": null,
              "name": "cirros-0.3.3-x86_64-uec-kernel",
              "property": [],
              "container_format": "aki",
              "min_ram": 0,
              "disk_format": "aki",
              "properties": [],
              "owner": "f72690e85b2a4ff095f50b7fad99429a",
              "protected": false,
              "checksum": "836c69cbcd1dc4f225daedbab6edc7c7",
              "min_disk": 0,
              "is_public": false,
              "size": 4969360,
              "id": "d00ea383-a1fa-48d3-b56c-880093730b54",
              "description": ""
          }
      },
      {
          "action": "update",
          "index": "search",
          "type": "image",
          "id": "75fbdd4c-3e5b-4552-8950-9bb5262babcd",
          "data": {
              "name": "cirros x86",
              "status": "inactive"
          }
      },
      {
          "action": "delete",
          "index": "search",
          "type": "image",
          "id": "95467ea8-dd34-4bdd-8a6a-f52e47ee9bce"
      }
  ]
}

Security impact

None to existing Glance API. Search queries will apply filters to return data that the user is authorized to see. See description.

Notifications impact

None to existing notifications. Will only consume notifications Need to add metadef notifications to Glance service.

Other end user impact

Update python-glanceclient as needed

Performance Impact

No changes to existing API or code Data from Glance DB will read once during initialization to index it inside search engine.

This is intended to improve performance of Glance API services while dramatically improving search capabilities. It will improve performance by offloading user search queries from existing API servers.

Other deployer impact

Glance Catalog Index service will be installed as a separate service with its own port and endpoint.

glance-manage will have new commands for indexing image, metadef, and artifact data

The deployment will be targeted as a single region service. In future if required an “Aggregate search” of all regions which can search across all the regions could be provided.

Developer impact

These are new API’s and will not impact any existing API’s.

Implementation

Assignee(s)

Primary assignee:: lakshmi-sampath, kamil-rykowski,
Other contributors:: wayne-okuma, travis-tripp

Reviewers

Core reviewer(s):: nikhil-komawar zhiyan
Other reviewer(s):: icordasc

Work Items

Installation of Elastic Search in Glance environment (single node)
Index Dictionary data in ElasticSearch
- Write a tool to index all metadata objects (namespaces objects, properties) from database into elasticsearch
- Write a tool to index all images from database into elasticsearch
- Merge used properties from Glance(optional)
- Listen to notifications/events form Glance on Image CRUD(optional) for continuous indexing of new/old data
Create Glance Search API - Interface to backend ElasticSearch
- Make Policy checks on requests
- Filter request based on RBAC with user token
Search Images
- List all the results by given search query string
Create Glance Index API
- Policy checks
Discuss with Openstack/Infra
- Test environment for elasticsearch
Devstack integration of single node elastic search.
Metadef notifications
- Generate and Listen to metadef notifications
Calls the tools (loaders)
Documentation update
Update glance client
Update glance manage

Dependencies

Depends on elasticsearch for search engine

Testing

Unit tests will be added for all possible code with a goal of being able to isolate functionality as much as possible.

Tempest tests will be added wherever possible.

Documentation Impact

Docs needed for new service and usage.

All document changes will indicate this as “Experimental API”

References

Elasticsearch Query DSL http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html

Delete an Image

Fri, 17 Oct 2014 00:00:00

DELETE /v2/images/<IMAGE_ID>

Encode the ID of the image into the request URI. Request body is ignored.

Images with the ‘protected’ attribute set to true (boolean) cannot be deleted and the response will have an HTTP 403 status code. You must first set the ‘protected’ attribute to false (boolean) and then perform the delete.

The response will be empty with an HTTP 204 status code.

Image API v2 HTTP PATCH media types

Fri, 17 Oct 2014 00:00:00

Overview

The HTTP PATCH request must provide a media type for the server to determine how the patch should be applied to an image resource. An unsupported media type will result in an HTTP error response with the 415 status code. For image resources, two media types are supported:

application/openstack-images-v2.1-json-patch
application/openstack-images-v2.0-json-patch

The application/openstack-images-v2.1-json-patch media type is intended to provide a useful and compatible subset of the functionality defined in JavaScript Object Notation (JSON) Patch RFC6902, which defines the application/json-patch+json media type.

The application/openstack-images-v2.0-json-patch media type is based on draft 4 of the standard. Its use is deprecated.

Restricted JSON pointers

The ‘application/openstack-images-v2.1-json-patch’ media type defined in this appendix adopts a restricted form of JSON-Pointers. A restricted JSON pointer is a Unicode string containing a sequence of exactly one reference token, prefixed by a ‘/’ (%x2F) character.

If a reference token contains ‘~’ (%x7E) or ‘/’ (%x2F) characters, they must be encoded as ‘~0’ and ‘~1’ respectively.

Its ABNF syntax is:

restricted-json-pointer = "/" reference-token
reference-token = *( unescaped / escaped )
unescaped = %x00-2E / %x30-7D / %x7F-10FFFF
escaped = "~" ( "0" / "1" )

Restricted JSON Pointers are evaluated as ordinary JSON pointers per JSON-Pointer.

For example, given the image entity:

{
    "id": "da3b75d9-3f4a-40e7-8a2c-bfab23927dea",
    "name": "cirros-0.3.0-x86_64-uec-ramdisk",
    "status": "active",
    "visibility": "public",
    "size": 2254249,
    "checksum": "2cec138d7dae2aa59038ef8c9aec2390",
    "~/.ssh/": "present",
    "tags": ["ping", "pong"],
    "created_at": "2012-08-10T19:23:50Z",
    "updated_at": "2012-08-10T19:23:50Z",
    "self": "/v2/images/da3b75d9-3f4a-40e7-8a2c-bfab23927dea",
    "file": "/v2/images/da3b75d9-3f4a-40e7-8a2c-bfab23927dea/file",
    "schema": "/v2/schemas/image"
}

The following restricted JSON pointers evaluate to the accompanying values:

"/name"        "cirros-0.3.0-x86_64-uec-ramdisk"
"/size"        2254249
"/tags"        ["ping", "pong"]
"/~0~1.ssh~1"  "present"

Operations

The application/openstack-images-v2.1-json-patch media type supports a subset of the operations defined in the application/json-patch+json media type. Specify the operation in the “op” member of the request object.

The supported operations are add, remove, and replace.
If an operation object contains no recognized operation member, an error occurs.

Specify the location where the requested operation is to be performed in the target image in the “path” member of the operation object.

The member value is a string containing a restricted JSON-pointer value that references the location where the operation is to be performed within the target image.

Where appropriate (that is, for the “add” and “replace” operations), the operation object must contain the “value” data member.

The member value is the actual value to add (or to use in the replace operation) expressed in JSON notation. (For example, strings must be quoted, numeric values are unquoted.)

The payload for a PATCH request must be a list of JSON objects. Each object must adhere to one of the following formats.

The add operation adds a new value at a specified location in the target image. The location must reference an image property to add to an existing image.

The operation object specifies a “value” member and associated value.

Example:

PATCH /images/{image_id}

[
   {
      "op":"add",
      "path":"/login-name",
      "value":"kvothe"
   }
]

You can also use the add operation to add a location to the set of locations that are associated with a specified image ID, as follows:

Example:

PATCH /images/{image_id}

[
   {
      "op":"add",
      "path":"/locations/1",
      "value":"scheme4://path4"
   }
]

remove

The remove operation removes the specified image property in the target image. If an image property does not exist at the specified location, an error occurs.

Example:

PATCH /images/{image_id}

[
   {
      "op":"remove",
      "path":"/login-name"
   }
]

You can also use the remove operation to remove a location from a set of locations that are associated with a specified image ID, as follows:

Example:

PATCH /images/{image_id}

[
   {
     "op":"remove",
     "path":"/locations/2"
   }
]

replace

The replace operation replaces the value of a specified image property in the target image with a new value. The operation object contains a “value” member that specifies the replacement value.

Example:

[
   {
      "op":"replace",
      "path":"/login-name",
      "value":"kote"
   }
]

This operation is functionally identical to expressing a “remove” operation for an image property, followed immediately by an “add” operation at the same location with the replacement value.

If the specified image property does not exist for the target image, an error occurs.

You can also use the replace operation to replace a location in the set of locations that are associated with a specified image ID, as follows:

Example:

PATCH /images/{image_id}

[
   {
      "op":"replace",
      "path":"/locations",
      "value":[
         "scheme5://path5",
        "scheme6://path6"
      ]
  }
]

General Image API v2.x information

Fri, 17 Oct 2014 00:00:00

The Image Service API v2 enables you to store and retrieve disk and server images.

Versioning

Two-part versioning scheme

The Image Service API v2 mimics API v1 and uses a major version and a minor version. For example, v2.3 is major version 2 and minor version 3.

Backwards-compatibility

Minor version releases expand and do not reduce the interface. For example, everything in v2.1 is available in v2.2.

Property protections

The Images API v2.2 enables a cloud provider to employ property protections, an optional feature whereby CRUD protections are applied to image properties.

Thus, in particular deployments, non-admin users might not be able to view, update, or delete some image properties.

Additionally, non-admin users might be forced to follow a particular naming convention when creating custom image properties.

It is left to the cloud provider to communicate policies concerning property protections to users.

HTTP response status codes

The following HTTP status codes are all valid responses:

200 - generic successful response, expect a body
201 - entity created, expect a body and a Location header
204 - successful response without body
301 - redirection
400 - invalid request (syntax, value, etc)
401 - unauthenticated client
403 - authenticated client unable to perform action
409 - that action is impossible due to some (possibly permanent) circumstance
415 - unsupported media type

Responses that don’t have a 200-level response code are not guaranteed to have a body. If a response does happen to return a body, it is not part of this spec and cannot be depended upon.

Authentication and authorization

This spec does not govern how one might authenticate or authorize clients of the v2 Images API. Implementors are free to decide how to identify clients and what authorization rules to apply.

Note that the HTTP 401 and 403 status codes are included in this specification as valid response codes.

Request and response content format

The Images Service API v2 primarily accepts and serves JSON-encoded data. In certain cases it also accepts and serves binary image data. Most requests that send JSON-encoded data must have the proper media type in their Content-Type header: ‘application/json’. HTTP PATCH requests must use the patch media type defined for the entity they intend to modify. Requests that upload image data should use the media type ‘application/octet-stream’.

Each call only responds in one format, so clients should not worry about sending an Accept header. It is ignored. The response is formatted as ‘application/json’ unless otherwise stated in this spec.

Image entities

An image entity is represented by a JSON-encoded data structure and its raw binary data.

An image entity has an identifier (ID) that is guaranteed to be unique within the endpoint to which it belongs. The ID is used as a token in request URIs to interact with that specific image.

An image is always guaranteed to have the following attributes: id, status, visibility, protected, tags, created_at, file and self. The other attributes defined in the image schema below are guaranteed to be defined, but is only returned with an image entity if they have been explicitly set.

A client may set arbitrarily-named attributes on their images if the image json-schema allows it. These user-defined attributes appear like any other image attributes. See documentation of the additionalProperties json-schema attribute.

JSON schemas

The necessary json-schema documents are provided at predictable URIs. A consumer should be able to validate server responses and client requests based on the published schemas. The schemas contained in this document are only examples and should not be used to validate your requests. A client should always fetch schemas from the server.

Property Protections

Version 2.2 of the Images API acknowledges the ability of a cloud provider to employ property protections. Thus, there may be image properties that will not appear in the list images response for non-admin users.

Image API Binary Data API calls

Fri, 17 Oct 2014 00:00:00

The following API calls are used to upload and download raw image data. For image metadata, see Metadata API.

Upload Image File

PUT /v2/images/<IMAGE_ID>/file

NOTE: An image record must exist before a client can store binary image data with it.

Request Content-Type must be ‘application/octet-stream’. Complete contents of request body will be stored and become accessible in its entirety by issuing a GET request to the same URI.

Response status will be 204.

Download Image File

GET /v2/images/<IMAGE_ID>/file

Request body ignored.

Response body will be the raw binary data that represents the actual virtual disk. The Content-Type header will be ‘application/octet-stream’.

The Content-MD5 header will contain an MD5 checksum of the image data. Clients are encouraged to verify the integrity of the image data they receive using this checksum.

If no image data has been stored, an HTTP status of 204 is returned.

Get an Image

Fri, 17 Oct 2014 00:00:00

GET /v2/images/<IMAGE_ID>

Request body ignored.

Response body is a single image entity. Using GET /v2/image/da3b75d9-3f4a-40e7-8a2c-bfab23927dea as an example:

{
    "id": "da3b75d9-3f4a-40e7-8a2c-bfab23927dea",
    "name": "cirros-0.3.0-x86_64-uec-ramdisk",
    "status": "active",
    "visibility": "public",
    "size": 2254249,
    "checksum": "2cec138d7dae2aa59038ef8c9aec2390",
    "tags": ["ping", "pong"],
    "created_at": "2012-08-10T19:23:50Z",
    "updated_at": "2012-08-10T19:23:50Z",
    "self": "/v2/images/da3b75d9-3f4a-40e7-8a2c-bfab23927dea",
    "file": "/v2/images/da3b75d9-3f4a-40e7-8a2c-bfab23927dea/file",
    "schema": "/v2/schemas/image"
}

Property Protections

Version 2.2 of the Images API acknowledges the ability of a cloud provider to employ property protections. Thus, there may be some image properties that will not appear in the image detail response for non-admin users.

Tag Catalog Support For Metadata Definitions

Mon, 13 Oct 2014 00:00:00

https://blueprints.launchpad.net/glance/+spec/metadefs-tags

This blueprint adds the basic tag catalog back after it was deferred from Juno due to time constraints. The original, approved spec included supporting tag libraries in the metadata definitions catalog. However, at the end of Juno, the original spec was updated to remove tags since they weren’t implemented.

This spec is actually a reduction of the tag concepts in the original approved Juno spec. In the original Juno spec, tags had a dynamic hierarchy capability. This spec does not include the tag hierarchy in order to simplify this spec. That aspect of tags will be deferred to a later spec.

The implemented juno spec is here for reference:

https://github.com/openstack/glance-specs/blob/master/specs/juno/metadata-schema-catalog.rst

Problem description

A challenge with using OpenStack is discovering, sharing, and correlating tags across services and different types of resources. We believe this affects both end users and administrators.

For example, a cloud operator or vendor may have a predefined set of “tags” they want to be used as a starting point for images and instances. Currently, OpenStack does not have a facility for the cloud operator to include that base set of tags. This means that every deployment and every project may end up with its own disparate set of tags. This leads to inconsistencies, but also is extra hassle for end users who end up reinventing all the “tags” in every project.

For example, is the tag “postgres” the same as “PostgreSQL”? If a base library of tags is used, any user typing “pos” would be prompted with the one that already exists in the tag library and would choose it. Future searches based on tags would ensure consistent results.

Terminology

The term metadata can become very overloaded and confusing. This proposed enhancement is about the additional metadata that is set as “tags” (name only) across various artifacts and OpenStack services.

Different APIs may use tags and key / value pairs differently. Tags typically are not used to drive runtime behavior. However, key / value pairs are often used by the system to potentially drive runtime, such as scheduling, quality of service, or driver behavior.

A few examples of metadata today:

Nova	Cinder	Glance
Flavor extra specs Host Aggregate metadata Instances metadata tags	Volume & Snapshot image metadata metadata VolumeType extra specs qos specs	Image & Snapshot properties tags

Proposed change

We are proposing enhancements to the Metadata Definitions Catalog. The following subsections detail the enhancements to the catalog.

Tags

A catalog of possible tags that can be used to help ensure tag name consistency across users, resource types, and services. So, when a user goes to apply a tag on a resource, they will be able to either create new tags or choose from tags that have been used elsewhere in the system on different types of resources. For example, the same tag could be used for Images, Volumes, and Instances. Tags are not case sensitive (BigData is equivalent to bigdata but is different from Big-Data).

Alternatives

A key use case is the collaboration on tags using a common catalog. This is complementary to tags being added ad-hoc across all services. We think the metadata API could also be backed by a search indexer across services to include ad-hoc metadata as well as defined metadata. However, that is not the focus of this blueprint.

Data model impact

This will use a relational database and exist in the same database as the existing Glance Metadata Definitions Catalog. It will be additive to the existing schema.

The following DB schema is the initial suggested schema. We will improve and take comments during code review. Constraints not shown for readability.

Suggested Basic Schema:

CREATE TABLE `metadef_tags` (
  `id`                     int(11) NOT NULL AUTO_INCREMENT,
  `namespace_id`           int(11) NOT NULL,
  `name`                   varchar(80) NOT NULL,
  `created_at`             timestamp NOT NULL,
  `updated_at`             timestamp
)

This will not include Tag descriptions in this revision.

REST API impact

In the REST API everything is referred by namespace and name rather than synthetic IDs. This helps to achieve portability (import / export using JSON).

APIs should allow coarse grain and fine grain access to information in order to control data transfer bandwidth requirements.

Working with Namespaces Basic interaction is:

Get list of namespaces with overview info based on the desired filters.

Get tags

Common Response Codes

Create Success: 201 Created
Modify Success: 200 OK
Delete Success: 204 No Content
Failure: 400 Bad Request with details.
Forbidden: 403 Forbidden
Not found: 404 Not found if specific entity not found

API Version

All URLS will be under the v2 Glance API. If it is not explicitly specified assume /v2/<url>

Namespace may optionally contain the following in addition to basic fields.

resource_types
properties
objects
tags

This spec adds Tags.

Tags

List All Tags in a namespace:: GET /metadefs/namespace/{namespace}/tags

Filters by adding query parameters:

limit          = Use to request a specific page size. Expect a response
                 to a limited request to return between zero and limit items.
marker         = Specifies the name of the last-seen tag.
                 The typical pattern of limit and marker is to make an initial
                 limited request and then to use the last tag from the
                 response as the marker parameter in a subsequent limited
                 request.

Design note: We want a format that allows for additional information such as description to be added without changing the base response. For this reason, we used a dictionary for each tag rather than just a flat list of tags.

Example Body:

{
    "tags": [
        {
            "name": "Databases"
        },
        {
            "name": "BigData"
        },
        {
            "name": "MySQL",
        },
        {
            "name": "PostgreSQL",
        },
        {
            "name": "MongoDB",
        }
    ]
}

Create / Replace all tags in a specific namespace:: POST /metadefs/namespaces/{namespace}/tags/
Add tag in a specific namespace:: POST /metadefs/namespaces/{namespace}/tags/{tag}
Delete tag in specific namespace:: DELETE /metadefs/namespaces/{namespace}/tags/{tag}
Delete all tags in specific namespace:: DELETE /metadefs/namespaces/{namespace}/tags

Security impact

None

Notifications impact

None

Other end user impact

We intend to expose this via Horizon and are working on related blueprints.

Update python-glanceclient as needed.

Performance Impact

None anticipated.

This is expected to be called from Horizon when an admin wants to annotate tags onto things likes images and instances. This API would be hit for them to get available tags or create new ones.

Other deployer impact

DB Schema Creation for new API

Default / Sample tag libraries will be checked into Glance.

Deployers can customize these and provide additional definition files suitable to their cloud deployment.

glance-manage will include loading tags.

Developer impact

None (New API)

Implementation

Assignee(s)

Primary assignee:: wayne-okuma
Other contributors:: None

Reviewers

Core reviewer(s):: zhiyan
Other reviewer(s):: lakshmi-sampath travis-tripp

Work Items

Changes would be made to:

The database API layer to add support for CRUD operations on tags

The REST API for CRUD operations on the namespaces (add tags)

The REST API for CRUD operations on the tags

The python-glanceClient to support operations

glance-manage to handle tags

Dependencies

Same dependencies as Glance.

Testing

Unit tests will be added for all possible code with a goal of being able to isolate functionality as much as possible.

Tempest tests will be added wherever possible.

Documentation Impact

Docs needed for new API extension and usage

References

Youtube summit recap of Graffiti Juno POC demo that included tags.

Current glance metadata definition catalog documentation.

Simple application category tags (no hierarchy)

Reload configuration files on SIGHUP signal

Thu, 09 Oct 2014 00:00:00

https://blueprints.launchpad.net/glance/+spec/sighup-conf-reload

We propose to eliminate the need to restart the glance api service when configuration files are modified. Operator can send SIGHUP signal to glance service which will reload the configuration file.

Problem description

In a production environment, an administrator will modify the glance-api.conf configuration parameters like filesystem_store_datadirs when the storage is almost full to add more capacity by adding more disks, or to increase the number of workers or log configuration etc. Then they need to restart the glance services explicitly for these changes to be loaded. Restarting service would break users connected to it which is not good from users point of view.

Proposed change

Add the ability to dynamically change configuration settings of a running glance server with no impact to service.

A running glance server consists of a parent process and one or more child processes.

On receipt of a SIGHUP signal the parent process will:

reload the configuration
send a SIGHUP to the original child processes
start new child processes with the new configuration
its listening socket will not be closed

On receipt of a SIGHUP signal each original child process will:

close the listening socket so as not to accept new requests
complete any in-flight requests
exit

This approach is based on nginx’s behaviour and avoids some of the disadvantages of the current oslo’s Launcher reload:

Race conditions: Launcher does not shutdown eventlet cleanly, existing requests can fail.
If all child processes are busy there can be a lengthy delay when new requests are not processed.
Long lived pre-SIGHUP idle client connections can stall request processing indefinitely.
Not all parameters can be changed, eg number of workers.
The wsgi pipeline cannot be changed, for example to enable caching.

Alternatives

An alternative may be to attempt to save and then restore long running tasks using taskflow. The process restart would then only need to deal with short lived requests (e.g. API DB lookups) and then no user visible downtime is required for regular restarts

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

If the reload takes too long (e.g., >50ms) then the API requests will be noticeably delayed.

We are proposing current worker processes to stop accepting requests and continue with what they are doing, while the parent process starts and spawn new child processes with the new configuration. So there is a possibility that the glance node will be running twice as many child processes as it is configured to run for a while. It could impact performance, especially if it is an underpowered node that is already configured to run as many child processes as it can handle without degradation.

In the author’s opinion, it is the responsibility of the operator to make sure the node will not be over-provisioned with child processes (workers). If an operator wants to run a node with no headroom for additional child processes, the author suggests that such an operator not use dynamic configuration via SIGHUP. Instead, such an operator should use the old fashioned technique of restarting the api service.

Other deployer impact

Need to document the impact of config changes for some params like workers, host, port etc.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: stuart-mclaren
Other contributors:: abhishek-kekane

Reviewers

Core reviewer(s):: nikhil-komawar flaper87
Other reviewer(s):: icordasc

Work Items

Add handler for SIGHUP signal
Reload configuration parameters
Unit and functional tests for coverage

Dependencies

None

Testing

None

Documentation Impact

Please refer to Other deployer impact

References

https://etherpad.openstack.org/p/sighup-conf-reload

Glance Swift Store to use Multiple Containers for Storing Images

Fri, 26 Sep 2014 00:00:00

https://blueprints.launchpad.net/glance/+spec/swift-store-multiple-containers

Glance, when configured to use Swift store in Single Tenant Mode, stores images in one container as indicated by the configuration option, swift_store_container. This approach of storing images in ONE container is subject to performance bottleneck.

Storing images in one container is prone to Swift rate-limiting on containers. Swift is equipped with container rate-limiting that can throttle concurrent POST, PUT and DELETE operations in a single container. This becomes a serious issue in a large scale deployments especially when coupled with smaller segment sizes.

Problem description

Swift is known to be capable of throttling incoming traffic[1]. The very fact that swift can throttle write operations on containers presents a performance bottleneck and hence large scale deployments need an alternative to get around Swift throttling.

When container rate-limiting is enabled for a Swift cluster, it throttles concurrent POST, PUT and DELETE requests after a certain configurable rate. This directly translates to a limit on concurrent image creation and deletion operations for Glance before experiencing performance degradation.

Proposed change

To reduce/overcome the performance bottleneck, we propose the use of multiple containers for storing images in Single Tenant Mode (this change will not affect Multi Tenant Mode because that setup stores each image in its own container). This leads to increased concurrency of image creation and deletion operations.

There are four major aspects to this change:

Container Selection - determining what container an image should go into
Container Creation - creating the new containers
Re-distribution of Existing Images - moving images from old to new containers
Database Migration - updating image locations as per new containers

1) Container Selection:

This change proposes to select containers based on image uuid. Images will be stored in multiple containers in order to avoid throttling during multiple simultaneous uploads. The first N characters of the image UUID, where N is a configurable integer between 1 and 32, the number of hex digits in a UUID, with the default value of 2, will be used to determine which container the image will be uploaded to. With the default value of the first two characters used, this gives 16*16=256 unique containers. At N=1, the smallest valid value for this configuration, 16 containers will be created and used for storing images. The containers will be named after the value set for swift_store_container with the first N chars of the image UUID as the suffix.

Example: if this config option is set to 3 and swift_store_container = ‘glance’, then an image with UUID ‘fdae39a1-bac5-4238-aba4-69bcc726e848’ would be placed in the container ‘glance_fda’. All dashes in the UUID are included when creating the container name but do not count toward the character limit, so in this example with N=10 the container name would be ‘glance_fdae39a1-ba’.

The number of containers can be easily increased or decreased by changing N in the configuration. However, a new set of containers will be created with every change to this configuration. Images created after a configuration change will go into new containers while older images remain in their previous containers. The older images do not need to necessarily be moved into new containers as their locations would still point to the existing older container they are stored in. This means that changing N in the configuration will never result in existing containers being deleted. However, if one wishes to move the older images to new containers, they may do so by re-distributing the images, which is described later in this section.

Note: ‘storing’ an image in a container implies storing the manifest and all the image segments in the same container. Unless otherwise mentioned, this continues to hold true all through this specification.

2) Container Creation:

Glance ships with a configuration option to dynamically create the container, if it doesn’t exist already at the time of uploading image data to Swift. This is indicated by configuration option, swift_store_create_container_on_put. If dynamic container creation is enabled, Glance would automatically create each container when the appropriate container for that image is not found.

However, if the config option for dynamic container creation is disabled, image uploads would fail if the appropriate containers are not created manually by the deployer. This behavior is consistent with how Glance currently handles missing containers if the config option to create them is not enabled.

3) Re-distribution of Existing Images (out of scope):

This spec will not provide code or scripts to migrate existing images since lazy loading is an existing effective method of distributing new images. However, if one wants to migrate images here is the process: Once the use of multiple containers is enabled or the number of containers is changed, all previously created images would remain in the older container(s). If desired, older images can be moved to new containers appropriately. This can be achieved as a separate batch job that can be run as and when desired. Subject to the number of older images, redistributing images may involve significant movement of data in the Swift cluster. Hence, it would be helpful to achieve this in phases and in a non-intrusive fashion. Once the images are re-distributed, their image locations need to be updated as well.

4) Database Migration (out of scope):

If images are re-distributed by operator choice, image location of each re-distributed image must be updated to reflect the new container name. This requires a db migration to replace the old container name in the location with the new container name as per the image id. This migration can go hand-in-hand with re-distribution.

Scope of this spec:

Of the four aspects discussed above, this specification only addresses container creation and selection while leaving re-distribution and the required db migrations out, which can be implemented as another concerted effort.

Alternatives

1) Instead of using image id as the basis for container selection, one can use other basis like tenant id, which would keep all images belonging to a certain tenant in the same container. While other container name basis are possible, using image id provides an easier way to correlate an image to its container.

2) An alternative to creating containers could be to allow the API to create all the required containers while it boots up. This requires the API to know all possible containers before hand, which may or may not be possible depending upon the container selection basis chosen. This places a certain limitation on the kind of bases one may opt for. Hence, going with dynamic container creation will eliminate this limitation as both container selection and creation could be dynamic. Also, dynamic container creation is in-line with current Glance behavior.

3) Instead of grouping multiple images together in a container, one alternative would be to give each image its own container. However, that doesn’t solve any problems, it just moves the cardinality issues from the container to the account. Additionally, some deployers limit the maximum number of containers allowed per account. White-listing certain accounts to bypass a container limit would defeat the purpose of swift ratelimits which are chosen by deployers in order to protect the entire cluster.

Data model impact

New containers will be created and used for storing images. However, this does not have any impact on the Glance image data model itself.

Database migrations:

No database migrations are required. The code supporting multiple containers would only affect the uploading of new images, determining which container they belong to based on uuid. For existing images (those uploaded before support for multiple containers), the image already contains a valid location in its metadata. Essentially, new containers will be populated by lazy loading: When an image is uploading, it will first check through a HEAD request if the appropriate container exists for that image based on its UUID, and if the container does not exist then the container will be created immediately with a PUT request. This image will then be the first image stored in that particular container.

REST API impact

None

Security impact

Given the scope of this spec, where image data is not being re-distributed among new containers and no migrations are being run, there is minimal to no security impact introduced.

Notifications impact

This change only impacts the image location property among all the image properties. And, since image location is not included in notifications, there should be no impact to Glance notifications.

Other end user impact

As image location is not accessible to either the end-user or from Glance client, there should be no end-user impact.

Performance Impact

The use of multiple containers will reduce throttling when multiple images are uploaded simultaneously. This leads to increased concurrency of image creation and deletion operations in large scale deployments.

Container selection would take place for every image upload request and thus adds an extra operation to the current set of operations to upload image data. However, selecting a container would be a simple substring operation to fetch the first few characters of an image id. The time incurred in determining the container would be significantly smaller than the time incurred to upload image data. Overall, the performance impact of container selection should be very minimal.

Container creation is a conditional operation that would take place only when the container is not present already. This would occur once for each combination of N characters as specified in the configuration. For example, the default configuration option is that the first 2 characters of the image UUID are used to select an appropriate container, leading to a total of 256 containers which should be optimal for mid size deployments. We found that in a large scale deployment, 4096 containers would be preferred over 256 containers if smaller segment sizes were chosen. The time incurred in creating a new container is significantly smaller than the time incurred in upload image data. Hence, the overall performance impact in image uploads should be minimal.

Other deployer impact

This change would begin taking effect upon enabling multiple containers in a configuration. When enabled, new images would be uploaded to new containers, while existing images would remain in their previously assigned container. This change is forwards and backwards compatible, such that the deployer can choose to enable or disable multiple containers at any time and images will still upload and download correctly.

Deployers should note that if their deployment limits the total number of containers per account, the seed for the total number of containers should be set such that this limit is not hit.

New configuration option in glance-api.conf

swift_store_multiple_containers_seed - default = 0

When set to 0, a single-tenant store will only use one container to store all images. When set to an integer value between 1 and 32, a single-tenant store will use multiple containers to store images, and this value will determine how many containers are created. Used only when swift_store_multi_tenant is disabled. The total number of containers that will be used is approximately equal to 16^N, so if this config option is set to 2, then 16^2=256 containers will be used to store images.

When choosing the value for swift_store_multiple_containers_seed, deployers should discuss a suitable value with their swift operations team. The authors of this spec recommend that large scale deployments use a value of ‘2’, which will create a maxiumum of ~256 containers. Choosing a higher number than this, even in extremely large scale deployments, may not have any positive impact on performance and could lead to a large number of empty, unused containers. If dynamic container creation is turned off, any value for this configuration option higher than ‘1’ may be unreasonable as the deployer would have to manually create each container.

Any diagnostic/monitoring scripts assuming images to be stored in a single container may need appropriate changes.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: hemanth-makkapati
Other contributors:: ben-roble

Reviewers

Core reviewer(s):: nikhil-komawar brian-rosmaita
Other reviewer(s):: None

Work Items

Implement new config options in Swift store driver
Implement container selection in Swift store driver
Implement unit, functional, and integration tests
Change glance-api sample conf in glance repo

Points to note:

All code changes would be limited to glance_store module.
Image download code wouldn’t require any changes.
Both manifest and segments would go into the same container.

Dependencies

None

Testing

No tempest tests needed

Documentation Impact

Document new configuration options

References

[1] http://docs.openstack.org/developer/swift/ratelimit.html#configuration

Refactoring Glance logging

Thu, 12 Jun 2014 00:00:00

Glance is currently logging lots of operator relevant information as DEBUG. Also the logged messages are translated in various ways. I would like to refactor the whole Glance codebase reflecting the current logging guidelines [1].

Problem description

OPS are complaining that OpenStack as whole does not utilize the logging levels appropriately and production implementations needs to run on DEBUG- level to get needed information out of the logs.

Glance is logging a lot of operator relevant information as DEBUG. [1]

Proposed change

I would like to see whole Glance codebase revisited and the logging changed to use appropriate logging level and reflect the translation functions to those levels as well. This would be also good opportunity to ensure that we have no logging left that would write sensitive information like URIs and credentials to logfiles.

Unify the way how exceptions are logged. Exception message should be always logged if anything to ensure that operators can find the event based on end user problem description.

Alternatives

We could implement error codes to differentiate the messages sent to user and logged having still unique linkage between the two. This would cause more documentation overhead but provide greater flexibility and granularity.

Data model impact

None

REST API impact

None

Security impact

This would reinforce the confidence that we do not log sensitive/confidential information. [2]

Notifications impact

Some notification messages might have minor changes.

Other end user impact

None

Performance Impact

This should not have performance hit outside of the corrected amount of translation function calls made.

Other deployer impact

Users who have their own logging filtering in place would need to review those rules after the logging level has been changed.

Developer impact

The logging on correct levels should make developer life easier when trying to debug the code.

Implementation

Assignee(s)

Primary assignee:: Erno Kuvaja <jokke>
Other contributors:: <launchpad-id or None>

Work Items

Review OpenStack logging guidelines [1][2]
Review the current logging pattern and make decision where changing the message content would be appropriate (sensitive data, etc.)
Determine the correct logging level message by message and align the translation functions used on that log level.

Dependencies

Related proposal in the Security Guideline draft [2] to implement some sanitizing and tagging to oslo. This is not dependency nor should affect this work other than possible adobtation if the proposed functionality gets implemented.

Testing

None

Documentation Impact

New log level use cases should be documented so deployers would know what to expect finding from their logs.

References

[1] https://wiki.openstack.org/wiki/LoggingStandards [2] https://wiki.openstack.org/wiki/Security/Guidelines/logging_guidelines

Restrict users from downloading image based on custom properties

Wed, 04 Jun 2014 00:00:00

https://blueprints.launchpad.net/glance/+spec/restrict-downloading-images-protected-properties

The goal of this blueprint is to restrict normal users from downloading the images on the basis of core or custom properties by using download_image policy.

Problem description

Presently images shared publicly with the users can download these images freely which could lead to piracy. Today, you can stop users from downloading images by configuring download_image policy with role constraint, but it will restrict all users having that particular role from downloading all of the images, this is not good. So what I want is to restrict users from downloading images on the basis of specific core or custom property is present in the image and users having certain specific roles.

Proposed change

We can achieve this by adding new rule in policy.json and apply that rule to ‘download_image’ policy.

For example: Add new rule in policy.json mentioned as below

‘restricted’: ‘not (ntt_3251:%(x_billing_code_ntt)s and role:member)’ ‘download_image’: ‘role:admin or rule:restricted’

So if ‘download_image’ policy is enforced then in above case only admin or user who satisfies rule ‘restricted’ will able to download image. Other users will not be able to download the image and will get 403 Forbidden response.

To avoid implementation of dict inspection via dot syntax and enforce the policy on v1 and v2 api’s in the same way, we can create a dictionary-like mashup of the image core and custom properties, in both v1 and v2 api and pass it directly as target to _enforce() method. In case if core and custom property is same for the image, then the core property value will be overwritten on the custom property.

For example: self._enforce(req, ‘download_image’, target=image_meta_mashup)

Alternatives

Instead of passing dictionary-like mashup of the image core and custom properties directly to target, we can pass image itself and can implement dict inspection via dot syntax. In this case the new rule in policy.json need to configured as follows,

‘restricted’: ‘not (ntt_3251:%(target.x_billing_code_ntt)s and role:member)’ ‘download_image’: ‘role:admin or rule:restricted’

Data model impact

None

REST API impact

GET:/v2/images/{image_id}/file
- Description: Downloads binary image data.
- Method: GET
- Normal response code(s): 200, 204
- Expected error http response code(s): 403
  
  When image having protected properties downloaded by user who doesn’t satisfy ‘download_image’ policy
- URL for the resource: /v2/images/{image_id}/file
- Parameters which can be passed via the url {image_id}, String, The ID for the image.
GET:/v1/images/{image_id}
- Description: Returns the image details as headers and the image binary
  in the body of the response.
- Method: GET
- Normal response code(s): 200
- Expected error http response code(s): 403
  When image having protected properties downloaded by user who doesn’t satisfy ‘download_image’ policy
- URL for the resource: /v1/images/{image_id}
- Parameters which can be passed via the url {image_id}, String, The ID for the image.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Need to add new rule in policy.json for restricting downloading of image.

“restricted”: “not (ntt_3251:%(x_billing_code_ntt)s and role:member)” “download_image”: “role:admin or rule:restricted”

Where ntt_3251 will be the value of property ‘x_billing_code_ntt’.

In our case it is necessary to ensure that normal users should not be able to delete the property (‘x_billing_code_ntt’) added to the image. If normal user is able to delete the property of the image then he can easily download the image as the rule ‘restricted’ will not work in this case.

So we need to restrict normal users from deleting the property using property protections.

Need to modify following options in glance-api.conf file to enable property-protections:

property_protection_file = property-protections-roles.conf property_protection_rule_format = roles

Changes in property-protections-roles.conf

[^x_billing_code_.*] create = admin,member read = admin,member,_member_ update = admin,member delete = admin,member

Need to ensure that to use this download restrictions feature, show_image_direct_url and show_multiple_locations parameter is not set to True in glance-api.conf file. If these options are True then, using this download restriction is potentially an inconsistent policy as user might be able to download the image using image location(direct url).

In order to deploy the above policy, service provider will need to deploy 2 sets of glance api services. One glance api service will be exposed to the external nova services(nova-compute) and other to the users. The one which is exposed to the users should enforce the download_image policy with the above “restricted” rule and the glance-api which used by nova need to be isolated/protected, e.g. separated by network, in order to avoid glance-client/end user connect it by standard API.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: abhishek-kekane
Other contributors:: None

Work Items

Add new rule in policy.json to restrict download of image.
Add method to create dictionary-like mashup of image properties
Modify v1 and v2 api to restrict download
Modify logic of caching to restrict download for v1 and v2 api
Sync openstack.common.policy of oslo-inc with Glance when the change of oslo-inc get merged.

Dependencies

None

Testing

Need to add tempest test to cover download operation.

Documentation Impact

Please refer Other deployer impact.

References

None