Cinder Certificate Validation¶

https://blueprints.launchpad.net/cinder/+spec/certificate-validate

OpenStack now supports signature verification for signed images. However, it does not support strong certificate validation for certificates used to generate image signatures. Specifically, cinder has no mechanism to identify trusted certificates. While cinder verifies the signature of a signed image using cursive, there is no way to determine if the certificate used to generate and verify that signature is a certificate that is trusted by the user. This change will introduce an addition to the cinder API allowing the user to specify a list of trusted certificates when creating volume from images. These trusted certificates will be used to conduct certificate validation in concert with signature verification in cursive, providing the user confidence in the identity and integrity of the image being created.

Problem description¶

Cinder is capable of verifying the signature of a signed image by using cursive, the OpenStack signature verification library [1]. Signature verification ensures that unmodified image data is retrieved from glance. However, the validation of the certificate used to generate the signature of the signed image is currently limited to a timestamp validity check, ensuring only that the certificate is valid for use at the time of signature verification. There is no mechanism to ensure that the certificate used is one approved by the end user. An attacker with access to glance could replace a user’s signed image with a modified, malicious image signed with the attacker’s certificate, stored in the OpenStack deployment’s certificate manager. If asked to create volume from the modified image, cinder would retrieve the image and its corresponding certificate, verify the image signature, and build a volume using the malicious image data. Providing support for certificate validation in cursive helps cinder detect this attack scenario and take steps to alert the user of the potential compromise.

Note that this threat model considers glance to be untrusted and does not include threats to the integrity, availability, or confidentiality of cinder. It assumes that: (1) an attacker has access to the certificate manager and is able to store certificates for use with image signing, and (2) this attacker is unable to access arbitrary certificate public/private key pairs belonging to other users. An attacker with such access would be able to impersonate the user, replacing signed images and perfectly updating the corresponding image signatures and metadata as needed to conceal the attack.

Use Cases¶

Cinder users want to ensure that they are creating from images they trust by controlling the set of certificates used to sign their images.

With this change, a cinder user can specify the identities of trusted certificates when creating volumes from a signed image if signature verification and certificate validation are enabled. One of these trusted certificates is expected to be the signing certificate of the certificate used to generate the image signature.

With certificate validation enabled, image signature verification will only succeed if the image signing certificate was generated by a trusted certificate. This allows users to place themselves in-the-loop of the signature verification process, requiring valid certificate information for boot to succeed. Note that if the deployment is configured to trust certain certificates that the user isn’t aware of, the deployer could still manipulate images in a way that the user wouldn’t detect with this feature.

Proposed change¶

Supporting certificate validation requires several changes. The initial change adds a pair of new configuration options. The first configuration option, enable_certificate_validation, will enable the use of the cursive certificate validation routine when conducting image signature verification (see the fifth change below). This option will only be used if verify_glance_signatures is set to enabled and will default to False. Certificate validation will be performed if this option is set to True.

The second configuration option, default_trusted_certificate_ids, will contain a list of certificate IDs that are designated as trusted by the cinder deployment. This list of trusted certificate IDs will only be used if certificate validation is enabled and if no trusted certificate IDs were provided by the user (see the second and third changes below). The list should be defined by an administrator as it will be the default set of trusted certificate IDs for the cinder deployment. The default value of this option will be an empty list, requiring a user-provided set of trusted certificate IDs if left empty. If the user does provide a list of trusted certificate IDs, the list of default trusted certificate IDs will be ignored.

The second change adds a parameter, trusted_image_certificates, to the volume create command of the cinder API. The value of the parameter is an array containing the string IDs of the trusted certificates used to validate the signed image’s signing certificate. These IDs are assigned by the certificate manager upon upload of the trusted certificates. Multiple IDs are allowed here to provide flexibility for the user. It may not be feasible for the user to know which specific certificate corresponds to their image. Allowing the user to define a set of trusted certificates removes the need for an image/certificate mapping, simplifying the user experience. When provided, cinder will pass these values to the certificate validation routine in cursive, overriding the default list of trusted certificate IDs (see the second configuration option above). Cursive will use them to fetch the trusted certificates using castellan. If provided, the value of the parameter will be saved in image metadata.

The third change integrates the certificate verification routine into the signature verification workflow. When the certificate verification routine fetches the image’s signing certificate, it builds the verification context using the trusted certificates provided by the user. It then passes the signing certificate through the context for certificate validation. If validation succeeds, signature verification can proceed as normal. If validation fails, signature verification fails as well, the volume is placed in an ERROR state.

NOTE: The corresponding feature [2] has been merged into Cursive already.

It is possible a silent fail scenario could occur if (1) cinder is not configured to conduct certificate validation, and (2) the user provides trusted certificate IDs expecting certificate validation to occur. In this case, cinder would not conduct certificate validation and would boot the instance, causing the user to believe certificate validation succeeded even though it never happened. To prevent this from happening, the create workflow will be updated to conduct signature and certificate verification if trusted certificate IDs are associated with the create request. This matches the user’s expectations and prevents a silent fail scenario from ever occurring. Note here that this override only occurs if the user specifies certificate IDs. The default list of certificate IDs is only used if the feature is enabled and therefore will never trigger the override.

The fourth change updates the cinderclient to support the trusted_image_certificates parameter for the volume create commands. This includes support for a new environment variable, OS_TRUSTED_CERTIFICATE_IDS, that can be used to define a comma-delimited list of trusted certificate IDs. If the trusted_image_certificates parameter is not used, the client will pull the value of the environment variable and use it instead. This value will be converted into a list before being passed on.

If the user does not provide a value for the trusted_image_certificates parameter, either explicitly or through the OS_TRUSTED_CERTIFICATE_IDS environment variable, cinder will pull the list of trusted certificate IDs from the default_trusted_certificate_ids configuration option. If this option is left as an empty list, there is no way for nova to obtain a trusted certificate for certificate validation. In this case there would be no way to determine if the image’s signing certificate is trusted so signature verification would fail, in turn failing volume creation.

To be clear, we will verify the image’s signature only when image is downloaded from glance and content is copied to volume on host. So when image volume is created via clone_image or clone_image_volume we will skip this verification process, in order not to confuse end users, we will also add a flag in volume image metadata to indicate whether we validated the certificates during volume creation, and the service will generate an user message if the certificate validation fails in the backend.

Note that these trusted certificates are stored in a certificate manager independent of the volume service. For this work, a certificate manager is any service backend supported by castellan that provides management operations for certificates objects. Certificate management is often a subset of the functionality provided by generic key managers, which are capable of managing different types of cryptographic secrets (e.g., encryption keys, passwords). As of the Pike release, barbican (the OpenStack key management service) is the only OpenStack service that satisfies the requirements for a certificate manager. In the future, any OpenStack or third-party service that is supported by castellan and provides certificate management could be used instead of barbican.

Alternatives¶

An alternative approach to certificate validation here would be to support certificate trust stores, collections of trusted certificates associated with individual users or projects. When creating a new server, the user would specify their trust store as a source of trusted certificates, replacing the list of certificates provided in the trusted_image_certificates parameter. There are many ways to support trust stores, including: a filesystem directory trust store containing trusted certificate files stored locally on the compute host, a metadata/managed resource approach supported under services like nova or keystone, and a container-based secret storage approach supported by services like barbican. While useful in defining collections of trusted certificates, a trust store approach would need to scale for large cloud deployments which may be difficult from a management and maintenance perspective. Trust stores also introduce a new construct that must be trusted by the user, especially if the user is not directly responsible for maintaining their trust store. These restrictions may not be feasible for some cloud deployments.

An alternative to the user providing trusted certificates, or storing trusted certificates in a trust store(or just a database table), would be to dynamically fetch certificates using information stored in the Private Internet Extension of the signed certificate being validated. This approach allows deployers and users to use signing certificates without needing to pre-fetch all of the root and intermediate certificates required to complete the certificate validation process. However, this approach requires the compute service have persistent network access to all possible certificate repositories where root and intermediate certificates may be stored. In many cases, this will include network access to the public Internet which may not be feasible for a generic deployment.

An enhanced certificate validation routine would include certificate revocation, supporting commonly used approaches like certificate revocation lists (CRLs) and/or the Online Certificate Status Protocol (OCSP). Supporting certificate revocation would allow the compute service to dynamically determine when certificates become invalid in real time due to compromise, further improving the security of booting signed images. However, supporting certificate revocation involves dynamically fetching and trusting network resources, often under the control and authority of third-parties. This may not be feasible for some deployments. It is possible that certificate revocation could be integrated outside of the compute service, for example within the certificate manager or through another third-party service. This would grant nova the benefits of timely revocation without complicating the signature verification and certificate validation features in nova itself.

It should be noted here that support for certificate revocation is intended to be added in future work for this feature.

Data model impact¶

None

REST API impact¶

Add a new microversion to support specifying trusted_image_certificates when creating volume:
```
POST: /V3/{tenant_id}/volumes
```

{
  "volume": {
      "size": 1,
      "name": "image_volume",
      "imageRef": "CAB56EC2-4BDC-45D4-9898-3F88A7003261",
      "trusted_image_certificates": [
          "00000000-0000-0000-0000-000000000000",
          "11111111-1111-1111-1111-111111111111",
          "22222222-2222-2222-2222-222222222222"
      ],
  }

Note that while in these examples the values in trusted_image_certificates are UUIDs they are not guaranteed to be so. Certificate managers use different ID allocation schemes; while some use strict UUIDs, others use simple incrementing integers or raw hex strings. For this feature, the type of trusted_image_certificates will be an array containing zero or more JSON string values.

The following is a JSON schema description of the trusted_image_certificates parameter:

{
    "type": "array",
    "minItems": 0,
    "maxItems": 50,
    "uniqueItems": true,
    "items": {
        "type": "string"
    }
}

Note the upper and lower bounds for the number of certificate IDs included in the trusted_image_certificates parameter. If an API call is made for a signed image and exceeds the maximum number of allowed certificate IDs, then the API call will fail.

API version bump will be required for these changes.

Security impact¶

With the added verification step provided by this feature when enabled, the security of the signed image verification feature is improved.

Notifications impact¶

None

Other end user impact¶

With support being added for the OS_TRUSTED_CERTIFICATE_IDS environment variable, users are encouraged to set the variable with the list of trusted certificate IDs through their openrc file, alongside their authentication credentials. The value of the OS_TRUSTED_CERTIFICATE_IDS environment variable is a comma-delimited string of trusted certificate IDs, which will be converted into a list of certificate IDs for the trusted_image_certificates parameter.

An example openrc file is shown below, using the same trusted certificate IDs as those used in the API example (see REST API Impact above):

export OS_USERNAME=username
export OS_PASSWORD=password
export OS_TENANT_NAME=projectName
export OS_AUTH_URL=https://identityHost:portNumber/v2.0
export OS_TRUSTED_CERTIFICATE_IDS=00000000-0000-0000-0000-000000000000,
11111111-1111-1111-1111-111111111111,22222222-2222-2222-2222-222222222222

Performance Impact¶

Cinder will load the user’s trusted certificates via cursive every time signature verification is performed. Depending upon the size and number of certificates, and the frequency of signature verification, this could introduce a performance burden on the volume service, such as consuming images from the Cinder image cache. To alleviate this, see “Alternatives” above regarding a persistent certificate trust store and dynamically loading certificates from remote storage.

Other deployer impact¶

The inclusion of two new configuration options, enable_certificate_validation and default_trusted_certificate_ids, will smooth the transition for deployments looking to enable this feature.

Developer impact¶

None

Implementation¶

Assignee(s)¶

Primary assignee:: Yikun Jiang(yikunkero@gmail.com)
Other assignee:: tommylikehu(tommylikehu@gmail.com)

Work Items¶

Add two new configuration options, enable_certificate_validation and default_trusted_certificate_ids. The first will enable the use of certificate validation if signature verification is enabled. The second will provide a default list of trusted certificate IDs that can be used if no trusted certificate IDs are provided with the volume request.
Update the existing signature verification workflow in cinder to incorporate certificate validation, using the verify_certificate routine in cursive to validate the signing certificate.
Add a new cinder API parameter, trusted_image_certificates, to the volume create command. The value of this parameter will need to be passed through to the signature verification step when downloading the image from glance.
Update cinderclient to support the trusted_image_certificates parameter.
Update cinderclient to pull the value of the OS_TRUSTED_CERTIFICATE_IDS environment variable when the trusted_image_certificates parameter is not provided by the user.

Dependencies¶

None

Testing¶

Unit tests will be included to test the functionality implemented in cinder, cinderclient. Tempest tests will also be implemented to test the end-to-end feature across glance and cinder.

Documentation Impact¶

Documentation on the trusted_image_certificates API parameter and the two new configuration options will need to be added, as will instructions defining the OS_TRUSTED_CERTIFICATE_IDS environment variable and its usage.

References¶

[1] “Cursive.” Online: https://launchpad.net/cursive

[2] “Add certificate validation.” https://review.openstack.org/#/c/357202/

Cinder Certificate Validation

Cinder Certificate Validation¶

Problem description¶

Use Cases¶

Proposed change¶

Alternatives¶

Data model impact¶

REST API impact¶

Security impact¶

Notifications impact¶

Other end user impact¶

Performance Impact¶

Other deployer impact¶

Developer impact¶

Implementation¶

Assignee(s)¶

Work Items¶

Dependencies¶

Testing¶

Documentation Impact¶

References¶

Cinder Specs 0.0.1.dev665