Nova Certificate Validation¶

https://blueprints.launchpad.net/nova/+spec/nova-validate-certificates

OpenStack now supports signature verification for signed images. However, it does not support strong certificate validation for certificates used to generate image signatures. Specifically, nova has no mechanism to identify trusted certificates. While nova verifies the signature of a signed image using cursive, there is no way to determine if the certificate used to generate and verify that signature is a certificate that is trusted by the user. This change will introduce an addition to the nova API allowing the user to specify a list of trusted certificates when creating or rebuilding a server. These trusted certificates will be used to conduct certificate validation in concert with signature verification in cursive, providing the user confidence in the identity and integrity of the image being booted.

Problem description¶

Nova is capable of verifying the signature of a signed image by using cursive, the OpenStack signature verification library [2]. Signature verification ensures that unmodified image data is retrieved from glance. However, the validation of the certificate used to generate the signature of the signed image is currently limited to a timestamp validity check, ensuring only that the certificate is valid for use at the time of signature verification. There is no mechanism to ensure that the certificate used is one approved by the end user. An attacker with access to glance could replace a user’s signed image with a modified, malicious image signed with the attacker’s certificate, stored in the OpenStack deployment’s certificate manager. If asked to boot this modified image, the compute service would retrieve the image and its corresponding certificate, verify the image signature, and proceed to boot a virtual machine using the malicious image data. Providing support for certificate validation in cursive helps nova detect this attack scenario and take steps to alert the user of the potential compromise.

Note that this threat model considers glance to be untrusted and does not include threats to the integrity, availability, or confidentiality of nova. It assumes that: (1) an attacker has access to the certificate manager and is able to store certificates for use with image signing, and (2) this attacker is unable to access arbitrary certificate public/private key pairs belonging to other users. An attacker with such access would be able to impersonate the user, replacing signed images and perfectly updating the corresponding image signatures and metadata as needed to conceal the attack.

Use Cases¶

Nova users want to ensure that they are booting images they trust by controlling the set of certificates used to sign their images.

With this change, a nova user can specify the identities of trusted certificates when creating/rebuilding a server from a signed image if signature verification and certificate validation are enabled. One of these trusted certificates is expected to be the signing certificate of the certificate used to generate the image signature.

With certificate validation enabled, image signature verification will only succeed if the image signing certificate was generated by a trusted certificate. This allows users to place themselves in-the-loop of the signature verification process, requiring valid certificate information for boot to succeed.

Note that these trusted certificates are stored in a certificate manager independent of the compute service. For this work, a certificate manager is any service backend supported by castellan that provides management operations for certificates objects. Certificate management is often a subset of the functionality provided by generic key managers, which are capable of managing different types of cryptographic secrets (e.g., encryption keys, passwords). As of the Pike release, barbican (the OpenStack key management service) is the only OpenStack service that satisfies the requirements for a certificate manager. In the future, any OpenStack or third-party service that is supported by castellan and provides certificate management could be used instead of barbican.

Note also that booting an instance from an existing volume is currently incompatible with trusted certificate validation. The block storage service does not have a method for associating trusted certificates with image data stored in a volume. If trusted certificate validation is enabled and an instance is booted from a volume, there is no way for certificate validation to proceed. Therefore, all attempts to boot-from-volume with certificate validation will result in a build error.

Proposed change¶

Supporting certificate validation requires several changes. The initial change adds a pair of new configuration options. The first configuration option, enable_certificate_validation, will enable the use of the cursive certificate validation routine when conducting image signature verification (see the fifth change below). This option will only be used if verify_glance_signatures is set to True and will default to False, allowing signature verification to work without certificate validation until a compute deployment has fully upgraded. Certificate validation will be performed if this option is set to True.

The second configuration option, default_trusted_certificate_ids, will contain a list of certificate IDs that are designated as trusted by the compute deployment. This list of trusted certificate IDs will only be used if certificate validation is enabled and if no trusted certificate IDs were provided by the user (see the second and third changes below). The list should be defined by an administrator as it will be the default set of trusted certificate IDs for the compute deployment. The default value of this option will be an empty list, requiring a user-provided set of trusted certificate IDs if left empty. If the user does provide a list of trusted certificate IDs, the list of default trusted certificate IDs will be ignored.

Both of the above configuration options have already been implemented and merged into nova. See [9] for implementation details.

The second change adds a parameter, trusted_image_certificates, to the server create command of the nova API. The value of the parameter is a list containing the string IDs of the trusted certificates used to validate the signed image’s signing certificate. These IDs are assigned by the certificate manager upon upload of the trusted certificates. Multiple IDs are allowed here to provide flexibility for the user. It may not be feasible for the user to know which specific certificate corresponds to their image. Allowing the user to define a set of trusted certificates removes the need for an image/certificate mapping, simplifying the user experience. When provided, nova will pass these values to the certificate validation routine in cursive, overriding the default list of trusted certificate IDs (see the second configuration option above). Cursive will use them to fetch the trusted certificates using castellan. If provided, the value of the parameter is saved and will persist for the lifetime of the server data.

The third change adds a parameter, trusted_image_certificates, to the server rebuild command of the nova API. This parameter is identical to the parameter described in the first change above. This parameter is optional and is ignored by nova when rebuilding a server with a non-signed image or when certificate validation is not enabled. If provided, the value of the parameter is saved and will persist for the lifetime of the server data. If the parameter is not provided when rebuilding a server with a signed image, the prior set of trusted certificate IDs associated with the server will be preserved.

Both of the above changes have been implemented and await additional nova feedback. See [14] for more information.

The fourth change adds a certificate verification context to conduct certificate validation. This work will live alongside the signature verification routine in the new cursive certificate_utils module. The verification context stores an arbitrary set of certificates and uses them to validate individual certificates submitted for certificate validation. The context attempts to build a certificate chain for submitted certificates, cryptographically verifying that each parent certificate in the chain has signed its child certificate. pyca/cryptography is used to conduct all certificate and cryptographic operations here. The context also checks various constraints to determine if a certificate is valid, including valid timestamp checks. If the context cannot build a valid certificate chain for a submitted certificate, or if any of the certificate constraint checks fail, the submitted certificate fails validation.

The above change has been implemented and merged into cursive. See [10] for more information.

The fifth change integrates the certificate verification routine into the signature verification workflow. When the certificate verification routine fetches the image’s signing certificate, it builds the verification context using the trusted certificates provided by the user (see the first and second changes above). It then passes the signing certificate through the context for certificate validation. If validation succeeds, signature verification can proceed as normal. If validation fails, signature verification fails as well, the server is placed in an ERROR state and a fault is returned to the user.

The above change has been implemented and awaits additional nova feedback. See [13] for more information.

It is possible a silent fail scenario could occur if (1) nova is not configured to conduct certificate validation, and (2) the user provides trusted certificate IDs expecting certificate validation to occur. In this case, nova would not conduct certificate validation and would boot the instance, causing the user to believe certificate validation succeeded even though it never happened. To prevent this from happening, the boot workflow will be updated to conduct signature and certificate verification if trusted certificate IDs are associated with the instance data. This matches the user’s expectations and prevents a silent fail scenario from ever occurring. Note here that this override only occurs if the user specifies certificate IDs. The default list of certificate IDs is only used if the feature is enabled and therefore will never trigger the override.

The sixth change updates the nova data model to support certificate validation during server operations, like server evacuation and cold or live migrations. More generally, this applies to any operation that may be done by an admin against the server without all the information the end user originally supplied to the nova boot command. To support these cases, the trusted certificate IDs used for certificate validation must be stored with the instance data, since the user cannot be expected to provide them. The InstanceExtra functionality in nova already supports keypairs associated with instances. This change will update the InstanceExtra schema to support a trusted_certs column, which will contain the list of trusted certificate IDs. The underlying storage will leverage oslo versionedobjects, adding a new TrustedCerts object for instances to reference.

The above change has been implemented and awaits additional nova feedback. See [11] and [12] for more information.

The seventh change updates the InstancePayload notification base, adding in trusted certificate IDs to the create and rebuild instance notifications. This will help users and administrators identify when their instances are leveraging certificate validation and can assist in diagnosing validation failures when they occur.

The eighth change updates the control flow for booting instances from volumes, intercepting these requests if certificate validation is enabled and generating a build failure instead. Certificate validation cannot be supported in these cases until the underlying block storage service supports mapping trusted certificate information to instance data [19].

The ninth change adds new policy rules around the use of trusted certificates, allowing nova administrators to easily enable/disable certificate validation if their deployments can/cannot support the feature. Specifically, new policy rules will be added for the server create and rebuild requests. If either request is made and trusted certificates are provided, the policy checker will verify that the operation is allowed. If not, a PolicyNotAuthorized exception will be raised to fail the request. This is useful in cases where a deployment supports boot-from-volume (see the seventh change above) or supports virtualization drivers aside from libvirt.

The tenth change updates the novaclient/openstackclient to support the trusted_image_certificates parameter for the server create/rebuild commands. This includes support for a new environment variable, OS_TRUSTED_CERTIFICATE_IDS, that can be used to define a comma-delimited list of trusted certificate IDs. If the trusted_image_certificates parameter is not used, the client will pull the value of the environment variable and use it instead. This value will be converted into a list before being passed on.

The above change has been implemented for both clients and awaits additional feedback. See [15] and [16] for more information.

If the user does not provide a value for the trusted_image_certificates parameter, either explicitly or through the OS_TRUSTED_CERTIFICATE_IDS environment variable, nova will pull the list of trusted certificate IDs from the default_trusted_certificate_ids configuration option. If this option is left as an empty list, there is no way for nova to obtain a trusted certificate for certificate validation. In this case there would be no way to determine if the image’s signing certificate is trusted so signature verification would fail, in turn failing server creation.

The eleventh and final change updates the output of the server show command to include the list of trusted certificate IDs stored with the server instance data. If no certificate IDs are stored with the server instance data, the output from the server show command will still contain the new key trusted_image_certificates in the response, just like for tags. This is to avoid confusing the end user who has made a request on a given microversion expecting to see whether or not the server has certificates associated with it.

Alternatives¶

An alternative approach to certificate validation here would be to support certificate trust stores, collections of trusted certificates associated with individual users or projects. When creating a new server, the user would specify their trust store as a source of trusted certificates, replacing the list of certificates provided in the trusted_image_certificates parameter. There are many ways to support trust stores, including: a filesystem directory trust store containing trusted certificate files stored locally on the compute host, a metadata/managed resource approach supported under services like nova or keystone, and a container-based secret storage approach supported by services like barbican. While useful in defining collections of trusted certificates, a trust store approach would need to scale for large cloud deployments which may be difficult from a management and maintenance perspective. Trust stores also introduce a new construct that must be trusted by the user, especially if the user is not directly responsible for maintaining their trust store. These restrictions may not be feasible for some cloud deployments.

An alternative to the user providing trusted certificates, or storing trusted certificates in a trust store, would be to dynamically fetch certificates using information stored in the Private Internet Extension of the signed certificate being validated. This approach allows deployers and users to use signing certificates without needing to pre-fetch all of the root and intermediate certificates required to complete the certificate validation process. However, this approach requires the compute service have persistent network access to all possible certificate repositories where root and intermediate certificates may be stored. In many cases, this will include network access to the public Internet which may not be feasible for a generic deployment.

An enhanced certificate validation routine would include certificate revocation, supporting commonly used approaches like certificate revocation lists (CRLs) and/or the Online Certificate Status Protocol (OCSP). Supporting certificate revocation would allow the compute service to dynamically determine when certificates become invalid in real time due to compromise, further improving the security of booting signed images. However, supporting certificate revocation involves dynamically fetching and trusting network resources, often under the control and authority of third-parties. This may not be feasible for some deployments. It is possible that certificate revocation could be integrated outside of the compute service, for example within the certificate manager or through another third-party service. This would grant nova the benefits of timely revocation without complicating the signature verification and certificate validation features in nova itself.

It should be noted here that support for certificate revocation is intended to be added in future work for this feature.

Data model impact¶

The InstanceExtra database model will be updated to include a new text column, trusted_certificate_ids, which will contain the list of trusted certificate IDs provided with server create/rebuild requests. As stated above, if the IDs are not included with the server request, they will be pulled from the default_trusted_certificate_ids configuration option. Like the existing fields in InstanceExtra, this addition will leverage oslo versionedobjects for storing the list, requiring the addition of a trusted_image_certificate_id field.

REST API impact¶

The following are example requests to (1) create a new server from a signed image and (2) rebuild a server from a signed image, including the new trusted_image_certificates parameter. This update will be done under a new API microversion.

{
    "server": {
        "name": "example-name",
        "imageRef": "70a599e0-31e7-49b7-b260-868f441e862b",
        "flavorRef": "http://openstack.example.com/flavors/1",
        "trusted_image_certificates": [
            "00000000-0000-0000-0000-000000000000",
            "11111111-1111-1111-1111-111111111111",
            "22222222-2222-2222-2222-222222222222"
        ],
        "metadata": {
            "My Server Name": "Example Signed Server"
        }
    }
}

{
    "rebuild": {
        "name": "example-name",
        "imageRef": "70a599e0-31e7-49b7-b260-868f441e862b",
        "trusted_image_certificates": [
            "00000000-0000-0000-0000-000000000000",
            "11111111-1111-1111-1111-111111111111",
            "22222222-2222-2222-2222-222222222222"
        ],
        "metadata": {
            "My Server Name": "Example Signed Server"
        }
    }
}

Note that while in these examples the values in trusted_image_certificates are UUIDs they are not guaranteed to be so. Certificate managers use different ID allocation schemes; while some use strict UUIDs, others use simple incrementing integers or raw hex strings. For this feature, the type of trusted_image_certificates will be an array containing zero or more JSON string values.

The following is a JSON schema description of the trusted_image_certificates parameter:

{
    "type": "array",
    "minItems": 0,
    "maxItems": 50,
    "uniqueItems": true,
    "items": {
        "type": "string"
    }
}

Note the upper and lower bounds for the number of certificate IDs included in the trusted_image_certificates parameter. If an API call is made for a signed image and exceeds the maximum number of allowed certificate IDs, then the API call will fail.

Security impact¶

With the added verification step provided by this feature when enabled, the security of the signed image verification feature is improved.

Notifications impact¶

With the addition of trusted certificate information to the InstanceExtra data model, create and rebuild instance notifications should be updated to include the trusted certificate IDs for a specific instance. Specifically, the InstanceCreatePayload and InstanceActionRebuildPayload should be updated to include a ‘trusted_image_certificates’ field that will contain the list of trusted certificate IDs obtained from the instance associated with the notification.

Other end user impact¶

This change imposes additional restrictions on the certificates that can be used to sign images, and may cause migration challenges if used with images signed before the feature is enabled.

Migration will require users to upload their trusted certificates to the certificate manager if they intend to specify them with the create or rebuild request. All image signing certificates must already be in the certificate manager to support signature verification.

With support being added for the OS_TRUSTED_CERTIFICATE_IDS environment variable, users are encouraged to set the variable with the list of trusted certificate IDs through their openrc file, alongside their authentication credentials. The value of the OS_TRUSTED_CERTIFICATE_IDS environment variable is a comma-delimited string of trusted certificate IDs, which will be converted into a list of certificate IDs for the trusted_image_certificates parameter.

An example openrc file is shown below, using the same trusted certificate IDs as those used in the API example (see REST API Impact above):

export OS_USERNAME=username
export OS_PASSWORD=password
export OS_TENANT_NAME=projectName
export OS_AUTH_URL=https://identityHost:portNumber/v2.0
export OS_TRUSTED_CERTIFICATE_IDS=00000000-0000-0000-0000-000000000000,111111
11-1111-1111-1111-111111111111,22222222-2222-2222-2222-222222222222

Note that in this example, the second certificate ID is split to satisfy line wrap formatting for this spec. No explicit linebreaks should be used in the actual openrc file.

Performance Impact¶

Nova will load the user’s trusted certificates via cursive every time signature verification is performed. Depending upon the size and number of certificates, and the frequency of signature verification, this could introduce a performance burden on the compute service. To alleviate this, see Alternatives above regarding a persistent certificate trust store and dynamically loading certificates from remote storage.

Other deployer impact¶

The inclusion of two new configuration options, enable_certificate_validation and default_trusted_certificate_ids, will smooth the transition for deployments looking to enable this feature. If these options are enabled, all prior usage of the server create/rebuild API when booting signed images will now fail if trusted certificates cannot be located.

The inclusion of new policy rules controlling the usage of certificate validation will make it easy for administrators to enable or disable the feature if their deployment supports other features that are incompatible with certificate validation, like boot-from-volume support and non-libvirt based virtualization.

Developer impact¶

None

Upgrade impact¶

None

Implementation¶

Assignee(s)¶

Primary assignee:: Peter Hamilton

Work Items¶

Add two new configuration options, enable_certificate_validation and default_trusted_certificate_ids. The first will enable the use of certificate validation if signature verification is enabled. The second will provide a default list of trusted certificate IDs that can be used if no trusted certificate IDs are provided with the server request. See [9].
Update cursive to support certificate validation. This includes the addition of the certificate verification context class and the verify_certificate routine which loads certificates from the certificate manager and uses the certificate verification context to conduct certificate validation. See [10].
Update the existing signature verification workflow in nova to incorporate certificate validation, using the verify_certificate routine in cursive to validate the signing certificate. See [13].
Update the InstanceExtra database model to include a new text column, trusted_certificate_ids. Database migrations will be included to add/remove this column when updating/downgrading the database schema. See [11] and [12].
Add a new nova API parameter, trusted_image_certificates, to the server create and rebuild commands. The value of this parameter will need to be passed through to the signature verification step when downloading the image from glance. See [14].
Add a new notification field, ‘trusted_image_certificates’, to the InstanceCreatePayload and InstanceActionRebuildPayload. See [20].
Modify the control flow for booting an instance from a volume to generate a build error when certificate validation is enabled.
Add new policy rules to allow simple enable/disable control for certificate validation if a deployment supports features that are incompatible with certificate validation.
Update novaclient to support the trusted_image_certificates parameter.
Update novaclient to pull the value of the OS_TRUSTED_CERTIFICATE_IDS environment variable when the trusted_image_certificates parameter is not provided by the user.
Update openstackclient to support the trusted_image_certificates parameter.
Update openstackclient to pull the value of the OS_TRUSTED_CERTIFICATE_IDS environment variable when the trusted_image_certificates parameter is not provided by the user.

Dependencies¶

This work is dependent on the creation and deployment of a gate-tempest-dsvm-security-ubuntu-xenial job which runs tempest with signed images and barbican as the certificate manager. For more information on this work, see the corresponding tempest blueprint [6].

Testing¶

Unit tests will be included to test the functionality implemented in nova, novaclient, and openstackclient. Tempest tests will also be implemented to test the end-to-end feature across glance and nova. See [17] and [18].

Documentation Impact¶

Documentation on the trusted_image_certificates API parameter and the two new configuration options will need to be added, as will instructions defining the OS_TRUSTED_CERTIFICATE_IDS environment variable and its usage.

References¶

[1] “Nova Signature Verification.” Online: http://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/image-verification.html

[2] “Cursive.” Online: https://launchpad.net/cursive

[3] “Cleanup of signature_utils code.” Online: https://blueprints.launchpad.net/nova/+spec/signature-code-cleanup

[4] “Use cursive for signature verification.” Online: https://review.openstack.org/#/c/351232/

[5] “Extend Extras Functionality.” Online: https://review.openstack.org/#/c/343939/

[6] “Create experimental gate job to test Nova’s image signature verification.” Online: https://blueprints.launchpad.net/tempest/+spec/image-signing-experimental-gate

[7] “Options for using trusted certificates in Nova image signature verification.” Online: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105454.html

[8] “pyca/cryptography.” Online: https://github.com/pyca/cryptography

[9] “Add configuration options for certificate validation.” https://review.openstack.org/#/c/457678/

[10] “Add certificate validation.” https://review.openstack.org/#/c/357202/

[11] “Add trusted_certs object.” https://review.openstack.org/#/c/489408/

[12] “Add trusted_certs to instance_extra.” https://review.openstack.org/#/c/537897/

[13] “Implement certificate_utils.” https://review.openstack.org/#/c/479949/

[14] “Add trusted_certificates to REST API.” https://review.openstack.org/#/c/486204/

[15] “Microversion 2.61 - Add trusted_image_certificates.” https://review.openstack.org/#/c/500396/

[16] “Add trusted_image_certificates to server.” https://review.openstack.org/#/c/501926/

[17] “Add new schema for Nova microversion 2.61.” https://review.openstack.org/#/c/526485/

[18] “Add certificate validation scenario tests.” https://review.openstack.org/#/c/515210/

[19] “Support image signature verification.” https://review.openstack.org/#/c/384143/

[20] “Add notification support for trusted_certs.” https://review.openstack.org/#/c/563269/

History¶

This specification has received extensive review from the OpenStack community given that it involves security features in nova. The following is a brief timeline of this proposal’s history, with major changes documented below during each development cycle.

..list-table:: Revisions

header-rows:: 1

- Release Name
- Description
- Newton
- Rough draft published
- Ocata
- Introduced for official review
- Pike
- Re-proposed for official review
- Approved
- Queens
- Re-proposed for official review
- Approved
- Rocky
- Re-proposed for official review
- Approved
- Amended and re-proposed for official review

Newton¶

The initial version of this spec was released towards the end of the Newton development cycle in preparation for Ocata, focusing on a certificate trust store implementation rooted on the compute host filesystem and managed by the cloud administrator. Versions 2, 3, and 4 involved minor formatting and grammatical updates.

Version 5 received feedback from the nova core team, focusing specifically on (1) the need for tighter integration between trusted certificate management and tenant users, and (2) the potential scalability issues with distributed certificate file management across large clouds. Further feedback was also solicited from the community through a post to the openstack-dev mailing list [7].

Version 6 updated the proposed approach, preserving the filesystem-based certificate trust store while adding an update to the nova API. This API change allowed users to specify the trusted certificate ID when creating new instances.

Ocata¶

Version 7 incorporated feedback received from the Ocata Design Summit, officially removing the filesystem-based certificate trust store approach and focusing solely on updating the nova API to allow the user to submit a set of trusted certificate IDs when creating new instances.

Version 8 addressed further feedback from the nova core team, including:

highlighting a dependency on barbican as the only supported castellan backend
updating the nova API changes to include the rebuild operation
updating the nova data model to support storing the set of trusted certificate IDs with the instance data in instance_extras, thereby supporting automatic operations like instance evacuation and cold/live migrations

Pike¶

Version 9 duplicated Version 8 as a clean slate for the Pike review process. Version 10 addressed minor whitespace and spec formatting errors.

Version 11 added the new History section from the Pike spec template and incorporated feedback received on Version 9, clarifying API details and reordering the Security, Other end user, and Other deployer impact sections.

Version 12 addressed further reviewer feedback, clarifying nova handling of the API changes and resolving discrepancies in spec details.

Version 13 updated the spec to reflect the integration of cursive into nova, moving the certificate validation code to cursive.

Version 14 added support for two new configuration options to smooth the transition to use for certificate validation, in addition to clarifying the use of oslo versionedobjects for the modification to InstanceExtra.

Queens¶

Version 15 and 16 duplicated Version 14 as a clean slate for the Queens review process. Version 17 added information on merged and active patches implementing the various changes detailed by the spec.

Version 17 added details addressing a silent fail scenario, conditionally including certificate information in server metadata, and other minor fixes.

Rocky¶

Version 18 duplicated Version 17 as a clean state for the Rocky review process. Version 19 added minor updates, including updated blueprint and code review links.

Version 20 incorporates reviewer feedback, including updates on notification changes for instances, the inclusion of new policy rules, and associated error handling when certificate validation is used with boot-from-volume.

Nova Certificate Validation