Barbican Specs

Secret Consumers

Fri, 09 Sep 2022 00:00:00

https://storyboard.openstack.org/#!/story/2005770

This spec proposes an addition to the Barbican Secrets API to allow other OpenStack projects to add references to individual Secrets when those secrets are being used by them.

This spec also proposes a change to both the Python and CLI clients in python-barbicanclient in how they handle the deletion of secrets. Clients would be changed such that deleting a secret will result in an error when they are still being consumed by another project unless a force parameter is provided.

This spec is part of a larger effort to provide Encrypted Images to OpenStack clouds.

Problem Description

Other OpenStack projects would like to make use of an end user’s secrets e.g. A Secret that contains an encryption key for Image Encryption. But there is currently no way for those projects to let the user know that they are using the Secret. This lack of awareness may lead to errors if the user deletes a Secret that is still in use by other projects.

On the other hand, users should be allowed to delete secrets whenever they want, so a Secret being used by other projects should not prevent deletion.

Proposed Change

Add a new API to Secrets to register Secret Consumers (similar, but not identical to the Containers Consumer API [1]).

With this new API, other OpenStack projects would register as a consumer of a secret by sending a request to Barbican. Barbican stores the service type of the requesting service, as well as both the resource type and resource ID of the resource that is using the Secret.

See REST API Impact below for details of the API changes.

Clients to barbican would change the semantics for deleting secrets by returning an error when trying to delete a secret if that secret has one or more consumers. Clients will also accept an additional boolean parameter to delete a secret regardless of how many consumers it has.

See Python and Command Line Client Impact below for details of the client changes.

Alternatives

One alternative would be to implement Secret Consumers just like Container Consumers, which uses a URL instead of the consuming entity type and ID.

Another alternative approach that was considered was to have each project clone the secret when they need to use it. This alternative has some downsides, however. For one, an end user may not be able to delete those copies.

Data model impact

A new model and associated data table will need to be added. For example, a new class SecretConsumerMetadatum with a secret_consumer_metadata table.

The new class will have references to both the secret_id as well as the project_id which owns the secret.

REST API impact

POST /v1/secrets/{secret_id}/consumers

Add a new resource as a consumer to a secret.

Body Parameters

Name	Type	Description
service	string	Consumer’s OpenStack service type as shown in https://service-types.openstack.org/service-types.json
resource_type (or resource_path?)	string	Name of the resource type using the secret e.g. “images” or “lbaas/loadbalancers”
resource_id	string	Unique identifier for the resource using this secret.

Barbican will consider the resource_id to be a unique together with the secret, service and resource_type. If the resource_id is a UUID, duplicate IDs for different projects are not likely to ever happen in a single cloud.

resource_type should be meaningful to the individual projects, and should be used to identify the resource in the consuming service. For example, Glance could use “images” as the value of the resource type to indicate that the resrouce_id refers to an image.

Request

POST /v1/secrets/{secret_id}/consumers
Headers:
    X-Auth-Token: {token}
    X-Content-Type: application/json

{
    "service": "image",
    "resource_type": "images",
    "resource_id": "{image_id}"
}

Responses

Code	Description
200	OK
401	Unauthorized - X-Auth-Token is invalid
403	Forbidden - X-Auth-Token is valid, but the associated project does not have the appropriate role/scope

GET /v1/secrets/{secret_id}/consumers

List consumers for a particular Secret.

Parameters

Name	Type	Default	Description
offset	integer	0	Offset to start consumer response
limit	integer	10	Number of consumer entries returned in response
service	string	None	Filter by service type

Request

GET /v1/secrets/{secret_id}/consumers
Headers:
    X-Auth-Token: {token}

OK Response

200 OK

{
    "total": 1,
    "consumers": [
        {
            "service": "image",
            "resource_type": "images",
            "resource_id" : "{image_id}"
        }
    ]
}

Other Responses

Code	Description
401	Unauthorized - X-Auth-Token is invalid
403	Forbidden - X-Auth-Token is valid, but the associated project does not have the appropriate role/scope

DELETE /v1/secrets/{secret_id}/consumers

Delete a consumer. ie. The resource is being deleted and it longer needs to access this secret.

Request

DELETE v1/secrets/{secret_id}/consumers
Headers:
    X-Auth-Token: {token}
    X-Content-Type: application/json

{
    "service": "image",
    "resource_type": "images",
    "resource_id": "{image_id}"
}

Responses

Code	Description
200	OK
401	Unauthorized - X-Auth-Token is invalid
403	Forbidden - X-Auth-Token is valid, but the associated project does not have the appropriate role/scope
404	Not Found - Consumer record for given resource_id was not found.

Security impact

Because the consumers are stored in the database, there is the possibility that a bad actor could add many consumers to try to fill the database disk space. Secret Consumers should be limited to the same quota as Container Consumers to mitigate this risk. For example:

[quota]
quota_consumers=10000

Would limit both Container Consumers and Secret Consumers to a maximum of 10,000 consumers each for both a single Container or a single Secret.

Notifications & Audit Impact

The new API endpoints should be audited as usual.

Python and Command Line Client Impact

The Secret class in python-barbicanclient should be updated to add new methods such as:

class SecretManager(...):
    ...

    def register_consumer(self, secret_ref, service_type, resource_type, resource_id):
        ...

    def remove_consumer(self, secret_ref, service_type, resource_type, resource_id):
        ...

Both methods should raise appropriate exceptions when the API returns an error. Additionally, the SecretManager.delete() method should be updated to take a new force parameter and throw an exception when delete() is called with force=False and the secret still has consumers:

class SecretManager(...):
    ...

    def delete(self, container_ref, force=False):
        ...

The CLI client should be changed to add new consumer options, such as:

openstack secret consumer add --service-type=image --resource-type=image \
    --resource-id=XXXX-XXXX-XXXX-XXXX

openstack secret consumer remove --service-type=image --resource-type=image \
    --resource-id=XXXX-XXXX-XXXX-XXXX

The secret delete command should be changed to take a –force parameter:

openstack secret delete --force {secret_uuid}

This command should return an error when a secret has one or more consumers and the –force flag is not used:

openstack secret delete {secret_uuid_with_consumers}
ERROR: Secret has one or more consumers.  Use --force to delete anyway.

These changes will require a new Major version for python-barbicanclient because the default –force=False option could cause some scripts to break in certain scenarios where secrets are currently being deleted that do have consumers associated with them.

Other end user impact

Currently there is no other impact to the end user other than the CLI changes listed above. In the future, when a barbican-ui for Horizon is developed, it should use the consumers to present confirmation dialogs to the user when deleting Secrets which have consumers.

It should be noted that Deleting Secrets in the Barbican REST API has not changed, and a client using the API directly will be able to delete a secret regardless of the presence of consumers.

Performance Impact

Deleting secrets using the CLI or the Python client will be affected as we will likely need to perform additional requests to the API to get the list of consumers for a secret before sending a DELETE request.

Other deployer impact

When python-barbican changes are merged, some automation scripts that use secret deletion may break if the secrets being deleted have consumers.

Any automation scripts should be updated to use the –force flag if needed.

Developer impact

Developers of other projects that want to make use of this feature will need to use python-barbicanclient to integrate with the Key Manager service.

Implementation

Assignee(s)

Primary assignee:: Douglas Mendizábal (OFTC: redrobot) <dmendiza@redhat.com>
Other contributors:: Moisés Guimarães (OFTC: moguimar) <moguimar@redhat.com> Grzegorz Grasza (OFTC: xek) <xek@redhat.com>

Work Items

Implement Model changes and database migration
Implement API changes
Implement python-barbicanclient changes (both python client and CLI)

Dependencies

None.

Testing

Tempest test cases should be added to test adding/removing Secret Consumers using a service-user that is not barbican.

Documentation Impact

All API changes should be documented in the API reference, as well as the API Guide.

References

[1] Container Consumers API: https://docs.openstack.org/barbican/stein/api/reference/consumers.html

Barbican Train PTG Etherpad: https://etherpad.openstack.org/p/barbican-train-ptg

Add Support For Mutable Generic Containers Resource

Thu, 09 Apr 2020 00:00:00

The URL of the associated launchpad blueprint:

https://blueprints.launchpad.net/barbican/+spec/api-containers-add-put

The original intent of the ‘generic’ container type was to support arbitrary secrets to be collected and referenced, similar to a file system folder. Hence it makes sense for an already-created ‘generic’ container to support changing this collection after the fact. This blueprint details how this API would look.

Problem Description

Currently Barbican allows a ‘generic’-typed container to be created via a POST call, but once that container is created it is not possible to modify it to change which secrets it contains. The ‘generic’ container was intended to be a convenient way to collect related secrets and as such should allow for modification after it is created.

For example continuous integration and deployment systems might wish to collect all secrets for a given environment such as database passwords and access tokens, for use by automation scripts when servers are created in the environment later. It would be convenient if each environment could have its own Barbican container that automation scripts could reference to retrieve secrets (by their fixed key names) as part of booting up servers in that environment. If secrets are later rotated or updated, the secret references for a given key name could be updated in the container, with no need to update the automation scripts.

Attempts to update containers of other types will be rejected.

Proposed Change

This blueprint calls for adding new sub-resources to generic containers. The API impact section details what these calls look like relative to clients. The actual service implementation is straightforward though, allowing clients to provide additional secrets or delete individual secrets by sending a POST or DELETE request to the sub-resources in a ‘generic’ container.

Alternatives

A previously-accepted but not implemented version of this blueprint called for adding PUT support for containers whereby clients must specify all secrets to be held in the container. For example when adding a secret to an existing container the PUT body would have to list all existing secrets in addition to the new secret. However, during the Newton cycle we discussed that this could be error-prone due to potential race conditions.

Utilize a PATCH call to provide partial updates to containers. This adds complexity to API processing, especially if JSONPatch [1] is used. If this feature is deemed necessary, that should be proposed as a separate blueprint.

Data model impact

None.

REST API impact

The following documentation specifies the proposed container sub-resource calls, used to add or remove a secret from an existing ‘generic’-type container.

The access policy for this call is similar to the container resource’s POST call, so users with the Barbican ‘admin’ or ‘creator’ role can modify containers.

POST /v1/containers/{container_uuid}/secrets

Add a secret to an existing container. This is only supported on generic containers.

Request Attributes

Name	Type	Description
name	string	(optional) Human readable name for identifying your secret within the container.
secret_ref	uri	(required) Full URI reference to an existing secret.

Request:

POST /v1/containers/{container_uuid}/secrets
Headers:
    X-Project-Id: {project_id}

Content:
{
    "name": "private_key",
    "secret_ref": "https://{barbican_host}/v1/secrets/{secret_uuid}"
}

Response:

{
    "container_ref": "https://{barbican_host}/v1/containers/{container_uuid}"
}

Note that the requesting ‘container_uuid’ is the same as that provided in the response.

HTTP Status Codes

In general, error codes produced by the containers POST call pertain here as well, especially in regards to the secret references that can be provided.

Code	Description
201	Successful update of the container
400	Missing secret_ref
401	Invalid X-Auth-Token or the token doesn’t have permissions to this resource

DELETE /v1/containers/{container_uuid}/secrets

Remove a secret from a container. This is only supported on generic containers.

Request:

DELETE /v1/containers/{container_uuid}/secrets
Headers:
    X-Project-Id: {project_id}

Content:
{
    "name": "private key",
    "secret_ref": "https://{barbican_host}/v1/secrets/{secret_uuid}"
}

Response:

204 No Content

HTTP Status Codes

Code	Description
204	Successful removal of the secret from the container.
400	Missing secret_ref
401	Invalid X-Auth-Token or the token doesn’t have permissions to this resource
404	Specified secret_ref is not found in the container.

Alternative

Alternatively, we could define the removal of a secret as a call to a resource that includes the secret_uuid as part of the URI, for example:

DELETE /v1/containers/{container_uuid}/secrets/{secret_uuid}

However, this would require the client to parse out UUIDs from secret URIs to be able to construct the correct URI for deletion. Because of this reason the DELETE with a body described above should be implemented instead.

Security impact

The proposed change does not pose a security impact because (1) it does not alter underlying secrets or their access restrictions, and (2) only ‘generic’- type containers can be modified hence sensitive grouped secrets such as ‘certificate’- and ‘asymmetric’-type containers remain immutable. Please also see the Performance Impact section for advice on rate limiting calls such as the one proposed in this blueprint to avoid denial of service attacks.

Notifications & Audit Impact

The proposed new POST and DELETE container request should be logged and audited the same as any other REST call to Barbican, and therefore does not need to be called out specifically in this blueprint.

Other end user impact

The Barbican Python client needs to be modified as well.

Performance Impact

As no cryptographic operations are needed for this blueprint, with only database references to secrets changed, the proposal should have minimal impact on performance. However, the call itself is not quota limited, so deployers might wish to utilize a rate limiting application in front of their Barbican API nodes, such as Repose [2].

Other deployer impact

No configuration or dependency changes are required to utilize the proposed operation, but as mentioned in the Performance Impact section above if rate limiting is deployed, the PUT operation on the containers resource should be limited as well to avoid denial of service attacks.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: TBD
Other contributors:: TBD

Work Items

The following work items are required to implement this blueprint:

Update container controllers to add the new POST and DELETE sub-resources.
Add a new policy entry to etc/policy.json for the new operations
Add a positive test to modify a ‘generic’-type container, and a negative test prove that a non-‘generic’-type container cannot be modified.
Add Barbican Python client support for the new feature.
Add sphinx documentation for the new POST and DELETE actions.

Dependencies

None.

Testing

See the Work Items section above for details on what to test. No special 3rd party systems are required to test the proposed functionality.

Documentation Impact

See the Work Items section above for details on what to document.

References

[1] http://jsonpatch.com/ [2] http://openrepose.org/

Date Filters

Thu, 09 Apr 2020 00:00:00

https://blueprints.launchpad.net/barbican/+spec/date-filters

Problem Description

In an effort to keep track of the changes being made to secrets or upcoming expiration dates, users such as IT Auditors or Operations Engineers need to be able to filter queries sent to the API based on the timestamped properties of a secret. These users require new filters to be able to query information about their secrets such as:

Secrets expiring in the near future
Secrets created in a particular time range
Secrets updated in a particular time range

Currently, Barbican does not provide any filters for querying secrets by using the properties that hold timestamp values. So a user currently needs to iterate through their entire secret collection to be able to gather this data.

Proposed Change

The barbican /secrets resource should be enhanced to allow sorting and filtering based on a secret’s created, updated, and expiration properties as described in the API Working Group’s guideline for Pagination, Filtering, and Sorting

The filters should allow limiting of returned values to specific times and time ranges by using the equality (=), greater-than (gt), greater-than-or-equal (gte), less-than (lt), and less-than-or-equal (lte) operators.

The filters should also allow the ordering of return values by using the sort query string parameter with both ascending (asc) and descending (desc) directions for the specified sort key.

Values passed in to these query parameters are assumed to be give in UTC time using the extended format described in ISO 8601. The UTC zone designation represented by appending the “Z” character will be required. Values that do not include the zone designation will result in an error response with status code 400. Values that specify a time offset from UTC will also result in a 400 error response even if the offset is zero to specify UTC.

Alternatives

Requiring the zone designation for UTC (“Z”) may be too stringent of a requirement. One alternative would be to accept time values without the “Z” zone designation and just assume that the values are all UTC.

Data model impact

This change will not affect the data model, since all values to be used in the filtering are already part of the data model.

REST API impact

Additional query parameters will be available for the GET /v1/secrets resource as described above.

Examples:

List secrets expiring in the next week (assuming current time is June 8, 2016 20:00 UTC) and sort by secrets expiring soonest:

GET /v1/secrets?expiration=gt:2016-06-08T20:00:00Z,lt:2016-06-15T20:00:00Z&sort=expiration:asc

List secrets created in the previous week assuming same current time as above:

GET /v1/secrets?created=gt:2016-06-01T20:00:00Z,lt:2016-06-08T20:00:00Z

List secrets updated in the previous week assuming same current time as above:

GET /v1/secrets?updated=gt:2016-06-01T20:00:00Z,lt:2016-06-08T20:00:00Z

The format of the data in the request and response will not change.

Security impact

This change should not impact the security of barbican, since it just provides a way to narrow the results of querying the API.

Notifications & Audit Impact

This change does not impact the notifications or auditing features of barbican.

Python and Command Line Client Impact

Both python-barbicanclient and the plugin for the unified CLI will need to be updated to provide a way for clients to use these new filters.

Other end user impact

No additional end user impact should result from this change.

Performance Impact

The implementation should not use additional database queries, but rather use the existing queries so that performance is not negatively impacted.

Other deployer impact

This change should not affect deployers.

Developer impact

Developers should not be impacted since these filters are optional. However, developers could make use of these filters when applicable to their use cases.

Implementation

Assignee(s)

Primary assignee:: Douglas Mendizábal
Other contributors:: Joe Savak

Work Items

Update controllers to accept and make use of the new filters
Update documentation to include the newly added filters
Update python-barbicanclient to make use of the new filters
Update the unified CLI plugin to make use of the new filters

Dependencies

None.

Testing

New functional and unit tests that exercise the new functionality should be included in the implementation of this spec.

Documentation Impact

The API change will need to be updated in the API reference as well as the user guide.

References

API Working Group’s Guideline for Pagination, Filtering and Sorting:

https://git.openstack.org/cgit/openstack/api-wg/tree/guidelines/pagination_filter_sort.rst

ISO 8601 in Wikipedia:

https://en.wikipedia.org/wiki/ISO_8601

Snakeoil CA Certificate Manager Plugin

Wed, 16 Jan 2019 00:00:00

https://blueprints.launchpad.net/barbican/+spec/barbican-snakeoil-ca

A certificate management plugin which auto approves all requests can be very useful for testing and development deployments. This is especially useful for tools (such as TripleO) which hope to use Barbican as an abstraction layer for certificate creation in production deployments but still require a configuration that will work in CI.

Problem Description

In order to use the certificate management interface in a development or testing environment a user must (currently) configure dogtag or another backend in a way which auto-approves certificate requests. This is an unnecessary burden in many environments (such as CI pipelines) which require none of the advanced features these tools provide.

Proposed Change

Create a plugin which implements the certificate manager interface and will sign any certificate request sent to it. Allow for configuration of an on-disk CA certificate and key. If no certificate or key is found at that path, a new CA certificate and key will be created and stored at that path encoded in PEM file format. If no certificate or key path is specified then a new CA key and certificate will be created and stored only in memory.

Serial numbers for certificates will be generated from UUIDs.

Alternatives

By limiting the scope of this plugin to non production uses, and only certificate signing (no revokation, for example) there ends up being a small amount of code using the PyOpenSSL library (an ffi for openssl). The amount of code here is comprable to the amount required to shell out to a tool like CA.pl and is considerable easier to read, understand, and test. If the scope of this tool was ever expanded then we may want to reconsider using PyOpenSSL.

Data model impact

None

REST API impact

A certificate order:

{
    "type": "certificate",
    "meta": {
        "request_data": "PEM encoded X509 Request with optional X509Name",
    }
}

Security impact

This feature is explicitly not intended for use in any type of production environment. This will be explicitly documented as such and is also named ‘Snakeoil’ to (hopefully) make this as apparent as possible.

Notifications & Audit Impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None. This feature should never be deployed to production.

Developer impact

This should ease the setup of development environments for certificate management. We may want to consider documenting setting up this type of environment for new users.

Implementation

This is implemented using PyOpenSSL and the motivation for this is explained in the ‘Alternatives’ section.

https://review.openstack.org/#/c/140575

Assignee(s)

Primary assignee:: greghaynes <greg@greghaynes.net>

Work Items

Implement the plugin. This plugin will implement the general certificate request API so barbican-client work will be completed in that fashion.

Dependencies

None

Testing

Documentation Impact

Documenting this feature for setting up development and testing environments could be useful but is not required.

References

Add cron job garbage collector for barbican database

Sun, 13 Jan 2019 00:00:00

Blueprint:

https://blueprints.launchpad.net/barbican/+spec/clean-db-soft-deletes

Problem Description

For all tables in the barbican database that use soft deletion, the entries in the table are not removed but flagged for deletion. We want to create a configurable command that will be called by a cron job to go through the database and delete entries where the deleted flag equals 1 or if the entry is not needed anymore (like zombie projects).

Proposed Change

Create a barbican command called ‘barbican-manage db cleanup’ that will be run by a cron job to clean up soft deletions in the barbican database. The command will utilize python sqlalchemy libary calls to go through the database and remove entries where deleted equals 1. The only tables that will be checked are tables which have a corresponding class in ‘barbican/model/models.py’ and the class has ‘SoftDeleteMixIn’ as a parent class. Any other table will not be checked due to the assumption that soft deletes are not used. The script will dynamically load in the table classes defined in models.py to a list. Once the list is created, the script will then check the configuration in barbican.conf to add or remove specific models.

An example cron job file will be provided so that the administrator can modify the contents to call the python script in intervals to their liking. It is intended that the barbican deployer calls the barbican-mange db cleanup command in a cron job.

The command is configurable for these options: minimum number of days to keep since soft deletion (default is 90 days) delete zombie projects (no associated resources and default is true) models to force check (no inheritence of SoftDeleteMixIn and default is None) models to ignore (have inheritence of SoftDeleteMixIn and default is None) log levels for log (default is INFO). location of cleanup log (default is None).

Note

note on zombie projects: Barbican will create a project entry anytime a user tries to make a request. If there are no resources associated with a project, then that project database entry can be removed.

Note

note on audit logging: DELETEs via API are already logged by the CADF middleware. So there should be no reason to add functionality to make an audit log of who performed deletions.

Note

For this iteration, orders will not be touched due to different available SLAs that barbican offerings could have and should be discussed further. Certificate orders should not be deleted because certificate renewals require the old orders.

Configuration - Option parameters can be passed with the barbican manage command to customize cleanup. For example: ‘barbican db cleanup –min_num_days_to_keep_softdeletes 30’

The command can also pass in a configuration file for a command. ‘barbican db cleanup -f “/etc/barbican/barbican.conf”

The option parameters will have higher precedence than the variables in the conf file if both the conf file and other options are given as parameters. So the barbican admin can give only option parameters, only a conf file, or both. The default value for the configuration file will be ‘None’.

Below is an example of how a configuration file can be configured:

[db_cleanup] #how long to keep the entry in the database after a soft deletion minimum_num_days_to_keep_soft_deletions = 90

#remove projects that have no associated resources cleanup_unassociated_projects = True

#table classes to force check e.g., CertificateAuthority,SecretACL,.. table_classes_to_force_check = None

# If a secret is expired, then it should be soft deleted soft_delete_expired_secrets = True

#table classes to ignore, their parent class is SoftDeleteMixIn #e.g, Project,Secret… table_classes_to_ignore = None

# Show more verbose log output (sets INFO log level output) verbose = True

# Show debugging output in logs (sets DEBUG log level output) #debug = True

#log file to store information on deletions db_cleanup_log_file = “/var/log/barbican/barbican-cleanup.log”

Alternatives

Add a REST API parameter for delete operations where users can specify hard deletion.
Leave it to the database admins to clean up their database manually.
Leave it to the database admin to create database triggers for cleanup.
Create a daemon instead of a cron job which is similar to the glance scrubber.

Data model impact

The data models should not change.

The tables that are checked are tables which have a class that has SoftDeleteMixIn as a parent class. Other tables will be ignored unless configured. Entries where ‘deleted == 1’ will be removed.

REST API impact

None

Security impact

The barbican manage command and cron job file should have valid barbican admin user permissions. No global user should be able to modify/run the script.

Notifications & Audit Impact

A log should be kept with total entries deleted for each table. The log location will be configurable.

If the level of logging is debug, then the table name, id of entry, soft deletion date, and hard deletion date will be recorded.

Since user deletions are already logged by CADF middleware, it will not be necessary to log who made the soft deletions.

Python and Command Line Client Impact

None

Other end user impact

None

Performance Impact

Since this is a recurring find and delete operation on a database, this may take up a great amount of compute cycles. It will be up to the admin/operator of Barbican to find the best time to run, what interval to run the job, and configure how much to delete. The admin will have to customize the cron job entry to their liking.

Other deployer impact

The deployer will have to configure the cron job file.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: edtubill
Other contributors:: None

Work Items

Each phase listed below is intended to be a CR. (Phase 2 will be split into different CRs)

Phase 1: Create simple barbican-manage command to go through the database and delete everything with ‘deleted == 1’. Create unit and functional tests for this first phase. Phase 2: Add logic to read configuration file and go through database based on the configuration. Create additional unit and functional tests. Phase 4: Create example cron job configuration files Phase 5: Document the cron job garbage collector on Barbican Wiki

Dependencies

None

Testing

Unit tests must be written for internal component testing. Functional tests must be created based on different possible configurations. The functional tests will check the entries of the database directly.

Documentation Impact

Wiki documentation will be created for usage of the ‘barbican-manage db cleanup’ command, how to configure, and examples on how to setup a cron job will be provided.

If a user does barbican-manage db cleanup –help, then usage documentation should be shown.

References

https://blueprints.launchpad.net/barbican/+spec/clean-db-soft-deletes

Add auditing capability via CADF based notification events

Tue, 27 Nov 2018 00:00:00

https://blueprints.launchpad.net/barbican/+spec/audit-cadf-events

This proposal is to add auditing capability in Barbican using CADF (Cloud Audit Data Federation) specification. The idea is to identify auditable attributes and construct audit data record as per CADF event specification and provide delivery option via OpenStack notification framework to interested services and consumers.

DMTF CADF standard provides auditing capabilities for compliance with security, operational and business processes. See References section in end for more details around CADF specification.

Problem Description

Large number of enterprise faces challenge when moving to OpenStack cloud. And most of these challenges have nothing to do with OpenStack service capabilities. Instead, the challenges are in terms of auditing and monitoring their workload and data in accordance with their strict corporate, industry or regional policies and compliance requirements (FIPS-140-2, PCI-DSS, SoX, ISO 27017 etc). Barbican, being a security component in a cloud, has similar auditing requirement.

In addition, Barbican has this concept of soft deletes where it keeps resources in database even after they are marked deleted and are no longer used in live systems. The idea is to have a sense of audit trail of its resources state. This approach does not provide much value in terms of audit other than when it was marked deleted.

Proposed Change

To add auditing capability in Barbican, proposal is to leverage a existing standard instead of creating own semantics and audit data model. In OpenStack ecosystem, number of services (Ceilometer, Keystone) are using CADF based event data model to report activities on its resources. Using CADF standard for auditing will allow consistent reporting across services and will allow customers to use common audit tools and processes for their audit data.

The CADF model can answer critical questions about activity or event happening on rest resources in a normative manner using CADF’s seven W’s of auditing (What, When, Who, On What, Where, From Where, To Where). See Auditing Event Details section below.

The overhead of adding audit data interaction can be minimized by publishing audit event as OpenStack notifications where event publisher does not have to wait for response/ acknowledgment. This way audit event delivery is decoupled and can still benefit from messaging infrastructure durability and delivery guarantees.

Auditing Event Details

Audit event data is constructed using who initiated request, request outcome, resource operated on, resource identifier and event type information. Following is sample of seven W's of auditing values for Barbican rest resources.

W Component

Resource

CADF Properties

Possible Values

What

POST/PUT/ DELETE v1/secrets

event.type event.outcome

activity/monitor/control success/failure/pending

When

event.eventTime

{event generation timestamp}

Who

initiator.id initiator.type initiator.project_id

{token user id} service/security/account/user {scoped token project id}

From Where

initiator.host initiator.agent

environemnt REMOTE_ADDR environment HTTP_USER_AGENT

On What

target.id target.type

resource id (secret/container) data/keymgr/secret

Where

observer.id observer.type

target service/keymgr

To Where

W Component	Resource	CADF Properties	Possible Values
What	POST/PUT/ DELETE v1/secrets	event.type event.outcome	activity/monitor/control success/failure/pending
When		event.eventTime	{event generation timestamp}
Who		initiator.id initiator.type initiator.project_id	{token user id} service/security/account/user {scoped token project id}
From Where		initiator.host initiator.agent	environemnt REMOTE_ADDR environment HTTP_USER_AGENT
On What		target.id target.type	resource id (secret/container) data/keymgr/secret
Where		observer.id observer.type	target service/keymgr
To Where

CADF extension properties (mentioned below) can be used to capture domain specific data available as part of rest request. Currently there is no plan to add these properties as this may need per resource awareness in common decorator logic.

event.attachments (structured or unstructured data)
event.tags (domain specific identifiers/ classifications)

pyCADF changes

Creation of audit event requires oslo pyCADF library. pyCADF library needs to be updated to reflect Barbican specific resource types. As per pyCADF contributors, either use existing types or create very few new generic types. Following Barbican specific resource types are going to be added in pyCADF resource taxonomy.

data/keymgr/secret
data/keymgr/container
data/keymgr/order
data/keymgr - general placeholder for all other remaining resourcs.
service/keymgr

Above types are going to be added in pyCADF taxonomy as mentioned in link below.

pyCADF resource taxonomy https://github.com/openstack/pycadf/blob/master/pycadf/cadftaxonomy.py#L111

Audit Event Generation

For API requests, audit event are going to be generated using audit middleware approach. This middleware is now available as part of keystonemiddleware (> 1.5) library. This library has Keystone middleware which is used by Barbican for Keystone token validation.

Audit middleware is going to create two events per Barbican REST API invocation. One with information extracted from request data and the second correlated one with request outcome (response). The mapping information is going to be managed via barbican specific configuration file which is going to be similar to files described in following pycadf samples link. https://github.com/openstack/pycadf/tree/master/etc/pycadf

For asynchronous task processing workers, audit event is constructed using task related data. It can be implemented as a decorator which can be added to each task common methods (handle_processing, handle_success and handle_error) or can be added on base task process method. This audit event is going to be published as notification to same queue which is used by audit middleware as well.

Audit Event Delivery

Internally audit middleware uses an oslo messaging based notifier to publish CADF events to configured messaging infrastructure. Audit decorators will use similar approach.

Audit middleware is added to Barbican request pipeline via additional filter in barbican-api-paste.ini where related audit mapping file path is defined. Delivery of audit event via decorator needs to be configurable as developer boxes may not necessarily have the needed setup. By default, audit delivery as notification is going to disabled.

The oslo messaging framework supports publish of this audit data to messaging queue via ‘messagingv2’ driver or can be written to log files via its ‘log’ driver.

Alternatives

We could rely on Barbican existing logging but that does not provide a complete and consistent picture of service audit data. Non-standard logging means that cloud provider need service specific audit tools to aggregate and analyze the logs.

Security impact

This improves security in the stack by providing audit capability. This will help in addressing some of compliance requirements expected from Barbican service.

Notifications & Audit Impact

Will have additional notification capability to publish audit events.

Other end user impact

None

Performance Impact

Audit events are published as notifications to queue and does not have to wait for response/acknowledgment so overall associated overhead should be very minimal. When notifications are written to log files, related overhead should still be low and can be comparable to logic of adding 2 log statements.

Other deployer impact

To enable audit event delivery via notifications, deployer will need to change default configuration.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: arunkant

Work Items

Add new blueprint and update oslo pyCADF library.
Define pipeline filter with audit map configuration file barbican_api_audit_map.conf
Add new decorator to create CADF event data for asynchronous worker processing logic. Add decorator to related worker task methods.
Add ability to turn on audit event generation. By default it needs to be off.

Dependencies

pyCADF library (with barbican specific updated taxonomy).

Testing

The new unit tests are going to be added for middleware and order processing flow. For middleware unit tests, configuration override is needed for paste and api ini files. Oslo test driver https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/notify/_impl_test.py can be used to verify message content in addition to mock patched methods approach.

Documentation Impact

CADF event usage should be documented. For order processing flow, document what audit related changes, if any, are needed to add audit support for new order types.

References

CADF Working Group. http://www.dmtf.org/standards/cadf
CADF representation for OpenStack http://www.dmtf.org/sites/default/files/standards/documents/DSP2038_1.0.0.pdf
Audit Middleware https://docs.openstack.org/developer/keystonemiddleware/audit.html
PyCADF developer docs https://docs.openstack.org/developer/pycadf/
CADF event model and taxonomies https://wiki.openstack.org/w/images/e/e1/Introduction_to_Cloud_Auditing_using_CADF_Event_Model_and_Taxonomy_2013-10-22.pdf
Ceilometer CADF support https://wiki.openstack.org/wiki/Ceilometer/blueprints/support-standard-audit-formats

Refactor Client Entity Models

Wed, 14 Nov 2018 00:00:00

https://blueprints.launchpad.net/python-barbicanclient/+spec/client-refactor-models

The current Entity Models in the client are a bit awkward to use. This blueprint proposes refactoring some functionality to make the API more usable and consistent.

Problem Description

The Entity Models in barbicanclient should be refactored to provide a more Pythonic api. This refactor will make the existing Entities consistent with the recently approved Containers blueprint. [1]

Proposed Change

Refactor existing Models to provide methods for actions that affect a single entity inside the entity class. This will allow for worflows that only affect a single entity to be completed without the need for a reference to the corresponding EntityManager subclass.

The Secret entity should be refactored to add a store() method and a payload property:

from barbicanclient import client

# Set up client connection
connection = client.Client(tenant_id="1", endpoint=ENDPOINT, insecure=True)

# Create a new Secret
my_secret = connection.secrets.Secret(name="My secret name",
                                      payload="the secret sauce")
my_secret.store()

# Alternatively set Secret properties instead of passing args
my_secret = connection.secrets.Secret()
my_secret.name = "My secret name"
my_secret.payload = "the secret sauce"
my_secret.store()

Similarly, Orders should allow both args to the constructor as well as setting properties directly. We should also add a submit() method to submit the order to the API:

from barbicanclient import client

# Set up client connection
connection = client.Client(tenant_id="1", endpoint=ENDPOINT, insecure=True)

# Create and submit a new Order
my_order = connection.orders.Order(
    name="My Order",
    payload_content_type="application/octet-stream"
    algorithm="AES",
    mode="CBC",
    bit_length=256,
    expiration=None
)
my_order.submit()

# Alternatively set the Order properties instead of passing args
my_order = connection.orders.Order()
my_order.name = "My Order"
my_order.payload_content_type = "application/octet-stream"
my_order.algorithm = "AES"
my_order.mode = "CBC"
my_order.bit_length = 256
my_order.expiration = None
my_order.submit()

Listing entities should still be handled via the corresponding EntityManager. The ability to decrypt a secret, however, should be moved to the Secret class, and removed from the SecretManager.

Retrieving entities should be moved from the EntityManager (replacing the get() function) to the Entity constructor. For example:

my_secret = connection.secrets.Secret(secret_ref=SECRET_REF)
my_order = connection.orders.Order(order_ref=ORDER_REF)

Deleting entities can either be done with the existing EntityManager delete(entity_ref) or with a new Entity function, delete(). An example using a Secret:

# New method
my_secret = connection.secrets.Secret(secret_ref=SECRET_REF)
my_secret.delete()

# Old way still works
connection.secrets.delete(secret_ref=SECRET_REF)

Alternatives

We could continue to use the objects as they currently exist.

Also note that the Orders functionality will need to be revisited once the Typed Orders implementation lands. [2]

Data model impact

None

REST API impact

None

Security impact

None

Notifications & Audit Impact

Logging should be done in a manner consistent with the rest of the library.

Other end user impact

This change will require rewriting how Secret objects are consumed, and will require a new major version for the client library.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Blueprint Draft: Douglas Mendizábal (redrobot) Implementation: Adam Harwell (rm_work)

Work Items

Refactor Secret entity
Refactor Order entity

Dependencies

None

Testing

Testing should be consistent with existing testing in the library.

Documentation Impact

Common workflows will have to be updated to give examples on how to use the refactored classes.

References

Containers in the Client etherpad: https://etherpad.openstack.org/p/python-barbicanclient-containers Containers Blueprint: https://specs.openstack.org/openstack/barbican-specs/specs/juno/client-add-containers.html

Consume Keystone Project Delete Events

Wed, 14 Nov 2018 00:00:00

https://blueprints.launchpad.net/barbican/+spec/consume-keystone-events

In most of openstack deployments, barbican is expected to be integrated with keystone for its resource authorization needs. Keystone and barbican share keystone project as a common entity reference. Which means, project needs to exist in keystone before barbican resources (e.g. secrets and containers) can be added for that project. But when project is deleted in keystone, there is no action taken by barbican to reflect the change and it keep maintaining that project dangling resource references as-is. This blueprint is addressing related issues in this area.

Problem Description

Barbican resources are managed at keystone project level. As part of barbican resource creation, barbican tenant (a.k.a. project) is mapped to keystone project and resource is associated with the tenant. Over time, many more resources are added to the tenant. Now when project is deleted from keystone, currently barbican stays unaware of that change and keeps its resources as-is. Those project resources are now invalid and are kind of dangling references to a non existing project.

As barbican feature set expands, the amount of such tenant resources will grow over time which can negatively impact performance of db operations.
Hardware Security Modules (HSM) has a upper limit on the number of key encryption keys it can store. These numbers are correlated to number of projects in barbican. Without project status awareness, related invalid keys cleanup cannot be done to reclaim space for new keys in HSM.
In a corner case of PKI token service usage, for a small time window, unexpired PKI token(s) can access a deleted project resource as it has yet to fetch and process related revocation events from keystone. By making barbican aware of deleted projects, even those PKI token would not have access to barbican resources. Just to be clear, in general PKI token would not have this issue if revocation events are fetched pretty frequently.

Proposed Change

Keystone publishes notifications whenever a project is created, updated and deleted. Proposal is to add notification listener in barbican to consume keystone public events.

Keystone event payload has project id as resource_info and barbican event listener logic will propagate it to make changes on resources associated with that project.

Following is keystone sample notification on project delete:

Message: ctxt = [{}], metadata = [{'timestamp': u'2014-06-12 00:20:03.584997', 'message_id': u'1ade0b2b-1584-48b9-a026-64bd06659baf'}]
Message: severity = [info], publisher_id = [identity.arunkant-uws], event_type = [identity.project.deleted]
Message: payload = [{u'resource_info': u'00ac7ea2a1a3486284c8e2af27b7bc9e'}]

Event listener will use oslo messaging library which provides a abstraction to allow usage of different type of messaging transport.

Deletion of a keystone project will invoke delete on related secret(s), container(s), tenant entity. Currently entity delete is a soft delete as specific fields are marked deleted but the record remains in datastore. In the event that a project is disabled, Keystone will invalidate the tokens for that project and the secrets owned by that project will be inaccessible. The secrets owned by Barbican won’t be soft deleted but they will be unreachable. Also, the keystone delete project and the barbican delete project should both generate CADF audit events and ideally a correlation identifier should be used between these two events. Also an extension field in the audit record should be used to declare the type of delete (soft or hard).
Keystone sends various notification events based on its own resource types (e.g. user, project, role etc.) and type of operation (e.g. created, updated, deleted etc.). This proposal scope is to act on keystone project delete event only.
Barbican will add a queue to exchange to listen keystone notifications. Queue specific attributes e.g. exchange name and type, queue name, binding pattern etc. are going to be configurable in barbican configuration. Some of these attribute values need to align with keystone notification configuration.
Keystone currently publishes notification only with info severity. The binding pattern of notifications.* will handle that.
Barbican queue is going to be durable to survive broker restart.
This feature is going to be configurable and not enabled by default similar to keystone authentication via pipeline configuration.
Notifications are acknowledged only when barbican listener has processed event information. Auto-acknoledgement is going to be false.
Barbican connection to message queue is authenticated using username and password. These credentials are going to be defined as part of barbican configuration.
Depending on SSL support available in oslo messaging for a specific transport provider, related configuration is going to be added in barbican configuration.
In initial version, there is no trust established between messaging queue and barbican instance(s). KDS (Key distribution service, Kite) may help around this in future once other services start using it for this kind of functionality.
Changes on resource side will be committed only when complete event processing is done otherwise change will be rolled back.
Once such event support is available in barbican, those events can be propagated to barbican’s external sub-components if it makes sense. One example of such sub-comonent can be clean up of key encryption keys in HSM (when this feature is added in future).

Alternatives

There can be other ways to get keystone project information like polling model or accessing keystone datastore directly but there are not scalable and may introduce tight coupling between services. Events based asynchronous approach is better as keystone notifications are primarily intended for other openstack services to take action based on events e.g. cleanup of their resources.

Data model impact

There is no data model impact as we are going to utilize existing entity soft delete functionality.

REST API impact

None

Security impact

A keystone public event does not contain sensitive data by itself. Event data has information about operation and resource id.

As implication of those events will result in deleting barbican resources so clear logging of events and action taken will be needed.
Once auditing support is added in barbican, the project delete event will be one type of audit activity.
Message queue connection information needs to be secured similar to other external resources connections e.g. database.

Notifications & Audit Impact

A new notification listener is added. Related auditing information is going to be added to system.

Other end user impact

None

Performance Impact

There should not be performance impact other than new message handling server is added on same host system.

As result of this change, barbican number of connections to db system may increase depending on load of actionable events. Generally the number of such events will be quite less considering keystone projects are likely not to deleted regularly.
Depending on keystone event activity, a deployer can choose to enable notification listener on some of barbican instances assuming there is pool of barbican instances in a setup.

Other deployer impact

This feature needs to be enabled as default configuration will have it disabled.
Related notification listener configuration needs to be configured as per deployer’s existing messaging infrastructure setup.
A message handling server is added, as a new process, to transport notification events from queue to barbican.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Arun Kant <arun.kant@hp.com>
Other contributors:: ??

Work Items

Need to add notification listener configuration
Implement notification listener and message handling server using oslo messaging packages.
Identify actionable events and process to make changes on barbican resources.
Implement these actions as a unit and rollback in case of a processing error. One option, needs to be investigated, is to do all operations within a SQLAlchemy session.
If missing, add checks in order and other resource API so that create and update of resource is not allowed for deleted tenants (a.k.a. keystone projects)

Dependencies

None

Testing

Add any integration test provided needed messaging support is available.

Documentation Impact

There will be additional documentation around notification listener configuration. Possibly a new option similar to following link

https://github.com/cloudkeep/barbican/wiki/Barbican-Options:-authentication-with-Keystone

References

Add worker retry and future updates support

Wed, 14 Nov 2018 00:00:00

Launchpad blueprint: https://blueprints.launchpad.net/barbican/+spec/add-worker-retry-update-support

The Barbican worker processes need a means to support retrying failed yet recoverable tasks (such as when remote systems are unavailable) and for handling updates for long-running order processes such as certificate generation. This blueprint defines the requirements for this retry and update processing, and proposes an implementation to add this feature.

Problem Description

Barbican manages asynchronous tasks, such as generating secrets, via datastore tracking entities such as orders (currently the only tracking entity in Barbican). These entities have a status field that tracks their state, starting with PENDING for new entities, and moving to either ACTIVE or ERROR states for successful or unsuccessful termination of the asynchronous task respectively.

Barbican worker processes implement these asynchronous tasks, as depicted on this wiki page: https://github.com/cloudkeep/barbican/wiki/Architecture

As shown in the diagram, a typical deployment can include multiple worker processes operating in parallel off a tasking queue. The queue invokes task methods on the worker processes via RPC. In some cases, these invoked tasks require the entity (eg. order) to stay PENDING, either to allow for follow on processing in the future or else to retry processing due to a temporary blocking condition (eg. remote service is not available at this time).

The following are requirements for retrying tasks in the future and thus keeping the tracking entity in the PENDING state:

R-1) Barbican needs to support extended workflow processes whereby an entity
     might be PENDING for a long time, requiring periodic status checks to
     see if the workflow is completed

R-2) Barbican needs to support re-attempting an RPC task at some point in
     the future if dependent services are temporarily unavailable

Note that this blueprint does not handle concurrent updates made to the same entity, say to perform a periodic status check on an order and also apply client updates to that same order. This will be addressed in a future blueprint.

Note also that this blueprint does not handle entities that are ‘stuck’ in the PENDING state because of lost messages in the queue or workers that crash while processing an entity. This will also be addressed in a future blueprint.

In addition, the following non-functional requirements are needed in the final implementation:

NF-1) To keep entity state consistent, only one worker can work on an
      entity or manage retrying tasks at a time.

NF-2) For resilience of the worker cluster:

    a) Any worker process (of a cluster of workers) should be able to
       handle retrying entities independently of other worker processes,
       even if these worker processes are intermittently available.

    b) If a worker comes back online after going down, it should be able to
       start processing retry tasks again, without need to synchronize with
       other workers.

NF-3) In the default standalone Barbican implementation, it should be
      possible to demonstrate the periodic status check feature via the
      SimpleCertificatePlugin class in
      barbican.plugin.simple_certificate-Manager.py.

The following assumptions are made:

A-1) Accurate retry times are not required:

    a) For example, if a task is to be retried in 5 minutes, it would be
       acceptable if the task was actually retried after more than 5
       minutes. For SSL certificate workflows, where some certificate types
       can take days to process, such retry delays would not be
       significant.

    b) Relaxed retry schedules allow for more granular retry checking
       intervals, and to allow for delays due to excessive tasks in queues
       during busy times.

    c) Excessive delays in retry times from expected could indicate that
       worker nodes are overloaded. This blueprint does not address
       this issue, deferring to deployment monitoring and scaling
       processes.

Proposed Change

This blueprint proposes that for requirements R-1 and R-2, the plugins used by worker tasks (such as the certificate plugin) determine if tasks should be retried and at what time in the future. If plugins determine that a task should be retried, then these tasks will be scheduled for a future retry attempt.

To implement this scheduling process, this blueprint proposes using the Oslo periodic task feature, described here:

https://docs.openstack.org/developer/oslo-incubator/api/openstack.common.periodic_task.html

A working example implementation with an older code base is shown here:

https://github.com/cloudkeep/barbican/blob/verify-resource/barbican/queue/server.py#L174

Each worker node could then execute a periodic task service, that invokes a method on a scheduled basis (configurable, say every 15 seconds). This method would then query which tasks need to be retried (say if current time >= retry time), and for each one issue a retry task message to the queue. Once tasks are enqueued, this method would remove the retry records from the retry list. Eventually the queue would invoke workers to implement these retry tasks.

To provide a means to evaluate the retry feature in standalone Barbican per NF-3, the SimpleCertificatePlugin class in barbican.plugin.simple_certificate_manager.py would be modified to have the issue_certificate_request() method return a retry time of 5 seconds (configurable). The check_certificate_status() method would then return a successful execution to terminate the order in the ACTIVE state.

This blueprint proposes adding two entities to the data model: OrderRetryTask and EntityLock.

The OrderRetryTask entity would manage which tasks need to be retried on which entities, and would have the following attributes:

1) id: Primary key for this record

2) order_id: FK to the order record the retry task is intended for

3) retry_task: The RPC method to invoke for the retry. This method could be
               a different method than the current one, such as to support
               a SSL certificate plugin checking for certificate updates
               after initiating the certificate process

4) retry_at: The timestamp at or after which to retry the task

5) retry_args: A list of args to send to the retry_task. This list includes
               the entity ID, so no need for an entity FK in this entity

6) retry_kwargs: A JSON-ified dict of the kwargs to send to retry_task

7) retry_count: A count of how many times this task has been retried

New retry records would be added for tasks that need to be retried in the future, as determined by the plugin as part of workflow processing. The next periodic task method invocation would then send this task to the queue for another worker to implement later.

The EntityLock entity would manage which worker is allowed to delete from the OrderRetryTask table, since per NF-1 above only one worker should be able to delete from this table. This entity would have the following attributes:

1) entity_to_lock: The name of the entity to lock ('OrderRetryTask' here).
                   This would be a primary key.

2) worker_host_name: The host name of the worker that has the
                     OrderRetryTask entity 'locked'.

3) created_at: When this table was locked.

This entity would only have zero or one records. So the periodic method above would execute the following pseudo code:

Start SQLAlchemy session/transaction
try:
    Attempt to insert a new record into the EntityLock table
    session.commit()
except:
    session.rollback()
    Handle 'stuck' locks (see paragraph below)
    return

try:
    Query for retry tasks
    Send retry tasks to the queue
    Remove enqueued retry tasks from OrderRetryTask table
    session.commit()
except:
    session.rollback()
finally:
    Remove record from EntityLock table
    Clear SQLAlchemy session/transaction

Lock tables can be problematic if the locking process crashes without removing the locks. The overall time a worker holds on to a lock should be brief however, so the lock attempt rollback process above should check for and remove a stale lock based on the ‘created_at’ time on the lock.

To separate coding concerns, it makes sense to implement this process in a separate Oslo ‘service’ server process, similar to the Keystone listener approach This service would only run the Oslo periodic task method, to perform the retry updating process. If the method failed to operate, say due to another worker locking resource, it could just return/exit. The next periodic call would then start the process again.

Alternatives

Rather than having each worker process manage retrying tasks, a separate node could be designated to manage these retries. This would eliminate the need for the EntityLock entity. However, this approach would require configuring yet another node in the Barbican network, adding to deployment complexity. This manager node would also be a single point of failure for managing retry tasks.

Data model impact

As mentioned above, two new entities would be required. No migrations would be needed.

REST API impact

None

Security impact

None

Notifications & Audit Impact

None

Other end user impact

None

Performance Impact

The addition of a periodic task to identify task to be retried presents an extra load on the worker nodes (assuming they are co-located processes to the normal worker processing, as expected). However, this process does not perform the retry work, but rather issues tasks into the queue to then evenly distribute back to the worker processes. Hence the additional load on a given worker should be minimal.

This proposal includes utilizing locks to deal with concurrency concerns across the multiple worker nodes that could be handling retry tasks. This can result in two performance impacts: (1) multiple workers might fight to grab the lock simultaneously leading to degraded performance for the workers that fail to grab the lock, and (2) a lock could become ‘stuck’ if a worker holding the lock crashes.

Regarding (1), locks are only utilized on the worker nodes involved in processing asynchronous tasks which are not time sensitive. Also, the time the lock is utilized will be very brief, just long enough to perform a query for retry tasks and to send those tasks to queue for follow on processing. In addition the periodic process of each worker node handles these retry tasks, so if the deployment of worker nodes is staggered the retry processes should not conflict. Another option is to randomly dither the periodic interval (eg. 30 seconds +- 5 seconds) so that worker nodes are less likely to conflict with each other.

Regarding concern (2) about ‘stuck’ locks, since the conditions which involve locks are either long-running orders that can suffer delays until locks are restored, or else are (hopefully) rare conditions when resources aren’t available, this condition should not be critical to resolve. The proposal does however suggest a means to remove stuck locks utilizing their created-at times.

Other deployer impact

The Barbican configuration file will need a configuration parameter to periodically run the retry-query process, called ‘schedule_period_seconds’, with a default value of 15 seconds. This parameter would be placed in a new ‘[scheduler]’ group.

A configuration parameter called ‘retry_lock_timeout_seconds’ would be used to release ‘stuck’ locks on the retry tasks table, as described in the ‘Proposed Change’ section above. This parameter would also be added to the ‘[scheduler]’ group.

A configuration parameter called ‘delay_before_update_seconds’ would be used to configure the amount of time the SimpleCertificatePlugin delays from initiating a demo certificate order to the time the update certificate method is invoked. This parameter would be placed in a new ‘[simple_certificate]’ group.

These configurations would be applied and utilized once the revised code base is deployed.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: john-wood-w
Other contributors:: Chelsea Winfree

Work Items

Add data model entities and unit tests, for OrderRetryTask and EntityLock
Add logic to SimpleCertificatePlugin per the Approach section, to allow demonstration of retry feature
Modify barbican.tasks.certificate_resources.py’s _schedule_retry_task to add retry records into OrderRetryTask table
Add Oslo periodic task support
Implement periodic method, that performs the query for tasks that need to be retried
Implement workers sending retry RPC messages back to the queue…see note below
Add new scripts to launch the Oslo periodic task called bin/barbican-task-scheduler.py and .sh, similar to bin/barbican-keystone-listener.py and .sh
Add to the Barbican Devstack gate functional tests a test of the new retry feature via the SimpleCertificatePlugin logic added above
Add logic to handle expired locks on the OrderRetryTask table

Note that for #6, the ‘queue’ and ‘tasks’ packages have to be modified somewhat to allow the server logic to send messages to the queue via the client logic, mainly to break circular dependencies. Again, see the example here for a working example of this server/client/retry processing.

Dependencies

None

Testing

In addition to planned unit testing, the functional Tempest-based tests in the Barbican repository would be augmented to add a test of the new retry feature for the default certificate plugin.

Documentation Impact

Developer guides will need to updated, to include the additional periodic retry process detailed above. Deployment guides will need to be updated to specify that a new process needs to executed (for the bin/barbican-task-scheduler.sh process).

References

None

Add Version Responses Consistent with Openstack

Wed, 14 Nov 2018 00:00:00

Launchpad blueprint: https://blueprints.launchpad.net/barbican/+spec/fix-version-api

Barbican currently returns one-off version information for GETs to the root resource. This blueprint calls for replacing that response with an OpenStack consistent response based on json-home (see http://http://tools.ietf.org/html/draft-nottingham-json-home-03), that includes which API versions are currently supported. Such version information could facilitate automatic discovery of services when appended with the endpoint information retrieved from Keystone service catalogs for example.

Problem description

When the root resource of Barbican is queried with a GET request, it should respond with a version response consistent with other OpenStack projects, based on json-home. This was discussed at a July 2014 mid-cycle meetup for Keystone and Barbican (see https://etherpad.openstack.org/p/keystone-juno-hackathon for details). The following is an example of implementing a json-home type of response from Keystone (documented here: https://docs.openstack.org/api/openstack-identity-service/2.0/content/Versions-d1e472.html):

{
    "choices": [
        {
            "id": "v1.0",
            "status": "DEPRECATED",
            "links": [
                {
                    "rel": "self",
                    "href": "https://identity.api.openstack.org/v1.0"
                }
            ],
            "media-types": {
                "values": [
                    {
                        "base": "application/xml",
                        "type": "application/vnd.openstack.identity+xml;version=1.0"
                    },
                    {
                        "base": "application/json",
                        "type": "application/vnd.openstack.identity+json;version=1.0"
                    }
                ]
            }
        },
        {
            "id": "v1.1",
            "status": "CURRENT",
            "links": [
                {
                    "rel": "self",
                    "href": "https://identity.api.openstack.org/v1.1"
                }
            ],
            "media-types": {
                "values": [
                    {
                        "base": "application/xml",
                        "type": "application/vnd.openstack.identity+xml;version=1.1"
                    },
                    {
                        "base": "application/json",
                        "type": "application/vnd.openstack.identity+json;version=1.1"
                    }
                ]
            }
        },
        {
            "id": "v2.0",
            "status": "BETA",
            "links": [
                {
                    "rel": "self",
                    "href": "https://identity.api.openstack.org/v2.0"
                }
            ],
            "media-types": {
                "values": [
                    {
                        "base": "application/xml",
                        "type": "application/vnd.openstack.identity+xml;version=2.0"
                    },
                    {
                        "base": "application/json",
                        "type": "application/vnd.openstack.identity+json;version=2.0"
                    }
                ]
            }
        }
    ],
    "choices_links": ""
}

When a GET is performed for a specific version, version details should be provided in response such as per this Keystone example:

 {
    "version": {
        "status": "stable",
        "updated": "2014-04-17T00:00:00Z",
        "media-types": [
        {
                "base": "application/json",
                "type": "application/vnd.openstack.identity-v2.0+json"
            },
            {
                "base": "application/xml",
                "type": "application/vnd.openstack.identity-v2.0+xml"
            }
        ],
        "id": "v2.0",
        "links": [
            {
                "href": "http://23.253.228.211:5000/v2.0/",
                "rel": "self"
            },
            {
                "href": "https://docs.openstack.org/api/openstack-identity-service/2.0/content/",
                "type": "text/html",
                "rel": "describedby"
            },
            {
                "href": "https://docs.openstack.org/api/openstack-identity-service/2.0/identity-dev-guide-2.0.pdf",
                "type": "application/pdf",
                "rel": "describedby"
            }
        ]
    }
}

Proposed change

However, as Chad Lung pointed out in the Launchpad blueprint, several projects document and implement the version responses shown above, so Barbican can follow the same approach. Barbican only supports requests and response in the JSON format, so XML formats will not be supported or implemented via this blueprint.

Keystone’s keystone/controllers.py implements the GET requests and builds the version responses. Note that the MEDIA_TYPE_JSON should be ‘application/vnd.openstack.keymanagement-%s+json’. The ‘Version’ controller defined in this file is mapped to the URI path via the keystone/routers.py module. Finally the keystone/service.py module builds the final WSGI application, and includes the version routers defined in routers.py.

For Barbican then, the barbican/api/controllers/versions.py VersionController class could be modified similarly to Keystone’s controllers.py, but should be renamed to VersionsController to handle the root requests, with a new VersionController added to return specific version detail responses.

The barbican/api/app.py file should be modified to use the new VersionsController wherever VersionController is used now. Note that while the root version resource would support unauthenticated requests, specific version resource requests would require authentication.

Also, the build version information that is responded with the root resource now would still need to be provided to support automated deployment and testing processes. This blueprint proposes adding a query parameter to the root resource such as ?build_version, that would return a JSON response similar to the current one but without the version information, such as this:

{
    "build": "2014.1.dev43.g22d1a96"
}

An alternative to retrieving this build information is detailed in the Alternatives section below.

Finally, the Barbican Python client should be updated to take advantage of this new version discovery information to augment the endpoint information it retrieves from Keystone service catalogs.

Alternatives

Regarding utilizing existing frameworks to generate the version information, there does not appear to be a Python library that implements json-home.

Regarding the build version information, that is not addressed by the json-home specification, although in Appendix C of the IETF reference in a section labeled ‘Open Issues’, there appears to be a placeholder called ‘release info?’ that might address this at a future date. So perhaps we could just add the build information to the root version response.

Data model impact

None.

REST API impact

The root version response will be changed from the one-off version information returned now to the OpenStack consistent response. A specific version details response (i.e. when performing a GET on v1/ for example) will need to be added. The current root response that includes the build version would need to be modified to only return this information if a query parameter is specified.

Security impact

None.

Notifications impact

None.

Other end user impact

Any processes that rely on the current build version response would need to be modified. Clients using the Barbican Python client will need to update to the release after the changes in this blueprint are made.

Performance Impact

None.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: jaosorior
Other potential contributors:: john-wood-w chad-lung

Work Items

The following CRs would build out this blueprint:

Modify version-related modules per proposal above.

2) Implement the build_version query parameter on the root resource to return the build version.

3) Update Barbican Python client code base to query the root resource for version information and then apply that to the endpoint information retrieved from the Keystone service catalog.

Dependencies

None.

Testing

Unit testing of the version resources will be added.

Documentation Impact

Update this page with API changes: https://github.com/cloudkeep/barbican/wiki/Application-Programming-Interface

References

See code and documentation references embedded in this blueprint above.

Information about json-home is found at http://http://tools.ietf.org/html/draft-nottingham-json-home-03.

Notes on the json-home discussion at the July 2014 Barbican and Keystone mid-cycle meetup can be found at https://etherpad.openstack.org/p/keystone-juno-hackathon.

Add barbican-manage command

Wed, 14 Nov 2018 00:00:00

Blueprint: https://blueprints.launchpad.net/barbican/+spec/add-barbican-manage-cmd

Client Blueprint: None

A new ‘barbican-manage’ command is introduced as Barbican admin tool. This command interacts with Barbican service for management operations which usually cannot be accomplished with REST APIs. This can improve usability and extensibility in the future.

Other OpenStack services like Keystone [1] and Nova [2] also provide similar commands for service admins.

Problem Description

Currently, Barbican uses individual admin commands for management functions. For example, using barbican-db-manage for database migration, using pkcs11-key-generation and pkcs11-kek-rewarp for HSM/pkcs11 related management, etc. More new admin functions will be added in future releases. It’s time to consolidate all these individual commands under a single tool for sake of simplicity.

Proposed Change

The syntax of new ‘barbican-manage’ command will be:

barbican-manage [options] category action [additional args]

category and action will be a list of subcommands that will be supported. The initial implementation of barbican-manage will be just refactoring current command code, and unify all functions into one command.

Note

Existing admin commands will be continue working and will be deprecated in future according to OpenStack standard deprecation policy [3]

Currently we have 2 categories: db for database management and hsm for HSM/PKCS11 management.

Category db replaces existing barbican-db-manage command:

db cleanup

Remove all soft-deleted and expired secrets from DB

db restore

Restore a soft-deleted secret from DB

db revision

Create a new DB version file

db upgrade

Upgrade to a future version DB version

db history

Show changeset history

db current

Show current revision for a database

Category db can take additional argument:

--dburl

URL to the database

--from-file

Secret garbage collection configuration file

Category hsm replaces existing pkcs11-key-generation and
pkcs11-kek-rewrap commands:

hsm gen-mkek

Generate HSM master key encryption key

hsm gen-mhmk

Generate HSM master HMAC key

hsm rewrap-pkek

Rewrap Project KEKs after rotating to a new MKEK

Category hsm can take following additional arguments:

--library-path

PKCS11 library path

--slot-id

Slot ID

--passphrase

PKCS11 login password

--label

Key label

--length

Key length

--dry-run

Displays changes that will be made (Non-destructive)

Note

–dry-run requires above 5 arguments be specified

General ‘options’ includes:

--help

show help message

--version

show command version

The command will read standard barbican.conf to get setting for debug, verbose and log_file options.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

User has to have appropriate privilege to run command barbican-manage.

Notifications & Audit Impact

Event log can be generated for audit support.

Python and Command Line Client Impact

No impact to Barbican client and OpenStack client.

A new CLI admin command will be added, so its user guide need to be added.

Other end user impact

None

Performance Impact

None

Other deployer impact

None if existing admin commands are not used in deploying script.

There is no immediate impact to deployers even if they use existing admin commands in script for deployment. The script need to be converted to use new barbican-manage command eventually before old commands are removed according to procedures in OpenStack standard deprecation policy.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <jianhua>
Other contributors:: <None>

Work Items

Work items or tasks

Create a new barbican-manage.py in barbican/cmd and call functions into scripts db_manage.py and pkcs11_*.py
Add unit testcases
Add barbican-manage command script in setup.cfg
Add user guide document for barbican-manage command
deprecate existing command scripts. Adding deprecation warning message in existing commands.

Dependencies

None

Testing

Unit tests will be added for all subcommands and various options.

Documentation Impact

A new barbican-manage command user guide will be added, which should include new user guide for database migration subcommands and user guide of pkcs11-related subcommands modified from existing https://docs.openstack.org/developer/barbican/api/userguide/pkcs11keygeneration.html

References

Add Certificate Generation and Management To Orders

Sun, 13 May 2018 00:00:00

Launchpad blueprint: https://blueprints.launchpad.net/barbican/+spec/add-ssl-ca-support

Barbican can be used to automate and streamline secure generation and storage of SSL certificates and associated private keys from client information (such as a CSR). This feature requires being able to coordinate between multiple possible certificate authorities (CA) for generation, Barbican for secure certificate storage and notification once generation is completed. This blueprint details a plugin-based approach to satisfy these requirements.

Problem description

To generate and manage SSL certificates within Barbican via its orders resource requires that Barbican perform these actions: (1) accept certificate information from a client, (2) initiate a certificate generation order with one of several possible certificate authorities (CA), (3) check on the progress of each CA order periodically, (4) securely store certificate information once generated by the CA, and finally (5) notify clients that the certificate is ready to provision. This wiki page details the flows involved: https://wiki.openstack.org/wiki/Barbican/Blueprints/ssl-certificates.

Barbican should be capable of interacting with a variety of different CAs, each having their own custom interaction workflows. Barbican should be able to store state on behalf of this workflow between states (such as to wait for a CA to generate a certificate). Barbican should also provide a means for workflow processes to reschedule calling back to the workflow process, such as retrying a failed attempt to create a certificate order in the CA.

Barbican should be capable of notifying clients about issues or generated certificates via a variety of eventing mechanisms available to a deployment. Examples include surfacing CADF events via Ceilometer, or issuing tickets into a corporate ticketing system.

Barbican should provide CA workflow processes the ability to securely store certificate information in Barbican without needing to directly interact with its datamodel.

Barbican should provide CA workflow processes the ability to reschedule themselves at a future time, perhaps to reattempt a failed process such as establishing an order with the CA.

Barbican should allow for validation of the certificate order fields prior to initiating the CA ordering process. Barbican should allow for clients to update information about their launched CA order, such as providing corrections, cancellations or approvals. This interface could also support revoking a certificate once it is created and installed in Barbican.

Proposed change

This blueprint proposes using a plugin approach to interact with specific CAs. The plugin contract will expose processing methods that represent workflow actions, such as ‘request_certificate(…)’ or ‘check_request_status(…)’. This blueprint will defer to implementing CRs the specific names and usages for these methods.

Because the workflow/state handling required to interact with a given CA is expected to vary significantly between CA vendors, it is expected that the plugin will need to manage its own state machine. However, the CA plugins shouldn’t have to manage persisting/retrieving state machine data for a given order instance. Hence Barbican should handle providing a dict of information into the plugin’s methods into which they can reference their state. For example, this might include a ‘state’ key that keeps track of the current state machine state for an order. Barbican would then store this dict as plugin-specific metadata about the order.

A separate scheduled process run from the worker nodes (via oslo incubator’s periodic_task feature) would poll CAs for updates to pending certificate orders, generating RPC tasks for each update, which would eventually invoke the CA plugin method above.

Also proposed is a plugin for surfacing certificate events from Barbican. Since CA plugins will know best within their workflows when an event should be issued (say when a certificate is generated) the proposal is to pass the event plugin into the CA plugin’s methods. This inversion of control (IoC) approach would allow plugins to be in control of event sequencing, and would allow separation from how the events are handled within and from Barbican. This plugin could have domain specific methods that make sense for certificate processing work, such as ‘notify_certificate_is_generated()’. The default out-of-the-box plugin would just log events. A provided optional event plugin implementation would create CADF events for Ceilometer to handle.

The CA workflow plugins also know best when to store a generated certificate. Therefore another IoC adapter would be passed into the CA plugin’s method providing specific methods such as ‘store_certificate()’ which would be implemented as a Barbican Container repository save call.

Finally, an IoC adapter would be passed into the CA plugin’s methods allowing the plugin to invoke one of its method at a future time, by enqueing an RPC call that launches the CA plugin’s method.

Alternatives

The certificate generation process detailed above is a workflow that could be expressed using a workflow framework. Several of them are discussed below. They all generally aim to execute a series of preconfigured tasks sequentially or in parallel until completion, potentially reverting the entire sequence if errors occur.

The certificate processing state machine does not seem to be a good fit for this approach for two reasons:

(1) Certificate processing involves potentially long delays (multiple days) between CA interaction state machine steps, including polling the CA for status updates or waiting for client updates. Delays and scheduled polling behaviors do not appear to be integrated into the existing workflow approaches below, so Barbican would need to implement such logic anyway.

(2) Some state steps may need to be repeated, such as retrying initiating an order with the CA when the CA is unavailable. So in this case, the order of tasks executed is not known a priori, and when an error happens tasks completed up to that point should not always be reverted.

Desired though not required is that the workflow process play nicely with the Barbican worker processes, including being able to react to RPC calls from the queue, or to scheduled update requests.

TaskFlow (https://wiki.openstack.org/wiki/TaskFlow) is an OpenStack Python library that can compose tasks and workflows using Python classes. These are intended to be run from within an ‘engine’, which manages the execution of the configured tasks. Hence TaskFlow might be best run as a separate process or node from the Barbican workers. It seems to be the most capable for this blueprint’s needs, but still has the ‘fitness’ issues mentioned above.

Mistral (https://wiki.openstack.org/wiki/Mistral) is an OpenStack Workflow as a Service project. It is in an early stage however, an may not be available by the Juno or K releases. It does not let projects upload and execute custom code, but rather calls back to Barbican to perform its tasks.

Lower level frameworks such as machinist (https://pypi.org/project/machinist/0.1) can operate state machines as defined via Python objects. Like TaskFlow this might be best run as a process separate from the Barbican workers. This framework seemed less flexible than TaskFlow.

Data model impact

The current Orders entity would need to have a new relationship added to store a CA plugin’s metadata, similar to the key/value metadata store that was added to the Secrets entity. This would be different than the Order’s ‘meta’ attribute, which is used to store user-provided ordering information.

No database migrations should be required as the metadata is both optional and controlled by the plugins themselves.

REST API impact

The ‘certificate’ orders type is being added per another change process, so this blueprint has not direct API impact.

Security impact

Barbican will interact with a third-party system (a certificate authority). This interaction is only initiated by Barbican, but care must be taken to validate both the user provided information (via the certificate plugin) as well as the response information back from the CA.

Notifications impact

This blueprint calls for using a plugin approach to surface events from the certificate generation process, in particular to notify when a certificate has been generated and is ready to install, and when an error has occurred. Errors could include the CA rejecting the order, is temporarily unavailable or is rate-limiting the number of requests made by the client. In some cases, Barbican would need to re-attempt the request at some point in the future.

This blueprint will probably be the first use case for event generation and notifications in Barbican.

Other end user impact

None.

Performance Impact

The impact of this certificate processing should be minimal, since even though it could take days to approve and generate a certificate, the vast majority of that time is spent waiting on either the CA to update a given certificate order, or else for users to provide corrections or approvals. When Barbican is processing a state machine step the computation load should be minimal.

Also, it is expected that not many certificate orders will be processed concurrently. Even if the load does increase over time, all certificate processing is performed asynchronously on worker nodes, so additional delays will be accommodated and are likely to be a fraction of the overall certificate workflow period.

Other deployer impact

None, as the current worker processes will be used.

Developer impact

New facilities will be added to the current Barbican worker code base, in particular oslo incubator’s periodic_task implementation, and the eventing plugin.

Implementation

Assignee(s)

Primary assignee:: john-wood-w
Other contributors:: alee-3 arvind-tiwari

Work Items

The following CRs would build out this blueprint:

– Add CA plugin and validation: 1) Add stevedore plugin manager for CA workflow plugin, and initial plugin abstract interface with workflow processing methods. Default implementation would just issue log messages. Call from the BeginOrder task for the ‘certificate’ orders type. Add initial unit testing.

Add validation processing to the CA plugin.

– Add eventing plugin: 3) Add eventing stevedore plugin manager for eventing plugin, and initial plugin abstract interface with default logging implementation.

– Add adapters: 4) Add datastore adapter and pass as IoC context to CA plugin. 5) Add task retry adapter and pass as IoC context to CA plugin.

– Add orders update: 6) Modify POST order process to handle validation processing (including using step #2 work above) 7) Add PUT handler to the ‘orders’ resource to handle client order updates. 8) Add ‘UpdateOrder’ task to asynchronously handle these updates and invoke CA plugin methods accordingly.

– Add scheduled processes: 9) Add oslo incubator’s periodic_task to worker process. 10) Add CA workflow support for periodically checking for CA status, and then generating update events back to the Barbican worker queue.

– Production plugin implementations: 11) Add CADF/Ceilometer event plugin implementation. 12) Add Symantec plugin implementation

Dependencies

The plugin restructure work associated with this blueprint should be completed prior to implementing this work: https://blueprints.launchpad.net/barbican/+spec/restructure-for-plugins

Testing

Unit testing of each plugin and adapter will be added. Integration testing of at least the default plugins will be added. Symantec testing will be needed, perhaps with a mock Symantec service.

Documentation Impact

Update https://github.com/cloudkeep/barbican/wiki/Application-Programming-Interface with API changes for the ‘certificate’ orders type, and for PUTs to orders.

References

A work in progress CR with strawman code related to this blueprint is available here: https://review.openstack.org/#/c/95023/

An analysis of the Symantec CA workflow is available here: https://review.openstack.org/#/c/95023/

Plugin design concepts related to this blueprint, including the IoC adapter usage, are detailed here: https://wiki.openstack.org/wiki/Barbican/Discussion-Plugin-Design

Add certificate to container type

Thu, 08 Feb 2018 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/barbican/+spec/add-certificate-to-the-container-type

Open Stack LBaaS is planning to use barbican for their certificate use case. We need to add ‘certificate’ to the container type option.

Problem Description

Barbican’s Container resource is capable of storing SSL certificate along with its private_key and passphrase. Current implementation of container only supports RSA and Generic type, due to this limitation containers holding certificate can not be distinguishable.

Proposed Change

Add the certificate option for containers. This augments the current type attribute on the containers resource and data model to accept certificate as a valid option.

Alternatives

None

Data model impact

The Container schema will be modified to have certificate in the container types constraint check.

REST API impact

The REST request and response structure for container may contain certificate in type field.

Example 01 - Container request with certificate type:

{
  "name": "my-cert",
  "type": "certificate",
  "secret_refs": [
    {
       "name": "certificate",
       "secret_ref":"http://localhost:9311/v1/b885a8320e4f48c69ddffb0364eeef36/secrets/46332930-8e52-4c40-b069-cc39ca65a221"
    },
    {
       "name": "intermediates",
       "secret_ref":"http://localhost:9311/v1/b885a8320e4f48c69ddffb0364eeef36/secrets/21f2234a-f65c-11e3-8791-002564955ea1"
    },
    {

       "name": "private_key",
       "secret_ref":"http://localhost:9311/v1/b885a8320e4f48c69ddffb0364eeef36/secrets/094e76ed-85e0-49b1-b6ce-6bde3cf6571c"
    },
    {
       "name": "private_key_passphrase",
       "secret_ref":"http://localhost:9311/v1/b885a8320e4f48c69ddffb0364eeef36/secrets/da5ccd5a-ef0f-4cb3-8d8f-dd12308b3109"
    }
  ]
}

Example 02 - Container response with certificate type:

{
    "status": "ACTIVE",
    "updated": "2014-06-10 17:21:37.477461",
    "name": "my-cert",
    "secret_refs": [
        {
            "secret_ref": "http://localhost:9311/v1/b885a8320e4f48c69ddffb0364eeef36/secrets/579d0ffe-a40c-47a6-b0c6-8978a441f661",
            "name": "private_key_passphrase"
        },
        {
            "secret_ref": "http://localhost:9311/v1/b885a8320e4f48c69ddffb0364eeef36/secrets/93688bf3-5101-47de-bdd5-46e52186038f",
            "name": "certificate"
        },
        {
           "name": "intermediates",
           "secret_ref":"http://localhost:9311/v1/b885a8320e4f48c69ddffb0364eeef36/secrets/21f2234a-f65c-11e3-8791-002564955ea1"
        },
        {

            "secret_ref": "http://localhost:9311/v1/b885a8320e4f48c69ddffb0364eeef36/secrets/a7a61669-c6fb-4375-9577-11744f4a88f7",
            "name": "private_key"
        }
    ],
    "created": "2014-06-10 17:21:37.477448",
    "container_ref": "http://localhost:9311/v1/b885a8320e4f48c69ddffb0364eeef36/containers/dc252a7d-600f-49a3-9df4-7bca35aa366d",
    "type": "certificate"
}

Security impact

None

Notifications & Audit Impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: atiwari (arvind-tiwari)

Work Items

CR to address data migration.
CR to add certificate type in container.

Dependencies

None

Testing

New unit test will be required to test this feature.

Documentation Impact

Documents has to be enhanced to add certificate in container type.

https://github.com/cloudkeep/barbican/wiki/Application-Programming-Interface#containers-resource

References

https://blueprints.launchpad.net/barbican/+spec/add-certificate-to-the-container-type

Adding per-secret policy to allow the storing of private secrets

Thu, 08 Feb 2018 00:00:00

https://blueprints.launchpad.net/barbican/+spec/add-per-secret-policy

Problem Description

This is a companion spec to the add-per-secret-policy spec. That spec proposed a mechanism for allowing users outside the secrets project to access a secret/container. This mechanism involved storing access control data in the database for each secret/container.

This spec solves a related problem. Currently, secrets are accessible by all users with the relevant role in the secret creator’s project. That means that all members of the same project can access all secrets stored for that project.

Recently there have been requests to be able to designate “private secrets”. These are secrets that only the secret creator (the user that created the secrets) would be able to extract.

Now, it is possible to create a project for each user that would like to store private secrets. This seems to be a very burdensome solution, with large admin overhead in managing all the required groups and roles.

Proposed Change

The proposal is to store additional access data with each secret/container. This data (“creator_only” with a value of True/False) will be stored in the SecretACL and ContainerACL data tables as a separate column. The details are in the Data Model Impact section below.

When a secret/container is accessed, the creator_only information and any whitelist information is passed up to the policy layer through target.* variables. Access is then determined by evaluating policy rules that use these parameters. Proposed policy rules are specified in the Policy section below.

Finally, the REST API will need to be augmented to allow users to specify and change the creator_only status of a secret/container.

Operations

Many different operations are restricted when creator_only = True:

get: If creator_only is true, only the secret/container’s creator will

be able to access the secret. Restricting access to a container though does NOT cascade the restriction down to the constituent secrets.

Note that this does not affect whitelisted users that have explicitly been allowed to get the secret/container.

delete: If creator_only is true, the secret/container can only be deleted by

the creator.

write: If creator_only is true, the secret/container can only be modified by

the creator.

change-acl: By default, this can only be modified by the creator.

list: If creator_only is true, only the creator will be able to list the secret.: This means that the link to the secret will not show up if a non-creator accesses GET /secrets

Data model impact

As per the per-secret spec, secrets, containers and orders tables will have a column added to store the creator of the secret. This would be populated by the creator’s user_id.

A boolean column (“creator_only”) will need to be added to the ContainerACL and SecretACL tables. The fields for the ACL tables will be as follows:

secret_id: foreign key to Secrets table operation: (in this case, get, write, delete, list) users: string, list of whitelisted users for the specified operation creator_only: True/False

When a secret is designated as “creator_only”, several fields will be created/ modified:

secret_id : get : users, True
secret_id:  write: None, True
secret_id:  delete: None, True
secret_id:  list:  None, True

Undesignating a secret as “creator-only” would either modify/delete the above entries.

REST API impact

A new parameter (“creator-only”), which would take the values “True” and “False” will be added to the ACL definition. It will default to False. Accordingly, the ACL definition will look like:
```
{
    "get": {
        "users": ["some_user",
                  "another_user"],
    },
    "creator-only": "true"
}
```
If creator-only is set to true, the fields mentioned in the section above would be created.
As described in the per-secret spec, the ACL could be specified as an optional acl parameter when creating the secret with a POST request. It can also be retrieved or modified through a GET or POST
```
POST <host>/v1/secrets/<secret-UUID>/acl
GET  <host>/v1/secrets/<secret-UUID>/acl
```

Policy

When a container or secret is accessed, the ACL data is retrieved from the database and provided to the RBAC layer as target.* attributes. For reading a secret, for example, we could pass in target.user_whitelist and target.creator_only.

The policy rule would then look something like:

(can_read_shared and user_in_user_whitelist) OR
((current_resource_permissions and not creator_only) OR user_is_creator)

The (current_resource_permissions) part is basically that the user has the relevant role in the secret creator’s project.

For deletes, modifications etc., the rule is much simpler because we do not need to account for the whitelists. Delete, for example, will likely look something like this:

((current_resource_permissions and not creator_only) OR user_is_creator)

Alternatives

As mentioned before, for private secrets, we could create a group for each user. Other than being cumbersome, this will entail a maintenance load on system administrators to keep track of new and removed users.

Security impact

This improves security and usability in the stack as a whole by allowing users to specify private secrets.

Notifications & Audit Impact

None.

Other end user impact

python-barbicanclient will need to be updated to provide an interface to populate the extra parameters.

Performance Impact

Accessing a secret/container will require two database calls: one to get secret/container whitelist as part of the RBAC engine’s rules enforcement, and one to actually get the secret.

These two database accesses are logically separate as the first process is controlled by middleware, and the second by Pecan. We might be able to utilize the same SQLAlchemy transaction, or else cache that secret entity data for the controllers to work with, so this might be moot.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: alee rm_work

Work Items

Add new field to the database tables, and new parameters/calls to the REST API.
Add logic to parse the data and store the data in the database.
Add logic to retrieve the data from the database and provide to RBAC layer as target.* attributes.
Modify policy rules based on these target.* attributes. It may be necessary to extend oslo policy here to account for the new boolean flag. Any changes will need to be communicated clearly to deployers as it is not guaranteed that they will deploy with default policy files.

Dependencies

None

Testing

The current unit tests will also be modified to have this change reflected upon them.

Documentation Impact

Barbican docs and API docs will need to be changed.

References

Earlier blueprint with similar ideas. https://blueprints.launchpad.net/barbican/+spec/secret-isolation-at-user-level

KMIP MKEK Model Plugin

Thu, 08 Feb 2018 00:00:00

https://blueprints.launchpad.net/barbican/+spec/babrican-mkek-model

This effort aims to improve on the scalability of the existing KMIP integration offered by Barbican. Specifically by removing possible scale limitations imposed by the number of project secrets that can be stored in a KMIP attached HSM. We propose adding a new plugin that implements a Master Key Encryption (MKEK) based model. Under this model, locally stored project secrets, termed Data Encryption Keys (DEKs), are protected by per-project Key Encryption Keys (KEKs) that are themselves protected by the use of a Master Key Encryption Key (MKEK). This functionality mirrors that offered by similar MKEK operations available to users of the PKCS11 ‘p11’ plugin.

Problem Description

Currently, Barbican can utilise HSMs that communicate via KMIP to generate and store key material on behalf of projects. However, several HSM appliances are restricted by the maximum number of keys that can be stored. This maximum could be easily exceeded in large cloud deployments.

Proposed Change

To combat this problem a model is presented here that stores project key material locally within the Barbican database. This will be implemented as a plugin following the same basic design as the PKCS11 plugin and deriving from the CryptoPluginBase interface.

To protect the local key material it is itself encrypted using a per-project Key Encryption Key. This process is referred to as wrapping the key and a wrapped key refers to the cipher text version of a given key value. All project key material is wrapped using that project’s unique KEK prior to storage, and this wrapping is performed by the attached HSM appliance. By wrapping all keys belonging to a project with a unique KEK we gain two advantages. Firstly, project separation is enhanced, a project acquiring another project’s wrapped keys will be unable to utilise them. Secondly, in the event that a project’s key material must be presented to a third party (on the issuing of a warrant or other access demand) only that specific project’s KEK must be surrendered, other projects remain fully protected.

The KEKs belonging to various projects are themselves stored locally within the Barbican database. To protect the KEKs from exposure should the database be compromised, they themselves will be wrapped using a single Master Key Encryption Key, or MKEK. No unwrapped project KEKs will be stored and this wrapping process will again be performed by the HSM. The MKEK used is global to the running Barbican deployment and is considered to be owned by the deployer, rather than by any project. It is generated on the backend HSM and stored there, the MKEK is never fetched locally into Barbican process memory as (un)wrapping operations are performed within the HSM.

The following operations are required:

Key Initialisation, MKEK generation. Initially a new MKEK will be created on the HSM and a reference will be stored to it in the local Barbican database. Note that the MKEK itself remains on the HSM and only its reference is stored. This logic will be placed within the bind_kek_metadata() method, when discovering that a MKEK is not available for use a new one will be generated.

Project Initialisation, KEK generation. A new project will be initialised by using the HSM to create a KEK for the project. This KEK will then be retrieved, as a wrapped key using the MKEK that resides on the HSM as a wrapping key. The wrapped KEK is stored in Barbican’s database and the KEK will be removed from the HSM. This logic will be placed within the bind_kek_metadata() method. Either a pre-existing KEK will be returned or a new one will be generated if no KEK exists for the project.

Project Data Encryption Key (DEK) generation. A new key will be generated by the HSM and its reference returned to Barbican. Barbican will then send the wrapped project KEK to the HSM as a wrapped key, and provide its reference to the MKEK as the wrapping key so the HSM may unwrap it. Finally, using the DEK reference, Barbican will request the newly generated DEK be returned, wrapped by the now unwrapped KEK. This wrapped DEK is stored in the local database and the DEK and KEK will be removed from the HSM. This logic will be placed within the generate_symmetric() and generate_asymmetric() methods for symmetric and asymmetric DEKs respectively.

Project Data Encryption Key (DEK) storage. Barbican will send the wrapped project KEK to the HSM as a wrapped key, and provide its reference to the MKEK as the wrapping key so the HSM may unwrap it. An unwrapped DEK, provided by the project, will be sent to the HSM and then retrieved, wrapped by the projects KEK. This wrapped DEK will be stored in the local Barbican database. The unwrapped DEK and KEK will now be removed from the HSM. This logic will placed within the encrypt() interface method.

Project Data Encryption Key (DEK) retrieval. A project wrapped DEK (its encrypted secret) will be retrieved from Barbican’s database by first sending the wrapped project KEK to the HSM and a reference to the MKEK to unwrap it, this results in a reference to the unwrapped KEK. Barbican will then send the wrapped DEK to the HSM and the reference to the unwrapped KEK to unwrap the DEK. Finally, Barbican can now retrieve the unwrapped DEK and return it to the requesting user. The unwrapped DEK and KEK will now be removed from the HSM. This logic will placed within the decrypt() interface method.

Master Key Encryption Key Rotation. The specifics of MKEK rotation and re-keying are beyond the scope of this spec and will be tackled independently. However, information is provided here to illustrate how this might be accomplished under the proposed model.

Should it be required that the MKEK be rotated out and a new one created, then this might be accomplish as follows. Firstly a new MKEK is generated on the HSM and its reference retrieved and stored as the active MKEK version, termed ‘originator-usage’. When a new project KEK must be created, the MKEK currently operating in originator-usage will be used for wrapping and its reference stored alongside the wrapped KEK as meta data. This MKEK reference data will allow Barbican to indicate the correct master key to be used during unwrapping. Former MKEKs are maintained in the HSM for retrieval of data only, termed ‘recipient-usage’. In this way new KEKs are protected by the new MKEK and older ones remain viable by referencing the earlier MKEKs stored in the HSM.

After a recipient-usage MKEK reaches a defined age, all KEKs wrapped by it should be re-keyed, that is unwrapped and then re-wrapped, using the originator-usage key. The period of time that an MKEK may remain valid for recipient-usage is termed the crypto-period and is defined via a config setting.

Alternatives

An existing plugin, the KMIPSecretStore is already available to Barbican and can be considered as an alternative to the MKEK model. It uses KMIP to communicate with a backend HSM and stores all key material within the HSM itself. As such it is limited by the HSMs own storage capacity.

Data model impact

No changes to the data model are required. All needed data can be stored as plugin specific meta-data. The wrapped per-project KEK will be stored, along with a reference to the MKEK in the kek_meta_dto object. If some form of key rotation is in effect (again, the specific details of key rotation are beyond the scope of this spec) then MKEK version info will be stored in the per-secret kek_extended_meta_dto object.

REST API impact

No changes need to be made to the REST API.

Security impact

This change uses encryption on the HSM to protect key material, however it introduces no additional security concerns over the existing KMIPSecretStore plugin. Project DEKs are stored in the local database but they are protected by the project KEK. The project KEK is store in the local database and this is protected by the MKEK. The MKEK is never stored or used outside of the HSM.

Notifications & Audit Impact

N/A

Other end user impact

N/A

Performance Impact

This change will incur a performance impact through additional communication to the backend HSM. These communications will need to be protected through the use of a TLS secured link. Additionally operations at scale will be limited to how rapidly the HSM can (un)wrap and return key material. Generally this is a fast operation but will be subject to network latency communicating to the HSM.

We expect this overhead to be exaggerated initially owing to the use of simple atomic KMIP operations. Once more complex compound operations are supported by the underlying library this situation will be improved.

Other deployer impact

No additional deployment impact is created by this proposed change over that required by the KMIPSecretStore. That is, a backend HSM that can communicate via KMIP must be available and contactable for successful operation.

Note also that although the plugin contract remains the same as the existing KMIPSecretStore plugin, the semantics of the stored data are incompatible. Since this is a new plugin rather than a modification to the existing KMIP plugin it is believed that no data versioning should be required since data generated by the respective plugins should be considered entirely incompatible.

Developer impact

N/A the plugin is self contained.

Implementation

Assignee(s)

Primary assignee:: tim-kelsey <tkelsey> rob-clark <hyakuhei>
Other contributors:: None

Work Items

Create spec (this spec).
Write the plugin and tests.
Confirm all Barbican tests, new and old, still pass.
Review the code.

Dependencies

This will require no new dependencies over KMIPSecretStore, that is it will require PyKMIP library v0.2 or greater.

Testing

A suite of unit tests will be produced to test the new code. In order to test the plugin functionality fully a KMIP capable HSM will need to be available to gate tests.

Documentation Impact

No changes to documented APIs will be needed. New plugin specific configuration options will be created, and these should be documented.

Option	Meaning
mkek_crypto_plugin.crt_file_path	KMIP client certificate file path.
mkek_crypto_plugin.key_file_path	KMIP client key file.
mkek_crypto_plugin.key_password	KMIP client key password.
mkek_crypto_plugin.user_name	KMIP user name.
mkek_crypto_plugin.user_pass	KMIP user password.
mkek_crypto_plugin.hsm_host	HSM host IPs, comma separated.
mkek_crypto_plugin.hsm_port	HSM port number.

References

Please refer to KMIPSecretStore plugin documentation. Please refer to the PKCS11 plugin MKEK implementation. Please refer to the previously merged spec for PKCS11 that solves a similar problem: https://review.openstack.org/#/c/107775/4/specs/juno/restructure-pkcs11-plugin.rst

The KMIP specification can be found at: http://docs.oasis-open.org/kmip/spec/v1.2/kmip-spec-v1.2.html

Rolling Upgrade

Sat, 10 Jun 2017 00:00:00

Barbican users would like to upgrade their existing environments with minimal downtime. This spec outlines the changes required to be able to assert the various upgrade tags. The goal for Pike is to assert assert:supports-rolling-upgrade tag.

Problem description:

There are currently six upgrade tags:

assert:follows-standard-deprecation
assert:supports-upgrade
assert:supports-accessible-upgrade
assert:supports-rolling-upgrade
assert:supports-zero-downtime-upgrade
assert:supports-zero-impact-upgrade

Status tags in Barbican project:

assert:follows-standard-deprecation: Done [1]

assert:supports-upgrade: Done [2]

assert:supports-accessible-upgrade: Not yet [3]

assert:supports-rolling-upgrade: Not yet [4]

assert:supports-zero-downtime-upgrade: Not yet [5]

assert:supports-zero-impact-upgrade: Not yet [6]

Proposed change:

To support rolling upgrade for Barbican, we need to compare feature lists support for upgrade and rolling upgrade as follows:

Maintenance Mode
Live Migration
Upgrade Orchestration
Multi-version Interoperability
Online Schema Migration
Graceful Shutdown
Upgrade Orchestration - Remove
Upgrade Orchestration - Tooling
Upgrade Gating
Project Tagging

For Barbican, upgrading from release N-1 to release N then to N+1, we are following supporting status and performing detail items as follows: https://github.com/openstack/development-proposals/blob/master/development-proposals/proposed/rolling-upgrades.rst

Maintenance Mode: No need to implemented

No need to implement it because of plugin mechanisms and no scheduler/placement services

Live Migration: No need to implemented

Because Barbican does not have data plane. All information will be stored in its database and secret store plugins.

Upgrade Orchestration: Not yet implemented
Multi-version interoperability

API microversion: Not yet - Barbican does not need microversions for rolling upgrade.

RPC versioning: Not yet implemented.

RPC object version (Oslo.VersionedObjects): Not yet implemented.

Online Schema Migration: Not yet implemented
Graceful Shutdown: Implemented

Barbican-api is using Pecan to expose API and Pecan has supported graceful shutdown so it means barbican-api has supported this feature as well

Other services in Barbican are using oslo.service to launch the services and oslo.service has implemented graceful shutdown so they have supported this feature also.

Upgrade Orchestration: Not yet implemented
Upgrade Orchestration: Tooling implemented
Upgrade Gating: Implemented
Project Tagging: Not yet implemented

Assert the tag and notify the OpenStack Technical Committee

Data model impact

Impacted because the O.VO codebase will change the way Barbican services communicate to database as well as the data flow between Barbican internal services.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

End users will have minimal downtime when upgrading barbican services.

Performance Impact

None

Other deployer impact

It is anticipated that a rolling upgrade will require the operators intervention.

Developer impact

Developers will need to be aware of Barbican features that enable rolling upgrades and make sure they are not removed (Developers will also need to work within the constraints of the database strategy for rolling upgrades, but that developer impact is covered by another spec).

Implementation

Assignee(s)

Primary assignee:

namnh

Other contributors:

daidv

hieulq

Work Items:

Apply O.VO in Barbican before rolling upgrade:
- Add oslo.versionedobjects to requirements docs.
- Implement barbican O.VO base objects.
- Migrate current database object to OVO objects.
- Implement extra fields in objects/fields.py.
- Implement RPC object registry and register all objects.
- Implement and attach the object serializer.
- Implement the indirection API(VersionedObjectIndirectionAPI) in the objects/base.py
- Implement Unit Test for Barbican O.VO module.
Perform rolling upgrade for O.VO.
- Providing serialized versions.
- Implement a new method to check version.
- Implement RPC pinning version.
- Implement methods to communicating with DB, such as query, save and create.
Perform rolling upgrade for OSM along with O.VO which completed before.
- Implementing DB upgrade mechanism for Barbican to perform barbican-db-manage command via CLI including three branches:
  - ‘–expand’ branch: To add columns, tables or triggers in barbican database.
  - ‘–contract’ branch: to delete columns, tables or trigger after all services in branch are upgraded to new release.
- Perform data gradually migrate to new schema. by multi-version interoperability (included O.VO and RPC Pinning).
- Ensure all data have migrated to new schema via online_data_migrations commandline.
Write documentation for rolling upgrade (operator docs).
Assert the tag and notify the OpenStack Technical Committee.

References

[1] https://governance.openstack.org/tc/reference/tags/assert_follows-standard-deprecation.html#tag-assert-follows-standard-deprecation

[2] https://governance.openstack.org/tc/reference/tags/assert_supports-upgrade.html#application-to-current-projects

[3] https://governance.openstack.org/tc/reference/tags/assert_supports-accessible-upgrade.html

[4] https://governance.openstack.org/tc/reference/tags/assert_supports-rolling-upgrade.html

[5] https://governance.openstack.org/tc/reference/tags/assert_supports-zero-downtime-upgrade.html

[6] https://governance.openstack.org/tc/reference/tags/assert_supports-zero-impact-upgrade.html

Multiple Secret Backend Support

Wed, 23 Nov 2016 00:00:00

https://blueprints.launchpad.net/barbican/+spec/multiple-secret-backend

Barbican provides capability to create and store secrets via its plugin mechanism. Barbican has KMIP, PKCS11 and Dogtag KRA plugins for creating and managing secrets in HSM (Hardware Security Module) devices and store_crypto plugin for storing secrets in its own database. Enterprises generally have HSM devices to safely secure their key material (data-at-rest) and to meet compliance requirements.

Problem Description

Barbican supports secret storage in HSM device as well as in a database. So far, Barbican has implicit concept of configuring one active plugin for secret store which means all of the new secrets are going to be stored via same plugin (i.e. same storage backend). This approach can limit the usage of barbican in a cloud deployment where not all services/applications have similar data sensitivity and hence don’t have necessity of using same mechanism to safeguard their key material. Also HSM are expensive devices which have limited storage capacity and performance characteristics in comparison to database. Current one active secret store plugin approach is inflexible for following use cases.

In a deployment, a deployer may be okay storing their dev/test resources using a low-security secret store, such as one backend by software-only crypto, but may want to use an HSM-backed secret store for production resources.
In a deployment, for certain use cases where a client requires high concurrent access of stored keys, HSM might not be a good storage backend. Also scaling them horizontally to provide higher scalability is a costly approach with respect to database.
HSM devices generally have limited storage capacity so a deployment will have to watch its stored keys size proactively to remain under the limit constraint. This is applicable in KMIP backend more than with PKCS11 backend because of plugin’s different storage approach. This aspect can also result from above use case scenario where deployment is storing non-sensitive (from dev/test environment) encryption keys in HSM.
Barbican running as IaaS service or platform component where some class of client services have strict compliance requirements (e.g. FIPS) so will use HSM backed plugins whereas others may be okay storing keys in software-only crypto plugin.

Currently there is no simple solution to support above use-cases. The suggestion of asking deployment to have a separate barbican endpoint per plugin is not a good usage experience. See Alternatives section for further details. As there is only one secret store backend available, so client (or class of services) does not have choice to choose backend regardless of its requirement related to safeguarding those keys.

Proposed Change

Proposal is to allow multiple secret store backends available in a single barbican deployment. As part of this change, client will have the choice to select preferred backend at a project level.

For this, a Barbican deployment will have at least 2 secret storage backend enabled and need to specify one backend to be a deployment level (global) default option. Project administrators will have choice of pre-selecting one backend as the preferred choice for secrets created under that project. Any new secret created under that project will use the preferred backend to store its key material. When there is no project level storage backend selected, then new secret will use the global secret storage backend.

As each secret already has metadata which captures its backend information, Barbican can allow clients to specify a specific storage backend when creating or storing new secret in Barbican. This can be a future enhancement if there is an actual use case for this requirement. Currently this feature is not in scope of this effort.

Also to manage this feature, a flag enable_multiple_secret_stores will be added. By default, this flag will be disabled (set as false). So any existing logic changes need to account for that flag value and modify only when this flag is enabled. When barbican is started with this flag enabled, then need to make sure that global default plugin exists in config file.

Change in New Secret Generate and Storage

Currently a secret store plugin expose its generate and store capability via its supports API mechanism. This mechanism checks that input secret specification (algorithm, length) is supported by plugin backend.

When a project has defined its preferred secret store plugin, then only that plugin is checked for supports criteria. If preferred plugin does not meets the given input criteria, then 400 error with related message is returned to the client. For this to happen, get_active_plugins logic will return project level preferred plugin if defined otherwise global default plugin.

This logic of preferred secret generate and store needs to be applied on order processing side as well. Synchronous or asynchronous order processing logic needs to be modified so that when generate plugin is looked up for new secret, it provides project level preferred plugin and then global default plugin. Same goes for secret store logic needed for symmetric key storage and certificate storage after certificate are generated successfully during order processing.

Barbican has key wrapping support (in few plugins) to ensure that secret is pre-encrypted in such a way that only client and backend store can decrypt the secret. For this client needs to have access to transport key to pre-encrypt while storing or reading that secret back. With multiple backend support, reading of transport key will not be impacted. For existing secrets, client invokes secret metadata API which returns secret transport key id in response when requested. This id is derived from secret plugin association and plugin transport key association. For new secrets, if there is project level preferred backend specified, then returned transport key is specific to that backend. If project does not have project preferred backend, then it will return transport key specific to global default backend. This way, there should not be case where transport key and secret store backends are different.

Alternatives

One alternative suggested earlier has been to deploy separate Barbican servers to have different endpoints for each supported plugin. This may work though it is not a good usage experience from client perspective. Now onus is on the client to maintain different barbican endpoints in their configuration and to remember on their side which secrets is accessed via which endpoint. Also some of current service integrations generally has support for one barbican endpoint so this may not be feasible for existing service integrations. Then there is standard issue of managing these endpoints in keystone service catalog.

Service Configuration impact

To support multiple plugin configuration, following configuration format is needed as current multiple plugin configuration does not define a clear relationship between secret store and crypto plugin:

[secretstore]
enable_multiple_secret_stores = True
stores_lookup_suffix = software, kmip, pkcs11, dogtag

[secretstore:software]
secret_store_plugin = store_crypto
crypto_plugin = simple_crypto

[secretstore:kmip]
secret_store_plugin = kmip_plugin
global_default = True

[secretstore:dogtag]
secret_store_plugin = dogtag_plugin

[secretstore:pkcs11]
secret_store_plugin = store_crypto
crypto_plugin = p11_crypto

When multiple secret stores support is enabled, then this new list property stores_lookup_suffix is going to be used for looking up relevant supported plugin names in oslo configuration section constructed using pattern ‘secretstore:{suffix}’. One of the plugin must be explicitly identified as global_default = True. There is going be a validation around this requirement. Ordering of suffix does not matter.

When multiple secret stores are enabled, then property enabled_secretstore_plugins and enabled_crypto_plugins are not used for instantiating supported plugins. Instead above mentioned mechanism is used to instantiate store and cryto plugins.

Data model impact

A new table, secret_stores, is going to be created for storing secret store backend information. At barbican application startup, this table is going to add new entry for plugins supported, if not present, based on secret_store_plugin and crypto_plugin combination. One of secret store plugin entry in db will have global_default column value as true.:

table_exists = ctx.dialect.has_table(con.engine, 'secret_stores')
if not table_exists:
   op.create_table(
         'secret_stores',
         sa.Column('id', sa.String(length=36), nullable=False),
         sa.Column('created_at', sa.DateTime(), nullable=False),
         sa.Column('updated_at', sa.DateTime(), nullable=False),
         sa.Column('deleted_at', sa.DateTime(), nullable=True),
         sa.Column('deleted', sa.Boolean(), nullable=False),
         sa.Column('status', sa.String(length=20), nullable=False),
         sa.Column('store_plugin', sa.String(length=255), nullable=False),
         sa.Column('crypto_plugin', sa.String(length=255), nullable=True),
         sa.Column('global_default', sa.Boolean(), nullable=False),
         sa.Column('name', sa.String(length=255), nullable=False),
         sa.PrimaryKeyConstraint('id'),
         sa.UniqueConstraint('store_plugin', 'crypto_plugin',
                             name='_secret_stores_plugin_names_uc'),
         sa.UniqueConstraint('name',
                             name='_secret_stores_name_uc'
   )

At service startup, logic is going to be added to populate secret_stores data. See Getting Global Default Backend for API. This new table is going to have crypto_plugin field to account for software-only and PKCS11 plugin case. Both plugins use store_crypto as secret store backend and difference is in crypto plugin name i.e. simple_crypto vs. p11_crypto plugin.

Each secret store and crypto plugins implementation going to have new config ‘plugin_name’ added in their plugin specific configuration. This name is intended to be user friendly name and we will add default for it which can be updated via config if needed. This plugin name is going to be used when creating new secret store entry. For PKCS11 and DB backend, crypto plugin names are going to be used in name field as they do not directly extend secret store base classes.

Another new table, project_secret_stores, is going to be created for storing per project secret store linking data.:

table_exists = ctx.dialect.has_table(con.engine, 'project_secret_stores')
if not table_exists:
   op.create_table(
         'project_secret_stores',
         sa.Column('id', sa.String(length=36), nullable=False),
         sa.Column('created_at', sa.DateTime(), nullable=False),
         sa.Column('updated_at', sa.DateTime(), nullable=False),
         sa.Column('deleted_at', sa.DateTime(), nullable=True),
         sa.Column('deleted', sa.Boolean(), nullable=False),
         sa.Column('status', sa.String(length=20), nullable=False),
         sa.Column('secret_store_id', sa.String(length=36), nullable=False),
         sa.Column('project_id', sa.String(length=36), nullable=False),
         sa.PrimaryKeyConstraint('id'),
         sa.ForeignKeyConstraint(['secret_store_id'], ['secret_stores.id'],),
         sa.ForeignKeyConstraint(['project_id'], ['projects.id'],),
         sa.UniqueConstraint('secret_store_id',
                             'project_id',
                             name='_project_secret_stores_id_project_uc')
   )
   op.create_index(op.f('ix_project_secret_stores_store_id'), 'project_secret_stores',
                   ['secret_store_id'], unique=False)
   op.create_index(op.f('ix_project_secret_stores_project_id'), 'project_secret_stores',
                   ['project_id'], unique=False)

This table is going to be populated whenever a project level preferred store is set. In case of update of existing project preferred value, existing entry will be removed and new entry is going to be created.

There is no database migration needed as its a new feature.

REST API impact

There is no impact on existing barbican API(s). Following new REST API are going to be added to support this new feature.

In case multiple secret store backend is not enabled and response is not applicable ( /secret-stores/{ID}/preferred ), secret-stores related API will result in error 404.

Listing All Secret Store Backend

A new API resource tree /secret-stores is going to be added for managing available secret storage backends. This API will provide list of available secret store backends. Only project administrator (users with admin role) are authorized to invoke this API.

REST API: GET /secret-stores:

Request:
   GET /secret-stores
   Headers:
      X-Auth-Token: 'f9cf2d480ba3485f85bdb9d07a4959f1'

Response:
   HTTP/1.1 200 OK

   {
      "secret-stores":[
         {
            "name": "PKCS11 HSM",
            "global_default": True,
            "secret_store_ref": "http://localhost:9311/v1/secret-stores/4d27b7a7-b82f-491d-88c0-746bd67dadc8",
            "store_plugin": "store_crypto"
            "crypto_plugin": "p11_crypto",
            "secret_store_id": "4d27b7a7-b82f-491d-88c0-746bd67dadc8",
            "status": "ACTIVE",
            "created": "2016-08-22T23:46:45.114283",
            "updated": "2016-08-22T23:46:45.114283"
         },
         {
            "name": "KMIP HSM",
            "global_default": False,
            "secret_store_ref": "http://localhost:9311/v1/secret-stores/93869b0f-60eb-4830-adb9-e2f7154a080b",
            "store_plugin": "kmip_plugin",
            "crypto_plugin": None,
            "secret_store_id": "93869b0f-60eb-4830-adb9-e2f7154a080b",
            "status": "ACTIVE",
            "created": "2016-08-22T23:46:45.124554",
            "updated": "2016-08-22T23:46:45.124554"
         },
         {
            "name": "Software Only Crypto",
            "global_default": False,
            "secret_store_ref": "http://localhost:9311/v1/secret-stores/0da45858-9420-42fe-a269-011f5f35deaa",
            "store_plugin": "kmip_plugin",
            "crypto_plugin": "simple_crypto",
            "secret_store_id": "0da45858-9420-42fe-a269-011f5f35deaa",
            "status": "ACTIVE",
            "created": "2016-08-22T23:46:45.127866",
            "updated": "2016-08-22T23:46:45.127866"
         }
   }

Reading A Secret Store Backend

Any barbican user can invoke a given secret store backend info. Returned name is derived from friendly plugin name defined in service configuration. Each secret store plugin and crpyto plugins have default name which can be overriden in plugin_name under that plugin section. Only project administrator (users with admin role) are authorized to invoke this API.

REST API: GET /secret-stores/{secret_store_id}:

Request:
   GET secret-stores/4d27b7a7-b82f-491d-88c0-746bd67dadc8
   Headers:
      X-Auth-Token: 'f9cf2d480ba3485f85bdb9d07a4959f1'

Response:
   HTTP/1.1 200 OK

   {
      "name": "PKCS11 HSM",
      "global_default": True,
      "secret_store_ref": "http://localhost:9311/v1/secret-stores/4d27b7a7-b82f-491d-88c0-746bd67dadc8",
      "store_plugin": "store_crypto"
      "crypto_plugin": "p11_crypto",
      "secret_store_id": "4d27b7a7-b82f-491d-88c0-746bd67dadc8",
      "status": "ACTIVE",
      "created": "2016-08-22T23:46:45.114283",
      "updated": "2016-08-22T23:46:45.114283"
   }

Preferred Secret Store Backend

Project administrator (users with admin role) can add, update or remove preferred secret store backend for their project. Project information is derived from project scoped token. In a barbican deployment without keystone setup, project information is obtained from X-Project-Id request header.

Getting Per Project Secret Store Backend

Only project administrator can request a reference to the preferred secret store if assigned previously. When a preferred secret store is set for a project, then new project secrets are stored using that store backend. If multiple secret store support is not enabled, then this resource will return 404 (Not Found title) error.

REST API: GET /v1/secret-stores/preferred:

Request:

GET /v1/secret-stores/preferred
Headers:
   X-Auth-Token: "f9cf2d480ba3485f85bdb9d07a4959f1"
   Accept: application/json

Response:

HTTP/1.1 200 Ok
Content-Type: application/json

{
   "name": "KMIP HSM",
   "global_default": False,
   "secret_store_ref": "http://localhost:9311/v1/secret-stores/93869b0f-60eb-4830-adb9-e2f7154a080b",
   "store_plugin": "kmip_plugin",
   "crypto_plugin": None,
   "secret_store_id": "93869b0f-60eb-4830-adb9-e2f7154a080b",
   "status": "ACTIVE",
   "created": "2016-08-22T23:46:45.124554",
   "updated": "2016-08-22T23:46:45.124554"
}

Setting Per Project Secret Store Backend

Only project administrator can set the per project backend. This API will set or update the per project backend for any new secrets created after that change.

REST API: POST /secret-stores/{secret_store_id}/preferred:

Request:

POST /secret-stores/7776adb8-e865-413c-8ccc-4f09c3fe0213/preferred
Headers:
   X-Auth-Token: '2f30ca66d27d4d0cbff51d44bc5ac66e'

Response:

HTTP/1.1 204 No Content

Deleting Per Project Secret Store Backend

Only project administrator can remove the per project backend.

REST API: DELETE /secret-stores/{secret_store_id}/preferred:

Request:

DELETE /secret-stores/7776adb8-e865-413c-8ccc-4f09c3fe0213/preferred
Headers:
   X-Auth-Token: '2f30ca66d27d4d0cbff51d44bc5ac66e'

Response:

HTTP/1.1 204 No Content

Global Default Secret Store Backend

Global default secret store is used when a project in barbican does not have a preferred secret store setting. In that case, new secrets created under that project will use global secret store to generate and store keys.

Global default secret store value is not expected to change that frequently so its value is managed via service configuration. At service startup, global_default value must be explicitly set in one of plugin related configuration section.

Changing global default secret store plugin in configuration should not impact project (without preferred plugin) existing secrets as those secrets metadata in db already has information about its associated plugin backend.

Getting Global Default Backend

Only project administrator can read the current global default backend value.

REST API: GET /secret-stores/global-default:

Request:

GET /secret-stores/global-default
Headers:
   X-Auth-Token: '2f30ca66d27d4d0cbff51d44bc5ac66e'
   Accept: 'application/json'

Response:

HTTP/1.1 200 Ok
Content-Type: application/json

{
   "name": "PKCS11 HSM",
   "global_default": True,
   "secret_store_ref": "http://localhost:9311/v1/secret-stores/4d27b7a7-b82f-491d-88c0-746bd67dadc8",
   "store_plugin": "store_crypto"
   "crypto_plugin": "p11_crypto",
   "secret_store_id": "4d27b7a7-b82f-491d-88c0-746bd67dadc8",
   "status": "ACTIVE",
   "created": "2016-08-22T23:46:45.114283",
   "updated": "2016-08-22T23:46:45.114283"
}

Operation POST or DELETE on /secret-stores/global-default is not allowed. So invoking that resource should generate 405 (Method Not Allowed) error.

Security impact

No impact as these are new APIs. Some of new APIs will require service administrator privileges to make changes. As result of this change, deployments will have choice to use database as secret store backend for some secrets. Database backend is considered less secure backend with reference to HSM storage option. But this feature in itself does not introduce a security risk as database backend is barbican existing backend and deployer can choose not to use it.

Notifications & Audit Impact

None

Python and Command Line Client Impact

These new APIs need to be supported in barbican client API and CLI (command line interface).

Other end user impact

There should not be impact on existing deployment as its a new feature. By default, this feature will be disabled. Once this feature is enabled, clients will have choice to define and use specific backend for a project. To use this, clients will initially have to use new APIs to set their project level preferred secret store plugins.

Performance Impact

Minimal. On API usage level, this is expected to be infrequent admin operation. Impact related to creating new secret and storing secret flow will be also very minimal.

Other deployer impact

This new feature is going to be disabled by default. There is new flag added to enable i.e. enable_multiple_secret_stores in secretstore section of barbican configuration. Also for multiple plugins support, deployer will need to define supported plugins configuration as mentioned above (different from way currently supported).

There is no migration needed and deployment should work with default configuration as-is.

Developer impact

There should not be any impact on plugin developers and existing API.

Implementation

Assignee(s)

Primary assignee:: arunkant

Other contributors:

Work Items

Add documentation for secret stores API.
Add model layer changes.
Logic to populate secret_stores table data.
Add logic to parse multiple plugin configuration.
Add controller layer changes for new api.
Modify secret create and store logic to leverage this feature.The feature flag needs to be used in related changes.
Add functional test for new APIs.
Update/Add existing functional test around multiple backend support.

Dependencies

None

Testing

Current unit and functional test will be modified to reflect related changes. New unit and functional test will be added for new secret-stores APIs.

Documentation Impact

This is a new feature that will be documented for new APIs and its usage.

References

1. Transport Key Wrapping https://github.com/openstack/barbican-specs/blob/master/specs/juno/add-wrapping-key-to-barbican-server.rst

Deployer Specific Metadata for Secrets

Mon, 04 Apr 2016 00:00:00

Blueprint: https://blueprints.launchpad.net/barbican/+spec/deployer-specific-secret-metadata

Problem Description

Deployers(Service Admins) may require the ability to add additional data to a Barbican Secret, which users cannot access/modify. The data must be immutable for to the user, but a deployer should be able to change it. Secrets contain immutable metadata such as created, updated, algorithm, etc.. The deployer may want to add some data which is also immutable for the user. For example, the deployer may want to store specifics such as the region where the secret is located.

Currently only user metadata can be used, and it is not immutable to a user[1].

Proposed Change

The proposed change will be to add a new attribute to Barbican Secrets in order for deployer meta-data to be stored. A new API endpoint must also be created for the manipulation of the metadata.

Alternatives

Lock usage of metadata and have database admins add keys and values.

Data model impact

A new table will be created called secret-deployer-metadata with secret_id, key`and `value columns.

At the database level a limit will also be placed on the size of the metadata.

Note

Values will all be stored as Strings.

REST API impact

Each current API call for Secrets will now have deployer-metadata as an added attribute. If no deployer-metadata is provided then an empty dictionary will be initialized.

The following will be added to the REST API:

Get deployer-metadata from a secret:

GET /v1/secrets/{uuid}/deployer-metadata
Headers:
    Accept: application/json
    X-Project-Id: {project_id}

200 OK

{
    'deployer-metadata': {
        'description': 'contains the AES key',
        'geolocation': '12.3456, -98.7654'
    }
}

Create/Update deployer-metadata for a secret:

PUT /v1/secrets/{uuid}/deployer-metadata
Headers:
    Accept: application/json
    X-Project-Id: {project_id}

Content:
{
    'deployer-metadata': {
        'description': 'contains the AES key',
        'geolocation': '12.3456, -98.7654'
    }
}

200 OK

{
    'deployer-metadata': {
        'description': 'contains the AES key',
        'geolocation': '12.3456, -98.7654'
    }
}

Note

Only Create/Update deployer-metadata will be needed. To remove the metadata a user can perform the PUT with an empty dict. If a partial model is sent then the whole metadata will be changed to the partial model which has been sent. Values that exist in the data model but not in the PUT will be deleted.

The following will be added to the REST API in order to address individual user metadata items:

Create an individual metadata item in a secret:

POST /v1/secrets/{uuid}/deployer-metadata
Headers:
    Accept: application/json
    X-Project-Id: {project_id}

Content:
{
  "key": "access-limit",
  "value": 11
}

201 Created

Secret Metadata Location: http://example.com:9311/v1/secrets/{uuid}/deployer-metadata/access-limit
{
    "key": "access-limit",
    "value": 11
}

Note

If the item already exists then a 409 Conflict error code will be returned.

Update an individual metadata item in a secret:

PUT /v1/secrets/{uuid}/deployer-metadata/access-limit
Headers:
    Accept: application/json
    X-Project-Id: {project_id}

Content:
{
  "key": "access-limit",
  "value": 11
}

200 OK

{
  "key": "access-limit",
  "value": 11
}

Note

access-limit must already have been created if not a 404 error code will be returned.

Get an individual metadata item in a secret:

GET /v1/secrets/{uuid}/deployer-metadata/access-limit
Headers:
    Accept: application/json
    X-Project-Id: {project_id}

200 OK

{
    "key": "access-limit",
    "value": 0
}

Note

If the access-limit key does not exist then a 404 error code will be returned.

Delete an individual metadata item in a secret:

DELETE /v1/secrets/{uuid}/deployer-metadata/access-limit
Headers:
    X-Project-Id: {project_id}

204 No Content

Note

If the access-limit key does not exist then a 404 error code will be returned.

Security impact

ACLs and Policy must be setup for the new API calls listed above.

Barbican’s policy.json will now include the following:

“secret-deployer-meta:get”: “rule:service_admin”
“secret-deployer-meta:post”: “rule:service_admin”
“secret-deployer-meta:put”: “rule:service_admin”
“secret-deployer-meta:delete”: “rule:service_admin”

Notifications & Audit Impact

If supported, adding/modifying secret-deployer-metadata should be audit events.

Other end user impact

None

Performance Impact

Minimal:

A new table will be added to the database. It will include new alembic scripts to create the new table and it’s associations.

Other deployer impact

Deployer will now have the ability to store secret specific metadata that may be consumed by an application.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: diazjf
Other contributors:: None

Work Items

Phase 1: Database alterations Phase 2: Current and New API alterations and Tests Phase 3: Documentation

Dependencies

None

Testing

Unit tests must be written for internal component testing. Functional tests must be written for testing this new feature as a whole.

Documentation Impact

Barbican API must be updated to include these changes.

References

[1] https://github.com/openstack/barbican-specs/blob/master/specs/mitaka/add-user-metadata.rst

Add KMIP key manager wrapper

Tue, 17 Nov 2015 00:00:00

https://blueprints.launchpad.net/castellan/+spec/kmip-key-manager

KMIP[1] devices are key managers and since Castellan is a key manager interface it seems likely that someone would want Castellan to go directly to a KMIP device instead of a Barbican with a KMIP device on the backend. This blueprint offers a solution to directly talk to a KMIP device using Castellan.

Problem Description

In order to currently talk to a KMIP device with Castellan, Castellan must first talk to a Barbican that is hooked up with a KMIP device on the backend. By creating a KMIP key manager interface, Castellan can interact with the KMIP device directly.

This has the added benefit of allowing a client to decide whether or not they require Barbican sitting in front of their KMIP device. Whether they have their own device or purchase one from the cloud provider, they might not need all of the Barbican functionality or resources and simply want a direct line to the KMIP device.

This will allow for a private or dedicated cloud to use Castellan without needing Barbican.

An important thing to keep in mind is that multi-tenancy is not guaranteed with KMIP devices. Some devices offer the ability to create separate users which does separate keys based on unique user accounts. However, not all devices support this feature. Moreover, KMIP devices do not currently support keystone as a means of authentication. The section below details the future changes and possible support for both multi-tenancy and keystone authentication.

Proposed Change

A module will be created titled kmip_key_manager. This module will live where the key manager interfaces live now and can be used to directly interface with a KMIP device.

In order for the KMIP key manager to take in a context object, two new credential types needs to be created. One is Certificate which will just take in the location of a certificate and the other is KMIP which will be an extension of Certificate. The KMIP credential object will take in all the necessary values in order to establish a connection to the KMIP device including a certificate.

The values used by the KMIP credential object will be the same variables used by Barbican to establish a connection to a KMIP device and can be modified via the castellan.conf file or through castellan itself. This will make it possible for KMIP devices that support separate user accounts to provide some multi-tenant capabilities.

When KMIP devices start supporting keystone tokens, the KMIP key manager class can be updated to take in either the KMIP credential object or keystone token credential object as a context. This will allow for full multi-tenant capabilities that we see today in Barbican.

Here is an example of the new values that will be added to the castellan.conf file:

[kmip]

#
# From castellan.config
#

# Username for user authorized to access KMIP device  (string value)
#username = admin

# Password for KMIP authenticated user (string value)
#password = password

# Use this endpoint to connect to the KMIP device (string value)
#host = localhost

# Port that KMIP device is listening on (string value)
#port = 5696

# Keyfile used by castellan (string value)
#keyfile = '/path/to/certs/cert.key'

# Certificate used by castellan to talk to KMIP device (string value)
#certfile = '/path/to/certs/cert.crt'

# List of certificates and CA's that castellan willaccept (string value)
#ca_certs = '/path/to/certs/LocalCA.crt'

These values can reside in the castellan.conf file or can be added to an already existing configuration file. Castellan just needs to be given the location of the file to read.

The rest of the code will be very similar to the KMIP secret store code within Barbican.[2]

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

KMIP devices do not guarantee multi-tenant functionality. This means users should be aware that when Castellan is the interface to the KMIP device, any user that has access to the KMIP credentials might be able to talk to the device. There is no differentiation based to the user’s role or project because keystone cannot currently be used as a form of authentication. This will need to be heavily documented.

Notifications & Audit Impact

Any CRUD request to the device will need to be logged. Establishing and ending the connection should also be logged to make sure the connection is successful and successfully terminated.

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: silos (Christopher Solis)

Work Items

Create a Certificate and KMIP credential object
Create a KMIP key manager
Add a KMIP section to the sample Castellan configuration file
Unit and functional test the key manager wrapper and credential objects
Documentation

Dependencies

None

Testing

New unit tests and functional tests need to be added.

Documentation Impact

Castellan documentation should be updated to reflect the addition of the new backend.

References

[1] https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=kmip

[2] https://github.com/openstack/barbican/blob/master/barbican/plugin/kmip_secret_store.py

Allow Castellan to Support different types of Keystone Auth

Mon, 02 Nov 2015 00:00:00

Blueprint: https://blueprints.launchpad.net/castellan/+spec/remove-keystone-dependency

Problem Description

Currently in Castellan in order to obtain access to Barbican via the Barbican Key manager a context containing a valid Keystone auth-token must be provided.

The Swift Keymaster under development[1] will access Castellan in order to obtain Secrets from Barbican, but would like to have a Keystone service user with access to all keys and in the future be able to have independent user access. For a service user, a context containing a Keystone username and password will be used. Castellan must support this.

Proposed Change

The proposed change is to allow Castellan to be able to check the context for the Barbican Key Manager and be able to determine what type of Keystone authentication function to use.

There will be a new type of hierarchal context object which will be passed from the user/service to Castellan. It will be used instead of oslo.context.

The hierarchal context will consist of a Credential object as the parent class and the children will be:

1.) TokenCredential, for authenticating with a token.

2.) PasswordCredential, for authenticating with a username and password.

3.) CertificateCredential, for authenticating with a certificate.

The context is first checked to see what type of object it is, after that we determine which Keystone auth-type to use.

Note

oslo.context still needs to be supported until it is deprecated in the future. tenant and project_id should be backwards compatible. Patch https://review.openstack.org/#/c/235671/ adds check for project_id, It allows avoidance of confusion between tenant and project_id and is necessary if passing a keystone auth-middleware object as context.

Example Usage within the swift keymaster:

from castellan.common.objects import symmetric_key
from castellan import credential
from castellan import key_manager

CONF = <swift conf>

# If keystone auth-middleware exists then it provides the authentication
# for the logged in user[2].
ks_context = env.get('keystone.token_auth').user

# Depending on how the configuration is setup, then a different 'credential` object
# will be created and processed by Castellan. More information on this can be
# found in examples below.
context = credential.credential_factory(context=ks_context, conf=CONF)

# create a key
key = symmetric_key.SymmetricKey('aes', 128, '==8hykeh')
manager = key_manager.API(configuration=CONF)
stored_key_id = manager.store(context, key)

The swift configuration must be altered to contain certain variables that Castellan will use for authentication.

For password-based authentication the user must provide the following in the Swift Configuration:

[castellan]
auth_type = 'password'
username = 'swift'
password = 'xswift'
project_id = '1a4a0618b306462c9830f876b0bd6af2'
domain_id = '5abc43'

Note

The above is an example on how it can be used in the Swift Keymaster. It is just provided for demonstration purposes.

The auth_type variable will be used in order to determine what type of credential Castellan should create. The possible auth_types will be token, password and certificate. Castellan will contain a context factory in order to process the different auth_types. Below is an example of the context factory.

def credential_factory(context=None, conf=None):
    if CONF.castellan.auth_type == 'token':
        if context:
            auth_token = context.auth_token
        else:
            auth_token = CONF.castellan.auth_token

        context = catstellan.KeystoneTokenContext(auth_token,
                                                  CONF.castellan.auth_url,
                                                  CONF.castellan.project_id)
    elif CONF.castellan.auth_type == 'password':
        context = castellan.PasswordContext(CONF.castellan.username,
                                            CONF.castellan.password,
                                            CONF.castellan.auth_url,
                                            CONF.castellan.project_id,
                                            CONF.castellan.domain_id)
    elif CONF.castellan.auth_type == 'certificate':
        context = castellan.CertificateContext(CONF.castellan.public_key,
                                               CONF.castellan.private_key)

    return context

Alternatives

A Keystone Auth-Token can be derived using Username and Password. An oslo.context object containing the Auth-Token can be created and passed to a Castellan call.

Data model impact

None

REST API impact

None

Security impact

If v3.Password is being used, then the username, password, project_id, and domain information(id or name) must be stored.

Notifications & Audit Impact

None

Python and Command Line Client Impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Developer has more options of context that can be passed to Castellan.

Implementation

Assignee(s)

Primary assignee:: diazjf
Other contributors:: rellerreller

Work Items

1. Create support for multiple Keystone auth types in Castellan for the Barbican Key manager. 2. Add documentation on how to generate context for the Barbican Key Manager.

Dependencies

None

Testing

New unit tests and functional tests need to be added.

Documentation Impact

Castellan documentation must be updated to include these changes.

References

[1] https://github.com/openstack/swift-specs/blob/master/specs/in_progress/at_rest_encryption.rst [2] https://github.com/openstack/keystonemiddleware/blob/1047cececf68c3c22add66b6a8a1c499a667c036/keystonemiddleware/auth_token/__init__.py#L171-L174

Add User Metadata to Barbican Secrets

Thu, 01 Oct 2015 00:00:00

Blueprint:

https://blueprints.launchpad.net/barbican/+spec/add-user-metadata

Client Blueprint:

https://blueprints.launchpad.net/python-barbicanclient/+spec/add-user-metadata

Currently the only unique information that can be generated for a Barbican Secret by a user is name. This blueprint will add a new field to each Barbican Secret to allow a user to add their own metadata to the Secret.

Problem Description

Users may require to add additional data to a Barbican Secret. A user may want to add a description of the Secret as well as other necessary data such as geolocation, rate, allowed time-access, etc. This will allow user applications/services to check the user-metadata for certain fields in order to allow/disallow an action to Barbican.

Proposed Change

The proposed change will be to add a new attribute to Barbican Secrets in order for user meta-data to be stored. A new API endpoint must also be created for the manipulation of the metadata.

Alternatives

Currently there are no alternatives, since user is not able to add any unique information other than to the attribute name.

Data model impact

Secrets will now have an additional parameter called metadata. metadata will be a dictionary in which a user can add a key and a value.

A limit must also be set as to how much meta-data a user can add to a secret. quota_secret_meta will be added to the Barbican config and set to -1, which allows for unlimited metadata.

At the database level a limit will also be placed on the size of the metadata.

A new table will be created called secret-metadata with secret_id, key and value columns.

This has the following benefits:

1. Searching Benefits The value column can be used to search and the search can stop once an invalid character is found. Indexing on keys can also be done to make searching for keys with a certain tag faster.

2. Extensibility If we wish to add new information, such as created-timestamp, updated-timestamp, etc. we can simply add a new column. This is cleaner than adding some other system.

3. Data Consistency Restrictions can be done at the database level, such as length of the key/value strings. Number of metadata items can be limited by adding column restrictions. This will provide good measures for data integrity, and strengthen security.

Note

Values will all be stored as Strings.

REST API impact

Each Current API call for Secrets will now have metadata as an added attribute. If no metadata is provided then an empty dictionary will be initialized.

POST /v1/secrets
Headers:
    Content-Type: application/json
    X-Project-Id: {project_id}

Content:
{
    "name": "AES key",
    "expiration": "2015-12-28T19:14:44.180394",
    "algorithm": "aes",
    "bit_length": 256,
    "mode": "cbc",
    "payload": "YmVlcg==",
    "payload_content_type": "application/octet-stream",
    "payload_content_encoding": "base64"
    "metadata": {
      'description': 'contains the AES key',
      'geolocation': '12.3456, -98.7654'
    }
}

201 Created

{
    "secret_ref": "https://{barbican_host}/v1/secrets/{uuid}"
}

GET /v1/secrets/{uuid}
Headers:
    Accept: application/json
    X-Project-Id: {project_id}

200 OK

{
    "status": "ACTIVE",
    "created": "2015-03-23T20:46:51.650515",
    "updated": "2015-03-23T20:46:51.654116",
    "expiration": "2015-12-28T19:14:44.180394",
    "algorithm": "aes",
    "bit_length": 256,
    "mode": "cbc",
    "name": "AES key",
    "secret_ref": "https://{barbican_host}/v1/secrets/{secret_uuid}",
    "secret_type": "opaque",
    "content_types": {
        "default": "application/octet-stream"
    }
    "metadata": {
      'description': 'contains the AES key',
      'geolocation': '12.3456, -98.7654'
    }
}

Note

If no metadata is input on the POST then metadata will not be shown.

The following will be added to the REST API:

Get user-metadata from a secret:

GET /v1/secrets/{uuid}/metadata
Headers:
    Accept: application/json
    X-Project-Id: {project_id}

200 OK

{
    'metadata': {
        'description': 'contains the AES key',
        'geolocation': '12.3456, -98.7654'
    }
}

Create/Update user-metadata for a secret:

PUT /v1/secrets/{uuid}/metadata
Headers:
    Accept: application/json
    X-Project-Id: {project_id}

Content:
{
    'metadata': {
        'description': 'contains the AES key',
        'geolocation': '12.3456, -98.7654'
    }
}

200 OK

{
    'metadata': {
        'description': 'contains the AES key',
        'geolocation': '12.3456, -98.7654'
    }
}

Note

Only Create/Update user-metadata will be needed. To remove the metadata a user can perform the PUT with an empty dict. If a partial model is sent then the whole metadata will be changed to the partial model which has been sent. Values that exist in the data model but not in the PUT will be deleted.

The following will be added to the REST API in order to address individual user metadata items:

Create an individual metadata item in a secret:

POST /v1/secrets/{uuid}/metadata
Headers:
    Accept: application/json
    X-Project-Id: {project_id}

Content:
{
  "key": "access-limit",
  "value": 11
}

201 Created

Secret Metadata Location: http://example.com:9311/v1/secrets/{uuid}/metadata/access-limit
{
    "key": "access-llimit",
    "value": 11
}

Note

If the item already exists then a 409 Conflict error code will be returned.

Update an individual metadata item in a secret:

PUT /v1/secrets/{uuid}/metadata/access-limit
Headers:
    Accept: application/json
    X-Project-Id: {project_id}

Content:
{
  "key": "access-limit",
  "value": 11
}

200 OK

{
  "key": "access-limit",
  "value": 11
}

Note

access-limit must already have been created if not a 404 error code will be returned.

Get an individual metadata item in a secret:

GET /v1/secrets/{uuid}/metadata/access-limit
Headers:
    Accept: application/json
    X-Project-Id: {project_id}

200 OK

{
    "key": "access-limit",
    "value": 0
}

Note

If the access-limit key does not exist then a 404 error code will be returned.

Delete an individual metadata item in a secret:

DELETE /v1/secrets/{uuid}/metadata/access-limit
Headers:
    X-Project-Id: {project_id}

204 No Content

Note

If the access-limit key does not exist then a 404 error code will be returned.

Security impact

ACLs and Policy must be setup for the new API calls listed above.

Barbican’s policy.json will now include the following:

“secret-meta:get”: “rule:secret_non_private_read or rule:secret_project_creator or rule:secret_project_admin or rule:secret_acl_read”
“secret-meta:post”: “rule:admin_or_creator and rule:secret_project_match”
“secret-meta:put”: “rule:admin_or_creator and rule:secret_project_match”
“secret-meta:delete”: “rule:admin_or_creator and rule:secret_project_match”

Notifications & Audit Impact

If supported, adding/modifying secret-metadata should be audit events.

Python and Command Line Client Impact

The following commands will need to be added to the CLI:

Secrets:

Secret Metadata Update
Secret Metadatum Update

Other end user impact

None

Performance Impact

Minimal:

A new table will be added to the database. It will include new alembic scripts to create the new table and it’s associations.

Other deployer impact

Deployer will now need to know to set quota_secret_meta if they wish users to be able to add less metadata keys to a secret.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: diazjf
Other contributors:: None

Work Items

Phase 1: Database alterations Phase 2: Current and New API alterations and Tests Phase 3: Add Functionality to CLI

Dependencies

None

Testing

Unit tests must be written for internal component testing. Functional tests must be written for testing this new feature as a whole.

Documentation Impact

Barbican API and Barbican Client must be updated to include these changes.

References

https://blueprints.launchpad.net/barbican/+spec/add-user-metadata https://blueprints.launchpad.net/python-barbicanclient/+spec/add-user-metadata

Add quota support for Barbican resources

Tue, 18 Aug 2015 00:00:00

https://blueprints.launchpad.net/barbican/+spec/quota-support-on-barbican-resources

Barbican REST API doesn’t impose any upper limit on the number of resources allowed per project. This could result in resource explosion. This blueprint proposes a way to specify and enforce quotas with projects. Quotas are operational limits so that cloud resources are optimized.

Problem Description

Here are few scenarios that could impact the normal functioning of the Barbican server:

A client could place requests for several thousands of orders to create secrets for a single project. This might overwhelm the Barbican server both in terms of processing time and disk space consumed
If a buggy client script runs amok and attempts to create a generic type container with no associated secret, it could quickly fill-up the Barbican database!
If a user creates a large number of projects and further creates a large number of Barbican resources per these projects, that could impact other genuine users

Note: The last point could be avoided if keystone could enforce a upper limit on the number of projects that an user could create. Hence this is treated as a lower priority for Barbican for now. This spec aims to implement project level quotas first. A later spec is expected to add support for user level quotas.

This is similar to the quota enforcement done by nova and cinder services.

Proposed Change

Introduce quotas for all Barbican resources. The quotas should have unlimited values as defaults, to ensure backwards compatibility with the current code that supports no quotas. The following resources will have quota support:

secrets
orders
containers
consumers

Note:

This proposal is a simpler subset of the quota implementation done by Nova and Cinder. Barbican does not have any reservable resources and so the quotas are much simpler in that there is no usage tracking and reservations. Also, project-user level quota enforcement is not covered by this spec.

*Enforcing quotas:*

Barbican API controllers will be updated with quota logic for the create resource methods. Once implemented, the quota check will work as follows for resource creation requests:

Get the quotas for the project retrieved from the auth context. If per-project quotas have not been setup, use default quotas
Get a count of the resources for the context project Note: Currently secrets that expire or are removed are not hard removed from the database, but rather are soft deleted, so they still are using resources within Barbican and would be counted against the project’s quota. However, this is deployment dependent. A deployment could have a process that hard removes such resources after a period of time
If the count equals or exceeds the quotas, reject the request with the following error message: HTTP 403 Forbidden {“error”: “Quota exceeded for <project_id>. Only <count> <resource>s are allowed” }
Continue with the resource creation

Update the Barbican config file to include the following section for quota limits:

# ====================== Quota Options ===============================

[quotas]
# For each resource, the default maximum number that can be used for
# a project is set below.  This value can be overridden for each
# project through the API.  A negative value means no limit.  A zero
# value effectively disables creation of the resource.

# default number of secrets allowed per project
quota_secrets = -1

# default number of orders allowed per project
quota_orders = -1

# default number of containers allowed per project
quota_containers = -1

# default number of consumers allowed per project
quota_consumers = -1

A number >=0 for the quota_<value> indicates the max limit for that resource and a negative value means unlimited. A value of zero indicates a maximum of zero instances of that resource, effectively disabling creation of that entity. To ensure backwards compatibility, the provided default for the defaults will be -1 (unlimited) for each resource.

While these generic quotas apply to all projects, there will also be support to define and enforce quotas per project. The priority in which the quotas are enforced is then: [per project quotas] => [default quotas]

where, “per project quotas” - these quotas are directly associated with a particular project id. Any changes to these quotas will impact only that project.

“default quotas” - if no per project quotas are specified, the default quotas are used. These would typically be the values supplied in the config file.

The default quotas are stored in the config file (as shown above) but per-project quotas are stored in db.

A REST API for Barbican administrators for the quota CRUD operations will be implemented as well. Non-admin users will be provided with a REST API to get their own effective quotas for various resources. Keystone RBAC checks will be employed to decide if a caller has the required role to perform these operations.

Quota management should be under the purview of a service level administrator, not a project level administrator. In the current Barbican implementation, there are four user roles: admin, creator, observer, and auditor. All of these roles are project level. Thus, to accomplish appropriate role based access to for quota a management, a new role will be created. The name of this role will be “key-manager:service-admin”.

Alternatives

An attempt was made to create an oslo.common library with quota support for all OpenStack projects. The first attempt was committed to oslo, however it has been deprecated and has not been adopted by any projects. A second attempt was started, but has been put on hold with no current plans to restart. There is no other common OpenStack library implementing quotas for Barbican to adopt.

The quota configuration and logic will be derived by looking at quota implementations done by other OpenStack projects like nova, cinder and neutron. A simplified implementation will developed for Barbican by using APIs and logic similar to Nova’s implementation, while removing unneeded features, such as pluggable backend drivers and resource reservation.

Another alternative is an initiative by Kevin Mitchell from Rackspace https://wiki.openstack.org/wiki/Boson. However, the Nova and Cinder design is more usable for Barbican.

In addition to the four resources specified above, a previous version of this spec described implementing quotas support for transport_keys. This is not possible based on the current implementation of transport_keys, because they are defined per-plugin and not per-project. A project admin should not be managing transport_keys at all.

The name of the role required to manage project quotas is decided to be “key-manager:service-admin”. This foreshadows a future change where all Barbican roles might be defined with a namespace of key-manager. A variety of other role names (“service-admin”, “barbican-admin”, “cloud-admin”) were are also possible alternatives.

Data model impact

The following new data models will be added:

ProjectQuota

Represents quotas override for a project.

If there is no row for a given project id, then the default for the deployment is used. If the quota value for a resource is null, then the default for that resource for the deployment is used. If the quota value for a resource is 0, creation of that resource is disabled. If the quota value for a resource is -1, creation of that resource is not limited by quota enforcement logic.

Schema: (table name: project_quotas)
- project_id: String(36) ForeignKey projects.id, nullable=False
- secrets: Integer, nullable=True
- orders: Integer, nullable=True
- containers: Integer, nullable=True
- consumers: Integer, nullable=True
Constraints: project_id must be unique
project_id must exist as projects.id
Changes to existing models:

No existing models will be impacted by this addition. However, it needs to be investigated if new indexes need to be built to speed up resource consumption lookups.

REST API impact

The following new REST API will be implemented to manage quotas CRUD operations. Please note that except for the first GET API, all the other APIs require the caller to have “key-manager:service-admin” role.

Get effective quotas (any Barbican user)

Returns effective resource quotas for the caller for the specified project. If there are no project specific quotas returns the deployment default resource limits.
GET /v1/quotas
Normal http response code(s) 200 OK
Expected error http response code(s)
- 401 Unauthorized - If the auth token is not present or invalid.
  Also, if using the unauthenticated context and the X-Project-Id header is not present in the request.
Required request headers

X-Auth-Token, if using keystone auth

X-Project-Id, if using unauthenticated context
Parameters

None
JSON schema definition for the body data if allowed

None

JSON schema definition for the response data if any

EXAMPLE:

{
  'type': 'object',
  'properties': {
      'quotas': {
        'type': 'object',
        'properties': {
          'secrets': {'type':'integer'}
          'orders': {'type':'integer'},
          'containers': {'type':'integer'},
          'consumers': {'type':'integer'}
         },
        'additionalProperties': False
      }
  },
  'additionalProperties': False
}

Example 1:

A non-admin user checking the resource quotas using a token scoped to a
particular project

Request:

  GET /v1/quotas

  X-Auth-Token:<token>

Response:

  200 OK

  Content-Type: application/json

  {
    "quotas": {
      "secrets": 10,
      "orders": 20,
      "containers": 10,
      "consumers": -1
    }
  }

List all project quotas (service-admin only)

Lists all configured project level resource quotas across all users for all projects. If a project does not have project specific quotas configured, that project is not included in the returned list. If there are only project specific quotas for a subset of resources for a project, this call will return null for those resources without a configured value in that project. The returned list will be sorted by create date, and support standard limit/offset paging.

The standard paging support includes adding three fields in the response body, when applicable.
- “total”: showing the number of project-quotas records
- “next”: giving a URL to the next page of records
- “prev”: giving a URL to the previous page of records
GET /v1/project-quotas?limit=x&offset=y (service-admin only)
Normal http response code(s) 200 OK
Expected error http response code(s)
- 401 Unauthorized - If the auth token is not present or invalid.
  Also, if using the unauthenticated context and the X-Project-Id header is not present in the request.
Required request headers

X-Auth-Token, if using keystone auth
Parameters

limit(optional), integer, maximum number of records retrieved offset(optional), integer, number of records to skip
JSON schema definition for the body data if allowed

None

JSON schema definition for the response data if any

EXAMPLE:

{
  'type': 'object',
  'properties': {
      'project_quotas': {
        'type': 'array'
        'items': {
          'type': 'object',
          'properties': {
             'project_id': {'type':'string'},
             'project_quotas': {
                  'type':'object',
                  'properties': {
                     'secrets': {'type': 'integer'},
                     'orders': {'type': 'integer'},
                     'containers': {'type': 'integer'},
                     'consumers': {'type': 'integer'}
                  }
             }
           }
         }
        }
     },
  'additionalProperties': False
}

Example 1:

A service-admin user listing all the project quotas

Request:

  GET /v1/project-quotas

  X-Auth-Token:<token>

Response:

  200 OK

  Content-Type: application/json

  {
    "project_quotas": [
      {
        "project_id": "1234",
        "project_quotas": {
             "secrets": 2000,
             "orders": 0,
             "containers": -1,
             "consumers": null
         }
      },
      {
        "project_id": "5678",
        "project_quotas": {
             "secrets": 200,
             "orders": 100,
             "containers": -1,
             "consumers": null
         }
      },
    ],
    "total" : 30,
  }

Example 2:

A service-admin user listing all the project quotas with paging

Request:

  GET /v1/project-quotas?limit=2&offset=6

  X-Auth-Token:<token>

Response:

  200 OK

  Content-Type: application/json

  {
    "project_quotas": [
      {
        "project_id": "1234",
        "project_quotas": {
             "secrets": 2000,
             "orders": 0,
             "containers": -1,
             "consumers": null
         }
      },
      {
        "project_id": "5678",
        "project_quotas": {
             "secrets": 200,
             "orders": 100,
             "containers": -1,
             "consumers": null
         }
      },
    ],
    "total" : 30,
    "next": "http://localhost:9311/v1/project_quotas?limit=2&offset=8",
    "prev": "http://localhost:9311/v1/project_quotas?limit=2&offset=4"
  }

Get quotas for a specific project (service-admin only)
- Returns a set of configured resource quotas for the specified project. If no project specific quota values have been configured (or if the project does not exist), the API responds with Not Found. If there are only project specific quotas for a subset of resources for a project, this call will return null for those resources without a configured value in that project.
- GET /v1/project-quotas/{project_id}
- Normal http response code(s) 200 OK
- Expected error http response code(s)
  - 401 Unauthorized - If the auth token is not present or invalid.
    Also, if using the unauthenticated context and the X-Project-Id header is not present in the request.
  - 404 Not Found - If there are no project quota settings to delete
    for the specified project.
- Required request headers
  
  X-Auth-Token, if using keystone auth
  
  X-Project-Id, if using unauthenticated context
- JSON schema definition for the body data if allowed None
- JSON schema definition for the response data if any:
```
{
  'type': 'object',
  'properties': {
       'project_quotas': {
          'type':'object',
          'properties': {
            'secrets': {'type': 'integer'},
            'orders': {'type': 'integer'},
            'containers': {'type': 'integer'},
            'consumers': {'type': 'integer'}
          }
     }
  },
  'additionalProperties': False
}
```
  - Example:
    Request: GET /v1/project-quotas/1234 X-Auth-Token:<token> Response: 200 OK Content-Type: application/json { "project_quotas": { "secrets": 10, "orders": 20, "containers": -1, "consumers": 10 } }
Update/Set quotas for a specific project (service-admin only)
- Creates or updates the configured resource quotas for the specified project. It is not required to specify limits for all Barbican resources. If a value for a resource is not specified, the default limits will be used for that resource. If the specified project is not previously known to Barbican, a new entry to the projects table will be created.
- PUT /v1/project-quotas/{project_id}
- Normal http response code(s)
  
  204 No Content
- Expected error http response code(s)
  - 401 Unauthorized - If the auth token is not present or invalid.
    Also, if using the unauthenticated context and the X-Project-Id header is not present in the request.
  - 400 Bad Request - If the request payload doesn’t confirm to schema
- Required request headers
  
  X-Auth-Token, if using keystone auth
  
  X-Project-Id, if using unauthenticated context
  
  Content-Type, application/json
- JSON schema definition for the body data if allowed:
```
{
  'type': 'object',
  'properties': {
     'project_quotas': {
          'type':'object',
          'properties': {
             'secrets': {'type': 'integer'},
             'orders': {'type': 'integer'},
             'containers': {'type': 'integer'},
             'consumers': {'type': 'integer'}
          }
     }
 },
 'additionalProperties': False
}
```
- JSON schema definition for the response data if any:: None
  - Example:
    Request: PUT /v1/project-quotas/1234 X-Auth-Token:<token> Body:: { "project_quotas": { "secrets": 50, "orders": 10, "containers": 20 } } Response: 204 OK
Delete quotas for a specific project (service-admin only)
- Deletes the configured resource quotas for the specified project. After this call succeeds, the default resource quotas will be returned for subsequent calls by the user to list effective quotas. If there are no project specific quota configuration, or the project is not previously known in Barbican, Not Found is returned.
- DELETE v1/project-quotas/{project_id}
- Parameters None
- Normal http response code(s) 204 No Content
- Expected error http response code(s)
  - 401 Unauthorized - If the auth token is not present or invalid.
    Also, if using the unauthenticated context and the X-Project-Id header is not present in the request.
  - 404 Not Found - If there are no project quota settings to delete
    for the specified project or the project is unknown to Barbican.
- Required request headers
  
  X-Auth-Token, if using keystone auth
  
  X-Project-Id, if using unauthenticated context
- Parameters
  
  None
- JSON schema definition for the body data if allowed
  
  None
- JSON schema definition for the response data if any
  
  None

Example 1:

Request:

  DELETE v1/project-quotas/1234

  X-Auth-Token:<token>

Response:

  204 No Content

Policy changes

For all service-admin-only APIs, the caller is expected to have a barbican key-manager:service-admin role. The check for this will be added to the Barbican policy.json file.

Once implemented and enforced, all Barbican resource creation API could return a new error message back to the client if the request exceeded the allowed quota limits.

Example:

Request::

  POST /v1/secrets

  X-Auth-Token: <token>

  Content-Type: application/json

  {
    # payload to create secret
  }

Response::

  403 Forbidden

  Retry-After: 0

  Content-Type: application/json

 {
  "error": "Quota exceeded for <project-id>. Only <count> <resource>s
            are allowed"
 }

Class Quotas

Class level quotas are not addressed in this spec. Need another spec to cover the data model impact and REST API for associated CRUD operations.

Security impact

None

Notifications & Audit Impact

None

Other end user impact

The Barbican client (python-barbicanclient) has to be enhanced to consume the Quota REST API mentioned. The following scenarios should be supported.

Quota commands that a regular non-admin barbican user can make:

List all quotas

barbican quota show

Quota commands that only a barbican service-admin can make

List the default quotas applicable to all new projects

barbican quota show
List quotas for a specific project

barbican quota show –project_id <project>
Update quotas for a specific project

barbican quota update –project_id <project> –secrets 50 –orders 10
Delete per-project quotas for a project

barbican quota delete –project_id <project>

Performance Impact

TBD

Other deployer impact

The new data models introduced will be added by a new Alembic version file. If automatic migration is turned OFF, the db migration tool has to be run manually to effect the changes.

Developer impact

Developers integrating with Barbican API/client now need to handle the case where the server could return a quota violation error

Implementation

Assignee(s)

Dave McCowan (dave-mccowan) will be leading the implementation of the code.

Primary assignee:: <dave-mccowan>

Other assignees:

Work Items

Quota db provider source code
Data model additions
Alembic migration version script
Updated default config file with quota section
python-barbicanclient enhancements to support quota operations
New unit tests to test quota related source changes
Update existing resource unit tests to handle quota violation errors
Functional tests

Dependencies

TBD

Testing

New unit tests and functional tests need to be added.

Documentation Impact

A new section about Quotas has to be documented
Existing resource API documentation needs to be updated with quota violation specific errors

References

TBD

Add List of Group-IDs to ACL for Secrets and Containers

Fri, 12 Jun 2015 00:00:00

The URL to the launchpad blueprint is here:

https://blueprints.launchpad.net/barbican/+spec/api-acl-add-group-list

The current access control list (ACL) approach in Barbican only allows for adding user IDs for access to a given secret or container. This blueprint proposes allowing group IDs to be added to ACLs to accommodate users within specified groups access to secrets/containers as well. Adding group support to ACLs would support LDAP group based access to secrets/containers. This blueprint depends on the approval of a Keystone blueprint [1].

Problem Description

Organizations with many users could find managing access to Barbican secrets at the per-user level via the current ACL approach a significant effort. Likewise for engineering systems that wish to manage access to secrets for many devices. An approach to simplify access configuration would be to manage secret and container access at the user group level, as proposed by this blueprint.

Proposed Change

The proposed approach is to add a new one-to-many reference on the current SecretACL entity called SecretACLGroup which would be very similar to the SecretACLUser entity, except with a group_id attribute that refers to a desired group-ID to allow secret access to. Likewise a new ContainerACLGroup entity would be added and referenced via the current ContainerACL entity.

In the barbican.api.controllers’s acls.py module, a new ‘groups’ element within each ACL operation would be added, and handled similarly to the ‘users’ element now.

Alternatives

None.

Data model impact

The ‘Proposed Change’ section calls for adding two new entities: SecretACLGroup and ContainerACLGroup, with additional one-to-many references from the existing SecretACL and ContainerACL entities respectively. No initial records or migrations are needed.

REST API impact

The API will be changed to add a ‘groups’ element to the current ACL JSON message.

Hence the response from GET /v1/secrets/{uuid}/acl would become:

HTTP/1.1 200 OK
{
  "read":{
    "updated":"2015-05-12T20:08:47.644264",
    "created":"2015-05-12T19:23:44.019168",
    "users":[
      {user_id1},
      {user_id2},
      .....
    ],
    "groups":[
      {group_id1},
      {group_id2},
      .....
    ],
    "project-access":{project-access-flag}
  }
}

The following sentence is proposed to be added to the description for the PUT /v1/secrets/{uuid}/acl call:

To delete existing groups from an ACL definition, pass an empty list [] for groups.

Also the following row is proposed added to the Attributes section:

Attribute Name	Type	Description	Default
groups	[string]	(optional) List of group ids. This needs to be group ids as returned by Keystone.	[]

Also the following should replace the ‘Body’ element in the ‘Request/Response’ section:

Body:
{
  "read":{
    "users":[
      {user_id1},
      {user_id2},
      .....
    ],
    "groups":[
      {group_id1},
      {group_id2},
      .....
    ],
    "project-access":{project-access-flag}
  }
}

In addition to the secrets resource API changes, the response from GET /v1/containers/{uuid}/acl would become:

HTTP/1.1 200 OK
{
  "read":{
    "updated":"2015-05-12T20:08:47.644264",
    "created":"2015-05-12T19:23:44.019168",
    "users":[
      {user_id1},
      {user_id2},
      .....
    ],
    "groups":[
      {group_id1},
      {group_id2},
      .....
    ],
    "project-access":{project-access-flag}
  }
}

The following sentence is proposed to be added to the description for the PUT /v1/containers/{uuid}/acl call:

To delete existing groups from an ACL definition, pass an empty list [] for groups.

Also the following row is proposed added to the Attributes section:

Attribute Name	Type	Description	Default
groups	[string]	(optional) List of group ids. This needs to be group ids as returned by Keystone.	[]

Also the following should replace the ‘Body’ element in the ‘Request/Response’ section:

Body:
{
  "read":{
    "users":[
      {user_id1},
      {user_id2},
      .....
    ],
    "groups":[
      {group_id1},
      {group_id2},
      .....
    ],
    "project-access":{project-access-flag}
  }
}

Security impact

The proposed change provides another optional mechanism to access secrets and containers, via a user’s group ID information.

Notifications & Audit Impact

No additional auditing beyond that planned for resource access should be required.

Other end user impact

Support for adding group IDs to the ACL for a secret or container should be added to the Barbican Python client as well.

Performance Impact

For each retrieve of a secret or container, an additional query for the group ACL information will be required.

Other deployer impact

No configuration is required. Changes should be applied in the sequence provided in the ‘Work Items’ section.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: TBD

Work Items

The following work sequence is proposed for a zero-down-time deployment:

Create a CR to modify the Sphinx documentation to include the new ‘groups’ API information, marked as ‘Work in Progress’
Create a CR to add the data model entities and relationships described in the ‘Data model impact’ section above, ensuring that current state code still operates correctly after the data model updates are applied
Create CR(s) to add the SQLAlchemy barbican.model’s models.py classes and relationships to match the above data model changes
Create CR(s) to add the new ‘groups’ list to the ‘acls’ controller module as detailed above, along with unit tests
Add CR(s) to add functional testing as detailed below
Add CR(s) to add ‘groups’ ACL support to the Barbican Python client

Dependencies

This blueprint depends on Keystone middleware providing the group ID information for a authenticated token via the X-Group-Ids HTTP header. A pending blueprint to add this capability to Keystone is available here [1].

Testing

The current functional tests for ACL will be augmented to also test out the new ‘groups’ list.

Documentation Impact

The current documentation for ACL will be augmented to add the ‘groups’ information as detailed in the ‘API impact’ section above.

References

[1] https://review.openstack.org/#/c/188564/

Allow Admins to add CAs

Mon, 01 Jun 2015 00:00:00

https://blueprints.launchpad.net/barbican/+spec/add-cas

Problem Description

In Kilo, code was added to allow clients to select a back-end CA server by specifying a Barbican defined ca_id. These ca_ids can be queried by doing a GET against the /cas interface.

In addition, it is now possible for a project admin to associate a set of CAs with a project, and to define a preferred (or default) CA in case a CA is not explicitly requested. All this amounts to the ability for project admins to constrain project clients to specific CAs.

It is now time to take the next step. Dogtag has implemented the ability to create back-end subordinate CA’s on the fly. We need to expose this functionality to project admins, so that they will be able to create project specific CAs. This would allow all certificates issued to clients in the project to be scoped to the project only.

This project specific scoping improves security by isolating clients, and gives project admins greater control over their certificates. Essentially, it is PKI-as-a-service.

Proposed Change

Creating a CA

Most of the infrastructure we need is already present. We will need to add a POST /cas call to request the creation of a new CA. This call will be restricted to project admins. This call will require at least three parameters:

subject DN for the new CA signing certificate.
name or handle for the new CA
parent_ca_ref - the reference of the CA to which this CA should be subordinate. That is, the CA that will issue the new CA’s signing certificate.

Additional parameters may be required to request specific attributes on the subordinate CA’s signing certificate. These are currently not exposed in Dogtag, and therefore will not be part of the initial implementation.

Two new methods will be added to the CertificateManager interface.

supports_create_ca() - which will be called to determine if a given plugin supports this operation. A default implementation returning False will be added to CertificateManager, Plugins that support this option will have to override this function.
create_ca(ca_create_dto) - which would be called to create the subCA, A default implementation that throws a NotSupported exception would be added to CertManager so that existing plugins would not need to change. The relevant parameters would be passed in the ca_create_dto.

Barbican core would select an appropriate CA plugin, and invoke create_ca(). If no plugin is appropriate, a 400 error will be returned.

Otherwise, once the new subCA is created, the data for that subCA will be returned to Barbican core, which will add the CA to the CA tables.

The newly created CA reference will be returned to the client.

CA Ownership

An additional “owner” field will be added to the CA table. This field will be populated with the project_id of the project admin when a CA is created using the mechanism described above. Changes will be needed to ensure that this field is not overwritten when Barbican-core syncs up its CA table with the CA plugins.

CAs which have been created through the /cas interface (and therefore owned by a project) will only be list-able and view-able by members of the owning project. Moreover, only members of that project will be able to submit certificate orders to that CA.

Deleting a CA

Only project admins will be able to delete a CA. The CA must be owned by the project to be deletable. To do this, a new REST method DEL /cas/{ca_id} will be exposed.

This method would look up the necessary CA plugin in the CA table, and would invoke a new delete_ca(plugin_ca_id) call. This call would make the relevant call on the back-end CA, which would, amongst other things, revoke the CA signing cert.

On success, the CA would be removed from the CA table.

Alternatives

None.

Data model impact

A new field “owner” will need to be added to the CA table.

REST API impact

We will need to add some calls to the /cas/ interface.

POST /cas - create a CA, as described above. The attributes name, parent_ca_ref and subject_dn are required, as seen in the example below.

POST /v1/cas
Headers:
    X-Project-Id: {project_id}

Content:
{
    "name": "Dogtag CA for Project yxz",
    "parent_ca_ref": "https://localhost/v1/cas/{parent_ca_uuid}",
    "subject_dn": "cn=CA Signing Cert for xyz, dc=example.com"
}

DEL /cas/{ca_id} - delete a CA, and revoke the CA signing cert, as described
above.

Also, GET /cas and GET /cas/{ca_id} will need to be modified (most likely in the DB query) so “owned CAs” are only accessible by project members.

Also, a validation will need to be added to the Orders interface to ensure that owned CA’s that are referenced by the request belong to the requestor’s project.

Security impact

None.

Notifications & Audit Impact

Make sure that all the actions above are audited as per the auditing spec. [1]

Other end user impact

python-barbicanclient will need to add methods to perform the new operations.

Performance Impact

Minimal. This is a relatively infrequent admin operation.

Other deployer impact

Migration scripts will need to be run on already existing deployments to add the new “owner” field. The owner field will be nullable, so that the code will handle cases with/without owners.

Developer impact

Plugin developers should not need to make any changes, unless they want to support the new methods.

Implementation

Assignee(s)

Primary assignee:: alee-3

Work Items

Update the CAs and Certificates API documentation.
Write the functional tests.
Add the new “owner” field and migration scripts.
Add the new REST API and internal Barbican core logic.
Add the Dogtag implementation.

Dependencies

None

Testing

The current unit and functional tests will also be modified to reflect these changes.

Documentation Impact

This is a new feature that will need to be documented.

References

Auditing spec https://review.openstack.org/#/c/159938/.

Add Transport Cert Reference

Mon, 01 Jun 2015 00:00:00

Launchpad blueprint: https://blueprints.launchpad.net/barbican/+spec/add-transport-cert-ref

Problem description

Transport keys are used to ensure that the secret is pre-encrypted in such a way that only the client and the back-end store can decrypt the secret.

This is for users which do not trust Barbican, but do trust the back-end secret store. Alternatively, clients that are required by FIPS or Common Criteria (CC) requirements only to use CC certified components may be able to argue for being able to use Barbican if the secrets remain opaque in transit through Barbican to their final storage back-ends.

Currently, the client gets the transport key from Barbican. But if the client does not trust Barbican, this is a potential vulnerability. We need to add the ability for the client to retrieve the transport key from the back-end store directly.

Proposed change

The change here is straightforward. Instead of only returning the transport certificate as a result of the GET /transportkeys/{key_id} call, we will also return a reference to the transport key in the header (TRANSPORT_KEY_URL) This reference will be provided by the plugin, and should be a link to a trusted location where the transport cert could be retrieved. In practice, this would most likely be an HTTPS connection to the backend secret store.

Internally, what this means is that the get_transport_key() method in the secret_store interface would be modified to return a dict containing both the value of the transport key and an external URL. We would use this dict to fill in the contents and header of the response.

As this method already has a None default configuration, this change should not require any changes in existing plugins.

Alternatives

This is a trivial change that makes the transport key story more complete.

Data model impact

None.

REST API impact

A new field will be added to the header in the response to GET /transportkeys/{key_id} as described above.

Security impact

Makes transport keys more secure.

Notifications impact

None.

Other end user impact

The client would need to decide whether to accept the value of the transport cert as returned by Barbican, or to retrieve the value from the provided URL.

Performance Impact

None.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: alee

Work Items

The server side changes can likely be done in a single CR.
Client side changes.

Dependencies

None.

Testing

Unit and functional tests will need to be written.

Documentation Impact

Docs on transport key use will need to be written and updated.

References

None

Add Crypto/HSM MKEK Rotation and Migration Support (Lightweight)

Fri, 08 May 2015 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/barbican/+spec/add-crypto-mkek-rotation-support-lightweight

Currently Barbican has no means to migrate secrets encrypted with a crypto/HSM-style plugin to a new master key encryption key (MKEK) and its associated wrapped project KEKs. This blueprint proposes adding a new Barbican utility that supports completing the rotation process by rewrapping the project KEKs with the new MKEK. Note that unlike the similarly-named blueprint at [1], this blueprint does not call for re-encrypting secrets and is therefore a ‘lightweight’ alternative to that blueprint.

Comparing the two approaches, this lightweight approach just rotates the MEKs and rewraps the project KEKs, leaving each secret’s encrypted data unchanged. The other blueprint rotates the MKEKs, project KEKs and the encrypted secret information.

Thus this blueprint has a less thorough rotation process for secrets which could increase the chances of decrypting the secret’s encrypted data. The likelihood of such an attack is expected to be small however. After all, MKEKs are infrequently rotated due to their low probability of being guessed or compromised. This blueprint’s less thorough approach is expected to be less process intensive and execute faster when there are many secrets stored in the database.

Similar to the other blueprint, this utility would be started after deployers, out of band:

generate new MKEK and HMAC signing keys with a binding to new labels, and then
Replicate these keys to other HSMs that may be in the high availability (HA) group, and then
Update Barbican’s config file to reference these new labels, and finally
Restart the Barbican nodes.

The proposed utility would then rewrap the project KEKs with the new MKEKs, updating the associated project KEK records with the new wrapped project KEKs. Note that only HSM/crypto-style plugins rotations are proposed for this blueprint.

Problem Description

When a secret is stored in Barbican using the crypto-style plugin, a KEKDatum entity (from barbican.model.models) is retrieved for the secret’s project. If no such entities are found for this project, then the following steps occur:

A new KEKDatum entity is created for the project and the specific crypto- style plugin with status of ACTIVE. The ACTIVE status indicates that this entity should be used for secret encryptions from then on.
The bind_kek_metadata() method is invoked on the crypto plugin. The plugin then creates a project-level KEK that will be used to encrypt new secrets for that project. For the PKCS11 HSM plugin, this is the project KEK that is wrapped/encrypted by the MKEK.
Information about this project-level KEK is then added to plugin_meta attribute of the new KEKDatum entity.
When a secret is finally stored, it has a Secret entity created for it to hold metadata, and also an associated EncryptedDatum entity that holds both the encrypted cypher text for the secret, and a reference to the project-level KEKDatum record.
Thereafter, new secrets that need to be encrypted for this same project will used this KEKDatum entity, with no further attempts made to generate a new entity.

Hence the KEKDatum entity is associated with project-level KEKs, which in turn are associated with a single MKEK in the HSM. As rotation involves creating a new MKEK (but not the project KEKs for this lightweight version), the KEKDatum entity needs to be updated per project to rewrap/encrypt the project KEK it contains.

Unlike the more stringent key rotation blueprint, no other steps are needed. The secrets, there UUIDs and their associated encrypted data, do not need to change at all, which should speed the overall migration process.

A requirement is that this utility be resilient, allowing for the utility to be re-run if it fails midway.

This blueprint details an approach to rewrap existing project KEKs with a new MKEK.

Proposed Change

This blueprint proposes the following steps be taken to complete the KEK rotation and migration effort:

Out of band to Barbican (so by deployers), new MKEK and HMAC keys are created in the HSM, with new unique labels. For high availability configurations such keys must be replicated across all HSMs in the cluster.
For each API and worker node in the network, update their /etc/barbican/barbican-api.conf files’ mkek_label and hmac_label attributes with the new unique labels generated above. Note that once Barbican is restarted, secrets added to new project-IDs will start to use the new MKEK to wrap their new project KEKs. Existing secrets will still utilize the old MKEK though hence the need for the new utility.
Restart the API and worker nodes. Barbican should be running normally at this point, with the following utility-based steps occurring while Barbican operates.
Via a new utility proposed in this blueprint, query for all KEKDatum entities for a specified crypto-style plugin class (e.g. barbican.plugin.crypto.simple_crypto.SimpleCryptoPlugin). These wrapped-project KEKs need to be rewrapped with the new MKEK.
For each KEKDatum entity, invoke a new method on the crypto-style plugin contract (defined in barbican.plugin.crypto.crypto.py) called ‘rewrap_project_kek()’, which takes the entity as input and updates it with the rewrapped project KEK. The plugin would need to load the wrapped project KEK into the HSM, decrypt it with the old MKEK, encrypt it with the new MKEK, and then return the new wrapped project KEK (but still containing the original project KEK).
The KEKDatum entity above would then be updated with the new wrapped project KEK information. Since this is an in-place update, the secrets associated with this entity do not have to be updated at all, unlike [1].
Info log the UUID of the migrated KEKDatum entity to produce a record of the migration.

The proposed utility would be implemented as a Python barbican.cmd script and added as a pbr entry point in the setup.cfg file, and hence available as a callable command once Barbican is deployed.

To preserve data integrity steps 5 and 6 should be performed in a database transaction.

Alternatives

See the more heavy-weight alternative approach to key rotation at [1].

Data model impact

No data model or repository changes are needed for the proposed solution. Data migrations will be required as detailed in the Proposed Change section above.

REST API impact

None.

Security impact

The proposed solution completes necessary key rotation processes by rewrapping project KEKs used to encrypt secrets. There are improbable but possible data loss risks with the proposed approach however, since project KEKs encrypted with old MKEKs are replaced with these same project KEKs encrypted with the new MKEKs. If the database update transaction fails and corrupts this record (very unlikely with ACID compliant databases, but possible), decryption would fail for all secrets derived from the failed project KEK. However, since these wrapped project KEKs do not change often (on the order of the MKEK rotation schedule) recovery from database backups is very likely, thus mitigating this risk.

Also, if the MKEK is compromised and if the attacker has access to the database backups for secrets they can then decrypt them by first unwrapping the project KEKs. Deployers should be mindful of this and securely store backups.

Notifications & Audit Impact

A log of each KEKDatum entity migrated is produced by the proposed utility, which could be used to prove to auditors that a migration/rotation occurred.

Other end user impact

None.

Performance Impact

The proposed utility could take significant time to process if there are many project-IDs to migrate, and thus it would represent a load on the HSM to re-wrap the project KEKs, impacting normal Barbican operations. This utility would be called quite infrequently however (maybe 1 to 4 times a year). Since there are fewer project IDs/KEKs then secrets, this utility would be more performant than the ‘all secrets’ migration proposed in [1].

Other deployer impact

The proposed utility would be executed as a new executable command available after deployment Barbican. The proposed changes would not require updates to the configuration file schema, but would require updates to provide new MKEK and HMAC key labels as detailed above.

Developer impact

None.

Implementation

Assignee(s)

TBD

Work Items

The proposed work items are:

Create a Python command script to implement the steps in the Proposed Change section.
Add unit testing to cover the new code lines.
Add integration unit test that, using the default simple_crypto.py plugin and an in-memory SQLite database, first creates a few known secrets with the default MEK in the config file, and then modifies this MEK, and then executes the migration script logic. The secrets should be decrypted and verified for accuracy. The KEKDatum entities should have their updated_at dates updated. The secrets should again be decrypted and verified for accuracy, which proves the migration of all secrets to the updated KEKDatum record occurred successfully.
Document the overall key rotation and migration process, including the usage of the new migration utility.

Dependencies

None.

Testing

No DevStack functional tests are expected at this time. Once an HSM gate job is added, a future blueprint add tests will be added.

Documentation Impact

The last work item details the required documentation.

References

[1] https://blueprints.launchpad.net/barbican/+spec/add-crypto-mkek-rotation-support

Remove the tenant-secret association table

Thu, 07 May 2015 00:00:00

https://blueprints.launchpad.net/barbican/+spec/data-remove-tenant-secret-assoc

Currently all secrets stored in Barbican are indirectly associated with Keystone ‘projects’ (formerly known as ‘tenants’) via a tenant-secret association table. This association was added early in Barbican’s lifecycle, when it seemed that such an association would aid in retrieving different components of grouped secrets with different RBAC behaviors. However, recent discussions with OpenStack projects have indicated that this feature would be better implemented in a different way. Hence this association adds an additional join on selects with no benefit, and therefore should be removed.

Problem Description

Secrets in Barbican are indirectly related to Keystone ‘projects’ via a TenantSecret association table. This association is not utilized for any features in Barbican currently but was envisioned to allow multiple ‘projects’ to access the same secret with different access privileges. The team has since had numerous discussions with OpenStack projects that wish to integrate with Barbican, and access control could either be accomplished via Keystone Trusts, or else via ‘user’ and ‘project’ level policy based on metadata configured at the secret level (see blueprint at [1])

Hence the TenantSecret association is not required for any envisioned Barbican use cases and therefore represents a wasteful join for all secret selects.

Proposed Change

This blueprint proposes removing the unused TenantSecret association.

Alternatives

None

Data model impact

The TenantSecret table would be removed, along with associated SQLAlchemy models and repository queries. This schema modification would require a non-trivial migration for zero downtime, as detailed in the ‘Work Items’ section below.

REST API impact

None

Security impact

None

Notifications & Audit Impact

None

Other end user impact

None

Performance Impact

Removing the TenantSecret association removes one join from secret-related queries, and removes Python code needed to create the SQLAlchemy TenantSecret model. Hence performance and code readability should be improved.

Other deployer impact

Migration scripts would need to be run on already existing deployments. There will be no other impact otherwise.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: john-wood-w

Work Items

The schema modification called for would need to be phased in to avoid breaking existing deployments with secrets stored using the current TenantSecret associations.

The first phase would add the new and initially nullable ‘tenant_id’ FK relationship to the secret entity, with all new secrets added setting this FK, and still adding the TenantSecret relationships (until older versions are rotated out). Secret retrievals would first attempt the simple query by FK, and fallback to the current TenantSecret join. The following mods would be needed:: * models.py: Add a ‘secrets_assoc’ relationship in the Tenant model class to the secrets model, similar to the ‘Container’ relationship. To the Secret model, add a ‘tenant_id’ FK reference similar to the Container model’s ‘tenant_id’ FK reference, but with nullable=True. * repositories.py: In the SecretRepo repository class add a query for secrets without the TenantSecret join, but on errors revert to the existing TenantSecret join query. * plugin/resources.py: Add setting the new ‘tenant_id’ FK on secrets to the ‘_save_secret()’ function. * store_crypto.py: Add setting the new ‘tenant_id’ FK on secrets to the ‘_store_secret_and_datum()’ function.

The second phase would provide an Alembic migration script to set the ‘tenant_id’ FK on all secrets that don’t already have it set. The ‘tenant_id’ FK would then be set as nullable=False.

The third phase would remove all logic related to the TenantSecret relationship. The following mods would be needed:: * models.py: Remove TenantSecret SQLAlchemy model class. * repositories.py: Remove TenantSecret joins from queries in the SecretRepo repository class. Remove the ProjectSecretRepo repository class. Remove ‘ProjectSecretRepo’ from the Repositories class. Remove the ‘get_project_secret_repository’ function. * plugin/resources.py: Remove ‘new_assoc’ lines from the ‘_save_secret()’’ function. * store_crypto.py: Remove ‘new_assoc’ lines from the ‘_store_secret_and_datum()’ function. * api/test_resources.py: Remove ‘project_secret_repo’ lines from the ‘_test_should_add_new_secret_metadata_without_payload’ test method. * test_repositories.py: Remove all ‘TenantSecret’ related lines. * test_store_crypto.py: Remove all ‘project_secret_repo’ lines.

The fourth phase would provide an Alembic migration script to remove the TenantSecret relationship from all secrets, and to then drop the TenantSecret table altogether.

Dependencies

None

Testing

The current unit tests will also be modified to have this change reflected upon them. Migration testing across the different phases would be needed, at least manually.

Documentation Impact

None

References

None

Define Content Types for API and Secret Store

Mon, 05 Jan 2015 00:00:00

The current implementation of Barbican is ambiguous to end users and developers as to the encoding that should be used to input new secrets and the encoding of secrets that are returned. This spec aims to define the encoding specifications for each of the secret types and define all of the secret types. The encoding specifications will allow end users and developers to know how to handle the secrets.

Problem Description

The biggest problem that currently exists is that data returned from the API and SecretStores is not clearly defined. An example would be public keys. When an end user requests a public key there were many options for how to return that data. One was to return just the public key (not the certificate) as a Base64 encoded string, but the question still remained of what was the format of the binary representing the key. However, most interpreted this to mean return a certificate. Then the question was raised as to how to return the certificate. Should it be PEM or DER format. Clearly there were lots of assumptions and unanswered questions.

There are two problems above. One is that we need to clearly define all of the possible secret types. This will eliminate the public key versus certificate issue. The other problem is that we need to define the encoding standard options for each type of secret. Then end users and SecretStore developers will know that if they have a secret of type X then it has an encoding of type Y.

The scope of this blueprint is limited to unencrypted secrets. In particular this spec does not cover the encoding standards for keys wrapped with a transport key nor does it cover the encoding scheme for private keys that are encrypted with a passphrase.

Proposed Change

Define All Secret Types

The first change will be to modify the SecretType class to include all of the possible secret types. The SecretType class represents an enum, and the currently defined values are symmetric, public, and private.

This spec proposes adding the types certificate, passphrase, and opaque data. The certificate type would represent X.509 certificates. The current spec would not support PGP certificates, but mostly because I do not know much about them. If support is needed then we can work through that. More than likely I think that if certificates other than X509 are to be supported then a subtype enum for certificates should be created to indicate the type, X.509, PGP, etc. That would be for another spec, and this one will assume only X.509 certificates.

The opaque data type would represent an unknown blob of data. This is reserved for types that are not keys or certificates. Barbican will not be able to understand any of the qualities about the data. It will simply be a blob. This is the type that would be used for secret data like IVs.

Here is a listing of the proposed secret types

symmetric key
public key
private key
passphrase
certificate
opaque data

Content Encoding for Each Secret Type

The second change will be to define the content encoding for each of the secret types. The types and proposed encodings are below.

symmetric key - Base64 encoding of byte array in network byte order
public key - Base64 encoding of DER-encoded SubjectPublicKeyInfo structure as defined by X.509 RFC 5280 with PEM header and footer
private key - Base64 encoding of PKCS#8 structures with PEM header and footer
passphrase - text with utf-8 encoding
certificate - Base64 encoding of DER-encoded ASN.1 X.509 structure with PEM header and footer
opaque data - Base64 encoding of byte array in network byte order

The formats for the public and private keys were chosen based on their popularity. The public key format of SubjectPublicKeyInfo was chosen because that is the default format for public keys by openssl and Java. Plus it is a part of the common X.509 standard.

The private key format of PKCS#8 is the default encoding for Java. The default encoding for generating new private keys in openssl is not PKCS#8. The default encoding in openssl is referred to as the “traditional” format in openssl documentation. The term traditional is ambiguous. It is in fact the plain private key structure, so for RSA it is the private key structure as defined in PKCS#1. This is a subset of the PKCS#8 structure.

The openssl library does support using PKCS#8 keys, and this does not require changing any command line options. The library automatically detects the encoding and uses the key if it is in PKCS#8 encoding or openssl “traditional” encoding. Because of this and the ambiguity of the term “traditional” I recommend using the PKCS#8 format. It is clear to developers and users of the expected format.

Alternatives

One alternative would be to leave everything as is. The downsides of this approach are identified above.

There may be other ways to represent the keys. We could define our own structures instead of using ASN.1. I chose to use ASN.1 and other standards because I think they are widely adopted and accepted. I think exporting keys from existing applications makes the choices the best option.

Data Model Impact

The only expected change is to modify the Secret entity to include a type column. The type column will be one of the predefined secret types of symmetric key, public key, private key, passphrase, certificate, or opaque data.

The secret stores currently expect an array of bytes that are Base64 encoded, so I do not anticipate changing anything with the secret store classes. The only difference now is that secret stores will have more insight as to the format of those bytes.

REST API Impact

The API for storing keys requires setting the payload_content_type and payload_content_encoding for the secret. For each type of secret there needs to a set of acceptable payload_content_type and payload_content_encoding values that are acceptable.

This also means that the secret type should be supplied as a new parameter. This was something that was missing before. The current implementation can only infer the secret type for symmetric keys that are uploaded because it can use the algorithm to determine that. The algorithm is not enough to infer the secret type for asymmetric keys since it could be a public or private key. Therefore a secret type should be provided. The current implementation currently stores them as unknown.

The table below lists the acceptable content types and content encoding values for each type of secret. Each pair is represented as (payload_content_type, payload_content_encoding). API calls that do not receive data in the specified encoding will receive a 406 error code response.

symmetric key (application/octet-stream, Base64)
public key (application/octet-stream, Base64)
private key (application/pkcs8, Base64)
passphrase (text/plain, utf-8)
certificate (application/pkix-cert, Base64)
opaque data (application/octet-stream, Base64), (text/plain, NULL)

Supporting Backwords Compatability

The new parameter for secret type will be an optional parameter to support backwards compatability. If it is not supplied then the key type opaque data will be used. Opaque data will be the only secret type to support multiple acceptable content types. This was chosen to support backwards compatability.

Secret Store Impact

The API content type and content encodings are defined above. However, the format of keys passed to the secret stores needs to be defined as well. The keys are passed from Barbican Core to the secret store using the SecretDTO object.:

+-------------+       +-------------+
| SecretType  |       |  KeySpec    |
+-------------+       +-------------+
| SYMMETRIC   |       | algorithm   |
| PUBLIC      |       | bit_length  |
| PRIVATE     |       | passphrase  |
| CERTIFICATE |       +-------------+
| OPAQUE      |        ^
+-------------+        |
           ^           | has
           | has       |
           |           |
        +---------------+
        | SecretDTO     |
        +---------------+
        | type          |
        | secret        |
        | key_spec      |
        | content_type  |
        | transport_key |
        +---------------+

The current SecretDTO has a content_type property and a secret property. The secret property is a Python string that is the Base64 encoding of the bytes that represent the secret. The content_type is the content_type parameter received from the API call. Since we are defining a single encoding for each SecretType of a SecretDTO then this parameter can be removed. The encodings will be the same as that of the API except that opaque data will be as a Base64 encoding of the bytes and not allowed to be text/plain.

Security impact

None

Notifications & Audit Impact

The ability to audit the types of keys that are managed by Barbican could be added.

Other End User Impact

We can validate secrets before they are stored to validate the structures are correct. This will help users by preventing them from uploading secrets of a specific type that are not formatted correctly.

Performance Impact

None, unless validation is done in which case there will be minor overhead to validate the secret structures.

Other deployer impact

None

Developer impact

Secret store developers will need to possibly modify their code to return secrets in the specified format.

The Barbican client will need to be updated to utilize the encodings and add the secret type parameter.

Implementation

Assignee(s)

Nathan Reller (rellerreller)

Work Items

Add secret type

Modify the API to accept a new secret_type parameter. The secret_type must be one of the predefined types, so a validator should be added for that. If not provided then use opaque data unless symmetric algorithm is provided.

Modify barbican.plugin.resources to create a SecretDTO with the appropriate secret type.

Add basic secret type validators

Add some basic validators that ensure the secrets passed to the API are formatted correctly. This first check would simply be to validate PEM headers. For example, when passing an RSA key this would check that the proper PEM header is included.

This step will _not_ do a deep inspection of the secrets passed to Barbican. A deep inspection would validate each of the structures to make sure the correct structures are received. For RSA public keys this would check that it contains a modulus and public exponent.

Update API documentation

Update the documentation to include notes on the expected encodings and include documentation on the new secret type argument for posting secrets.

Update barbicanclient

Update barbicanclient to take advantage of the new secret types and comply with the API changes.

Dependencies

None

Testing

Add validator tests to verify the API is correctly parsing the new secret type and catching the instances where the incorrect encoding is used for keys passed to the API.

Documentation Impact

Update the API documentation to include information on the new secret type API parameter and the encodings for each of the secret types.

References

PKCS#8 -https://tools.ietf.org/html/rfc5958 X.509 - https://tools.ietf.org/html/rfc5280 Java Key - http://docs.oracle.com/javase/7/docs/api/java/security/Key.html openssl to pkcs8 - https://www.openssl.org/docs/apps/pkcs8.html

Add ability for functional tests to run as different users

Wed, 19 Nov 2014 00:00:00

https://blueprints.launchpad.net/barbican/+spec/add-run-as-for-functional-tests

Currently all Barbican functional tests run under the same userid. This blueprint adds the ability for a Barbican functional test to run under a different userid from that default. This is needed to support the ability to execute tests in parallel and ensure that the underlying system state doesn’t change as a result of other concurrently executing tests.

Problem Description

Today all Barbican functional tests run under the same user. Since these tests run in parallel then any assumptions about the underlying system state are invalid - for example, a test for paging secrets could fail if another tests does secret operations in parallel.

Proposed Change

The proposed change not only solves this problem, but also provides a generalized way to run tests in an isolated manner, under their own userid.

The first step involves determining which test(s) need to run as a non-default user. This is done at testcase creation time, by the testcase author.

Next the testcase author will create their test within class that extends a proposed new class (functionaltests.api.base.DynamicUserTestCase) which extends the existing functionaltests.api.base.TestCase to indicate that the test(s) should not be run under the default user, but rather under a new user.

The user name, password, and tenant name for this new user are derived from settings in barbican’s etc/dev_tempest.conf file. Under a new [key-manager] section are three new settings:

dynamic_user_userid_prefix=testuser_
dynamic_user_password=topsecret
dynamic_user_tenant_prefix=testtenant_

As part of each invocation of the test, the oslotest fixture for the DynamicUserTestCase setUp() code will:

generate a new, unique user - new userid = dynamic_user_userid_prexfix concatenated with a UUID - new password = value of dynamic_user_password - new tenant name = value of dynamic_user_tenant_name_prefix concatenated with the same UUID that was used for the new userid.
login that user
run the test
logout and delete the user we created above

Next the test itself will be run as usual.

When the test completes, the oslotest fixture tearDown() code will logout and delete the new user.

The components of barbican that will need to be updated to support this new function are:

new class added to functionaltests.api.base.py called DynamicUserTestCase that will create and login a new user for use with that testcase.
DynamicUserTestCase.setup() will create and login the new user
DynamicUserTestCase.tearDown() will delete that user
Support functions in functionaltests.api.base.TestCase to talk to the auth provider for userid creation, login and deletion.

Alternatives

One alternative that was discussed was to create a new decorator called @run_as for which could be used for any test, not requiring a new subclass of Testcase.

We rejected that approach for several reasons:

Tests requiring this decorator generally have some other focus of their testing. For example, pagination tests for secrets have their focus on pagination, not on secrets. It makes sense that these tests aren’t grouped in the same class as secrets, but rather in their own subclass.
Adding decorators where they aren’t necessarily the best mechanism can lead to confusing and hard to debug code. Care should be taken to ensure that a decorator is really the best way to implement.

Data model impact

No data model impact.

REST API impact

No REST API impact.

Security impact

This change allows dynamic user creation and deletion on a per-testcase basis. It also allows a testcase to specify an existing user to run under. There are no changes to any API as all of these updates are focused under the functionaltests and do not affect the Barbican API.

Notifications & Audit Impact

No notification/audit impacts for this change.

Other end user impact

The test logging will show the user id that each test runs under in the test logs. They are not part of the API, but could be useful for problem determination and debugging.

Performance Impact

The only performance impact of this CR is on the functionaltests path. It has no effect on the product’s performance.

Tests that request a different user will incur additional pathlength to:

create and login the user (before the test runs)
delete the user (after the test completes)

Other deployer impact

Need to ensure that the fields for dynamic user creation in etc/dev_tempest.conf are acceptable for the tester.

Developer impact

Developers adding functional tests can choose to have those tests run under a different userid, if they desire. Otherwise, all functional tests will run under the default user.

Unit tests are NOT impacted by this change.

Implementation

Assignee(s)

Primary assignee:: sheyman (hockeynut)

Work Items

Upon approval of this blueprint we will code and post for review the implementation of the new class as well as the supporting code in the utilities and fixtures.

Dependencies

This function depends on keystone for the user and tenant creation and deletion API.

Testing

This code is part of the functional testing of Barbican and will be exercised in the devstack gate as well as locally on a developer/tester machine by running “tox -e functional”

Documentation Impact

Currently there is no documentation on how to write Barbican tests (it is an open story in the Barbican work backlog). The prose in this blueprint can be used as a starting point to describe the usage of the DynamicUserTestCase.

An important item to cover in the documentation is guidance as to when a test needs to run as a DynamicUserTestCase. We should use an example such as:

test A that creates a large number of secrets, then fetches the list of all secrets and validates the size of that list against the number created.

test B that simply creates a new secret.

If test B runs in the middle of the execution of test A then the secret count expected by test A will be off by one, and test A will fail.

However, if test A runs as a different user from test B then there will be no interference between the two and both tests will pass as expected.

References

Barbican wiki: https://wiki.openstack.org/wiki/Barbican
Barbican source code: https://github.com/openstack/barbican
Barbican functional tests: https://github.com/openstack/barbican/tree/master/functionaltests

Common Certificate API

Tue, 18 Nov 2014 00:00:00

https://blueprints.launchpad.net/barbican/+spec/certificate-order-api

Problem Description

We currently have a mechanism for enrolling a certificate request by creating a new order, and passing parameters in the metadata of the order. Currently, the parameters in the metadata are passed through to the CA plugin unchanged. This means that - as long as the parameters passed in match the parameters expected by a particular back-end, then a certificate request can be created for that back-end.

This diminishes the power of the abstraction though. A client should not have to know which CA is served by the Barbican server. Rather, a client should be able to request a certificate through a common API, which should be supported by all known certificate plugins.

The trick is to define an interface/metadata parameters that would be supported by most CA’s. An attempt was made to do this in [1].

Recently, it was suggested to use RFC 7030 [4], as a standards-based approach for a common API. RFC7030 basically proposes an HTTP interface for submitting CMC (RFC 5272) requests, and receiving the relevant responses.

RFC 7030 is too different from the current Orders interface and functionality for us to consider adding for Kilo.

On the other hand, CMC requests - both simple and full - as defined in RFC 5272 [2] - have been standards for some time now, and are supported by most, if not all, Certificate Authorities. This makes them ideal candidates for objects in a common set of parameters supported by multiple back-end CA’s.

Ideally, the CMC request could be passed intact from barbican core through the plugin to the backend CA. The following CA’s are known to support CMC requests directly: Dogtag, Microsoft CA.

Even if a CA does not support CMC requests directly, the CMC request provides a convenient standard-based vechicle for describing all the requested x509 extensions, request data and authentication mechanisms like POP. These are things that we do not want to try to re-invent.

And of course, as simple CMC == PKCS10 - which should be supported by all CAs, writing a plugin method to support simple CMC requests should be trivial.

Proposed Change

We will continue to use the Orders resource to enroll certificate requests, and the order metadata to contain the request parameters. Order requests can look like this:

Option 1: Order using a Simple PKI Request (See Section 3.1 in RFC 5272):

{
    "type": "certificate",
    "meta": {
        "ca": "CA identifier (optional, see note in reference [3] below)",
        "request_type": "simple-cmc",
        "request_data": "base64 encoded simple CMC request (PKCS10)"
        "requestor_name": "(optional) string"
        "requestor_email": "(optional) email address - RFC 5322, 5321"
        "requestor_phone" : "(optional) string"
     }
}

Option 2: Order using a Full PKI Request (PKCS7, see section 3.2 in RFC 5272):

{
    "type": "certificate",
    "meta": {
        "ca": "CA identifier (optional, see note in reference [3] below)",
        "request_type": "full-cmc",
        "request_data": "base64 encoded full CMC request (PKCS7)"
        "requestor_name": "(optional) string"
        "requestor_email": "(optional) email address - RFC 5322, 5321"
        "requestor_phone" : "(optional) string"
     }
}

Option 3: Order using keys already stored in Barbican:

{
    "type": "certificate",
    "meta": {
        "ca": "CA identifier (optionali, see note in reference [3] below)",
        "request_type": "stored-key",
        "public_key_ref": "barbican UUID for public key",
        "subject_dn": "subject DN (RFC1485 [5])"
        "extensions": "base 64 DER encoded ASN.1 values (RFC 5280)"
        "requestor_name": "(optional) string"
        "requestor_email": "(optional) email address - RFC 5322, 5321"
        "requestor_phone" : "(optional) string"
     }
}

Extensions data defined by the following in RFC 5272 (section 3.1), where
the specific ASN.1 structures for each extension are defined in RFC 5280
(section 4.2) [6]:

Extensions  ::=  SEQUENCE SIZE (1..MAX) OF Extension

Extension  ::=  SEQUENCE  {
     extnID      OBJECT IDENTIFIER,
     critical    BOOLEAN DEFAULT FALSE,
     extnValue   OCTET STRING
                 -- contains the DER encoding of an ASN.1 value
                 -- corresponding to the extension type identified
                 -- by extnID
     }

Order 4: Orders using parameters for a specific CA.

This mechanism is already supported by the existing code. There is a blueprint for an API to retrieve the possible parameters from the server [7]:

{
    "type": "certificate",
    "meta": {
        "ca": "CA identifier (optional, see note in reference [3] below)",
        "request_type": "custom",
        "ca_specific_parameter1": "value",
        "ca_specific_parameter2": "value2",
        ...
     }
}

In the Barbican server, logic will need to be added to differentiate between these different request types. For backward compatibility, if request_type is not provided, then we will assume that it is “custom”.

On the server, we can add the following logic:

For Orders 1 and 2, we may be able to validate whether the request_data is valid PKCS10/ PKCS7. To do this, we could use PyASN.1 to attempt to decompose the request into the relevant ASN.1 structure, and fail if the decomposition is unsuccessful.
For Order 3, we should retrieve the relevant public key, create a PKCS10 request using the public key, subject DN and extensions data. Submit to backend as a simple CMC request.
For Orders 1,2,3, submit to plugin through new CertPlugin API calls: issue_simple_cmc_request(), issue_full_cmc_request().
Plugins will interact with the back-end CA, which presumably will return PKCS7 full CMC responses.

The plugin will be responsible for interpreting the response and returning a ResultDTO object to barbican-core. Processing would then proceed as already coded in Barbican-core.

In order to simplify the plugin code, and to avoid unnecessary duplication of code, barbican-core will provide some utility functions to parse a CMCResponse and convert the result into a ResultDTO object.

The details of this work need not be specified in this spec, but essentially this involves mapping the CMCStatus field to a Barbican CertificateStatus values, and extracting certs, intermediates, Query Pending control/ pendInfo data, and/or error messages as appropriate.
The new and existing certificate request issuance methods (issue_*_cmc_request, issue_certificate_request) will be provided default implementations (which would raise NotImplemented exceptions) in case any plugins have not yet implemented the relevant methods.

In addition, the supports() on each plugin would need to be modified to specify whether the plugin supports each certificate issuance method.

Moreover, the API that identifies the available CAs will also include information on which methods are supported by each plugin, so that clients can determine which method to use for a particular CA a priori. This information can be extracted from the supports() method. See [3] for more information.

Note on Certificate Request Updates:

Typically, a CA will not allow a certificate request to be amended once it has been submitted, with the exception of the optional requestor data.

For the certificate update API, therefore, I propose the following method:

PUT /orders/{order_id}:
```
{
    "type": "certificate",
    "meta": {
        "requestor_name": "(optional) string"
        "requestor_email": "(optional) email address - RFC 5322, 5321"
        "requestor_phone" : "(optional) string"
     }
}
```
This unambiguously specifies the fields that we expect to be able to change in a certificate request for all request types above.

Alternatives

One possibility is to implement an API that implements RFC 7030, which is also an HTTP interface around CMC requests. RFC 7030 expects clients to send CMC requests to specific URLs, and process CMC responses.

The main problem with adopting a RFC 7030 interface is that it is very different from what we have already implemented in Barbican, and we would have to jettison/rewrite much of the code we already have. That is not doable in kilo.

Some of the workflows are quite different.

For example, if the cert is pending, there is logic within Barbican to poll for the certificate until it is approved and issued. Then the certificate and its intermediates are stored within Barbican for later retrieval.

For RFC7030, the client needs to handle the polling by decoding the pkcs7-mime response – which is 202 – and re-submitting the same request until it is approved.

Moreover, there is no support for storing the certificate on the server, and returning things like order_id, secret_id etc. Nor is there support for amending/ canceling the order.

Instead of supporting CMC, we could try to define a common API with a basic set of parameters for different profiles of certificates. This is the approach started in [1]. If we adopt this blueprint, we will abandon [1].

This is simpler, but seems a little arbitrary. Using CMC is likely to be supported by most CA’s because it is standards-based. Also, there are likely to be existing clients that already do CMC.

The simpler mechanism is also limited, having no support for encryption or POP. Ultimately, I think we would need to evolve to support something like CMC.

Data model impact

We may need to add addtional data in the order_metadata, but this should not result in database changes.

REST API impact

See above in proposed changes.

Security impact

None

Notifications & Audit Impact

None, other than additional audit changes needed.

Other end user impact

There will need to be changes in either barbican-client or certmonger to support the new API.

Performance Impact

There will be some additional processing that will need to be done to parse the CMCRequest. These changes are not covered in the spec.

Other deployer impact

None.

Developer impact

Plugin developers will need to modify the supports() method to specify which issuance methods they will support. This information will be used to appropriately route certificate requests, and to notify the client of the CA’s capabilities when interacting with the interface defined in [3].

Plugin developers will need to implement two new methods: issue_simple_cmc_request() and issue_full_cmc_request() if they want to support these requests. We expect simple CMC to be easy to implement because simple CMS == PKCS10.

Implementation

Assignee(s)

Primary assignee:: alee chellygel dave-mccowan woodster

Work Items

Modify API code to parse the new Order parameters and pass the data to lower levels.
Add code to validate CMC request data.
Add code to certificate_manger.py to send the requests to the plugins. Modify the plugin interface contract to provide default implementations.
Plugins should modify the supports() method and implement the new methods as appropriate.
Add code to certificate_manager.py to parse a CMC Response and convert into a ResultDTO object to return to barbican-core.
Add client code. This probably requires a separate blueprint.

Dependencies

This work should do done in conjunction with the work on the interface to get CA identifiers in [3].

Testing

More unit and functional tests will be needed.

Documentation Impact

Docs (including sphinx and plugin docs) will need to be changed to reflect this API.

References

[1] https://review.openstack.org/#/c/129695 Existing Blueprint to define a set of common cert API parameters.

[2] http://tools.ietf.org/html/rfc5272 Certificate Management over CMS

[3] API for getting CA identifiers is defined in this blueprint. In addition, there is an API for getting the CA signing certificate and intermediates. https://review.openstack.org/129048

[4] https://tools.ietf.org/html/rfc7030 Enrollment over Secure Transport

[5] http://tools.ietf.org/html/rfc1485 A string representation of distinguished names

[6] http://tools.ietf.org/html/rfc5280 Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile

[7] https://review.openstack.org/129377 API for getting CA specific certificate enrollment parameters.

[8] LibEST implementation https://github.com/cisco/libest

Expose CA enrollment templates

Fri, 17 Oct 2014 00:00:00

https://blueprints.launchpad.net/barbican/+spec/expose-ca-enrollment-templates

Problem Description

Barbican will allow interaction with multiple Certificate Authorities through multiple CA plugins. Each plugin expects the certificate requests to conform to a specific format and to have specific input parameters. A common CA interface will make it easier for clients to construct certificate requests. Nonetheless, a mechanism is needed to expose the parameters required to make a certificate request.

Proposed Change

A new resource has been proposed for certificate authorities in [1]. This interface can be used to expose the parameters needed to create a certificate request. The details on the interface changes are shown in the REST API section below.

The data would be obtained from the plugins by implementing two new API calls: get_enrollment_templates() and get_enrollment_template(template). These calls can have default implementations that return None so that not all plugins need implement these calls immediately.

For the common API, the methods get_common_enrollment_templates() and get_common_enrollment_template() will be implemented to return the correct data.

Alternatives

Leave the code-base as it is.

Data model impact

None.

REST API impact

The following REST API operations will be added:

GET /cas/templates/enrollment – returns the request template for the common CA API. This will end up returning the data from get_common_enrollment_templates().
GET /cas/{ca_id}/templates/enrollment – returns the list of request templates for a specific CA (as defined by ca_id). This will end up returning the data from calling get_enrollment_templates(plugin_ca_id) on the specific plugin.

The ca_id values are defined in the spec for identifying ca_ids [1], which has already been implemented in the server.

This approach also leaves open the possibility to exposing other types of templates in future as needed (like renewal and revocation templates).

The data returned can consist of parameters and/or links for additional data. All relevant links will be subordinate to the request template URL.

For example, for the Dogtag CA, the call:

GET /cas/{ca_id}/templates/enrollment could return:

{“templates”:[

{“id”: “caUserCert”,
“name”: “Manual User Dual-Use Certificate Enrollment”, “description”: “This certificate profile is for enrolling user certificates.”, “link”: “https://host:port/cas/ca1/templates/enrollment/caUserCert”},

{“id”: “caServerCert”,
“name”: “Server Certificate Enrollment”, “description”: “This certificate profile is for enrolling server certificates.”, “link”: “https://host:port/cas/ca1/templates/enrollment/caServerCert”}

]}
GET /cas/{ca_id}/templates/enrollment/caServerCert could return:

{“parameters”:[

{“id”: “profile_id”,
“description”: “Profile ID”, “syntax”: “string”},

{“id”: “cert_request_type”,
“description”: “Certificate Request Type”, “syntax”: “cert_request_type”},

{“id”: “cert_request”,
“description”: “Certificate Request”, “syntax”: “cert_request”},

{“id”: “requestor_name”,
“description”: “Requestor Name”, “syntax”: “string”},

{“id”: “requestor_email”,
“description”: “Requestor Email”, “syntax”: “string”},

{“id”: “requestor_phone”,
“description”: “Requestor Phone”, “syntax”: “string”}

]}

The idea would be that a client that filled in all the parameters listed above would create a certificate request with all the required parameters.

An example request using the above parameters would look like:

{ 'type': 'certificate',
  'meta': { 'profile_id': 'caServerCert',
            'ca_id': '{ca_id}'
            'cert_request_type': 'pkcs10',
            'cert_request': '<put PEM formatted CSR here>',
            'requestor_name': 'John Wood',
            'requestor_email': 'jwood@rackspace.com',
            'requestor_phone': '555-1212'
          }
}

Note that specification of ‘profile_id’ requires the specification of the ‘ca_id’. This check is already enforced in the server.

Security impact

None. The data that are exposed through this interface are parameters which the CA chooses to expose to show users how to interact with it. These are the things that would be exposed by the CA on a web UI , say, or as part of the CA’s client API. This data is necessarily public - otherwise no one would know how to interact with the CA.

All this info is queried by the CA plugin to the CA. Barbican only reveals what the CA chooses to expose.

Notifications & Audit Impact

None.

Other end user impact

The python-barbicanclient will need to be enhanced to take advantage of this new feature. In particular, it may be possible to recurse through the parameters either interactively or otherwise to generate valid cert requests.

Performance Impact

None.

Other deployer impact

None.

Developer impact

Plugin developers may choose to implement the new methods to take advantage of the new functionality.

Implementation

Assignee(s)

Primary assignee:: alee-3

Work Items

Add code to implement the new API calls and call get_enrollment_templates() etc. Add these calls and their default implementation to the plugin interface.
Add code for get_common_enrollment_templates() and get_common_enrollment_template(x).
Implement get_enrollment_templates() and get_enrollment)template() for each plugin, as needed.
Document all, including sphinx documentation.

Dependencies

None

Testing

The current unit and functional tests will also be modified to reflect this change.

Documentation Impact

New feature that will need to be documented.

References

[1] Spec for identifying CAs: https://review.openstack.org/#/c/129048: This spec was implemented in the server in Kilo.

Identify available CAs

Thu, 16 Oct 2014 00:00:00

https://blueprints.launchpad.net/barbican/+spec/identify-cas

Problem Description

It is possible to have multiple CA plugins, each potentially talking to multiple backend CA servers. A mechanism is therefore needed to allow the client to select a backend CA server.

In addition, Dogtag plans to implement the ability to configure lightweight sub CA’s - subordinate CA’s that can exist within the same CA instance. This opens up the possibility of configuring a separate CA instance for each project, so that the project could have certificates that are scoped to the project only. Thus, a mechanism is also required to associate a project with a preferred CA, so that if a client does not request a specific CA, the preferred CA is selected.

Also, a mechanism should be added to allow clients to discover the CA servers available for a particular Barbican instance.

Proposed Change

We need to add a new resource CertificateAuthorities that will specify the CAs that are available to Barbican. This resource will permit listing CAs and associating projects with a CA. See the REST API section for details.

Four new tables will be required:

CertificateAuthority, which will list the CA’s available. This table will include the Barbican generated ca_id, the plugin name, plugin_ca_id, information about the CA and a cache expiration time. The data will be assumed to be stale and to require a refresh when the cache expiration time expires.
CertificateAuthorityMetadatum, which contains data about the CAs, to be provided to the client.
ProjectCertificateAuthority, which will list the set of CA’s defined per project. If a project chooses to do so, it can specify a set of CA’s (referenced by ca_ids) that a particular project can use. The first entry for a project will automatically be added to the PrefCA table for that project to ensure that there will always be a preferred CA for a project.
PreferredCertificateAuthority (PrefCA), which contains the preferred CA entry for each project. This table requires the project_id to be unique so that there is only one preferred CA per project. A special entry for the preferred global CA will have “project_id” = 0.

See the Data model section for details.

The ProjectCertificateAuthority table will be updated through REST API calls by a project administrator.

The CertificateAuthority table will be updated as follows:

On startup, Barbican should iterate through all the CA plugins and execute certificate_resources.update_ca_list(plugin_name). This will trigger a call to a provides() method for each plugin.
The provides() call will query the backend-CA to determine which CAs are provided. It will return the relevant data ie. a list/dict of (plugin_ca_id, description, ca_ski etc.) as needed to populate/update the CertificateAuthority and CertificateAuthorityMetadatum table. It will also return an expiration time for the data.
The CA and CA metadata table would then be updated by a method defined in Barbican core. (eg. certificate_resources.update_ca_list(plugin_name)). Barbican core will use the plugin_ca_id to map the CA entry to the correct ca_id. At this point: new CA’s will be added; decommisioned CA’s will be removed; and existing CA’s will be updated. In each case, the expiration time will be updated.
A method is provided below for the user to obtain a list of available CA’s. If the expiration time for a CA has elapsed, the plugin for that particular CA will be queried by calling the provides() method. The CA table and CA metadata tables will be updated accordingly.
If a certificate request is made and a ca_id is specified, the relevant plugin will be looked up in the CA table. If the entry for this ca_id is stale, then the provides() method will be invoked for this CA to update the CA table.

Logic when a certificate is requested:

When a certificate is requested,

If a ca_id is provided:

If the ca_id is not defined in the CA table, a 400 error is returned.

If any entries for the project exists in the PCA table, and the ca_id is not one of those entries, a 403 error is returned.

Otherwise the request is passed through to the plugin specified by the entry in the CA table.

If a ca_id is not provided:

If at least one entry for the project exists in the PrefCA table, then barbican core will select the preferred CA, and pass the request through to that CA.

If no entry exists in the PrefCA table for the project, then the global preferred CA specified in the prefCA table will be selected. If no preferred CAs are defined, then the first CA plugin will be selected.

The plugin_ca_id will be passed to the plugin as part of the order_metadata.

Alternatives

None.

Data model impact

Four new tables will be added: CertificateAuthority (CA table), CertificateAuthorityMetadatum (CAM table), ProjectCertificateAuthority (PCA table) and PreferredCertificateAuthority (PrefCA table).

The CA table will have the following fields:

ca_id - unique identifier for the CA generated by Barbican
plugin_name
plugin_ca_id - (string) identifier for the CA generated by the plugin This identifier must be unique for the plugin.
expiration_time
ca_metadata (see below)

To store metadata about each CA, data will be stored in a CertificateAuthorityMetadatum (CAM) table. This table will consist of the following fields:

ca_id - foreign key to CA table

key

value

The following key-value data will be stored to begin with:

name - human readable name
description
ca signing certificate
intermediates

A note about ids:

The ca_id that will be presented to the client will be the Barbican generated ca_id. However, because the CAs that are available are determined by the back-end CA, the plugin needs to provide a plugin_ca_id that is unique to the plugin, so that the plugin_ca_id can be mapped to the Barbican generated ca_id.

The PCA table lists the CA’s defined for each project. There can be multiple entries for each project. It will have the following fields:

project_id

ca_id (foreign key to the CA table)

The PrefCA table lists the preferred CA either globally or for any particular project. This is to enforce the requirement that only one CA can be defined to be preferred either globally or pre-project. This table will have the following fields:

project_id – required to be unique

ca_id – foreign key to CA table

REST API impact

We need a new resource to expose the CA’s available. This resource will allow the following operations:

GET /cas - List all the available CAs.
GET /cas/{ca_id} - Get details about CA referenced by ca_id. This would include things like a HATEOAS link to the CA information, name and description.
GET /cas/{ca_id}/cacert - Get the base 64 encoded signing certificate for the CA in PKCS7 format
GET /cas/{ca_id}/intermediates - Get the CA cert and intermediates certs in base 64 encoded PKCS7 format.
GET /cas/{ca_id}/projects - List projects which use the CA with the specified ca_id. Restricted to global admins. This is an operation that will allow global admins to clean up the PCA table, in case a CA is decommisioned.
POST /cas/{ca_id}/add-to-project - Add entry in PCA table for ca_id. The project_id is determined from authz. Restricted to project admins. If this is the first CA to be added to a project, it will be set as preferred by adding an entry for that project in the PrefCA table.
POST /cas/{ca_id}/remove-from-project - Remove entry in PCA table for ca_id. The project_id is determined from authz. Restricted to project admins. If the CA is the preferred CA for the project and no other CAs exist in the PCA table, then the PrefCA entry will be removed too. If other entries exist in the PCA table, then a 400 error is returned. (“Cannot remove a preferred CA. Select another project CA to be preferred first.”)
POST /cas/{ca_id}/set-preferred - Set ca_id to be a preferred CA for the project. A 400 error should be returned if a PCA entry does not already exist for this project_id/ca_id. Restricted to project admins.
POST /cas/{ca_id}/set-global-preferred - Set ca_id as a global preferred CA. This CA is invoked if no ca_id is specified in a request, and no PCA entries exist for the project. Restricted to global admins.
POST /cas/{ca_id}/unset-global-preferred - Unset ca_id as a global preferred CA. Restricted to global admins.

Also, an optional metadata parameter (ca_id) will be provided in the Order for a Certificate, and will be processed as described above.

Security impact

None.

Notifications & Audit Impact

None.

Other end user impact

python-barbicanclient will need new methods to check the list of CAs and to invoke a specific ca_id if requested.

Performance Impact

This feature requires database access to get the CA/project/plugin information on each certificate order request, which will have an impact on both the API and worker nodes.

Other deployer impact

Migration scripts will need to be run on already existing deployments to add the new tables.

Developer impact

Plugin developers should write a provides() method that returns the correct information. We can provide a default implementation that creates a CA table entry based on the plugin_name.

Implementation

Assignee(s)

Primary assignee:: alee-3 dave-mccowan

Work Items

Add new data models (PCA, PrefCA, CA, CAM tables)
Add default provides() call to the back-end plugin contract. Add the logic that populates the CA table on startup.
Add /cas and the relevant REST API calls to retrieve/set these resources.
Add CA selection logic to the certificate request flow.
Add each plugin specific provides() method.

Dependencies

None

Testing

The current unit and functional tests will also be modified to reflect these changes.

Documentation Impact

This is a new feature that will need to be documented.

References

None

Storing metadata to allow the use of per-secret policy

Thu, 09 Oct 2014 00:00:00

https://blueprints.launchpad.net/barbican/+spec/add-per-secret-policy

This is a proposal to add per-secret specific policy in Barbican to augment the generic operation based policy in policy.json. The idea is to store attributes that help determine whether or not a secret or container could be accessed (like an include_list of users) per secret.

Oslo policy allows you to access those attributes and pass them to the RBAC rules engine as a dictionary of target.* attributes, and used in policy enforcement.

Problem Description

Most recently, Neutron LBaaS tried to figure out how to grant the LBaaS system user the ability to retrieve secrets (cert containers and secrets) in order to create load balancer configuration. The solution that was decided upon was trusts. That is, a trust was created on the keystone server for the LBaaS user for the relevant project ID.

While this mechanism will work, a trust is pretty heavy-handed. Essentially, it gives permission to the LBaaS user to impersonate the user for the relevant project, as long as the trust is valid (which could easily be forever). If the LBaaS user were ever hijacked, they would not only be able to access all the Barbican data, but they would be able to interact with all the other services as that user.

Indeed, as more projects integrate with Barbican, the requirement for service users to obtain barbican secrets will come up again and again. Proliferating trusts throughout the services simply to do something as basic as retrieving secrets seems like a recipe for disaster.

In addition, recently there have been requests to be able to specify that some secrets were private. That is, only the secret creator (the user that created the secrets) should be able to extract them. Now, one might argue that one could create a project for each user - but that seems like a very burdensome solution.

Both of these problems (and perhaps others) can be solved by using per-secret access attributes to further refine the policy associated with a secret. In policy parlance, this is using attributes of the target (resource being accessed, in this case, a secret or a container) to further refine policy.

This is something that is done, for instance, in Keystone. See the rules in https://github.com/openstack/keystone/blob/master/etc/policy.v3cloudsample.json which refer to target. These values are populated in keystone/common/controller.py (in the definition of protected()).

To make things clearer, I will separate the solution for this second problem (“private secrets”) into a separate follow-on spec.

Proposed Change

The spec deals only with the problem posed by LBaaS. That problem is: how do we allow access to a secret by a user that is not part of the secret’s project.

The proposal is to store access control data with each secret or container that could be used to augment the existing policy for retrieval on a per-resource basis.

This feature can be broken down in terms of four subproblems:

What operations will the proposed ACLs cover, and what will be the default policy for each operation? See the Operations section below.
What does the client send into the server when setting the ACLs for a secret or container? See REST API section below.
How is this ACL information stored within the database? See the Database section below.
How is this data extracted and presented to the Oslo layer to do RBAC enforcement? See the Policy section below.

Operations

The ACL permissions here correspond to actions that some user outside of the creator’s project can perform on a secret or container. These actions include:

get: For secrets, someone with ‘get’ permission will be able to retrieve: secret metadata and payload. For containers, they will be able to retrieve the container (which is essentially all metadata). Accessing secrets referenced within the container will require ‘get’ permission on each secret. For ‘Kilo’, get permission on the container will not automatically cacade down to the secret. This is the only action to be implemented for K.
delete: Ability to delete a secret or container. Currently, this permission: is restricted to a particular role (“creator”?) in the secret’s project. For Kilo, this will not change. In particular, its not clear that we would ever want someone outside the project to be able to delete a secret or container.
write: Ability to modify a project/container. Currently typed containers and: secrets are immutable. Even if they were not, its not clear someone from a different project should be able to modify a secret/container. This will not be implemented for K.
change-acl: Ability to change the ACL for a secret. For K, this will be: restricted to the secret/container’s creator (ie. the user that created the secret/container).

REST API impact

A new parameter (“acl”) will be added to POST for secrets and containers. The data in this field has the following format:
```
{
    "get": {
            "users": ["some_user_id",
                      "another_user_id"],
           }
}
```
The data in those fields will be placed directly into the ContainerACL and SecretACL tables below.

There has been some discussion of adding parameters for whitelisted groups and projects to the acl. This requires additional discussion because groups are not specified in the keystone token, and roles should also be defined within projects. As there is no current need for these parameters, we will defer adding them till L+.
In addition, a new endpoint will be provided to allow a user to change the ACL on a secret or container:
```
POST <host>/v1/secrets/<secret-UUID>/acl
```
where the body would be an ACL in the same format as above. This endpoint would be limited to uid == <uid of creator> by default.
ACL data will not be returned by default with a GET request. Rather, a new route will be provided to retrieve the ACL:

GET /secrets/<secret_uuid>/acl

which would return the ACL as shown above. This endpoint would be limited to uid: <creator>.
When orders are created, the Barbican server will need to keep track of the order’s creator to ensure that the order’s creator has permissions to access the secrets.

For example, when generating a certificate using the stored key mechanism, the asynchronous worker will need to access a container and the private key referenced within that container. This should only be allowed if the order’s creator has access to the relevant resources.

Data model impact

For secrets and containers, we will need add a column to store the creator of the secret. This would be populated by the creator’s user_id. It may also be useful to store the creator’s project_id, given that we plan to remove the secret/project table.

We will also need to add column to store the creator of an Order, so that any secrets/containers that are generated when an Order is processed will be stamped with their creator.

We also need to create two new tables: ContainerACL and SecretACL, in which to store ACL data. The fields for the SecretACL table will be as follows:

secret_id: foreign key to Secrets table action: currently, only “read” will be implemented users: string, list of whitelisted users for specified action

In L+, we may decide to add columns to allow for whitelisted groups or projects. For now, only users are whitelisted.

Policy

When a container or secret is accessed, the ACL data is retrieved from the database and provided to the RBAC layer as target.* attributes. For getting a secret, for example, we could pass in target.user_whitelist.

The policy rule would then look something like:

(can_read_shared and user_in_user_whitelist) OR current_resource_permissions

The current_resource_permissions part is basically that the user has the relevant role in the secret creator’s project.

We may need to extend (and upstream) oslo-policy to allow lists to be processed.

Alternatives

We could use trusts, but they are, as we described before, very heavy handed. All in all, this is something that is simply required in Barbican.

Security impact

This improves security in the stack as a whole by eliminating the requirement for a trust, and rather, providing granular per-secret permissions.

Notifications & Audit Impact

None.

Other end user impact

python-barbicanclient will need to be updated to provide an interface to populate the extra parameters.

Performance Impact

Accessing a secret/container will require two database calls: one to get secret/container include)list as part of the RBAC engine’s rules enforcement, and one to actually get the secret/container.

Other deployer impact

Neutron and other service users that will be accessing secrets in Barbican will be able to do so using the new mechanism, rather than a trust.

Developer impact

Changes will likely need to be made in the neutron code to take advantage of this mechanism. The current mechanism of using trusts should still work.

Implementation

Assignee(s)

Primary assignee:: alee rm_work

Work Items

Add new fields to the database tables, and new parameters to the REST API.
Add logic to stamp secrets/containers created through the REST API or through Orders with their creator.
Add logic to retrieve ACL data from the database and provide to RBAC layer as target.* attributes.
Modify policy rules based on these target.* attributes. It may be necessary to extend oslo policy here to process lists correctly.
Modify neutron code/barbican client to take advantage of this new mechanism.

Dependencies

None

Testing

The current unit tests will also be modified to have this change reflected upon them.

Documentation Impact

Neutron and Barbican docs will need to be changed.

References

Diagram showing Neutron LBaaS detailed flow. http://goo.gl/Wc8oIj
Earlier blueprint with similar ideas. https://blueprints.launchpad.net/barbican/+spec/secret-isolation-at-user-level

Change GET decrypted secrets to unique URI

Thu, 02 Oct 2014 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/barbican/+spec/api-change-get-secrets-decrypted

Currently retrieving the metadata and actual decrypted data for a secret stored in Barbican uses the same URI, with only the Accept header used to determine which response to return. This complicates deployments with RBAC systems in front of Barbican (such as Repose - http://openrepose.org/) or as middleware (such as EOM - https://github.com/racker/eom). It also requires adding logic like this to the Barbican app when it is used to enforce RBAC: https://github.com/openstack/barbican/blob/master/barbican/api/controllers/__init__.py#L53

This blueprint proposes using a unique URI to access decrypted secrets, such as this: <host>/v1/secrets/<secret-UUID>/payload

Problem Description

Currently to retrieve a secret’s metadata and its decrypted data, the same URI is used, such as this:

GET <host>/v1/secrets/<secret-UUID>

It would be easier to configure RBAC systems that operate in front of a Barbican deployment if these two retrieval types were distinguished by URI.

Proposed Change

This blueprint calls for adding a new URI structure to retrieve decrypted secrets as follows:

GET <host>/v1/secrets/<secret-UUID>/payload

So this would include adding a new Pecan sub-controller to handle the ‘payload’ requests, adding unit and functional tests for the new ‘payload’ resource, and updating API documentation.

The existing Accept-based decryption approach would not be removed in the current ‘v1’ API version to avoid breaking the current API contract, but could be removed in the next version (‘v2’) of the API (not part of this blueprint).

Alternatives

Another option is to use a URI such as this to retrieve decrypted secrets:

GET <host>/v1/secrets/<secret-UUID>.payload

This approach would require parsing within the Pecan controller logic however.

Data model impact

None

REST API impact

This blueprint would add a new URI to retrieve decrypted secrets:

GET <host>/v1/secrets/<secret-UUID>/payload

The Accept header would still be used as now to specify the desired format to return decrypted secrets in, but the ‘application/json’ format would no longer be needed/used.

The current URI to retrieve secrets would still function as it does now, but the decryption options could be removed in the next API version (‘v2’) by a future blueprint.

Security impact

None, as the same policy rule (‘secret:decrypt’) for the new decrypt action will be used as for the current header-based decryption mechanism.

Notifications & Audit Impact

None, though auditing might actually be easier due to the unique-URI per action approach taken by this blueprint.

Other end user impact

The Python Barbican client will also need to be updated to work with this change.

Performance Impact

None

Other deployer impact

None

Developer impact

The proposed change would clean up the Pecan controllers code a bit, segregating secret metadata and decrypted payload concerns from each other.

Implementation

Assignee(s)

Primary assignee:: jaosorior
Other contributors:: john-wood-w

Work Items

Add new ‘payload’ sub-resource controller off the ‘secrets’ resource, using the ‘secret:decrypt’ policy action
Add associated unit and functional tests per below
Update API documentation per below (currently located here: https://github.com/cloudkeep/barbican/wiki/Application-Programming-Interface)
Update the Python Barbican Client to utilize this new resource. If it is not available (due to older Barbican deployments), it should fail over to using the current decrypt mode

Dependencies

None

Testing

Add unit and functional tests to test decrypted secret retrievals using the new ‘payload’ resource off of the ‘secrets’ resource. These new tests should function properly in the Devstack gate job.

Documentation Impact

API documentation will need to be updated to reflect these new secret decryption API call, and to mark as ‘deprecated’ the current Accept header based approach.

References

None

Replace the concept of tenants in the code-base in favor of projects

Tue, 30 Sep 2014 00:00:00

https://blueprints.launchpad.net/barbican/+spec/replace-concept-of-tenants-for-projects

To enforce a uniform and consistent code-base, it is also necessary to make use of concepts in a consistent way. As a notorious example of this, the incidental usage of the concepts ‘tenant’ and ‘project’ throughout the code-base. The proposition provided in this blueprint is to finally replace one concept for the other, and thus end this lack of uniformity.

Problem Description

Even though, it has been talked of following the steps taken by the Keystone team of starting to enforce the usage of the concept of ‘projects’ instead of tenants; there are still 580 occurrences of the word ‘tenant’ in the Barbican code-base alone (At the time that this blueprint was written), as opposed to the 103 occurrences of the word ‘project’. This shows lack of uniformity, which degrades the consistency throughout the code-base.

Proposed Change

Thus, to address the aforementioned issue, I propose to replace every instance of ‘tenants’ from the code-base, and enforce the usage of ‘projects’. This is to, not only have a consistent code-base, but also have consistency towards other projects such as Keystone.

Alternatives

Leave the code-base as it is.

Data model impact

There will be impacts in the data model, since the concept of ‘tenants’ is actually being used there. Thus, the Tenant and the TenantSecret classes will be replaced with a Project and ProjectSecret classes respectively. On the other hand, tables containing columns with the word ‘tenant’ will also need to be replaced, and with those, migration scripts will need to be added (the TenantSecret, Tenant, EncryptedDatum, KEKDatum, Order and Container classes/tables will be affected).

On the other hand, keystone_id will also be replaced from the Tenant (which will be called Project) in favor of calling it ‘project_id’, since this was initially named due to the confusion as to whether it should be ‘tenant_id’ or ‘project_id’. The proper business logic around this which involves changing how the context is filled, and how it is passed to the controllers will also reflect this change. So, in summary, ‘keystone_id’ will also be replaced by ‘project_id’

REST API impact

None

Security impact

None

Notifications & Audit Impact

If the word ‘tenant’ is being used while emitting notifications, this will be changed to ‘project’. Other than that, there will not be any extra impact.

Other end user impact

There are certain arguments taken by the python-barbicanclient that will change due to this.

Performance Impact

None

Other deployer impact

Migration scripts will need to be run on already existing deployments. There will be no other impact otherwise.

Developer impact

Developers hopefully will stop using the term ‘tenant’ in barbican.

Implementation

Assignee(s)

Primary assignee:: juan-osorio-robles

Work Items

Replace all trivial instances in the code-base
Replace instances in the data model (write the proper migration scripts)
Replace all instances in the python-barbicanclient

Dependencies

None

Testing

The current unit tests will also be modified to have this change reflected upon them.

Documentation Impact

If there exist instances of the term in the wiki, these should be changed also.

References

None

Creation of a Babrican Plugin to use HP Atalla ESKM.

Fri, 15 Aug 2014 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/barbican/+spec/hp-eskm-plugin

This effort will enhance Barbican by adding a crypto plugin to allow use of Atalla ESKM from HP. Atalla ESKM is a HSM appliance for generating and managing cryptographic keys. By adding an optional plugin to Barbican, users have the option of integrating OpenStack with existing or new Atalla ESKM installations.

Problem Description

Deployers of an OpenStack cloud may have, or wish to have an Atalla ESKM Appliance for key management. In this case, it is desirable to utilise the appliance alongside the new OpenStack installation for enhanced security.

Proposed Change

The proposed changes to Barbican are, creation of an optional plugin to talk to an ESKM appliance. This plugin will be based of off the CryptoPlugin base and encapsulate all necessary functionality. The plugin will communicate over a secure network link to a remote appliance using a vendor specific protocol.

No changes to the core of Barbican are required.

Alternatives

Alternatively, Atalla ESKM supports the KMIP industry standard protocol for key exchange. KMIP is something planned for Barbican in the future but at the time of writing has not yet been implemented.

Data model impact

None

REST API impact

None

Security impact

This change touches sensitive data (tokens, keys, and user data) since it will pass key data through to an ESKM server and receive the results prior to encryption. User data is protected using a key encryption key and 256 bit AES CBC encryption. The key encryption key is never stored and is fetched on demand from the ESKM appliance.

This change utilises encryption algorithms from the ‘cryptography’ python module but does not introduce any additional cryptographic dependencies into Barbican.

This change requires a network connection to be established between Barbican and an ESKM appliance. This connection is protected using TLSv1 and client identification certificates. Key material is transmitted over this connection.

Notifications & Audit Impact

None

Other end user impact

None

Performance Impact

A performance impact is introduced by this change, communication over a network link to a remote ESKM appliance may take time. This is multiplied by the need to request a key encryption key from ESKM as part of all requests. This performance impact is accepted in return for the improved security gained by not caching the key encryption key locally.

Other deployer impact

The following new config options are added, they are specific to the plugin and need not be used otherwise. All options live within the ‘eskm_crypto_plugin’ group.

Option	Meaning
eskm_crypto_plugin.crt_file_path	ESKM client certificate file path.
eskm_crypto_plugin.key_file_path	ESKM client key file.
eskm_crypto_plugin.key_password	ESKM client key password.
eskm_crypto_plugin.user_name	ESKM user name.
eskm_crypto_plugin.user_pass	ESKM user password.
eskm_crypto_plugin.eskm_host	ESKM host IPs, comma separated.
eskm_crypto_plugin.eskm_port	ESKM port number.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: tim-kelsey
Other contributors:: None

Work Items

Create spec (this spec).
Write the plugin and tests.
Confirm all Barbican tests, new and old, still pass.
Review the code.

Dependencies

No compulsory dependencies are introduced, but an Atalla ESKM appliance must be available if a deployer wishes to use this plugin.

Testing

A suite of unit tests will be produced to test the new code.

Documentation Impact

Documentation may need updating to reflect the existence of the plugin and its configuration options.

References

None

Add containers to python-barbicanclient

Mon, 28 Jul 2014 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/python-barbicanclient/+spec/client-add-containers

The Secret Containers resource is now available in Barbican, so it should also be added to the client.

Problem Description

We need to be able to consume and create Container resources in the Python library using Python objects to represent the containers.

Proposed Change

Add new Classes to the client to represent the different types of Containers available in Barbican in a new containers module:

barbicanclient.containers.Container
barbicanclient.containers.CertificateContainer
barbicanclient.containers.RSAContainer

The classes should be usable thusly:

from barbicanclient.containers import Container, RSAContainer

# New Generic Container
my_container = Container()
my_container.add("Secret Name in Container", Secret(...))
my_container.save()

# New RSA Container
my_rsa_container = RSAContainer()
my_rsa_container.public_key = Secret(...)
my_rsa_container.private_key = Secret(...)
my_rsa_container.private_key_passphrase = Secret(...) #optional
my_rsa_container.save()

# RSAContainer should be subclassed from Container
my_rsa_container.add("public_key", Secret(...)) # should work
my_rsa_container.add("unsupported_secret_name", Secret(...)) # should raise exception

# New Certificate Container
my_cert_container = CertificateContainer()
my_cert_container.certificate = Secret(...)
my_cert_container.private_key = Secret(...)
my_cert_container.private_key_passphrase = Secret(...)
my_cert_container.intermediates = Secret(...)
my_cert_container.save()

# CertificateContainer should be subclassed from Container
my_cert_container.add('certificate', Secret(...)) # should work
my_cert_container.add('unsupported_secret_name', Secret(...)) # should raise exception

Objects should be lazy, so requests should not be sent until the save() method is called. The save() method should also recursively store secrets if they have not yet been stored to the Barbican instance.

Once the save() method is called for the first time the Container becomes immutable, subsequent calls to add() or save() should raise exceptions. Basically, any action that would modify the container, except for delete(), should raise an exception.

Retrieving existing containers should be done via a ContainerManager accessible from the client object:

from barbicanclient import client

barbican = client.Client(...)
my_container = barbican.containers.get('https://ref_to_container/UUID')

The get() method should return an instance of the appropriate Container class. Listing existing methods should be done via the ContainerManager as well:

my_containers = barbican.containers.list(limit=10, offset=0)
isinstance(my_containers, list) # should return True

Container deletion should be done on either the Container object or using the ContainerManger:

my_container = barbican.containers.get('https://ref_to_container/UUID')
my_container.delete()
# or
barbican.containers.delete('https://ref_to_container/UUID')

Alternatives

The proposed functionality above pushes most of the actions to the Container objects, while keeping the ContainerManager functionality somewhat light. As an alternative, the ContanerManager could perform all of the functionality, while making the Container objects simple.

I think container creation in a single method could get ugly really fast, so I would prefer to avoid it.

Data model impact

None

REST API impact

None

Security impact

None

Notifications & Audit Impact

Logging should be done in a manner consistent with the rest of the library.

Other end user impact

None as this is new functionality.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Blueprint Draft: dougmendizabal Implementation: TBD

Work Items

Implement the new containers module.

Dependencies

None

Testing

Testing should be consistent with existing testing in the library.

Documentation Impact

All new Container functionality needs to be documented.

References

Containers in the Client etherpad: https://etherpad.openstack.org/p/python-barbicanclient-containers

Restructure PKCS11 Plugin

Thu, 17 Jul 2014 00:00:00

https://blueprints.launchpad.net/barbican/+spec/restructure-pkcs11-plugin

The current PKCS11 plugin assumes that it has sufficient storage to create a KEK per project/tenant. This assumption is untrue for many HSMs so we propose to introduce a master KEK stored in the HSM that wraps per project KEKs to resolve the problem.

Problem Description

The PKCS11 plugin creates a key encrypting key (KEK) per project/tenant. When connecting to an HSM with limited storage, this system can exhaust available space on the HSM and fail to create KEKs for new project/tenants.

Proposed Change

A master KEK wrapping a project KEK will resolve the storage problem by keeping a minimum of keys in the actual HSM while still using a different encryption key per project/tenant. The hierarchy would look like this:

+--------------+
|  Master KEK  |
+-----+--------+
      |
      |
      |
+-----+-----+       +-----------+
|Wrapped KEK+-------+Encrypted  |
+-----------+       |Secret     |
                    +-----------

The sequence diagram of the decryption process will look like this:

::: barbican DB HSM + Get master KEK handle + + | <—————————————————–> | | Get wrapped project KEK | | | <—————————-> | | | Unwrap project KEK (stays|in HSM) | +——————————————————-> | | project KEK handle | | | <——————————————————-+ | Get encrypted secret | | | <—————————-> | | | Decrypt secret using project KEK handle | | <—————————————————–> |

Alternatives

None.

Data model impact

The data model may need to be expanded to support storing a per project KEK. We propose that this be accomplished by creating a new table that stores the wrapped KEK bytes with a foreign key to project/tenant.

REST API impact

None

Security impact

The risks of the PKCS11 plugin remain largely the same. While the KEKs are no longer stored exclusively in the HSM, the KEK is always encrypted when outside the HSM.

Notifications & Audit Impact

None

Other end user impact

None

Performance Impact

There is an additional decryption round trip to the HSM. Previously decrypting a secret required finding the project KEK and then sending the encrypted bytes to the HSM for decryption. With the new system we must find the master KEK, decrypt the wrapped project KEK, then decrypt the user secret using that unwrapped KEK.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Paul Kehrer (reaperhulk)

Work Items

Add new model to barbican
Update PKCS11 plugin to do wrap/unwrap and consume the new model.

Dependencies

None

Testing

The unit tests for the PKCS11 plugin will change to reflect the new calls made to the HSM and the data stored in the models.

Documentation Impact

Update the documentation to describe the manner of operation of the plugin.

References

https://blueprints.launchpad.net/barbican/+spec/restructure-pkcs11-plugin

Transport Key Wrapping

Thu, 10 Jul 2014 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/barbican/+spec/add-wrapping-key-to-barbican-server

Secrets are currently passed from Barbican clients to the Barbican server encrypted only by TLS. For Common Criteria environments, this is insufficient. Secrets need to be encrytped within the TLS stream at the point of origin, and ideally only decrypted when the secret is stored. In the case where a hardware token is used, this would happen in the token, so that even if an attacker were able to gain access to the Barbican server, and was able to introspect the process memory, no secrets could be deciphered.

Problem Description

See above.

Proposed Change

Mechanism for Storing Keys using Key Wrapping:

Client elects to use key wrapping to store a secret, and therefore will use a two step storage mechanism for its secret.
Client does a POST to /secrets without a payload. Client also specifies transport_key_needed=true in the request.
Server notices that transport_key_needed=true and looks for a plugin that supports storage with key wrapping.

The logic is something like this

for (plugin; list_of_plugins):
    call plugin.get_transport_key()
    if transport_key_found:
        put transport_key URL in transport_key_ref in the response.
        return success
return failure (No plugin available 400)

Note that plugin.get_transport_key() will be defined within the abstract base secret_store class to return None. This means that plugins that wish to support key wrapping will need to override this method to return a transport_key_id (a reference to the key in the transport key repo).
It is expected that the plugin will be responsible for populating the TransportKeyRepo with the correct key. This could occur, for example, on module startup, or when the first call to get_transport_key() is made.
Client gets the transport key from the transport key resource. To improve performance, the client will probably cache this key.
Client uses transport key to do key wrapping as expected. Specifically - wrap the secret with a session key, then wrap session key with transport key, then place in ASN1 structure.
Client returns encrypted blob and the transport key reference in the PUT /secret/{id} request. We can use the parameter transport_key_ref.
Server notices that transport_key_ref has been set. It looks up the plugin associated with that transport key from the TransportKeyRepo. If the plugin is not available or the transport key does not exist, throw failure (400)
Pass the parameter transport_key_id in the SecretDTO, and call store_secret().
Plugin detects that transport_key_id is set, and processes accordingly.

Mechanism for Retrieving Keys using Key Wrapping:

Client elects to retrieve a secret using key wrapping. It first retrieves the metadata using a GET.
If the plugin that stored the secret supports key_wrapping and a transport key exists, return the URL for the transport key in transport_key_ref.
The client fetches the transport key and caches it.
Client generates a symmetric key (session key) and wraps it with the transport key. Client sends the trans_wrapped_session_key to the server as part of the GET request for the secret.
The server writes this key (trans_wrapped_session_key) into the secret_metadata and calls get_secret()
Plugin detects that this parameter has been written and retrieves the secret accordingly. The storing server will extract the session key and will encrypt the response with the session key.
On returning the secret, the plugin will remove the trans_wrapped_session_key from the secret_metadata.
Client will decrypt the response from the server using the session key.

Alternatives

The above solution is the culmination of a series of design discussions at the Atlanta summit. An old design can be found here: http://pki.fedoraproject.org/wiki/Barbican_server_add_wrapping_key

Data model impact

This change requires a new resource to be made available through the REST interface and stored in the database - transport keys.

The code to implement this new resource/model has already been merged. https://review.openstack.org/94875 - Add transport key as a resource.

These are new objects. They will be populated by any plugins that support key wrapping whenever the get_transport_key() method is invoked.

REST API impact

The following API methods will be changed:

POST /secrets
- A new optional parameter will be added “transport_key_needed” which will have the possible values of “true|false”. The parameter defaults to “false”.
- This parameter will have an effect when the two step storage mechanism is used. In this case, when there is no payload, if transport_key_needed is set to true, then a suitable plugin will be found and a transport key URL will be returned in the response in the field “transport_key_ref”.
- Return Error 400 if a suitable plugin/transport key cannot be found when transport_key_needed=true.
POST /secrets (with a payload)
- A new optional parameter will be added: “transport_key_ref”. This will contain the URL for the transport key used to encode the secret. Use of this parameter will imply that the secret has been wrapped.
- Return error 400 if the suitable plugin/transport key cannot be found.
PUT /secrets/foo
- A new optional parameter will be added: “transport_key_ref”. This will contain the URL for the transport key used to encode the secret. Use of this parameter will imply that the secret has been wrapped.
- Return error 400 if the suitable plugin/transport key cannot be found.
GET /secrets/foo (metadata only)
- If the secret is stored by a plugin that supports key_wrapping, then the a new parameter “transport_key_ref” will be added to the metadata response. This will contain a reference to the transport key for that plugin.
GET /secrets/foo (the actual secret)
- A new optional query parameter trans_wrapped_session_key will be added to this method. This parameter will contain a session key wrapped by a transport key to be used for retrieving the secret.
GET/DEL/POST /transport-keys
- This is mentioned here for completeness. The methods for these resources have already been merged into the code.
- GET will be used by clients to retrieve a transport key.
- POST/DEL can be used by plugin administrators to add or remove transport keys. These operations will not be available to ordinary users.

Security impact

This change should improve security in providing end-to-end encryption for secrets.

Notifications & Audit Impact

None, other than probably some changed notifications if this mechanism is used instead of the regular mechanism.

Other end user impact

None

Performance Impact

Doing this extra encryption will mean that storing and retrieving secrets will take two steps, rather than one, so that the correct transport key can be retrieved. This could be mitigated though by having the client cache the transport key - especially given that transport keys change very infrequently.

There will be some addtional work performed by the client, both in encrypting and decrypting the secrets, but this is likely not to be onerous.

Other deployer impact

Client work is required. We will cover this in a separate spec.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: alee-3
Other contributors:: redrobot (client side)

Work Items

CR performing the following for storage using transport key: * Add transport_key_needed boolean to POST request flows * Add get_transport_key() method to SecretStore * Return transport key if requested. * Modify secret PUT to add transport_key_ref. * Lookup secret_store plugin associated with that transport_key * Modify the secret_store SecretDTO to accept an optional transport key
CR performing the following for retrieval using transport key * Modify secret metadata GET to add a reference to the transport key * Add trans_wrapped_session_key to secret GET. * Write this parameter into the secret_metadata and provide to the plugin. * Modify the plugin to act accordingly.
CR to implement fetching the transport key and end to end testing in Dogtag.

Dependencies

https://review.openstack.org/94875 Add transport key as a resource (Done)
Barbican Client work

Testing

All new functionality and changes in the API/ secret store / Dogtag plugin will be unit tested. In addtion, we will perform end to end integration tests using Dogtag as the backend. Ideally, these end to end tests will be made part of the gate testing, with Dogtag being installed using a Chef recipe.

Also eventually, transport key functionality will be added to the dev plugin and will therefore be tested too.

Documentation Impact

This is a brand new feature and will need to be described in the docs.

References

https://review.openstack.org/94875 Add transport key as a resource CR.
An old (since abandoned) implementation: https://review.openstack.org/93165
Discussion at meetup: https://etherpad.openstack.org/p/barbican-juno-meetup
Discussion at Atlanta summit: https://etherpad.openstack.org/p/barbican-juno-roadmap

Introduce Oslo’s cliff as cli framework for Barbican

Thu, 10 Jul 2014 00:00:00

Introduce cliff as a cli framework for python-barbicanclient, and, as a result promote usage of common OpenStack libraries.

Problem Description

Currently the Barbican cli utilities (such as the migration script and even the python-barbicanclient itself) rely on the argparse library, yet, their current implementations do not provide sufficient information for their usage in production. Usage of “standard” libraries should be encouraged in order to make Barbican’s integration easier. On the other hand, subcommand help is lacking, and input is not properly validated.

Proposed Change

The proposal is to take into use cliff, which is a framework for building command line programs introduced by Oslo. This will enable Barbican’s scripts to have more standard outputs (akin to other OpenStack projects) and while moving the code to the new framework, better documentation will be added to the commands as part of the work.

Alternatives

The argparse library could still be used, yet, more input validation and documentation needs to be added.

Data model impact

None.

REST API impact

None.

Security impact

None for the Barbican server. Barbican client input though should be properly validated as part of the work.

Notifications impact

None.

Other end user impact

This will hopefully have a positive impact in the client’s usability, as the main focus of this work is to make it better.

Performance Impact

None.

Other deployer impact

This change takes immediate effect after it’s merged, and adds a dependency (which is cliff) to the python-barbicanclient.

Developer impact

If other developers are working with the client, it might affect their work, since the changes are potentially big.

Work Items

Replicate the current functionality of python-barbicanclient (with help messages and subcommands) to the new framework.
Complete missing sub-command help over the new framework.

Testing

Unit tests are assumed.

Documentation Impact

None.

Enforce content type on barbican REST API

Tue, 17 Jun 2014 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/barbican/+spec/barbican-enforce-content-type

Before barbican moved from falcon to pecan, content types were coerced to application/json. This meant that a user could use a tool such as curl, omit the content type, and the call would succeed. After moving to pecan, the default content-type changed to application/x-www-form-urlencoded. This meant that a curl request without a content type specified would end up coming into barbican as a urlencoded string and would fail during JSONifying.

Problem Description

As a result of the move from falcon to pecan, the default content type for requests has changed from application/json to application/xxx-www-form-urlencoded. That means that calls without a content-type which used to work with the falcon implementation will now fail with the pecan implementation.

If a caller sets their content-type header to application/json then everything works as expected.

However, if a caller omits the content-type header then it now defaults to application/xxx-www-form-urlencoded. This means that pecan will URLencode the request body, and that causes barbican to fail when it tries to JSONify it.

Proposed Change

The proposed change is to check the content-type header on the following types of Barbican requests, and reject any request (via pecan abort with HTTP 415) that does not specify the correct value:

Resource	HTTP Verb	Enforced Content Types
Secret	PUT	application/octet-stream text/plain
Secrets	POST	application/json
Orders	POST	application/json
Containers	POST	application/json
TransportKeys	POST	application/json

Alternatives

There are alternatives to the proposed change:

Change the content-type header to the correct value. This would resolve the problem described above, but could have unintended consequences as the user’s input data would be changed from what they provided/expected.

Detect the situation where the default was taken for content-type and compensate by:

URL-decoding the string

cleanup any padding characters that may have been added

Data model impact

None.

REST API impact

Users of the following resources will have to provide a valid value in their content-type header on REST API calls as indicated below:

Resource	HTTP Verb	Required Content Types
Secret	PUT	application/octet-stream or text/plain
Secrets	POST	application/json
Orders	POST	application/json
Containers	POST	application/json
TransportKeys	POST	application/json

Security impact

None.

Notifications & Audit Impact

None.

Other end user impact

None.

Performance Impact

Each REST call will now incur a check of the content-type. Calls without the correct content-type will fail with pecan.abort and HTTP 415.

Other deployer impact

None.

Developer impact

Developers who use a tool such as curl will need to ensure that they pass content-type:application/json in their HTTP headers, otherwise they will fail with HTTP 415.

Implementation

Assignee(s)

Primary assignee:: sheyman
Other contributors:: john-wood-w arunkant-uws

Work Items

There are 3 work items required for this change:

updating the barbican code to detect incorrect content types and pecan abort with HTTP 415
updating the barbican documentation to describe this change and provide recovery action
updating the barbican tests to validate the behavior of the new code. This includes unit and functional tests.

Dependencies

None.

Testing

The following positive test scenarios will be needed for this proposed change:

Resource	Verb	Content-Type	Expected Result
Secret	PUT	application/octet-stream	success
Secret	PUT	text/plain	success
Secrets	POST	application/json	success
Orders	POST	application/json	success
Containers	POST	application/json	success
Transport Keys	POST	application/json	success

In addition, the following negative tests will be used to verify behavior:

Resource	Verb	Content-Type	Expected Result
Secret	PUT	none (omitted)	HTTP 415
Secret	PUT	application/octet-streamx	HTTP 415
Secret	PUT	text/plainx	HTTP 415
Secret	PUT	applicationx/octet-stream	HTTP 415
Secret	PUT	textx/plain	HTTP 415
Secret	PUT	application/json	HTTP 415

Resource	Verb	Content-Type	Expected Result
Secrets	POST	none (omitted)	HTTP 415
Secrets	POST	text/plain	HTTP 415
Secrets	POST	application/octet-stream	HTTP 415
Secrets	POST	application/jsonx	HTTP 415
Secrets	POST	applicationx/json	HTTP 415
Secrets	POST	application/jsonx	HTTP 415

Resource	Verb	Content-Type	Expected Result
Orders	POST	none (omitted)	HTTP 415
Orders	POST	text/plain	HTTP 415
Orders	POST	application/octet-stream	HTTP 415
Orders	POST	application/jsonx	HTTP 415
Orders	POST	applicationx/json	HTTP 415
Orders	POST	application/jsonx	HTTP 415

Resource	Verb	Content-Type	Expected Result
Containers	POST	none (omitted)	HTTP 415
Containers	POST	text/plain	HTTP 415
Containers	POST	application/octet-stream	HTTP 415
Containers	POST	application/jsonx	HTTP 415
Containers	POST	applicationx/json	HTTP 415
Containers	POST	application/jsonx	HTTP 415

Resource	Verb	Content-Type	Expected Result
Transport Keys	POST	none (omitted)	HTTP 415
Transport Keys	POST	text/plain	HTTP 415
Transport Keys	POST	application/octet-stream	HTTP 415
Transport Keys	POST	application/jsonx	HTTP 415
Transport Keys	POST	applicationx/json	HTTP 415
Transport Keys	POST	application/jsonx	HTTP 415

Documentation Impact

Documentation will need to specify that the content-type header is now required for the API/verbs shown above under “Proposed Change”. Omitted content-type, or content-type other than the allowed values will result in an HTTP 415.

NOTE: current documentation says that a secret PUT “should” include the appropriate content-type. That will have to be updated to “must”. See https://github.com/cloudkeep/barbican/wiki/Application-Programming-Interface#put

References

The following bugs and proposed fixes led to the creation of this blueprint:

Remove the project-id from Barbican resource URIs

Mon, 16 Jun 2014 00:00:00

https://blueprints.launchpad.net/barbican/+spec/api-remove-uri-tenant-id

OpenStack projects appear to be moving away from requiring a project-id in the URI, as this is redundant information when Keystone authentication is used. All Barbican resource URIs includes project-id as part of their URI. This need to be reviewed.

Problem Description

All Barbican resources have a project-id in their URI. This helps to correlate the resource with the specified tenant when no external authentication mechanism is used.

However, all Barbican deployments would use Keystone to authenticate the API requests. With Keystone, the project-id information is already obtained when the X-Auth-Token header from the request is validated. This makes the project-id in the URI redundant.

Need a solution to remove the project-id from the URI.

Proposed Change

All Barbican resource URIs will drop project-id from their URI. For example, /v1/<project-id>/secrets will become /v1/secrets, /v1/<project-id>/secrets/<secret-ref> will become /v1/secrets/<secret-ref> and so on.
For all successfully authenticated client requests, the project-id (a.k.a tenant-id) would be obtained by looking up the authentication context associated with the request. If it is not found (unscoped token), the request will be denied with a HTTP 401 error.
If no authentication mechanism has been configured with the Barbican deployment, the client is expected to pass in a “X-Project-Id” request header, set to the desired project-id. If this header is not passed in, the request will be denied with a HTTP 400 error. In other words, the client is expected to spoof the data presented to Barbican by keystone.middleware.auth_token, which Barbican trusts.

Backwards compatability note:

Once the changes proposed by this spec are in place, callers of the API should no longer specify project-id in the URI. Otherwise, the request will be denied with a HTTP 404 error.

Alternatives

None.

Data model impact

None.

REST API impact

All Barbican REST resources will be impacted by this change. However, this change is limited to dropping the project-id from the uri. All other request parameters (including limit and filter parameters) will remain the same. If Barbican is configured for unauthenticated request flow, then X-Project-Id is required to be passed by the caller.

API

Secrets

Create Secret: POST /v1/secrets

Example:

Request:

    Headers:

    X-Auth-Token:<token>
    Content-Type:application/json

    POST /v1/secrets

    {
      "name": "AES key",
      "expiration": "2014-02-28T19:14:44.180394",
      "algorithm": "aes",
      "bit_length": 256,
      "mode": "cbc",
      "payload": "gF6+lLoF3ohA9aPRpt+6bQ==",
      "payload_content_type": "application/octet-stream",
      "payload_content_encoding": "base64"
    }


Response:

    Status: 201 Created

    {
        "secret_ref": "http://localhost:9311/v1/secrets/a8957047-16c6-4b05-ac57-8621edd0e9ee"
    }

Two-step secret creation: PUT /v1/secrets/<secret-id>

Example:

Request:

    Headers:

    X-Auth-Token:<token>
    Content-Type:text/plain

    PUT /v1/secrets/a8957047-16c6-4b05-ac57-8621edd0e9ee

    'mysecret'

Response:

    Status: 201 Created

    {
        "secret_ref": "http://localhost:9311/v1/secrets/a8957047-16c6-4b05-ac57-8621edd0e9ee"
    }

List Secrets: GET /v1/secrets

Example:

Request:

    Headers:

    X-Auth-Token:<token>

    GET /v1/secrets

Response:

    Status: 200 Ok

    {
      "secrets": [
        {
          "status": "ACTIVE",
          "updated": "2013-06-28T15:23:30.668641",
          "mode": "cbc",
          "name": "Main Encryption Key",
          "algorithm": "AES",
          "created": "2013-06-28T15:23:30.668619",
          "secret_ref": "http://localhost:9311/v1/secrets/e171bb2d-f14f-433e-84f0-3dfcac7a7311",
          "expiration": "2014-06-28T15:23:30.668619",
          "bit_length": 256,
          "content_types": {
            "default": "application/octet-stream"
          }
        },
        {
          "status": "ACTIVE",
          "updated": "2013-06-28T15:23:32.210474",
          "mode": "cbc",
          "name": "Backup Key",
          "algorithm": "AES",
          "created": "2013-06-28T15:23:32.210467",
          "secret_ref": "http://localhost:9311/v1/secrets/6dba7827-c232-4a2b-8f3d-f523ca3a3f99",
          "expiration": null,
          "bit_length": 256,
          "content_types": {
            "default": "application/octet-stream"
          }
        },
        {
          "status": "ACTIVE",
          "updated": "2013-06-28T15:23:33.092660",
          "mode": null,
          "name": "PostgreSQL admin password",
          "algorithm": null,
          "created": "2013-06-28T15:23:33.092635",
          "secret_ref": "http://localhost:9311/v1/secrets/6dfa448d-c35a-4158-abaf-e4c249efb580",
          "expiration": null,
          "bit_length": null,
          "content_types": {
            "default": "text/plain"
          }
        }
      ],
      "next": "http://localhost:9311/v1/secrets?limit=3&offset=5",
      "previous": "http://localhost:9311/v1/secrets?limit=3&offset=0"
    }

Get individual secret: GET /v1/secrets/<secret-id>

Example:

Request:

    Headers:

    X-Auth-Token:<token>

    GET /v1/secrets/e171bb2d-f14f-433e-84f0-3dfcac7a7311

Response:

    Status: 200 Ok

    {
      "status": "ACTIVE",
      "updated": "2013-06-28T15:23:30.668641",
      "mode": "cbc",
      "name": "Main Encryption Key",
      "algorithm": "AES",
      "secret_ref": "http://localhost:9311/v1/secrets/e171bb2d-f14f-433e-84f0-3dfcac7a7311",
      "expiration": "2014-06-28T15:23:30.668619",
      "bit_length": 256,
      "content_types": {
        "default": "application/octet-stream"
      }
    }

Delete individual secret: DELETE /v1/secrets/<secret-id>

Example:

Request:

    Headers:

    X-Auth-Token:<token>

    DELETE /v1/secrets/e171bb2d-f14f-433e-84f0-3dfcac7a7311

Response:

    Status: 204 No Content

Create Secret(no authentication) : POST /v1/secrets

Example:

Request:

    Headers:

    X-Project-Id:<project-id>
    Content-Type:application/json

    POST /v1/secrets

    {
      "name": "AES key",
      "expiration": "2014-02-28T19:14:44.180394",
      "algorithm": "aes",
      "bit_length": 256,
      "mode": "cbc",
      "payload": "gF6+lLoF3ohA9aPRpt+6bQ==",
      "payload_content_type": "application/octet-stream",
      "payload_content_encoding": "base64"
    }


Response:

    Status: 201 Created

    {
        "secret_ref": "http://localhost:9311/v1/secrets/a8957047-16c6-4b05-ac57-8621edd0e9ee"
    }

Orders

Create Order: POST /v1/orders

Example:

Request:

    Headers:

    X-Auth-Token:<token>
    Content-Type:application/json

    POST /v1/orders

    {
      "secret": {
        "name": "secretname",
        "algorithm": "AES",
        "bit_length": 256,
        "mode": "cbc",
        "payload_content_type": "application/octet-stream"
      }
    }

Response:

    Status: 201 Created

    {
        "order_ref": "http://localhost:9311/v1/orders/a8957047-16c6-4b05-ac57-8621edd0e9ee"
    }

Get individual order: GET /v1/orders/<order-id>

Example:

Request:

    Headers:

    X-Auth-Token:<token>

    GET /v1/orders/f9b633d8-fda5-4be8-b42c-5b2c9280289e

Response:

    Status: 200 Ok

    {
      "secret": {
        "name": "secretname",
        "algorithm": "aes",
        "bit_length": 256,
        "mode": "cbc",
        "payload_content_type": "application/octet-stream"
      },
      "order_ref": "http://localhost:8080/v1/orders/f9b633d8-fda5-4be8-b42c-5b2c9280289e",
      "secret_ref": "http://localhost:8080/v1/secrets/888b29a4-c7cf-49d0-bfdf-bd9e6f26d718",
      "status": "ERROR",
      "error_status_code": "400 Bad Request",
      "error_reason": "Secret creation issue seen - content-encoding of 'bogus' not supported."
    }

Get list of orders per tenant: GET /v1/orders

Example:

Request:

    Headers:

    X-Auth-Token:<token>

    GET /v1/orders


Response:

    Status: 200 Ok

    {
      "orders": [
        {
          "status": "ACTIVE",
          "secret_ref": "http://localhost:9311/v1/secrets/bf2b33d5-5347-4afb-9009-b4597f415b7f",
          "updated": "2013-06-28T18:29:37.058718",
          "created": "2013-06-28T18:29:36.001750",
          "secret": {
            "name": "secretname",
            "algorithm": "aes",
            "bit_length": 256,
            "mode": "cbc",
            "payload_content_type": "application/octet-stream"
          },
          "order_ref": "http://localhost:9311/v1/orders/3100078a-6ab1-4c3f-ab9f-295938c91733"
        },
        {
          "status": "ACTIVE",
          "secret_ref": "http://localhost:9311/v1/secrets/fa71b143-f10e-4f7a-aa82-cc292dc33eb5",
          "updated": "2013-06-28T18:29:37.058718",
          "created": "2013-06-28T18:29:36.001750",
          "secret": {
            "name": "secretname",
            "algorithm": "aes",
            "bit_length": 256,
            "mode": "cbc",
            "payload_content_type": "application/octet-stream"
          },
          "order_ref": "http://localhost:9311/v1/orders/30b3758a-7b8e-4f2c-b9f0-f590c6f8cc6d"
        }
      ]
    }

Delete individual order: DELETE /v1/orders/<order-id>

Example:

Request:

    Headers:

    X-Auth-Token:<token>

    DELETE /v1/orders/e171bb2d-f14f-433e-84f0-3dfcac7a7311

Response:

    Status: 204 No Content

Containers

Create Container: POST /v1/containers

Example:

Request:

    Headers:

    X-Auth-Token:<token>
    Content-Type:application/json

    POST /v1/containers

    {
      "name": "container name",
      "type": "rsa",
      "secret_refs": [
        {
           "name": "private_key",
           "secret_ref":"http://localhost:9311/v1/secrets/05a47308-d045-43d6-bfe3-1dbcd0c3a97b"
        },
        {
           "name": "public_key",
           "secret_ref":"http://localhost:9311/v1/secrets/05a47308-d045-43d6-bfe3-1dbcd0c3a97b"
        },
        {
           "name": "private_key_passphrase",
           "secret_ref":"http://localhost:9311/v1/secrets/05a47308-d045-43d6-bfe3-1dbcd0c3a97b"
        }
      ]
    }


Response:

    Status: 201 Created

    {
        "container_ref": "http://localhost:9311/v1/containers/a8957047-16c6-4b05-ac57-8621edd0e9ee"
    }

Get individual container: GET /v1/containers/<container-id>

Example:

Request:

    Headers:

    X-Auth-Token:<token>

    GET /v1/containers/f9b633d8-fda5-4be8-b42c-5b2c9280289e

Response:

    Status: 200 Ok

    {
       "name":"rsa container",
       "secret_refs":[
          {
             "secret_ref":"http://localhost:9311/v1/secrets/059805d5-b400-47da-abc5-cae7286d3ede",
             "name":"private_key_passphrase"
          },
          {
             "secret_ref":"http://localhost:9311/v1/secrets/28704f0f-3273-40d4-bc40-4de2691135ea",
             "name":"private_key"
          },
          {
             "secret_ref":"http://localhost:9311/v1/secrets/29d89344-10ad-4f92-8aa2-adebaf7556ee",
             "name":"public_key"
          }
       ],
       "container_ref":"http://localhost:9311/v1/containers/888b29a4-c7cf-49d0-bfdf-bd9e6f26d718",
       "type":"rsa"
    }

Get list of containers per tenant: GET /v1/containers

Example:

Request:

    Headers:

    X-Auth-Token:<token>

    GET /v1/containers


Response:

    Status: 200 Ok

    {
       "total":42,
       "containers":[
          {
             "status":"ACTIVE",
             "updated":"2014-02-11T18:05:58.909411",
             "name":"generic container_updated",
             "secret_refs":[
                {
                   "secret_id":"123",
                   "name":"private_key"
                },
                {
                   "secret_id":"321",
                   "name":"public_key"
                },
                {
                   "secret_id":"456",
                   "name":"private_key_passphrase"
                }
             ],
             "created":"2014-02-11T18:05:58.909403",
             "container_ref":"http://localhost:9311/v1/containers/d4e06015-4f6e-4626-ac3d-4ece6621f96d",
             "type":"rsa"
          },
          {
             "status":"ACTIVE",
             "updated":"2014-02-11T18:08:58.160557",
             "name":"generic container_updated",
             "secret_refs":[
                {
                   "secret_id":"321",
                   "name":"public_key"
                },
                {
                   "secret_id":"456",
                   "name":"private_key_passphrase"
                }
             ],
             "created":"2014-02-11T18:08:58.160551",
             "container_ref":"http://localhost:9311/v1/containers/bb24fa61-0b5f-4d40-8990-846e95cd7b12",
             "type":"rsa"
          },
          {
             "status":"ACTIVE",
             "updated":"2014-02-11T18:25:58.198072",
             "name":"generic container_updated",
             "secret_refs":[
                {
                   "secret_id":"1df433d6-c2d4-480d-90fb-0bfd9c5da3dd",
                   "name":"private_key"
                },
                {
                   "secret_id":"321",
                   "name":"public_key"
                },
                {
                   "secret_id":"456",
                   "name":"private_key_passphrase"
                }
             ],
             "created":"2014-02-11T18:25:58.198063",
             "container_ref":"http://localhost:9311/v1/containers/38f58696-5013-4bd6-ab2b-fbea41dc957a",
             "type":"rsa"
          },
          {
             "status":"ACTIVE",
             "updated":"2014-02-11T18:44:06.296957",
             "name":"generic container_updated",
             "secret_refs":[
                {
                   "secret_id":"1df433d6-c2d4-480d-90fb-0bfd9c5da3dd",
                   "name":"private_key"
                },
                {
                   "secret_id":"321",
                   "name":"public_key"
                },
                {
                   "secret_id":"456",
                   "name":"private_key_passphrase"
                }
             ],
             "created":"2014-02-11T18:44:06.296947",
             "container_ref":"http://localhost:9311/v1/containers/a8d1adfd-0d36-4eb0-8762-99787eb4a7ff",
             "type":"rsa"
          }
       ],
       "next":"http://localhost:9311/v1/containers?limit=10&offset=10"
    }

Delete individual container: DELETE /v1/containers/<container-id>

Example:

Request:

    Headers:

    X-Auth-Token:<token>

    DELETE /v1/containers/e171bb2d-f14f-433e-84f0-3dfcac7a7311

Response:

    Status: 204 No Content

Security impact

This change will require that all requests have a valid keystone token which will be used for authorization.
Requests without the token will be considered as unauthenticated requests.

Notifications & Audit Impact

None.

Other end user impact

Barbican python client will need to be modified to accommodate API changes. This work could happen in parallel to the server side tasks.

Performance Impact

None.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Venkat Sundaram (tsv)

Work Items

Modify PecanAPI routing mechanism and remove logic that parses project-id from the URI (tsv)
Enhance the enforce_rback decorator for the controllers to get the keystone_id from the authentication context if it is not passed in(tsv)
Update the barbican/common/util module method hostname_for_refs to strip project-id from the URI links returned in response body(tsv)
Modify barbican client and replace all tenant-id references from the URI it calls (tsv)
Enhance the tests (tsv)

Dependencies

None.

Testing

Update unit tests to drop project-id from uri.
Tempest tests need to be added for functional testing

Documentation Impact

Existing documentation has to be updated to remove the project-id from the uri.
Explain how to specify project-id when no authentication mechanism is used.

References

Barbican Spec - Barbican Consumer Registration

Wed, 11 Jun 2014 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/barbican/+spec/api-add-container-registration

Allow consumers to register as “interested parties” on Barbican Containers and Secrets. This information would then be available when the container is retrieved, which a client process could use to either warn about these interested parties prior to performing a delete operation, or else use to determine what parties to update if a new Container is created that needs to replace an old one.

Problem description

In some workflows, multiple clients will be accessing the same Container/Secret information in Barbican. This workflow might involve a client updating or deleting this ‘shared’ information. If this client could check to see what entities are interested in this information, they might be able to perform follow on operations (such as updating or notifiying the affected entities of the update), or else warn the user that an operation could impact other entites (such as before performing a delete).

An example workflow is when a client creates certificate information within Barbican (which creates a Container with a unique UUID), and then shares that Container UUID with LBaaS as part of creating a load balancer instance. LBaaS could both retrieve the Container by UUID to get the certificate to install, and also register itself as an interested consumer for that Container. Now if a client wishes to delete that same Container later, they can check to see if one or more consumers are interested in the Container first. They might choose (for example) to remove the load balancer first, or else abort the delete process altogether.

Proposed change

Allow consumers to register their interest for a Barbican Container or Secret.

This will require a new REST Pecan API sub-resource under the ‘containers’ and ‘secrets’ resource, so consumers can POST to register and DELETE to deregister with their consumer data in the JSON body. This is detailed in the API impact section below.

We will also need two new tables in the data model for storing this consumer data. This is detailed in the Data model impact section below.

Alternatives

This could also be done without a new resource, and using query parameters like ‘register’ and ‘deregister’ on the existing GET call for a Container, but that does not conform to a true RESTful scheme and would add side effect bahaviors to the simple GET call.

We could avoid the admin-api and instead add the functionality to the standard user-api, restricting access using keystone roles.

We could use a single generic table for the Data Model instead of one for each entity type, but it seems to be consensus that it is worth an extra table in order to maintain a clear ForeignKey relationship.

Alternatives to the entire service registration concept include the following:

1) Keep Barbican simple and force clients to handle switching from one Container UUID to a new one, in all the dependent consumers such as LBaaS. Orchestration systems such as Heat might be able to handle this conversion. However the LBaaS community is concerned that since clients would still have access to the Containers, they could in turn delete them from under such orchestrations. LBaaS attempts to failover to a new load balance instance if needed would then fail, as the original Container UUID would no longer be available.

2) Allow Containers to be mutable such that certificates (for example) could be updated without changing Container UUIDs. Barbican could then fire off events to interest parties (like LBaaS) for them to update certificates in load balancer instances. If new certificate is flawed however, LBaaS would need to be able to revert back to a previous version of the certificate. Adding versioning to Barbican would be a significant effort with minimal benefit versus effort.

Data model impact

A new model and repository will be created for ContainerConsumerMetadatum containing the following:

container_id (ForeignKey reference to Container.id)

name (String, freeform data specified by the consumer, eg: “LBaaS”, “VPNaaS”)

URL (String, resource locator for the object that depends on this entity, eg: “https://lbaas.myurl.net/loadbalancer/<lbid>”)

A Unique constraint would be put on the combination of “container_id”, “name”, and “URL”. An Index would be created on “container_id” and “name”.

The same will be done for SecretConsumerMetadatum, with column names updated appropriately.

As this is an optional one to many relationship off of Containers/Secrets, data migrations should not be required.

REST API impact

This BP impacts both the admin API and the user API.

Changes that affect the admin API:

Need to add a new resource in the admin-api named “consumers”, supporting GET, POST and DELETE. This would be a subresource of “containers” and “secrets”, so we would also need to create a stub resource for “containers” and “secrets” on the admin-api. Example: http://admin-api/v1/containers/{container-uuid}/consumers

A simple registration JSON Body would be structured like the following using the POST method:

{
    "name": "LBaaS",
    "URL": "https: //lbaas.myurl.net/loadbalancer/4124/"
}

The response of this registration POST would be the current entity response as available from a GET call. Hence after a client creates a Container and hands that UUID to LBaaS (for example) to create a load balancer instance, LBaaS will make the above POST call to the ‘consumers’ sub-resource of that Container UUID. LBaaS would then synchronously receive the Container’s information as if they had performed a GET call, and can then install the certificate on the load balancer.

Deregistration would use the same resource and JSON Body as the registration, except using the DELETE method. Hence the delete would be performed by value rather than by a specific UUID reference to a consumer entity (since the registration UUID is never exposed to the consumer).

Changes that affect the user API:

The JSON Body for the GET response of a Container/Secret would include the current GET response for that entity, plus a new ‘consumers’ element, that will be structured as follows:

{
    <current GET response data>,
    "consumers": [
        {
            "name": "LBaaS",
            "URL": "https://lbaas.myurl.net/loadbalancer/4124/"
        },
        {
            "name": "LBaaS",
            "URL": "https://lbaas.myurl.net/loadbalancer/4125/"
        },
        {
            "name": "VPNaaS",
            "URL": "https://vpn.myurl.net/vpn/345634/"
        }
    ]
}

If significant performance degredation is observed in testing, the current Container/Secret API could change as follows in a future CR:

1) The GET on the root (list) call can be modified to not return the ‘consumers’ attribute to reduce the overall size of the response message.

Security impact

This blueprint would not require changes to cryptographic resources. We might need to impose an upper limit on the number of consumers that can be registered per Container/Secret to prevent denial of service attacks.

There is also some concern that any metadata stored on Barbican entities should be encrypted for security, but that concern applies to much more than this functionality, and should probably be handled as its own blueprint.

Notifications impact

None.

Other end user impact

None.

Performance Impact

Initially, there could be a performance impact on the GET for /containers/ and /secrets/ (the list, not individual entities) because of an increase in the amount of data returned. If this is significant, then consumer registration information will only be retrieved for GETs on specific Containers/Secrets, and not on the GET list of all Containers/Secrets.

Other deployer impact

None.

Developer impact

Developers of services that consume Barbican Containers/Secrets would use the admin-api consumer service instead of using the standard api GET resource when they wish to register their interest in an entity. This feature is optional however, and would not impact current workflows.

Implementation

Assignee(s)

Primary assignees:

adam-harwell
vivek-jain

Work Items

The initial implementation will consist of two CRs:

One CR will be created for the initial implementation with Containers.
A subsequent CR will be created to add this functionality to Secrets.

There may be an additional CR for the above mentioned performance changes.

There may be an additional BP/CR for encryption of this (and other) metadata.

Dependencies

None.

Testing

Add unit and integration testing for the new feature.

Documentation Impact

Update API guide here: https://github.com/cloudkeep/barbican/wiki/Application-Programming-Interface

References

https://wiki.openstack.org/wiki/Neutron/LBaaS/SSL#TLS_Certificates_Management (At the time this was written, this wiki is still assuming Mutable Containers, which will be corrected soon.)

Add more types to the orders resource

Thu, 05 Jun 2014 00:00:00

https://blueprints.launchpad.net/barbican/+spec/api-orders-add-more-types

Barbican’s orders resource is used to generate secrets on behalf of clients. Currently (as of Icehouse M2) only specific symmetric key secrets can be generated. This blueprint addresses how the orders resources can be modified to generate other useful secret types, such as asymmetric key-pairs.

Problem Description

Barbican is currently lacking generation support for following secret types which is highly expected from a Key Manager:

There are three main types of secret information that could be generated by Barbican via the Orders resource:

Key:
- Symmetric encryption keys: AES, 3DES, Camellia, RC4.
- Other types of keys: HMAC, byte stream
Asymmetric:
- RSA, DSA, EC.
- Can include public and private keys, and a passphrase
Certificate:
- Generation by CSR signing by CA.
- Generation by Barbican and its back-end plug-in using different parameters.

Proposed Change

This is a multi phase initiative as described below:

1. Enhance Barbican crypto plug-in framework to support generation of different types of secrets. (Completed)

Make appropriate changes to the Order data model to hold required parameters.
Enhance Orders REST API to support Key and Asymmetric secrets.
Enhance Orders REST API to support Certificate type secrets which includes
- API changes to support CSR post.
- API changes to support post or certificate parameters.
- Barbican core subsystem to integrate with CA.
- Barbican event subsystem to emit events for certificate life-cycle.

Alternatives

None.

Data model impact

Barbican Order model has to be enhanced to support these requirements. Following new fields will need to be added in the model:

Type (String): This field will represent the order type. Allowed options are
key, asymmetric and certificates
Meta (Text): This field will hold JSON string and store all meta attributes
for secrets.
container_id: An optional field depending on generated artifacts, a particular
type of secret can have multiple sub secrets. Such secrets requires Container object to group multiple secrets. e.g. Private Key, Public Key and Passphrase for a asymmetric key.

Note: Either the container_id or the existing secret_id will be filled out once the order is ACTIVE, depending on the type of secret generated by the order.

REST API impact

The ordering resource allows for the generation of secret material by Barbican. The ordering object encapsulates the work flow and history for the creation of a secret. This interface is implemented as an asynchronous process since the time to generate a secret can vary depending on the type of secret.

Resource Structure:

{
   "type": "--type--",
   "meta": {
      "name": "--name--",
      "algorithm": "--algorithm--",
      "bit_length": --bit_length--,
      "mode": "--mode--",
      "pass_phrase": "--pass_phrase--",
      "expiration": "--expiration--",
      "payload_content_type": "--type--"
   }
}

Note: Structure for meta is not yet decided for type certificate.

Required Attribute:

type (string)

Type of the secret, supported options are key, asymmetric and certificate:

meta (string)

Describes additional information regarding the secret:

meta.name (string)

Human readable name for the secret:

meta.algorithm (string)

Cryptographic algorithm type used to generate the secret. e.g. aes, 3des, hmacsha1, hmacsha256,hmacsha384, hmacsha512, rsa and dsa:

meta.bit_length (int)

The bit-length of the secret.

Optional Attribute:

meta.mode (string)

The type/mode of the algorithm associated with the secret information:

meta.pass_phrase (string)

Pass phrase to be associated with secret. Used in conjunction when type is asymmetric:

meta.expiration (string)

The expiration date for the secret in ISO-8601 format. Once the secret has expired, it will no longer be returned by the API or agent. If this field is not supplied, then the secret has no expiration date:

meta.payload_content_type (string)

API

Order

Create Order: POST /{tenant_id}/orders

Example 01 - Symmetric key generation:

Request:

    POST /v1/{tenant_id}/orders

    {
       "type": "key",
       "meta": {
          "name": "secretname",
          "algorithm": "aes",
          "bit_length": 256,
          "mode": "cbc",
          "expiration": "2015-02-28T19:14:44.180394",
          "payload_content_type": "application/octet-stream"
       }
    }

Response:

    Status: 201 Created

    {
        "order_ref": "http://localhost:9311/v1/1234/orders/daf3c6de-095f-46a8-94ec-65ec0d98eb68"
    }

Example 02 - Asymmetric key generation:

Request:

    POST /v1/{tenant_id}/orders

    {
       "type": "asymmetric",
       "meta": {
          "name": "secretname",
          "algorithm": "RSA",
          "bit_length": 2048,
          "expiration": "2015-02-28T19:14:44.180394",
          "payload_content_type": "application/octet-stream"
       }
    }

Response:

    Status: 201 Created

    {
        "order_ref": "http://localhost:9311/v1/1234/orders//a8957047-16c6-4b05-ac57-8621edd0e9ee"
    }

Example 03 - Certificate generation:

Request: (TBD)

Response: (TBD)

List Orders: GET /{tenant_id}/orders

List orders per tenant:

Request:

    GET /v1/{tenant_id}/orders

Response:

    Status: 200 OK

    {
      "orders": [
        {
          "status": "ACTIVE",
          "secret_ref": "http://localhost:9311/v1/12345/secrets/bf2b33d5-5347-4afb-9009-b4597f415b7f",
          "updated": "2013-06-28T18:29:37.058718",
          "created": "2013-06-28T18:29:36.001750",
            "type": "key",
            "meta": {
              "name": "secretname",
              "algorithm": "AES",
              "bit_length": 256,
              "mode": "cbc",
              "expiration": "2015-02-28T19:14:44.180394",
              "payload_content_type": "application/octet-stream"
          },
          "order_ref": "http://localhost:9311/v1/1234/orders/daf3c6de-095f-46a8-94ec-65ec0d98eb68"
        },
        {
          "status": "ACTIVE",
          "container_ref": "http://localhost:9311/v1/12345/containers/fa71b143-f10e-4f7a-aa82-cc292dc33eb5",
          "updated": "2013-06-28T18:29:37.058718",
          "created": "2013-06-28T18:29:36.001750",
            "type": "asymmetric",
            "meta": {
              "name": "secretname",
              "algorithm": "RSA",
              "bit_length": 2048,
              "expiration": "2015-02-28T19:14:44.180394",
              "payload_content_type": "application/octet-stream"
          },
          "order_ref": "http://localhost:9311/v1/1234/orders//a8957047-16c6-4b05-ac57-8621edd0e9ee"
        }
      ]
    }

Note: There is no change to the existing “limit” and “offset” filter parameters.

Get Order: GET /{tenant_id}/orders/{order_id}

Get order by order_id:

Request:

    GET /v1/{tenant_id}/orders/{order_id}

Response:

    Status: 200 OK

   {
            "type": "asymmetric",
            "meta": {
              "name": "secretname",
              "algorithm": "RSA",
              "bit_length": 2048,
              "expiration": "2015-02-28T19:14:44.180394",
              "payload_content_type": "application/octet-stream"
          },
          "status": "ACTIVE",
          "container_ref": "http://localhost:9311/v1/12345/containers/fa71b143-f10e-4f7a-aa82-cc292dc33eb5",
          "updated": "2013-06-28T18:29:37.058718",
          "created": "2013-06-28T18:29:36.001750",
          "order_ref": "http://localhost:9311/v1/1234/orders//a8957047-16c6-4b05-ac57-8621edd0e9ee"
        }

Security impact

None

Notifications & Audit Impact

None.

Other end user impact

Barbican-python client will need to be enhanced to accommodate API changes.

Performance Impact

None

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Arvind Tiwari (atiwari) is leading this feature enhancement.

Work Items

Enhance crypto plug-in framework to support secret generation for different types. (atiwari, DONE)
Data migration to add Type, Meta and container_id in the Orders Model. (atiwari, DONE)
Enhance Order REST API to support key and asymmetric order type. (atiwari, IN_PROGRESS, https://review.openstack.org/#/c/87405/)
Enhance Order REST API to support certificate order type. (TBD)
Enhance Barbican python client to support key and asymmetric order type. (TBD)
Enhance Barbican python client to support certificate order type. (TBD)

Dependencies

None

Testing

Need to add unit testing for the new order types.
This change has to be backwards compatible by allowing the current order contract to be used to generate symmetric keys, in addition to the new way outlined in this blueprint.

Documentation Impact

Order API request and response structure has to be explained in the docs.
https://github.com/cloudkeep/barbican/wiki/Application-Programming-Interface#orders-resource
Barbican WADL and developer guide will be update for this API change.

References

Restructure project to better accommodate all plugin types

Thu, 05 Jun 2014 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/barbican/+spec/restructure-for-plugins

The current project structure only naturally accommodates HSM-style plugins in the ‘crypto’ package. This blueprint accommodates all the plugin types required by Barbican by reorganizing the project structure under a ‘plugin’ package.

Problem description

The current project structure places the only plugin currently supported by Barbican (used to interface with HSMs) into a ‘crypto’ package. Planning for new Barbican features has revealed the need for new types of plugins, that are either awkward to implement with the HSM contract (such as Dogtag or KMIP) or else are entirely different types of plugins. The latter includes SSL certificate workflow processing plugins, and eventing plugins. These new plugins do not fit well into the current ‘crypto’ package, and hence Barbican has outgrown it.

Proposed change

This blueprint replaces the current ‘crypto’ package with the following package structure:

common/

resources.py - API and Worker processes call into this module to: generate, store or get secrets.
plugin_managers.py - Contains base-level stevedore plugin lookup logic as: needed. Extended by interfaces in plugin package.

plugin/

interface/ - Stores Plugin contracts used by other Barbican packages

(as abc abstracts). Also has stevedore lookup methods.

secret_store.py - Nate’s current data storage plugin, that handles
securely storing/retrieving secrets (such as Dogtag and KMIP). Would also include a class called SecretStorePluginManager that uses stevedore to look up Nate’s secret store plugin implementations.

certificates.py - SSL certificate workflow plugins. Would also include
a class called CertGenerationPluginManager that uses stevedore to look up certificate generation workflow plugin implementations.

crypto/ - Similar to the current ‘crypto’ package. Stores

HSM and security-module related modules. Because this is an ‘inner’ interface not intended to be called from outside the plugin package, these modules are segregated from the ‘interface’ package above.

crypto.py - Key and secret generation plugin for HSM-style
interfaces. Similar to the current crypto/plugin.py::CryptoPluginBase interface. Would also include a class called SecretGenerationPluginManager that uses stevedore to look up the symmetric/asymmetric key generation plugins (the so called ‘second level plugin lookup’). This logic is very similar to the current crypto/extension_manager.py.

p11_crypto.py - Paul’s HSM plugin implementation.

simple_crypto.py - Simple implementation of security-module plugin.

store_crypto.py - A secret_store.py implementation that adapts between
the higher-level secret_store.py interface and the lower-level crypto.py interface. Would utilize SecretGenerationPluginManager to locate the ‘second level’ HSM-style plugin implementation.

dogtag.py - Dogtag implementation of the secret_store.py interface.

kmip.py - A KMIP implementation of the secret_store.py interface.

symantec.py - A Symantec implementation of the certificates.py
interface.

This effort first introduces the new structure only. Then existing plugins are moved over to the new structure, verifying unit tests still pass.

To provide concrete context to this new structure, the following sequence provides an example flow to store a secret after this refactoring effort:

A call is made to POST /secrets (using the one-step method)
The API code in barbican.api.controllers.secrets.py makes a call to…
Common code in barbican.common.resources.py that has a line such as this:

storer = SecretStorePluginManager.getPlugin(
              plugin_managers.STORE_SECRET, meta_data)
storer.store_secret(...)

4. SecretStorePluginManager will look at the config file and see which plugin is available. Initially it will be one of the implementations defined in plugin.dogtag, plugin.kmip, or plugin.store_crypto.

5. If the plugin is dogtag or kmip, then store_secret() will implemented by the plugin code directly.

6. If the plugin is the store_crypto adapter, then store_secret() will in turn call common.plugin_manager.py to find a crypto/security-module type of plugin (i.e. a second-stage stevedore lookup). Initially this will be either plugin/crypto/p11_crypto.py or simple_crypto.py, and the encrypt() method will be called on it as done now.

7. The plugin returns, updates meta_data, creates the secret record and returns the url to the user.

Alternatives

More detailed discussion occurred on this etherpad: https://etherpad.openstack.org/p/extension-manager

An original approach discussed elevating the interfaces to top level Barbican packages. However, that would place them at the same level as the ‘common’, ‘api’ and ‘task’ packages (for example) which would make the plugin source code locations harder to locate. Hence it seemed that centralizing all plugins under a clear ‘plugin’ package would provide this proper delineation relative to the rest of the source code. Under the ‘plugin’ package, the structure is designed to organize interface, default and external modules.

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

None.

Other deployer impact

None.

Developer impact

This first phase of this change will not be impactful as it sets up the new folder structure only. Once existing plugin implementations are moved to the new structure, pending CRs will be impacted as the file structure has changed. We should be able to sequence this change with our contributors however.

Implementation

Assignee(s)

Primary assignee:: john-wood-w
Other contributors:: alee-3 rellerreller

Work Items

CR #1 - Add new structure in parallel to current structure, should not impact: current code base.
CR #2 - Move existing plugins over to the new structure, including updating: api/ and tasks/ modules to refer to the new locations, and then verifying unit tests still pass.

Dependencies

None.

Testing

Current unit testing will need to be changed to reference the new location of items in the revised project structure.

Documentation Impact

Update this wiki section, and the sections thereafter it: https://github.com/cloudkeep/barbican/wiki/Developer-Guide-for-Contributors#detailed-explanation

References

More detailed discussion occurred on this etherpad: https://etherpad.openstack.org/p/extension-manager

db cleanup	Remove all soft-deleted and expired secrets from DB
db restore	Restore a soft-deleted secret from DB
db revision	Create a new DB version file
db upgrade	Upgrade to a future version DB version
db history	Show changeset history
db current	Show current revision for a database

hsm gen-mkek	Generate HSM master key encryption key
hsm gen-mhmk	Generate HSM master HMAC key
hsm rewrap-pkek	Rewrap Project KEKs after rotating to a new MKEK