Keystone Fernet Token Implementation¶

Fernet tokens were added to OpenStack to solve the problem of keystone being required to persist tokens to the a common database (cluster) like UUID tokens, and solve the problem of size for PKI or PKIZ tokens.

From Fernet - Frequently Asked Questions

A fernet token is a bearer token that represents user authentication. Fernet tokens contain a limited amount of identity and authorization data in a MessagePacked payload. The payload is then wrapped as a Fernet message for transport, where Fernet provides the required web safe characteristics for use in URLs and headers. The data inside the fernet token is protected using symmetric encryption keys, or fernet keys.

Problem Description¶

Keystone has had support for Fernet tokens since Kilo, so all services support fernet token authorization. In the upcoming Rocky release the sql token driver and uuid token provider will be removed. The main part of the work on this task is changing the keystone charm to enable configuration of fernet tokens, provide the initialisation of fernet tokens, provide rotation of fernet keys and their subsequent synchronisation to other keystone units.

There are also some issues around what aspects should be configurable vs controlled by the charm. The Proposed Change section provides more details about this.

Finally, upgrading an existing OpenStack implementation from another token system to Fernet tokens is discussed, and the support the charm(s) will need to implement to support this, along with documentation.

Proposed Change¶

Theory of Operation¶

The keystone-charm will, with the appropriate configuration options, configure keystone to generate fernet tokens.

In order to generate tokens, fernet keys are used. These are generated by keystone and have an expiry date. The key repository is a directory, and each key is an integer number, with the highest number being the primary key. Key ‘0’ is the staged key, that will be the next primary. Other keys are secondary keys.

New tokens are only ever generated from the primary key, whilst they secondary keys are used to validate existing tokens. The staging key is not used to generate tokens, but can be used to validate tokens as the staging key might be the new primary key on the master due to a rotation and the keys have not yet been synchronised across all the units.

Fernet keys need to be rotated at periodic intervals, and the keys need to be synchronised to each of the other keystone units. Keys should only be rotated on the master keystone unit, and must be synchronised before they are rotated again. “Over rotation” occurs if a unit rotates its keys such that there is no suitable decoding key on another unit that can decode a token that has been generated on the master. This happens if two key rotations are done on the master before a synchronisation has been successfully performed. This should be avoided. Over rotations can also cause validation keys to be removed before a token’s expiration which would result in failed validations.

There are 3 parts to the Key Rotation Strategy:

The rotation frequency
The token lifespan (set with [token] expiration)
The number of active keys: (set with [fernet_tokens] max_active_keys)

There needs to be at least 3 keys as a minumum. The actual number of keys is determined by the token lifespan and the rotation frequency. The max_active_keys must be one greater than the token lifespan / rotation frequency

To quote from the FAQ:

The number of max_active_keys for a deployment can be determined by dividing the token lifetime, in hours, by the frequency of rotation in hours and adding two. Better illustrated as:

token_expiration = 24
rotation_frequency = 6
max_active_keys = (token_expiration / rotation_frequency) + 2

In the keystone charm, there is already the config parameter token-expiration in seconds, and there will be a proposed config item of fernet-max-active-keys. Thus, the rotation frequence will be calucated as:

token_expiration = 24   # actually 3600, as it's in seconds
max_active_keys = 6
rotation_frequency = token_expiration / (max_active_keys - 2)

Thus, the fernet-max-active-keys can never be less than 3 (which will be enforced in the charm), which would make the rotation frequency the same as the token expiration time.

Upgrading an Existing System¶

To upgrade an existing system to use Fernet tokens, the keystone charm should be upgraded first. The (new) configuration option token-provider has a no default value set, which means on a pre-rocky install the token type will remain as UUID. In order to move a pre-rocky install to Fernet, the token-provider option should be set to fernet.

Note that if a pre-rocky system is upgraded to rocky, then the default token type will be fernet. The rocky cycle removes support for UUID tokens. Thus upgrading a system to rocky will automatically use fernet as the only token type. The token-provider option is only valid for pre-rocky systems.

Note that althought the charms enable token cachinng with memcache by default, this is only for the default of 300 seconds as the token_cache_time is not being set (see (Keystone) Middleware Architecture: Improving response time for further details. The impact of this is that some services may try to re-authenticate during the upgrade, and depending on which keystone unit they “pick”, and what stage the upgrade is at, will determine whether the action succeeds. As different services may be part of the same action, this might lead to odd failure modes. However, once the system is upgraded, after the default of 300 seconds, all services will continue to operate normally.

Therefore, there may be some percieved outage of the openstack control plane. Already running instances will not be affected and all services will continue to operate normally as soon as they request new tokens from keystone.

Additional Configuration Items¶

The following configuration items will be needed in the keystone charm.

token-provider - the token system to use: Either ‘uuid’ or ‘fernet’. The default will not be set. Pre-rocky systems will have a default of uuid. On rocky systems, the configuration option has no effect. As the default is uuid for pre-rocky systems, the token-provider won’t change on an upgrade unless the operator sets the configuration value to fernet.
fernet-max-active-keys - the maximum active keys configured in keystone. This controls the key rotation trigger times based on this config item and the config item token-expiration.

Keystone Actions¶

The following action will be required:

purge-tokens – purge existing tokens from the database. This is used after upgrading from UUID to Fernet tokens,

Internal Cron Jobs¶

The charm will set up a cron job to rotate the keys and then synchronise them to the other peered units. The cron job will call juju run from within the charm to rotate the keys and then synchronise the keys to the other peered units. It will also only perform this action if it is the leader. The cron job will run on all peered units, but only have an effect on the leader.

Synchronisation of the Fernet keys will be via Juju leader settings. The keys are small, and “leader settings” provides a convenient and secure mechanism to synchronise the keys between units without having to explicitly provide networking for all keystone peered units. The delay in transferring the keys using hooks is not an issue as the synchronisation does not need to be immediate; indeed, it could be just before the next key rotation in the worst case, although, this is extremely unlikely to be the case.

Alternatives¶

In the Openstack rocky release, fernet is the only token provider available. Therefore, there is no alternative.

Implementation¶

Assignee(s)¶

Primary assignee:: ajkavanagh
Secondary assignees:: fnordahl

Gerrit Topic¶

Use Gerrit topic “fernet-keystone-charm” for all patches related to this spec.

git-review -t fernet-keystone-charm

Work Items¶

Add fernet token functionality to the keystone-charm. This includes: * setup * upgrade * rotate / sync actions * cron job for automatic rotate / sync.
Add Fernet token information to the documentation: * charm-store text for keystone charm * Notes in the charm guide re: uuid vs Fernet.
Update tests: * Amulet/bundle for actions / verify installation. * Update other bundles to ensure defaults * Upgrade from uuid to Fernet tokens.

Repositories¶

No new git repositories required.

Documentation¶

Documentation will be provided as part of the keystone charm and notes in charm guide.

Security¶

A change of token provider does have security implications and well tested and proved best practices for using the fernet token provider will be implemented.

Testing¶

Unit tests will be developed along with new code. Functional tests will be implemented. A scenario test for change of token provider will also be written.

Dependencies¶

No external dependencies.

Keystone Fernet Token Implementation