API to migrate a secret to a secret store

The URL of the associated launchpad blueprint:

https://blueprints.launchpad.net/barbican/+spec/admin-migrate-secret-store

Provide a REST operation that re-encrypts an existing, active secret onto a named secret store and keeps the same secret UUID. Consumers, ACLs, container membership, creator_id, and user metadata are not modified. The secrets row is never deleted.

This is independent of the in-review custom-UUID / disaster-recovery spec (Gerrit change 997056). That spec restores a deleted UUID. This spec moves live key material from one backend to another.

Problem Description

Use case (Deployer / operator)

A user has secrets stored on backend A (for example PKCS11 / store_crypto + p11_crypto) and needs them on backend B (for example KMIP) without changing the secret UUID. Volume and image encryption, Castellan clients, and container secret_refs all persist that UUID. Recreating the secret with a new id is not acceptable.

Today it is not possible to change the backend used to store an existing secret’s payload. Delete-and-recreate is the only workaround, and it is the wrong tool:

  • DELETE soft-deletes the secrets row and children, and plugin.delete_secret may destroy the KMIP/Vault object.

  • ACLs are hard-deleted.

  • Re-POSTing consumers stamps the migrator’s project_id, not Cinder/Glance’s, so later remove_consumer fails.

  • Re-POSTing the same payload generates a new UUID. After DELETE, the original UUID permanently 404s unless restored from soft-delete.

  • Preferred-store only affects new POST /v1/secrets. It does not move existing secrets.

Use case (Self-service to preferred store)

GET /v1/secret-stores is allowed for role:reader. Every project member can list every configured backend, including customer-specific HSMs. A user must not be able to park material on an arbitrary backend they discover from that list.

A common case: a secret was created while store AAAA was the project preferred store; the admin later changes preferred to BBBB. The secret owner should be able to move their secret onto BBBB without a cloud-operator ticket. They must not be able to migrate to CCCC, another store they saw listed but that is not their project’s preferred store.

Setting a project preferred store is already role:admin (this policy predates the Domain Manager persona). A future self-service model could let Domain Managers set preferred backends for projects in their domain; this spec’s policy hooks should not block that.

Actors

  • Cloud / project admin: migrates a secret onto any configured secret store (or scripts a list). Does not need to know which store the secret is on today. If it is already there, the call is a success no-op. May use a system-scoped token to avoid assigning admin on every project that holds secrets to migrate.

  • Secret owner / member: under the default policy, may migrate a secret they can access onto the project’s current preferred secret store only. A request to any other store returns 403.

  • Domain manager (optional deployer policy): custom policy may allow migrate for secrets in their domain (see Policy).

  • External consumer (Cinder, Glance, Castellan): unchanged. The UUID, ACLs, and consumer registrations stay in place.

Proposed Change

Add a microversioned operation on the secret resource that takes a target secret-store identifier, decrypts the payload with the plugin already recorded on the secret, stores it on the named store, then removes the old plugin object only if it is a different object.

The secrets row stays. Identity (UUID, ACLs, consumers, container joins, user metadata, creator_id) is not rewritten.

Scope of this spec

This spec covers:

  • REST API for per-secret migration to a named secret_stores row

  • Returning the current secret-store id on secret metadata GET, computed from the same plugin identity retrieve already uses

  • Returning secret_store_id on GET /v1/secret-stores (list, get-by-id, preferred, and global-default) so clients can migrate without parsing secret_store_ref

  • Policy (secret:migrate_secretstore; admin to any store; members to project preferred only under default)

  • Atomicity and plugin-delete rules so the secret is not lost and a newly stored KMIP/Vault object is not immediately destroyed

  • Idempotency when the secret is already on the target store

  • Microversion, errors, tests, and documentation

Out of scope:

  • Custom UUID / tombstone recovery (change 997056)

  • Making secret payloads generally mutable (no user-facing PUT/PATCH of ciphertext)

  • Bulk “migrate this whole project” API (an admin client can loop)

  • Persisting secret_store_id or backend_id on the secret (see Data model impact). A stored UUID would dangle across reinstalls until multiple instances of the same driver are first class (bug #2164266, bug #2164268).

  • PATCH of secret-store placement (clients must not drive migrate via metadata update)

  • Changing who may list secret stores

  • Orders, CAs, transport keys

  • Barbican-to-Barbican copy across deployments

REST API

The operation is a PUT on the secret, not on /v1/secret-stores. The destination store UUID is in the URL; the body is empty.

PUT /v1/secrets/{secret-id}/secret-store/{secret-store-id}
OpenStack-API-Version: key-manager 1.3

  • Method: PUT

  • Request body: none (empty)

  • Success: 204 No Content (secret href unchanged)

  • Microversion: next after current max (1.2 → 1.3). Requests without the microversion get 404 on this path. The same microversion adds computed secretstore fields on secret metadata GET (below), and adds secret_store_id on secret-store catalogue GET responses.

URL parameters:

  • secret-id — existing secret UUID.

  • secret-store-id — UUID of the destination secret_stores row.

Normal response codes: 204

Error response codes:

  • 400 — multiple secret stores not enabled; secret-store-id is not a UUID; secret has no stored payload (metadata-only); nothing to migrate

  • 401 / 403 — unauthenticated / policy deny; or target store is not the project preferred store and caller is not permitted to migrate to arbitrary backends

  • 404 — secret not found in scope, or secret store id not found (including a UUID-shaped id that does not exist)

  • 503 / plugin errors — decrypt or store failed; secret must remain readable from the original backend

Secret metadata GET (same microversion)

GET /v1/secrets/{secret-id} (JSON metadata) and secret list responses include the store the payload currently lives on, when that store can be resolved. This is not a stored column. It is computed the same way payload retrieve already finds a plugin: match generate_fullname_for(plugin) (and the crypto plugin for store_crypto) to secret_store_metadata['plugin_name'] / kek_data.plugin_name, then map that identity onto the unique secret_stores row (Newton UNIQUE(store_plugin, crypto_plugin)).

New fields, only when the request asks for microversion 1.3 or newer (same gating as consumers in 1.1):

{
  "secret_ref": "https://…/v1/secrets/{secret-id}",
  "secret_store_id": "93869b0f-60eb-4830-adb9-e2f7154a080b",
  "secret_store_ref":
    "https://…/v1/secret-stores/93869b0f-60eb-4830-adb9-e2f7154a080b"
}

  • secret_store_id / secret_store_ref — the live secret_stores row for this secret right now. After migrate, GET shows T. After a DB rebuild that re-syncs the same plugin pair, GET shows the new catalogue UUID, not a stale id.

  • Always include both fields on microversion ≥ 1.3 (clients can rely on the keys being present). Values are JSON null when multiple secret stores are not enabled, or when the secret has no stored payload (metadata-only).

  • Return HTTP 500 when multiple backends are enabled, the secret has a live payload, and no secret_stores row matches the plugin identity (or more than one row matches). That is a server misconfiguration / removed backend, not a client error.

  • Do not add these fields to payload GET (/v1/secrets/{id}/payload).

  • Do not accept them on POST/PUT of the secret. Placement changes only via PUT …/secret-store/{secret-store-id}.

This is derived at read time from existing plugin metadata, not a new attribute on the secrets row. Unlike optional transport_key_id, the keys are always present on ≥ 1.3 so clients do not need “field missing vs null” branching.

Secret-store catalogue responses (same microversion)

Today GET /v1/secret-stores (and get-by-id / preferred / global-default) return secret_store_ref but deliberately omit the store UUID, so clients must parse the href before they can call migrate. Microversion 1.3 also returns secret_store_id on those responses:

{
  "secret_store_id": "93869b0f-60eb-4830-adb9-e2f7154a080b",
  "secret_store_ref":
    "https://…/v1/secret-stores/93869b0f-60eb-4830-adb9-e2f7154a080b",
  "name": "Software Only Crypto",
  "secret_store_plugin": "store_crypto",
  "crypto_plugin": "simple_crypto",
  "global_default": true,
  "status": "ACTIVE"
}

Requests without microversion 1.3 keep the previous response shape (secret_store_ref only). The field name is secret_store_id (Barbican catalogue naming), not a bare id, matching the computed fields on secret metadata GET.

Policy

New rule: secret:migrate_secretstore.

Default check:

{
  "secret:migrate_secretstore": "role:admin or (role:member and secret_store_id:%(target.preferred_secret_store_id)s)"
}

with scope_types=['project', 'domain', 'system']. role:admin may target any configured store (including with a system-scoped token). role:member may migrate only when the URL’s secret-store-id matches the project’s current preferred store; a request to any other store returns 403 after policy evaluation.

Unlike secretstore_preferred:post, this rule is not limited to project scope only. secretstore_preferred:post rejects domain- and system-scoped tokens because POST …/preferred has no request body and must take the project id from the Keystone token. Migrate identifies the secret (and thus its project) from the URL, so system-scoped operators can migrate secrets across projects without holding admin on every project.

Deployers may replace the default with a stricter rule (for example admin-only) or extend it (for example Domain Managers — see below). There is no tighter built-in policy than admin-only; tightening is done by overriding the rule in policy.yaml.

Target dict / request context — include at least:

  • project_id and domain_id for the secret’s project

  • secret_store_id (destination from the URL)

  • preferred_secret_store_id for the secret’s project

  • source_secret_store_id when resolvable from plugin metadata

This enables deployer policies such as:

"secret:migrate_secretstore": "role:admin or (role:manager and domain_id:%(target.secret.domain_id)s)"

which would let Domain Managers migrate secrets in their domain.

Do not gate this on secret:decrypt or secrets:post alone. Decrypt is too broad (ACL readers); post is create, not mutate.

Algorithm

For secret S and target store T (from URL):

  1. Authorise (secret:migrate_secretstore). Load S (active only; 404 if deleted). Reject metadata-only secrets with 400. Load T from secret_stores (404 if missing). If multiple secret stores are disabled, return 400.

  2. If the caller is role:member (not role:admin) and T is not the project’s preferred store, return 403.

  3. If S is already on T’s plugin, return 204. Resolve T to a store/crypto plugin instance and compare generate_fullname_for(plugin) (and crypto plugin identity for store_crypto) to the plugin_name already on S. Do not persist a store id on the secret.

  4. Decrypt the payload using the plugin already on S (get_plugin_retrieve_delete / existing GET-payload path). Failure: abort, no writes.

  5. Snapshot old_meta (plugin name + plugin-specific object ids) before store.

  6. Store the plaintext on the plugin for T. This is not the current store_secret() helper: that helper refuses secrets that already have data. Implementation needs a dedicated “store onto this secret_stores row” path (resolve T → store_plugin / crypto_plugin, then store_secret on that plugin).

  7. Persist new secret_store_metadata / encrypted_data (and any plugin metadata) so subsequent GET uses T. Keep this in the same DB transaction as retiring the old datum rows (soft-delete old encrypted datum / store metadata).

  8. If store or DB commit fails: delete the new plugin object if one was created; leave old_meta as the live pointer. The secret stays available on the original backend.

  9. After commit: plugin.delete_secret(old_meta) only if old_meta names a different plugin object than new_meta. Skip if old_meta is missing, or if it is the same object. store_crypto.delete_secret is a no-op; that is fine.

  10. If step 9 fails: log at ERROR/WARNING. The secret is already available on T. Leaking an orphan KMIP object is better than deleting the only copy.

Do not call plugin.delete_secret on the secrets row. Do not DELETE the secret. Do not change consumers, ACLs, or containers.

Same-plugin trap

A naive “store, then delete using the old plugin” on the same KMIP/Vault plugin stores a new object and then destroys it, because delete_secret uses whatever metadata it is given. Always compare old vs new object identity. PKCS11 via store_crypto hides this today (delete is a no-op); KMIP and Vault will not.

Idempotency and concurrency

  • Already on T: 204, no decrypt required if T’s resolved plugin fullname matches the secret’s existing plugin_name (and crypto plugin for store_crypto).

  • Two concurrent migrates of S: at most one should rewrite metadata; the other should 204 or 409. Implementation can use a row lock on the secret.

  • Secret DELETE during migrate: migrate must 404/409; DELETE must not observe a half-rewritten metadata set.

Containers and consumers

Migrate one secret. An RSA container is three secrets (private, public, optional passphrase); migrate each id separately. Consumer registrations stay; Cinder/Glance keep their original project_id.

Alternatives

  1. DELETE then POST with caller-supplied UUID (change 997056). Rejected for live secrets. Destroys identity, ACLs, and consumer ownership. Correct for DR of a tombstone, not for migration.

  2. User-facing PUT of arbitrary payload to change backend. Rejected. Makes secrets mutable for anyone who can update payload. Migrate to the project preferred store via this subresource is in scope (see Policy).

  3. PUT /v1/secret-stores/{ss-id}/secrets/{secret-id}. Rejected as the primary spelling. Semantically this mutates the secret, not the store catalogue. A store-centric bulk API can be a later addition.

3a. PUT /v1/secrets/{id}/secret-store with store id in the body.

Rejected. Prefer PUT …/secret-store/{secret-store-id} with an empty body so the destination is unambiguous in the URL.

  1. PATCH ``secret_store_id`` on the secret to change placement. Rejected. A writable field on every secret would let clients drive placement without policy checks. Migrate stays a dedicated subresource. Read-only computed fields on GET are in scope (see REST API) so operators and clients can see where a secret lives without a stored foreign key.

  2. Persist ``secret_store_id`` (or ``backend_id``) on the secret for idempotency or GET. Rejected for this spec. secret_stores.id is a random UUID4 assigned at first INSERT of a (store_plugin, crypto_plugin) row, not a stable plugin or HSM identity. Reinstall, or delete-and-re-add a backend from barbican.conf, yields a new UUID. A column on the secret would then dangle. That only becomes a reasonable stored key once multiple instances of the same driver are first class (bug #2164266, bug #2164268). Until then, compute the catalogue UUID on GET from plugin identity. The backend-registry spec may later persist a stable backend_id on metadata; migrate’s no-op check can switch to that in a small follow-up.

  3. Server-side “migrate all secrets in this project”. Deferred. Timeouts, partial failure, and progress reporting need a job API. An admin script calling this endpoint per secret is enough for the first iteration.

Data model impact

None. No new tables, columns, or Alembic migrations. GET does not write secret_store_id onto the secret.

The migrate no-op check and GET store fields both use existing plugin metadata only (see Algorithm). secret_stores.store_plugin is a Stevedore/config name; plugin_name on the secret is a Python class path. The comparison is after resolving a catalogue row to an instance, not string equality of those two columns.

This is sufficient while UNIQUE(store_plugin, crypto_plugin) holds (Newton): two stores cannot share the same plugin pair, so plugin identity is a 1:1 stand-in for the store row. GET therefore returns at most one secret_store_id. If several rows ever share a plugin pair, GET returns HTTP 500 rather than guess (that layout is out of scope until bugs 2164266 / 2164268).

The backend-registry spec (later) will persist a stable backend_id on secret metadata so multiple instances of the same driver can be distinguished. When that lands, migrate should switch the no-op check to backend_id (a small follow-up, not a field added here and discarded). Operators should expect a reconfig when moving to the registry layout.

Security impact

  • Touches secret payload in API-worker memory for the duration of the call, same as GET payload + POST secret. Must not log plaintext.

  • New way to cause Barbican to write to a named backend. Default policy limits members to the project preferred store; admins may target any configured backend.

  • Does not weaken cross-project isolation: the secret must already be in the caller’s authorised scope.

  • Encryption changes with the destination plugin. PKCS#11 via store_crypto uses pKEK/MKEK; KMIP stores cleartext in the appliance; SimpleCrypto uses Fernet in the database. Migrate re-wraps through the API worker; it does not preserve the old on-disk/on-HSM representation.

  • Resource exhaustion: one decrypt + one store per call. No fan-out. Bulk migrate is client-side.

  • Audit: log project, actor, secret id, source plugin identity, target secret_store id at INFO (WARNING if old-object delete fails after commit).

  • GET metadata (1.3) tells any secret:get caller which catalogue store currently holds the payload. That is no stronger than GET /v1/secret-stores (already role:reader) plus knowing the secret exists. It does not allow choosing a backend.

Notifications impact

None required. A future CADF notification for “secret re-wrapped” is optional and can be a follow-up.

Python and command line client impact

Follow-up, not this spec:

  • openstacksdk: method on Secret (for example migrate_secret_store); parse secret_store_id / secret_store_ref on secret GET.

  • OSC: admin command wrapping migrate; show the new GET fields.

python-barbicanclient is deprecated for new work; do not add the primary client there.

Other end user impact

Members who can secret:get see which catalogue secretstore currently holds the payload (microversion ≥ 1.3). Under default policy they may migrate only to the project preferred store. Operators can GET before/after migrate instead of inferring from plugin names.

Performance impact

  • One decrypt and one store per migrated secret. PKCS11 unwrap + KMIP create is HSM/network bound.

  • One extra read of secret_stores and current plugin metadata on migrate, and the same cheap lookup on secret metadata GET when microversion ≥ 1.3 (the secret_stores table is small).

  • No extra work on payload GET. POST create is unchanged.

Other deployer impact

  • Requires enable_multiple_secret_stores = True and a populated secret_stores table. Migrate when the flag is False returns 400. Longer term, consider deprecating enable_multiple_secret_stores = False and phasing out single-store-only deployments (out of scope for this spec).

  • No new config flags required. Optional later: max payload size already enforced by existing limits.

  • Rolling upgrade: older API workers do not expose the route. Microversion 1.3 must be served only by new workers.

  • After migrate, operators may decommission backend A only once they have migrated every secret that still decrypts with A’s plugin.

  • Follow-up (not this spec): a barbican-manage subcommand summarising how many secrets use each backend would help migration planning.

Developer impact

  • plugin.resources.store_secret cannot be reused as-is (_secret_already_has_stored_data). Add a sibling helper, for example rewrap_secret(secret_model, project_model, secret_store).

  • SecretStorePluginManager.get_plugin_store selects preferred / default / plugin class name, not secret_stores.id. Need a resolver from store row → plugin instance (including store_crypto + named crypto plugin). Reuse it for: store onto T, “already on T”, and computed GET fields.

  • Plugin authors: no interface change if store_secret / get_secret / delete_secret already work. The comparison of old vs new metadata is in the resource layer.

Worked example (PKCS11 → KMIP)

GET  /v1/secret-stores                 (admin notes KMIP store id)
GET  /v1/secrets/{secret-uuid}       (secret_store_id is PKCS11)
PUT  /v1/secrets/{secret-uuid}/secret-store/{kmip-ss-id}
GET  /v1/secrets/{secret-uuid}       (secret_store_id is now KMIP)

GET  /v1/secrets/{secret-uuid}/payload  (still works; same UUID)
GET  /v1/secrets/{secret-uuid}/consumers
     (Cinder registration unchanged)

Internal sequence:

decrypt via p11_crypto (ciphertext in DB, MKEK on HSM)
         |
         v
KMIP store_secret  →  new KMIP UUID in secret_store_metadata
         |
         v
commit DB (new metadata live, old encrypted_data soft-deleted)
         |
         v
delete_secret(old_meta)  →  store_crypto no-op; leftover ciphertext
                            already soft-deleted

Implementation

Assignee(s)

Primary assignee:

alee

Other contributors:

None yet.

Work Items

  1. Spec review and barbican implementation.

  2. Microversion 1.3 and API-guide / microversion history.

  3. Policy secret:migrate_secretstore and target dict fields.

  4. Resolver: secret_stores row → store/crypto plugin instance; use it to store onto T, decide “already on T” via plugin_name, and compute GET secret_store_id.

  5. rewrap_secret helper with old-vs-new plugin-delete guard.

  6. Controller PUT /v1/secrets/{id}/secret-store/{ss-id}.

  7. Secret metadata GET/list: always add computed secret_store_id and secret_store_ref for microversion ≥ 1.3 (null when N/A; HTTP 500 when a live payload is unresolvable). No schema change.

  8. Secret-store catalogue GET/list/preferred/global-default: return secret_store_id alongside secret_store_ref for microversion ≥ 1.3 (omit the id on older microversions).

  9. Unit tests: already-on-target; decrypt fail; store fail rolls back; KMIP-style metadata; store_crypto no-op delete; member 403 to non-preferred store; member 204 to preferred; admin to any store; GET returns store id matching T after migrate; GET returns null store fields for metadata-only and when multiple backends are disabled; GET omits fields on microversion 1.2; GET returns 500 when a live payload cannot be resolved; metadata-only migrate returns 400; secret-stores list includes secret_store_id on 1.3 and omits it on older microversions.

  10. Functional tests with two configured stores.

  11. Release note.

  12. Follow-up: SDK/OSC. Follow-up (with backend registry): switch no-op check to backend_id. Follow-up: barbican-manage backend usage summary.

Dependencies

  • Does not depend on custom UUID spec 997056.

  • Does not depend on the backend-registry / admin-backend-API drafts. Those land after this spec. Migrate compares plugin identity until registry stores backend_id on secrets; a small follow-up then switches the no-op check. Operators reconfigure when moving to the registry layout.

Testing

  • Unit: algorithm steps 2–10, policy, validation, concurrent migrate.

  • Functional: two stores (simple-crypto + second plugin, or two store_crypto configs where feasible); migrate; GET payload; GET metadata secret_store_id matches destination; consumers still listed; ACL still enforced; repeat PUT is 204.

  • Negative: member 403 to non-preferred store; member 204 to preferred; admin to arbitrary store; bad store id 404; metadata-only migrate 400 and GET returns null store fields; multiple backends disabled migrate 400 and GET returns null store fields; unresolved live payload GET 500; microversion 1.2 does not expose the migrate route and does not include store fields on GET.

  • Gate: no extra hardware required for a simple-crypto pair if the second store can be a second crypto plugin. KMIP job remains optional extra coverage.

Documentation Impact

  • API reference: new subresource and errors; secret GET/list fields secret_store_id and secret_store_ref.

  • Admin guide: operator procedure (list stores, GET secret to see current store, PUT …/secret-store/{ss-id} per secret, then decommission old backend). Warn that GET payload during migrate is briefly served from the old backend until commit. Document that the id is computed and is not durable across a secret_stores re-sync.

  • doc/source/api/microversion_history.rst — 1.3.

  • Release note.

References