Audit Pipeline¶
https://blueprints.launchpad.net/watcher/+spec/audit-pipeline
This specification proposes the introduction of Audit Pipeline, a new
feature that allows administrators to define an ordered set of strategy stages
that are evaluated against a projected cluster state. The first implementation
will support ONESHOT pipelines using cascade planning and will produce one
optimized action plan.
Problem Description¶
Currently, Watcher supports three audit types: ONESHOT, CONTINUOUS, and EVENT. Each audit executes a single strategy targeting a single goal. However, complex optimization scenarios often require multiple strategies to be executed in sequence, where each subsequent strategy operates on the expected state of the cluster after the previous strategy’s proposed actions are applied.
Cloud operators may need to combine optimization strategies to achieve their operational goals. For example, for maintenance purposes, an operator may need to migrate workloads from specific hosts and then rebalance workloads. To save energy, an operator may need to consolidate workloads, power off unused nodes, and still keep enough capacity available for future workloads.
Use Cases¶
As a cloud operator, I need to perform scheduled maintenance on specific hosts without affecting the running workloads. As a result, I need to migrate those workloads without pessimizing the workload distribution by using workload balancing strategies after the maintenance evacuation. This requires coordinating multiple optimization strategies in sequence to achieve a composite goal. Each strategy needs to operate on a scoped and updated state of the cluster based on the future state created by the previous strategy.
As a cloud operator, I want to save energy by consolidating workloads while also ensuring enough nodes remain online to support new workloads. To achieve this, the workloads first need to be consolidated using server or VM consolidation goals. Then compute nodes can be powered off based on the Saving Energy strategy.
Proposed Change¶
This spec proposes the introduction of an Audit Pipeline API resource,
which models the composition of multiple strategies executed in sequence.
Each stage is evaluated based on the projected outcome of the prior stages.
A mutable cluster data model, and optionally a transparent metric data cache,
will be passed to each stage to allow simulation of the future state.
This functionality will be implemented through a new Audit Pipeline
resource with dedicated APIs. It builds on the existing Audit Template
default_parameters field, which is already available and is used to define
the strategy parameters of each pipeline stage.
The initial implementation is intentionally scoped to ONESHOT audit pipelines using cascade execution (where each stage runs sequentially on the projected state produced by the previous stage). CONTINUOUS and EVENT audit pipelines, composite execution, and recurring pipeline scheduling are out of scope for the first implementation and may be proposed in a future spec.
Audit Pipeline Resource¶
A new Audit Pipeline resource will be introduced. The first implementation
will support ONESHOT pipelines only. A pipeline starts in PENDING state and
is triggered by the decision engine in the same general way as an ONESHOT
Audit. The result of a successful pipeline planning run is one Action Plan.
The Audit Pipeline state machine will reuse the existing Audit states. The initial ONESHOT implementation will expose this reduced transition set:
PENDING -> ONGOING, CANCELLED
ONGOING -> SUCCEEDED, FAILED, CANCELLING
CANCELLING -> CANCELLED
SUCCEEDED -> DELETED
FAILED -> DELETED
CANCELLED -> DELETED
All pipelines start in PENDING state. DELETED is the final soft-delete
state. SUSPENDED remains an Audit state only, and suspending and resuming
pipelines is out of scope until recurring CONTINUOUS or EVENT pipelines are
supported.
CANCELLING is an intermediate cancellation state for pipelines that are
already ONGOING. When a cancel request arrives for an ONGOING pipeline,
the state is set to CANCELLING and the Pipeline Handler detects this
between stages, aborts the remaining stages without interrupting the strategy
that is currently executing, and then sets the pipeline to CANCELLED. A
PENDING pipeline is moved directly to CANCELLED without passing through
CANCELLING because no strategy is executing.
An Audit Pipeline is similar to an Audit, with these differences:
It accepts a list of stage objects. Each stage references one Audit Template. The goal, strategy, scope, and parameters used by the stage are snapshotted from that Audit Template when the pipeline is created.
Unlike an Audit, an Audit Pipeline will not accept strategy
input_parametersdirectly. Strategy parameters must be defined asdefault_parametersin the Audit Template for each stage.It does not directly accept a goal or strategy. Those are defined by the referenced Audit Templates.
It supports only
CASCADEexecution mode in the initial implementation.It supports a minimum of two stages and a hard-coded maximum of ten stages. This limit is not configurable in this first implementation.
The stage object accepted by the create API will have this shape:
{
"name": "maintenance",
"description": "Migrate instances away from compute-1",
"audit_template": "zone-migration-template"
}
name and description are optional user-facing metadata. The
audit_template field may be the UUID or name of an existing Audit Template.
The response representation will also include a stable stage uuid.
Stage-specific input_parameters are not part of the initial API. The stage
object leaves room to add them in a later microversion if a future use case
requires per-pipeline parameter overrides.
Both the pipeline name and each stage name are persisted as non-null
values. When the request omits them, the API generates a default. Pipeline
names are unique among non-deleted pipelines, and stage names are unique
within a pipeline; a collision is rejected at create time.
Cascade Execution Mode¶
Cascade execution runs each stage sequentially. Before each stage, the Pipeline Handler performs two pre-stage steps: it applies the previous stage’s solution to a mutable copy of the cluster data model, and it updates the metric cache with projected values derived from that solution. This ensures each strategy receives an up-to-date context reflecting the expected cluster state after prior stages.
Each stage strategy reads the injected context as if it were the current cluster state. Strategies remain pipeline-agnostic: they should not need to know whether they are being evaluated as part of a normal Audit or an Audit Pipeline. The pipeline implementation is responsible for injecting the scoped CDM used by each stage and for pre-populating the metric cache with projected values before each stage executes.
Mutable Cluster Data Model¶
The Pipeline Handler will manage CDM mutation between stages. In cascade mode, before each stage, a copy of the CDM is updated based on the previous stage’s solution. The lifecycle of the mutable CDM is tied to a single Audit Pipeline execution. It is created at the start of the pipeline workflow and discarded after the final Action Plan is generated. The CDM is not persisted and is not reused by any other pipeline execution.
The CDM simulation layer applies actions from previous stage solutions to
project the expected cluster state. It does not alter anything beyond what
those solutions explicitly define. Preventing later stages from reverting
earlier decisions is the responsibility of the strategies themselves: a
strategy must include in its solution any actions needed to make the projected
state durable. For example, a host-maintenance strategy should emit a
change_nova_service_state action to disable the maintained node; once
that action is reflected in the projected CDM, the next stage will treat the
node as disabled and will not schedule instances back onto it. Without that
action in the solution, the simulated CDM would still show the node as
available, and a subsequent balancing stage could undo the intended
evacuation.
Metric Data Cache¶
The metric data cache is an in-memory data structure scoped to one Audit Pipeline execution. It is not stored in the database and is discarded after the final Action Plan is generated or the pipeline planning run fails.
The cache will be implemented transparently inside the datasource layer without
requiring any changes to existing strategies. The public
statistic_aggregation method, currently implemented independently by each
datasource, will be refactored:
each datasource will rename its current implementation to a private method, and
statistic_aggregation will become a base-class method that implements the
caching logic and delegates to that private method on a cache miss. When a
strategy calls statistic_aggregation, the base implementation will first
check the per-execution cache for a matching result. If a cached or projected
value is present it is returned immediately; otherwise the private datasource
method is called and the result is stored in the cache before being returned.
The cache itself is a MetricDataCache keyed on the tuple of resource id,
meter name, aggregate, period, and granularity. Entries are tagged as either
observed or simulated, which lets handlers drop only the projected values
while retaining real datasource reads.
This approach keeps the strategy interface unchanged and makes caching transparent to both existing and future strategies. The same mechanism applies to normal Audits, where the cache acts as a lightweight per-audit cache.
Strategy Execution Context¶
BaseStrategy is extended to allow the Pipeline Handler to inject
pre-populated cluster data models before executing each stage. Setter methods
are added for compute_model and storage_model so the Pipeline Handler
can supply projected CDMs directly. When a setter has been used, the property
returns the injected model instead of invoking the collector. If no model has
been injected, the existing lazy collector behaviour is preserved.
The Pipeline Handler initialises the compute CDM from the stage-0 strategy on
the first stage, then injects a deep copy into every subsequent stage. After
each stage, the compute CDM is mutated to reflect migrate and
change_nova_service_state actions based on the stage’s solution.
The Pipeline Handler also initialises the storage CDM from the stage-0
strategy on the first stage and injects a deep copy into every subsequent
stage. After each stage, the storage CDM is mutated to reflect move_volume
actions based on the stage’s solution.
Injection for baremetal_model is deferred to a future iteration.
This makes pipeline support transparent to strategies that use the documented
BaseStrategy accessors. Requiring a concrete strategy to opt in to Audit
Pipeline execution is not acceptable. If an existing strategy bypasses those
accessors and talks directly to a collector or service client, that is a
compatibility gap to fix in the framework or strategy implementation, not a
separate pipeline-compatibility mode.
Pipeline Planner¶
The Pipeline Planner will combine all stage solutions into the final Action Plan. It is responsible for:
merging the solutions from all stages into a single one, tracking which stage each action originated from for the duration of the merge;
aggregating the per-stage efficacy indicators into pipeline-level indicators;
detecting conflicts between stage solutions;
collapsing only actions that are explicitly safe to collapse;
removing circular migrations (e.g., A->B followed by B->A, both eliminated);
ordering actions and preserving dependencies in the final Action Plan.
Migration chain optimization must be conservative. For example, an optimization sequence such as A -> B followed by B -> C may be collapsed to A -> C only when no earlier stage constraint is violated. A B -> A move must not be generated or kept if A was removed from the valid candidate set by a host maintenance stage. If the planner cannot prove that a collapse is safe, it must keep the required actions or fail planning with a clear status message.
RPC and Service Routing¶
Starting a pipeline requires RPC coordination with the decision engine. The
API service will add a decision-engine RPC cast method that mirrors the
existing trigger_audit asynchronous pattern while passing the pipeline
UUID instead of an Audit UUID:
def trigger_audit_pipeline(self, context, audit_pipeline_uuid=None):
if not utils.is_uuid_like(audit_pipeline_uuid):
raise exception.InvalidUuidOrName(name=audit_pipeline_uuid)
cctxt = self.conductor_client.prepare(version='1.1')
cctxt.cast(
context, 'trigger_audit_pipeline',
audit_pipeline_uuid=audit_pipeline_uuid)
The decision-engine AuditEndpoint RPC interface is bumped from 1.0 to
1.1 to add trigger_audit_pipeline.
The decision engine will add an Audit Pipeline endpoint that loads the pipeline object, records the worker hostname for the ONGOING run, and executes the Pipeline Handler in the decision-engine worker pool.
Strategy Scopes¶
Each Audit Template may define a scope. In cascade mode, stage scopes are snapshotted into the pipeline stage records when the pipeline is created. The first stage establishes the initial scoped CDM. Later stages are evaluated against the projected CDM after previous stage solutions have been applied. The Pipeline Handler may further restrict later-stage candidate sets based on constraints introduced by earlier stages.
Efficacy Indicators¶
Each stage strategy produces efficacy indicators as part of its solution.
Because different goals use different indicator names for the same concept
(e.g. zone_migration reports planned_live_migrate_instance_count
while consolidation strategies report instance_migrations_count), the
Pipeline Planner normalises the per-stage indicators through a grouping
layer before persisting them.
A PipelineIndicatorGroup declares a set of source indicator names that
are semantically equivalent and sums their values across all stage
solutions into a single combined pipeline-level indicator. The default
groups are:
pipeline_instance_migrations_count— aggregatesinstance_migrations_count,planned_live_migrate_instance_count, andplanned_cold_migrate_instance_count.pipeline_released_compute_nodes_count— aggregatesreleased_compute_nodes_count.pipeline_volume_migrations_count— aggregatesplanned_volume_migrate_count.
The combined indicators are stored as EfficacyIndicator rows linked to
the Action Plan, using the existing data model without requiring new
database columns or per-stage storage.
Additionally, when the pipeline optimiser collapses or removes redundant
migration chains, the number of eliminated migrations is recorded in the
Action Plan’s global_efficacy field as
pipeline_migrations_collapsed_count.
Future Work¶
The initial implementation is limited to ONESHOT cascade pipelines. The following items describe the expected future evolution of the feature, but are out of scope for this spec:
Composite execution mode: group independent stages that do not require CDM or metric simulation between them. This would allow operators to schedule a set of independent audits as one unit, for example workload stabilization scoped separately to multiple availability zones.
CONTINUOUS and EVENT pipeline triggers: extend pipelines beyond ONESHOT execution, including interval handling, webhook or event triggering,
next_run_timemanagement, and interaction with the existing audit scheduler.Stage-specific input parameters: allow per-pipeline overrides for a stage without editing the source Audit Template.
Additional action simulation handlers: expand the set of actions whose projected effects can be represented in the mutable CDM and metric cache.
Architecture Overview¶
The following diagram shows the sequence of processes that happen when an Audit Pipeline using cascade execution is triggered:
+-------------------------------------------------------------------+
| PIPELINE HANDLER |
| Orchestrator and Simulator |
+-------------------------------------------------------------------+
| |
| 1. Set up execution context |
| - Fetch initial scoped compute and storage CDMs |
| - Initialize transparent metric cache |
| |
| 2. Cascade simulation loop, stage 1 to N |
| +---------------------------------------------------------+ |
| | (check for CANCELLING; abort remaining stages if set) | |
| | | |
| | a. Project current world (pre-stage steps) | |
| | - Apply previous solution to compute and storage CDM | |
| | - Update projected metric values in cache | |
| | - Stage 1 applies null updates to both | |
| | | |
| | b. Execute isolated strategy | |
| | +-----------------------------------------------+ | |
| | | WATCHER STRATEGY | | |
| | | - Reads injected CDM as current state | | |
| | | - Calls statistic_aggregation on datasource | | |
| | +----------------------+------------------------+ | |
| | | | |
| | v | |
| | +-----------------------------------------------+ | |
| | | DATASOURCE (statistic_aggregation) | | |
| | | - Hit: return cached or projected metric | | |
| | | - Miss: call private impl and store result | | |
| | +-----------------------------------------------+ | |
| | | |
| | c. Capture solution | |
| | - Store solution for final planning | |
| | - Use solution to drive next stage context | |
| +---------------------------------------------------------+ |
+--------------------------------+----------------------------------+
|
v
+-------------------------------------------------------------------+
| PIPELINE PLANNER |
| |
| - Merge stage solutions |
| - Preserve constraints and dependencies |
| - Optimize only safe action chains |
| - Generate the final Action Plan |
+--------------------------------+----------------------------------+
|
v
+-------------------------------------------------------------------+
| FINAL ACTION PLAN |
| Ready for persistence and execution |
+-------------------------------------------------------------------+
Alternatives¶
An external tool could orchestrate the execution of multiple Audits. This is an alternative to the current manual process of executing strategies one by one. However, this approach still requires each Action Plan to be executed before computing the next strategy solution, so later planning cannot use a projected future state before execution.
Watcher could be extended to support Audit creation as a final Action from a given Action Plan. This still requires a first Action Plan to execute before calculating the next one, and it does not provide a single optimized plan.
For each use case, a new strategy could implement the combination of the required strategy algorithms. This duplicates existing strategy logic and does not provide a reusable composition mechanism.
Data Model Impact¶
New Audit Pipeline Resource
Create a new AuditPipeline database model:
class AuditPipeline(Base):
"""Represents an audit pipeline for multi-strategy execution."""
__tablename__ = 'audit_pipelines'
__table_args__ = (
UniqueConstraint('uuid', name='uniq_audit_pipelines0uuid'),
UniqueConstraint('name', 'deleted',
name='uniq_audit_pipelines0name'),
table_args()
)
id = Column(BigInteger().with_variant(Integer, 'sqlite'),
primary_key=True, autoincrement=True)
uuid = Column(String(36), nullable=False)
name = Column(String(63), nullable=False)
description = Column(String(255), nullable=True)
execution_mode = Column(String(20), nullable=False)
state = Column(String(20), nullable=False)
audit_type = Column(String(20), nullable=False)
auto_trigger = Column(Boolean, nullable=False, default=False)
hostname = Column(String(255), nullable=True)
status_message = Column(String(255), nullable=True)
stages = orm.relationship(
'AuditPipelineStage',
foreign_keys='AuditPipelineStage.audit_pipeline_id',
order_by='AuditPipelineStage.position',
back_populates='audit_pipeline',
lazy=None)
The model intentionally omits interval, next_run_time,
start_time, and end_time because recurring CONTINUOUS and EVENT
pipelines are out of scope for the initial implementation.
Audit Pipeline Stage Resource
Create a new AuditPipelineStage model to store the ordered stage list
as a normalised table:
class AuditPipelineStage(Base):
"""Represents one stage in an audit pipeline."""
__tablename__ = 'audit_pipeline_stages'
__table_args__ = (
UniqueConstraint('uuid', name='uniq_audit_pipeline_stages0uuid'),
UniqueConstraint('name', 'audit_pipeline_id', 'deleted',
name='uniq_audit_pipeline_stages0name'),
UniqueConstraint('position', 'audit_pipeline_id',
name='uniq_audit_pipeline_stages0position'),
table_args()
)
id = Column(BigInteger().with_variant(Integer, 'sqlite'),
primary_key=True, autoincrement=True)
uuid = Column(String(36), nullable=False)
audit_pipeline_id = Column(
BigInteger().with_variant(Integer, 'sqlite'),
ForeignKey('audit_pipelines.id'), nullable=False)
position = Column(Integer, nullable=False)
name = Column(String(63), nullable=False)
description = Column(String(255), nullable=True)
goal_id = Column(Integer, ForeignKey('goals.id'), nullable=False)
strategy_id = Column(
Integer, ForeignKey('strategies.id'), nullable=True)
scope = Column(JSONEncodedList, nullable=True)
parameters = Column(JSONEncodedDict, nullable=True)
audit_pipeline = orm.relationship(
AuditPipeline, foreign_keys=audit_pipeline_id,
back_populates='stages', lazy=None)
goal = orm.relationship(Goal, foreign_keys=goal_id, lazy=None)
strategy = orm.relationship(
Strategy, foreign_keys=strategy_id, lazy=None)
The stage stores a snapshot of the referenced Audit Template’s goal, strategy, scope, and default parameters at pipeline creation time. Later edits or deletes of the Audit Template do not change existing pipelines. The source template reference is intentionally not stored because the Audit Template may be updated or deleted independently after the pipeline is created.
Action Plan Changes
Update the existing ActionPlan model to make audit_id and
strategy_id nullable and add a foreign key to AuditPipeline:
class ActionPlan(Base):
...
audit_id = Column(Integer, ForeignKey('audits.id'), nullable=True)
strategy_id = Column(
Integer, ForeignKey('strategies.id'), nullable=True)
audit_pipeline_id = Column(
BigInteger().with_variant(Integer, 'sqlite'),
ForeignKey('audit_pipelines.id'), nullable=True)
audit_pipeline = orm.relationship(
'AuditPipeline', foreign_keys=audit_pipeline_id, lazy=None)
For normal Audit-created Action Plans, audit_id and strategy_id remain
populated. This feature does not deprecate strategy_id for normal Audits.
The ActionPlan object version is bumped to 2.4: audit_id and
strategy_id become nullable, and the audit_pipeline_id field plus the
audit_pipeline object field are added.
The Action model is left unchanged. Stage provenance is tracked in memory
while the Pipeline Planner merges stage solutions, but it is not persisted per
action.
The object save path will enforce that exactly one of audit_id or
audit_pipeline_id is set. This invariant is enforced in the object layer,
which raises an exception when neither or both are set. When audit_id
is set, strategy_id must also be set.
When listing Action Plans in an older microversion, pipeline-created Action
Plans are excluded (the list is filtered on audit_pipeline_id being null)
because their audit_id and strategy_id fields cannot be represented by
the older response contract, and the audit_pipeline_uuid field is unset in
the response. In the new microversion, Action Plan responses include
audit_pipeline_uuid for pipeline-created Action Plans, with audit_uuid
and strategy_uuid null, and the list API accepts an
audit_pipeline_uuid filter.
The database migration adds the nullable columns and new tables in a
backwards-compatible way, and relaxes action_plans.audit_id and
action_plans.strategy_id to nullable. No data migration is required for
existing Audits, Action Plans, or Actions.
REST API Impact¶
Audit Pipeline - New Resource APIs
All Audit Pipeline endpoints are gated behind API microversion 1.8.
Clients negotiating an older microversion get 404 Not Found for every
/v1/audit_pipelines route.
Create Audit Pipeline:
Method: POST
URL:
/v1/audit_pipelinesNormal response: 201 Created
Error responses: 400 Bad Request, 404 Not Found
Request body:
{
"name": "maintenance-pipeline",
"description": "Maintenance and rebalance pipeline",
"audit_type": "ONESHOT",
"execution_mode": "CASCADE",
"auto_trigger": false,
"stages": [
{
"name": "maintenance",
"description": "Move workloads from compute-1",
"audit_template": "zone-migration-template"
},
{
"name": "rebalance",
"audit_template": "workload-balance-template"
}
]
}
The create schema requires audit_type, execution_mode, and stages.
audit_type only accepts ONESHOT in the initial microversion.
execution_mode only accepts CASCADE. stages must contain at least
two and no more than ten entries. name, description, and
auto_trigger are optional, and auto_trigger defaults to false.
Stage audit_template is required. Stage name and description are
optional. Additional properties are rejected at the pipeline and stage levels.
When name is omitted at either level, the API generates one. A
pipeline name that collides with an existing non-deleted pipeline is rejected,
and duplicate stage names within one request are rejected with
400 Bad Request.
ONESHOT pipelines are triggered automatically by the API at creation time,
mirroring the behavior of ONESHOT Audits. The auto_trigger field controls
whether the resulting Action Plan is automatically launched after the pipeline
execution succeeds. When auto_trigger is false (the default), the
Action Plan is created in RECOMMENDED state and the operator must launch
it manually.
Response body:
{
"uuid": "pipeline-uuid",
"name": "maintenance-pipeline",
"description": "Maintenance and rebalance pipeline",
"audit_type": "ONESHOT",
"execution_mode": "CASCADE",
"state": "PENDING",
"auto_trigger": false,
"hostname": null,
"status_message": null,
"stage_uuids": ["stage-uuid-1", "stage-uuid-2"],
"stages": [
{
"uuid": "stage-uuid-1",
"position": 0,
"name": "maintenance",
"description": "Move workloads from compute-1",
"audit_pipeline_uuid": "pipeline-uuid",
"goal_uuid": "goal-uuid-1",
"goal_name": "hardware_maintenance",
"strategy_uuid": "strategy-uuid-1",
"strategy_name": "zone_migration",
"scope": [],
"parameters": {"compute_nodes": [{"src_node": "compute-1"}]},
"created_at": "2026-07-14T10:00:00+00:00",
"updated_at": null,
"deleted_at": null
},
{
"uuid": "stage-uuid-2",
"position": 1,
"name": "rebalance",
"description": null,
"audit_pipeline_uuid": "pipeline-uuid",
"goal_uuid": "goal-uuid-2",
"goal_name": "workload_balancing",
"strategy_uuid": "strategy-uuid-2",
"strategy_name": "workload_balance",
"scope": [],
"parameters": {},
"created_at": "2026-07-14T10:00:00+00:00",
"updated_at": null,
"deleted_at": null
}
],
"created_at": "2026-07-14T10:00:00+00:00",
"updated_at": null,
"deleted_at": null,
"links": []
}
stage_uuids is always returned and gives the ordered list of stage UUIDs.
The expanded stages list is returned by the create, show, and detail views;
it is omitted from the summary list view.
The API will reject unsupported audit_type values, unsupported
execution_mode values, fewer than two stages, more than ten stages, and
duplicate or over-long names with 400 Bad Request. A missing Audit Template
returns 404 Not Found. If stage persistence fails after the pipeline row is
created, the pipeline is destroyed so no orphaned rows remain.
List Audit Pipelines:
Method: GET
URL:
/v1/audit_pipelinesNormal response: 200 OK
The list API supports the common pagination and sorting query parameters used
by other Watcher collection APIs (marker, limit, sort_key,
sort_dir), plus filtering by state, audit_type, and
execution_mode. The summary view returns stage_uuids but not the
expanded stages list.
List Audit Pipelines Detailed:
Method: GET
URL:
/v1/audit_pipelines/detailNormal response: 200 OK
Same query parameters as the list API, with the expanded stages list
included for every pipeline, following the existing Watcher detail
collection pattern.
Get Audit Pipeline:
Method: GET
URL:
/v1/audit_pipelines/{audit_pipeline}Normal response: 200 OK
Error responses: 404 Not Found
The response body has the same shape as the create response. As with other
Watcher resources, {audit_pipeline} may be either the UUID or the name of
the pipeline; this applies to the show, update, cancel, and delete endpoints.
Update Audit Pipeline:
Method: PUT
URL:
/v1/audit_pipelines/{audit_pipeline}Normal response: 200 OK
Error responses: 400 Bad Request, 404 Not Found, 409 Conflict
PUT updates the mutable user fields. The mutable pipeline fields are name,
description, and auto_trigger; all are optional in the request body
and only the supplied fields are changed. name and description may be
updated in any pipeline state. auto_trigger may only be updated while the
pipeline is in PENDING state; supplying it in any other state returns
409 Conflict. All other fields, including stage fields, are immutable.
Stage name and description updates are not supported in the initial
implementation. Renaming to a name already used by another non-deleted pipeline
is rejected.
Cancel Audit Pipeline:
Method: POST
URL:
/v1/audit_pipelines/{audit_pipeline}/cancelNormal response: 202 Accepted
Error responses: 404 Not Found, 409 Conflict
Cancel is valid for PENDING and ONGOING pipelines and returns the
updated pipeline representation. A PENDING pipeline is moved directly to
CANCELLED. An ONGOING pipeline is moved to CANCELLING; the
Pipeline Handler detects that state between stages and before persisting the
final Action Plan, aborts the remaining stages without interrupting the
currently executing strategy, and then sets the pipeline to CANCELLED.
Cancelling a pipeline already in the terminal SUCCEEDED, FAILED, or
intermediate CANCELLING state returns 409 Conflict.
Suspend and resume actions are intentionally not part of the initial ONESHOT pipeline API.
Delete Audit Pipeline:
Method: DELETE
URL:
/v1/audit_pipelines/{audit_pipeline}Normal response: 204 No Content
Error responses: 404 Not Found, 409 Conflict
Delete is allowed from SUCCEEDED, FAILED, and CANCELLED, and
performs a soft delete. Pipelines in PENDING or ONGOING state cannot be
deleted and return 409 Conflict; they must be cancelled first.
Audit Pipeline is introduced in API microversion 1.8.
Security Impact¶
The security posture remains the same as a normal Audit. New policy rules are
added for Audit Pipeline API endpoints and follow the same pattern as the Audit
API, requiring administrator privileges (RULE_ADMIN_API).
Notifications Impact¶
Watcher currently sends versioned notifications for Audit lifecycle operations. Audit Pipeline should mirror that behavior.
A new watcher.notifications.audit_pipeline module will define payloads and
notifications equivalent to the Audit notifications:
class AuditPipelineStagePayload(NotificationPayloadBase):
VERSION = '1.0'
fields = {
'uuid': wfields.UUIDField(),
'position': ovo_fields.IntegerField(),
'name': ovo_fields.StringField(nullable=True),
'goal_uuid': wfields.UUIDField(),
'strategy_uuid': wfields.UUIDField(nullable=True),
}
class TerseAuditPipelinePayload(NotificationPayloadBase):
VERSION = '1.0'
fields = {
'uuid': wfields.UUIDField(),
'name': ovo_fields.StringField(nullable=True),
'audit_type': ovo_fields.StringField(),
'execution_mode': ovo_fields.StringField(),
'state': ovo_fields.StringField(),
'auto_trigger': ovo_fields.BooleanField(),
'hostname': ovo_fields.StringField(nullable=True),
'status_message': ovo_fields.StringField(nullable=True),
'created_at': ovo_fields.DateTimeField(nullable=True),
'updated_at': ovo_fields.DateTimeField(nullable=True),
'deleted_at': ovo_fields.DateTimeField(nullable=True),
}
class AuditPipelinePayload(TerseAuditPipelinePayload):
VERSION = '1.0'
fields = {
'stages': ovo_fields.ListOfObjectsField(
'AuditPipelineStagePayload')}
class AuditPipelineStateUpdatePayload(NotificationPayloadBase):
VERSION = '1.0'
fields = {
'old_state': ovo_fields.StringField(nullable=True),
'state': ovo_fields.StringField(nullable=True),
'status_message': ovo_fields.StringField(nullable=True),
}
class AuditPipelineCreatePayload(AuditPipelinePayload):
VERSION = '1.0'
fields = {}
class AuditPipelineUpdatePayload(TerseAuditPipelinePayload):
VERSION = '1.0'
fields = {
'state_update': ovo_fields.ObjectField(
'AuditPipelineStateUpdatePayload')}
class AuditPipelineActionPayload(TerseAuditPipelinePayload):
VERSION = '1.0'
fields = {'fault': ovo_fields.ObjectField(
'ExceptionPayload', nullable=True)}
class AuditPipelineDeletePayload(AuditPipelinePayload):
VERSION = '1.0'
fields = {}
The stage list is carried only by the create and delete payloads, which
describe the full pipeline. Update and state-transition events derive from
TerseAuditPipelinePayload and omit the stages, because stages are immutable
after creation and would only repeat data the consumer already has.
The corresponding notification wrappers are
AuditPipelineCreateNotification, AuditPipelineUpdateNotification,
AuditPipelineDeleteNotification, and AuditPipelineActionNotification.
The action notification covers the start, end, and error phases of
a pipeline execution.
The Action Plan notification payloads are bumped to version 1.3:
audit_uuid and strategy_uuid become nullable, audit_pipeline_uuid
is added to TerseActionPlanPayload, and a nested audit_pipeline object
field is added to ActionPlanPayload. For pipeline-created Action Plans the
nested Audit payload is absent and the nested Audit Pipeline payload is
populated; for Audit-created Action Plans the reverse holds.
Other End User Impact¶
python-watcherclient will need updates for the new Audit Pipeline API resource.
Performance Impact¶
Cascade planning adds computational overhead because multiple strategies are evaluated during one planning run.
The metric data cache reduces redundant datasource API calls by caching metrics retrieved during strategy execution.
The cache increases memory usage during the pipeline planning run. The cache is bounded by a single pipeline execution and is discarded when planning completes or fails.
Other Deployer Impact¶
None
Developer Impact¶
None
Upgrade Impact¶
None
Implementation¶
Assignee(s)¶
- Primary assignee:
dviroel
Work Items¶
Phase 1: Metric Data Cache
Refactor
statistic_aggregationin each datasource to a private method.Implement the caching logic in the base-class
statistic_aggregation.Ensure caching happens transparently when strategies call datasource methods.
Make the cache usable by normal Audits and Audit Pipelines.
Add unit tests and documentation with the implementation changes.
Phase 2: Audit Pipeline Database Changes
Create the Audit Pipeline database model.
Create the Audit Pipeline Stage database model.
Update Action Plan to reference either an Audit or an Audit Pipeline.
Add backwards-compatible database migrations for the new tables and nullable columns.
Add object and database API tests for the new models and invariants.
Phase 3: Object and RPC Changes
Add Audit Pipeline and Audit Pipeline Stage objects.
Add state transition handling for Audit Pipeline.
Add
trigger_audit_pipelineRPC method to the decision-engine API.Add a decision-engine messaging endpoint for Audit Pipeline operations.
Record the decision-engine hostname that owns an ONGOING pipeline run.
Add notifications for Audit Pipeline lifecycle and action phases.
Extend Action Plan notifications for pipeline-created Action Plans.
Add unit tests with each object and RPC change.
Phase 4: Pipeline Handler and Planner
Implement the Pipeline Handler for orchestrating cascade stage execution.
Implement mutable CDM updates and metric cache projection between stages.
Inject scoped CDM into strategy execution via
BaseStrategysetter methods.Implement the Pipeline Planner for conservative action consolidation and conflict detection.
Implement cancellation checkpoints between pipeline stages.
Track stage origin while merging actions and aggregate per-stage efficacy indicators into pipeline-level indicators.
Add unit tests and developer documentation with the implementation changes.
Phase 5: Pipeline API Changes and Integration Tests
Implement Audit Pipeline POST, GET, detail, PUT, DELETE, and cancel endpoints behind API microversion 1.8.
Add policy rules for every Audit Pipeline endpoint.
Expose
audit_pipeline_uuidon Action Plan responses and as a list filter, and hide pipeline-created Action Plans from older microversions.Add API sample tests for request and response formats.
Add integration or functional tests for the end-to-end ONESHOT cascade flow.
Update API reference and user documentation.
Phase 6: Client Support
Update python-watcherclient with Audit Pipeline commands.
Add client-side tests and documentation.
Dependencies¶
No new external dependencies required
Testing¶
Unit tests will be added with each implementation phase. API sample tests will cover all new request and response formats, including Audit Pipeline create, show, list, update, delete, and cancel endpoints.
Functional or integration tests will cover a full ONESHOT cascade pipeline flow that creates a pipeline, triggers planning, and verifies that one Action Plan is created. Tempest scenario coverage should be added for the full user-visible workflow when the feature is available end to end.
Documentation Impact¶
User Documentation
New section in the User Guide: “Audit Pipeline”.
Audit Pipeline lifecycle and state transitions.
Cascade execution examples for strategy combinations.
Known limitations section describing which action types are fully simulated by the CDM mutation layer and which are carried as constraints.
API Reference
Document the Audit Pipeline resource and APIs.
Include request and response examples for all new API methods.
Developer Documentation
Describe the Pipeline Handler and Pipeline Planner design.
Describe transparent metric cache and datasource refactor.
Describe mutable CDM simulation and supported action types.
Explain that strategies should remain pipeline-agnostic.
References¶
History¶
Release Name |
Description |
|---|---|
2026.2 |
Introduced |
2027.1 |
Re-proposed. Dropped the Audit Template |