https://blueprints.launchpad.net/fuel/+spec/graph-concept-extension
There is introduced a new opportunity that allows to execute graphs for different purposes by the Fuel graph concept extension.
Currently, the Fuel graph concept is tied to the deployment process. For example, we can’t use graphs for provisioning, deletion or verification. Those actions are hardcoded in Nailgun and Astute, and there’s no way to extend extend them easily.
Meantime we want to see every action as a graph in order to make it pluggable and extendable, since end users usually want to somehow change them. For instance, some of them want to use torrent protocol for image delivering instead of HTTP and there’s no way to change it so far.
Another problem is that we can’t verify advanced network configuration in bootstrap mode. The problem lies in our approach where network-checker is responsible only for basic configuration while we need l23network manifest to be applied in order to verify network against real configuration. Having everything in the graphs allows to reuse that puppet manifest, and hence prepare network for verification.
There’re plenty of places where we have hardcoded actions instead of declarative ones. Moving them into graphs will help to clean and simplify our code base, as well as provide opportunity to customize them manually or via plugins.
Transaction Manager
Nailgun should have a general transaction manager for running graphs as well as a bunch of them within a single transaction.
The transaction manager must be used by the new RESTful API endpoint for executing graphs. See REST API section for details.
Default Graphs for Basic Actions
At minimum we want to see the following actions as graphs:
Hence, fuel-library should provide tasks for those graphs the same way they provide them for deployment. The proposed way is to separate them on filesystem (drop into different directories) and sync them one be one by passing additional argument to Fuel CLI. Example:
fuel rel --sync-deployment-tasks --dir /etc/puppet/ --graph provision
Scenarios
Scenarios is the way to run specified graphs one-by-one, each on pre-defined set of nodes. A set of nodes could be specified either explicitly or by using YAQL expression.
Scenarios is a good way to provide a high level orchestration flows such as “Deploy Changes” in declarative manner.
New Astute tasks
In order to support existing scenarios as graphs we need to implement the following tasks in task-based format in Astute:
New method of nodes statuses update
In order to get rid of hardcoded state machine of node statuses, we need to provide a way to set node statuses in a data driven format. Hence, it’s proposed to add a set of callbacks: on_success, on_error and on_stop.
graph_metadata:
on_success:
node_attributes:
status: ready
on_error:
node_attributes:
status: error
error_type: deploy
on_stop: null
Custom graphs management in Fuel UI was described and implemented within the [1], although the ability to execute a sequence of graphs is introduced in this spec as extension.
Working in ‘Custom Scenarios’ deployment mode, user should be able to specify a sequence of space-separated graph types, that he wants to execute.
Also, it is necessary to use a new /api/v1/graphs/execute/ handler (that works with transactions manager) in Fuel UI to run a graph/graphs.
Having everything defined as a graph and mechanism to run few graphs within a single transaction, simple means we can’t rely on task’s name anymore. It makes more sense to distinguish runs by two criteria: graph_type and dry_run. So it’s proposed to extend tasks table with those columns and mark tasks.name as deprecated column.
Transient node statuses shouldn’t be persisted in database. That means nodes::status attribute should contain either discover or provisioned or deployed. Statuses provisioning, deploying and error should be calculated based on node attributes.
When any action is committed the progress should be resetted to 100.
error_type should not be limited to pre-defined set of types.
In order to implement scenarios, we need to design a database schema for new entity. Here’s a proposed solution:
.
SCENARIOS_ACTS
SCENARIOS +--------------------+
+-----------+ | + id (pk) |
| + id (pk) |<------------| + scenario_id (fk) |
| + name | | + order |
+-----------+ | + graph_type |
| + nodes |
+--------------------+
where:
Executing scenarios mean: run its graphs on corresponding set of nodes within a single transaction.
Graphs Execution
Execute passed graphs.
Request:
POST /graphs/execute HTTP/1.1
{
"cluster": <cluster-id>,
"graphs": [
{
"type": "graph-type-1",
"nodes": [1, 2, 3, 4],
"tasks": ["task-a", "task-b"]
},
{
"type": "graph-type-2",
"nodes": [3, 4],
"tasks": ["task-c", "task-d"]
},
],
"dry_run": false,
"force": false
}
where:
Response:
HTTP/1.1 202 Accepted
{
"task_uuid": "transaction-uuid",
...
}
where:
As the graph term was extended, some requests should be modified to avoid misunderstanding. In the following requests the deployment/deploy word should be removed:
Scenarios
Create a new workflow.
Request:
POST /scenarios HTTP/1.1
{
"name": "deploy-changes",
"scenario": [
{
"graph_type": "provision",
"nodes": {
"yaql_exp": "select nodes for provisioning"
}
},
{
"graph_type": "deployment"
"nodes": ...,
}
...
]
}
List available scenarios.
Response:
HTTP/1.1 200 Ok
[
{
"id": 1,
"name": "deploy-changes",
"scenario": [
... scenario's acts ...
]
},
{
"id": 2,
...
}
]
Run a scenarios with a given name. If successful a transaction ID is returned.
Response:
HTTP/1.1 202 Accepted
{
"task_uuid": "transaction uuid"
}
For listing/uploading/downloading will be used the common custom graph commands [0].
The graph execution command should stay practically the same, however it is necessary to be able to define several graph types to run them one by one. Also it should be possible to enforce execution of tasks without skipping and to run only specific tasks ignoring dependancies.
fuel2 graph execute --env 1 [--nodes 1 2 3]
[--graph-types gtype1 gtype2]
[--task-names task1 task2]
[--force]
[--dry-run]
where
None
None for the whole approach.
For the verification tool:
Some API endpoints are renamed so it breaks backward compatibility.
None
None
Ability to:
None
The whole mechanism is more flexible. The provisioning part is configurable and easier to debug. Thanks to the verification graph mechanism, errors detection before the deployment stage may save a lot of time in case of reconfiguration necessity.
None
None
Custom graph management on UI [1].