Create and run workflows within a namespace¶

Launchpad blueprint:

https://blueprints.launchpad.net/mistral/+spec/create-and-run-workflows-within-a-namespace

Creating and running workflows within a namespace will allow users to create many workflows with the same name. This is useful when a user already has many workflows that are connected to each other implemented and one of the workflow names is already in use and the user does not want to edit that workflow and all the ones referencing it or merge them to a workbook. This is possible because the namespace is not a part of the Mistral language.

Problem description¶

When a workflow name is already in use, it takes editing the workflow the user wants to add to mistral or delete the existing one.

Use Cases¶

When there are many users writing workflows in the same tenant simultaneously each can use a different namespace, so there will be no clashes preventing any of the users to add the workflow he created to mistral.
If a workflow definition is not allowed to be modified, but a workflow with the same name already exists in Mistral and the user wants to upload the workflow and create an execution from it.

Proposed change¶

Add a new namespace parameter to both workflow definition creation, workflow definition deletion, workflow execution creation and other relevant APIs.

If a workflow definition creation request has a namespace specified in it, the workflow name should be unique only within the namespace. If no namespace is passed, the name should be unique within the group of workflow definitions without a namespace (will be referred as the default namespace from here on).

If a workflow execution request from the user, has a namespace specified in it, mistral will search for the workflow definition within the namespace only. However, if the workflow execution came from the mistral engine (e.g sub-workflow execution), mistral will first try finding the workflow definition within the namespace, and only if one does not exist, mistral will try finding it the default namespace. By default the namespace passes to the sub-workflows execution. If a workflow definition only exists in another namespace that is not the default namespace, it will not be found, and the create workflow execution request will fail.

In the future we might want to add a namespace for actions and workbooks (but for now workbooks and workflows created from them, are always in the default namespace).

An example to explain how the namespace moves recursively to sub-workflow executions.

Given there are 3 workflows in the workflow table. Two workflows definitions called ‘wf’ and ‘sub_sub_workflow’are in the same namespace - a namespace we will call ‘abc’, and one workflow definition called ‘sub_workflow’ in the default namespace.

Visualization of a partial workflow definitions table:

ID

name

namespace

1

wf

abc

2

sub_wf

3

sub_sub_wf

abc

4

sub_sub_wf

Workflow definition for ‘wf’ with ID 1:

---
version: '2.0'
wf:
  tasks:
    t1:
      workflow: sub_wf

Workflow definition for ‘sub_wf’ with ID 2:

---
version: '2.0'
sub_wf:
  tasks:
    t2:
      workflow: sub_sub_wf

Workflow definition for ‘sub_sub_wf’ with ID 3:

---
version: '2.0'
sub_sub_wf:
  tasks:
    t3:
      action: std.noop

Workflow definition for ‘sub_sub_wf’ with ID 4:

---
version: '2.0'
sub_sub_wf:
  tasks:
    should_not_run:
      action: std.fail

As you notice, namespace is not and should never be a part of the language.

By calling the execution of workflow with name ‘wf’ within namespace ‘abc’, it is required for workflow with name ‘wf’ in namespace ‘abc’ to run, and when task t1 is executed to call workflow ‘sub_wf’ within the default namespace (since no workflow with name ‘sub_wf’ exist within namespace ‘abc’), but still remember the namespace is ‘abc’ so that when task t2 will be executed, the workflow that will be executed is workflow ‘sub_sub_wf’ in namespace ‘abc’ with ID ‘3’, rather than workflow ‘sub_sub_wf’ in the default namespace with ID ‘4’. The execution described above should result in success.

More strictly speaking, when it comes to calling nested workflows the namespace of the top most workflow is propagated down to its children. So that when Mistral needs to resolve a workflow name, it first searches the configured name in that propagated namespace, and if it doesn’t exist there, Mistral will try to find it in the default namespace.

A workflow execution can only trigger an execution of a workflow within both the same tenant and namespace or within both the same tenant and the default namespace.

For workbooks that means that all workflows within the workbook could only call workflows in the default namespace.

Leading suggestion for the creation API of the ‘wf’ execution is this:

POST /v2/executions
{
  "workflow_name": "wf",
  "workflow_namespace": "abc"
}

Leading suggestion for passing the namespace from execution to sub-execution recursively is putting it in the params of the execution possible under env. A user is not allowed to add any key that starts with two underscores to the env.

Example of how such row might look like in the database:

mysql> select * from workflow_executions_v2 where id='3'\G;
*************************** 1. row ***************************
          created_at: 2017-06-19 10:59:29
          updated_at: 2017-06-19 10:59:30
               scope: private
          project_id: 1
                  id: 3
                name: sub_sub_wf
         description:
       workflow_name: sub_sub_wf
  workflow_namespace: abc
         workflow_id: 3
                spec: {"tasks": {"t3": {"action": "std.noop", "version": "2.0", "type": "direct", "name": "t3"}}, "name": "sub_sub_wf", "version": "2.0"}
               state: SUCCESS
          state_info: NULL
                tags: NULL
     runtime_context: {"index": 0}
            accepted: 1
               input: {}
              output: {}
              params: {"env": {"__namespace": "abc"}}

Notice the last line where under params->env we have a key called ‘__namespace’

In the example described above, if a user decides to add a workflow with the name ‘sub_wf’ to the ‘abc’ namespace, the next time the workflow will be executed, the new workflow called ‘sub_wf’ from the ‘abc’ namespace will be triggered by the workflow with the name ‘wf’ from the ‘abc’ namespace, instead of the workflow ‘sub_wf’ from the default namespace.

Regarding the results of the current APIs see the next examples that all assume that the workflows described in the next table are the only one that exist.

Table:

ID	name	namespace
1	wf	abc
2	sub_wf
3	sub_sub_wf	abc
4	sub_sub_wf
5	example_wf	example_1
6	example_wf	example_a

Examples:

GET /v2/workflows Will return all 6 workflows

GET /v2/workflows/wf Will return an error “workflow not found [workflow_identifier=wf]

GET /v2/workflows/sub_wf Will return workflow ‘sub_wf’ from the default namespace (ID=2).

GET /v2/workflows/sub_sub_wf Will return workflow ‘sub_sub_wf’ from the default namespace (ID=4).

GET /v2/workflows/example_wf Will return an error “workflow not found [workflow_identifier=example_wf]

DELETE /v2/workflows/wf Will throw an exception, because no namespace supplied and no such workflow exist in the default namespace

DELETE /v2/workflows/sub_wf Will delete the workflow with the name ‘sub_wf’ from the default namespace (ID=2). This should be allowed in order to let users that don’t use namespaces to work as they are used to.

DELETE /v2/workflows/sub_sub_wf Will delete the workflow with the name ‘sub_sub_wf’ from the default namespace (ID=4).

DELETE /v2/workflows/example_wf Will return an error “workflow not found [workflow_identifier=example_wf]

PUT will have similar results to DELETE

Alternatives¶

We can try and use workbooks, but the down side is it forces the user to merge his workflows, and might result in a hugh file, that a user might find to be hard to edit and read.

For the described namespace design, we can use different names. For example in the create execution API we can call the new key ‘workflow_namespace’ instead of ‘namespace’. Also the default namespace currently described is the empty string (‘’), but it can be something like “<default-namespace>”. We should also consider saving some namespaces to future system use (for example namespaces that starts with 2 underscores ‘__’)

Data model impact¶

The proposed change must come with a change to the data model.

For workflow definition, a namespace should be added to the model and the DB workflow_definitions_v2 table. And the same for workflow execution, plus it should also be under env in params, so it will seep easily to the sub-workflow executions. In the case of workflow execution there is also the option of just adding it to the env under params.

In the future we might create a namespace table. Migration from current suggested model to one that includes a separate table for namespace, should be easy using SQLAlchemy.

REST API impact¶

Optional namespace parameter will be added to relevant requests:

create workflow definition within a namespace:
POST /v2/workflows?namespace=NAMESPACE
RAW_WF_DEFINITION
delete workflow definition within a namespace:
DELETE /v2/workflows/WORKFLOW_IDENTIFIER?namespace=NAMESPACE
get a workflow definition within a namespace:
GET /v2/workflows/WORKFLOW_IDENTIFIER?namespace=NAMESPACE
get all the workflow definitions within a given namespace:
GET /v2/workflows?namespace=NAMESPACE
update a workflow definition within a given namespace:
PUT /v2/workflows?namespace=NAMESPACE
RAW_WF_DEFINITION
create an execution of a workflow where the workflow belongs to given:
POST /v2/executions
{
  "workflow_name": "WORKFLOW_NAME",
  "workflow_namespace": "NAMESPACE"
}
get a list of all the namespaces:
GET /v2/namespaces

End user impact¶

The new namespace request parameter should be added to the python-mistralclient as well.

Performance Impact¶

None.

Deployer impact¶

Database migration should be done when upgrading mistral to a version that includes this change.

Implementation¶

Assignee(s)¶

Primary assignee:: michal-gershenzon
Other contributors:: melisha

Work Items¶

Adding namespace parameter to create workflow definition, delete workflow definition and create workflow execution requests.
Change the way workflow definition are queried during execution.
Tests
Database migration script
Documentation
Add new parameter to python-mistralclient

Nice to have work items:

Adding namespace as a filter parameter of get workflow definition
Adding namespace as a filter parameter of update workflow definition
Adding namespace as a filter parameter of get workflow executions
Supporting the namespace feature with workbooks
Adding namespaces API endpoint

Dependencies¶

None.

Testing¶

Create a workflow under some namespace that already exist in the default namespace.
Create a workflow under the default namespace that calls the workflow above. Run it once under the default namespace and once under the namespace from previous section and see each time the expected sub-workflow execution is created.
Create a workflow under some namespace that does not exist in the default namespace and see trying to execute it without specifying a namespace fails.

References¶

None.

Create and run workflows within a namespace

Create and run workflows within a namespace¶

Problem description¶

Use Cases¶

Proposed change¶

Alternatives¶

Data model impact¶

REST API impact¶

End user impact¶

Performance Impact¶

Deployer impact¶

Implementation¶

Assignee(s)¶

Work Items¶

Dependencies¶

Testing¶

References¶

Mistral Specs 0.0.1.dev81

Page Contents