Scalable VIM Monitor using Mistral¶
https://blueprints.launchpad.net/tacker/+spec/refactor-vim-monitor
Problem description¶
currently, a thread will be started by tacker server once a vim is registered, which is not scalable:
if we have a lot of vims, the tacker server will have to run a lot of threads to monitor them, which will impact the API function of tacker server
if the tacker server restarts, the monitor thread will not start
cannot run more than one tacker server
Proposed change¶
asciiflow:
+---------------------+
| Mistral workflow |
+-------> VIM monitor action |
+--------------+ | | |
| Tacker server+-----+ +------------------+--+
+-----+--------+ |
| |
| |
| +-------v----------+
| | |
| | Conductor server |
| +------------+ +------+ |
| | | | +------------------+
+-------> Tacker DB <-----+
| |
+------------+
Since Mistral is an integral part of tacker system, a long-live Mistral workflow action can be used to do this kind of task.
Tacker server will generate a VIM reachability test workflow and execute it if a new vim is registered. The workflow and execution will be removed once the vim is de-registered from tacker server.
Vim monitor actions cannot access tacker database directly, so we introduce a conductor server to do database access for the mistral actions.
Mistral does not stop long-live running action even if the workflow execution is deleted. So a mechanism is devised for action to exit. Every 1 minute, action will contact the conductor to see if it is the wanted action class, if it is not the one, the action should exit the loop and exit. To deal with scalability of lots of VIM, load balancer can be placed before conductor(s).
Mistral action will use RPC to communicate with conductor server. Ping action communicates with the conductor via the following interface VimMonitor:
class VimMonitorAction2ConductorRPC(object)
# update vim's status, action_id is the task action
def update_vim (vim_id, status, action_id):
# The response message containing the operation result
# the status should be the passed in by request or 'bad_action'
# if the action is not the wanted action.
status = rpc call conductor
return status
sequence diagram for register vim:
sequence diagram for de-register vim:
sequence diagram for update vim with auth url change:
Alternatives¶
Another way to use mistral is to use a loop workflow:
asciiflow:
start_task -----> ping_task ------> update_task
^ |
| |
| |
| on_succes |
+---------------+
But Mistral will save task executions into Mistral database, so the loop workflow will populate mistral db with thousands of ping_task and update_task records for each VIM, which will impact Mistral DB.
Data model impact¶
VIM database will be extended to contain a Mistral action id column to record the current action id.
REST API impact¶
None
Security impact¶
None
Notifications impact¶
None
Other end user impact¶
None
Performance Impact¶
None
Other developer impact¶
A new RPC server will be started, and Load balancer can be used for more than one tacker conductor deployment.
And this will help to deploy more than one tacker servers
Developer impact¶
None
Implementation¶
Assignee(s)¶
- Primary assignee:
gongysh
- Other contributors:
<launchpad-id or None>
Work Items¶
refactor work flow codes in tacker server
implement workflow version of vim reachability monitor
Unit Tests
Dependencies¶
rabbitmq
oslo message
Testing¶
this feature can be tested by the following steps:
setup up tacker system which will start tacker conductor servers
register a vim
check the vim monitor workflow is setup and there is a ping action is running on mistral executor component
de-register the vim to check if the mistral action will exit
Documentation Impact¶
change tacker deployment document
add a document about mistral workflow way to do actions in tacker server
