Support Watcher HA active-active Mode¶
Watcher Decision Engine and Applier currently don’t support HA active-active mode.
There is often more than one controller in the real environment. Watcher Decision Engine and Applier are deployed on the controller. Now there is only one active Watcher-decision-engine or Applier. Otherwise Watcher Decision Engines will make duplicate actionplans for CONTINUOUS audit. And we don’t know which Applier is doing the actionplan. Another problem is how to sync Data Models of Decision Engines to each other. Now Decision Engine updates its Data Model based on notifications from Nova. If there are many Decision Engines, we need to find a way to broadcast notifications. One solution don’t depend on notifications is to update CDM before each audit.
As an operator, I want to enable all Watcher-decision-engine and Applier on all controllers.
If we enable more than one Watcher Decision Engine, the question is how to know the relation between Audit and Watcher Decision Engine. We can add a new ‘host’ field to the Audit table to solve this question. When we update the audit state from PENDING to ONGOING, we set the host field to record the Watcher Decision Engine hostname.
So, the changes will include:
Watcher Decision Engine CDM should consume notifications from Nova in broadcast mode to make CDMs synced.
Add a new ‘host’ field to the audit table
Add a new ‘host’ field to the actionplan table
Record the host when updating the audit’s state from PENDING to ONGOING
Record the host when updating the actionplan’s state from PENDING to ONGOING
For CONTINUOUS audit, we need to check the host that running decision engine and recorded host value in audit table. If they are different, just skip the audit.
When starting Applier process, if there are ONGOING actionplan with the same host, cancel these stale actionplans. Actions of these stale actionplans should also be marked as CANCELLED.
Action Plan includes different resources which are represented in Actions. If user wants to run more than one action plan at the same time, Watcher should check whether new action plan’s resources overlaps with already running one or not. If so, Watcher should prevent running of new Action Plan by raising appropriate error.
Data model impact¶
Add a new ‘host’ filed in the audit table
Add a new ‘host’ filed in the actionplan table
REST API impact¶
Add ‘host’ field to AuditPayload, ActionPlanPayload
Other end user impact¶
Other deployer impact¶
- Primary assignee:
- Other contributors:
Add broadcast notifications for DE
Update database to add a new field ‘host’
Record host when changing state from PENDING to ONGOING
Add checking host for CONTINUOUS audit
Check stale actonplans when starting Applier
Unittest for each change.
Appropriate documentation should be added with new HA section.