Since Newton, we have the following services managed by pacemaker:
It is currently not possible to compose the above service in the same way like we do today via composable roles for the non-pacemaker services This spec aims to address this limitation and let the operator be more flexible in the composition of the control plane.
Currently tripleo has implemented no logic whatsoever to assign specific pacemaker managed services to roles/nodes.
The proposal here is to keep the existing cluster in its current form, but to extend it in two ways: A) Allow the operator to include a specific service in a custom node and have pacemaker run that resource only on that node. E.g. the operator can define the following custom nodes:
With the above definition the operator can instantiate any number of A, B or C nodes and scale up to a total of 16 nodes. Pacemaker will place the resources only on the appropriate nodes.
B) Allow the operator to extend the cluster beyond 16 nodes via pacemaker remote. For example an operator could define the following:
This second scenario would allow an operator to extend beyond the 16 nodes limit. The only difference to scenario 1) is the fact that the quorum of the cluster is obtained only by the nodes from Node A.
The way this would work is that the placement on nodes would be controllerd by location rules that would work based on node properties matching.
A bunch of alternative designs was discussed and evaluated: A) A cluster per service:
One possible architecture would be to create a separate pacemaker cluster for each HA service. This has been ruled out mainly for the following reasons:
This would be still a single cluster, but unlike today where the cloned and master/slave resources run on every controller we would introduce variables to control the maximum number of nodes a resource could run on. E.g. GaleraResourceCount would set clone-max to a value different than the number of controllers. Example: 10 controllers, galera has clone-max set to 3, rabbit to 5 and redis to 3. While this would be rather simple to implement and would change very little in the current semantics, this design was ruled out:
No changes regarding security aspects compared to the existing status quo.
No particular impact except added flexibility in placing pacemaker-managed resources.
The performance impact here is that with the added scalability it will be possible for an operator to dedicate specific nodes for certain pacemaker-managed services. There are no changes in terms of code, only a more flexible and scalable way to deploy services on the control plane.
This proposal aims to use the same method that the custom roles introduced in Newton use to tailor the services running on a node. With the very same method it will be possible to do that for the HA services managed by pacemaker today.
We need to work on the following:
No additional dependencies are required.
We will need to test the flexible placement of the pacemaker-managed services within the CI. This can be done within today’s CI limitations (i.e. in the three controller HA job we can make sure that the placement is customized and working)
Mostly internal discussions within the HA team at Red Hat