Enable deployment of availability monitoring¶

https://blueprints.launchpad.net/tripleo/+spec/tripleo-opstools-availability-monitoring

TripleO should be deploying out-of-the-box availability monitoring solution to serve the overcloud.

Problem Description¶

Currently there is no such feature implemented except for possibility to deploy sensu-server, sensu-api and uchiwa (Sensu dashboard) services in the undercloud stack. Without sensu-client services deployed on overcloud nodes this piece of code is useless. Due to potential of high resource consumption it is also reasonable to remove current undercloud code to avoid possible problems when high number of overcloud nodes is being deployed.

Instead sensu-server, sensu-api and uchiwa should be deployed on the separate node(s) whether it is on the undercloud level or on the overcloud level. And so sensu-client deployment support should be flexible enough to enable connection to external monitoring infrastructure or with Sensu stack deployed on the dedicated overcloud node.

Summary of use cases:

1. sensu-server, sensu-api and uchiwa deployed in external infrastructure; sensu-client deployed on each overcloud node 2. sensu-server, sensu-api and uchiwa deployed as a separate Heat stack in the overcloud stack; sensu-client deployed on each overcloud node

Proposed Change¶

Overview¶

The sensu-client service will be deployed as a composable service on the overcloud stack when it is explicitly stated via environment file. Sensu checks will have to be configured as subscription checks (see [0] for details). Each composable service will have it’s own subscription string, which will ensure that checks defined on Sensu server node (wherever it lives) are run on the correct overcloud nodes.

There will be implemented a possibility to deploy sensu-server, sensu-api and uchiwa services on a stand alone node deployed by the undercloud. This standalone node will have a dedicated purpose for monitoring (not only for availability monitoring services, but in future also for centralized logging services or performance monitoring services)

The monitoring node will be deployed as a separate Heat stack to the overcloud stack using Puppet and composable roles for required services.

Alternatives¶

None

Security Impact¶

Additional service (sensu-client) will be installed on all overcloud nodes. These services will have open connection to RabbitMQ instance running on monitoring node and are used to execute commands (checks) on the overcloud nodes. Check definition will live on the monitoring node.

Other End User Impact¶

None

Performance Impact¶

We might consider deploying separate RabbitMQ and Redis for monitoring purposes if we want to avoid influencing OpenStack deployment in the overcloud.

Other Deployer Impact¶

Sensu clients will be deployed by default on all overcloud nodes except the monitoring node.
New Sensu common parameters:
- MonitoringRabbitHost
  RabbitMQ host Sensu has to connect to
- MonitoringRabbitPort
  RabbitMQ port Sensu has to connect to
- MonitoringRabbitUseSSL
  Whether Sensu should connect to RabbitMQ using SSL
- MonitoringRabbitPassword
  RabbitMQ password used for Sensu to connect
- MonitoringRabbitUserName
  RabbitMQ username used for Sensu to connect
- MonitoringRabbitVhost
  RabbitMQ vhost used for monitoring purposes.
New Sensu server/API parameters
- MonitoringRedisHost
  Redis host Sensu has to connect to
- MonitoringRedisPassword
  Redis password used for Sensu to connect
- MonitoringChecks:
  Full definition (for all subscriptions) of checks performed by Sensu
New parameters for subscription strings for each composable service:
- For example for service nova-compute MonitoringSubscriptionNovaCompute, which will default to ‘overcloud-nova-compute’

Developer Impact¶

Support for new node type should be implemented for tripleo-quickstart.

Implementation¶

Assignee(s)¶

Martin Mágr <mmagr@redhat.com>

Work Items¶

puppet-tripleo profile for Sensu services
puppet-tripleo profile for Uchiwa service
tripleo-heat-templates composable service for sensu-client deployment
tripleo-heat-templates composable service for sensu-server deployment
tripleo-heat-templates composable service for sensu-api deployment
tripleo-heat-templates composable service for uchiwa deployment
Support for monitoring node in tripleo-quickstart
Revert patch(es) implementing Sensu support in instack-undercloud

Dependencies¶

Puppet module for Sensu services: sensu-puppet [1]
Puppet module for Uchiwa: puppet-uchiwa [2]
CentOS Opstools SIG repo [3]

Testing¶

Sensu client deployment will be tested by current TripleO CI as soon as the patch is merged, as it will be deployed by default.

We should consider creating CI job for deploying overcloud with monitoring node to test the rest of the monitoring components.

Documentation Impact¶

Process of creating new node type and new options will have to be documented.

References¶

[0] https://sensuapp.org/docs/latest/reference/checks.html#subscription-checks [1] https://github.com/sensu/sensu-puppet [2] https://github.com/Yelp/puppet-uchiwa [3] https://wiki.centos.org/SpecialInterestGroup/OpsTools

Enable deployment of availability monitoring