Track DBaaS(Trove) Notifications

https://blueprints.launchpad.net/ceilometer/+spec/track-dbaas-notifications

The goal of this blueprint is to capture the notification events emitted by Trove (DBaaS) during create, modify volume, resize, delete operations and periodic exist events.

Problem description

Trove is Database as a Service for OpenStack. It automates the creation, configuration and management of relational or non-relational database without the burden of handling complex administrative tasks.

The trove-taskmanager service dispatches tasks such as provisioning database instances, managing the lifecycle of database instances, and performing operations on the database instance and emits notification event for each such task.

These notifications are CRUD events, that notify the creation, update or modification of a trove instance and its underlying volume, and also have measurement values such as instance size and volume size.

Ceilometer needs to be updated to consume and record these notifications as samples and events (traits).

From Trove perspective they are running instances of a database, not compute instances. Metering and billing will be based on hours that the database has been running, not necessarily how long the hosting instance has been run. This requires a unique set of metering records(samples) to be generated and stored to enable usage tracking and billing of the individual database instances.

If Ceilometer processes and record these notifications, other services will be able to use this for aggregation and billing purpose.

Proposed change

To represent as events, define event definitions and field mappings (trait definitions) for Trove resource type - instance in the /etc/ceilometer/event_definitions.yaml file.

The trait field syntax in trait definitions follow a variant of the JSONPath.

Example of the event definition and field mappings for the domain notifications that need to go in event_definitions.yaml,

  • event_type: trove.instance.*
  • traits: &trove_traits
  • instance_size:
  • fields: payload.instance_size
  • resource_id:
  • fields: payload.instance_id
  • instance_name:
  • fields: payload.instance_name
  • nova_instance_id:
  • fields: payload.nova_instance_id
  • created_at:
  • fields: payload.created_at
  • launched_at:
  • fields: payload.launched_at
  • volume_size:
  • fields: payload.volume_size
  • volume_id:
  • fields: payload.nova.volume_id
  • modify_at:
  • fields: payload.modify_at
  • deleted_at:
  • fields: payload.deleted_at

To enable event processing in Ceilometer, configuration is needed in ceilometer.conf,

  • Configure store_events as True under the notification section,

eg - [notification] store_events = True

  • Specify the event_definitions file name under the event section,

eg - [event] definitions_cfg_file = event_definitions.yaml

Using events provides better querying of metadata and avoids the effort to aggregate samples with a fixed measurement value of 1.

To represent as samples, a notification plugin is required in Ceilometer, to hear notifications at an exchange and topic and transform them into samples.

This is needed to account for the database uptime hours, which is based on the trove.instance.exists event. Specifically, the audit_period_beginning and audit_period_ending for each exists event will be used to compute the uptime hours

A counter type - database can be used to associate the trove samples

Add a new listener class - TroveMetricsNotificationBase that extends plugin.NotificationBase for DBaaS service to transform Trove event data into samples

A sub-class of TroveMetricsNotificationBase that emits samples for the trove.instance.create and trove.instance.exists events and another for the trove.instance.modify* and trove.instance.delete events

The Trove event types have the following pattern,

  • trove.<resource>.create
  • trove.<resource>.modify_volume
  • trove.<resource>.modify_flavor
  • trove.<resource>.delete
  • trove.<resource>.exists

, where resource is instance.

Alternatives

Data model impact

None.

REST API impact

None.

Security impact

None.

Pipeline impact

None.

Other end user impact

None.

Performance/Scalability Impacts

No new impacts. Pre-existing concerns with capacity at the notification and storage handling layers remain.

Other deployer impact

Enable Trove exchange in ceilometer.conf

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:
rjaiswal
Ongoing maintainer:
rjaiswal

Work Items

  • Event definitions and field mappings for all supported Trove resource types.
  • Event validation.
  • Trove Notifications Listener class to process create/update/delete/exists events for trove instance.
  • Test coverage for the above listener class and sample validation.

Future lifecycle

There could be new types of notifications when new resource types are added or new features are supported like backup/restore, patching, monitoring etc.

When Trove conforms to the PaaS event format, the field mappings for integrated events will need to be refactored to adjust to the new event format.

With the impending move towards declarative metering in Ceilometer, the proposed notification plugin for Trove will become redundant/deprecated and replaced with declarative meter definitions.

Dependencies

DBaaS is construed as a PaaS service by some and might have it own message bus where it is sending notifications. Ceilometer recently was extended to consume notification from multiple message bus - https://review.openstack.org/#/c/77612/

If there are multiple message buses, then ceilometer will need multiple transport endpoints configured in ceilometer configuration. (ceilometer.conf)

Testing

Unit and integration Tests will be added to validate events generated.

Documentation Impact

None.