Send Event Notifications

https://blueprints.launchpad.net/masakari/+spec/notifications-in-masakari

Problem description

Currently, Masakari doesn’t send any event notifications on hosts, failover segments and notifications RestFul API operation requests made by an user.

It would be useful to receive create, update and delete event notifications on any of this information changing, and the payload should contain the same information for create and update as accessible from the API.

Use Cases

Operators will be able to know following things by event notifications:

  • Begin/End of API process

  • Errors in event of failure to process APIs request

  • Payload which will contain the contents of the API request

Operators can use event notifications for analyzing, monitoring and metering purposes.

Proposed change

Versioned notifications will be emitted for the following events:

  1. Segments

    Create segment

    segment.create.start

    segment.create.end

    segment.create.error

    Update segment

    segment.update.start

    segment.update.end

    segment.update.error

    Delete segment

    segment.delete.start

    segment.delete.end

    segment.delete.error

  2. Hosts

    Create host

    host.create.start

    host.create.end

    host.create.error

    Update host

    host.update.start

    host.update.end

    host.update.error

    Delete host

    host.delete.start

    host.delete.end

    host.delete.error

  3. Notifications

    Create notification

    notification.create.start

    notification.create.end

    notification.create.error

    Process notification

    notification.process.start

    notification.process.end

    notification.process.error

Note

Process notification event is emitted only when masakari-engine starts processing received notifications by executing recovery workflow.

Event notification generally contain following information:

{
    "priority": <string, selected from a predefined list[2] by the sender>,
    "event_type": <string, defined by the sender>,
    "timestamp": <string, the isotime of when the notification emitted>,
    "publisher_id": <string, defined by the sender>,
    "message_id": <uuid, generated by oslo>,
    "payload": <json serialized dict, defined by the sender>
}

Versioned notifications:

Similar to the other OpenStack services Masakari will emit event notification to the message bus with the Notifier class provided by oslo.messaging-doc.

In versioned notification, the payload isn’t a free form dictionary but a serialized oslo versioned object. In other words, the payload should be the format which is serializable by oslo-versionedobjects library.

This is a sample of segment.create.start versioned notification:

{
    "event_type": "segment.create.start",
    "timestamp": "2018-11-22 09:25:12.393979",
    "payload": {
        "masakari_object.name": "SegmentApiPayload",
        "masakari_object.data": {
            "service_type": "compute",
            "fault": null,
            "recovery_method": "auto",
            "description": null,
            "name": "test"
        },
        "masakari_object.version": "1.0",
        "masakari_object.namespace": "masakari"
    },

    "publisher_id": "masakari-api:fake-mini",
    "message_id": "e44cb15b-dcba-409e-b0e1-9ee103b9a168"
}

Alternatives

None

Data model impact

Add osloversioned.objects for hosts, failover segment and notification. No changes will be made to the database schema.

REST API impact

None

Security impact

None

Notifications impact

Masakari doesn’t support event notification feature. This spec will add this new feature.

Other end user impact

None

Performance Impact

There will be a slight performance impact due to the overhead of sending event notifications during processing of each RestFul API request. Operator can also disable event notifications completely using configuration options.

Other deployer impact

Following config section and option will be added in masakari.conf:

[oslo_messaging_notifications]
driver=messaging

driver
Type: multi-valued
Default:''
The Drivers(s) to handle sending notifications. Possible values are messaging, messagingv2, routing, log, test, noop

Deployers should prepare the messaging system (e.g.RabbitMQ) to receive event notifications if they want to use event notification feature.

Developer impact

After this feature lands in the code, henceforth, developers will need to add new event notifications if they decide to add new RestFul APIs.

Implementation

Assignee(s)

Primary assignee:

Work Items

  • Add base classes for event notification mechanism

  • Add osloversioned.objects to be used to send event notifications

  • Add methods to send notifications for each newly added osloversioned.object

  • Send event notification for create/update/delete operations

  • Add unit tests for code coverage

  • Add documentation on how to use this feature

Dependencies

None

Testing

No need to write tempest tests as unit tests are sufficient to check whether the event notifications are sent or not for each create, update and delete operations.

Documentation Impact

Add documentation to explain how to use event notification feature so that operator can write code to receive these events for their own purpose.

References

History

Revisions

Release Name

Description

Queens

Introduced

Rocky

Approved

Stein

Re-proposed