Create table publishing middleware¶
https://blueprints.launchpad.net/congress/+spec/table-publishing-middleware
To lower the barrier to entry for other projects to publish data to Congress on the (extended) DSE, create middleware that automates as much of the process as possible.
Problem description¶
Currently Congress periodically polls the services it is managing to get their data. It would be preferable if services sent data to Congress only when that data changes. For example, instead of Congress pulling the list of Nova’s servers every minute, Nova would send its servers to Congress whenever a new server is created, an existing server is deleted, or an existing server is updated.
oslo.messaging already makes it possible for projects to publish data to other projects, but that mechanism is underutilized today.
Proposed change¶
Solving this problem means making it even easier for other projects to send their data to Congress than oslo.messaging does. This change will include middleware, perhaps made available via oslo, that publishes all changes to the underlying database tables on the bus. Moreover, it will send not the entire table but rather the delta on the table that occurred. It will be tightly integrated into existing oslo.db so that existing projects need make no code changes; they need only include and configure the code.
There will need to be a synchronization mechanism: Congress pulls all the data once and then needs deltas published on the bus from that point on. The deltas sent on the bus will include timestamps that allow Congress to synchronize the initial pull of data with the deltas.
Alternatives¶
Another option is to have a separate effort, say oslo.publish_tables, that is not so tightly integrated with oslo.db. The upside to this approach is that projects not using oslo.db (even those outside of OpenStack) could leverage the code. The downside to this approach is that projects already using oslo.db would need to write and maintain a bunch of special-purpose code for publishing their data–an approach that has seen little success til now.
Perhaps we will find that we can implement both approaches with little extra effort, but the primary goal of this spec is the tight integration with oslo.db.
Policy¶
None
Policy Actions¶
None
Data Sources¶
None
Data model impact¶
None
REST API impact¶
None
Security impact¶
None
Notifications impact¶
The goal of this spec is to improve the ability of other projects to utilize the notifications mechanism provided by oslo.messaging.
Other end user impact¶
None
Performance impact¶
There will be minimal performance impact on the projects that utilize this code because only the deltas that hit the database will be published on the bus. Moreover, because each project can configure which tables are published on the bus, the project owner can tune any possible performance impact.
Other deployer impact¶
When configuring the middleware, we propose one key configuration option: which tables should be published on the bus.
Developer impact¶
None
Implementation¶
Assignee(s)¶
- Primary assignee:
<launchpad-id or None>
- Other contributors:
<launchpad-id or None>
Work items¶
Write basic middleware functionality
Integrate it into oslo.db
Install it into an existing service, say Nova.
Standalone documentation
Dependencies¶
enable-delta-driven-datasources: enable the datasources/DatasourceDriver class to connect to datasources that send deltas.
Testing¶
Unit testing the basic functionality
Tempest tests o Verify that changes to tables configured to be published actually get published o Verify that changes to tables configured NOT to be published do not get published
Documentation impact¶
Need documentation in oslo for the basic middleware. Need documentation update to oslo.db
References¶
None
