The ceilometer polling agents currently perform two roles: They poll various services to generate samples and they optionally transform those samples, using pipelines defined by pipeline.yaml, to compose other samples eventually publishing them as measurements. This latter transformation functionality raises the complexity of the pollster code. An alternative would be for the pollsters to send notifications for the meters it creates and have them be processed by the notification agent (which has a transformation pipeline of its own) and then be published.
Including the transformation pipeline in the pollsters increases complexity in the pollsters and duplicates functionality found in the notification agent.
One way to resolve this complexity is to not do transformations in the pollsters. Instead when new samples are polled, format them as notifications and push them onto the notification bus to be retrieved by the notification agent. The pipeline within that agent will then do any required transformations.
Besides clarifying the focus of the polling agents it is likely this change will make it easier to test and build performance improvements both in code and in deployments. This is murky speculation but it is challenging to test the theory without doing the work.
We also get an opportunity to review and audit the polling code, which is a useful thing to do on a regular basis.
The primary alternative is to leave things as they are. Obviously if we do this we lose the benefits described above.
The proposal does not impact the structure of the data, but it would impact the flow of the data. Testing will be required to see what kind of impact this will have on the timeliness of data acquisition.
No direct impact on the resources presented by the API.
None beyond existing concerns.
Describing pipeline impact is somewhat complex because the word pipeline can be used:
In this change transformation and publishing will be localized in the notification agents. Gathering and producing of samples will be done by polling agents, the api server (via posting meters) and the notification agents.
In order to ease migration the polling agents will continue to use the pipeline.yaml file to configure the sources they will poll and the intervals at which that polling will be done. The same file may be used, or a new file without any sinks. The idea is that both should work.
The most significant difference is that the existing validation which confirms that there is a pairing between source and sink will be removed. The polling agents may poll whatever they like. If the notification agent doesn’t want the samples that are generated it can simply drop them on the floor. This provides two benefits:
There is hope that this will decrease the footprint of the pollsters making it easier to deploy many of them, thus increasing the overall performance of the polling system.
On the negative side there could be increased load on both the messaging and the notification agents but this could be compensated for by:
Deployers will need to be aware of their options for deploying suitable numbers of notification agents.
Testing sample transformation from point of sample creation to point of storage will become more complex as it will traverse several services. With the advent of in-tree functional testing there is a place where this testing can and should be done. Having those tests will be very helpful in the long run despite the initial pain in setting them up.
The members of the Telemetry program will do any required ongoing maintenance for this feature.
No new dependencies.
Testing as described above.
Deployment documentation will need to be updated to reflect the connection between the polling and notification agents, the option to use a custom messaging bus and other opportunities for customization.
None at this time.