Resource Meta-data Caching

This was discussed at the YVR summit as follow up to issues raised during the Operators feedback session.

Problem description

Ceilometer polls regularly via the compute-agent to get resource meta-data’ for all instances. This polling adds load to Nova, Neutron, and Keystone APIs, which at scale causes a significant performance impact. This meta-data does not change very often, and the load is likewise often unnecessary.

Proposed change

Lets take advantage of the changes-since parameter in the nova api call. We’ll start by creating an in-memory cache to store the resource meta-data on the initial polling and the time of that polling. Then the compute agent will use the changes-since parameter in all subsequent pollings. This will reduce the amount of data queried and returned by the Nova API. The results of this call will update the cache, and the cache will be used to return the data. Also, we should populate the instance flavor and image data in the cache as well.

Alternatives

Do nothing

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

Should provide greater overall scalability due to the reduction in API load and reduced traffic flows.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:
jasonamyers
Other contributors:
None
Ongoing maintainer:
jasonamyers

Work Items

  • Implement an in-memory cache in the compute agent.
  • Change compute-agent to populate an in-memory cache.
  • Change compute-agent to use the changes-since parameter in the Nova API call .
  • Change compute-agent to record the polling timestamp in the cache.
  • Change compute-agent to cache the flavor and image metadata as well.

Future lifecycle

None

Dependencies

None

Testing

We will need to add tests for meta-data cache flag, API changed-since queries, compute-agent cache control, and image/flavor caching.

Documentation Impact

We will need to document the changes to the polling process, the new meta-data cache config option, and create new documentation that describes the caching work-flow.