Reworking Prometheus Collector

CloudKitty’s Prometheus collector needs some rework. This spec aims at defining what is to be done and why. Everything in this document is open to discussion, equally on the associated storyboard and gerrit.

Problem Description

In the present state, Prometheus collector is not consistent with the other collectors present in CloudKitty in terms of configuration and operations.

  • The end user fully defines, on its own, the PromQL query to be executed in the metrics.yml file in a query option under extra_args subkey. Permitting the end user such complexity for building its request at the expense of genericity is not relevant regarding the use cases covered by CloudKitty.

  • The collector does not use native PromQL groupby capabilities. The process is done post Prometheus service response causing an overhead operational cost.

  • The collector ignores scope_key configuration option, disallowing scoping requests however it is the standard behaviour to be found in other collectors of the project. Besides, it is a major concern for orchestration. Let’s say you deploy several CK processors. Without scoping, you will end up with data duplications likely to result in false reports.

The collector also needs the following new features:

  • It needs authentication configuration options to dial into some secured Prometheus service.

  • It needs a configuration option to take in account a custom certificate authority file in order to authorize HTTPS requests signed with custom certificate.

  • It needs an insecure configuration option to explicitly trust HTTPS requests signed with an untrusted certificate.

These features will be useful to end users for configuring CloudKitty to effectively connect to a Prometheus service.

Proposed Change

In cloudkitty.conf, under [collector_prometheus] section:

  • Add a prometheus_user and a prometheus_password option to provide credentials for authentication (both defaults: none).

  • Add a cafile option to allow custom certificate authority file (default: none).

  • Add an insecure option to explicitly allow insecure HTTPS requests (default: false).

Possible deprecation of extra_args/query in metrics.yml for a defined metric (see Alternatives section).

In metrics.yml, for a defined metric, add an aggregation_method option under extra_args subkey to define which aggregation method to use collecting metric data.

For instance:

# metrics.yml

  # [...]

    unit: GiB
      - id
      - project_id
      - volume_type
      aggregation_method: max

Valid aggregation methods will be the following :

  • avg: the average value of all points in the time window.

  • min: the minimum value of all points in the time window.

  • max: the maximum value of all points in the time window.

  • sum: the sum of all values in the time window.

  • count: the count of all values in the time window.

  • stddev: the population standard deviation of the values in the time window.

  • stdvar: the population standard variance of the values in the time window.


Time window is computed from the period configuration option under [collect] section in cloudkitty.conf.

Other changes at this stage are mostly focused on the query building process happening inside the Prometheus collector:

  • [collect]/scope_key option value will be added to filter every query with the corresponding scope_id.

  • The collector will now use the groupby capabilities offered by PromQL using the syntax by (scope_key, groupby, metadata) instead of deferring it to the collector implementation.


For the following cloudkitty.conf :

# Some options have been omitted for the sake of clarity

collector = prometheus
scope_key = namespace
period = 3600

prometheus_url = http://prometheus-dn.tld/api/v1

And the following metrics.yml :

  # [...]

  # Let's say we want to rate the following metric
    unit: GiB
      - container_id
      - volume_type
      aggregation_method: max

The PromQL request will look the following :

max(max_over_time(container_memory_usage_bytes{namespace="foobar"}[3600s])) by (namespace, container_id, volume_type)

Start and end timestamps will be computed from the period option. Let’s say that we are starting from January 30th, 2019 at 1pm (timestamp: 1548853200). Then the end timestamp will be 1548853200 + period (= 1548856800).

The HTTP URL with the url-encoded PromQL will then look the following :



Instead of deprecation for the extra_args/query option defined for a metric in metrics.yml file, we could remove it.

Data model impact


REST API impact


Security impact


Notifications Impact


Other end user impact

End users will probably have to reconfigure their metrics definition if they are using the Prometheus collector in the present state.

However, after these changes are made, they will have less overhead configuration to do in order to switch from a collector to another, due to the consistency improvement in the way metrics are defined in metrics.yml for Prometheus collector.

They will also benefit from the added configuration options for the [collector_prometheus] section in cloudkitty.conf.

Performance Impact


Other deployer impact


Developer impact




Justin Ferrieu is assigned to work on the spec as well as to develop the evoked points.

Primary assignee:


Other contributors:


Work Items

  • Change configuration schema, query building process and query response formatting process accordingly inside Prometheus collector.

  • Add HTTPS and authentication support.




The proposed changes will be tested with unit tests.

Documentation Impact

We will add an entry for detailing the Prometheus collector configuration in Administration Guide/Configuration Guide/Collector/Prometheus.