File based Metric Map

https://blueprints.launchpad.net/watcher/+spec/file-base-metric-map

Problem description

Watcher supports loading metrics for various datasources however the metrics it uses to retrieve data are stored directly in the codebase and do not allow flexibly changing these names. And customers/admin/user are forced to use a naming scheme expected by watcher. Also it hinders Watcher ease of integration if a particular metric is already in use under a different name.

Use Cases

As a Watcher user, I want that Watcher should be allow choosing a metric name as defined in current datasource being used.

As a Watcher user, I want that Watcher to be able to change the metrics names as required.

Proposed change

An option of metric_map_path should be added to watcher_decision_engine section of watcher.conf

Allow Watcher to load a yaml file which contains a map of internal metric names (as expected by Watcher) to real metrics names in use in the datasources.

The file is optional and needs to contain only those metrics names that the user expects to override.

A method in DataSourceManager class could be called in its __init__ method.

For example:

[watcher_decision_engine]
# ...
metric_map_path = /etc/watcher/metricmap.yaml
# ...

monasca:
  - instance_cpu_usage: VM_CPU
gnocchi:
  - instance_cpu_usage: cpu_vm_util

class DataSourceManager(object):
    def __init__(self):
        #...
        # 1. Initial loading is necessary to initialize the correct defaults
        self.metric_map = {
            mon.MonascaHelper.NAME: mon.MonascaHelper.METRIC_MAP,
            gnoc.GnocchiHelper.NAME: gnoc.GnocchiHelper.METRIC_MAP,
            ceil.CeilometerHelper.NAME: ceil.CeilometerHelper.METRIC_MAP
        }
        new_metric_map = self.metrics_from_file():
        # 2. overide the loaded default by using yaml
        # update self.metric_map recursively using new_metric_map

    def get_backend(self, metrics):
        #...
        if not no_metric:
            ds = getattr(self, datasource)
            # 3. Pass the re-loaded metric map to the datasource
            ds.METRIP_MAP.update(self.metric_map[ds.NAME])
            return ds

    def metrics_from_file(self):
        """Load metrics from the config.metric_map_path"""
        if not os.path.exists(config.metric_map_path or ''):
            return {}
        with open(config.metric_map_path, 'r') as f:
            try:
                return yaml.safe_load(f.readall())
            except yaml.YAMLERROR as e:
                log.info('Could not load %(s): %s' % (
                    config.metric_map_path, str(e)))
                return {}

Comments about implementation:

Step 1: This loading can be avoided if the defaults are to be never stored in python files. But that is outside the scope of this spec as that will make the yaml file mandatory.

Step 2: self.metric_map is a nested dict so will need special handling for recursive update to datasources.

Step 3: This loading can be moved to the individual getter’s we have already defined an “API for metric_map communication” by making METRIC_MAP dictionary a class variable.

Alternatives

Instead of updating the METRIC_MAP directly in the datasource, the map could be passed to each datasoruce classes, but that spills the change over to other classes and files and also this delegation of responsibility is not contributing to any reasonable advantage.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

A configuration option should be added to the ‘watcher_decision_engine’ section, however this is not a requirement, but rather a good-to-have as this can lead to potential name conflict.

A user/admin can configure a new parameter in the config, but this is a non-binding config.

Performance Impact

None

Other deployer impact

No impact as the file creation/existence is optional

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

<sumitjami>

Work Items

• Add a configuration option ‘metric_map_file’

• Add new method to DataSourceManager class

• Load the file and update the metric_map variable

• update the METRIC_MAP dict of datasource in get_backend method

Dependencies

None

Testing

• Add unit test for file existance error check.

• Add unit test for checking if file contents are loaded correctly.

Documentation Impact

Update Watcher developer documents.

References

None

History

Revisions

Release Name	Description
Train	Introduced