Create functional and integration tests in ceilometer project

https://blueprints.launchpad.net/ceilometer/+spec/ceilometer-functional-tests

At this moment there are no integration tests for Ceilometer outside of Tempest, which are fully covering all its functionality. On Paris summit it was decided to move part of the tests (and the implementation of new ones) in the code tree of Ceilometer itself. Also tests for different databases should be moved to this module as well (currently they are living in the folder of unit tests and this doesn’t correspond, generally speaking, to their aim). And all unit tests should be moved from ceilometer/tests/ path to ceilometer/tests/unit folder, so that ceilometer/tests tree contains only three folders: unit, functional, integration.

It is why we should develop integration tests and create Jenkins job for them, which will run these tests, located in a separate module within the Ceilometer code tree (tests for different backends will be moved to this module as well). These tests will be designed in DSVM-style. Tests that require communication with real other OpenStack services, will follow Tempest practice to use master branches for these components.

Problem description

Implementation of the plan will take a long time because of many subtasks and interaction with other projects (in particular with OpenStack infra). In this approach we may implement tests which are not assumed in tempest: * grey/white-box tests that are not pure API tests and may assumed some knowledge of the implementation * tests that require acceleration of the polling interval in order to complete within a reasonable time, e.g. changing the cpu_source interval in the pipeline.yaml and restarting the compute agent so as to gather sufficient cpu_util datapoints to drive alarming tests * tests that depend on the timeliness of incoming notifications from other OpenStack services The implementation of this task will greatly increase test coverage for ceilometer project, including the features which weren’t tested before.

Proposed change

Alternatives

Contribute new integration tests to the tempest project. This approach does not satisfy us because OpenStack QA team rejects too specific integration tests for projects, as only projects teams can decide if they are good or not.

Data model impact

None

REST API impact

None

Security impact

None

Pipeline impact

None

Other end user impact

None

Performance/Scalability Impacts

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:
vrovachev <vrovachev@mirantis.com>
Other contributors:
chdent <chdent@redhat.com>

Work Items

Work items: * Add ceilometer/tests/{integration|functional} testcases that use the python-clients directly. * Move ceilometer/tests testcase to ceilometer/tests/unit. * Move all existing unit tests to ceilometer/tests/unit. * Move existing DB scenario tests to ceilometer/tests/{integration|functional}. * Add new tox target(s)(functional, integration and others if necessary) to ceilometer/tox.ini. * Add {pipeline}-ceilometer-dsvm-functional-{datastore} job template to os-infra/project-config/jenkins/jobs/ceilometer.yaml where datastore in [‘sqlalchemy’, ‘mongodb’, ‘hbase’].

Future lifecycle

None

Dependencies

None

Testing

None

Documentation Impact

None

References

None