OpenStack projects vary on their impact to the underlying infrastructure that they rely on greatly. This is hard to measure without going to a full scale deployment, but we should be able to measure the impact by inspecting counters already maintained by the system.
Create a new “OpenStack QA Tools” repository to house small tools written in python for purposes such as this.
Leverage existing subunit/statsd/graphite infrastructure to record results of several tests in the devstack gate.
Monitor counters for stable indicators and identify the best predictors of problems.
Once stable counters are identified, create an upper bounds for these counters to help prevent new changes in the system from accidentally introducing an inordinate amount of cost into the tested code paths.
Since there are daunting social issues around failing gate tests on global collisions, warnings and bugs about said warnings are likely the only reasonable outcome we can achieve. It will take a considerable amount of community agreement to make these limits hard.
A new python repo, os-performance-tools, has already been created, and will be maintained for the purposes of extracting and pushing counters into statsd/graphite. This will include a subunit2sql attachments plugin and code to output the counters as a subunit attachment.