Triple CI improvements
Tripleo CI is painful at the moment, we have problems with both reliability
and consistency of running job times, this spec is intended to address a
number of the problems we have been facing.
Developers should be able to depend on CI to produce reliable test results with
a minimum number of false negatives reported in a timely fashion, this
currently isn’t the case. To date the reliability of tripleo ci has been
heavily effected by network glitches, availability of network resources and
reliability of the CI clouds. This spec is intended to deal with the problems
we have been seeing.
- Problem : Reliability of hp1 (hp1_reliability)
- Intermittent failures on jobs running on the hp1 cloud have been causing a
large number of job failures and sometimes taking this region down
altogether. Current thinking is that the root of most of these issues is
problems with a mellanox driver.
- Problem : Unreliable access to network resources (net_reliability)
- Gaining reliable access to various network resources has been inconsistent
causing a CI outage when any one network resource is unavailable. Also
inconsistent speeds downloading these resources can make it difficult to
gauge overall speed improvements made to tripleo.
- Problem : (system_health) The health of the overall CI system isn’t
- immediately obvious, problems often persist for hours (or occasionally days)
before we react to them.
- Problem : (ci_run_times) The tripleo devtest story takes time to run,
- this uses up CI resources and developer’s time, where possible we should
reduce the time required to run devtest.
- Problem : (inefficient_usage) Hardware on which to run tripleo is a finite
- resource, there is a spec in place to run devtest on an openstack
deployment, this is the best way forward in order to use the resources we
have in the most efficient way possible. We also have a number of options to
explore that would help minimise resource wastage.
- Problem : (system_feedback) Our CI provides no feedback about trends.
- A good CI system should be more than a system that reports pass or fail, we
should be getting feedback on metrics allowing us to observe degradations,
where possible we should make use of services already provided by infra.
This will allow us to proactively intervene as CI begins to degrade?
- Problem : (bug_frequency) We currently have no indication of which CI
- bugs are occurring most often. This frustrates efforts to make CI more
- Problem : (test_coverage) Currently CI only tests a subset of what it
There are a number of changes required in order to address the problems we have
been seeing, each listed here (in order of priority).
- Temporarily scale back on CI by removing one of the overcloud jobs (so rh1 has
the capacity to run CI Solo).
- Remove hp1 from the configuration.
- Run burn-in tests on each hp1 host, removing(or repairing) failing hosts.
Burn-in tests should consist of running CI on a newly deployed cloud matching
the load expected to run on the region. Any failure rate should not exceed
that of currently deployed regions.
- Redeploy testing infrastructure on hp1 and test with tempest, this redeploy
should be done with our tripleo scripts so it can be repeated and we
are sure of parity between ci-overcloud deployments.
- Place hp1 back into CI and monitor situation.
- Add back any removed CI jobs.
- Ensure burn-in / tempest tests are followed on future regions being deployed.
- Attempts should be made to deal with problems that develop on already
deployed clouds, if it becomes obvious they can’t be quickly dealt with after
48 hours they should be temporarily removed from the CI infrastructure and will
need to pass the burn-in tests before being added back into production.
- Deploy a mirror of pypi.openstack.org on each Region.
- Deploy a mirror of the Fedora and Ubuntu package repositories on each region.
- Deploy squid in each region and cache http traffic through it, mirroring
where possible should be considered our preference but having squid in place
should cache any resources not mirrored.
- Mirror other resources (e.g. github.com, percona tarballs etc..).
- Any new requirements added to devtest should be cachable with caches in
place before the requirement is added.
- Monitor our CI clouds and testenvs with Icinga, monitoring should include
ping, starting (and connecting to) new instances, disk usage etc....
- Monitor CI test results and trigger an alert if “X” number of jobs of the
same type fail in succession. An example of using logstash to monitor CI
results can be found here.
Once consistency is no longer a problem we will investigate speed improvements
we can make on the speed of CI jobs.
- Investigate if unsafe disk caching strategies will speed up disk image
creation, if an improvement is found implement it in production CI by one of
- run “unsafe” disk caching strategy on ci cloud VM’s (would involve exposing
this libvirt option via the nova api).
- use “eatmydata” to noop disk sync system calls, not currently
packaged for F20 but we could try and restart that process.
- Abandon on failure : adding a feature to zuul (or turning it on if it already
exists) to abandon all jobs in a queue for a particular commit as soon as a
voting commit fails. This would minimize usage of resources running long
running jobs that we already know will have to be rechecked.
- Adding the collectl element to compute nodes and testenv hosts will allow us
to find bottle necks and also identify places where it is safe to overcommit
(e.g. we may find that overcommitting CPU a lot on testenv hosts is viable).
- Using a combination of logstash and graphite
- Output graphs of occurrences of false negative test results.
- Output graphs of CI run times over time in order to identify trends.
- Output graphs of CI job peak memory usage over time.
- Output graphs of CI image sizes over time.
- In order to be able to track false negatives that are hurting us most we
should agree not to use “recheck no bug”, instead recheck with the
relevant bug number. Adding signatures to Elastic recheck for known CI
issues should help uptake of this.
- Run tempest against the deployed overcloud.
- Test our upgrade story by upgrading to a new images. Initially to avoid
having to build new images we can edit something on the overcloud qcow images
in place in order to get a set of images to upgrade too.
- As an alternative to deploying our own distro mirrors we could simply point
directly at a mirror known to be reliable. This is undesirable as a long
term solution as we still can’t control outages.
Other End User Impact
- No longer using recheck no bug places a burden on developers to
investigate why a job failed.
- Adding coverage to our tests will increase the overall time to run a job.
Performance of CI should improve overall.
Other Deployer Impact
- Primary assignee:
- Other contributors:
- looking for volunteers...
- hp1 upgrade to trusty.
- Potential pypi mirror.
- Fedora Mirrors.
- Ubuntu Mirrors.
- Mirroring other non distro resources.
- Per region caching proxy.
- Document CI.
- Running an unsafe disk caching strategy in the overcloud nodes.
- ZUUL abandon on failure.
- Include collectl on compute and testenv Hosts and analyse output.
- Mechanism to monitor CI run times.
- Mechanism to monitor nodepool connection failures to instances.
- Remove ability to recheck no bug or at the very least discourage its use.
- Monitoring cloud/testenv health.
- Expand ci to include tempest.
- Expand ci to include upgrades.
CI failure rate and timings will be tracked to confirm improvements.
The tripleo-ci repository needs additional documentation in order to describe
the current layout and should then be updated as changes are made.