We must complete all package validation (including system tests) before
package change patch set is merged, and then publish packages to the mirrors
immediately after review is merged.
Proposed flow for change requests to MOS packages code and/or spec is below:
- Change request is proposed for code review to Gerrit
- Change request parameters (repository name, branch, change ID) are passed
by Zuul to the Jenkins build job that fetches sources and specs from
Gerrit and pushes sources and specs to the build system in a respective
format (SRPM for RPM packages, and unpacked sources for DEB packages)
- On success build job triggers in series publish job, and then test jobs -
install and systest. If any of is jobs failed, next jobs are not run and
build job fails
- publish job moves package (or packages), built from resulting sources and
specs by the build system, into a test repository. Test repository location
is created as a Jenkins job artifact
- Test repository is run through install test - all built packages are
installed and removed one by one
- Test repository is run through systest test - built packages are used to
deploy some set of servers.
- Change request is peer-reviewed
- Change request is accepted or rejected by core team
- Change request is rebased and run through pre-merge automated checks -
gate job . Because gate job should check several change requests at
once, a set of build jobs are triggered with parameters preventing tests,
so each build job triggers only publish job
- If all build jobs are succeed then gate job triggers systest job using
all created repositories at once to check together packages built by
- Change request is merged or rejected
- Merge event from Gerrit is picked up by Zuul to run Jenkins build job that
builds final package and publish it to released repository. publish job
detects if change request is merged, selects appropriate repository, puts
packages to selected repository, and signes them (for RPM - a package
itself, for DEB - repository metadata)
- Repository with rebuilt package is propagated to the external MOS mirrors
by the means of Jenkins job that uses transactional syncronization
Test repository format
Test repository with packages should be passed to the system tests as
parameter using the following conventions:
- RPM packages. Plain RPM repository accessible by HTTP.
- DEB packages. Single-component APT repository accessible by HTTP,
with distribution name that equals to the current upstream OS codename.
Automated functional and integration tests
Each package built into a test repository will undergo the following tests:
- Basic syntax test
- rpmlint and lintian, for RPM and DEB packages, respectively
- Basic package health test - install job
- Uses plain VM with respective OS type that
contains only the minimal set of packages. Package install, run basic
shell script, and uninstall tests are performed.
- Full system test - systest job
- All packages are tested using the same
scenarios that are used in current BVT tests on the Product CI. In
order to keep the actual test run as short and reliable as
possible, the VM snapshots for Fuel master node and cluster nodes
are prepared ahead of time and kept on slave servers ready for
immediate use. The process of preparing the VM snapshots ahead of
time reduces the network traffic and influence of an external
depencencies during the run. The job that prepares the VM
snapshots is executed on a daily basis or, alternatively, triggered
on ISO update, depending on the frequency if ISO updates.
Depending on the package flavor, there are different artifacts that could be
supplied to the integration test job.
- RPM and DEB packages for clusters. Test repository is added to the
Nailgun with highest priority (in terms of a respective package
- Fuel Master host OS packages. Rebuilding of a Fuel ISO with an extra test
repository and redeployment of Fuel Master node must precede the system
- Fuel bootstrap packages. Rebuilding of bootstrap package and
installing it on the Fuel Master node created from snapshot prepared
ahead precedes the system test.
- Fuel Docker container packages. Rebuilding of Fuel Docker containers
late-package and installing it on the Fuel Master node created from
snapshot prepared ahead precedes the system test.
All tests will be running in parallel. In case if basic package health test
for a given patch set is failed, the respective full system test must be
aborted to prevent the waste of a CI system resources.
Full logs for a system test run must be available in the Jenkins as a job
artifact. In order to support debugging of system test issues, snapshots of
VMs created during the test run are stored on Jenkins slaves for 20 days.
In order to support enhanced debug cases, there should be an ability to
replicate all the test parameters, including job artifacts, on a
After merge event in Gerrit, package built on a gate will be pushed to a
respective stable repository on an internal OSCI mirror and signed in
terms of a respective package manager. Right after that, this
resulting repository will be synced to the external OSCI mirrors
without any further verification. RPM and DEB mirrors are synced
independently, list of synced packages must be included as an artifact
to an appropriate Jenkins job.
Testing of concurrent requests
To ensure that no regressions are introduced by merging of concurrent
requests, on merge event, every outstanding request must be speculatively
retested against the tip of the package code and specs branches exactly
as it is going to be merged.
To optimize handling of merge queue we are going to use Zuul as a
The following auxiliary jobs must be supported on the Jenkins as the
part of this spec:
- Job to prepare environment for system tests on the Jenkins slave servers
- Jobs to support debug sessions workflow
- Job to clean expired VM snapshots on Jenkins slaves
As an alternative to Zuul we considered managing pre-merge checks via
set of custom scripts, but it appears that we need a lot of Zuul
functionality and in long-term we will benefit from reusing the
existent tooling, which is supported by wide community including
OpenStack Infra team.
Other end user impact
Other deployer impact
Due to more complex scenarios used in deploy tests configuration, time to
complete all CI jobs for a single commit will increase approx. by 60%-120%.
A speed of publishing of packages decreased from hours to minutes, should
compensate this inconvenience, though.
- Changes in system tests will require additional servers to be used as
- Logs of a system tests will be stored on the Jenkins Master, therefore
we will need to increase the storage capacity for this server (at least 1 TB
is required). Alternatively, this requirement could be addressed by the
centralized diagnostic snapshot storage feature.
- Changes described in this document will only apply to 7.0+ CI environments,
no backporting for older releases is planned.