https://blueprints.launchpad.net/fuel/+spec/replace-obs
https://blueprints.launchpad.net/fuel/+spec/enable-gpg-check-and-sign
We’ve found following fundamental limitations of OBS:
This specification introduces replacement for existing OBS infrastructure using new build system Perestroika.
Solution will use standard upstream Linux distribution tools to build packages (sbuild/mock), publish packages to repositories and manage package repositories (reprepro/createrepo).
Every package will be built in a clean and up-to-date buildroot. Packages, their dependencies and build dependencies will be fully self-contained for each MOS release. Any package included in any release can be rebuilt at any point in time using only the packages from that release.
Package build CI will be reproducible and can be recreated from scratch in a repeatable way.
New build system is based on Docker, which provides easy distribution. There will be created proper Docker Images for each supported Linux distribution with necessary tools and scripts.
Puppet will be used for configuration of those images.
We can wrap host side scripts of interaction in a package for easy deployment.
None
None
None
None
None
None
None
New build system will contain following parts:
Code storage We use gerrit code review system as code storage.
Gerrit projects structure:
MOS+master-node Openstack packages code projects:
[customer-name]/openstack/{package name}
spec projects:
[customer-name]/openstack-build/{package name}
MOS linux packages code+spec projects:
[customer-name]/packages/{distribution}/{packagename}
Master-node linux packages (separated from MOS linux in 7.0) code+spec projects:
[customer-name]/packages/fuel/{distribution}/{package name}
Versioning scheme will be supported by project branches openstack:
openstack-ci/fuel-{fuel version}/{openstack version}
MOS linux/master-node:
{fuel version}
Scheduler This part is based on Jenkins CI tool. All jobs will be configured via jenkins-job-builder Jenkins has a separate set of jobs for each [customer name]+[fuel version] case. Gerrit-trigger configured to track events from {version} branch of all [customer-name] gerrit projects.
Each set of jobs will contain:
- Jobs for openstack packages for cluster (rpm/deb)
- Jobs for MOS linux packages for cluster (rpm/deb)
- Jobs for openstack packages for master-node (optional in case of using cluster packages) (rpm)
- Jobs for non-openstack master-node packages (rpm)
- Jobs for fuel packages (rpm/deb)
- Job for package publishing
Build workers Hardware nodes with preconfigured build tools for all supported distribution. Will be configured as Jenkins slave.
Each worker will contain:
- preconfigured docker images with native build tools for each distro type: mockbuild: will build packages by mock (centos6/7 target distributions) sbuild: will build packages by sbuild (trusty target distribution)
- prepared minimal build chroots for all supported distribution These chroots will be updated on daily basis in order to be up-to-date against upstream state.i
- precofigured packages caching system (optional) All packaged downloaded from upstream repositories should be cached on build host in order to be reused by build stages. This part will reduce building time. Could be done with squid/polipo/approx
Build system will use short-lived docker containers to perform package building. Docker images contain preconfigured build tools only. No chroots inside images. Build chroots will be mounted to docker container on start in read-only mode. Additionally tmpfs partition will be mounted over read-only chroot folder with AUFS overlays inside docker container. Docker container will be destroyed after build stage is done.
All worker nodes will be joined by jenkins slave label
Publisher Publisher node will contain all repositories for all customer projects. Will be configured as Jenkins slave. Repositories will be maintained by native tools of respective distribution (reprepro/createrepo). Publisher node will be fully private and available from Jenkins master node only because of containing secret GPG key. All packages and repositories will be signed in terms of respective distribution by GPG key, stored on Publisher node.
Mirror node All repositories should be available via http/rsync protocols. All repositories will be synced by Publisher to Mirror host
None
In case of using new build system we should change workflow documentation where OBS mentioned.
All of the scripts and Jenkins jobs should be tested in a sandbox environment for building packages.