Reduce an OpenStack release size inside upgrade tarball

Let’s reduce size of our fuel upgrade tarball, since it’s huge and uncomfortable for distribution.

Problem description

We need to reduce upgrade tarball’s size in order to simplify distribution workflow, boost up downloading/unpacking speed and reduce used space on master node’s file system.

Proposed change


openstack bundle

An OpenStack bundle is a set of artifacts which together constitute an openstack release in terms of Nailgun. The bundle contains next artifacts:

  • puppet modules and manifests
  • ubuntu and centos repos
  • openstack.yaml
full repo
A full repo is a self-sufficient repo which contains all packages that are required for both master node and openstack deployment.
diff-based repo
A diff-based repo is a kind of repo that contains only those packages that were changed regarding to some previous repo. Please note, that a previous repo may be either full or diff-based one.
base repo
A base repo term is used only in context of diff-based repos, when we want to point the repo on which a diff-based repo based.


Since Fuel 5.1 an upgrade tarball may contain few OpenStack bundles and each OpenStack bundle includes a self-sufficient repo. The “self-sufficient” here means that the repo contains not only OpenStack packages, but system.

The system packages are rarely changed so in most cases we can distribute within OpenStack bundle a small repo which contains only those packages that were changed regarding to some base previous release.

The changes have to be detected by packages’ checksum, since we can’t rely on package versions due to the following issue - LP1376694.

To achieve this goal we need to resolve next points:

  • A build system should have a make centos-diff-repo ubuntu-diff-repo targets to create a diff-based mirror between current mirror and some previous one. As it was mentioned in previous paragraph, a diff-based mirror should be created by comparing packages’ checksums from both mirrors.
  • A Jenkins job should be added to create a diff-based repo and provide it as an artifact.
  • The Jenkins Job which builds ISO/tarball for current version (6.0) should be able to include additional openstack bundles in order to deliver more than one openstack release.
  • The Fuel Upgrade script has to be able to add new “OpenStack releases” with one, two or more repos. In other words, the script should detect whether we’re dealing with a diff-based repo and if so to add a path to base repo too.

Information about base repo should be added through metadata.yaml. The YAML should be generated by make system and should be in the following format:

   2014.2-6.0: 2014.1.1-5.1

where 2014.2-6.0 is a release to be installed and 2014.1.1-5.1 is a base release for 2014.2-6.0. The metadata.yaml should be located in upgrade/releases/metadata.yaml folder inside upgrade tarball.

There should be only one major Fuel master node upgrade chain. For example:

5.1 -> 6.0
5.1.1 -> 6.0

In other words, there shouldn’t be a situation when we upgrade to 6.0 from 5.0.x series.


We can distribute just a set of changed packages instead of a small repo. In that case we will need to create a new repo on fly during fuel upgrade procedure. Creating repos on fly is not trivial procedure and may fail time to time due to outside factors.

Data model impact

A deployment info (a one that Nailgun sends to Astute and Astute saves it as astute.yaml) should have a repo_metadata field, which is a JSON object (or Python dictionary).

The repo_metadata has next structure:

"repo_metadata": {
    "repo_name_1": "repo_path_1",
    "repo_name_2": "repo_path_2",

and will be converted into following yum’s repo file:



and into next apt’s sources list:

deb repo_path_1
deb repo_path_2

The changes that was described above are already implemented in both Nailgun and Astute, and therefore should be kept as is.

REST API impact

Master node IP is not a constant value, so we need to support some sort of macro replacement in the release handler. In other words if we want to add a Nailgun’s release with the following orchestrator data:



    "repo_metadata": { ... }

the {MASTER_IP} and the {OPENSTACK_VERSION} will be replaced by Nailgun with an actual master node ip address and an openstack version respectively.

Upgrade impact

The Fuel Upgrade script has to be able to add new “OpenStack releases” with one, two or more repos.

Security impact


Notifications impact


Other end user impact


Performance Impact

  • Unpacking time of tarball should be reduced.
  • Downloading time of tarball should be reduced.

Other deployer impact

  • Slave nodes may have more than one repo in the operating system.

Developer impact




Primary assignee:

Other contributors:

  • Fuel OSCI
  • Fuel DevOps

Work Items

  • Add make centos-diff-repo ubuntu-diff-repo targets to build system.
  • Configure a Jenkins Job for providing diff-based mirrors as an artifact.
  • Add ability to existing Jenkins Job to build an upgrade tarball with few openstack bundles.
  • Add support of diff-based repos to fuel-upgrade script.
  • Resolve issue - LP1376694.




Existing Fuel Upgrade / OpenStack patching tests are enough since it’s about improvements, not about entirely new feature.

But next tests may be added:

  • Test that there’re more than one repo on slaves.
  • Test that an upgrade tarball contains only changed packages, not all.
  • Test that a diff-based release passes the same tests as the full release.

Acceptance criteria

  • Build an upgrade tarball with a diff-based repo. The file upgrade/releases/metadata.yaml must be exist.
  • After master node upgrade a diff-based release must have at least two repos in repo_metadata.
  • A cluster (with a diff-based release) must be deployed successfully and with packages from a diff-based repo.
  • Each slave in that cluster must have at least two repos.

Documentation Impact

The documentation about release management should be added and it should resolve next questions:

  • Which components includes a release?
  • Where are the components stored?
  • How releases reuse packages from older releases.


  • #fuel-dev on freenode