.. This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode ======================================================= Reduce an OpenStack release size inside upgrade tarball ======================================================= https://blueprints.launchpad.net/fuel/+spec/reduce-tarball-size Let's reduce size of our fuel upgrade tarball, since it's huge and uncomfortable for distribution. Problem description =================== We need to reduce upgrade tarball's size in order to simplify distribution workflow, boost up downloading/unpacking speed and reduce used space on master node's file system. Proposed change =============== Terms ----- *openstack bundle* An OpenStack bundle is a set of artifacts which together constitute an openstack release in terms of Nailgun. The bundle contains next artifacts: - puppet modules and manifests - ubuntu and centos repos - openstack.yaml *full repo* A full repo is a self-sufficient repo which contains all packages that are required for both master node and openstack deployment. *diff-based repo* A diff-based repo is a kind of repo that contains only those packages that were changed regarding to some previous repo. Please note, that a previous repo may be either full or diff-based one. *base repo* A base repo term is used only in context of diff-based repos, when we want to point the repo on which a diff-based repo based. Changes ------- Since Fuel 5.1 an upgrade tarball may contain few OpenStack bundles and each OpenStack bundle includes a self-sufficient repo. The "self-sufficient" here means that the repo contains not only OpenStack packages, but system. The system packages are rarely changed so in most cases we can distribute within OpenStack bundle a small repo which contains only those packages that were changed regarding to some base previous release. The changes have to be detected by packages' checksum, since we can't rely on package versions due to the following issue - `LP1376694 `_. To achieve this goal we need to resolve next points: * A build system should have a ``make centos-diff-repo ubuntu-diff-repo`` targets to create a diff-based mirror between current mirror and some previous one. As it was mentioned in previous paragraph, a diff-based mirror should be created by comparing packages' checksums from both mirrors. * A Jenkins job should be added to create a diff-based repo and provide it as an artifact. * The Jenkins Job which builds ISO/tarball for current version (6.0) should be able to include additional *openstack bundles* in order to deliver more than one openstack release. * The Fuel Upgrade script has to be able to add new "OpenStack releases" with one, two or more repos. In other words, the script should detect whether we're dealing with a diff-based repo and if so to add a path to base repo too. Information about base repo should be added through ``metadata.yaml``. The YAML should be generated by make system and should be in the following format:: diff_releases: 2014.2-6.0: 2014.1.1-5.1 where *2014.2-6.0* is a release to be installed and *2014.1.1-5.1* is a base release for *2014.2-6.0*. The ``metadata.yaml`` should be located in ``upgrade/releases/metadata.yaml`` folder inside upgrade tarball. There should be only one major Fuel master node upgrade chain. For example:: 5.1 -> 6.0 5.1.1 -> 6.0 In other words, there shouldn't be a situation when we upgrade to ``6.0`` from ``5.0.x`` series. Alternatives ------------ We can distribute just a set of changed packages instead of a small repo. In that case we will need to create a new repo on fly during fuel upgrade procedure. Creating repos on fly is not trivial procedure and may fail time to time due to outside factors. Data model impact ----------------- A deployment info (a one that Nailgun sends to Astute and Astute saves it as ``astute.yaml``) should have a ``repo_metadata`` field, which is a JSON object (or Python dictionary). The ``repo_metadata`` has next structure:: "repo_metadata": { "repo_name_1": "repo_path_1", "repo_name_2": "repo_path_2", } and will be converted into following yum's repo file:: [repo_name_1] name=repo_name_1 baseurl=repo_path_1 gpgcheck=0 [repo_name_2] name=repo_name_2 baseurl=repo_path_2 gpgcheck=0 and into next apt's sources list:: deb repo_path_1 deb repo_path_2 The changes that was described above are already implemented in both Nailgun and Astute, and therefore should be kept as is. REST API impact --------------- Master node IP is not a constant value, so we need to support some sort of macro replacement in the release handler. In other words if we want to add a Nailgun's release with the following orchestrator data:: { "puppet_manifests_source": "rsync://{MASTER_IP}:/puppet/{OPENSTACK_VERSION}/manifests/", "puppet_modules_source": "rsync://{MASTER_IP}:/puppet/{OPENSTACK_VERSION}/modules/", "repo_metadata": { ... } } the ``{MASTER_IP}`` and the ``{OPENSTACK_VERSION}`` will be replaced by Nailgun with an actual master node ip address and an openstack version respectively. Upgrade impact -------------- The Fuel Upgrade script has to be able to add new "OpenStack releases" with one, two or more repos. Security impact --------------- None Notifications impact -------------------- None Other end user impact --------------------- None Performance Impact ------------------ * Unpacking time of tarball should be reduced. * Downloading time of tarball should be reduced. Other deployer impact --------------------- * Slave nodes may have more than one repo in the operating system. Developer impact ---------------- None Implementation ============== Assignee(s) ----------- Primary assignee: * Igor Kalnitsky * Sergey Kulanov * Vladimir Kozhukalov Other contributors: * Fuel OSCI * Fuel DevOps Work Items ---------- * Add ``make centos-diff-repo ubuntu-diff-repo`` targets to build system. * Configure a Jenkins Job for providing diff-based mirrors as an artifact. * Add ability to existing Jenkins Job to build an upgrade tarball with few *openstack bundles*. * Add support of diff-based repos to fuel-upgrade script. * Resolve issue - `LP1376694 `_. Dependencies ============ None Testing ======= Existing Fuel Upgrade / OpenStack patching tests are enough since it's about improvements, not about entirely new feature. But next tests may be added: * Test that there're more than one repo on slaves. * Test that an upgrade tarball contains only changed packages, not all. * Test that a diff-based release passes the same tests as the full release. Acceptance criteria ------------------- * Build an upgrade tarball with a diff-based repo. The file ``upgrade/releases/metadata.yaml`` must be exist. * After master node upgrade a diff-based release must have at least two repos in ``repo_metadata``. * A cluster (with a diff-based release) must be deployed successfully and with packages from a diff-based repo. * Each slave in that cluster must have at least two repos. Documentation Impact ==================== The documentation about release management should be added and it should resolve next questions: * Which components includes a release? * Where are the components stored? * How releases reuse packages from older releases. References ========== * #fuel-dev on freenode