.. This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode ====================================== Separate MOS packages from CentOS ones ====================================== https://blueprints.launchpad.net/fuel/+spec/separate-mos-from-centos -------------------- Problem description -------------------- * As a Cloud Operator I would like to see what RPM packages are provided by Mirantis OpenStack and what RPM packages are provided by base distro * As a Cloud Operator I would like to get security updates as fast as possible independently for base distro as well as for Mirantis OpenStack * As a Package Maintainer I would like to keep track of sources to all Mirantis OpenStack packages and minimize the number of modified ones as much as possible ---------------- Proposed changes ---------------- Web UI ====== None Nailgun ======= Data model ---------- None REST API -------- None Orchestration ============= RPC Protocol ------------ None Fuel Client =========== None Plugins ======= None Fuel Library ============ * Fuel Puppet manifests have to be adjusted to support CentOS 7 deployments ------------ Alternatives ------------ There is no alternative to the repositories separation approach due to considerations related to distribution policies of major OS vendors. Regarding the helper script to download base distro repositories, there could be a different approach implemented, by downloading only particular packages that required by MOS. However, we consider that providing a full upstream repository would make customer experience a bit better, especially in cases when additional upstream packages that are not a part of MOS need to be installed). -------------- Upgrade impact -------------- When Fuel master node is upgraded to a version that supports Linux distro separation, package repositories for old versions of MOS deployed by previous version of Fuel will keep using the old mirror structure. Package repositories for the new versions of MOS will use the structure defined in the mos-rpm-repos-iface_ specification. .. _mos-rpm-repos-iface: https://github.com/stackforge/fuel-specs/blob/master/specs/7.0/mos-rpm-repos-iface.rst --------------- Security impact --------------- None -------------------- Notifications impact -------------------- None --------------- End user impact --------------- None ------------------ Performance impact ------------------ None ----------------- Deployment impact ----------------- Changes described in this document allow to increase product flexibility, by making possible to choose an operating system and install it independent of MOS. ---------------- Developer impact ---------------- None -------------------------------- Infrastructure impact -------------------------------- * The mixed repository that contains both upstream OS and MOS RPM packages ("fwm") will be no longer propagated across the MOS mirrors. * Jobs for creating a product ISO will be modified to reflect changes to the repositories structure -------------------- Documentation impact -------------------- None ------------------------------- Package versioning requirements ------------------------------- Package version string, as well as package metadata for a *MOS specific* or *divergent* package must not include registered trademarks of base distro vendors, and should include "mos" keyword. ----------------------- RPM packages versioning ----------------------- Package name constructs from:: -- For example:: python-iso8601-0.1.10-1.el7 Where: - python-iso8601 - name - 0.1.10 - version - 1.el7 - release All modifications should be made in release section. **1** - first digits in *release* represents actual package revision/release number and should be incremented in case of package update(spec modification, patching etc). Example:: python-iso8601-0.1.10-1.el7 -> python-iso8601-0.1.10-2.el7 **el7** - represents distribution that was used during package building process and generated by %{?dist} macro. For packages maintained by MOS special suffix *mos* must be add after %{?dist} macro during package build process. This will shows that package belongs to MOS. Example:: python-iso8601-0.1.10-1.el7 -> python-iso8601-0.1.10-1.el7~mos1 **Options/tags should be modified by CI/Build:** Below provided example with options from python-iso8601.spec file:: Name: python-iso8601 Version: 0.1.10 Release: 1%{?dist} CI/Build system should modify *Version:* and *Release:* values before build process to ensure that package version and release represents truth: - *Version:* for **OpenStack projects** must be substituted with last tag in code branch from where package will be built. - *Release:* value should be preserved and concatenated with MOS specific attributes. Example:: was: Release: 1%{?dist} became: Release: 1%{?dist}~mosX This modification leads to transformations as follows:: python-iso8601-0.1.10-1.el7 -> python-iso8601-0.1.10-1.el7~mos1 **Subsequent version:** This number represents amount of commits into code since last tag change in current code branch and must be added after **mos**. Example:: python-heat-2015.2-1.el7~mos123 -> python-heat-2015.2-1.el7~mos124 **Structure of release part for packages maintained by Mirantis:** python-iso8601-0.1.10-1.%{?dist}~mos1 Where: - ~ separator from base Linux distro version - mos - shows that package belongs to MOS and maintained by Mirantis. - X - represents commits number since last tag/branch update in code. For example we have python-iso8601 package with code version = *0.1.10* - package release = *1*, - %{?dist} = Linux distro name(el7), - package maintained by Mirantis = *mos*, - commits number into code within code version 0.1.10 = *1*. Only packages from *security* repository should have security update bundle number at the very end! Regular packages should only have commits number for the very last value in version string. ------------------------------ Backport from external sources ------------------------------ The name and the upstream version of a package backported from external sources (EPEL, newer Fedora releases, etc) - name and version must be kept. Modification required for *release* part, initial revision of a package also should be preserved. Any further modifications of package will be represented in commits number which follows after *mos*. By default this value will be always set to 1 and will be increased in case of package modification. Example:: python-iso8601-0.1.10-1.el7 -> python-iso8601-0.1.10-1.el7~mos1 python-iso8601-0.1.10-1.el7~mos1 -> python-iso8601-0.1.10-1.el7~mos2 -------------- Package update -------------- If required to update package SPEC file or add patch or make any other modifications not related to code version update, package revision / release number must be increased. If a major change (new version of the software being packaged) occurs, the version number is changed to reflect the new software version, and the release number is reset to 1. In case of packages maintained by MOS this is **valid for OpenStack** projects. For **non OpenStack** projects, like dependencies and back-ported packages all updates will be represented in commits number part of release. After code version update Commits number value resets to 1 and will be increased in cases of further modifications of a package. Update of dependencies within one code version(*non OpenStack*):: python-iso8601-0.1.10-1.el7~mos1 -> python-iso8601-0.1.10-1.el7~mos2 Update of dependencies in case of code version update(*non OpenStack*):: python-iso8601-0.1.10-1.el7~mos1 -> python-iso8601-0.1.11-1.el7~mos1 Update of OpenStack project - SPEC changed:: python-heat-2015.2-1.el7~mos123 -> python-heat-2015.2-2.el7~mos123 Update of OpenStack project - code tag/branch changed:: python-heat-2015.2-1.el7~mos123 -> python-heat-2015.3-1.el7~mos0 ---------------------------------------------- Versioning of packages in post-release updates ---------------------------------------------- **Updates:** Since MOS reaches GA status, ie officially released, all updated packages will be published into separate *updates* repository. Updated package will have higher commit number value in the release part then package from stable repository. Example:: python-iso8601-0.1.10-1.el7~mos200 -> python-iso8601-0.1.11-1.el7~mos201 python-heat-2015.2-1.el7~mos200 -> python-heat-2015.2-1.el7~mos201 **Security updates:** Security updates will also be published in a separate repository and based on package from *updates* repository. Additional subsequent tag will be added to the version of a package which includes ".sec." prefix followed by the security bundle number. Example:: python-iso8601-0.1.10-1.el7~mos201 -> python-iso8601-0.1.11-1.el7~mos201.1 python-heat-2015.2-1.el7~mos201 -> python-heat-2015.2-1.el7~mos201.1 **Work with branches within updates:** Branches example: - openstack-ci/fuel-8.0/stable - freezes after GA - openstack-ci/fuel-8.0/updates - branch for maintenance updates between main releases - openstack-ci/fuel-8.0/security-1 - branch for security updates Any changes into *updates* and *security* branches are undergoing the full acceptance cycle. All *security* fixes should be proposed to particular branches called "security-" where ID corresponds to the number of a current update bundle. These branches must be based on a particular commits that correspond to the previously released version of a package. Such branches are generated each time when a fix is based on a code released in terms of a current update bundle. Example for python-iso8601 0.1.10 package: Stable branch:: project: python-iso8601 branch: openstack-ci/fuel-8.0/stable number of commits: 1 tag: 0.1.10 After GA, *stable* branch should be frozen and do not accept any changes. All further work is moving into "updates" branch, this means all next maintenance updates will be published from this branch. Updates branch:: project: python-iso8601 branch: openstack-ci/fuel-8.0/updates number of commits: 2 tag: 0.1.10 In case of critical vulnerabilities found for project, the *security-1* branch in the python-iso8601 project will be created, pointing to the same commit from which the GA version of python-iso8601 was built. Patches will be committed into the *security-1* branch, built package will be published into security-updates package repositories and also pushed into *updates* branches to keep these changes. Security updates branch:: project: python-iso8601 branch: security-1 number of commits: 2 security update tag: 1 tag: 0.1.10 Transformations within ongoing MOS releases as for dependencies as for OpenStack projects:: mos8.0: python-iso8601-0.1.10-1~mos1 mos8.0: python-heat-2015.2-1.el7~mos1 mos8.0-updates: python-iso8601-0.1.10-1~mos2 mos8.0-updates: python-heat-2015.2-1.el7~mos2 mos8.0-security-updates: python-iso8601-0.1.10-1~mos2.1 mos8.0-security-updates: python-heat-2015.2-1.el7~mos2.1 Next version of MOS released: mos9.0: python-iso8601-0.1.10-1~mos3 mos9.0: python-heat-2015.2-1.el7~mos3 All the current security fixes should be included into upcoming update bundle. This means that if a new security fix gets into repository while new update bundle is going through acceptance testing, the update bundle code should include this fix, acceptance testing should be reset and new update bundle should be retested again. ------------------------------ Prioritization of repositories ------------------------------ From out of the box *YUM* package manager has no ability to use repository priorities. This functionality is accessible via yum plugin named **yum-plugin-priorities** and accessible from Base repository. Also this makes us able to use priorities for *Holdback* repositories. --------------------------------- EXTRA_RPM_REPOS variable behavior --------------------------------- In the current Fuel make system, the EXTRA_RPM_REPO variable is used to specify one or more yum repositories to be used as additional sources during creation of mixed repository that consists of MOS and upstream packages. EXTRA_RPM_REPOS is a space-separated list of: name,url,priority With the introduction of separate MOS and upstream repositories, the approach to handling of the EXTRA_RPM_REPOS variable will change as well. The following extra actions will be taken for each of yum repositories specified in the EXTRA_RPM_REPOS variable: * download to the local mirror created during an ISO build process using reposync tool from yum-utils, along with the respecive comps.xml Downloaded repository is placed to the $(LOCAL_MIRROR)/extra-repos/$reponame folder, where $reponame is the repo name taken from EXTRA_RPM_REPOS * rebuild metadata with the createrepo command * copy extra repository to the "extra-repos" folder in the ISO root * add repo entry to the Fuel node kickstart file (ks.cfg) with the priority specified in the EXTRA_RPM_REPOS variable * create extra.repo file in yum.conf(5) format to be used on the Fuel node .. note:: In current implementation, source packages and debug symbols packages will be excluded during repositories download, to save diskspace on the ISO. Repositories metadata will be rebuilt automatically in order to keep the consistency. **Note** As this approach is not fully aligned with the artifacts separation strategy, it will be revised in the upcoming implementations. Extra repositories added to the Fuel node kickstart, will be used both during node provisioning, and node deployment. -------------- Implementation -------------- Assignee(s) =========== Primary assignee: Vitaly Parakhin QA assignee: TBD Other contributors: TBD Mandatory design review: Roman Vyalov Vladimir Kozhukalov Work Items ========== * Modify MOS mirroring Jenkins jobs * Modify ISO creation Jenkins jobs * Modify make system to allow Fuel node installation from multiple repositories Dependencies ============ ------------ Testing, QA ------------ TBD Acceptance criteria =================== * Fuel node can be installed from an ISO with multiple RPM repositories * Only MOS RPM packages are stored on Fuel mirrors ---------- References ---------- TBD