TripleO Repos Single Source

This proposal lays out the plan to use tripleo-repos as a single source to install and configure non-base OS repos for TripleO - including setting the required DLRN hashes.

https://blueprints.launchpad.net/tripleo/+spec/tripleo-repos-single-source

Problem Description

Reviewing the code base, there are multiple places where repos are specified. For example,in the release files we are setting the configuration that is applied by repo setup role. Some of the other repo/version configurations are included in:

The process of setting repo versions requires getting and transforming DLRN hashes, for example resolving ‘current-tripleo’ to a particular DLRN build ID and specifying the correct proxies. Currently a large portion of this work is done in the release files resulting in sections of complicated and fragile Bash scripts - duplicated across numerous release files.

This duplication, coupled with the various locations in use for setting repo configurations, modules and supported versions is confusing and error prone.

There should be one source of truth for which repos are installed within a tripleo deployment and how they are installed. Single-sourcing all these functions will avoid the current problems of duplication, over-writing settings and version confusion.

Proposed Change

Overview

This proposal puts forward using tripleo-repos as the ‘source of truth’ for setting repo configurations, modules and supported versions - including setting the DLRN hashes required to specify exact repo versions to install for upstream development/CI workflows.

Having a single source of truth for repo config, modules, etc. will make development and testing more consistent, reliable and easier to debug.

The intent is to use the existing tripleo-repos repo for this work and not to create a new repo. It is as yet to be determined if we will add a v2/versioned api or how we will handle the integration with the existing functionality there.

We aim to modularize the design and implementation of the proposed tripleo-repos work. Two sub systems in particular have been identified that can be implemented independently of, and ultimately to be consumed by, tripleo-repos; the resolution of delorean build hashes from known tags (i.e. resolving ‘current-tripleo’ to a particular DLRN build ID) and the configuration of dnf repos and modules will be implemented as independent python modules, with their own unit tests, clis, ansible modules etc.

Integration Points

The new work in tripleo-repos will have to support with all the cases currently in use and will have to integrate with:

  • DLRN Repos

  • release files

  • container and overcloud image builds

  • rdo config

  • yum/dnf repos and modules

  • Ansible (Ansible module)

  • promotion pipeline - ensuring the correct DLRN hashes

Incorporating the DLRN hash functionality makes the tool more complex. Unit tests will be required to guard against frequent breakages. This is one of the reasons that we decided to split this DLRN hash resolution into its own dedicated python module ‘tripleo-get-hash’ for which we can have independent unit tests.

The scope of the new tripleo-repos tool will be limited to upstream development/CI workflows.

Alternatives

Functionality to set repos, modules and versions is already available today. It would be possible to leave the status quo or:

  • Use rdo config to set one version per release - however, this would not address the issue of changing DLRN hashes

  • Create an rpm that lays down /etc/tripleo-release where container-tools could be meta data in with that, similar to /etc/os-release

Security Impact

No security impact is anticipated. The work is currently in the tripleo open-source repos and will remain there - just in a consolidated place and format.

Upgrade Impact

Currently there will be no upgrade impact. The new CLI will support all release versions under support and in use. At a later date, when the old CLI is deprecated there may be some update implications.

However,there may be work to make the emit_releases_file https://opendev.org/openstack/tripleo-ci/src/branch/master/scripts/emit_releases_file/emit_releases_file.py functionality compatible with the new CLI.

Other End User Impact

Work done on the new project branch will offer a different version of CLI, v2. End users would be able to select which version of the CLI to use - until the old CLI is deprecated.

Performance Impact

No performance impact is expected. Possible performance improvements could result from ensuring that proxy handling (release file, mirrors, rdoproject) is done correctly and consistently.

Other Deployer Impact

Developer Impact

See `Other End User Impact` section.

Implementation

The functionality added to tripleo-repos will be writen as a Python module with a CLI and will be able to perform the following primary functions:

  • Single source the installation of all TripleO related repos

  • Include the functionality current available in the repo-setup role including creating repos from templates and files

  • Perform proxy handling such as is done in the release files (mirrors, using rdoproject for DLRN repos)

  • Get and transform human-readable DLRN hashes - to be implemented as an independent module.

  • Support setting yum modules such as container-tools - to be implemented as an independent module.

  • Support enabling and disabling repos and setting their priorities

The repo-setup role shall remain but it will invoke tripleo-repos. All options required to be passed to tripleo-repos should be in the release file.

Work done on the new project branch will offer a different version of CLI, v2. Unit tests will be added on this branch to test the new CLI directly. CI would be flipped to run in the new branch when approved by TripleO teams. All current unit tests should pass with the new code.

An Ansible module will be added to call the tripleo-repos options from Ansible directly without requiring the end user to invoke the Python CLI from within Ansible.

The aim is for tripleo-repos to be the single source for all repo related configuration. In particular the goal is to serve the following 3 personas:

  • Upstream/OpenStack CI jobs

  • Downstream/OSP/RHEL jobs

  • Customer installations

The configuration required to serve each of these use cases is slightly different. In upstream CI jobs we need to configure the latest current-tripleo promoted content repos. In downstream/OSP jobs we need to use rhos-release and in customer installations we need to use subscription manager.

Because of these differing requirements we are leaning towards storing the configuration for each in their intended locations, with the upstream config being the ‘base’ and the downstream config building ontop of that (the implication is that some form of inheritance will be used to avoid duplication). This was discussed during the Xena PTG session

Assignee(s)

  • sshnaidm (DF and CI)

  • marios (CI and W-release PTL)

  • weshay

  • chandankumar

  • ysandeep

  • arxcruz

  • rlandy

  • other DF members (cloudnull)

Work Items

Proposed Schedule

Investigative work will be begin in the W-release cycle on a project branch in tripleo-repos. The spec will be put forward for approval in the X-release cycle and impactful and integration work will be visible once the spec is approved.

Dependencies

This work has a dependency on the DLRN API and on yum/dnf.

Testing

Specific unit tests will be added with the python-based code built. All current CI tests will run through this work and will test it on all releases and in various aspects such as:

  • container build

  • overcloud image build

  • TripleO deployments (standalone, multinode, scenarios, OVB)

  • updates and upgrades

CLI Design

Here is an abstract sketch of the intended cli design for the new tripleo-repos.

It covers most of the needs discussed at multiple places.

Scenario 1

The goal is to construct a repo with the correct hash for an integration or a component pipeline.

For this scenario:

  • Any combination of hash, distro, commit, release, promotion, url parameters can passed

  • Use the tripleo-get-hash module to determine the DLRN build ID

  • Use the calculated DLRN build ID to create and add a repo

Scenario 2

The goal is to construct any type of yum/dnf repo.

For this scenario:

  • Construct and add a yum/dnf repo using a combination of the following parameters

  • filename - filename for saving the resulting repo (mandatory)

  • reponame - name of repository (mandatory)

  • baseurl - base URL of the repository (mandatory)

  • down_url - URL to download repo file from (mandatory/multually exclusive to baseurl)

  • priority - priority of resulting repo (optional)

  • enabled - 0/1 whether the repo is enabled or not (default: 1 - enabled)

  • gpgcheck - whether to check GPG keys for repo (default: 0 - don’t check)

  • module_hotfixes - whether to make all RPMs from the repository available (default: 0)

  • sslverify - whether to use a cert to use repo metadata (default: 1)

  • type - type of the repo(default: generic, others: custom and file)

Scenario 3

The goal is to enable or disable specific dnf module and also install or remove a specific package.

For this scenario:

  • Specify

  • module name

  • which version to disable

  • which version to enable

  • which specific package from the module to install (optional)

Scenario 4

The goal is to enable or disable some repos, remove any associated repo files no longer needed, and then perform a system update.

For this scenario:

  • Specify

  • repo names to be disabled

  • repo names to be enabled

  • the files to be removed

  • whether to perform the system update

Documentation Impact

tripleo-docs will be updated to point to the new supported repo/modules/versions setting workflow in tripleo-repos.

References to old sources of settings such as tripleo-ansible, release files in tripleo-quickstart and the repo-setup role will have to be removed and replaced to point to the new workflow.