MOS APT repositories: URLs, metadata, and other interface details

Improve the API (URLs and metadata) of MOS APT repositories

Problem description

The APT URLs and repository metadata is kind of an API (a contract between the repo users and its maintainers). Quite a lot of Fuel components depend on this interface. Building IBP target images, bootstrap images, and regular OpenStack deployment is going to break if APT URLs or repository metadata (such as a codename) gets changed.

Currently the codename is bound to the MOS release number, that is, the repo URLs look like (as documented in separate_mos_from_linux)

deb http://${host}/mos/ubuntu mos${version}          main
deb http://${host}/mos/ubuntu mos${version}-security main
deb http://${host}/mos/ubuntu mos${version}-updates  main
deb http://${host}/mos/ubuntu mos${version}-proposed main
deb http://${host}/mos/ubuntu mos${version}-holdback main

This stucture yields several issues:

• it’s impossible to distinguish between repositories targeted for different Ubuntu versions (i.e. for trusty and vivid)
• it’s impossible to distinguish between Ubuntu and Debian
• it’s difficult to support per customer repositories

Proposed change

Change the codename to mos${version}[-${distro_codename}], so the URLs are

deb http://${host}/mos-repos/${distro}/{version} mos${version}          main
deb http://${host}/mos-repos/${distro}/{version} mos${version}-security main
deb http://${host}/mos-repos/${distro}/{version} mos${version}-updates  main
deb http://${host}/mos-repos/${distro}/{version} mos${version}-proposed main
deb http://${host}/mos-repos/${distro}/{version} mos${version}-holdback main

and the repository metadata is

Origin: Mirantis Codename: mos${version} Label: mos${version} Suite: mos${version}-${component}

Example: MOS 7.0

deb http://${host}/mos-repos/ubuntu/7.0 mos7.0          main
deb http://${host}/mos-repos/ubuntu/7.0 mos7.0-security main
deb http://${host}/mos-repos/ubuntu/7.0 mos7.0-updates  main
deb http://${host}/mos-repos/ubuntu/7.0 mos7.0-proposed main
deb http://${host}/mos-repos/ubuntu/7.0 mos7.0-holdback main

The Release files are:

Origin: Mirantis Codename: mos7.0 Label: mos7.0 Suite: mos7.0{,-security,-updates,-proposed,-holdback}

Example: MOS 7.0/vivid

deb http://${host}/mos-repos/ubuntu/7.0 mos7.0-vivid          main
deb http://${host}/mos-repos/ubuntu/7.0 mos7.0-vivid-security main
deb http://${host}/mos-repos/ubuntu/7.0 mos7.0-vivid-updates  main
deb http://${host}/mos-repos/ubuntu/7.0 mos7.0-vivid-proposed main
deb http://${host}/mos-repos/ubuntu/7.0 mos7.0-vivid-holdback main

The Release files are:

Origin: Mirantis Codename: mos7.0-vivid Label: mos7.0-vivid Suite: mos7.0-vivid{,-security,-updates,-proposed,-holdback}

Example: MOS 7.0/vivid-fuel

deb http://${host}/mos-repos/ubuntu/7.0 mos7.0-vivid-fuel          main
deb http://${host}/mos-repos/ubuntu/7.0 mos7.0-vivid-fuel-security main
deb http://${host}/mos-repos/ubuntu/7.0 mos7.0-vivid-fuel-updates  main
deb http://${host}/mos-repos/ubuntu/7.0 mos7.0-vivid-fuel-proposed main
deb http://${host}/mos-repos/ubuntu/7.0 mos7.0-vivid-fuel-holdback main

The Release files are:

Origin: Mirantis Codename: mos7.0-vivid-fuel Label: mos7.0-vivid-fuel Suite: mos7.0-vivid-fuel{,-security,-updates,-proposed,-holdback}

Example: Customer 7.0

deb http://${host}/customer/ubuntu/7.0 customer7.0          main
deb http://${host}/customer/ubuntu/7.0 customer7.0-security main
deb http://${host}/customer/ubuntu/7.0 customer7.0-updates  main
deb http://${host}/customer/ubuntu/7.0 customer7.0-proposed main
deb http://${host}/customer/ubuntu/7.0 customer7.0-holdback main

The Release files are:

Origin: Customer Codename: customer7.0 Label: customer7.0 Suite: customer7.0{,-security,-updates,-proposed,-holdback}

Advantages:

• MOS release can target arbitrary number of Ubuntu/Debian versions (limited only by available resources).
• It’s possible to create arbitrary number of per customer (or per team) APT repositories using codenames and custom url, but still keeping the overall structure.
• It’s possible to maintain a separate set of repositories which are not intended for OpenStack nodes (say, packages relevant for Fuel master node only).

Alternatives

Decouple the codename from the MOS release number and use the OpenStack release codename instead, i.e

deb http://${host}/mos-repos/ubuntu/7.0 kilo-trusty main

Data model impact

Default set of APT repositories for OpenStack nodes should be changed.

REST API impact

None.

Upgrade impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

None.

Plugin impact

None.

Other deployer impact

EXTRA_DEB_REPOS should provide a compatible metadata in order for repo priorities to work properly.

Developer impact

None.

Infrastructure impact

Implementation

We need to update every release in transaction way. Each release should be a symlink to particular snapshot:

mos-repos/ubuntu/{version} -> snapshots/{version}-{datetime} mos-repos/ubuntu/snapshots/{version}-{datetime}

Each snapshot should contain all the data related to corresponding relese

mos-repos/ubuntu/snapshots/{version}-{datetime}/
 ├─ dists
 │  ├─ mos7.0
 │  │  ├─ main
 │  │  ├─ resticted
 │  │  ├─ Release
 │  │  └─ Release.gpg
 │  └─ mos7.0-updates
 │     ├─ main
 │     ├─ resticted
 │     ├─ Release
 │     └─ Release.gpg
 └─ pool

Updating steps:

• create new snapshot: snapshots/{version}-{newdatetime}/{dists,pool} based on previous one (in order to reduce uploading traffic, all unchanged files will be linked from previous snapshot with rsync --link-dest option)
• update {version} symlink to new snapshot {version} -> snapshots/{version}-{newdatetime}

As far as current development suite is updating very often (up to ten times per minute), we need a way to freeze its state for all CI processes. We could use snapshots as freezed suite state. Just dereference current suite symlink to actual snapshot. In order to get the actual target of symlink we need to have a kind of dereference mechanism. It can be plain text file in the same directory:

• mos-repos/ubuntu/{version}.target.txt

which contains target of {version} symlink:

• snapshots/{version}-{timestamp}

We could use this value instead of symlink:

• current repository string: deb {host}/mos-repos/ubuntu/{version} {suite} main
• dereference suite symlink: {version} -> snapshots/{version}-{datetime}
• new repository string: deb {host}/mos-repos/ubuntu/snapshots/{version}-{datetime} {suite} main

Assignee(s)

Work Items

Dependencies

None.

Testing

Acceptance criteria

Documentation Impact

• New APT URLs and repo metadata (Release files) should be documented so people can create their repositories the right way.

References