Simple Container Generation

Simple container generation is an initiative to reduce complexity in the TripleO container build, deployment, and distribution process by reducing the size and scope of the TripleO container build tools.

The primary objective of this initiative is to replace Kolla, and our associated Kolla customization tools, as the selected container generation tool-kit. The TripleO community has long desired an easier solution for deployers and integrators alike and this initiative is making that desire a reality.

The Simple container generation initiative is wanting to pivot from a tool-chain mired between a foundational component of Kolla-Ansible and a general purpose container build system, to a vertically integrated solution that is only constructing what TripleO needs, in a minimally invasive, and simple to understand way.

3

Problem Description

TripleO currently leverages Kolla to produce container images. These images are built for Kolla-Ansible using an opinionated build process which has general purpose features. While our current images work, they’re large and not well suited for the TripleO use-case, especially in distributed data-centers. The issue of container complexity and size impacts three major groups, deployers, third party integrators, and maintainers. As the project is aiming to simplify interactions across the stack, the container life cycle and build process has been identified as something that needs to evolve. The TripleO project needs something vertically integrated which produces smaller images, that are easier to maintain, with far fewer gyrations required to tailor images to our needs.

Proposed Change

Overview

Implement a container file generation role, and a set of statically defined override variable files which are used to generate our required container files. 2

Layering

tripleo-base+---+
                |
                |
                +---+-openstack-${SERVICE}-1-common-+-->openstack-${SERVICE}-1-a
                    |                               |
                    |                               +-->openstack-${SERVICE}-1-b
                    |                               |
                    |                               +-->openstack-${SERVICE}-1-c
                    +-->openstack-${SERVICE}-2
                    |
                    +-->ancillary-${SERVICE}-1
                    |
                    +-->ancillary-${SERVICE}-2

User Experience

Building the standard set of images will be done through a simple command line interface using the TripleO python client.

$ openstack tripleo container image build [opts] <args>

This simple sub-command will provide users the ability to construct images as needed, generate container files, and debug runtime issues.

CLI Options

The python TripleO client options for the new container image build entry point.

Option

Default

Description

config-file

$PATH/overcloud_containers.yaml

Configuration file setting the list of containers to build.

exclude

[]

Container type exclude. Can be specified multiple times.

work-dir

/tmp/container-builds

Container builds directory, storing the container files and logs for each image and its dependencies.

skip-push

False

Skip pushing images to the registry

skip-build

False

Only generates container files without producing a local build.

base

centos

Base image name.

type

binary

Image type.

tag

latest

Image tag.

registry

localhost

Container registry URL.

namespace

tripleomaster

Container registry namespace.

volume

[]

Container bind mount used when building the image. Should be specified multiple times if multiple volumes.

Container Image Build Tools

Container images will be built using Buildah, The required Buildah functionality will leverage BuildahBuilder via python-tripleoclient integration and be exposed though CLI options.

Image layout

Each image will have its own YAML file which has access to the following parameters. Each YAML file will have one required parameter (tcib_from for the source image to build from) and optional parameters.

Option

Default

Type

Required

Description

tcib_path

{{ lookup(‘env’, ‘HOME’) }}

String

Path to generated the container file(s) for a given image.

tcib_args

Dict[str, str]

Single level key:value pairs. Implements arg.

tcib_from

centos:8

Str

True

Container image to deploy from. Implements from.

tcib_labels

Dict[str, str]

Single level key:value pairs. Implements label.

tcib_envs

Dict[str, str]

Single level key:value pairs. Implements env.

tcib_onbuilds

List[str]

<item>=String. Implements onbuild.

tcib_volumes

List[str]

<item>=String. Implements volume.

tcib_workdir

Str

Implements workdir.

tcib_adds

List[str]

<item>=String. Implements add.

tcib_copies

List[str]

<item>=String. Implements copy.

tcib_exposes

List[str]

<item>=String. Implements expose.

tcib_user

Str

Implements user.

tcib_shell

Str

Implements shell.

tcib_runs

List[str]

<item>=String. Implements run.

tcib_healthcheck

Str

Implements healthcheck.

tcib_stopsignal

Str

Implements stopsignal.

tcib_entrypoint

Str

Implements entrypoint.

tcib_cmd

Str

Implements cmd.

tcib_actions

List[Dict[str, str]]

Each item is a Single level Dictionary key:value pairs. Allows for arbitrary verbs which maintains ordering.

tcib_gather_files

List[str]

Each item is a String. Collects files from the host and stores them in the build directory.

Application packages are sorted within each container configuration file. This provides a programmatic interface to derive package sets, allows overrides, and is easily visualized. While the package option is not processes by the tripleo_container_image_build role, it will serve as a standard within our templates.

Option

Description

tcib_packages

Dictionary of packages to install.

common:
  - openstack-${SERVICE}-common
distro-1:
  common:
    - openstack-${SERVICE}-proprietary
  x86_64:
    - $dep-x86_64
  power:
    - $dep-power
distro-2:
  common:
    - openstack-${SERVICE}
    - $dep

This option is then captured and processed by a simple RUN action.

tcib_actions:
  - run: "dnf install -y {{ tcib_packages['common'] }} {{ tcib_packages[ansible_distribution][ansible_architecture] }}"

Example Container Variable File

tcib_from: ubi8
tcib_path: "{{ lookup('env', 'HOME') }}/example-image"
tcib_labels:
  maintainer: MaintainerX
tcib_entrypoint: dumb-init --single-child --
tcib_stopsignal: SIGTERM
tcib_envs:
  LANG: en_US.UTF-8
tcib_runs:
  - mkdir -p /etc/ssh && touch /etc/ssh/ssh_known_host
tcib_copies:
  - /etc/hosts /opt/hosts
tcib_gather_files:
  - /etc
tcib_packages:
  common:
    - curl
  centos:
    x86_64:
      - wget
tcib_actions:
  - run: "dnf install -y {{ tcib_packages['common'] }} {{ tcib_packages[ansible_distribution][ansible_architecture] }}"
  - copy: /etc/resolv.conf /resolv.conf
  - run: ["/bin/bash", "-c", "echo hello world"]

Container File Structure

The generated container file(s) will follow a simple directory structure which provide an easy way to view, and understand, build relationships and dependencies throughout the stack.

tripleo-base/${CONTAINERFILE}
tripleo-base/ancillary-${SERVICE}-1/${CONTAINERFILE}
tripleo-base/ancillary-${SERVICE}-2/${CONTAINERFILE}
tripleo-base/openstack-${SERVICE}-1-common/${CONTAINERFILE}
tripleo-base/openstack-${SERVICE}-1-common/openstack-${SERVICE}-1-a/${CONTAINERFILE}
tripleo-base/openstack-${SERVICE}-1-common/openstack-${SERVICE}-1-b/${CONTAINERFILE}
tripleo-base/openstack-${SERVICE}-1-common/openstack-${SERVICE}-1-c/${CONTAINERFILE}
tripleo-base/openstack-${SERVICE}-2/${CONTAINERFILE}

Alternatives

  • Use Ansible Bender

Ansible Bender was evaluated as a tool which could help to build the container images. However it has not been productized downstream; which would make it difficult to consume. It doesn’t generate Dockerfiles and there is a strong dependency on Bender tool; the container image build process would therefore be more difficult to do in a standalone environment where Bender isn’t available. 1

  • Leave the container image build process untouched.

We could leave the container image generate process untouched. This keeps us a consumer of Kolla and requires we maintain our complex ancillary tooling to ensure Kolla containers work for TripleO.

Security Impact

While security is not a primary virtue in the simple container generation initiative, security will be improved by moving to simplified containers. If the simple container generation initiative is ratified, all containers used within TripleO will be vertically integrated into the stack, making it possible to easily audit the build tools and all applications, services, and files installed into our containerized runtimes. With simplification we’ll improve the ease of understanding and transparency which makes our project more sustainable, thereby more secure. The proposed solution must provide layers where we know what command has been run exactly; so we can quickly figure out how an image was built.

Upgrade Impact

There is no upgrade impact because the new container images will provide feature parity with the previous ones; they will have the same or similar injected scripts that are used when the containers start.

Other End User Impact

None

Performance Impact

We should expect better performance out of our containers, as they will be smaller. While the runtime will act the same, the software delivery will be faster as the size of each container will smaller, with better constructed layers. Smaller containers will decrease the mean time to ready which will have a positive performance impact and generally improve the user experience.

Other Deployer Impact

The simplified container generation initiative will massively help third party integrators. With simplified container build tools we will be able to easily articulate requirements to folks looking to build on-top of TripleO. Our tool-chain will be capable of bootstrapping applications where required, and simple enough to integrate with a wide variety of custom applications constructed in bespoke formats.

Developer Impact

In the first phase, there won’t be any developer impact because the produced images will be providing the same base layers as before. For example, they will contain all the Kolla scripts that are required to merge configuration files or initialize the container at startup.

These scripts will be injected in the container images for backward compatibility:

  • kolla_extend_start

  • set_configs.py

  • start.sh

  • copy_cacerts.sh

  • httpd_setup.sh

In a second phase, we will simplify these scripts to remove what isn’t needed by TripleO. The interface in the composable services will likely evolve over time. For example kolla_config will become container_config. There is no plan at this time to rewrite the configuration file merge logic.

Implementation

Assignee(s)

Primary assignee:
  • Cloudnull

  • EmilienM

Work Items

First phase

  • Ansible role to generate container file(s) - https://review.opendev.org/#/c/722557

  • Container images layouts - https://review.opendev.org/#/c/722486

  • Deprecate “openstack overcloud container image build”

  • Implement “openstack tripleo container image build” which will reuse the BuildahBuilder and the same logic as the deprecated command but without Kolla.

  • Build new images and publish them.

  • Switch the upstream CI to use the new images.

Second phase:

  • Simplifying the injected scripts to only do what we need in TripleO.

  • Rename the configuration interfaces in TripleO Heat Templates.

Dependencies

The tooling will be in existing repositories so there is no new dependency. It will mainly be in tripleo-ansible, tripleo-common, python-tripleoclient and tripleo-heat-templates. Like before, Buildah will be required to build the images.

Testing

  • The tripleo-build-containers-centos-8 job will be switched to be using the new “openstack tripleo container image build” command.

  • A molecule job will exercise the container image build process using the new role.

  • Some end-to-end job will also be investigated to build and deploy a container into a running deployment.

Documentation Impact

Much of the documentation impact will be focused on cleanup of the existing documentation which references Kolla, and the creation of documentation that highlights the use of the vertically integrated stack.

Since the changes should be transparent for the end-users who just pull images without rebuilding it, the manuals will still be updated with the new command and options if anyone wants to build the images themselves.