Upgrade Fuel Admin node 7.0 to 8.0 with CentOS7
https://blueprints.launchpad.net/fuel/+spec/upgrade-master-node-centos7
Upgrade of the Fuel Admin node is a way to deliver latest version of
Fuel installer and OpenStack to users without breaking their existing
environments. It is necessary to accommodate the upgrade method for
Fuel Admin node to changes made in 8.0 release cycle.
This spec is addressing only the upgrade of the Fuel Admin node up to
the point when it is able to deploy and manage environments with the
new version of OpenStack. Upgrade of the OpenStack environments is out
of scope of this proposal.
Problem description
In release 8.0, CentOS 7 is introduced as a base operating system for the
Fuel Admin node, replacing CentOS 6.6 in release 7.0.
Due to the level of customization of the basic operating system in release
7.0, it is impossible to upgrade the operating system by standard tools
(e.g. YUM package management system).
Users of Fuel installer should be able to keep their OpenStack environments
managed by Fuel through the upgrade of the Fuel Admin node.
Proposed changes
Current approach to the upgrade in release 7.0 assumes that the new version
of docker images are packaged in RPM. Upgrade script installs the package and
rebuilds the containters.
New approach shall be data-driven in a sense that the Fuel Admin node
will be reinstalled from scratch, while configuration data of OpenStack
environments shall be retained and applied to the fresh installation of
Fuel. This allows the Fuel Admin node to manage existing OpenStack
clusters deployed by previous versions of Fuel.
For data-driven upgrade of the Fuel Admin node, the following procedure
must be developed:
- Backup configuration of the Fuel Admin node and all OpenStack clusters
managed by this node, including the following items:
- /etc/fuel/ directory that contains configuration files for the
Fuel Admin node’s installation and operation.
- /etc/puppet/ directory to preserve Puppet modules for former
release(s) to retain support for old clusters.
- /root/.ssh/ directory contains private/public keys used to access
all target nodes in environments.
- /var/lib/cobbler/config/systems.d directory contains configuration
files for target nodes discovered by the Fuel Admin node. File
default.json contains definition for bootstraping nodes and must not
be backed up or restored in the new release version.
- /var/lib/fuel/keys/ certificates and keys for SSL/TLS
encryption of web UI traffic.
- text dump of nailgun database from Postgres server
- text dump of keystone database from Postgres server
- ostf database is not backed up/restored, since its contents
are incremental and the new release DB already has everything for
legacy releases
- fuel-octane tool shall be used for backup purposes. The original
version shall be patched and published as an update to 7.0 release. User
installs it in the host and uses it to create backup tarball.
- Backup log files separately from the configuration, since they could be
bulky and will slow down the backup/restore process. Logs backup can
be done via Nailgun diagnostic snapshot as usual.
- Install the Fuel Admin node as usual.
- Supply file astute.yaml from backup to the freshly installed Fuel
Admin node via octane fuel-restore command. It will restore
credentials from the supplied /etc/fuel/astute.yaml file.
- Command octane fuel-restore shall inject database dump into
PostgreSQL DB and run migration scripts on the database.
- Command octane fuel-restore shall upload openstack.yaml fixture
with metadata of the new release via Nailgun API.
- After installation of the Fuel Admin node, inject Cobbler configuration
files, keys and certs in corresponding places and restart appropriate
services.
- After configuration is restored, rebuild the bootstrap image with restored
SSH keys.
Plugins
Fuel Operator will have to install plugins of the compatible version onto
the Fuel Admin node after the restore is done.
Alternatives
Alternative way is to backup and restore Docker containers, as per current
version of Fuel Admin node backup. Given the decision to drop containers
support and additional value of the data-based backup/restore for cases when
the master node is reinstalled, we abstain from this approach.
Another path is to upgrade operating system of the Fuel Admin node in-place
with centos-upgrade-tool provided by CentOS. This path is unstable
considering modifications to base operating system, including changed package
versions and rebuilt packages. It will require modifications to the upgrade
tool and supporting third-party upgrade scripts.
Upgrade impact
This proposal covers change of upgrade workflow. The new workflow shall
reuse the database upgrade capabilities provided by Alembic migrations
in Nailgun. Migrations will be applied automatically with the restart of
container docker-nailgun.
Security impact
Backup file contains high sensitive data, including SSH private keys and
access credentials to all components in both Fuel and OpenStack environments.
This file must be handled with extreme care. It must not be published to
externally accessible location (e.g. HTTP server). Preferred way to transfer
the file between old and new instance of the Fuel Admin node is removable
storage device.
Backup file shall be encrypted and protected with user-supplied secret.
Root password for the fresh installation of the Fuel 8.0 Admin node must be
changed via fuel-menu or immediately after the installation. fuel-octane
backup shall not save /etc/passwd and /etc/shadow files and thus
won’t preserve root password and/or other users credentials from the original
node.
Notifications impact
None.
End user impact
User experience for upgrading Fuel Admin node changes significantly. Instead
of running single shell script upgrade.sh, operator will have to:
- install new version of fuel-octane package from mos-updates
repository onto Fuel Admin node version 7.0.
- create backup with octane fuel-backup command from fuel-octane
package and copy resulting file to external location (e.g. USB drive or
another server).
- install new Fuel Admin node with version 8.0 using elements of backup in
installation procedure. Operator have 2 options:
- install the new 8.0 Fuel Admin node onto existing physical server or VM,
replacing the original 7.0 Fuel Admin node.
- install the Fuel Admin node onto new physical/virtual server in parallel
with the original 7.0 Fuel Admin node.
- use octane fuel-restore to extract the backup file and restore
credentials and other settings of the Admin node from astute.yaml
from the backup and inject remaining elements of backup into
fresh installation of the Fuel Admin node.
Rollback option is available through installing fresh 7.0 Fuel Admin node and
restoring configuration on that node according to the procedure outlined
in Operations Documentation (see Documentation Impact section for details).
Deployment impact
Fuel 8.0 Admin node is installed as usual. The restoration procedure must
be applied immediately afterwards, before any changes to settings of the
new Admin node.
This proposal doesn’t impact the deployment of new OpenStack environments.
To be able to scale and manage environments of version 7.0 using upgraded
Fuel Admin, operator must save repositories with OpenStack packages from
the 7.0 Admin node and copy them to the 8.0 Admin node. Workflow for the
repository transfer is described in Operations Guide.
Infrastructure impact
To accommodate to this change, an extra virtual machine has to be used in the
testing environment when testing this function and the upgrade of an OpenStack
environment following the upgrade of the Fuel Admin node.
Documentation impact
- New upgrade workflow shall be documented in respective section of Operations
Guide.
- New rollback workflow based on octane fuel-restore shall be described
in Operations Guide:
- ** To rollback the Fuel Admin installation, deploy fresh Fuel 7.0 Admin node,
- retrieve backup of Docker containers from external store and use command
octane fuel-restore to restore configuration and data of containers.
- The transfer mechanism for repository with OpenStack packages shall be
described in Operations Guide.
Implementation
Assignee(s)
- Primary assignee:
- sryabin
- Other contributors:
- gelbuhos
- Mandatory design review:
- vkozhukalov
sgolovatyuk
ikalnitskiy
dborodaenko
Work Items
- Implement octane fuel-backup command to create a backup file from
the Fuel Admin node
- Implement injection of astute.yaml file into boostrap process of the
Fuel Admin node in octane fuel-restore utility
- Implement injection of DB dump from backup file into PostgreSQL database
server during the bootstrap in octane fuel-restore command
- Implement restore of Cobbler configuration files and key/cert files from
backup in octane fuel-restore command
- Implement rebuild of Docker containers with the restored data in
octane fuel-restore command
- Implement upload of openstack.yaml fixtures for 8.0 release using
octane fuel-restore command
- Implement translations for the backup data according to Predictable
Interfaces Naming feature in fuel-web DB migration scriptsa
- Implement post-restore actions to:
** Upload releases data for 8.0 release from openstack.yaml file
- ** Rebuild bootstrap image with restored SSH public key for accessing
- target node
- Implement system test to verify the new upgrade workflow
- Prepare documentation on the new upgrade workflow
Dependencies
- Centos7 on the Fuel Admin node
- Enable Predictable Interfaces Naming schema
Testing, QA
- Current test plans must be be updated with new upgrade procedure.
- Rollback-scenarios must be adapted for using restore feature.
- New tests must be written for covering upgrading cluster with new features
introduced in 7.0:
- Network templates
- Node groups
- Separate services
- Node reinstallation
- Chain-upgrade scenarious for upgrading fuel master node 6.1->7.0->8.0
must be written to ensure the ability to manage Kilo cluster with
deprecated or removed features:
- nova-network FlatDHCP
- Neutron GRE network
- CentOS as base OS for cluster
- Classic provisioning
Acceptance criteria
- Backup created by octane fuel-backup command is a tarball that
contains all files and data according to this specification on the
Fuel Admin node of version 7.0.
- On fresh installation of Fuel 8.0 Admin node, octane fuel-restore
command restores contents of /etc/fuel/astute.yaml file in the
following sections from values in backup astute.yaml file:
- HOSTNAME
- DNS_* parameters
- ADMIN_NETWORK
- FUEL_ACCESS
- FEATURE_GROUPS
- keystone credentials
- postgres credentials
- astute credentials
- cobbler credentials
- mcollective credentials
- Command octane fuel-restore uploads data from the backup to nailgun
and keystone databases at Fuel 8.0 Admin node.
- Configuration files in systems.d directory of Cobbler configuration
directory restored from backup and match the actual nodes in the test
environment.
- Proper access credentials are restored across the system, including DB
accounts, SSH keys and certificates for Cobbler and Nginx.
- Changes implementing the functions listed above are properly submitted,
reviewed and merged into fuel-octane repository.
- Documentation describing the new upgrade workflow submitted and merged
in the main Fuel documentation.