Configurable directory for persistent and stateful data

https://blueprints.launchpad.net/tripleo/+spec/tripleo-juno-configurable-mnt-state

Make the hardcoded /mnt/state path for stateful data be configurable.

Problem Description

1. A hard coded directory of /mnt/state for persistent data is incompatible with Red Hat based distros available mechanism for a stateful data path. Red Hat based distros, such as Fedora, RHEL, and CentOS, have a feature that uses bind mounts for mounting paths onto a stateful data partition and does not require manually reconfiguring software to use /mnt/state.

2. Distros that use SELinux have pre-existing policy that allows access to specific paths. Reconfiguring these paths to be under /mnt/state, results in SELinux denials for existing services, requiring additional policy to be written and maintained.

3. Some operators and administrators find the reconfiguring of many services to not use well known default values for filesystem paths to be disruptive and inconsistent. They do not expect these changes when using a distro that they’ve come to learn and anticipate certain configurations. These types of changes also require documentation changes to existing documents and processes.

Proposed Change

Deployers will be able to choose a configurable path instead of the hardcoded value of /mnt/state for the stateful path.

A new element, stateful-path will be added that defines the value for the stateful path. The default will be /mnt/state.

There are 3 areas that need to respect the configurable path:

os-apply-config template generation

The stateful-path element will set the stateful path value by installing a JSON file to a well known location for os-collect-config to use as a local data source. This will require a new local data source collector to be added to os-collect-config (See Dependencies).

The JSON file’s contents will be based on $STATEFUL_PATH, e.g.:

{‘stateful-path’: ‘/mnt/state’}

File templates (files under os-apply-config in an element) will then be updated to replace the hard coded /mnt/state with {{stateful-path}}.

Currently, there is a mix of root locations of the os-apply-config templates. Most are written under /, although some are written under /mnt/state. The /mnt/state is hard coded in the directory tree under os-apply-config in these elements, so this will be removed to have the templates just written under /. Symlinks could instead be used in these elements to setup the correct paths. Support can also be added to os-apply-config’s control file mechanism to indicate these files should be written under the stateful path. An example patch that does this is at: https://review.openstack.org/#/c/113651/

os-refresh-config scripts run at boot time

In order to make the stateful path configurable, all of the hard coded references to /mnt/state in os-refresh-config scripts will be replaced with an environment variable, $STATEFUL_PATH.

The stateful-path element will provide an environment.d script for os-refresh-config that reads the value from os-apply-config:

export STATEFUL_PATH=$(os-apply-config –key stateful-path –type raw)

Hook scripts run at image build time

The stateful-path element will provide an environment.d script for use at image build time:

export STATEFUL_PATH=${STATEFUL_PATH:-“/mnt/state”}

The use-ephemeral element will depend on the stateful-path element, effectively making the default stateful path remain /mnt/state.

The stateful path can be reconfigured by defining $STATEFUL_PATH either A) in the environment before an image build; or B) in an element with an environment.d script which runs earlier than the stateful-path environment.d script.

Alternatives

None come to mind, the point of this spec is to enable an alternative to what’s already existing. There may be additional alternatives out there other folks may wish to add support for.

Security Impact

None

Other End User Impact

End users using elements that change the stateful path location from /mnt/state to something else will see this change reflected in configuration files and in the directories used for persistent and stateful data. They will have to know how the stateful path is configured and accessed.

Different TripleO installs would appear different if used with elements that configured the stateful path differently.

This also adds some complexity when reading TripleO code, because instead of there being an explicit path, there would instead be a reference to a configurable value.

Performance Impact

There will be additional logic in os-refresh-config to determine and set the stateful path, and an additional local collector that os-collect-config would use. However, these are negligible in terms of negatively impacting performance.

Other Deployer Impact

Deployers will be able to choose different elements that may reconfigure the stateful path or change the value for $STATEFUL_PATH. The default will remain unchanged however.

Deployers would have to know what the stateful path is, and if it’s different across their environment, this could be confusing. However, this seems unlikely as deployers are likely to be standardizing on one set of common elements, distro, etc.

In the future, if TripleO CI and CD clouds that are based on Red Hat distros make use of this feature to enable Red Hat read only root support, then these clouds would be configured differently from clouds that are configured to use /mnt/state. As a team, the tripleo-cd-admins will have to know which configuration has been used.

Developer Impact

1. Developers need to use the $STATEFUL_PATH and {{stateful-path}} substitutions when they intend to refer to the stateful path.

2. Code that needs to know the stateful path will need access to the variable defining the path, it won’t be able to assume the path is /mnt/state. A call to os-apply-config to query the key defining the path could be done to get the value, as long as os-collect-config has already run at least once.

Implementation

Assignee(s)

Primary assignee:

james-slagle

Work Items

tripleo-incubator

  • Update troubleshooting docs to mention that /mnt/state is a configurable path, and could be different in local environments.

tripleo-image-elements

  • Add a new stateful-path element that configures stateful-path and $STATEFUL_PATH to /mnt/state

  • Update os-apply-config templates to replace /mnt/state with {{stateful-path}}

  • Update os-refresh-config scripts to replace /mnt/state with $STATEFUL_PATH

  • Update all elements that have os-apply-config template files under /mnt/state to just be under /.

    • update os-apply-config element to call os-apply-config with a –root $STATEFUL_PATH option

    • update elements that have paths to os-apply-config generated files (such as /etc/nova/nova.conf) to refer to those paths as $STATEFUL_PATH/path/to/file.

  • make use-ephemeral element depend on stateful-path element

Dependencies

  1. os-collect-config will need a new feature to read from a local data source directory that elements can install JSON files into, such as a source.d. There will be a new spec filed on this feature. https://review.openstack.org/#/c/100965/

  2. os-apply-config will need an option in its control file to support generating templates under the configurable stateful path. There is a patch here: https://review.openstack.org/#/c/113651/

Testing

There is currently no testing that all stateful and persistent data is actually written to a stateful partition.

We should add tempest tests that directly exercise the preserve_ephemeral option, and have tests that check that all stateful data has been preserved across a “nova rebuild”. Tempest seems like a reasonable place to add these tests since preserve_ephemeral is a Nova OpenStack feature. Plus, once TripleO CI is running tempest against the deployed OverCloud, we will be testing this feature.

We should also test in TripleO CI that state is preserved across a rebuild by adding stateful data before a rebuild and verifying it is still present after a rebuild.

Documentation Impact

We will document the new stateful-path element.

TripleO documentation will need to mention the potential difference in configuration files and the location of persistent data if a value other than /mnt/state is used.

References

os-collect-config local datasource collector spec:

Red Hat style stateful partition support this will enable: