This specification describes the next stage of the image generation epic, and proposes a CLI to generate images from the plugin-defined recipes described in the validate-image-spi blueprint.
Sahara has historically used a single monolithic repository of DIB elements for generation of images. This poses several problems:
For these reasons, a means to encapsulate image generation logic within a plugin, a command line utility to exercise these recipes, and a backing toolchain that emphasizes reliability over flexibility and speed are indicated.
The https://blueprints.launchpad.net/sahara/+spec/validate-image-spi blueprint describes the first steps of this epic, enabling us to define recipes for image validation and generation of clusters from “clean” images.
The current specification proposes that a CLI script be added to sahara at sahara.cli.sahara_pack_image.py. This script will not start a new service as others do; rather, it will use the pre-existent sahara.services.images module to run the image generation recipe on a base image through a new remote implementation. This remote implementation will use libguestfs’python API to alter the image to the plugin’s specifications.
–help text for this script follows:
Usage: sahara-image-create --image IMAGE_PATH [--root-filesystem] [--test] PLUGIN PLUGIN_VERSION [More arguments per plugin and version] * --image: The path to an image to modify. This image will be modified in-place: be sure to target a copy if you wish to maintain a clean master image. * --root-filesystem: The filesystem to mount as the root volume on the image. No value is required if only one filesystem is detected. * --test-only: If this flag is set, no changes will be made to the image; instead, the script will fail if discrepancies are found between the image and the intended state. [* variable per plugin and version: Other arguments as specified in the generation script. * ...] * -h, --help: See this message.
Both PLUGIN and PLUGIN_VERSION will be implemented as required subcommands.
If –test-only is set, the image will be packed with the reconcile option set to False (meaning that the image will only be tested, not changed.)
Each image generation .yaml (as originally described in the validate-images-spi spec,) may now register a set of ‘arguments’ as well as a set of ‘validators’. This argument specification should precede the validator declaration, and will take the following form:
arguments: - java-version: description: The java distribution. target_variable: JAVA_VERSION default: openjdk required: false choices: - oracle-java - openjdk
A set of arguments for any one yaml must obey the following rules:
An ImageArgument class will be added to the sahara.service.images module in order that all plugins can use the same object format. It will follow the object model above.
In order to facilitate the retrieval of image generation arguments, a get_image_arguments SPI method will be added to the plugin SPI. The arguments returned from this method will be used to build help text specific to a plugin and version, and will also be used to validate input to the CLI. It will return a list of ImageArguments.
A pack_image method will also be added to the plugin SPI. This method will have the signature:
def pack_image(self, hadoop_version, remote, reconcile=True, image_arguments=None)
This method will take an ImageRemote (see below). In most plugins, this will:
However, the implementation is intentionally vague, to allow plugins to introduce their own image packing tools if desired (as per the validate-images-spi spec.)
Now that these image definitions need to be able to take arguments, a new ArgumentCaseValidator will be added to the set of concrete SaharaImageValidators to assist image packing developers in writing clean, readable recipes. This validator’s yaml definition will take the form:
argument_case: argument_name: JAVA_VERSION cases: openjdk: - [action] oracle-java: - [action]
The first value case key which matches the value of one of the variable will execute its nested actions. All subsequent cases will be skipped.
A new ImageRemote class will be added to a new module, at sahara.utils.image_remote. This class will be encapsulated in its own module to allow distribution packagers the option of externalizing the dependency on libguestfs into a subpackage of sahara, rather than requiring libguestfs as a dependency of the main sahara python library package in all cases.
This class will represent an implementation of the sahara.utils.remote.Remote abstraction. Rather than executing ssh commands from the provided arguments, however, this class will execute scripts on a target image file using libguestfs’ python API.
The CLI will use generate a remote targeting the image at the path specified by the ‘image’ argument, and use it to run the scripts which would normally be run over ssh (on clean image generation) within the image file specified.
We have discussed the option of bringing DIB elements into the Sahara plugins. However, this was nixed due to the issues above related to tradeoff between speed and power and usability, and because of certain testing issues (discussed in an abandoned spec in the Mitaka cycle).
It is also possible that we could maintain our current CLI in sahara-image-elements indefinitely. However, as more plugins are developed, a single monolithic repository will become unwieldy.
The new CLI will be (at present) the only means of interacting with this feature.
Packaging this script as a separate python (and Debian/RPM package) is an option worth discussing, but at this time it is intended that this tool be packed with the Sahara core.
This feature will reuse definitions specified in the validate-image-spi spec, and thus should have minimal developer impact from that spec.
This feature will hopefully, once it reaches full maturity and testability, supplant sahara-image-elements, and will provide the baseline for building image packing facility into the sahara API itself.
No dashboard representation is intended at this time.
This feature will introduce a dependency on libguestfs, though it will only use libguestfs features present in all major distributions.
As this feature does not touch the API, it will not introduce new Tempest tests. However, testing the image generation process itself will require attention.
It is proposed that tests be added for image generation as each plugin implements this generation strategy, and that nightly tests be created to generate images and run these images through cluster creation and EDP testing.
These should be implemented as separate tests in order to quickly differentiate image packing failure and cluster failure.
As these recipes stabilize for any given plugin, we should begin to run these tests when any change to the sahara repository touches image generation resources for a specific plugin (which should be well-encapsulated in a single directory under each version for each plugin.) Toward the end of this epic (as we are nearing the stage of authoring the API to pack images, we may consider removing integration tests for SIE to save lab time. Still, these tests will be, compared to sahara service tests, very resource-light to run.
This feature should be documented in both devref (for building image generation recipes) and in userdoc (for script usage).