Machine Readable Sample Config

https://blueprints.launchpad.net/oslo?searchtext=machine-readable-sample-config

Currently the sample configs generated by oslo.config are targeted for consumption by humans. However, many deployment tools would benefit from a sample config that is machine readable.

Problem description

All deployment tools have to solve a similar problem: how to generate the config files for each service at deployment time. There are various approaches to this today, including manually updated templates, semi-automatically generated templates, and using an ini file tool to update the existing sample config for each project.

These approaches all have various drawbacks. Template approaches generally require some level of tedious human editing to template all of the values the deployment tool needs to set. Ini file tools are not aware of the sample config format and thus are only able to set values in the correct section, which usually separates them from the sample config documentation of the option. None of these approaches make it easy for a deployment tool to provide a user-friendly interface to the config options.

The problem with the current ini format sample config in this case is that it is not easily machine-readable. The content is all written in commented blocks which would have to be parsed by a deployment tool. Since that comment format is not guaranteed to remain the same, it would require ongoing maintenance by each deployment tool using this approach.

Proposed change

The proposal is to write machine-readable sample config files that output the same data as the existing ini files, but in a YAML or JSON format that can be more easily consumed by deployment tools.

Example of proposed output:

generator_options:
    args: --config-file=etc/nova/nova-config-generator.conf
    output_file: etc/nova/nova.conf.yaml
    wrap_width: 80
    namespace:
        - nova.conf
        - oslo.log
        ...
    summarize: False
    # minimal intentionally omitted as I don't think it makes sense in
    # this context
    [more generator details can be added as desired]
options:
    DEFAULT:
        image_service:
            # The list of parameters is for example only.  The actual output
            # can include anything deployment tools would find useful.
            default: nova.image.glance.GlanceImageService
            documentation: Service used to search for and retrieve images.
            type: string
            required: True
            deprecated: False
            deprecated_reason: blabla
            deprecated_for_removal: False
            deprecated_since: XXX
            deprecated_name: old_parameter
            [all other parameters passed to the opt constructor]
        new_opt:
            deprecated_name: old_opt
            ...
        ...
    conductor:
        workers: (...)
        ...
    ...
deprecated_options:
    DEFAULT:
        old_opt:
            default: foo
            documentation: Example deprecated opt
            type: string
            replacement_name: new_opt
            replacement_group: DEFAULT
            ...

This structure would contain all of the details needed to write a well-formatted config file, and it would be easier to inject values in the appropriate place because we would be starting from the actual opt data instead of an ini file full of comments. It should require little to no manual maintenance because it would be generated from sample config data already provided by each project.

Another suggested change in this format from the existing generated sample config is to write deprecated opts as first-class citizens rather than just as a deprecated_name/group on the new opt. This way the structure will still align with the previous config, which should make it easier to transition from release to release. These opts would, of course, be marked deprecated so it is clear that they should no longer be used. An example of this is included above.

Note on the implementation: In order to easily support both YAML and JSON formats, I think the generator should work with nested dicts that can simply be output using the appropriate module. Since YAML and JSON formatters already exist we shouldn’t need to reinvent those wheels.

Alternatives

Some of the existing methods for doing this and their drawbacks are discussed above in the problems section. These methods work, but there is a lot of duplication of effort across all deployment projects so it is not desirable to continue this way.

Each project could automate their config file generation, similar to the OpenStack Helm tool in the References section, but there again we have a duplication problem that oslo.config can solve.

Impact on Existing APIs

We would need to add a –format parameter to the oslo-config-generator so users can generate sample configs with the new format(s). This would default to the existing ini-style format for backwards compatibility.

Security impact

Should be none. For the most part this is just a new format for data that was already being generated. One possible exception would be capturing the parameters to the config generator, but I don’t see anything in there that should be sensitive.

Performance Impact

Generating the new file formats will take additional time, but by default only the existing format will be created so there should be no change.

Configuration Impact

Outside of the –format CLI param, no new config opts are proposed.

Developer Impact

None. Developers are already exposing config opts to the config generator, and this is simply adding another use for that.

Testing Impact

This functionality would be unit tested in the same way as the existing config generator. We would need to add YAML/JSON scenarios to the unit tests.

Implementation

Assignee(s)

Primary assignee:

bnemec

Other contributors:

emacchi

Milestones

Target Milestone for completion: We would like to complete this work in Pike, probably for consumption by deployment tools in the Queens cycle.

Work Items

  • Implement YAML output

  • Implement JSON output

  • Add test scenarios for new formats

Incubation

Not being incubated.

Adoption

NA

Library

NA

Anticipated API Stabilization

NA

Documentation Impact

This work is targeted at deployment tools, so there should be little to no user-facing documentation changes required. We would want to reference the existence of these new sample formats in the oslo.config documentation.

The format and structure of the output files will need to be documented for deployment tool consumption.

Dependencies

None

References

Cross-Deployment Tool PTG Session, specifically the Problem #1 section.

https://github.com/alanmeadows/gen-oslo-openstack-helm

Note

This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode