OpenStack manuals project migration

updated: 2018-12-06 13:55

OpenStack manuals project migration

Problem description

The documentation team are rapidly losing key contributors and core reviewers. We are not alone, this is happening across the board. It is making things harder, but not impossible. Since our inception in 2010, we’ve been climbing higher and higher trying to achieve the best documentation we could, and uphold our high standards. However, we now need to take a step back and realise that the amount of work we are attempting to maintain is out of reach for the team size that we have. At the moment we have 13 cores, of whom none are full time contributors or reviewers.

Until this point, the documentation team has owned several manuals that include content related to multiple projects, including an installation guide, admin guide, configuration guide, networking guide, and security guide. Because the team no longer has the resources to own that content, we want to invert the relationship between the doc team and project teams, so that we become liaisons to help with maintenance instead of asking for project teams to provide liaisons to help with content. As a part of that change, we plan to move the existing content out of the central manuals repository, into repositories owned by the appropriate project teams. Project teams will then own the content and the documentation team will assist by managing the build tools, helping with writing guidelines and style, but not writing the bulk of the text.

We currently have the infrastructure set up to empower project teams to manage their own documentation in their own tree, and many do. As part of this change, the rest of the existing content from the install guide and admin guide will also move into project-owned repositories.

Proposed change

Combine all of the documentation builds for a given repository, so that each repository has a single doc/source directory, a single sphinx conf.py, and a single job for building and publishing the content including developer, contributor, and user documentation. This option would reduce the number of build jobs we have to run, and cut down on the number of separate sphinx configurations in each repository.

Move most of the content from various guides in openstack-manuals into the same repository as the thing being documented – the documentation should live with the code. For example, the installation instructions for glance move to the glance repository and the reference for using the glance client command line tool move to the python-glanceclient repository.

In order to make it easy to include links from the landing pages on docs.openstack.org, we need to ensure a minimum level of consistency in the organization of the docs directory. The sections are based on the existing high-level groupings for which there are already landing pages on docs.openstack.org that will need to be updated. Within each top-level directory, project teams are free to organize their content however seems most appropriate to them. This is the proposed layout for phase 1:

  • doc/source/
    • install/ – anything having to do with installation of the project
    • contributor/ – anything related to contributing to the project or how the team is managed. Applies to some of the current content under /developer, we are changing the name to emphasize that not all contributors are developers and sometimes developers are users but not contributors. Service projects should place their automatically generated class documentation under this part of the tree, e.g. in contributor/api or contributor/internals.
    • configuration/ – automatically generated configuration reference information based on oslo.config’s sphinx integration (or manually written for projects not using oslo.config). Step-by-step guides for doing things like enabling cells or configuring a specific driver should be placed in the admin/ section.
    • cli/ – command line tool reference docs, similar to man pages These may be automatically generated with cliff’s sphinx integration, or manually written when auto-generation is not possible. Tutorials or other step-by-step guides using these tools should go in either the user/ or admin/ sections, depending on their audience. Because the documentation for each project should live in the repository with the code, this directory may not be present for all service repositories but will be present for most of the client library repositories.
    • admin/ – any content from the old admin guide or anything else that discusses how to run or operate the software
    • user/ – end-user content such as concept guides, advice, tutorials, step-by-step instructions for using the CLI to perform specific tasks, etc.
    • reference/ – any reference information associated with a project that is not covered by one of the above categories. Library projects should place their automatically generated class documentation here.

This layout is the minimum set. Projects are free and encouraged to add whatever other docs they need beyond this list, but these items are listed here explicitly because there are already links to most of them from landing pages, and landing pages can be created for the others.

During a later phase, we will merge the API reference and release notes builds into the same job, along with the rest of the documentation for a project. Both of those builds have custom considerations, though, and it is more important to move the content that is no longer going to be maintained by the documentation team.

  • doc/source/
    • api/ – the REST API reference and Guide content, when it exists
    • releasenotes/ – reno directions (the actual release notes inputs will stay in /releasenotes/notes, where they are now)

Note

Further discussion of the layout of the api/ and releasenotes/ directories is deferred until we are farther along with the initial migration work.

A new documentation build job will be set up to take the output produced from tox -e venv -- python setup.py build_sphinx (matching the existing Consistent Testing Interface”) and publish it to a new location under http://docs.openstack.org The results will go to:

  • docs.openstack.org/$project-name/latest – build from master
  • docs.openstack.org/$project-name/$series – build from stable/$series
  • docs.openstack.org/$project-name/latest/$lang – build translated version from master
  • docs.openstack.org/$project-name/$series/$lang – build translated version from stable/$series

Because we plan to reuse the existing doc/source directory in each project, some of the existing content will need to be rearranged as part of importing the content from openstack-manuals.

Ultimately, this changes the way we publish results, and redirects will be required to be setup from all of the existing locations to the new locations, and move all of the existing documentation under the new structure. We will retain landing pages for the high level categories such as the install guides, the configuration reference, and contributor documentation. Those pages will continue to be maintained by the documentation team, and will deep-link into the project team documentation.

For example, we will keep pages like https://docs.openstack.org/project-install-guide/ocata/ but they will provide lists of links to URLs like https://docs.openstack.org/nova/ocata/install, which will be part of the single doc build for nova.

What is happening to each guide?

  • Installation Guide
    • Most of the content will move from openstack-manuals into each project tree.
    • The chapters not directly related to specific OpenStack projects, such as the parts related to installing ntp and RabbitMQ, will be retained in openstack-manuals in a shared guide for setting up common dependencies so that content does not need to be reproduced several times.
    • The current guide is actually built 3 separate times, to cover 3 separate deployment platforms. The new build will not support that, so the migration for the installation guide will involve breaking the content up into separate pages for each platform (as needed). Patch https://review.openstack.org/473579 splits the content up into separate patches, one per OS.
  • Project Installation guides, already in tree
    • We recommend any installation guides already in-tree also move to the new organization.
  • Administrator Guide
  • High Availability Guide
  • Operations Guide
    • This guide will eventually move from openstack-manuals into the wiki. Nothing will be done with it until a volunteer is found to manage that move.
  • Security Guide
    • This content will stay in openstack-manuals, and be managed by the security team.
    • A notice is being added to indicate the last time it was updated and which release is relevant (https://review.openstack.org/#/c/470059).
  • Architecture Design Guide
  • Networking Guide
    • This content will move from openstack-manuals to the neutron repository under docs/source/admin.
  • Configuration Reference
  • API Documentation
    • No changes.
  • End User Guide
    • This content will be divided between the horizon repository and python-openstackclient repository.
  • Command-Line Reference
    • This content will move the project-specific client documentation trees under doc/source/cli. For example, the information about using the glance command line tool would move to the python-glanceclient repository.
  • Virtual Machine Image Reference
    • This content will stay in openstack-manuals.

Migration process

We will need to parallelize the migration work as much as possible if we are going to complete it by the end of the Pike cycle. We will therefore need project teams to find volunteers to “pull” the content into their repositories, instead of having the documentation team “push” it.

Note

Use the topic doc-migration for all patches.

Note

Repeat these steps for all server projects, clients, and other libraries.

  1. Move the existing contributor-focused content to fit the layout above. Submit that change with Depends-On: Ia750cb049c0f53a234ea70ce1f2bbbb7a2aa9454 to tie it to this spec.

  2. If your project docs are not already building using warning-is-error in setup.cfg, turn that on and fix any build errors. Submit these as patches on top of the first patch.

  3. Pull in the content being migrated, following the layout above.

    • Go through the list of manuals in https://etherpad.openstack.org/p/doc-migration-tracking and take any actions needed to import content.
    • Prepare one patch per manual (so one to import the install guide, one to import the user guide, etc.). Submit these as patches on top of any previous patches.
    • :term: needs to be removed when importing and performing the migration. This is due to the glossary remaining in the openstack-manuals repo. Teams can choose to link to the glossary on their own project pages if they so desire.

  4. Ensure that there is an index.rst in each subdirectory of doc/source so that the various landing pages managed by the documentation team can link directly to that portion of the documentation for your project. For example, in addition to moving installation documentation into install/ create install/index.rst with a toctree directive that shows all of the installation.

  5. Ensure that there is a top-level index.rst in doc/source that incorporates all of the documentation for the project by including all of the subdirectories in a toctree.

  6. Update the theme for the in-tree docs to use the openstackdocstheme instead of oslosphinx.

  7. Add auto-generated config reference section(s) under the configuration/ directory.

  8. If pbr’s autodoc feature is being used, update the api_doc_dir setting in the pbr section of setup.cfg to point to either reference/api (for libraries) or contributor/api (for other types of projects).

  9. Update project-config to have the doc build use the new jobs instead of the old jobs by replacing ‘openstack-server-publish-jobs’ with ‘openstack-unified-publish-jobs’.

    Set Depends-On to the Change-Id from the patch created in step 1. This ensures that we do not publish the old content to the new location.

  10. Add links to the reviews for individual TODO items below those items in the sections dedicated to each manual. That way the docs team will know when it is safe to start deleting content.

Alternatives

  1. We could retain the existing trees for developer and API docs, and add a new one for “user” documentation. The installation guide, configuration guide, and admin guide would move here for all projects. Neutron’s user documentation would include the current networking guide as well. This option would add 1 new build to each repository, but would allow us to easily roll out the change with less disruption in the way the site is organized and published, so there would be less work in the short term.
  2. We could move the content under separate repositories owned by the project teams, rather than in-tree with the code. This would allow project teams to delegate management of the documentation to a separate review project-sub-team, but would complicate the process of landing code and documentation updates together so that the docs are always up to date.
  3. Do nothing, and watch the world burn.

We did consider using “service type” instead of “project name” for the publishing URLs, but not all of the projects that need documentations are services. We will have user-facing documentation coming from several Oslo libraries, for example.

Implementation

Assignee(s)

Primary assignee:

  • Alexandra Settle (asettle)
  • Doug Hellmann (dhellmann)
  • Project teams
  • Documentation team PTL for Queens
  • Documentation team

Work items

The task list is quite long, so rather than repeat it here we give a summary. There is more detail in the tracking pad mentioned in step 3.

  1. Define new doc build and gate jobs that work like the current job, using “tox -e venv – python setup.py build_sphinx`” in a repository, but publish to the new location of docs.o.o/$project-name/latest (dhellmann)
  2. Define doc build jobs for stable branches that run the same command but publish to docs.o.o/$project-name/$series (dhellmann)
    • The same job will work for all branches.
  3. In parallel, in each repository, perform the migration steps listed above to copy the new content into the doc/source directory. Refer to https://etherpad.openstack.org/p/doc-migration-tracking for details about which pages go into which project trees.
  4. Define new translation jobs based on the ones for the release notes build but using the main doc build.
  5. Create a separate build for the openstack-manuals glossary.

Dependencies

  • Project team(s) collaboration
  • Infra team assistance
  • Reviews from multiple sources
updated: 2018-12-06 13:55
Creative Commons Attribution 3.0 License

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.

questions?

docs-specs