Our current requirements management policy is causing significant gate downtime, developer issues and packaging headaches. We can do better.
There are a number of interacting things we do today which are causing issues.
We run our tests with unpinned dependencies. This means that any patch being tested actually tests two things: the change in the patch, and all new releases of dependencies (within the ranges relevant to the branch in question). With nearly 300 (transitive) libraries in use, an incompatibility rate of only one per library per year can break us daily. We are suffering regular firedrills, and we’ve burnt out many gate fixers so far.
We require that projects have their install_requires exactly match our top level package specifiers that we use in gate jobs. This leads to releases of stable branches that have much narrower dependency specifiers than may work in practice. This becomes a problem when one package of a set needs to upgrade a given dependency post-release to fix a bug: the new package can be outside the set of versions dictated by the union of specifiers across all our packages that use it, which causes a cascade where new releases are required across the entire set to permit it to be used.
We override project local install_requires during testing, which means that our co-installability check is often returning a false positive: our actual install_requires may be incompatible but the gate won’t report on this.
Additionally we have some constraints on solutions:
Finally there are plenty of things that could be done that aren’t addressed in this specification: it’s the minimal self consistent set of improvements to address the ongoing firedrills we currently suffer.
tl;dr: Use exact pins for testing. Use open ended specifiers in project install_requires.
Globally we need to maintain global-requirements as we do today. This remains our policy control for which libraries are acceptable in OpenStack projects. As we start using extras, we’ll need to track extras in global-requirements as well. We want to preserve an axiom: that projects have wider-or-equal requirements than the coordinated release which has wider-or-equal requirements to the test pinned list.
We’ll add a new pip freeze file to openstack/requirements, called upper-constraints.txt. This will contain a pinned list of the entire set of transitive requirements for that branch of OpenStack (defined by the projects within the projects.txt file in openstack/requirements). All CI jobs will use this to constrain the versions of Python projects that can be tested with. Changes to that file will be tested by running a subset of the same jobs that would consume it, as well as a policy checker that checks it is compatible with global-requirements.txt. Changes to either global-requirements.txt or upper-constraints.txt will have to be compatible with each other.
We’ll tighten up the policy checks on projects to require that there be no dependencies outside of those listed in global-requirements. This is needed to allow centralised calculations about potential upgrades and co-installability calculations. We’ll change the check from ‘identical line to global-requires’ to be ‘compatible with both global-requires and upper-constraints’. The existing requirements sync job will continue to propose converged requirements for projects that have opted into converged dependency management.
We’ll create a periodic job that takes global-requirements, expands it out transitively, and then proposes a merge to global-requirements bringing in any new releases (and any new or removed transitive-only dependencies that that implies) as a patch to upper-constraints.
Releases are a multi week process in OpenStack as different servers fork and setup their branches one at a time. During that period we’ll gate any requirements changes on both master and any branched projects, branching openstack/requirements last when we’re finally ready to decouple the release from master. This is aimed at the changes involved in using new releases of oslo etc.
Lastly, the pip resolver work will increase the accuracy of our constraints pinning, but this spec is not dependent on that.
The null option will continue to burn people out.
No other options have been proposed.
This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode