Tempest historically has been treated like a core service, with a stable/foo release branch cut at every release of OpenStack. However, Tempest is largely black box testing for the OpenStack API. The API is represented by major versions, not by release tags, so Tempest should not need branches. This change would have a number of interesting cascading impacts, all of which should be positive for the health of the project.
Our current release method for Tempest is to create a stable/foo branch at every release of the OpenStack core services. This has a number of side effects.
Tempest has organically grown up as a tool that works really well in our upstream gate, and medium well for people testing real deployments. One of the reasons for this difference is that we use the stable/foo Tempest branches to make assumptions about the relationship of services, and devstack, which set up Tempest. Breaking this convenience relationship will make us be much more dilligent in being explicit about Tempest as a tool designed to run under any setup environment.
Under the proposed branchless environment we’d do the following:
This has cascading implications, probably best described as scenarios (I’ll use nova as the example service for all scenarios, mostly because I’m more familiar with the extensions model, not because it’s being picked on):
If a new feature is added to Nova in Juno, and a new test is added to Tempest for this feature, this test will need to be behind a feature flag (as the feature wasn’t available in Icehouse).
This has the added benefit of making sure that a new Nova change is added in such a way that it’s somehow discoverable that it’s not there (i.e. behind an unloaded extension), and that Tempest has a knob for configuring or not configuring it.
Previously a behavior change in nova master that needed a Tempest change could be changed via the tempest 2 step:
In this new model the change will also need to be backported to stable/icehouse simultaneously. Or we decide that this is behavior that should not be tested for, and drop the test entirely.
When adding new tests for existing features the new tests will need to simultaneously pass when tested against nova master and nova stable/icehouse. If we discover there is a difference of behavior between the two, we’ll need to harmonize that behavior before landing the test. We end up with 3 options:
A feature, like Nova XML v2 is deprecated is icehouse, and (hypothetically) removed in juno. Howeve the XML v2 tests are in Tempest, and we still need to test icehouse releases.
This would be handled by a slow deprecation / feature flag:
This will mean that Tempest will contain more legacy code, however that’s a trade off we take with the benefits this provides.
This will have some interesting additional fallout:
The alternative is to continue with the current approach which uses stable/* branches in Tempest. However that has caused more unintentional coupling with devstack that we’d like, and I believe that long term it’s the wrong approach for a test suite like Tempest.
Can optionally can list additional ids if they intend on doing substantial implementation work on this blueprint.
Target Milestone for completion:
The work items span projects
Only those listed above