Heat Upgrade Tests¶
The new direction of the grenade project is to use devstack-style plugins to support upgrade testing. This spec aims to get upgrade testing infrastructure in Heat’s tree.
Currently, there aren’t any tests that check if a patch in-review would break the upgrade of Heat from a previous release.
Some of the issues a deployer may face during or post upgrade are as follows:
The deployed database not successfully migrating to the newest schema.
Stacks created in an older release not updatable/deletable in new release. A change in the update/delete workflow may render existing stacks useless or be in an IN_PROGRESS state forever.
Physical resources disappearing when the control plane is taken down.
Configuration options removed prematurely without notice.
Get upgrade testing in-tree by using grenade’s external plugin mechanism. The upgrade tests should follow what grenade calls the Theory of Upgrade - https://github.com/openstack-dev/grenade#theory-of-upgrade
To ease developers writing upgrade tests, we will have (pre/during/post)-upgrade tox envs that will run tests respectively during the (pre/during/post)-upgrade phases in grenade.
As an example, the pre-upgrade phase will create stacks before an upgrade. The during-upgrade phase will check if the resources created by a stack are still alive even when heat’s services are down. The post-upgrade phase could update/delete these stacks to verify that those old stacks are still operable.
Once this infrastructure is in place, we should be able to add support for rolling upgrades of heat-engine with the help of the existing versioned objects mechanism in Heat. This will use Grenade’s partial-upgrade strategy similar to what is being done for Nova.
- Primary assignee:
- Target Milestone for completion:
In-tree grenade testing.
Project config changes to add voting grenade job.
Tag smoke tests in heat_integrationtests that must be run during the grenade sanity-check(verify) phase.
pre/during/post-env env sections in tox that will respectively invoke the tests before, during and after the upgrade of Heat.
Get devstack to use the system level heat binaries instead of the ones in the repository to emulate what a user would face when upgrading heat.
Add Upgrade Impact section to heat-specs.
Support rolling upgrade of heat-engine.
Grenade external plugin mechanism https://review.openstack.org/#/c/185050