The new direction of the grenade project is to use devstack-style plugins to support upgrade testing. This spec aims to get upgrade testing infrastructure in Heat’s tree.
Currently, there aren’t any tests that check if a patch in-review would break the upgrade of Heat from a previous release.
Some of the issues a deployer may face during or post upgrade are as follows:
Get upgrade testing in-tree by using grenade’s external plugin mechanism. The upgrade tests should follow what grenade calls the Theory of Upgrade - https://github.com/openstack-dev/grenade#theory-of-upgrade
To ease developers writing upgrade tests, we will have (pre/during/post)-upgrade tox envs that will run tests respectively during the (pre/during/post)-upgrade phases in grenade.
As an example, the pre-upgrade phase will create stacks before an upgrade. The during-upgrade phase will check if the resources created by a stack are still alive even when heat’s services are down. The post-upgrade phase could update/delete these stacks to verify that those old stacks are still operable.
Once this infrastructure is in place, we should be able to add support for rolling upgrades of heat-engine with the help of the existing versioned objects mechanism in Heat. This will use Grenade’s partial-upgrade strategy similar to what is being done for Nova.