This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode
Tempest test resource cleanup
The cleanup/release of test resources created/allocated in the class level test fixtures is invoked in the class level tearDown fixture. However tearDownClass is invoked by the unittest framework only in case setUpClass is successful. This is causing resources being leaked when:
The test-accounts bp introduces the possibility to run parallel tests using a configured list of pre-provisioned test accounts. Test accounts are allocated and released by each test class, and a failure to release leads to exhaustion of test accounts.
Disallow overriding setUpClass defined in BaseTestCase with a hacking rule. Define setUpClass so that it calls one (or more) other methods to be overridden by descendants. Decorate it with the @safe_setup decorator. This way tearDownClass will always be invoked.
@classmethod @safe_setup def setUpClass(cls): cls.setUpClassCalled = True cls.resource_setup()
While doing this change, an extra benefit can be gained by structuring the setup in a series of methods, to enforce the least possible resource allocation before failure and thus quick cleanup as well.
PoC for this is available here: https://review.openstack.org/#/c/115353.
@classmethod @safe_setup def setUpClass(cls): cls.setUpClassCalled = True # All checks that may generate a skip cls.setup_skip_checks() # Any setup code that does not require / generate test resources cls.setup_pre_resources() # Allocation of all required credentials cls.setup_allocate_credentials() # Shortcuts to clients cls.setup_clients() # Allocation of shared test resources cls.setup_create_resources() # Any setup code to be run after resource allocation cls.setup_post_resources()
The tearDownClass fixture requires fixing in several places, because several tearDownClass implementation would become unsafe, as they expect attributes defined during setUpClass, which may not be there anymore.
Disallow overriding tearDownClass defined in BaseTestCase with an hacking rule. Define tearDownClass so that it invokes a descendant specific cleanup code, and finally cleans-up credentials.
@classmethod def tearDownClass(cls): at_exit_set.discard(cls) try: cls.resource_cleanup() finally: cls._cleanup_credentials() # Defined in BaseTestCase
Two alternatives have been identified.
Decorate all setUpClass implementation with @safe_setup and all tearDownClass implementation with @safe_teardown. This approach requires a mass change to tempest, which as the benefit of being almost scriptable (PoC: https://review.openstack.org/#/c/115123/). It has the downfall of requiring every new test class to add those two decorators.
This may still be an option on the long term, but at the moment the effort of the migration would be more than the benefit from it. Additional work to ensure cleanup of resources would still be required anyways.