This spec is to define the deadlines, requirements and process of bringing third party continuous integration testing to ironic.
The ironic project wants to make sure that all the drivers in tree are maintained and of high quality. In order to do this, it has become necessary to require that all drivers have a continuous integration test system to test the driver against every code change proposed for ironic, unless that change is to code or documentation that can not impact the driver. Required tests to be run by each driver test system will be documented in the ironic CI requirements documentation. Not all drivers are provided or maintained by third party vendors.
Third party vendors must provide and maintain this CI for drivers they maintain. Any driver whose maintainer is not able to implement a reliable CI system to test will be removed from the ironic tree.
A driver test system will be deemed reliable if it runs the expected tests and reports the results of those tests consistently over a period of time. Tests will be expected to complete and report back to gerrit within 8 hours of the patch submission until the end of Newton release and within 4 hours by the end of Ocata development cycle.
A driver test system will be deemed reliable if:
The ironic team has decided to require all driver maintainers to run and maintain CI test systems in order to have their driver code remain in the ironic source tree. When there are problems with the CI system, the driver test team will make a best effort to respond to questions about their system and to fix the system quickly. If the driver test team is not responsive, any voting privileges for the test system may be removed, and may eventually result in the driver being removed from the source tree.
The process to implement and start enforcing third party driver CI is to be completed by the end of the Newton development cycle.
Please refer to the Mitaka release schedule for specific dates. Note that the ironic project does not strictly follow the official OpenStack release cycle deadlines, but we are using the official deadlines as a reference point.
Infra CI will continue to test the ssh drivers already being used in the gate.
Third party driver teams that do not implement a reliable reporting CI test system by the Newton release feature freeze (see Deliverable milestones above) will be removed from the ironic source tree. Driver test systems that miss any of the milestones may be subject to immediate removal from the source tree.
Driver test systems will be required to initially run a test similar to the gate-tempest-dsvm-ironic-pxe_ipa test, with the only difference being changes to drivers loaded, etc, to make ironic work with the driver under test.
More information about this and other required tests will need further documentation.
Initial driver test systems will not be allowed to vote on changes to ironic. A driver test team may ask that their CI system be allowed to vote only if the ironic team approves based on the test team’s participation, availability and history of reliable operation.
Driver test systems will be expected to follow the Infrastructure Third Party test requirements, unless overridden here or in the ironic third party driver testing documentation produced as a result of this spec.
Driver test systems must test patches to master, and patches to branches that were created since the third party CI began testing. For example, if a CI began testing during the Mitaka cycle, then during the N cycle that CI will be expected to test changes to the master and stable/mitaka branches, but not the stable/liberty branch.
Community-maintained drivers that do not have CI testing will also be removed from the ironic source tree.
From Newton feature freeze forward, all new ironic drivers must show that they have a system performing CI testing and meet all infra  and ironic requirements in order to land code into the ironic tree. Drivers in progress before Newton feature freeze have until Newton feature freeze to meet these requirements. CI testing does not need to be shown in order to propose a driver spec, but must be in place before landing code.
When upgrading to the release that drops untested drivers, if a deployer is using a driver that is removed from the tree, they will need to change to an in-tree driver or install the removed driver from a new location, if one exists.
The Ironic team must communicate which drivers are being removed, and when. We should note that these drivers may be available at a new location, and that driver authors may be communicating that information.
Authors of a driver removed from tree may communicate the new location, if one exists, and document how to install the driver into an ironic environment.
Developer impacts may include core reviewers needing to wait until testing for a system completes before approving a patch for merge. Developers that had a test fail will need to review the test artifacts for their patch linked to the comment left in the patch comment log. If necessary, the developer may need to coordinate with the driver test team for help with debugging the problem. See Infra requirements in the References section below.
As described in this spec.
There will be a major upgrade impact on deployers using drivers that are removed from tree; see the “Deployer impact” section for more info.
A deprecation process will be documented including timeline.
There will be several areas impacted:
 Third Party CI working group https://wiki.openstack.org/wiki/ThirdPartyCIWorkingGroup
 Third party CI meetings https://wiki.openstack.org/wiki/Meetings/ThirdParty
 Infra requirements documentation for implementing a third party system http://docs.openstack.org/infra/system-config/third_party.html#requirements
 Discussion at Mitaka summit https://etherpad.openstack.org/p/summit-mitaka-ironic-third-party-ci
 Mitaka release schedule: https://wiki.openstack.org/wiki/Mitaka_Release_Schedule