Power fault recovery¶
This spec proposes adding a new node field to reflect any faults detected by the system. For nodes put into maintenance mode due to power-sync failures, this proposes a mechanims to try to recover the nodes from the failure.
Currently, ironic’s API and data model do not differentiate between ironic setting maintenance on a node (for instance, when cleaning fails, or when the power status loop cannot contact the BMC) and an operator setting maintenance on a node (an explicit API action).
This situation makes it difficult to apply any power fault recovery mechanism.
A string field named
fault will be added to ironic’s data model for a Node.
It will be used to record any detected faults.
The field will be set to one of the following values, depending on the situation (when ironic puts the node into maintenance due to one of these):
power failure: when a node is put into maintenance due to power sync failures that exceed max retries.
clean failure: when a node is put into maintenance due to failure of a cleaning operation.
rescue abort failure: when a node is put into maintenance due to failure of cleaning up during rescue abort.
None: there is no fault detected for the node; this is the default value. Since the field is set to a fault when ironic puts the node into maintenance due to a fault, it gets set to None when the node is taken out of maintenance (by ironic or the user).
For nodes that were in maintenance prior to upgrading to this release, the field will also be None, since we don’t know for sure, if (or what) there had been a fault. Even if there had been, we don’t know whether the operator wants the node in maintenance for other reasons as well.
A periodic task will be added to the conductor. This will operate on nodes
node.fault == power failure. It will try to get the node’s power
state. If successful, the node will be taken out of maintenance.
An integer configuration option named
[conductor]power_failure_recovery_interval will be added. This is the
interval (number of seconds) between each run of the periodic task. The
default value is 300 seconds (5 minutes). Setting it to 0 will disable
power fault recovery.
Proposals have already been made and rejected to use a combination of maintenance being set and a given maintenance_reason or last_error to do self-healing due to inconsistency.
Specific faults support 1 is proposed but it’s too big and to-date, no consensus has been reached on it.
An earlier version of this spec had proposed a
field for the node, instead of a
fault field. We prefer the
fault field since it better reflects that it is about faults.
maintenance_type caused the field to be tied in with the
fault makes it distinct from
maintenance; it will
make it easier to move forward in the future, should we decide to
provide more enhancements that are fault-related, or to separate
the notion and handling of faults from maintenance (which some
would argue should only be operator-initiated).
Data model impact¶
A string field named
fault will be added to ironic’s node table.
The field will be added to
Node object; object version will be
State Machine Impact¶
REST API impact¶
fault will be added to
Node API object, ironic
API version will be bumped.
Node-related API endpoints will be affected:
For requests with older microversion,
node.fault is hidden from
the response. This will be done in the
fault is read-only from the API, and can only be modified
internally. Any POST/PATCH request to Node with
fault set will
get 400 (Bad Request).
There is no other impact to API behaviors.
Client (CLI) impact¶
“openstack baremetal” CLI¶
The CLI will be updated to support field
by ironic API microversion.
RPC API impact¶
Driver API impact¶
Nova driver impact¶
None, although we should update the ironic-virt driver code to
also look at this new node.fault field, in addition to the
Other end user impact¶
If the periodic task for recovery is enabled, it will consume some resources on conductor node.
More will be consumed in an environment containing some nodes in maintenance due to power failure.
Other deployer impact¶
A new option
[conductor]power_failure_recovery_interval is introduced to
support power failure recovery. The default value is 300 (5 minutes), you
have to set to 0 if this feature is not needed.
- Primary assignee:
- Other contributors:
Update db layer to include the
Change places where nodes enter/leave maintenance, to set
[conductor]power_failure_recovery_intervaloption to ironic configuration, add periodic task to handle power recovery.
The feature will be covered by unit tests, API change will be covered by tempest test.
Upgrades and Backwards Compatibility¶
ironic API change is guarded by microversion.
When upgrading, any nodes in maintenance will have
node.fault set to None.
This is because there is no easy/guaranteed way to determine if a node had
been (previously) put into maintenance due to one of our internal faults.
Furthermore, even if we could determine whether it was due to a fault, we don’t
know whether the operator wants to keep the node in maintenance for other
reasons besides the fault.
Should the operator want to take advantage of the power fault recovery mechanism, they could take the nodes out of maintenance. If there are still issues, ironic will do its thing – detect the fault and try to recover.
New option will be generated by config generator.