High-Level Workflows for BMC Management

https://bugs.launchpad.net/ironic/+bug/2162051

Problem description

Ironic does a great job managing the entire lifecycle of machines themselves, but enrolling new hardware often involves several actions to prepare BMC for operations. When an Ironic operator has a high number of machines entering and leaving the cluster(s), it can become a significant burden.

Imagine this scenario. I order a large number (hundreds to thousands) of machines. They came with factory BMC credentials, and the BMCs are capable of receiving networking information via DHCP. This is enough to use the machines with Ironic but is not enough for secure operation:

  • Factory BMC credentials are well known. And while most users will receive machine-specific credentials from the vendor instead, they’ll probably prefer to change them initially and then rotate on regular basis.

  • Some deployments may prefer to use LDAP for authentication to ensure that all credentials are managed in a central place.

  • For availability reasons, BMC networking must not depend on the DHCP server operation.

Similar concerns are important for already provisioned machines:

  • Credentials rotation may be required by the site policy or jurisdiction.

  • LDAP settings may need updating.

This proposal focuses on Redfish.

Proposed change

While the general BMC settings specification proposes low-level actions on BMCs, this specification covers a few common workflows. All of them have certain pitfalls that justify adding higher-level logic. These include both implementation caveats and drastic vendor differences.

The Redfish management interface gains a number of new clean/service steps. See Redfish details for how the steps are actually implemented in the Redfish level.

All of the steps will be marked with requires_ramdisk=False. We’ll document that it’s not recommended to run these steps together with any steps that require IPA.

management.set_bmc_password(new_password)

This step is designed to update the password of the current BMC user (identified by redfish_username).

A delicate process ensures that Ironic is not left with invalid credentials. The process is roughly as follows:

Recovery phase (in case of a step retry):

  • Ironic checks for driver_internal_info[redfish_new_password].

  • If not found, proceed with the update phase. Use the cached session, if it exists.

  • If found, check if the cached session exists and works. If yes, proceed with the update phase, use the cached session.

  • Otherwise, create new sessions with redfish_password, redfish_new_password, redfish_previous_password, and figure out which works. Use the working session for the update phase.

Update phase:

  • Ironic stores the requested password in driver_internal_info[redfish_new_password].

  • Ironic uses Sushy to find the corresponding Account and update its password by a PATCH request.

  • Ironic sets driver_internal_info[redfish_previous_password] to the current value of driver_info[redfish_password].

  • Ironic updates redfish_password to the requested value.

  • Ironic removes the Redfish session from the session cache and closes it.

  • Ironic opens a new session to verify credentials.

  • On success, redfish_new_password and redfish_previous_password are removed.

An automatic rollback on failure is possible here but is left up to a follow-up RFE.

management.configure_authentication(ldap=None, active_directory=None, new_password=None)

This step is designed to update the account service in the BMC. This is the Redfish service responsible for authentication methods and policies.

For example, it can be used to configure LDAP or ActiveDirectory on the BMC and optionally update Ironic with a new password. If new password is not provided, the old password must be valid after the change.

The ldap and active_directory object is a JSON representation of updated fields as defined by the DMTF AccountService schema. Ironic will validate it against a trimmed down version of the schema and PATCH it into the AccountService.

If new password is provided, it’s updated in Sushy and Ironic in a similar way to the set_bmc_password step.

Note

We’ll consider adding OpenID Connect configuration for hardware that supports it, which currently includes Dell. See proof-of-concept OIDC patch.

management.disable_dhcp(persist_addresses=True, persist_nameservers=True, persist_hostname=True, ip_version=None)

Saves the currently used addresses (obtained from the Manager’s EthernetInterface) as statically configured and disables DHCP.

The three parameters control which parts of the configuration needs to be persisted. Only IP versions with DHCP currently enabled are modified. Additionally, ip_version can be used to limit the changes to one IP version.

This step may have to reset the BMC to make the changes take effect.

management.configure_static_networking(addresses=None, gateway=None, nameservers=None, fqdn=None, new_redfish_address=None)

Changes static networking configuration for the BMC. If DHCP is turned on for any IP address family, it’s turned off in the process similarly to the disable_dhcp step.

The addresses argument is a list where both IPv4 and IPv6 addresses can be presented. If addresses is set, and this list does not include the current redfish_address, then new_redfish_address must be provided with one of the addresses from the list. The current redfish_address will be replaced in a lockstep with updating BMC following a process similar to updating password in change_password.

The gateway field provides the default gateway for the interface. It can also be a list, in which cases it must contain exactly one IPv4 address and one IPv6 address.

The nameservers argument configures nameservers.

The fqdn argument allows to set the current host name.

This step may have to reset the BMC to make the changes take effect.

management.configure_ntp(enabled=True, servers=None)

Configures NTP settings on the BMC. If enabled is True and the current list of servers is empty, then a list must be provided as servers argument.

Note

Current time and timezone can already be set via the existing step set_bmc_clock.

management.configure_bmc_access(ipmi=None, host_interface=None)

Allows enabling or disabling access to the BMC via IPMI and via the emulated ethernet host interface.

Non-goals

  • Ironic will not become the primary authority for passwords or network configuration. We’ll expect these to be stored in the user’s inventory database and merely passed to Ironic.

  • Only minimal changes to address Problem description are proposed here. It’s not a goal of this proposal to enable general purpose BMC management in Ironic.

  • General BMC settings are addressed in a different spec.

  • BMCs managing several systems won’t be supported.

Alternatives

The alternative is to rely on 3rd party software for BMC initialization. This has two major downsides:

  • Increased operational complexity and security surface since two components are now responsible for managing BMCs.

  • Potential conflicts because of outdated information in Ironic. Ironic processes that run when updating BMC settings will be interrupted, and there will be a period when Ironic has invalid credentials.

There is a discussion on the general BMC settings spec on whether we should reuse the management interface or create a new bmc interface. This spec will follow the outcome of that discussion.

Redfish details

Credentials are handled by PATCH requests against the Account object linked from AccountService. The account name must match redfish_username. If the AccountService provides MinPasswordLength and MaxPasswordLength, the new password will be checked against them.

LDAP settings are specified in the AccountService itself in the LDAP and ActiveDirectory keys. We’ll PATCH in the entire payload provided by the user (verified against the schema first).

IP addresses are read from IPv4Addresses/IPv6Addresses on an EthernetInterface linked from the Manager and set via IPv4StaticAddresses/IPv6StaticAddresses. Some of the entries may contain placeholders (::), which will be ignored on read and overwritten on write. Similarly, there are NameServers and StaticNameServers.

On some hardware, a Manager can have several interfaces. The code will detect which one is actually used.

DHCP can be managed in two different places. In my testing, HPE supports DHCPv4 and DHCPv6 fields on EthernetInterface, while Dell configures DHCP via the NetworkProtocol object linked directly from the Manager. We’ll support both paths.

Gateway is handled different in case of IPv4 and IPv6. In case of IPv4, it’s a property of an IPAddress, but only one value is allowed for the entire interface per documentation. In case of IPv6, a separate IPv6StaticDefaultGateways field is used.

NTP is configured in NetworkProtocol linked from the manager. Some older BMCs, such as iLO 5, may require changing NTP settings via OEM resources. We’ll consider implementing it based on user demand.

Similarly, IPMI access can be enabled or disabled via NetworkProtocol.

Finally, host interface is controlled via a separate HostInterface collection linked from a Manager. Each interface has an InterfaceEnabled field that can be set to turn it on or off.

Check the following Sushy patches for an idea how these resources look:

As evident in the proof-of-concept OIDC patch, Dell machines may require modifying OEM settings to achieve some of these functionalities. We’ll handle it transparently from the user.

Data model impact

No impact. Some new fields are temporarily stored in driver_internal_info.

State Machine Impact

No impact. Existing steps mechanism is used.

REST API impact

No impact. Existing steps mechanism is used.

Client (CLI) impact

No impact. Existing steps mechanism is used.

“openstack baremetal” CLI

No impact. Existing steps mechanism is used.

“openstacksdk”

No impact. Existing steps mechanism is used.

RPC API impact

No impact. Existing steps mechanism is used.

Driver API impact

The new methods described in Proposed change are added to RedfishManagement. There are no changes to the base interface.

Nova driver impact

None

Ramdisk impact

None. All steps are out-of-band and don’t require ramdisk.

Security impact

This spec together with the general BMC settings spec create a precedent of modifying the BMC itself, not just the machine it manages. The proposed actions are highly privileged and, if misused, can result in a machine takeover or unavailability.

Configuring arbitrary BMC addresses carries a risk of accidentally (or purposely) interfering with other BMCs, potentially requiring manual intervention.

Enabling IPMI or host interfaces can be a major security risk. However, enabling IPMI may also be needed for integration with serial console, which is a tradeoff infrastructure operators need to make.

Finally, there is a (mostly theoretical) risk of a DoS attack against an arbitrary IP address by configuring BMCs with it as an LDAP or NTP server.

Mitigations

Ironic recently got an ability to disable steps (e.g. disable_service_steps). The release note for this feature will mention this option.

The new steps will deviate from the usual RBAC for steps. While normal clean/service steps use SYSTEM_OR_OWNER_MEMBER_AND_LESSEE_ADMIN, here we’ll use SYSTEM_ADMIN_OR_OWNER_ADMIN by default.

The provided steps will have further configuration knobs to reduce the chance of interfering with other BMCs - see Other deployer impact.

We’ll create a new notification type NodeBMCSettings that will be used by all of the new steps. Its payload will include:

  • action - the name of the invoked step,

  • affected_user - current user in case of password change, None otherwise,

  • and potentially modified fields (to be determined).

Finally, we’ll ensure that new_password or any passwords in LDAP/ActiveDirectory payload are not exposed in plain text in any places like steps listing in driver_internal_info or the current step.

Other end user impact

None

Scalability impact

No impact

Performance Impact

No major impact expected. New steps will be used once, set_bmc_password may be used occasionally.

Other deployer impact

Operators may decide to disable some or all of the new steps via the existing mechanism.

Tuning configure_static_networking

We’ll provide knobs for operators to control what networking settings can be changed in the deployment and how:

[redfish]configure_static_networking_allow_arbitrary_addresses=True

If set to True (the default), then a user can provide their own values for addresses and nameservers. Otherwise, the only option available to users will be to copy values from the DHCP list by settings addresses and nameservers to boolean True.

[redfish]configure_static_networking_check_duplicates=True

When addresses are provided, Ironic will check for duplicated BMC addresses in the same conductor group and fail if any are found. Host names will not be resolved for performance reasons.

This option has no effect if configure_static_networking_allow_arbitrary_addresses is False.

Tuning configure_authentication

[redfish]allowed_authentication_servers

When configured, it specifies the list of allowed servers to be used as part of LDAP or ActiveDirectory configuration. Entries starting with a dot a treated as wildcards (e.g. .example.com matches ldap.example.com). IP subnets can be provided as well.

The default is to allow any values.

Tuning configure_ntp

[redfish]allowed_ntp_servers

When configured, it specifies the list of allowed servers to be used as NTP servers for BMCs. Entries starting with a dot a treated as wildcards (e.g. .example.com matches ldap.example.com). IP subnets can be provided as well.

The default is to allow any values.

Tuning configure_bmc_access

[redfish]allow_enabling_ipmi

When set to false, configure_bmc_access cannot be used to enable IPMI, only to disable it. The default is true.

[redfish]allow_enabling_host_interface

When set to false (the default), configure_bmc_access cannot be used to enable host interface, only to disable it.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

Dmitry Tantsur <dtantsur@protonmail.com>, IRC: dtantsur

Other contributors:

TBD

Work Items

  • Implement AccountService API in sushy-tools.

  • Implement the set_bmc_password step.

  • Implement the configure_authentication step with schema validation.

  • Implement fake DHCP settings in sushy-tools.

  • Implement the disable_dhcp step with support for both EthernetInterface and NetworkProtocol DHCP paths.

  • Implement the configure_static_networking step.

  • Implement fake NTP, IPMI, and HostInterface settings in sushy-tools.

  • Implement the configure_ntp step.

  • Implement the configure_bmc_access step.

  • Add the NodeBMCSettings notification type.

  • Add CI tests where possible.

Dependencies

All features need to be available in the hardware. In most cases, we’ll be able to detect their presence and adapt the process or fail before changes are made.

Testing

Password change can be simulated with sushy-tools. To make it CI-compatible, we’ll probably have to implement creating users in the emulator. Then the CI test can create a temporary user, enroll a node that uses it, and update credentials.

We can also test disable_dhcp since it does not change the actual IP address of sushy-tools.

The configure_ntp and configure_bmc_access steps can be emulated on sushy-tools by read/write values that don’t change anything in reality.

Everything else will need to be tested on real hardware.

Upgrades and Backwards Compatibility

No concerns here.

Documentation Impact

The features will be documented together with security trade-offs they make.

References

DMTF schemas: https://redfish.dmtf.org/schemas/v1/