High-Level Workflows for BMC Management¶
https://bugs.launchpad.net/ironic/+bug/2162051
Problem description¶
Ironic does a great job managing the entire lifecycle of machines themselves, but enrolling new hardware often involves several actions to prepare BMC for operations. When an Ironic operator has a high number of machines entering and leaving the cluster(s), it can become a significant burden.
Imagine this scenario. I order a large number (hundreds to thousands) of machines. They came with factory BMC credentials, and the BMCs are capable of receiving networking information via DHCP. This is enough to use the machines with Ironic but is not enough for secure operation:
Factory BMC credentials are well known. And while most users will receive machine-specific credentials from the vendor instead, they’ll probably prefer to change them initially and then rotate on regular basis.
Some deployments may prefer to use LDAP for authentication to ensure that all credentials are managed in a central place.
For availability reasons, BMC networking must not depend on the DHCP server operation.
Similar concerns are important for already provisioned machines:
Credentials rotation may be required by the site policy or jurisdiction.
LDAP settings may need updating.
This proposal focuses on Redfish.
Proposed change¶
While the general BMC settings specification proposes low-level actions on BMCs, this specification covers a few common workflows. All of them have certain pitfalls that justify adding higher-level logic. These include both implementation caveats and drastic vendor differences.
The Redfish management interface gains a number of new clean/service steps. See Redfish details for how the steps are actually implemented in the Redfish level.
All of the steps will be marked with requires_ramdisk=False. We’ll document
that it’s not recommended to run these steps together with any steps that
require IPA.
management.set_bmc_password(new_password)This step is designed to update the password of the current BMC user (identified by
redfish_username).A delicate process ensures that Ironic is not left with invalid credentials. The process is roughly as follows:
Recovery phase (in case of a step retry):
Ironic checks for
driver_internal_info[redfish_new_password].If not found, proceed with the update phase. Use the cached session, if it exists.
If found, check if the cached session exists and works. If yes, proceed with the update phase, use the cached session.
Otherwise, create new sessions with
redfish_password,redfish_new_password,redfish_previous_password, and figure out which works. Use the working session for the update phase.
Update phase:
Ironic stores the requested password in
driver_internal_info[redfish_new_password].Ironic uses Sushy to find the corresponding
Accountand update its password by a PATCH request.Ironic sets
driver_internal_info[redfish_previous_password]to the current value ofdriver_info[redfish_password].Ironic updates
redfish_passwordto the requested value.Ironic removes the Redfish session from the session cache and closes it.
Ironic opens a new session to verify credentials.
On success,
redfish_new_passwordandredfish_previous_passwordare removed.
An automatic rollback on failure is possible here but is left up to a follow-up RFE.
management.configure_authentication(ldap=None, active_directory=None, new_password=None)This step is designed to update the account service in the BMC. This is the Redfish service responsible for authentication methods and policies.
For example, it can be used to configure LDAP or ActiveDirectory on the BMC and optionally update Ironic with a new password. If new password is not provided, the old password must be valid after the change.
The
ldapandactive_directoryobject is a JSON representation of updated fields as defined by the DMTF AccountService schema. Ironic will validate it against a trimmed down version of the schema and PATCH it into theAccountService.If new password is provided, it’s updated in Sushy and Ironic in a similar way to the
set_bmc_passwordstep.Note
We’ll consider adding OpenID Connect configuration for hardware that supports it, which currently includes Dell. See proof-of-concept OIDC patch.
management.disable_dhcp(persist_addresses=True, persist_nameservers=True, persist_hostname=True, ip_version=None)Saves the currently used addresses (obtained from the
Manager’s EthernetInterface) as statically configured and disables DHCP.The three parameters control which parts of the configuration needs to be persisted. Only IP versions with DHCP currently enabled are modified. Additionally,
ip_versioncan be used to limit the changes to one IP version.This step may have to reset the BMC to make the changes take effect.
management.configure_static_networking(addresses=None, gateway=None, nameservers=None, fqdn=None, new_redfish_address=None)Changes static networking configuration for the BMC. If DHCP is turned on for any IP address family, it’s turned off in the process similarly to the
disable_dhcpstep.The
addressesargument is a list where both IPv4 and IPv6 addresses can be presented. Ifaddressesis set, and this list does not include the currentredfish_address, thennew_redfish_addressmust be provided with one of the addresses from the list. The currentredfish_addresswill be replaced in a lockstep with updating BMC following a process similar to updating password inchange_password.The
gatewayfield provides the default gateway for the interface. It can also be a list, in which cases it must contain exactly one IPv4 address and one IPv6 address.The
nameserversargument configures nameservers.The
fqdnargument allows to set the current host name.This step may have to reset the BMC to make the changes take effect.
management.configure_ntp(enabled=True, servers=None)Configures NTP settings on the BMC. If
enabledis True and the current list of servers is empty, then a list must be provided asserversargument.Note
Current time and timezone can already be set via the existing step
set_bmc_clock.management.configure_bmc_access(ipmi=None, host_interface=None)Allows enabling or disabling access to the BMC via IPMI and via the emulated ethernet host interface.
Non-goals¶
Ironic will not become the primary authority for passwords or network configuration. We’ll expect these to be stored in the user’s inventory database and merely passed to Ironic.
Only minimal changes to address Problem description are proposed here. It’s not a goal of this proposal to enable general purpose BMC management in Ironic.
General BMC settings are addressed in a different spec.
BMCs managing several systems won’t be supported.
Alternatives¶
The alternative is to rely on 3rd party software for BMC initialization. This has two major downsides:
Increased operational complexity and security surface since two components are now responsible for managing BMCs.
Potential conflicts because of outdated information in Ironic. Ironic processes that run when updating BMC settings will be interrupted, and there will be a period when Ironic has invalid credentials.
There is a discussion on the general BMC settings spec on whether we should
reuse the management interface or create a new bmc interface. This spec
will follow the outcome of that discussion.
Redfish details¶
Credentials are handled by PATCH requests against the Account object linked
from AccountService. The account name must match redfish_username.
If the AccountService provides MinPasswordLength and
MaxPasswordLength, the new password will be checked against them.
LDAP settings are specified in the AccountService itself in the LDAP
and ActiveDirectory keys. We’ll PATCH in the entire payload provided by the
user (verified against the schema first).
IP addresses are read from IPv4Addresses/IPv6Addresses on an
EthernetInterface linked from the Manager and set via
IPv4StaticAddresses/IPv6StaticAddresses. Some of the entries may
contain placeholders (::), which will be ignored on read and overwritten on
write. Similarly, there are NameServers and StaticNameServers.
On some hardware, a Manager can have several interfaces. The code will
detect which one is actually used.
DHCP can be managed in two different places. In my testing, HPE supports
DHCPv4 and DHCPv6 fields on EthernetInterface, while Dell
configures DHCP via the NetworkProtocol object linked directly from the
Manager. We’ll support both paths.
Gateway is handled different in case of IPv4 and IPv6. In case of IPv4, it’s a
property of an IPAddress, but only one value is allowed for the entire
interface per documentation. In case of IPv6, a separate
IPv6StaticDefaultGateways field is used.
NTP is configured in NetworkProtocol linked from the manager. Some older
BMCs, such as iLO 5, may require changing NTP settings via OEM resources. We’ll consider implementing it
based on user demand.
Similarly, IPMI access can be enabled or disabled via NetworkProtocol.
Finally, host interface is controlled via a separate HostInterface
collection linked from a Manager. Each interface has an
InterfaceEnabled field that can be set to turn it on or off.
Check the following Sushy patches for an idea how these resources look:
As evident in the proof-of-concept OIDC patch, Dell machines may require modifying OEM settings to achieve some of these functionalities. We’ll handle it transparently from the user.
Data model impact¶
No impact. Some new fields are temporarily stored in driver_internal_info.
State Machine Impact¶
No impact. Existing steps mechanism is used.
REST API impact¶
No impact. Existing steps mechanism is used.
Client (CLI) impact¶
No impact. Existing steps mechanism is used.
“openstack baremetal” CLI¶
No impact. Existing steps mechanism is used.
“openstacksdk”¶
No impact. Existing steps mechanism is used.
RPC API impact¶
No impact. Existing steps mechanism is used.
Driver API impact¶
The new methods described in Proposed change are added to
RedfishManagement. There are no changes to the base interface.
Nova driver impact¶
None
Ramdisk impact¶
None. All steps are out-of-band and don’t require ramdisk.
Security impact¶
This spec together with the general BMC settings spec create a precedent of modifying the BMC itself, not just the machine it manages. The proposed actions are highly privileged and, if misused, can result in a machine takeover or unavailability.
Configuring arbitrary BMC addresses carries a risk of accidentally (or purposely) interfering with other BMCs, potentially requiring manual intervention.
Enabling IPMI or host interfaces can be a major security risk. However, enabling IPMI may also be needed for integration with serial console, which is a tradeoff infrastructure operators need to make.
Finally, there is a (mostly theoretical) risk of a DoS attack against an arbitrary IP address by configuring BMCs with it as an LDAP or NTP server.
Mitigations¶
Ironic recently got an ability to disable steps (e.g. disable_service_steps). The release note for this feature will mention this option.
The new steps will deviate from the usual RBAC for steps. While normal
clean/service steps use SYSTEM_OR_OWNER_MEMBER_AND_LESSEE_ADMIN, here we’ll
use SYSTEM_ADMIN_OR_OWNER_ADMIN by default.
The provided steps will have further configuration knobs to reduce the chance of interfering with other BMCs - see Other deployer impact.
We’ll create a new notification type NodeBMCSettings that will be used by
all of the new steps. Its payload will include:
action- the name of the invoked step,affected_user- current user in case of password change,Noneotherwise,and potentially modified fields (to be determined).
Finally, we’ll ensure that new_password or any passwords in
LDAP/ActiveDirectory payload are not exposed in plain text in any places like
steps listing in driver_internal_info or the current step.
Other end user impact¶
None
Scalability impact¶
No impact
Performance Impact¶
No major impact expected. New steps will be used once, set_bmc_password may
be used occasionally.
Other deployer impact¶
Operators may decide to disable some or all of the new steps via the existing mechanism.
Tuning configure_static_networking¶
We’ll provide knobs for operators to control what networking settings can be changed in the deployment and how:
[redfish]configure_static_networking_allow_arbitrary_addresses=TrueIf set to
True(the default), then a user can provide their own values foraddressesandnameservers. Otherwise, the only option available to users will be to copy values from the DHCP list by settingsaddressesandnameserversto booleanTrue.[redfish]configure_static_networking_check_duplicates=TrueWhen
addressesare provided, Ironic will check for duplicated BMC addresses in the same conductor group and fail if any are found. Host names will not be resolved for performance reasons.This option has no effect if
configure_static_networking_allow_arbitrary_addressesisFalse.
Tuning configure_authentication¶
[redfish]allowed_authentication_serversWhen configured, it specifies the list of allowed servers to be used as part of LDAP or ActiveDirectory configuration. Entries starting with a dot a treated as wildcards (e.g.
.example.commatchesldap.example.com). IP subnets can be provided as well.The default is to allow any values.
Tuning configure_ntp¶
[redfish]allowed_ntp_serversWhen configured, it specifies the list of allowed servers to be used as NTP servers for BMCs. Entries starting with a dot a treated as wildcards (e.g.
.example.commatchesldap.example.com). IP subnets can be provided as well.The default is to allow any values.
Tuning configure_bmc_access¶
[redfish]allow_enabling_ipmiWhen set to false,
configure_bmc_accesscannot be used to enable IPMI, only to disable it. The default is true.[redfish]allow_enabling_host_interfaceWhen set to false (the default),
configure_bmc_accesscannot be used to enable host interface, only to disable it.
Developer impact¶
None
Implementation¶
Assignee(s)¶
- Primary assignee:
Dmitry Tantsur <dtantsur@protonmail.com>, IRC: dtantsur
- Other contributors:
TBD
Work Items¶
Implement AccountService API in sushy-tools.
Implement the
set_bmc_passwordstep.Implement the
configure_authenticationstep with schema validation.Implement fake DHCP settings in sushy-tools.
Implement the
disable_dhcpstep with support for bothEthernetInterfaceandNetworkProtocolDHCP paths.Implement the
configure_static_networkingstep.Implement fake NTP, IPMI, and HostInterface settings in sushy-tools.
Implement the
configure_ntpstep.Implement the
configure_bmc_accessstep.Add the
NodeBMCSettingsnotification type.Add CI tests where possible.
Dependencies¶
All features need to be available in the hardware. In most cases, we’ll be able to detect their presence and adapt the process or fail before changes are made.
Testing¶
Password change can be simulated with sushy-tools. To make it CI-compatible, we’ll probably have to implement creating users in the emulator. Then the CI test can create a temporary user, enroll a node that uses it, and update credentials.
We can also test disable_dhcp since it does not change the actual IP
address of sushy-tools.
The configure_ntp and configure_bmc_access steps can be emulated on
sushy-tools by read/write values that don’t change anything in reality.
Everything else will need to be tested on real hardware.
Upgrades and Backwards Compatibility¶
No concerns here.
Documentation Impact¶
The features will be documented together with security trade-offs they make.
References¶
DMTF schemas: https://redfish.dmtf.org/schemas/v1/