Physical Network Awareness¶
Ironic ports and portgroups correspond to the physical NICs that are available on baremetal nodes. Neutron ports are logical ports attached to tenant and provider networks. When booting baremetal nodes in a system with multiple physical networks, the mapping between logical and physical ports must account for the physical network connectivity of the system.
This feature aims to make ironic aware of the
physical_network attribute of
neutron networks and network segments.
A neutron port on a provider network or network segment is associated with a physical network. The physical network concept is used to specify the physical connectivity of networks and network segments in the system.
There are many reasons why an operator might use multiple physical networks, including:
routed provider networks 1
security, through physical segregation of traffic
redundancy, through independence of multiple networks
traffic isolation (e.g. storage vs. application)
different characteristics (bandwidth, latency, reliability)
different technologies (Ethernet, Infiniband, Omnipath, etc.)
It is possible that some or all nodes may be connected to more than one physical network. OpenStack adoption is growing rapidly in the scientific computing community, an area in which the use of nodes connected to multiple physical networks is prevalent.
For context, we provide some examples of how the physical network attribute is
used in other OpenStack components. The neutron ML2 Open vSwitch agent is made
physical network-aware via the
[OVS] bridge_mappings configuration option.
This option maps physical network names to Open vSwitch bridges that have a
physical network interface as a port. This option is used for flat and VLAN
network types, and is taken into consideration when binding ports to network
segments in the neutron server ML2 driver. This acts both to ensure that
physical connectivity allows for the requested logical topology, and also
supports hosts being connected to multiple physical networks.
A similar mapping exists in nova for pass-through of PCI physical devices or
SR-IOV virtual functions via the
Physical Networks in Ironic¶
An ironic port is connected to a physical network. As portgroups are a layer-2 construct, all ports in a portgroup should be in the same physical network. If a neutron port is mapped to a port or portgroup and is attached to a neutron network or network segment on a different physical network, there will be no connectivity between the bare metal node’s NIC and other neutron ports on the network. This is perhaps most obvious when it results in a failure to acquire a DHCP lease for the interface.
The mapping between logical ports in neutron and physical ports and portgroups in ironic has always been somewhat unpredictable 2. The ironic-neutron integration work added support for local link information for ports 3. In the interface attach/detach API work 4 ironic moved the responsibility for attaching virtual interfaces from nova to ironic. In both of these features physical network-awareness was seen as out of scope.
Currently when a virtual interface is attached to a node, the procedure used to map it to an ironic port or portgroup is roughly as follows:
if there is a free portgroup, select one
else if there is a free port with PXE enabled, select one
else if there is a free port with PXE disabled, select one
This algorithm takes no account of the physical network to which the ports and portgroups are connected, and consequently can result in invalid mappings. Further, there is currently no standard way in ironic to specify the physical network to which a port or portgroup is connected.
Provisioning and Cleaning Networks¶
The ironic pluggable network provider 5 work added support for attaching nodes to dedicated provisioning and cleaning networks during provisioning and cleaning operations respectively. Currently, all ironic ports and portgroups with PXE enabled are attached to the provisioning or cleaning network. While this ensures that the node has a port on the provisioning or cleaning physical network, it may result in unnecessary neutron ports being created if some of the ironic ports or portgroups are connected to a different physical network. In practice this can be avoided by disabling PXE on ports where it is not required.
In a system with multiple physical networks where not all nodes are connected to every physical network, it becomes possible for a user to request a logical network topology that cannot be realised. Without awareness of the physical networks that each ironic node is connected to, nova cannot reliably schedule instances to ironic nodes. This problem is considered beyond the scope of this spec.
There are four parts to the proposed solution to this problem.
It must be possible to specify the physical network to which an ironic port is connected.
The ironic network interfaces must account for the physical network of an ironic node’s ports and portgroups when attaching tenant virtual interfaces.
When attaching a node to a provisioning or cleaning network, neutron ports should be created only for ironic ports and portgroups on the same physical network as the provisioning or cleaning network.
The binding profiles of attached neutron ports should be updated with the physical network of the ironic port or portgroup to which the port was mapped.
There are a few options for how ports might be tagged with a physical network:
a new attribute of the
a new attribute of the
a new first class field
Reusing an existing field would certainly be easier to implement than adding a
new one, but OpenStack history has frequently shown that buried fields often
end up needing to become first class citizens. A relevant example here is the
provider:physical_network extension field on neutron networks, which later
became a first class field on network segments 1. Further, if ironic
intends to support physical network-aware scheduling in future, the ability to
efficiently filter ports by their physical network may be advantageous. This
spec therefore proposes to add a new first class
physical_network field to
Port object. For backwards compatibility, this field will be
The process of mapping physical networks to ironic ports is out of scope for ironic. This could be done either through a manual procedure or through an automated process using information gathered during a hardware introspection process. For example, if using ironic inspector to perform introspection it would be possible to create an introspection plugin 6 that maps switch IDs discovered via LLDP to physical networks.
The physical network of a portgroup will be determined through the physical network of its constituent ports. All ports in the portgroup must have the same physical network, and this will be enforced in the ironic API when creating and updating ports.
This has the unfortunate consequence of making it rather unwieldy to update the physical network of the ports in a portgroup, since the ports must be removed from the portgroup while their physical network is updated. This may be improved upon in future through the use of a virtual physical network field in the portgroups API that allows simultaneous update of the physical network field of all the ports in the group.
Mapping Logical Ports to Physical Ports¶
In order to account for physical network connectivity, the virtual
interface attachment algorithm must determine the physical networks that the
neutron port being attached can be bound to. This information is available via
the neutron API as the
physical_network field on network segments in the
port’s network or as
provider:physical_network on the port’s network.
The virtual interface attachment mapping algorithm will be modified to the use the following set of criteria listed in order of descending priority:
reject ports and portgroups with a non-null physical network that is different than all of the network’s physical networks
prefer ports and portgroups with a non-null physical network to ports with a null physical network
prefer portgroups to ports
prefer ports with PXE enabled to ports with PXE disabled
This algorithm provides backwards compatibility for environments in which the
port(s) and/or portgroup(s) associated with the ironic node do not have a
physical_network property configured.
Provisioning and Cleaning Networks¶
In ironic network drivers that support network flipping for provisioning and cleaning operations, we will create neutron ports only for those ironic ports and portgroups that have PXE enabled and are on the same physical network as the provisioning or cleaning network in question, or do not have a physical network specified.
Neutron Port Binding Profiles¶
When attaching virtual interfaces to physical or virtual functions of PCI
network devices, nova sets a
physical_network attribute in the
binding:profile field of the neutron port. Further research is required
to determine what effect it would have if ironic were to do the same.
We could continue to use an unpredictable mapping between logical ports and physical ports. This limits the use of ironic to environments in which there is only one physical network.
We could continue with the existing mapping algorithm in ironic but provide
neutron with the information required to determine whether a mapping is valid
local_link_connection binding information. Ironic would then be
modified to retry interface attachment with a different neutron port if neutron
determined the mapping to be invalid. This method would be inefficient due to
the retries necessary.
We could avoid the need to tag ironic ports with a physical network by
providing a mechanism to map from the information in their
local_link_connection fields to a physical network. This would require
either an addition to ironic’s data model to support Switch objects or a new
neutron API providing a lookup from switch ID to physical network.
Data model impact¶
physical_network field will be added to Port object. In neutron the
physical_network field is defined as
sqlalchemy.String(64), so the same will be used in ironic.
State Machine Impact¶
REST API impact¶
The port REST API will be modified to support the new
field. The field will be readable by users with the baremetal observer role
and writable by users with the baremetal admin role. If the port is a member
of a portgroup, the API will enforce that all ports in the portgroup have the
same value in their physical network field.
Updates to the physical network field of ports will be restricted in the
same way as for other connectivity related fields (link local connection, etc.)
- they will be restricted to nodes in the
The API microversion will be bumped.
Client (CLI) impact¶
The ironic CLI will not be updated.
“openstack baremetal” CLI¶
The openstack baremetal CLI will be updated to support getting and setting the
physical_network field on ports.
RPC API impact¶
Driver API impact¶
Nova driver impact¶
This change should increase the potential security level of an ironic bare metal cloud by supporting multiple segregated physical networks and honoring the physical network restrictions assigned by the operator.
Other end user impact¶
Other deployer impact¶
In order to make use of this feature, deployers must tag ironic ports and portgroups with the physical network to which they are attached. This implies that they must have a mechanism to correctly determine this information.
Modify the ironic port model to include a
Modify the ironic ports REST API to support the
Modify the openstack baremetal CLI to support the
Modify the ironic
VIFPortIDMixinplugin with the new port mapping algorithm.
Modify the ironic
NeutronNetworknetwork driver to be physical network- aware when creating neutron ports for cleaning and provisioning.
Modify the ironic network drivers to add the physical network to neutron ports’ binding profiles.
Add support for multiple (virtual) physical networks to DevStack.
Update the ironic developer documentation to cover the use of physical networks.
Support will be added to DevStack for ironic environments with multiple (virtual) physical networks.
Upgrades and Backwards Compatibility¶
The proposed data model and algorithm changes are backwards compatible. A
database migration will be provided to add the
physical_network field to
existing ports with a null value.
The ironic developer documentation will be updated to cover the use of this feature.