vRouter Hardware Offload Enablement

https://blueprints.launchpad.net/nova/+spec/vrouter-hw-offloads

SmartNICs allow complex packet processing on the NIC. In order to support hardware acceleration for them, Nova core and os-vif needs modifications to support the combination of VIF and vRouter plugging they support. This spec proposes a hybrid SR-IOV and vRouter model to enable acceleration.

Note

In this spec, Juniper Contrail, OpenContrail and Tungsten Fabric will be used interchangeably.

Problem description

SmartNICs are able to route packets directly to individual SR-IOV Virtual Functions. These can be connected to instances using IOMMU (vfio-pci passthrough) or a low-latency vhost-user virtio-forwarder running on the compute node. The vRouter packet processing pipeline is managed by a Contrail Agent. If Offload hooks in kernel vRouter are present, then datapath match/action rules can be fully offloaded to the SmartNIC instead of executed on the hypervisor.

For a deeper discussion on datapath offloads, it is highly recommended to read the Generic os-vif datapath offloads spec.

The vrouter VIF type has not been converted to the os-vif plugin model. This spec proposes completing the conversion to an os-vif plugin as the first stage.

Currently, Nova supports multiple types of Contrail plugging: TAP plugs, vhost-user socket plugs or VEB SR-IOV plugs. Neutron and the Contrail controller decides what VIF type to pass to Nova based on the Neutron port semantics and the configuration of the compute node. This VIF type is then passed to Nova:

  • The vrouter VIF type plugs a TAP device into the kernel vrouter.ko datapath.

  • The vhostuser VIF type with the vhostuser_vrouter_plug mode plugs into the DPDK-based vRouter datapath.

  • The hw_veb VIF type plugs a VM into the VEB datapath of a NIC using vfio-pci passthrough.

In order to enable full datapath offloads for SmartNICs, Nova needs to support additional VNIC types when plugging a VM with the vrouter VIF type, while consuming a PCIe Virtual Function resource.

Open vSwitch offloads recognises the following VNIC types:

  • The normal (or default) VNIC type indicates that the Instance is plugged into the software bridge. The vrouter VIF type currently supports only this VNIC type.

  • The direct VNIC type indicates that a VF is passed through to the Instance.

In addition, the Agilio OVS VIF type implements the following offload mode:

  • The virtio-forwarder VNIC type indicates that a VF is attached via a virtio-forwarder.

Use Cases

  • Currently, an end user is able to attach a port to an Instance, running on a hypervisor with support for plugging vRouter VIFs, by using one of the following methods:

    • Normal: Standard kernel based plugging, or vhost-user based plugging depending on the datapath running on the hypervisor.

    • Direct: PCI passthrough plugging into the VEB of an SR-IOV NIC.

  • In addition, an end user should be able to attach a port to an Instance running on a properly configured hypervisor, equipped with a SmartNIC, using one of the following methods:

    • Passthrough: Accelerated IOMMU passthrough to an offloaded vRouter datapath, ideal for NFV-like applications.

    • Virtio Forwarder: Accelerated vhost-user passthrough, maximum software compatibility with standard virtio drivers and with support for live migration.

  • This enables Juniper, Tungsten Fabric (and partners like Netronome) to achieve functional parity with the existing OVS VF Representor datapath offloads for vRouter.

Proposed change

  • Stage 1: vRouter migration to os-vif.

    • The vRouter os-vif plugin has been updated with the required code on the master branch. Changes in Nova for this stage are gated on a release being issued on that project in order to reflect the specific version required in the release notes.

      Progress on this task is tracked on the vRouter os-vif conversion blueprint.

    • In nova/virt/libvirt/vif.py:

      Remove the Legacy vRouter config generation code, LibvirtGenericVIFDriver.get_config_vrouter(), and migrate the plugging code, LibvirtGenericVIFDriver.{plug,unplug}_vrouter(), to an external os-vif plugin.

      For kernel-based plugging, VIFGeneric will be used.

    • In privsep/libvirt.py

      Remove privsep code, {plug,unplug}_contrail_vif():

      The call to vrouter-port-control will be migrated to the external os-vif plugin, and further changes will be beyond the scope of Nova.

  • Stage 2: Extend os-vif with better abstraction for representors.

    os-vif’s object model needs to be updated with a better abstraction model to allow representors to be applicable to the vrouter datapath.

    This stage will be covered by implementing the Generic os-vif datapath offloads spec.

  • Stage 3: Extend the vrouter VIF type in Nova.

    Modify _nova_to_osvif_vif_vrouter to support two additional VNIC types:

    • VNIC_TYPE_DIRECT: os-vif VIFHostDevice will be used.

    • VNIC_TYPE_VIRTIO_FORWARDER: os-vif VIFVHostUser will be used.

    Code impact to Nova will be to pass through the representor information to the os-vif plugin using the extensions developed in Stage 2.

Summary of plugging methods

  • Existing methods supported by Contrail:

    • VIF type: hw_veb (legacy)

      • VNIC type: direct

    • VIF type: vhostuser (os-vif plugin: contrail_vrouter)

      • VNIC type: normal

      • details: vhostuser_vrouter_plug: True

      • os-vif object VIFVHostUser

    • VIF type: vrouter (legacy)

      • VNIC type: normal

  • After migration to os-vif (Stage 1):

    • VIF type: hw_veb (legacy)

      • VNIC type: direct

    • VIF type: vhostuser (os-vif plugin: contrail_vrouter)

      • VNIC type: normal

      • details: vhostuser_vrouter_plug: True

      • os-vif object: VIFVHostUser

    • VIF type: vrouter (os-vif plugin: vrouter)

      • VNIC type: normal

      • os-vif object: VIFGeneric

  • Additional accelerated plugging modes (Stage 3):

    • VIF type: vrouter (os-vif plugin: vrouter)

      • VNIC type: direct

      • os-vif object: VIFHostDevice

      • port_profile.datapath_offload: DatapathOffloadRepresentor

    • VIF type: vrouter (os-vif plugin: vrouter)

      • VNIC type: virtio-forwarder

      • os-vif object: VIFVHostUser

      • port_profile.datapath_offload: DatapathOffloadRepresentor

Additional notes

  • Stage 1 and Stage 2 can be completed and verified in parallel. The abstraction layer will be tested on the Open vSwitch offloads.

  • Selecting between the VEB passthrough mode and the offloaded vRouter datapath passthrough mode happens at the Contrail Controller. This is keyed on the provider network associated with the Neutron port.

  • The vRouter os-vif plugin has been updated to adopt vrouter as the new os-vif plugin name. contrail_vrouter, is kept as a backwards compatible alias. This prevents namespace fragmentation. Tungsten Fabric, OpenContrail and Juniper Contrail can use a single os-vif plugin for the vRouter datapath.

  • No corresponding changes in Neutron are expected. The Contrail Neutron plugin and agent require minimal changes in order to allow the semantics to propagate correctly.

  • This change is agnostic to the SmartNIC datapath: should Contrail switch to TC based offloads, eBPF or a third-party method, the Nova plugging logic will remain the same for full offloads.

  • A deployer/administrator still has to register the PCI devices on the hypervisor with pci_passthrough_whitelist in nova.conf.

  • SmartNIC-enabled nodes and standard compute nodes can run side-by-side. Standard scheduling filters allocate and place Instances according to port types and driver capabilities.

Alternatives

Alternatives proposed require much more invasive patches to Nova:

  • Create a new VIF type:

    • This would add three VIF types for Contrail to maintain. This is not ideal.

  • Add glance or flavor annotations:

    • This would force an Instance to have one type of acceleration. Code would possibly move out to more VIF types and Virtual Function reservation would still need to be updated.

Data model impact

None

REST API impact

None

Security impact

os-vif plugins run with elevated privileges.

Notifications impact

None

Other end user impact

End users will be able to plug VIFs into Instances with either normal, direct or virtio-forwarder VNIC types on hardware enabled Nova nodes running Contrail.

Performance Impact

This code is likely to be called at VIF plugging and unplugging. Performance is not expected to regress.

On accelerated ports, dataplane performance between Instances is expected to increase.

Other deployer impact

A deployer would still need to configure the SmartNIC components of Contrail and configure the PCI whitelist in Nova at deployment. This would not require core OpenStack changes.

Developer impact

Core Nova semantics will be slightly changed. vrouter VIFs will support more VNIC types.

Upgrade impact

New VNIC type semantics will be available on compute nodes with this patch.

A deployer would be mandated to install the os-vif plugin to retain existing functionality in Nova. This is expected to be handled by minimum required versions in Contrail.

Implementation

Assignee(s)

Primary assignee:

Jan Gutter <jan.gutter@netronome.com>

Work Items

Dependencies

The following dependencies on Tungsten Fabric have been merged on the master branch and are awaiting a release tag:

The following items can occur in parallel:

  • os-vif extensions for accelerated datapath plugin modes need to be released. Consult the Generic os-vif datapath offloads spec for more details. The os-vif library update is planned for the Stein release.

  • Pending release tags on the Contrail os-vif plugin, the vRouter os-vif conversion blueprint can be completed. This is currently planned for the Tungsten Fabric 5.1 release.

Once both of the preceding tasks have been implemented, the following items can occur in parallel:

  • Nova can implement the VNIC support for the contrail os-vif plugin.

  • The contrail os-vif plugin can be updated to use the new os-vif interfaces.

Testing

  • Unit tests have been refreshed and now cover the VIF operations more completely.

  • Third-party CI testing will be necessary to validate the Contrail and Tungsten Fabric compatibility.

Documentation Impact

Since this spec affects a non-reference Neutron plugin, a release note in Nova should suffice. Specific versions of Contrail / Tungsten Fabric need to be mentioned when a new plugin is required to provide existing functionality. The external documentation to configure and use the new plugging modes should be driven from the Contrail / Tungsten Fabric side.