neutron-specs

Example Spec - The title of your RFE

Thu, 12 Mar 2026 00:00:00

Include the URL of your launchpad RFE:

https://bugs.launchpad.net/neutron/+bug/example-id

Introduction paragraph – why are we doing this feature? A single paragraph of prose that deployers, and developers, and operators can understand.

Do you even need to file a spec? Most features can be done by filing an RFE bug and moving on with life. In most cases, filing an RFE and documenting your design in the devref folder of neutron docs is sufficient. If the feature seems very large or contentious, then the drivers team may request a spec, or you can always file one if desired.

Problem Description

A detailed description of the problem:

For a new feature this should be a list of use cases. Ensure that you are clear about the actors in each use case: End User vs Deployer. Ensure that you identify which area of the core is being affected; for something completely new, it should be clear why you are considering it being part of the core.
For a major reworking of something existing it would describe the problems in that feature that are being addressed.

Note that the RFE filed for this feature will have a description already. This section is not meant to simply duplicate that; you can simply refer to that description if it is sufficient, and use this space to capture changes to the description based on bug comments or feedback on the spec.

Proposed Change

How do you propose to solve this problem?

This section is optional, and provides an area to discuss your high-level design at the same time as use cases, if desired. Note that by high-level, we mean the “view from orbit” rough cut at how things will happen.

This section should ‘scope’ the effort from a feature standpoint: how is the ‘neutron end-to-end system’ going to look like after this change? What Neutron areas do you intend to touch and how do you intend to work on them? The list below is not meant to be a template to fill in, but rather a jumpstart on the sorts of areas to consider in your proposed change description.

Am I going to see new CLI commands? * Is OpenStack CLI covered in addition to neutronclient?
How do you intend to support or affect aspects like: * Address Management, e.g. IPv6, DHCP * Routing, e.g. DVR/HA * Plugins, ML2 Drivers, e.g. OVS, LinuxBridge * Agents, e.g. metadata * High level services, e.g. *-aas. * Scheduling, quota, and policy management, e.g. admin vs user rights * API and extensions * Clients * Impact on services or out-of-tree plugins/drivers
What do you intend to not support in the initial release?

You do not need to detail API or data model changes. Details at that level of granularity belong in the neutron devref docs.

References

Please add any useful references here. You are not required to have any reference. Moreover, this specification should still make sense when your references are unavailable.

PVLAN Semantics for Provider Networks

Fri, 30 Jan 2026 00:00:00

RFE: https://bugs.launchpad.net/neutron/+bug/2138746

Private VLAN (PVLAN) [1] is a device isolation mechanism through the application of special Layer 2 forwarding constraints.

This spec describes how to implement the PVLAN mechanism by creating a service plugin on Neutron, that can be extensible to any ML2 mechanism driver, but that will be mainly focused on the implementation of the ML2/OVN logic.

Problem Description

Current implementations of Security Groups or other networking isolation tools do not comply with this security-focused approach to device isolation. PVLAN is an established mechanism [1] that has well-defined boundaries and rules for consistent networking management. There is currently not a straight-forward way of replicating these boundaries and rules to achieve a consistent implementation of PVLAN in Neutron with existing plugins and/or extensions. A mix of security groups and security group rules could make a certain replication but it would not be flexible or robust enough compared to implementing its own mechanism.

What PVLAN proposes is a consistent semantic model [2] [3] to enforce port isolation rules:

Network attributes

pvlan {{enabled | disabled}}: enable PVLAN semantics for a network.

Port attributes

Roles for ports (Enforced label if pvlan is enabled):
- Promiscuous: Can reach any other port.
- Community: Can reach only ports within their community + promiscuous. One port can only be part of one community.
- Isolated: Can only reach promiscuous, if any.
Community label (if applies) {{ string }}: To subgroup community ports

In the following diagram taken from the OVN proof-of-concept [4] we can understand how the different VMs would be configured depending on their role.

┌────────────────────────────────────────────────────────────────────────┐
│                                                                        │
│                      ┌─────────────────────┐                           │
│                      │   Promiscuous       │ ← Can communicate with    │
│                      │   192.168.1.30      │   ALL ports               │
│                      └──────────┬──────────┘                           │
│                                 │                                      │
│           ┌─────────────────────┼──────────────────────┐               │
│           │                     │                      │               │
│           ▼                     ▼                      ▼               │
│    ┌──────────────────┐ ┌──────────────────┐    ┌──────────────────┐   │
│    │  Isolated        │ │  Community-1     │    │  Community-2     │   │
│    │                  │ │                  │    │                  │   │
│    │  ┌────┐  ┌────┐  │ │  ┌────┐  ┌────┐  │    │  ┌────┐  ┌────┐  │   │
│    │  │iso1│  │iso2│  │ │  │1-c1│◄►│2-c1│  │    │  │1-c2│◄►│2-c2│  │   │
│    │  │.1  │  │.2  │  │ │  │.10 │  │.11 │  │    │  │.20 │  │.21 │  │   │
│    │  └────┘  └────┘  │ │  └────┘  └────┘  │    │  └────┘  └────┘  │   │
│    │                  │ │                  │    │                  │   │
│    └──────────────────┘ └──────────────────┘    └──────────────────┘   │
│   (Can only talk          (Members can talk     (Members can talk      │
│    to Promiscuous)         within community      within community      │
│                            + Promiscuous)        + Promiscuous)        │
│                                                                        │
│                                                                        │
└────────────────────────────────────────────────────────────────────────┘

Proposed Change

A new service plugin of Neutron that will implement new API extension and will be extensible by the various backend drivers.

Note

The scope of this spec is to complete the driver implementation of ML2/OVN. Any interested contributor is welcomed to implement the ML2/OVS backend.

REST API Impact

As described before, there are new attributes present on both networks and ports.

New table networkpvlan extending networks:

network_id - UUID type.
pvlan - boolean type. True if PVLAN is enabled. Default: False.

Example:

network_id	pvlan
2c1b70d6-1ffb-43e8-8069-53f5597c446e	0

POST /v2.0/networks
Accept: application/json

{
"network": {
    "name": "pvlan-net-1",
    "admin_state_up": true,
    "pvlan": true}
}

Response:

{
 "network": {
     "id": "a9254bdb-2613-4a13-ac4c-adc581fba50d",
     "name": "pvlan-net-1",
     "status": "ACTIVE",
     "subnets": [],
     "admin_state_up": true,
     "shared": false,
     "project_id": "fb0e16da8ce7435c8a444647738e2166",
     "pvlan": true,
     "tenant_id": "fb0e16da8ce7"
     }
 }

New table portpvlan extending ports:

port_id - UUID type.
pvlan_type - string type. Allowed values: “isolated”, “community”, “promiscuous”. Default: Promiscuous.
pvlan_community - string type. Allowed values: Any string that complies with Port Group name restriction notes in OVN [5] — Names are ASCII and must match [a-zA-Z_.][a-zA-Z_.0-9]*. Default: None.

Example:

port_id	pvlan_type	pvlan_community
7abca15f-a19a-4f4d-a27c-b695ca1ec44b	community	community_1
e66abfad-a681-4a90-a6f7-8efe5e4bd2ae	isolated	NONE

POST /v2.0/ports
Accept: application/json

{
"port": {
    "name": "pvlan-port-1",
    "network_id": "03848825-c4fc-44c1-b510-fc6bf64cc8ef",
    "pvlan_type": "community",
    "pvlan_community": "community_1"}
}

Response:

{
"port": {
    "id": "e66abfad-a681-4a90-a6f7-8efe5e4bd2ae",
    "name": "port_2",
    "network_id": "03848825-c4fc-44c1-b510-fc6bf64cc8ef",
    "mac_address": "fa:16:3e:a8:70:16",
    "status": "DOWN",
    "device_id": "f51da0f6-f1e2-48cb-b46d-d37a262f383e",
    "pvlan_type": "community",
    "project_id": "fb0e16da8ce7435c8a444647738e2166",
    "pvlan_community": "community_1"}
}

Sub Resource Extension

It will be created as an extension that extends the parameters for network and for port:

RESOURCE_ATTRIBUTE_MAP = {
    network.COLLECTION_NAME: {
        pvlan_const.PVLAN_MODE: {
            'allow_post': True,
            'allow_put': True,
            'is_visible': True,
            'default': False,
            'enforce_policy': True,
            'validate': {'type:boolean': None},
            'convert_to': converters.convert_to_boolean}
    },
    port.COLLECTION_NAME: {
        pvlan_const.PVLAN_TYPE: {
            'allow_post': True,
            'allow_put': True,
            'is_filter': True,
            'default': pvlan_const.PROMISCUOUS_TYPE,
            'is_visible': True,
            'validate': {'type:values': pvlan_const.PVLAN_TYPES}
        },
        pvlan_const.PVLAN_COMMUNITY: {
            'allow_post': True,
            'allow_put': True,
            'is_visible': True,
            'default': None,
            'enforce_policy': True,
            'validate': {
                'type:regex_or_none': pvlan_const.COMMUNITY_NAME_REGEX}
        }
    }
}

Service Plugin

The new service plugin pvlan will implement the API extension and will extend Neutron resources with the attributes needed for port and network. It will also define the new DB objects and will call the backend driver for the attribute operations (create / update / delete) implementation.

ML2/OVN Driver Implementation details

The backend driver needs to implement the operations by consuming Neutron notifications about creation, deletion and update of the Networks and Ports.

The operations to be implemented are:

Network create (enable pvlan) -
1. Create pvlan_drop_all, pvlan_isolated and pvlan_promiscuous Port Groups for the network (Logical Switch).
2. Create ACLs for those Port Groups.

Port Group example:

_uuid               : 6dfac905-892c-44f6-880c-7168399e7240
acls                : [f21e358d-ce0b-4741-80bf-a13abeefc221, 41cfb444-c744-4d4f-be2f-706f17bab37e]
external_ids        : {"neutron:network"="cb33079f-9309-4312-bd0f-9d57d20c8c0a"}
name                : pg_pvlan_community_1
ports               : [e9c40083-d420-47c9-86ec-d4de363d8c1b]

ACL example:

_uuid               : f21e358d-ce0b-4741-80bf-a13abeefc221
action              : allow-stateless
direction           : to-lport
external_ids        : []
label               : 0
log                 : false
match               : "outport == @pg_pvlan_community_1 && inport == @pg_pvlan_community_1 && (ip || arp)"
meter               : []
name                : []
options             : {}
priority            : 1010
severity            : []
tier                : 0

Network delete (disable pvlan) - delete pvlan_drop_all, pvlan_isolated and pvlan_promiscuous Port Groups for the network (Logical Switch), delete ACLs created for those Port Groups.
Port create -
- If pvlan_type=isolated or pvlan_type=promiscuous - add port to the specific port group.
- If pvlan_type=community - ensure that Port Group and ACLs for that Port Group are created and add port to that Port Group.
Port update - Depending on the updated value, remove port from one Port Group and add to another.
Port delete - Delete port from the corresponding Port Group. If pvlan_type=community and it is the last port in the Port Group corresponding to that community name, remove that Port Group as well.

In ML2/OVN Port Groups can serve as a tool for defining Communities within a PVLAN domain. In this implementation there would also be a specific Port group for ports from isolated VMs, and another one for ports from promiscuous VMs. Each Port group will be then associated with ACLs that will define their behaviour.

Since ACLs have a priority parameter, we can tune them so that they will interact as intended e.g. right now the ‘neutron_pg_drop’ related ACLs have lower priority than the security group ACLs. With PVLAN the pg_drop_group would still exist (or we would create a similar one just for PVLAN logic, depending on what is best). This Port group would have the lowest priority. Security group rules could coexist with PVLAN by placing on PVLAN a higher priority on its ACLS, making the PVLAN ACLs be the ones to be used if they exist.

We don’t need to track state of the connection so ACLs actions will be stateless, which is better in terms of the performance in some cases.

Note

Ports with disabled port security in the network will be forbidden if pvlan is enabled, as there is no a better solution that we can think of for its coexistence.

Warning

Since the implementation in ML2/OVN uses ACLs, there is no compatibility with external PVLAN networks.

Remaining questions on Implementation

Should we disable arp_responder for LSP to be able to block that traffic too? Or maybe it is possible to configure OVN to respond to ARP traffic “selectively”?
Would pg_drop_group be different to the general drop port group of PVLAN or can we use the same one?

Upgrading

This feature does not affect how current components work, so no extra effort should be done regarding upgrades.

Implementation

Assignee(s)

Slawek Kaplonski
Elvira García

Work Items

API Definition
Service implementation
ML2/OVN logic implementation

Testing

Unit/functional tests.
Scenario test in neutron tempest.

Documentation

User Documentation: How to set up PVLAN in Neutron with ML2/OVN

References

Routed Provider Networks with Multiple Segments per Host for ML2/OVN

Wed, 05 Nov 2025 00:00:00

https://bugs.launchpad.net/neutron/+bug/2130453

As implemented by the ML2/OVN driver, routed provider networks are limited to one segment per host [1] [2]. This specification proposes to remove such limitation, enabling multiple segments per host per network.

Problem Description

Originally, all Neutron networks were single L2 broadcast domains. In such networks, performance degrades as traffic grows as a result of an increasing number of active VMs/ports. Routed provider networks were implemented to overcome this limitation and enable users to create “large networks”, where a large number of VMs/ports can be connected without incuring the performance penalty of large single L2 broadcast domains. As shown in the following diagram, routed provider networks are constituted by several L2 segments (broadcast domains) stitched together by a router into one L3 “large network”. In this arrangement, each individual segment handles only a portion of the total network traffic, improving overall performance. The router is not part of Neutron; it is provided by the underlying networking infrastructure [3].

In routed provider networks each segment is connected to a group of hosts (a Nova aggregate), as shown in the following diagram. In the optimal situation, the network traffic generated by the workload running in the hosts doesn’t exceed the capacity of the corresponding segment.

Recently, though, some operators have found this not to be the case. They can comfortably accomodate in the hosts workloads that generate network traffic that exceeds their segment capacity. In such situations, more than one segment per host is necessary if the deployers are going to fully utilize their compute resources, while at the same time achieving the benefits of the routed provider networks, by effectively limiting the size of the individual L2 broadcast domains. This is shown in the following diagram.

Proposed Change

This specification proposes to implement each segment in a routed provider network as an OVN Logical Switch. Each one of these logical switches will be associated with a Logical Switch Port of type localnet, that will map the segment to a physical network in the hosts connected to it. As a consequence, for routed provider networks, the one-to-one mapping between a Neutron network and an OVN Logical Switch will no longer be true. The following diagram summarizes the proposed approach, using a Neutron network named public and segments with vlan-ids 42, 43 and 44 as examples:

At the chassis level, this design will be implemented as depicted in the following diagram:

In this diagram, the key features of the proposed design are:

The four bridges br-ex* represent the routed provider network named public depicted in the previous diagram with its three segments.
Each of the bridges br-ex-42, br-ex-43 and br-ex-44 represents one of the segments in the routed provider network.
For each segment, there will be a key-value pair in OVS’s external_ids:ovn-bridge-mappings. In the public network example we are using in this specification, for the physnet identified as physnet-43, the corresponding mapping is physnet-43:br-ex-43. It is the presence of these mappings that triggers the ovn-controller to configure the patch ports on the br-int side of the segment bridges. It is important to note that br-ex is not part of external_ids:ovn-bridge-mappings. The ovn-controller doesn’t interact with that bridge.
There are two alternatives for the creation of the br-ex* bridges and the configuration of the br-ex side of the patch ports. They can be created automatically by Neutron or they can be created as a result of system administration activities. To select from these two alternatives, a large user of routed provider networks with the ML2/OVS driver was asked how frequently they have added segments to their hosts. They responded that they add segments every month. Based on this information, this specification proposes to develop an agent that will create bridges and configure them. It is important to note that since br-ex provides connectivity to the underlay network, it will still be created by the cloud operator.

To implement the proposed new functionality, the following changes to the code are expected:

When a segment is created for a routed provider network, an associated Logical Switch and Logical Switch Port of type localnet will need to be created in the OVN NBDB. Correspondingly, these OVN resources will have to be removed when the segment is deleted.
When a port is created for a routed provider network, the creation of the associated Logical Switch Port will have to be deferred until the moment when the segment to which it is bound is known. Correspondingly, when a port is deleted, its associated Logical Switch Port will have to be removed from the correct Logical Switch.
The OVN maintenance and DB synchronization periodic jobs must be updated to account for the changes described in the previous two points.
For routed provider networks, the logical switch associated to each segment will have its own localport to serve metadata to VMs. This means that the metadata agent will be updated to provision the datapath in each chassis for localports associated to segments.
A neutron-ovn-agent extension will be developed that will be responsible for creating and configuring the bridges that represent the routed provider network segments at the chassis level. When a segment is added, the extension will add the corresponding bridge and add it to the OVS OpenvSwitch table ovn-bridge-mappings attribute. Patch ports will be created between the new bridge and br-int and br-ex. The br-ex side of the patch port will be configured with the correct tag and trunk attributes. When a segment is removed, these steps will be undone.

The implementation will be carried out in two phases. In the first phase, the core functionality described in the first four points above will be implemented. In the second phase, the neutron-ovn-agent extension to manage the segment bridges at the chassis level will be developed. This approach will allow us to start testing the core funcionality as soon as possible while giving us more time to develop the neutron-ovn-agent extension.

References

Core OVN BGP integration

Wed, 18 Jun 2025 00:00:00

https://bugs.launchpad.net/neutron/+bug/2111276

OVN 25.03 introduces BGP-related capabilities that provide parity with the current ovn-bgp-agent underlay exposing method. This spec is to introduce a design using the OVN capabilities integrated with Neutron to replace the ovn-bgp-agent.

Problem Description

The ovn-bgp-agent is another process running on compute and network nodes. There is a lot of processing happening when new workloads are created or moved around because the agent needs to pay attention to these changes and rewire configuration on the node as needed. As OVN is well aware of the locality of its resources we can leave all the processing up to OVN and only manage underlying BGP OVN topology in Neutron and still use the pure L3 spine-and-leaf topology for the dataplane traffic.

Acronyms used in this spec

BGP: Border Gateway Protocol
ECMP: Equal-Cost Multi-Path
LRP: Logical Router Port
LSP: Logical Switch Port
LS: Logical Switch
LR: Logical Router
VRF: Virtual Routing and Forwarding
FRR: Free Range Routing (https://github.com/FRRouting/frr)

Proposed Change

This spec proposes to introduce a new Neutron service plugin that manages the underlying BGP topology in OVN. Its main purpose is to make sure the OVN resources related to BGP are correctly configured at all times by being a reconciler over those resources. Additionally it takes care of scaling in and out the compute nodes because every compute node needs to have their own bound resources, such as a router and a logical switch with a localnet port.

There is no need to make any changes to API or database models. However, there is a need to modify Neutron OVN DB sync scripts to not monitor the underlying BGP resources. Possibly this was already planned to exist in Neutron with the spec at [1] so we need to revive the work. That can be achieved by setting an explicit tag in the external_ids column of the BGP managed resources that Neutron would not touch. Also, we need to make sure on the presentation layer that all the underlying BGP resources are not exposed to the users through an API. For example, a router list command must not return the BGP routers.

Each compute node requires a running FRR instance that monitors the local VRF and advertises the routes to the BGP peers. It is the installer’s responsibility to configure the FRR instance to use the correct BGP parameters and to connect to the correct BGP peers.

As it is easier to understand the topology visually than through description, the following diagram shows the underlying BGP logical topology in OVN. For better resolution, it is recommended to open the image in a new tab.

OVN BGP Logical Topology (click for full resolution)

BGP distributed logical router

A new router with the OVN BGP enabled capabilities is introduced, the router is named “BGP distributed router” in the diagram above, with dynamic routing flag enabled. The router is connected to the provider logical switch with a dummy connection. This connection is not used for any traffic and serves only to logically connect the logical switch and the BGP router so the northd can create entries in the Advertised_Route table in the Southbound DB for the IPs that need to be advertised.

The router also connects to a LS with a localnet port. This LS is connected to the provider bridge br-bgp that needs to be configured on every chassis since the traffic here is distributed and can happen on any node. This bridge connects to the ls-public LS through the localnet port created by Neutron. This LS is what typically connects the physical network in the traditional deployments. We need to have localnet ports to avoid OVN sending traffic over the geneve tunnel to the node hosting the logical router gateway.

The BGP distributed router is connected to the per-chassis logical routers through peered LRPs and bound to the corresponding chassis. The LRs per chassis are described in the next section. Because the BGP router is distributed we need to pick the right LRP so the traffic is not forwarded to a different chassis. For example, if there is egress traffic coming from a tenant LSP on chassis A, the BGP distributed router needs to route the traffic to the LRP on chassis A. For this we will use logical routing policy and the is_chassis_resident match. An example of the logical routing policy is shown below:

action              : reroute
bfd_sessions        : []
chain               : []
external_ids        : {}
jump_chain          : []
match               : "inport==\"lrp-bgp-main-router-to-ls-interconnect\" && is_chassis_resident(\"cr-lrp-bgp-main-router-to-bgp-router-r0-compute-0\")"
nexthop             : []
nexthops            : ["169.254.0.1"]
options             : {}
priority            : 10

The nexthop in this case is the LRP on the chassis A and for now must be an IPv4 as OVN currently contains a bug that prevents the use of IPv6 LLAs as nexthops, reported at [2]. The policy is implemented only on the chassis defined in is_chassis_resident and hence the traffic will always remain local to the chassis. Because the policy is at a later stage in the LR pipeline we need to create a logical router static route in order to pass the routing phase. Hence the BGP distributed logical router needs to contain two static routes. One to route ingress traffic to the provider network and one unused route that serves only to pass the routing stage in the pipeline until the reroute policy is hit.

The first static route can look like this:

bfd                 : []
external_ids        : {}
ip_prefix           : "192.168.111.0/24"
nexthop             : "192.168.111.30"
options             : {}
output_port         : lrp-bgp-main-router-to-ls-interconnect
policy              : []
route_table         : ""
selection_fields    : []

where the ip_prefix is the provider network prefix and the output_port is the LRP connecting to the provider LS. The nexthop is the LRP of the Neutron router port that serves as a gateway.

The second static route is unused and can look like this:

bfd                 : []
external_ids        : {}
ip_prefix           : "0.0.0.0/0"
nexthop             : "192.168.111.30"
options             : {}
output_port         : []
policy              : []
route_table         : ""
selection_fields    : []

The route needs to match all traffic and the nexthop doesn’t matter because it will be determined by the reroute policies based on the chassis locality. The ingress logical router pipeline with the route implemented looks like this:

... the other routes are here but none matches 0.0.0.0/0 ...
table=15(lr_in_ip_routing   ), priority=4    , match=(reg7 == 0 && ip4.dst == 0.0.0.0/0), action=(ip.ttl--; reg8[0..15] = 0; reg0 = 192.168.111.30; reg5 = 192.168.111.30; eth.src = 00:de:ad:10:00:00; outport = "lrp-bgp-main-router-to-ls-interconnect"; flags.loopback = 1; reg9[9] = 1; next;)
table=15(lr_in_ip_routing   ), priority=0    , match=(1), action=(drop;)
table=16(lr_in_ip_routing_ecmp), priority=150  , match=(reg8[0..15] == 0), action=(next;)
table=16(lr_in_ip_routing_ecmp), priority=0    , match=(1), action=(drop;)
table=17(lr_in_policy       ), priority=10   , match=(inport=="lrp-bgp-main-router-to-ls-interconnect" && is_chassis_resident("cr-lrp-bgp-main-router-to-bgp-router-r0-compute-0")), action=(reg0 = 169.254.0.1; reg5 = 169.254.0.2; eth.src = 00:de:ad:00:10:00; outport = "lrp-bgp-main-router-to-bgp-router-r0-compute-0"; flags.loopback = 1; reg8[0..15] = 0; reg9[9] = 1; next;)

As we need to get to the stage where the reroute policy is hit, we need to pass the lr_in_ip_routing stage first and this stage is implemented with a static route. That means we match the 0.0.0.0/0 prefix using the first rule and then later we change the output_port with the last rule with its reroute action. If the static route would not be present, the traffic would be dropped with the second rule containing the drop action.

Per-chassis logical routers

There is also a logical router created and bound to each chassis. These routers serve to learn ECMP routes from the BGP peers and to forward traffic between the provider bridges and the BGP distributed router.

For cases where the compute nodes share data plane and control plane traffic over the same spine-and-leaf topology, there is a need to maintain openflow rules on the provider bridge that differentiate traffic between control plane, and hence forward traffic to the host, and the dataplane traffic that needs to go to the OVN overlay. The following openflow rules could be used to achieve this:

priority=10,ip,in_port=eth0,nw_dst=<host IPs> actions=NORMAL
priority=10,ipv6,in_port=eth0,ipv6_dst=<host IPv6s> actions=NORMAL
priority=10,arp actions=NORMAL
priority=10,icmp6,icmp_type=133 actions=NORMAL
priority=10,icmp6,icmp_type=134 actions=NORMAL
priority=10,icmp6,icmp_type=135 actions=NORMAL
priority=10,icmp6,icmp_type=136 actions=NORMAL
priority=10,ipv6,in_port=eth0,ipv6_dst=fe80::/64 actions=NORMAL
priority=8,in_port=eth0 actions=mod_dl_dst:<LRP MAC>,output:<patch_port_to_ovn>

Those rules match traffic that is destined to the host and forward it to the host. Everything else is forwarded to the OVN overlay. The patch_port_to_ovn is a patch port that ovn-controller created based on the ovn-bridge-mappings configuration.

The router itself needs to implement routes for traffic coming from the provider network and for traffic coming from the OVN overlay. For ingress provider network traffic, the routes can look as follows:

bfd                 : []
external_ids        : {}
ip_prefix           : "192.168.111.0/24"
nexthop             : "169.254.0.2"
options             : {}
output_port         : lrp-bgp-router-r0-compute-0-to-bgp-main-router
policy              : []
route_table         : ""
selection_fields    : []

where ip_prefix matches the subnet of the provider network and the nexthop is set to the address of the LRP attached to the BGP distributed router and the output_port is set to its peer LRP.

The egress traffic from the OVN overlay needs to be routed with ECMP to the BGP network. This can be achieved with the following static routes for each BGP peer:

bfd                 : []
external_ids        : {}
ip_prefix           : "0.0.0.0/0"
nexthop             : "100.64.0.1"
options             : {}
output_port         : lrp-bgp-router-r0-compute-0-to-ls-r0-compute-0-eth0
policy              : []
route_table         : ""
selection_fields    : []

bfd                 : []
external_ids        : {}
ip_prefix           : "0.0.0.0/0"
nexthop             : "100.65.0.1"
options             : {}
output_port         : lrp-bgp-router-r0-compute-0-to-ls-r0-compute-0-eth1
policy              : []
route_table         : ""
selection_fields    : []

Traffic flow

This section describes the traffic flow from and to a LSP hosted on a chassis.

An example of the traffic from the external network to a VM with a Floating IP on chassis 1

Because of the dummy connection between the ls-public LS and the BGP distributed router, OVN creates an Advertised_Route entry for the Floating IP. Because the associated logical port is bound to the chassis 1, OVN populates the local VRF on the chassis 1 with the route to the Floating IP and the local FRR instance advertises the route to the BGP peers.

The fabric learns the route to the Floating IP from the BGP peers and forwards the traffic to the chassis 1 to either eth0 or eth1 because of the ECMP routes.

The traffic does not match any of the higher priority openflow rules on the provider bridge and matches the last rule. The rule changes the destination MAC to the LRP MAC address of the per-chassis router associated with the NIC and the traffic is forwarded to OVN. The traffic enters the per chassis logical router that has Logical_Static_Route configured to forward the traffic to the distributed BGP router. The BGP distributed router is configured to forward the traffic to the ls-inter-public switch with a Logical_Static_Route matching the destination IP with the provider network subnet and through the br-bgp provider bridge the traffic gets to the ls-public logical switch. From here the traffic follows the same path as without BGP and is NAT’ed by the Neutron router.

An example of the traffic from a VM with a Floating IP on chassis 1 to the external network

The egress VM traffic is NAT’ed by the Neutron router and the traffic is forwarded to the provider network gateway which is connected to the ls-public LS. Because of the presence of the localnet ports the traffic gets through the br-bgp bridge to the distributed BGP router where it matches the artificial Logical_Router_Static_Route to skip the lr_in_ip_routing stage in the pipeline and will be matched with the BGP router policy based on the chassis locality. The reroute action of the policy will pick the right LRP that is connected to the per-chassis router. Here the traffic matches static routes per peer and with the ECMP is forwarded to the BGP networks.

Testing

Existing tempest tests should provide good regression testing. We can reuse the existing topology from the ovn-bgp-agent project that peer with VMs simulating a BGP router.

Implementation

The implementation is split into two parts. The first part creates the service plugin that takes care of the BGP topology in OVN including the configuration of static routes and router policies.

The second part is an OVN agent extension [3] that configures per chassis host configurations. The OVN agent itself is orthogonal to the BGP service plugin and can be replaced with any third-party tool that takes care of the node dynamic configuration. The OVN agent extension is responsible for steering the traffic to the OVN and for configuring the per chassis host configurations such as adding ovn-bridge-mappings per the BGP peer, and for implementing local openflow rules to differentiate traffic between the control plane and the dataplane. It also monitors patch ports on the br-bgp and creates direct connection between the localnet ports to avoid any FDB learning on the bridge. An example of the simple openflow rules is shown below:

priority=10,in_port=2, actions=output:3
priority=10,in_port=3, actions=output:2

where 2 is the patch port to the logical switch connected to the BGP distributed router and 3 is the patch port connected to the Neutron public switch.

Where BGP is used with Neutron or not is determined by enabling the service plugin and the OVN agent extension.

As it is written in the first paragraph of this spec, the BGP support in OVN was introduced in 25.03. Therefore the BGP service plugin requires OVN 25.03 or later.

Assignee(s)

Jakub Libosvar <jlibosva@redhat.com>

Documentation

Deployment guide will be written in describing how to enable the service plugin and what needs to be configured on the nodes, such as steering traffic to the OVN or configuration of a BGP speaker advertising the routes to its peer. An example using FRR configuration will be introduced so the operators have a reference for the configuration.

Distributed DNS Forwarder with Openvswitch Agent

Tue, 17 Jun 2025 00:00:00

RFE: https://bugs.launchpad.net/neutron/+bug/2112446

Neutron DHCP agents (using dnsmasq) can act as DNS forwarders or provide DNS resolution for instances by using resolvers configured on the host where the DHCP agent runs. However, when users deploy Distributed DHCP alongside the Open vSwitch (OVS) agent, they are responsible for ensuring that instances can reach DNS servers or for setting up their own forwarding mechanism.

This spec describes how to implement a DNS forwarder extension for Neutron openvswitch agent to provide a simple, fully distributed DNS forwarding capability for virtual machines by leveraging OpenFlow rules and os-ken applications.

Problem Description

DNS plays a critical role in ensuring high availability for services such as databases, VPN clusters, or any failover scenarios where DNS records are updated dynamically whether within OpenStack or through external systems. In addition, DNS resolution is essential in OpenStack environments to allow instances on external or project networks to those services, even if they don’t have a direct path to DNS servers. For instance, a web server running in an isolated VXLAN network might need to connect to a database service (provided by the cloud provider) on the same network using an internal domain name.
Distributed DHCP for Openvswitch Agent is good for large-scale deployments, but it lacks built-in DNS resolution capabilities, unlike the DHCP agent that uses dnsmasq. [1]

Proposed Change

A new extension of Neutron openvswitch agent will be added to achieve the Distributed DNS Forwarder with Openvswitch Agent.

Note

This extension is only for openvswitch agent, other mechanism drivers will not be considered, because this new extension will rely on the openflow protocol and principle. For OVN, it has supported similar DNS resolution mechanism with ovn-controller.

Solution Proposed

Developing a new OS-Ken application in OVS-agent which will act as an intermediary between instance and a DNS server, so it can response DNS query to instance without direct connection to the actual DNS server.

The basic data pipeline can be described as this:

                        +--------------+                +---------------------+                +---------------------+
+-------+ DNS Query     |              |    packet-in   |  +----------------+ | DNS Query      |                     |
|  VM   +--------------->  Flows       +---------------->  |   os-ken app   | +---------------->   Local resolvers   |
|       <---------------+ (capture     <----------------+  |                | <----------------+        or           |
+-------+ DNS Answer    | DNS query    |    packet-out  |  | DNS Forwarder  | | DNS Answer     |   Real DNS servers  |
                        |   here)      |                |  +----------------+ |                |                     |
                        |   br-int     |                |      OVS-agent      |                |                     |
                        +--------------+                +---------------------+                +---------------------+

After this we will have:

Higher level availability than DNS resolution with dnsmasq in DHCP Agent.
OpenStack with distributed DHCP extension [1] can now support DNS lookup.
Great combination with Distributed metadata datapath for Openvswitch Agent [2].

DNS Forwarder

We will extract DNS payload from the packet, then send it to upstream DNS server.

Server side changes

None

OpenvSwitch Agent side changes

For Neutron openvswitch agent, we will add a new agent extension which will process the basical flow installation for any DNS query.

There will be some basic flows which will check if the DNS query is matched with destinations, then submit it to the controller. The default destinations are 169.254.169.254:53 and [fd00::254]:53. We will make it configurable.

table=60, priority=102,udp,nw_dst=169.254.169.254,tp_dst=53 actions=CONTROLLER:0
table=60, priority=102,udp6,nw_dst=fd00::254,tp_dst=53 actions=CONTROLLER:0

For the new extension of openvswitch agent, it will register a local packet_in_handler which will do the following works:

Listen on the EventOFPPacketIn event
Verify each packet if it’s matched above rules
Extract DNS payload from the packet
Sent DNS payload to upstream DNS server
Assemble the DNS answer response and send_msg to in_port

The response DNS answer packet structure will be:

+------------------------------------------+
|           *Source Mac Address            |
|   169.254.169.254/fd00::254 MAC Address  |
+------------------------------------------+
|       *Destination Mac Address           |
|        Neutron Port Mac Address          |
+------------------------------------------+
|           *Source IP Address             |
|        169.254.169.254/fd00::254         |
+------------------------------------------+
|         *Destination IP address          |
|             Neutron Port IP              |
+------------------------------------------+

Source Mac Address will be 169.254.169.254/fd00::254 MAC Address.
Destination Mac Address will be the port’s MAC.
Source IP Address will be 169.254.169.254/fd00::254.
Destination IP address will be the port IP(v4/v6) address.

Note

Unlike DHCP Request which typically rely on broadcast packets, this solution works more like the metadata service which is need to have a route to 169.254.169.254/fd00::254 via somewhere to sent out DNS query packet. These packets will then be captured on the br-int bridge. The necessary route can be provided by the DHCP agent, distributed DHCP, or via cloud-init. Additionally, the specified next hop must be reachable, meaning the instance should be able to resolve its MAC address using ARP and in most cases, this next hop is the default gateway.

Potential configurations

A new extension alias name dns_forwarder will be added for neutron openvswitch agent:

[agent]
extensions = ...,dns_forwarder

Config section [dns_forwarder] will be added for neutron openvswitch agent and register some common options:

dns_forwarder_opts = [
  cfg.ListOpt('upstream_dns_server_ports',
              default=['1.1.1.1:53', '[2606:4700:4700::1111]:53'],
              help=_('Comma-separated list of the Upstream DNS server
                     'in ip:port format which will be used as resolvers.')),
  cfg.IntOpt('upstream_dns_query_timeout', default=5,
             help=_("Query timeout in seconds for each Upstream DNS servers")),
  cfg.ListOpt('client_dns_server_ports',
              default=['169.254.169.254:53', '[fd00::254]:53'],
              help=_('Comma-separated list of the Client DNS server
                     'in ip:port format which will be used inside client instances.')),
]

Data Model Impact

None

REST API Impact

None

Upgrading

This feature does not affect any current features, so we simply enable it to use.

Implementation

Assignee(s)

Primary assignees:

Dai, Dang Van <daikk115@gmail.com>

Work Items

Implement a new DNS Responder OS-Ken application.
Implement some new configuration options.
Implement relevant unit and functional tests.
Write documentation.

Testing

Unit/functional tests.

Documentation

User Documentation: How to setting up internal DNS resolution with OVS.

References

Add more information to network-ip-availabilites response

Tue, 22 Apr 2025 00:00:00

https://bugs.launchpad.net/neutron/+bug/2107316

This RFE intends to provide detailed information on the number of IPs in the subnet from the response of GET /v2.0/network-ip-availabilites - the number of available IPs in the subnet cidr and in the allocation pools, used IPs in the subnet cidr and in the allocation pools.

Problem Description

Now the response of network-ip-availabilites api contains: (1) total_ips (2) used_ips

(1) total_ips - subnet cidr, when there are no allocation pools in the subnet - the sum of IPs in each allocation pool, when there are allocation pools in the subnet

(2) used_ips - the number of used IPs in the subnet (does not consider allocation pools)

We cannot know how many available IPs are in the allocation pools from the response. There are use cases that we need to know the number of available IPs in the allocation pools, such as check ip availability before creating a vm in the network.

Using IPv4 subnet, total_ips does not show the exact number of available IPs. total_ips includes .0 and .255, which IPs can’t be used as ports’ ip.

Proposed Change

To solve the problem described above, the proposal is to add a new API extension network-ip-availability-details to extend network-ip-availability resource in Neutron with ip_availability_details attribute.

Server side changes

A new API extension network-ip-availability-details will be added. The extension adds a new attribute ip_availability_details.

The attribute consists of: (1) total_ips_in_subnet: the number of available IPs in the subnet cidr (except .0, .255, in IPv4 subnet) (2) total_ips_in_allocation_pool: the sum of IPs in each allocation pool, 0 in case of no allocation pools (3) used_ips_in_subnet: the sum of used IPs in the subnet (does not consider allocation pools) (4) used_ips_in_allocation_pool: the sum of used IPs in each allocation pool, 0 in case of no allocation pools

Response of GET /v2.0/network-ip-availabilities/{network_id}:

{
    "network_ip_availability": {
        "subnet_ip_availability": [
            {
                "subnet_id": "44e70d00-80a2-4fb1-ab59-6190595ceb61",
                "subnet_name": "private-subnet",
                "ip_version": 4,
                "cidr": "10.0.0.0/24",
                "total_ips": 256,
                "used_ips": 2,
                "ip_availability_details": {
                  "total_ips_in_subnet": 254,
                  "total_ips_in_allocation_pool": 0,
                  "used_ips_in_subnet": 2,
                  "used_ips_in_allocation_pool": 0
                }
            },
            {
                "subnet_id": "a90623df-00e1-4902-a675-40674385d74c",
                "subnet_name": "ipv6-private-subnet",
                "ip_version": 6,
                "cidr": "fdbf:ac66:9be8::/64",
                "total_ips": 18446744073709551600,
                "used_ips": 2,
                "ip_availability_details": {
                  "total_ips_in_subnet": 18446744073709551616,
                  "total_ips_in_allocation_pool": 18446744073709551600,
                  "used_ips_in_subnet": 2,
                  "used_ips_in_allocation_pool": 1
                }
            }
        ],
        "network_id": "6801d9c8-20e6-4b27-945d-62499f00002e",
        "project_id": "d56d3b8dd6894a508cf41b96b522328c",
        "tenant_id": "d56d3b8dd6894a508cf41b96b522328c",
        "network_name": "private",
        "total_ips": 18446744073709551856,
        "used_ips": 4,
        "ip_availability_details": {
          "total_ips_in_subnet": 18446744073709551870,
          "total_ips_in_allocation_pool": 18446744073709551600,
          "used_ips_in_subnet": 4,
          "used_ips_in_allocation_pool": 1
        }
    }
}

Response of GET /v2.0/network-ip-availabilities:

{
    "network_ip_availabilities": [
        {
            "network_id": "4cf895c9-c3d1-489e-b02e-59b5c8976809",
            "network_name": "public",
            "subnet_ip_availability": [
                {
                    "subnet_id": "44e70d00-80a2-4fb1-ab59-6190595ceb61",
                    "subnet_name": "private-subnet",
                    "ip_version": 4,
                    "cidr": "10.0.0.0/24",
                    "total_ips": 256,
                    "used_ips": 2,
                    "ip_availability_details": {
                      "total_ips_in_subnet": 254,
                      "total_ips_in_allocation_pool": 0,
                      "used_ips_in_subnet": 2,
                      "used_ips_in_allocation_pool": 0
                    }
                },
                {
                    "subnet_id": "a90623df-00e1-4902-a675-40674385d74c",
                    "subnet_name": "ipv6-private-subnet",
                    "ip_version": 6,
                    "cidr": "fdbf:ac66:9be8::/64",
                    "total_ips": 18446744073709551600,
                    "used_ips": 2,
                    "ip_availability_details": {
                      "total_ips_in_subnet": 18446744073709551616,
                      "total_ips_in_allocation_pool": 18446744073709551600,
                      "used_ips_in_subnet": 2,
                      "used_ips_in_allocation_pool": 1
                    }
                }
            ],
            "project_id": "1a02cc95f1734fcc9d3c753818f03002",
            "tenant_id": "1a02cc95f1734fcc9d3c753818f03002",
            "total_ips": 18446744073709551856,
            "used_ips": 4,
            "ip_availability_details": {
              "total_ips_in_subnet": 18446744073709551870,
              "total_ips_in_allocation_pool": 18446744073709551600,
              "used_ips_in_subnet": 4,
              "used_ips_in_allocation_pool": 1
            }
        },
        {
            "network_id": "6801d9c8-20e6-4b27-945d-62499f00002e",
            "network_name": "private",
            "subnet_ip_availability": [
                {
                    "cidr": "10.0.0.0/24",
                    "ip_version": 4,
                    "subnet_id": "44e70d00-80a2-4fb1-ab59-6190595ceb61",
                    "subnet_name": "private-subnet",
                    "total_ips": 256,
                    "used_ips": 2,
                    "ip_availability_details": {
                      "total_ips_in_subnet": 254,
                      "total_ips_in_allocation_pool": 0,
                      "used_ips_in_subnet": 2,
                      "used_ips_in_allocation_pool": 0
                    }
                },
                {
                    "ip_version": 6,
                    "cidr": "fdbf:ac66:9be8::/64",
                    "subnet_id": "a90623df-00e1-4902-a675-40674385d74c",
                    "subnet_name": "ipv6-private-subnet",
                    "total_ips": 18446744073709551600,
                    "used_ips": 2,
                    "ip_availability_details": {
                      "total_ips_in_subnet": 18446744073709551614,
                      "total_ips_in_allocation_pool": 18446744073709551600,
                      "used_ips_in_subnet": 2,
                      "used_ips_in_allocation_pool": 2
                    }
                }
            ],
            "project_id": "d56d3b8dd6894a508cf41b96b522328c",
            "total_ips": 18446744073709551856,
            "used_ips": 4,
            "ip_availability_details": {
              "total_ips_in_subnet": 18446744073709551868,
              "total_ips_in_allocation_pool": 18446744073709551600,
              "used_ips_in_subnet": 4,
              "used_ips_in_allocation_pool": 2
            }
        }
    ]
}

DB Impact

None

REST API Impact

New API extension: network-ip-availability-details introducing new attribute ip_availability_details.

Client Impact

Relevant changes in openstackclient and openstacksdk to add support for new attribute ip_availability_details.

Implementation

Add new API extension network-ip-availability-details. This extension extends network ip availability with field ip_availability_details consists of attributes - total_ips_in_subnet, total_ips_in_allocation_pool, used_ips_in_subnet, and used_ips_in_allocation_pool.

New class IpAvailabilityDetailsMixin will be added for the extension. The class calculates value of the four attributes, and extends network ip availability.

Assignee(s)

jimin3-shin <jimin3.shin@samsung.com>

Testing

New test classes and methods will be added to test network-ip-availability-details extension.
Tempest tests in neutron-tempest-plugin.

References

Provide Port Binding Information for Nova Live Migration

Wed, 20 Nov 2024 00:00:00

https://bugs.launchpad.net/neutron/+bug/1580880

Nova Live Migration consists of 3 stages:

pre_live_migration - executed before migration starts; the migration target is already known, but the instance still resides on the source.
live_migration_operation - migration itself which consists of 2 substages:
- before the VM is running on the migration target compute.
- after VM is running on the destination compute, but migration is still in progress (post copy migration).
post_live_migration - executed after migration; source VM does not exist anymore. Used for finalizing the migration.

Today, port binding occurs on the target compute host during migration, considered now in the post_live_migration stage. This, unfortunately, is too late in the process as there are cases where Nova requires this information in the pre_live_migration stage. We simply cannot move the port binding to the pre_live_migration stage as the original port binding would be deleted, causing issues due to the instance being active still on the original host. Further details are contained in the ‘Alternatives’ section.

Problem Description

For live migration improvements, it is required that Neutron allows port binding on the target compute host during the pre_live_migration stage, without removing the original port binding on the source compute.

The proposal is to have a port binding on the source AND target compute hosts where only one port binding is active. After instance migration is complete, the target port binding is activated and the source port binding is deactivated, but not deleted. The original port binding on the source compute host is kept for a possible move of the instance back to the source, and will only be removed during the post_live_migration stage.

The issues can be divided in 2 categories:

#1 Keep the source instance running if the port binding on the target fails:

Instance error state when port binding fails:

Today, the port binding is triggered in the post_live_migration stage, after the migration has completed. If the port binding fails, the instance is then stuck in an error state.

The solution is to move the port binding to the pre_live_migration stage, where there is an opportunity to fail the port binding earlier. This would prevent the instance from being stuck in an errored state after the migration is complete and fail the migration at a pre_live_migration stage.

The issue with moving the port binding to the pre_live_migration stage is that some drivers will shutdown the port binding on the source compute host, even though the instance is still active. To achieve a cleaner migration solution, Neutron needs to be modified to allow a port binding on both the source and target compute hosts in an active/inactive state.

#2 Handling the differences in port binding details between hosts:

Live migration between hosts running different l2 agents:

Another case to consider is the chance that the migration occurs between two hosts running different l2 agents. The requirement here is on Nova to update the instance definition before the migration is executed. In the case of libvirt, Nova would update the domain.xml with the target interface definition. For more information, refer to [3] and [4].

A special case to consider is where a migration occurs between agents with differing firewall drivers, i.e. from a host running the ovs hybrid-fw driver to a host running the new ovs conntrackd firewall driver.

Such a migration must only be allowed with source and target binding using the same VNIC type.
Live migration with MacVTap agent when different physnet mappings are used:

MacVTap today has restrictions with live migration in certain situations [1] and requires an update to the instance definition (libvirt domain.xml) before starting a migration.

Updating the instance definition occurs during the live_migration_operation stage, before the instance is running on the target compute host. Two port bindings, one active at the source and one inactive at the target host are required for this operation to succeed.

Proposed Change

For migration, Nova will require the port binding information from the target compute during the pre_live_migration stage. This can be achieved by allowing a compute port to be bound to the migration source and the migration target host.

This can be achieved by the following steps:

#1 Expand upon existing API entities under port, allowing CRUD bindings.

#2 Update ML2 to support the changes.

#3 Update the DB to support the changes.

Usage by Nova

Nova will utilize the expanded API, and the potential flow is as follows:

pre_live_migration: Create the inactive binding for target host.
live_migration_operation: Use information gathered from the inactive binding to modify the instance definition.
live_migration_operation: Once the instance is active, set the inactive binding to active, and the previous active binding to inactive.
post_live_migration: Remove the inactive binding on the source compute host.

If rollback is performed after the instance is active on target: From a Neutron standpoint, if the binding is active on the target host, Nova will need to set the source binding back to active:
```
PUT /v2.0/ports/{port_id}/bindings/{host_id}/activate
```

For more details on the Nova implementation , see the related Nova Blueprint [3] and its Spec [4]. Neutron will not dictate the implemented capabilities of Nova live migration and will support either path, to rollback or to not rollback.

Binding API Extension for Ports

List Bindings

GET /v2.0/ports/{port_id}/bindings

{
    "bindings": [
        {
            "host_id": "source-host_id",
            "vif_type": "ovs",
            "vif_details": {
                "port_filter": true,
                "ovs_hybrid_plug": true
                },
            "profile": {},
            "vnic_type": "normal",
            "status": "active"
        },
        {
            "host_id": "target-host_id",
            "vif_type": "bridge",
            "vif_details": {
                "port_filter": true,
                },
            "profile": {},
            "vnic_type": "normal",
            "status": "inactive"
        },
    ]
}

Response Parameters
Parameter	Style	Type	Description
bindings	plain	xsd:list	A list of binding objects

More parameters see Show Binding

Important key features of list bindings:

Compute bindings will currently be listed and a request for unsupported bindings will return ‘NotImplemented’ until the capability is introduced.
All bindings will be listed and pagination will be used when many bindings are returned.

Show Binding

GET /v2.0/ports/{port_id}/bindings/{host_id}

Response Parameters
Parameter	Style	Type	Description
binding	plain	xsd:dict	A binding object
host_id	plain	xsd:string	Hostname where the port is allocated.
vif_type	plain	xsd:string	The VIF type for this port binding determined during portbinding
vif_details	plain	xsd:dict	A dictionary containing additional details for this specific binding. The details are set by a mechanism driver.
vnic_type	plain	xsd:string	The VNIC type for this port binding.
profile	plain	xsd:dict	A dictionary holding the vif profile.
status	plain	xsd:String	Status of the binding Status Usage

{
    "binding": {
         "host_id": "target-host_id",
         "vif_type": "target-vif-type",
         "vif_details": {
             "port_filter": true,
             },
         "vnic_type": 'NORMAL',
         "profile": {},
         "status": "active"
     }
}

Important key features of show binding:

DVR ports exposed in this resource will show the real vif_type of ‘distributed’ ports as they are stored in DistributedPortBindings, i.e. ovs.

Create Binding

POST /v2.0/ports/{port_id}/bindings

Request Parameters
Parameter	Style	Type	Description
binding	plain	xsd:dict	A binding object
host_id (mandatory)	plain	xsd:string	Hostname where the port is allocated.
vnic_type (optional, default = ‘normal’)	plain	xsd:string	The VNIC type for this port binding.
profile (optional)	plain	xsd:dict	A dictionary holding the vif profile.

{
    "binding": {
        "host_id": "target-host_id"
    }
}

Response parameters

see List Bindings

{
    "binding": {
        "host_id": "target-host_id",
        "vif_type": "ovs",
        "vif_details": {
            "port_filter": true,
            "ovs_hybrid_plug": true
            },
         "vnic_type": 'NORMAL',
         "profile": {},
         "status": "active"
    }
}

If the binding fails, a new return code of 4xx or 5xx should be returned. This differs from today where a failed binding returns a 2xx response code and the vif_type is set to “binding_failed”.

Important key features of update/create binding:

By default, the status will be active when creating a port binding. If a binding is created, doesn’t exist already, and an existing binding is already active, the binding will default to inactive, requiring the operator to activate the new binding.
If a binding being added already exists, a 4xx will be returned.
A compute port can only have 1 active binding at a time. This is not an enforcement by Neutron, but a result of the operation surrounding PortBinding. This feature expands the capability of having multiple bindings, but will only allow for 1 active binding for compute ports.
At this time, creation of a binding will be limited to compute ports.
The existing API is not touched, which will return host_id:{host_id} as the current active binding.
Activating an inactive compute binding will deactivate the current active binding.

Update Binding

PUT /v2.0/ports/{port_id}/bindings/{host_id}

Request Parameters
Parameter	Style	Type	Description
binding	plain	xsd:dict	A binding object

All create parameters are valid for update as well. See Create Binding.

{
    "binding": {
        "vnic_type": 'NORMAL',
        "profile": {}
    }
}

Response parameters

see Show Binding

{
    "binding": {
        "host_id": "target-host_id",
        "vif_type": "ovs",
        "vif_details": {
            "port_filter": true,
            "ovs_hybrid_plug": true
            },
        "vnic_type": 'NORMAL',
        "profile": {"foo": "bar"},
        "status": "active"
    }
}

On failed binding, a 4xx or 5xx return code should be returned.

Activating an Inactive Binding

PUT /v2.0/ports/{port_id}/bindings/{host_id}/activate

Response parameters

see Show Binding

{
    "binding": {
        "host_id": "target-host_id",
        "vif_type": "ovs",
        "vif_details": {
            "port_filter": true,
            "ovs_hybrid_plug": true
            },
        "vnic_type": 'NORMAL',
        "profile": {"foo":"bar"},
        "status": "active"
    }
}

Important key features of activate binding:

Activating a compute binding that is inactive will deactivate the existing active binding, as a compute port can only have 1 binding active at a time.
Operation will be limited to compute ports.
Attempting to activate an existing active binding will return a 4xx.
Returns 5xx if activating the binding fails.

Delete Binding

DELETE /v2.0/ports/{port_id}/bindings/{host_id}

This operation does not accept a request body and does not return a response body.

Important key features of delete binding:

Active/Inactive bindings can be removed.
Deleting an active compute port binding, where an inactive binding exists does not activate the binding. The operator will be required to explicitly activate the binding.

Overlap Between Existing vs New APIs

All the functionality of the existing API will be covered by the new API as well. This section describes the overlap.

Overlap existing vs. new API
Existing API	New API
Show port with active binding	n/a
Create port: directly with an active binding	n/a
Update port: add host_id (which adds the active binding)	Add Binding
Create port: without any binding	n/a
Update port: change host_id (re-trigger port binding for another host)	Update binding
Update port: set host_id to ‘’(remove the active binding)	Delete active binding
Delete port: Remove port with all its bindings	n/a

Effects on Existing APIs

Slight adjustment to existing APIs:

Show Port will still just show the binding like today. For compute ports, it would only show the active binding.
Create Port will create an unbound binding as before, but with the status of active.
Update Port with host_id will still re-trigger port binding for a host. The difference will be update_port() will only action on the active binding.
vif_type is set to “binding_failed” and http code 2xx is still used on failed port binding when binding is triggered via the existing port binding extension.

API Visibility

A normal user should not be able to trigger any create/update/delete/activate actions. This should only be possible via some special service user or the admin role.

Sub Resource Extension

Neutron ports will be extended with a sub resource bindings, having a member name of port to preserve portbindings and ports extensions. The new sub resource extension will be portbindings_extended.py and have a parent resource of ports.

The following methods will be added to the newly created service plugin bindings_plugin.py:

get_port_bindings()
get_port_binding()
create_port_binding()
update_port_binding()
delete_port_binding()
update_port_binding_activate()

ML2 Changes

Existing methods create_port() and update_port() will need to be updated to support actioning only on active status bindings. In addition, a new status of inactive will be introduced to neutron-lib for use in PortBinding.

Status Usage

The status column in PortBinding will store an additional state ‘inactive’ where the current states are ‘active’ and ‘down’. Neutron-lib will only require the addition of PORT_BINDING_STATUS_ACTIVE and PORT_BINDING_STATUS_INACTIVE.

from neutron_lib import constants as const

const.PORT_BINDING_STATUS_ACTIVE
const.PORT_BINDING_STATUS_INACTIVE

Create/Update/Delete Port

New methods will be introduced in support of the new sub resource under ports, but the current create_port(), update_port() and delete_port() will be modified to only act on active bindings in the PortBinding table.

Today, create_port() adds an empty unbound binding in PortBinding and the following changes will be made in support of this spec:

Create an unbound binding with status active in the PortBinding table.

In addition, update_port() will be adjusted for active status with the following changes:

Update will only change binding information on the active binding in the PortBinding table.

Finally, delete_port() will be adjusted for active status with the following changes:

Delete will only act on the active binding in the PortBinding table.

Data Model Changes

The PortBinding table will expand the primary key to column host, allowing selection of the binding based on port_id and host.

In addition, a status column will be introduced in the expansion where states active, down, and inactive will be values.

Online upgrades, Blueprint [9], requires the addition of host to primary_keys and a new field status for Port Binding OVO. Version of the object should be bumped if push-notifications, under Blueprint [10], will be merged first, and PortBinding object will be present on the RPC wire. Defining a default value for the status field would not require online data migration.

Changes to Mechanism Drivers

A new mechanism driver method is required to determine if the new way of binding things is supported. This must be validated for all binding levels and allow fallback to previous methods. By default, new method will return unsupported.

Besides the in tree mechanism drivers for l2 agents (ovs, lb, sr-iov, macvtap) the following drivers need to be considered:

l2pop (however there are ideas to eliminate l2pop)
ironic
third party mechanism drivers

Activate RPC Port Update/Delete

The existing port_update and port_delete RPC message will be adjusted to send, when the agent retrieves device information with get_devices_details_list_and_failed_devices, specific binding information for the host regardless of binding state. This will allow the addition of additional plumbing to occur as follows:

Activate will result in a port_update, which will pass the relative binding information for the host and dictate the transition from inactive to active in the get_devices_details_list_and_failed_devices response. This will allow for a GARP to be sent out, updating the topology to a change in status.
Activate will result in a port_delete rpc call to the source host, removing the source VIF. This will need to be accomplished due to Nova not being able to issue a delete port, as the port still exists on a different host. The binding will remain in an inactive state where binding information has been populated, but the port, from an agent perspective, will not exist. In addition, the transition from active to inactive will be indicated in the rpc call, influencing the update_device_list to not update the port state.

In the case where push-notifications are implemented for ports under Blueprint [10] the get_devices_details_list_and_failed_devices would not be adjusted for transition state. Instead, the binding transition state would be sent to the agent as part of the port object. The remaining actions are the same.

Other changes

Neutron/Openstack Python Client support.
Neutron-Lib support of new constant PORT_STATUS_INACTIVE see Activating an Inactive Binding.

Command Line Client Impact

Support for port bindings will be needed in OSC. The following will be added:

$ openstack port binding list {ARGS} <port>

$ openstack port binding show {ARGS} <port> <host>

$ openstack port binding create {ARGS} <port>

$ openstack port binding update {ARGS} <port> <host>

$ openstack port binding delete {ARGS} <port> <host>

$ openstack port binding activate {ARGS} <port> <host>

Security Impact

None.

Notifications Impact

None.

Other End User Impact

None.

Performance Impact

There should be no performance impact.

IPv6 Impact

None.

Other Deployer Impact

None.

Developer Impact

Impact to Nova live migration, and is directly in support of their efforts.

Community Impact

Yes. This change has been discussed on the ML, in Neutron meetings (especially ML2), at mid-cycles, and at the design summit.

Alternatives

An alternative is to use the current resources under ports to facilitate this change to live migration. The problem is, current dict structures would need to be expanded to accommodate the ‘bindings’ key. This may cause some confusion as the user already receives ‘bindings:profile’ and various other values.

Implementation

Assignee(s)

Primary assignee: * Jakub Libosvar

Please add your name here and attend the ML2 Subteam Meeting if you’d like to contribute.

Dependencies

None.

Testing

Tempest Tests

Addition of a scenario test, test_bindings.py, to walk through the creation of a source migration instance, creation of the inactive binding on a secondary host, creation of a secondary target migration instance, activating the inactive binding, deactivating the source migration active binding, and then validating connectivity is still working.

Functional Tests

Additional functional tests will be added to ml2/test_plugin.py to expand on the current port binding tests. This will accommodate for a status check in the case of adding an inactive binding.

API Tests

Bindings resource (CRUD)
Bindings resource (CRUD) and validation of active binding under current ports extension.

Documentation Impact

Yes.

User Documentation

None.

Developer Documentation

Create detailed explanation of active/inactive binding operation in devrefml2_port_bindings.rst. This should detail changes to ml2, and the extended ports resource.

References

Layer 3 service plugin for the Cisco APIC

Wed, 20 Nov 2024 00:00:00

https://blueprints.launchpad.net/neutron/+spec/cisco-apic-l3

This blueprint is to implement Layer 3 features in the network fabric via a L3 router service plugin using the Cisco APIC.

Flows

Problem description

The APIC (Application Policy Infrastructure Controller) together with Cisco Nexus 9000 switches provides programmable, policy-driven network control.

The Service plugin proposed will interact with the APIC to dynamically configure Layer 3 inter and intra tenant communications in the network fabric.

Proposed change

This proposal is to introduce a new Layer 3 service plugin that communicates with the APIC.

The driver implements the following neutron events: * Add a new router interface * Delete a router interface

The plugin will implement Layer 3 communications using a construct called a contract that provides communications in the fabric between various end point groups (Neutron networks)

Events that trigger the creation of end point groups and subnets (i.e. create_network and create_subnet) will be handled by the APIC ML2 mechanism driver and id’s will be stored in a database accessed by a common client and manager class.

Due to hardware limitations as of this writing this service plugin will only handle add/remove router_interface (internal gateway) events. The service plugin may be expanded in the future to also handle all neutron L3 events.

Alternatives

The alternative approach is to use the open source agent based layer 3 router plugin. The agent based approach does not implement any policies that are centric to the APIC’s ACI (Application Centric Infrastructure) based design.

Data model impact

n.a.

REST API impact

n.a

Security impact

n.a

Notifications impact

n.a.

Other end user impact

n.a.

Performance Impact

The service plugin is triggered instead of polled, there are no changes to any existing code patterns. The potential bottleneck for this plugin would be the link between neutron and the APIC.

Other deployer impact

There are no config options specific to the Layer 3 plugin, it relies on the configuration options of the ML2 apic mechanism driver.

Developer impact

n.a.

Implementation

Assignee(s)

Arvind Somya <asomya>

Work Items

Single work item for the L3 APIC service plugin.

Dependencies

Depends on the APIC ML2 blueprint: https://blueprints.launchpad.net/neutron/+spec/ml2-cisco-apic-mechanism-driver

Testing

Complete unit test coverage of the code is included.

For tempest test coverage, third party testing is provided. The Cisco CI reports on all changes affecting this driver. The testing is run in a setup with an OpenStack deployment (devstack) connected to a live APIC and a Cisco Nexus 9000 physical switch.

Documentation Impact

Deployment documentation on how to configure and deploy this service plugin will be documented in the Openstack wiki.

References

http://www.cisco.com/go/apic

Database Migration Refactoring

Wed, 20 Nov 2024 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/neutron/+spec/db-migration-refactor

The neutron database schema will be made consistent. It will be independent of the configured core plugin and service plugins.

Problem description

The database schema is different based on which core and service plugins are configured. If a new or different plugin is configured after a deployment is started, the earlier migrations that correspond with the new configuration will be missing.

This means that the migration process is not idempotent, resulting in schema versions which depend on the current configuration. In other words, when one environment’s schema is at a specific version it may not be the same as another environment at the same version. This even happens in the same environment during a downgrade if the configuration changes.

Proposed change

Make all migrations unconditional. Make a single point in the migration timeline where all tables are created and the database schema is the same no matter which plugins are used.

This solution will introduce a “healing” migration that will call the DDLs needed to complete the database schema so that it contains all tables for all database models. This migration will occur in the timeline between the icehouse_release version and the juno_release version.

The refactoring will comprise the following:

Investigation of models and current migrations to establish what is needed to achieve the complete database schema.
A healing migration that ensures that the database schema is complete and consistent. The version will be named db_healing.
All migrations after db_healing will be unconditional. This requires updating the behavior of the –autogenerate for neutron-db-manage, and documentation for developers.

The healing migration will work like any other alembic migration, i.e. it will allow upgrade and downgrade. However, it will be different in the following aspects:

In the upgrade direction the healing migration will introspect the schema and only add tables that are missing. Therefore it cannot be run in offline mode. (See also Online Requirement.)

Upgrade Notes:
- When the healing migration starts, all table schema changes for existing tables will already be in place (either directly from the model, or from previous migrations). For each missing table, the healing migration will add the table by creating its schema from the model.
- The healing migration may need to alter some existing tables to match its model. We won’t know for sure until the proposed implementation is tested in more detail. These alterations, if needed, will be dealt with on a case-by-case basis. For more details, see the Implementation section.
In the downgrade direction the healing migration will make no schema changes. This means it will not remove any tables.

Downgrade Notes:
- If a deployment is upgraded and then downgraded through the healing migration, the schema will contain tables that were not present before. In other words, if a deployment downgrades back to Icehouse or Havana, Neutron will appear and behave as before, yet if you look at the DB it is not the old Icehouse or Havana DB but a healed one.
- If during the upgrade a table was altered to match its model, then the downgrade will reverse the alteration if needed.

Migration Timeline

Online Requirement

More reasons why we cannot support offline mode for the healing migration:

We cannot use “create table if not exists” because it is only supported by specific sql dialects.
Dependencies between tables (e.g. foreign keys) mean we would need to check that all tables with dependencies on the current one are already created.
Alembic has no support for conditional DDL.

Alternatives

Instead of a healing migration we could break the migration timeline after Icehouse and create a new one beginning at Juno that includes all tables. A manual script could be provided to convert the schema from the old timeline to the new. This alternative approach has the following disadvantages:

It would not allow a downgrade via migration.
Switching from the old timeline to the new timeline is more a complex process for the deployer and DB Administrator than a simple migration in one timeline.
We would need to support two migration timelines in neutron until the Icehouse release is deprecated.

Data model impact

Some database models may need to be updated in order to have non-conflicting models based on core and service plugins.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

End (non-admin) users should see no impact.

Performance Impact

None

Other deployer impact

The healing migration is an online operation that must be run by the DB Administrator. Thus the deployer must co-ordinate with the DBA when upgrading or downgrading through Juno.

The healing migration will be run when migrating from Icehouse or earlier to Juno. It behaves like a normal migration, but it does not support upgrade in offline mode.

Downgrade of the healing migration does nothing. Thus all tables are present in the schema if downgrading from db_healing to a previous version.

Note: Greenfield deployment of the Juno release or later will start at the new migration timeline and therefore no healing will be involved.

Developer impact

None

Implementation

Most of the work lies in developing a robust healing migration. Alembic will be used where possible to maximize automation, but some healing may need to be manually coded.

Examples of conflicts which may need manual coding to resolve:

If two different migrations for two different plugins add an attribute with the same name but different type.
If an ENUM was modified in an earlier migration but its specification was not updated for PostgreSQL.

Assignee(s)

akamyshnikova
libosvar
rpodolyaka

Work Items

Dependencies

Testing

Unit (and functional?) testing of migrations shall be added. We plan to utilize the unit test framework from the graduated oslo.db package.

Documentation Impact

Create Release Note for this change.
Update Operators Guide for upgrading.
Update Developer Documentation for creating migration scripts.

References

Provider Networking - upstream SLAAC Support

Wed, 20 Nov 2024 00:00:00

It should be possible to create a Provider Network in Neutron, that uses an upstream switch or router that advertises RA’s - so that instances can use SLAAC to configure their IPv6 networking, while still utilizing the Neutron APIs for security group rules.

https://blueprints.launchpad.net/neutron/+spec/ipv6-provider-nets-slaac

Problem description

A deployer of OpenStack that wishes to use IPv6 and already has configured IPv6 on northbound networking devices, may wish to advertise IPv6 routes by having non-OpenStack hardware transmit ICMPv6 Router Advertisement packets.

Proposed change

OpenStack Neutron should support this configuration, and display the correct IP address for instances, since Neutron has enough information to generate the EUI-64 address.

Alternatives

None

Data model impact

None

REST API impact

The Neutron REST API for IPv6 Subnets allows the following combinations, as well as disallowing certain combinations. In short, when both ipv6_ra_mode and ipv6_address_mode are set, they must be equal. When only one attribute is set, the setting is automatically valid.

ipv6_ra_mode	ipv6_address_mode	valid	RA Settings
ATTR_NOT_SPECIFIED	ATTR_NOT_SPECIFIED	yes	ANY
ATTR_NOT_SPECIFIED	slaac	yes	“A=1,M=0,O=0”
ATTR_NOT_SPECIFIED	dhcpv6-stateful	yes	“A=0,M=1,O=1”
ATTR_NOT_SPECIFIED	dhcpv6-stateless	yes	“A=1,M=0,O=1”
slaac	ATTR_NOT_SPECIFIED	yes	“A=1,M=0,O=0”
dhcpv6-stateful	ATTR_NOT_SPECIFIED	yes	“A=0,M=1,O=1”
dhcpv6-stateless	ATTR_NOT_SPECIFIED	yes	“A=1,M=0,O=1”
slaac	slaac	yes	“A=1,M=0,O=0”
dhcpv6-stateful	dhcpv6-stateful	yes	“A=0,M=1,O=1”
dhcpv6-stateless	dhcpv6-stateless	yes	“A=1,M=0,O=1”
slaac	dhcpv6-stateful	no	UNDEF
slaac	dhcpv6-stateless	no	UNDEF
dhcpv6-stateful	slaac	no	UNDEF
dhcpv6-stateful	dhcpv6-stateless	no	UNDEF
dhcpv6-stateless	slaac	no	UNDEF
dhcpv6-stateless	dhcpv6-stateful	no	UNDEF

This blueprint will implement the functionality required to satisfy IPv6 Subnets that are set with ipv6_address_mode set to ‘slaac’, and ipv6_ra_mode not set.

Security impact

None

Notifications impact

None

Other end user impact

This change will require support in python-neutronclient for the two IPv6 subnet attributes - which is currently in review

Performance Impact

None

Other deployer impact

This change will change the behavior of Neutron in specific configurations, when the IPv6 attributes for Subnets are set. Previously, the attributes were no-ops.

Developer impact

None

Implementation

Subnets will be created with the ipv6_address_mode set to slaac and ipv6_ra_mode not set.
Neutron will calculate the IP address of a port, using the MAC address and the CIDR.

Assignee(s)

Primary assignee:: scollins

Work Items

Dependencies

The IPv6 Subnet Attributes must be returned in API calls.

Testing

Add unit tests to support Subnets created with only the ipv6_address_mode set.
Verify that ports with allocations from a subnet with ipv6_address_mode set are not touched by the DHCP agent.

Documentation Impact

Documentation about this network configuration will need to be written.

References

Devstack for IPv6 in the Comcast lab

‘security_group_rules_for_devices’ RPC call refactoring

Wed, 20 Nov 2024 00:00:00

https://blueprints.launchpad.net/neutron/+spec/security-group-rules-for-devices-rpc-call-refactor

Security group rules synchronization from neutron-server to L2 agents scales poorly for high density clouds, leading to a blocked neutron-server in some situations.

Problem description

The security_group_rules_for_devices RPC call from L2 agents to neutron-server doesn’t scale well because all the security group rule entries are expanded with each specific IP address (see [1]) in a security group, when that group is referenced in rules like:

allow all from ‘default’ group for IPv4 and IPv6

This leads to:

huge AMQP messages (>20-600 MB)
very long processing time at neutron-server side when we have lots of instances under the same tenant/security group (>60 seconds)
neutron-server lockups, when RPC call times out at agent side, and the same security_group_rules_for_devices call is issued back to neutron.

For a more detailed insight:

security_group_rules_for_devices is an RPC call from the L2 agents to neutron-server, see [1].

This call’s arguments are a list of device_ids, device_ids are connected to ports. Neutron builds a list [1] of security group rules and returns the list of security group rules per device_id

The message size between neutron-server and L2 agent will grow according to the following formula:

MessageSize ~= base + L * Instances_in_host * (Instances_in_security_group-1)

Where L = len(str(security_group_rule)) ~=440 bytes

This problem is more likely to happen in big clouds, or denser ones, but it’s an issue, as nova can scale to 10x instances without incident: see [2].

Proposed change

Refactor the security_group_rules_for_devices into a new call ‘security_group_rules_for_devices_compact’, which instead of returning [1] , returns a more compact result that can be expanded by the l2 agent.

The new ‘security_groups_rules_for_devices_compact’ RPC call returns:

{'security_groups': {'sg-id1': {'rules': [ {},{},{},{}]}
                    },
 'security_group_member_ips':  { 'sg-id1' : {'ipv4': ['192.168.11.2/32'],
                                             'ipv6': [] },
                                 'sg-id2-referenced-from-id1' : {...},
                               },

 'devices': {'dev-id1': { ... ,
                         'fixed_ips': ['192.168.11.4'],
                         'security_groups': ['sg-id1', 'sg-id2', ...] }}
}

Security group rules would be passed in a non-expanded way, which implies that any reference to src or dst security groups in rules would be stored as security group ids.

A response like this (old result):

{'dev-id1':
 {...,
  'security_group_rules': [{'direction': u'egress',
                            'ethertype': u'IPv6',
                            'security_group_id':
                             u'1809f907-4b0c-4445-a366-ff28eaab9c2e'},
                          {'direction': u'egress',
                           'ethertype': u'IPv4',
                           'security_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e'},
                          {'direction': u'ingress',
                           'ethertype': u'IPv4',
                           'protocol': u'icmp',
                           'security_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e'},
                          {'direction': u'ingress',
                           'ethertype': u'IPv4',
                           'remote_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'security_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'source_ip_prefix': '192.168.11.2/32'},
                          {'direction': u'ingress',
                           'ethertype': u'IPv4',
                           'remote_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'security_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'source_ip_prefix': '192.168.11.3/32'},
                          {'direction': u'ingress',
                           'ethertype': u'IPv4',
                           'remote_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'security_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'source_ip_prefix': '192.168.11.4/32'},
                          {'direction': u'ingress',
                           'ethertype': u'IPv4',
                           'remote_group_id':
                                   u'23138476-4fde-454e-33ad-abc123456782',
                           'security_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'source_ip_prefix': '192.168.33.4/32'}
                            ]
 },
'dev-id2': {
 ...,
 'security_group_rules': [{'direction': u'egress',
                           'ethertype': u'IPv6',
                           'security_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e'},
                          {'direction': u'egress',
                           'ethertype': u'IPv4',
                           'security_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e'},
                          {'direction': u'ingress',
                           'ethertype': u'IPv4',
                           'protocol': u'icmp',
                           'security_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e'},
                          {'direction': u'ingress',
                           'ethertype': u'IPv4',
                           'remote_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'security_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'source_ip_prefix': '192.168.11.2/32'},
                          {'direction': u'ingress',
                           'ethertype': u'IPv4',
                           'remote_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'security_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'source_ip_prefix': '192.168.11.3/32'},
                          {'direction': u'ingress',
                           'ethertype': u'IPv4',
                           'remote_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'security_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'source_ip_prefix': '192.168.11.4/32'},
                          {'direction': u'ingress',
                           'ethertype': u'IPv4',
                           'remote_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'security_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'source_ip_prefix': '192.168.11.5/32'},
                          {'direction': u'ingress',
                           'ethertype': u'IPv4',
                           'remote_group_id':
                                   u'23138476-4fde-454e-33ad-abc123456782',
                           'security_group_id':
                                   u'1809f907-4b0c-4445-a366-ff28eaab9c2e',
                           'source_ip_prefix': '192.168.33.4/32'}
                            ]
 }
}

Would be like this in the new version:

{'security_groups': {u'1809f907-4b0c-4445-a366-ff28eaab9c2e':
                      {'rules': [
                         {'direction': u'egress', 'ethertype': u'IPv6'},
                         {'direction': u'egress', 'ethertype': u'IPv4'},
                         {'direction': u'ingress',
                          'ethertype': u'IPv4',
                          'protocol': u'icmp',},
                         {'direction': u'ingress',
                          'ethertype': u'IPv4',
                          'remote_group_id':
                               u'1809f907-4b0c-4445-a366-ff28eaab9c2e'},
                         {'direction': u'ingress',
                          'ethertype': u'IPv4',
                          'remote_group_id':
                               u'23138476-4fde-454e-33ad-abc123456782'}
                        ]
                       }
                    },
 'security_group_member_ips':  { u'1809f907-4b0c-4445-a366-ff28eaab9c2e' :
                                 {u'ipv4': ['192.168.11.2/32',
                                           '192.168.11.3/32',
                                           '192.169.11.4/32',
                                           '192.168.11.5/32'],
                                  u'ipv6': []
                                 },
                                 u'23138476-4fde-454e-33ad-abc123456782' :
                                 {u'ipv4': ['192.168.33.2/32'],
                                  u'ipv6': []
                                 }
                               },

 'devices': {'dev-id1': { ... ,
                         'fixed_ips': ['192.168.11.4'],
                         'security_groups':
                          ['1809f907-4b0c-4445-a366-ff28eaab9c2e'] },
             'dev-id2': { ... ,
                         'fixed_ips': ['192.168.11.4'],
                         'security_groups':
                          ['1809f907-4b0c-4445-a366-ff28eaab9c2e'] },

             }

All security groups referenced from devices will be included in the response.

All security group members ip addresses from all remote_group_id referenced groups will be included in the response.

The old call could be marked as deprecated during this J cycle, and removed during K cycle.

Making the refactor into a new call would have the following advantages:

Compatibility during neutron-server upgrade with older agents
Ability to split patches (server/agents) in more steps, as we will have the ability to address the new call, while keeping the agents calling the old one, and then refactor the agents in further steps.

The resulting message size would be:

MessageSize ~= base +: D * Instances_in_host + L * Referenced_security_groups + I * Instances_in_referenced_security_groups
Where L = len(str(compact_security_group_rule)) ~= 220 bytes: D = len(str(device_description_including_ips_and_sg_ids)) I = len(str(ip_address + ‘,’)) ~= 17 bytes

In the new message format no data is replicated, thus now two variables become multiplication factors.

Next steps:

There is a proposal from Édouard Thuleau to use an RPC topic per security group [3] which would be addressed in a second iteration after this one.

Alternatives

Instead of including all the security groups in one rpc call, this could be split to a second call, ‘security_groups_and_referenced_members’, which would receive a list of security groups ids, and would return a list of security groups and a list of security group ip addresses. A full sync would require 2 calls to neutron server.
We could have a ‘security_groups’ and a ‘security_groups_members’, which would provide the security groups, without member IP addresses and, a list of IPv4 and IPv6 addresses members for each security group. A full sync would require 3 calls to neutron server. But this approach would allow separate communication of new members in security groups, or new rules in security groups, further reducing the information transmited in those cases. As for the first alternative, reducing the traffic volume by increasing the number of calls seems like a bad tradeoff due to the overhead/latency generated by each call.
We could just compact CIDR ranges in rules generation, that wouldn’t require modifications to the agents, but that would increase the rpc request processing time.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

The performance impact should be very positive in the next situations:

Security group changes: neutron-server load, and AMQP message sizes. At this moment oslo messaging serializes structures to JSON because of AMQP version limitations (string sizes for dictionaries is one of those). Reducing the message size will reduce the dictionary building time and also the serialization time.
Creation time for new ports

Even higher performance impact in packet processing would be achieved by the optimization at iptables level which is proposed in the ipset spec [4].

If tranmission times become our bottleneck instead of processing times we may consider compression if that’s available at the AMQP level.

Other deployer impact

None

Developer impact

I’m unsure if there are proprietary l2-agents talking to the RPC. We would allow some time for those to be upgraded by introducing it as a new RPC call instead of modifying the existing one.
All the agents that use this RPC call would need to be updated to the new call before removing the old one.

Implementation

Assignee(s)

Primary assignee:: https://launchpad.net/~mangelajo
Other contributors:: http://launchpad.net/~shihanzhang

Work Items

Refactor the rpc call into a new one in neutron-server and and the matching rpc call at the agent mixin.
Add functional testing to validate the approach.
Upgrade the agents to use the new call, one by one.
Analyze db access and file a new spec if improvements can be made in this area.

Dependencies

None

Testing

Functional testing will validate the approach and make sure the result that was possible with the old method can be consistently replicated (only faster) with the new method. Testing both rpc calls with functional testing will avoid regressions in both of the code paths, while testing this in Tempest only allows testing the default rpc call.

Documentation Impact

None

References

Agent child processes status

Wed, 20 Nov 2024 00:00:00

https://blueprints.launchpad.net/neutron/+spec/agent-child-processes-status

Neutron agents spawn external detached processes which run unmonitored, if anything happens to those processes neutron won’t take any action, failing to provide those services reliably.

We propose monitoring those processes, and taking a configurable action, making neutron more resilient to external failures.

Problem Description

When a ns-metadata-proxy dies inside an l3-agent [4], subnets served by this ns-metadata-proxy will have no metadata until there are any changes to the router, which will recheck the metadata agent liveness.

Same thing happens with the dhcp-agent [3] and also in lbaas and vpnaas agents.

This is a long known bug, which generally would be triggered by bugs in dnsmasq, or the ns-metadata-proxy, and specially critical on big clouds and HA environments.

Proposed Change

I propose to monitor the spawned processes using the neutron.agent.linux.external_process.ProcessMonitor class, which relies on the ProcessManager to check liveness periodically.

If a process that should be active is not, it will be logged, and we could take any of the following admin configured actions, in the configuration specified order.

Respawn the process: The failed external process will be respawned.
Exit the agent: for use when an HA service manager is taking care of the agent and will respawn it, optionally in a different host. During exit action, all other external processes will be left running as for any other agent stop. So there is no downtime for the unaffected tenant network until the HA solution takes care of failing over the agent. In case of failover, responsibility for cleanup (processes and ports) lies on neutron-netns-cleanup and neutron-ovs-cleanup.

In future follow ups, we plan to implement a notify action to the process manager when the corresponding piece lands in oslo [6].

Examples of configurations could be:

Disabled, external processes are not polled for liveness

check_child_processes_period  = 0

Log (implicit) and respawn

check_child_processes_action = respawn
check_child_processes_period = 60

Log (implicit) and notify

check_child_processes_action = notify
check_child_processes_period = 60

Log (implicit), and exit

check_child_processes_action = exit
check_child_processes_period = 60

This feature will be enabled by default (60 seconds), and default action will be ‘respawn’.

Data Model Impact

None

REST API Impact

None

Security Impact

None

Notifications Impact

None

Other End User Impact

None

Performance Impact

Some extra periodic load will be added by checking the underlying children. Locking of other green threads will be diminished by starting a green thread pool for checking the children. A semaphore is introduced to avoid several check cycles from starting concurrently.

As there were concerns on polling /proc/$pid/cmdline, I implemented a simplistic benchmark:

i=10000000
while  i>0:
  f = open ('/proc/8125/cmdline','r')
  f.readlines()
  i = i - 1

Please note that the cmdline file is addressed by kernel functions [9] in memory and does not rely on any I/O over a block device, that means there is no cache speeding up the read of this file which would invalidate this benchmark.

root@ns316109:~# time python test.py
real  0m59.836s
user  0m23.681s
sys 0m35.679s

That means, 170.000 reads/s using 1 core / 100% CPU on a 7400 bogomips machine.

If we had to check 1000 children processes we would need 1000/170000 = 0.0059 seconds plus the overhead of the intermediate method calls and the spawning of greenthreads.

I believe ~ 6ms CPU usage to check 1000 children is rather acceptable, even though the check interval is tunable, and it’s disabled by default to let the deployers balance the performance impact with the failure detection latency.

Polling isn’t ideal, but the alternatives aren’t either, and we need a solution for this problem, specially for HA environments.

IPv6 Impact

No effect on IPv6 expected here.

Other Deployer Impact

People implementing their own external monitoring of the subprocesses, may need to migrate into the new solution, taking advantage of the exit method, or a later notify one when that’s available.

Developer Impact

Developers which spawn external processes may start using ProcessMonitor instead of using ProcessManager directly.

Community Impact

This change has been discussed several times on the mailing list, IRC, and previously accepted for Juno, but didn’t make it to the deadline on time. It’s something desired by the community, as it makes neutron agents more resilient to external failures.

Alternatives

Use popen to start services in the foreground and wait on SIGCHLD instead of polling. It wouldn’t be possible to reattach after we exit or restart an agent because the parent will detach from the child and it’s not possible to reattach when agent restarts (without using ptrace which sounds too hackish). This is a POSIX limitation. In our design, when an agent exits, all the underlying children stay alive, detached from the parent and continue to run to make sure there is no service disruption during upgrades. When the agent starts again, it will check in /var/neutron/{$resource}/ for the pid of the child that serves each resource, and it’s configuration, and make sure that it’s running (or restart it otherwise). This is the point we can’t re-attach, or wait [7] for an specific non-child PID [8].
Changing the restart mechanism of agents to an execve from inside the agent itself (via signal capture). The execve system call retains original PID and children PID relationship, thus we could wait on children pid. But this prevents stop/start capability of agents which could be handy during maintenance and development. If we decide to change this in the future, ProcessMonitor implementation could be easily modified to non-polling-wait on pids without changing any of it’s API.
Use a intermediate daemon to start long running processes and monitor them via SIGCHLD as a workaround for the problems in the first alternative. This is very similar to the soon-to-be available functionality in oslo rootwrap daemon, but rootwrap daemon won’t be supporting long running processes yet, even though the problem with this alternative is the case when the intermediate process manager dies or gets killed. In that case we lose control over the spawn children (that we would be monitoring via SIGCHLD).
Instead of periodically checking all children, spread the load in several batches over time. That would be a more complicated implementation, which probably could be addressed on a second round or as a last work item if the initial implementation doesn’t perform as expected for a high amount of resources (routers, dhcp services, lbaas..).
Initially, the notification part was planned to be implemented within neutron itself, but the design has been modularized in oslo with drivers for different types (systemd, init.d, upstart..).

Implementation

Assignee(s)

Adding brian-haley as I’m taking a few of his ideas, and reusing partly his work on [5].

Work Items

ProcessMonitor, and functional testing: done
Implement in dhcp-agent, refactoring the code duplication with neutron.agent.linux.external_process. [1]
Implement in l3-agent [2]
Implement in lbaas-agent
Implement in vpnaas-agent

Notes: a notify action was planned, but it’s depending on a new oslo feature, this action can be added later via bug process once the oslo feature is accepted and implemented.

Dependencies

The notify action depends on the implementation of [6], but all the other features/actions can be acomplished without that.

Testing

Tempest Tests

Tempest tests are not capable of doing arbitrary execution of command in the network nodes (killing processes for example). So we can’t use tempest to check this without implementing some sort of fault injection in tempest.

Functional Tests

Functional testing is used to verify the ProcessMonitor class, in charge of the core functionality of this spec.

API Tests

None

Documentation Impact

User Documentation

The new configuration options will have to be documented per agent.

This are the proposed defaults:

check_child_processes_action = respawn
check_child_processes_period  = 0

Developer Documentation

None

References

Add Subnet Allocation to IPAM

Wed, 20 Nov 2024 00:00:00

https://blueprints.launchpad.net/neutron/+spec/subnet-allocation

Author:: Carl Baldwin <carl.baldwin@hp.com>
Co-Author:: Ryan Tidwell <ryan.tidwell@hp.com>

The current implementation of IPAM does not provide a mechanism to set up an address space from which subnets can be allocated. This will be needed in order to automatically allocate addresses for subnets instead of requiring subnet details at the time of creation.

This becomes very important with IPv6 where floating IPs are not implemented. A tenant may want to create a network with addresses that are routable: not just that the addresses are globally unique – because all IPv6 addresses should be – but that the operator can actually route them.

Problem Description

IPAM in Neutron cannot allocate subnets. Subnet details must be specified by the End User at the time of subnet creation.

End Users may want to offload the burden of keeping track of subnets and which addresses are in use. In this case, the End User should be able to set up a private address space from which these are automatically allocated. For IPv4, this will often be a portion of the RFC1918 address space but doesn’t need to be. It might be part of a corporate address space which has been delegated to the cloud. For IPv6, the End User may want Neutron to automatically calculate a useable ULA subnet using a pseudo-random algorithm in harmony with RFC4193 [1]. This implies that the algorithm for the selection of subnets within the space is pluggable in some way.

Deployers will set up external networks and may have a chunk of routable addresses that could be leased or delegated to tenants for use on their networks.

Neutron needs an API for creating and managaging address spaces and making them available to tenants.

Proposed Change

Overview

The purpose of this blueprint is to enable creation and management of subnet allocation pools. This will require changes to the core API and will also require a modification to subnet creation to allow specifying an address space id and a prefix length in lieu of actual subnet address details.

A reference implementation for this new feature will be added to implement this along with the included reference IPAM implementation.

A subnet pool can be shared or not shared. Only admins can create shared pool.

A quota mechanism will be added for shared pools. Quotas will be expressed in terms of the number of minimum atomically allocatable address units. To keep the math simple, the unit size will be hard-coded at /32 for IPv4 and /64 for IPv6. Counting the total number of addresses with IPv6 will make things cumbersome since even an unsigned 64 bit integer is not sufficient to express numbers this large. It would also require extra complexity around presentation in order to present these numbers to a user in a way that makes any sense at all. The implementation will share code between IP versions. The only difference will be the prefix size constant.

The resource quotas are applied to is not the SubnetPool, but rather IP addresses. As such, the current quota engine is not able to perform this operation so management and enforcement should occur in a custom fashion.

Operators may want to charge for allocations (hopefully not with IPv6) but the mechanism by which they can do this is beyond this bp’s scope.

A quota mechanism will be used for the number of SubnetPools which can be created per tenant just to avoid excessive abuse of the API.

Address Scoping

There should be one SubnetPool for each address scope. The pool simply tells us what addresses are available within the scope for allocation. It is out of this bp’s scope to implement anything more. However, in the future, there may be more use cases for allowing routing across scopes. For example:

NAT may be used in IPv4 even without external networks.
Two scope owners may agree that routing should be allowed. This might be useful when RBAC is implemented to allow tenants to cross-plug networks.
A tenant has some globally routable addresses and wants to work it out with the cloud operator to route it externally.

Neutron has been implicitly using a single scope per tenant. Address uniqueness is not explicitly enforced in this scope. Most tenants likely already do this on their own. However, Neutron has not been enforcing uniqueness within a tenant’s address scope.

Overlapping addresses will be allowed for IPv4 pools. By default, the pool will allow overlapping. This can be changed by setting a flag when creating the pool. Overlapping will not be allowed under any circumstances inside IPv6 pools. However, there will be no enforcement of uniqueness across pools.

Multinetting on a network using subnets from different pools will be allowed.

Data Model Impact

New objects will be added:

SubnetPool

Attribute	Type	Description
id	UUID	Unique identifier for the pool.
name	String	Name of the pool
tenant id	UUID	The tenant owning the space.
shared	boolean	Whether the pool is shared.
version	4 or 6
allow overlap	boolean	Whether the pool allows overlapping
min prefix len	Integer	Largest subnet that can be allocated.
default prefix len	Integer	The default subnet size to allocate

SubnetPoolRanges

Attribute	Type	Description
Subnet Pool	UUID	Unique identifier of the SubnetPool
first ip	Integer
last ip	Integer

SubnetAllocation

Attribute	Type	Description
id	UUID	Unique identifier for the allocation.
Subnet Pool	UUID	Unique identifier of the SubnetPool
IP Version	4 or 6
prefix	Integer	The allocated network address prefix
prefix len	Integer	The prefix length

The pool_id field will be added to Subnet:

Subnet

Attribute	Type	Description
All fields as currently implemented
pool id	UUID	ID of the pool the subnet was allocated from

When the subnet is either not allocated from a pool or is migrated during upgrade, pool_id will be ‘null’.

No data migration is necessary. The standard script to create the initial empty tables will be provided.

Needing an availability table like in existing address IPAM is not anticipated. This will be computed dynamically. Subnet allocation will be performed much less often than port allocation and won’t be as contentious in the database. This is an implementation detail to be worked out later. The availability table or some other clever solution may be necessary after all.

Avoiding overlap in the reference implementation will be a little trickier than with simple IP address allocation. It can’t use a simple unique constraint on the database because we want to support allocating subnets of different sizes, especially for IPv4. It could store allocations in terms of the minimum allocation size. So, a larger subnet allocation would be stored using multiple rows in the database. If SubnetPools were always of reasonable size, storing availability and allocations in the same table by prepopulating the entire table might be feasible. This is an implementation detail that can be worked out during code review.

If we stored allocations in terms of the minimum allocation size then we will have problems if that size is updated after the pool was created. For example, if the minimum size is increased, then what happens to existing allocations that are already smaller and don’t align with the new size?

REST API Impact

Subnet details become optional in subnet-create. Instead, an address space can be chosen along with a prefix length indicating the size of the subnet desired. This table summarizes changes to the subnet creation API.

Attribute Name	Type	Access	Default Value	Validation Conversion	Description
cidr			allocated automatically
address pool	string (UUID)	same as cidr	none	Must exist and tenant must have access	Pool to allocate from
allocation pools					-Use 0.0.0.0/NN
gateway ip					Use 0.0.0.0

Allocation pools and gateway ips can still be specified as they are with current subnet creation. Since the actual subnet address is not known, they must be specified using 0 as a wildcard prefix (0.0.0.0/NN) for the subnet where NN is the prefix length chosen. The actual network prefix will be filled in when it has been allocated. For example, if I send this in a successful subnet create call:

cidr = 0.0.0.0/25
address pool = <some id>
gateway_ip = 0.0.0.1
allocation_pool = 0.0.0.64 - 0.0.0.126

I might get this back:

cidr = 10.10.10.128/25
address pool = <some id>
gateway_ip = 10.10.10.129
allocation_pool = 10.10.10.192 - 10.10.10.254

In some cases, the tenant is perfectly content with a default prefix length determined by the pool. This has utility with IPv6 where the tenant just needs to be allocated a /64. In such cases,the pool is configured with a default prefix length and tenants have no need to supply a prefix length when requesting a subnet from the pool. For example, a subnet create call using the default prefix length of the pool (/25 in this case) would look like this:

Request:

cidr = <not specified>
address pool = <some id>
gateway_ip = 0.0.0.1
allocation_pool = 0.0.0.192 - 0.0.0.254

Allocation:

cidr = 10.10.10.128/25
address pool = <some id>
gateway_ip = 10.10.10.129
allocation_pool = 10.10.10.192 - 10.10.10.254

The following explains how cidr and address pool can be used together. The basic rules are that cidr and prefix are mutually exclusive and one must be specified. If a prefix length is specified, a pool must be specified too. The exception to this is if an option global default pool is defined in neutron.conf.

cidr	address pool	action
		Error when no global default pool defined	else subnet with default prefix length is allocated
	specified	Allocate using the pool’s default prefix length
specify specific CIDR		Same as before. Uses Implicit tenant address pool.
specify specific CIDR	specified	Tries allocating specified subnet from pool.
specify wildcard CIDR	specified	Subnet with requested prefix length from pool.
specify wildcard CIDR		Error when no global default pool defined	else allocate subnet from global pool

New errors from the API are possible: SubnetPoolNotFound, PrefixLengthTooBig/Small, NoAddressesAvailable.

New methods need to be added to create and manipulate SubnetPools.

Attribute	Access	Type	Required	CRUD	Default Value	Validation Constraints	Notes
id	RO, all	string(UUID)	N/A	R	generated	N/A	UUID representing the address space
name	RW, owner	string	Yes	CR	N/A	name of the pool
shared	RO, all (if True); RW, admin	bool	No	CRU	False	True/False	whether other tenants see it
version	RW, owner	integer	Yes	CR	N/A	4/6	The IP version
allow overlap	RW, admin	bool	No	CR	True if version=4	False if version=6	True/False	allow overlapping subnets
min prefix len	RW, owner	integer	Yes	CRU	N/A	viable prefix lengths for IP version	The IP version
default prefix len	RW, owner	integer	Yes	CRU	determined by min_prefix_len and version	> min_prefix_len & < max version prefix	default prefix allocation len
ranges	RW, owner	list(2-tuples or CIDR’s)	Yes	CRU	N/A	valid non-overlapping ranges

Basically, if shared is True then all tenants can read all fields. If it is not true then only the owner can see the pool. Only the owner is able to write the fields in any case. Only admin can write to the shared field.

For IPv4, the prefix lengths should be between 8 and 30. For IPv6, likely between 32 and 64.

There are certain parts of the IPv6 address space that are simply not specified for any kind of use (i.e. outside 2000::/3, fc00:/7 for ULA, and other specified scopes) [2]. Some validation is called for. The following address spaces should be allowed:

Addresses	Description
2000::/3	Global unicast
fc00::/7	ULA addresses which can be routable within sites.

If ranges are updated after the initial creation, nothing will be done about existing subnet allocations that happen to fall outside of the new ranges.

This bp will start with the same shared model that we have now for networks. However, an RBAC mechanism can be added later similar to the one proposed for networks [3].

Security Impact

There is an new API. With any new API, there is the potential for new attacks on the system. For example, if someone could obtain control over an address space, they could shrink it down to nothing and prevent further allocations.

As long as only admins can create shared pools and quotas are in place on the shared pools, there should be no new vulnerabilities introduced. With that said, particular attention should be paid during code review to guard against the introduction of new ones.

Someone with unlimited control of an address space could potentially fill it up as another way to prevent any further allocations.

Notifications Impact

None

Other End User Impact

Performance Impact

Allocation is performed on subnet create. This may involve a couple of significant database queries. Subnet create is not nearly as common as a port create so this is not expected to be a problem. Use of optimistic locking techniques should mitigate the impact.

The new API methods for creating and updating pools aren’t expected to be called often enough to have any significant impact.

We should consider atypical use cases in addition to the typical. If the implementation performs very poorly, it could be used as a denial of service attack and pose a Security Risk. This should be addressed during code review.

IPv6 Impact

This new feature must work for IPv6 equally as well as IPv4. This is intended to enhance the IPv6 experience in Neutron. There will be no effect on existing IPv6 features in Neutron.

Other Deployer Impact

By default, the cloud system will work like it does today except that tenants will have the ability to create their own pools without any deployer action.

The deployer may use the shared pools feature to create pools of addresses that will be available to tenants for use on their networks but is not required to. They will likely want to use this feature if they want to route to tenant networks either globally or within the datacenter.

External IPAM may be used with this new API. Development of external IPAM drivers is out of the scope of this blueprint.

Developer Impact

None

Community Impact

This change was discussed at the Kilo design summit in the pluggable IPAM session. It has also been discussed with the IPv6 subteam. This has been recognized as a need for the community, especially for IPv6 routing. With this API, the IPv6 subteam can just use prefix delegation as a mechanism for handing out allocations, after a user has made a request to this API.

Alternatives

N/A

Implementation

Assignee(s)

Primary assignee:: ryan-tidwell
Other contributors:: carl-baldwin L3 Subteam

Work Items

subnetpools REST API and corresponding DB support
Adjustments to subnets API and DB schema
Minimal Horizon enablement to support basic subnet allocation (v4 & v6)
Tempest test
Functional Test
API test

Dependencies

This blueprint depends on the work in the neutron-ipam [4] one. That blueprint adds some of the frame-work necessary to implement this new feature. For further information, see the Etherpad [5].

Testing

Unit test all new code of course. This means that the code structure must be testable. Will use TDD.

Tempest Tests

Create v4 subnetpool
Create v6 subnetpool
Allocate v4 subnet from subnetpool
Allocate v6 subnet from subnetpool

Functional Tests

Verify quota enforcement
Lower tenant quota, verify previously allocated resources intact, verify enforcement of new value
Assert applicable subnet allocation details not leaking cross-tenant with shared pools
Create subnetpools with allow_overlap=True and allow_overlap=False, verify allocation of subnets

API Tests

Verify defaults on subnetpool creation of v4 pools
Verify defaults on subnetpool creation of v6 pools
Verify allow_overlap is constrained to False for v6 pools
Create v4 subnetpool, verify shared and allow_overlap values
Create v6 subnetpool, verify shared and allow_overlap values
Allocate v4 subnet from subnetpool, assert success and subnet details
Allocate v6 subnet from subnetpool, assert success and subnet details

Documentation Impact

User Documentation

Update networking API reference Update admin guide

Developer Documentation

N/A

References

Pluggable IPAM Subsystem in Neutron

Wed, 20 Nov 2024 00:00:00

Launchpad blueprint: https://blueprints.launchpad.net/neutron/+spec/neutron-ipam

This blueprint introduces the pluggable IPAM subsystem in Neutron that allows flexible control over the lifecycle of network resources like the following:

Fixed IP addresses assigned to Neutron ports
Floating IP addresses
Network address ranges, i.e. sub-networks

Problem Description

Users have a requirement to integrate OpenStack into their existing infrastructure that uses external IPAM.
Currently most (if not all) Neutron plugins leverage an IPAM implementation that is embedded in the db_base_plugin implementation. While this works well in a self-contained system, it makes it difficult or impossible to integrate with an external IPAM backend without terrible hacks.

The current architecture of Neutron does not allow users to implement their own IPAM mechanism in any other way other than introducing their own core plugin. That is, while the DHCP provider can be changed, the actual allocation logic cannot.

Proposed Change

Overview

The proposed change is the addition of a well-defined, abstract IPAM interface, and the refactoring of the existing NeutronDbPluginV2 to utilize that interface rather than directly perform IPAM actions on its own. A related blueprint [1] is proposed to implement a reference IPAM driver that captures the current behavior.

IPAM Interface

IPAM subsystem architecture overview is shown below. There is an abstract base class defined, IPAMDriver. There will be a reference implementation of this class - NeutronIPAM - that will encapsulate the current behavior. The IPAMDriver can be subclassed by third parties to implement different IPAM behaviors, such as different subnet or IP allocation strategies, or access to an external IPAM system.

The Neutron Plugins will call into the driver for IPAM functionality.

The driver interface will be synchronous.

The IPAM implementation will define abstract Request classes for subnets and addresses. These classes enable the caller to request subnet or IP allocations using criteria other than the explicit subnet or address, though implementation of request criteria other than the existing behavior is not within scope of this blueprint.

See [2] for the interface definition. The details of this interface are not part of this spec, and work on this interface may continue beyond approval of this spec.

Interaction Examples

An example interaction scenario between Neutron Plugins and IPAM are shown in the following diagram. In this diagram and the one that follows, “Pluggable IPAM” represents an optional external IPAM system. These diagrams are intended as examples; the specific details of each flow will be defined during implementation.

Here is another flow demonstrating the call to create a port. In this case, the IPAM driver is called in order to retrieve a subnet, or allocate the subnet if not found. That is, the “get_subnet” call may simply retrieve an existing subnet from the Neutron database, or it may go to an external IPAM system to query or allocate the subnet. The behavior is left to the discretion of the driver.

The IPAMSubnet object is used for the IP allocation.

Driver Creation

The IPAMDriver will contain a factory method to generate specific driver instances. There will be a driver instance per SubnetPool (see [3]). However, in this release only a single driver will be supported across the deployment.

Note that this BP will not provide any database or API for the SubnetPool. As part of this BP only a minimal implementation will be created.

The IPAM driver to use for a SubnetPool is specified through the configuration file /etc/neutron/neutron.conf. The default value will point to the reference NeutronIPAM driver.

Refactoring

The existing NeutronDbPluginV2 must be refactored to utilize the new IPAM interface. Several core plugins make calls to IPAM-related private methods in the NeutronDbPluginV2. Stub versions of those methods must be left in place and be refactored to utilize the IPAM interface, or the plugins must themselves be refactored to avoid calling private methods of the base class. The current implementation within those methods will be moved to the reference driver.

While the driver will enable an external IPAM system to provide the authoritative response on whether to allocate a new address or subnet, the Neutron database will still be required to have an accurate representation of the currently allocated subnets and IP addresses. Queries for existing allocations will still access the local database rather than call out through the driver. The synchronization of external IPAM and Neutron during initial migration and for ongoing verification purposes is the responsibility of the driver author, either within the driver or external to it.

The DB activities currently done in NeutronDbPluginV2 would better be handled via composition rather inheritance. The base plugin could have a database handler object that performs these functions. This would enable the database transaction to be performed outside (after) the addressing decision is made by the external system. This avoids a call involving I/O during an open transaction, which can lead to deadlock issues due to a MySQL connector flaw.

This goes beyond the IPAM functions, of course. The idea here being that all the plugins still need the core data in the Neutron DB even if they may need to store additonal data or perform additional actions during these calls.

Out-of-Scope Items

Several related functions have been discussed in relation to this blueprint.

DHCP options such as nameservers and host routes are intentionally de-coupled from the IPAM implementation. Pluggable DHCP would require a separate effort, or must be addressed within the individual drivers.

Similarly, integrations with Designate or other external DNS services during IPAM activities is out-of-scope.

IPAM Regional Internet Registries (RIRs) life-cycle management and automation is beyond the scope of this blueprint.

Data Model Impact

There will be no data model changes for this implementation, only the addition of interfaces and non-persistent classes. Rather, related data model updates are captured in [4], though this is not strictly required for this blueprint.

REST API Impact

None.

Security Impact

None.

Notifications Impact

None.

Other End User Impact

None.

Performance Impact

IPAM subsystem implementation of the default Neutron driver should have similar performance to the current Neutron IPAM. The performance impact of external IPAM drivers is beyond the scope of this document.

IPv6 Impact

Support for IPv6 is a requirement of this specification.

Other Deployer Impact

A new configuration option to specify the desired IPAM driver will be available in the neutron.conf file. If this value is not specified Neutron Server will fallback to the default Neutron IPAM driver in the default location. This choice was made to support backward compatibility with older neutron.conf files that do not have this option specified.

Developer Impact

By default the Neutron should work as it does today. Supplied reference IPAM driver should encapsulate current functionality.
As core plugins override several methods from the base plugin class, we will evaluate impact of the IPAM changes to those plugins.

Community Impact

This change was discussed at the Juno and Kilo Design summits. There was support for Pluggable IPAM, see link to the Etherpad in the Reference section of document.

Alternatives

None.

Developer Impact

Implementation

Assignee(s)

Primary assignee:: John Belamaric (jbelamaric)
Other contributors:: Salvatore Orlando (salvatore-orlando) Carl Baldwin (carl-baldwin) Ryan Tidwell (ryan-tidwell) Hosung Hwang (hhwang-2) Yue Ko (yko) Pavel Bondar (pasha117)

Work Items

Create IPAM abstract interfaces.

Create pluggable IPAM for db_base_plugin_v2.

Move all the IPAM-related functionality from db_base_plugin_v2 to the Neutron IPAM plugin.

Dependencies

https://blueprints.launchpad.net/neutron/+spec/reference-ipam-driver

Testing

The existing unit test will be used when appropriate and unit test coverage will be expanded to cover refactored Neutron IPAM code.

Functional Tests

Existing Functional Tests will be used when appropriate, refactoring of IPAM may require additional or refactored functional tests.

Tempest Tests

The existing Neutron Tempest tests will be utilized to test the default Neutron IPAM that will be developed.

API Tests

No change to API proposed.

Documentation Impact

User Documentation

Admin guide will be updated.

Developer Documentation

API guide will be updated.

References

Dynamic Advertising Routes for Public Ranges

Wed, 20 Nov 2024 00:00:00

Blueprint Link

The goal of this blueprint is to add dynamic routing capability to OpenStack deployment. This feature would allow Neutron to advertise its own routes to an external router using the BGP protocol.

Problem Description

This is a new feature and this section will describe use cases where dynamic routing may be useful.

Expose External Networks Dynamically

Allow Neutron to dynamically announce to external uplink routers.

An OpenStack cloud would run a routing protocol (for example, BGP) against at least one router in each uplink network provider. By announcing network prefixes to those peers, the Neutron network would be reachable by the rest of the internet via both paths. If the link to an uplink provider broke, the failure information would propagate to routers further up the stream, keeping the cloud reachable through the remaining healthy link.

Sample Topology

Please note that in this diagram, the red gateway router is actually a linux machine with a software BGP speaker and a software router. This blueprint proposes to configure this BGP speaker through the Neutron API (see below).

This also would be a valid scenario for an internal cloud, where the uplink router is actually one’s provider router and the routes are advertised using iBGP.

Routed Model for Floating IPs on External Network

This use case will allow an external network with a large public IP space to possibly span more than one L2 network. This could allow for improved scale and isolation between AZs while maintaining the appearance of a large external network where floating IPs and networks can float freely.

This use case may also include announcing public networks behind Neutron routers to an upstream router. For example, this will be useful for IPv6 networks when using the subnet-allocation mechanism to allocate routable IPv6 prefixes to tenant networks. It does not require learning routes from the upstream router.

Address Scopes blueprint adds a new L3 context to make public routable tenant networks (among other advantages). Address Scopes may leverage this development to advertise the new routable subnets to uplink routers.

Sample Topology

The topology of this use case can be seen as a generalization of the previous one, with a multi-homed OpenStack installation and leverage the fact that a floating IP can be seen as a /32 network.

Proposed Change

Overview

A new system that dynamically advertises routes to other peers outside the OpenStack deployment is proposed. From the Neutron API, cloud administrator should be able to define these peers as well as the way to interact with them.

Routing Peer

A system that supports dynamic routing must be able to advertise its own routes to its peers. It order to achieve this goal the system must know the list of its peers and be able to trust the information that it receives from them.

Route Advertisement

Routes added manually by administrator will be advertised to external dynamic routing peers.

Dynamic Routing System

In default implementation a new system will be used to manage dynamic routing information at the edge of OpenStack deployment. Dynamic peering will not be performed by each individual Neutron router due to scaling concerns. Such an approach could create an inordinately high number of peering relationships. Instead, the model proposed sets up a speaker to represent the Neutron deployment as a whole to external routers.

This system will allow different implementations (develop your own BGP speaker implementation, for instance) to fit into third party requirements.

IPv6 Impact

The implementation must be able to exchange IPv4 and IPv6 routes. This blueprint may be even more important with IPv6 to complete routing from outside the cloud to tenant networks running IPv6.

Solution Proposed

Reference implementation will employ a dynamic routing agent (dr_agent). This agent will have direct connectivity with configured peers and it will be responsible to update the dynamic routes of some defined routers. Another spec will be filled to define the details of this implementation.

So, through the Neutron API, cloud administrator should be able to define the peer connections for each agent according to his needs.

The cloud administrator will be able to execute the following new actions, through the Neutron API:

Create a BGPSpeaker entity with configuration options of the BGP connection. First approach will only need the local_as attribute. Future implementations, such as the policy support for dynamic routing, will need to add more attributes here. The implementation driver need to read this configuration options an establish BGP sessions to its peers.
Create a RoutingPeer entity.
Associate a RoutingPeer entity to a BGPSpeaker. The BGPSpeaker implementation driver will be able to connect to this RoutingPeer
Associate a Router to a BGPSpeaker. That means the router will have dynamic routing capabilities that will let it advertise its routes
Associate a Network to a BGPSpeaker. All the routers with external gateways attached to this network will use the same BGPSpeaker attributes.

At this point, Neutron will compare the address scope of the subnets to which assigned routers (step 5 implies all the routers of the external network) have their external gateways connected to the internal subnets of the tenants to see if they belong to the same address scope. If so, the BGP speaker implementation will advertise these tenant subnets to its configured RoutingPeers. Floating IPs will be included implicitly since they are allocated from the external network.

Considerations

Alternatives

Multi-homed clouds can be handled using classic networking infrastructure, configuring manually the vendor router with BGP outside the OpenStack deployment. This is limited. It can’t handle the routed floating ip model proposed above.

Data Model Impact

This document proposes modifying data objects and schema in the following way. For a quick glance of the Data Object Model, check out this etherpad.

Data Object Changes

Three new data model classes will be added: BGPSpeaker, RoutingPeer and AdvertiseRoute.

We will need the binding entities:

RoutingPeerBGPSpeakerBinding to associate peers to BGPSpeaker.

RouterBGPSpeakerBinding to associate routers BGPSpeaker.

NetworkBGPSpeakerBinding to associate networks to BGPSpeaker.

New BGPSpeaker class will contain the following attributes:

id: UUID of the entity.

local_as: Local AS value.

Now we only need these values. In the future, more advanced configuration options of BGPSpeaker will be able to be added here.

New RoutingPeer class that represents a peer connection will contain the following attributes:

id: UUID of the entity

ip: IP Address of the peer.

remote_as: Remote Peer’s AS value.

auth: Authentication data of the connection that can be serialized as a
dictionary. { type: ‘MD5’, password: ‘234a23d10234’ } could be a simple example and first approach.

Another data object called AdvertiseRoute will be created extending the Route entity and associated to a router. Will have the following attributes:

nexthop: IP address of the next hop.

destination: CIDR prefix.

router_id: UUID of the Routing Instance.

RoutingPeerBGPSpeakerBinding is an n-to-n relationship between bgp peer and bgp speaker and only will have:

peer_id: UUID of the BGPPeer

bgpspeaker_id: UUID of the Dynamic Routing Agent

RouterBGPSpeakerBinding is an n-to-1 relationship with the attributes:

router_id: UUID of the Router

bgpspeaker_id: UUID of the Dynamic Routing Agent.

NetworkBGPSpeakerBinding is an n-to-1 relationship with the attributes:

network_id: UUID of the Network

bgpspeaker_id: UUID of the Dynamic Routing Agent.

RouterBGPSpeakerBinding it will be used when you want to propagate a single router’s AdvertiseRoutes. This option is suitable for the Expose External Networks Dynamically use case, explained above. It can be seen as adding dynamic routing capabilities to a Router

NetworkBGPSpeakerBinding is the Routed Model for Floating IPs on External Network use case: in this case you propagate tenant routes (IPv6) or Floating IPs into an upstream router. These routes are more dynamic to assign, because it is up to the tenant use to set them. You will want any router attached to a network, to automatically be added to propagate its routes.

REST API Impact

API endpoints should be implemented according to the Solution Proposed section.

Security Impact

This feature will allow an external system to manipulate routing information within Neutron network. The external system should be trusted and may be authenticated using a shared secret.

Dynamic routing may only be configured by the system administrator.

Notifications Impact

A notification should be provided when connectivity of control channel over which routes are exchanged is interrupted

Other End User Impact

The following CLI commands will be added to manage dynamic routing specification for connecting OpenStack to outside networks:

bgp-speaker-list: List configured bgpspeakers.
bgp-speaker-show: Show detailed bgpspeaker configuration.
bgp-speaker-create Create new bgpspeaker connection.
bgp-speaker-update: Update bgpspeaker specification.
bgp-speaker-delete: Delete bgpspeaker specification.
bgp-speaker-peer-add: Associate a peer to a bgpspeaker.
bgp-speaker-peer-list: List Peers on bgpspeaker
bgp-speaker-peer-remove: Remove peers on bgpspeaker.
bgp-speaker-network-add: Associate a network to a bgpspeaker.
bgp-speaker-network-list: List networks on bgpspeaker
bgp-speaker-network-remove: Remove networks on bgpspeaker.
bgp-speaker-router-add: Associate a peer to a bgpspeaker.
bgp-speaker-router-list: List routers on bgpspeaker
bgp-speaker-router-remove: Remove router on bgpspeaker.
bgp-peer-list: List configured peers.
bgp-peer-show: Show detailed peer configuration.
bgp-peer-create Create new peer connection.
bgp-peer-update: Update peer specification.
bgp-peer-delete: Delete peer specification.
router-advertiseroutes-list: List advertise routes.

Horizon Requirements

A new screen will be added to configure gateway configuration for connecting OpenStack to outside networks. This screen will allow routes and peer configuration to be added to gateway configuration.

An external network will have an option to be linked to a routing instance.

Usage Example

Configure 2 uplinks for the routing instance serving an external network to advertise its routes and update the discovered ones.

Sample configuration using Neutron CLI commands:

neutron bgp-speaker-create --local-as 12345

neutron bgp-peer-create --ip 123.23.43.4 --remote-as=1234

neutron bgp-speaker-peer-add peer1 (previous step created peer. should be an
uuid, but modified for the sake of understanding)

neutron bgp-speaker-network-add bgpspeaker1 network1 (same here, with uuids)

Performance Impact

This feature describes an out of band mechanism to negotiate routing configuration. This feature should not have a performance impact on Neutron network.

Other Deployer Impact

This feature would have to explicitly enabled and configured before it will take effect. There are no changes to configuration files.

Developer Impact

This change does not affect current developments or any plugin development.

Neutron API exposed is agnostic of the exchange routing protocol used. If another developer want to provide other driver than BGP with exabgp, only the dr_agent part will be affected with new code.

Community Impact

This change does not impact community.

Alternatives

Taking into account the use case (BGP connectivity), we think that the agent approach is the only one that fits into Neutron. Although maybe the functionality would be solved using another entities and workflow.

Implementation

Assignee(s)

This is a pre-liminary contributor list

Primary assignee:: tidwellr vikram
Other contributors:: devvesa YAMAMOTO

Work Items

Create the dr_agent, exposing the API and implemented with the chosen BGP speaker. (BGP Comparison)
Model tables and API resources.
Periodically scheduled process to communicate with dr_agent.
Testing.
Devstack.
Documentation.

Dependencies

Depending on the implementation, new system library or python library will need to be installed.

Testing

Tempest Tests

Dynamic routing testing may be performed in an isolated environment. An external autonomous system may be simulated with an instance of BGP capable software router (for example, quagga).

The following dynamic routing scenarios could be tested:

Verify that when BGP is enabled on the gateway and one peer is configured the agent establishes BGP session with the peer, receives a list of routes, and submits advertised routes to the peer.

Verify that when BGP is disabled on the gateway and one peer is configured the dr_agent establishes no BGP sessions.

Verify that when BGP is enabled on the agent and 3 BGP peer connections are configured, the agent establishes 3 BGP sessions, one to each of the configured peers.

When 2 or more peers are configured, verify that BGP implementation is able to detect when the BGP session is interrupted the routes received from that BGP session are automatically removed from the routing table.

Functional Tests

Full top-down Neutron API internal logic must be developed by mocking the agent.

API Tests

All API exposed endpoints by Neutron extensions must be tested.

Documentation Impact

New documentation for the whole functionality.

User Documentation

User documentation explaining the functionality must be provided.

Developer Documentation

Developer documentation about how to develop a new driver for the dr_agent must be provided.

References

Previous work:

Links and helpers:

Port Forwarding API

Wed, 20 Nov 2024 00:00:00

https://blueprints.launchpad.net/neutron/+spec/port-forwarding

Port forwarding is a common feature in networking and more specifically in PaaS and SaaS cloud systems which aim at reusing the same public IP for different clients that use different VMs for their services.

This is especially relevant for deployments which lack a large number of public IPs they can assign.

Common use case for this feature is a client requesting a specific service, where the serving platform (PaaS, SaaS) allocates a VM to run the service and then allocates a client port to access this service. This means that various clients use the same public IP, but the TCP/UDP destination port is used to distinguish between the end point VMs.

Example, a mapping for web servers:

client1 172.24.4.2:4001 TCP => maps to 10.0.0.2 port 80 TCP (VM1)
client2 172.24.4.2:4002 TCP => maps to 10.0.0.3 port 80 TCP (VM2)

This spec will focus on port forwarding based on Floating IPs. A future spec will be submitted for port forwarding based on routers external gateway interface.

Problem Description

In environments constrained with limited IPs, operators would like to reuse public IPs instead of assigning to each VM its own public IP (Floating IP).
Docker supports a port-mapping feature and hence a big eco-system of automation orchestration and management plugins leverage it. We would like to make Neutron compatible for these tools and systems and provide a similar API [1].

Proposed Change

Introduce port forwarding API and implementation to Floating IPs.

The user can define various port forwarding rules on the Floating IPs containing the internal/client port and the external/destination port, connected with the VM they want to expose. And users would be allowed to create port forwarding rules only on a “free” Floating IP, i.e. a Floating IP which is not directly associated with a Fixed IP of a tenant’s VM.

We will have four deployment variants:

a) Legacy Router: For the generic Router deployment, the port forwarding rules would be installed in the Router namespace on the network node and forwarding functions would be performed.

b) HA Router: For the HA Router deployment, the port forwarding rule will be installed in both the ACTIVE and the BACKUP Router namespaces on the network nodes they are located on.

c) DVR: As [2] has been merged, we now have the ability to create centralized Floating IPs in a DVR supported deployment. This helps in mapping the Compute nodes where the destination VM is present with the centralized FIP for Port forwarding. This mechanism will centralize a floating IP not only when it is associated with a port bound to a host with the dvr_no_external option enabled, but also when port forwarding attributes are added to it.

d) DVR + HA: If the created router is not an HA router, then we can proceed with option (c). While if the router is an HA router, then we will use the centralized HA router to install the port forwarding rules.

In all deployment variants, the port forwarding entry NATs a specific Floating IP:Port and protocol to a specific Neutron port (and a private IP that is attached to this port). That means it will maintain a mapping like “FIP:extport protocol” to “Neutron Port Fixed IP:intport protocol”. So if a Neutron Port contains multiple Fixed IPs, then it will be allowed to create multiple port forwarding entries for a particular Neutron port, with different external ports, such as:

FIPX:EXTPORTX PROTOCOLA => Neutron PortQ Fixed IPA:INTPORTA PROTOCOLA (VM1)
FIPX:EXTPORTY PROTOCOLA => Neutron PortQ Fixed IPB:INTPORTA PROTOCOLA (VM1)

However, the same Fixed IP:intport socket cannot be mapped with different protocols.

If the Neutron port is deleted, the port forwarding entries that match this port are also deleted. Same is applicable in case the Floating IP is deleted.

Data Model Impact

The following new table is added as part of the port forwarding feature:

CREATE TABLE port_forwarding (
    id CHAR(36) NOT NULL PRI KEY,
    floating_ip_id VARCHAR(36) NOT NULL,
    external_port INT NOT NULL,
    internal_neutron_port_id VARCHAR(36) NOT NULL FOREIGN KEY,
    protocol CHAR(4) NOT NULL,
    socket VARCHAR(20) NOT NULL,
    CONSTRAINT floating_ip_id_external_port_constraint UNIQUE (floating_ip_id, external_port),
    CONSTRAINT internal_neutron_port_id_socket_constraint UNIQUE (
        internal_neutron_port_id, socket)
);

The socket column will store the string like ‘Fixed IP:Port’.

Note

This table lacks project_id, as the owner of this port_forwarding must be the owner of associated Floating IP. So there is a project_id check for preventing association of Floating IP to internal Neutron Port if their project_id are different. Also, allow the association that Floating IP/internal Neutron Port exists on a shared network for admin users in different project_id cases, such as FloatingIP from a shared public network created by a admin user and a Neutron Port from a particular internal tenant network created by the tenant user, then admin user want a association of them which have different project_ids. For general users, there is only the same project_id case.

Sub Resource Extension

Neutron floatingips will be extended with a sub resource port_forwarding, it will contain some fields to expose the port_forwarding assigned to the floatingip resource.

For this new feature, a new service plugin will be introduced, and the following methods will be added:

‘create_floatingip_port_forwarding()’
‘delete_floatingip_port_forwarding()’
‘get_floatingip_port_forwarding()’
‘get_floatingip_port_forwardings()’

For update operation, we will extend the function in the future if possible. But for now, we will just support create/delete/get functions.

So the attributes map of new sub resource would be like:

SUB_RESOURCE_ATTRIBUTE_MAP = {
    'port_forwarding': {
        'parent': {'collection_name': 'floatingips',
                    'member_name': 'floatingip'},
        'parameters': {
            'external_port': {'allow_post': True, 'allow_put': False,
                                 'convert_to':
                                     convert_validate_port_value,
                                 'is_visible': True},
            'internal_port': {'allow_post': True, 'allow_put': False,
                                 'convert_to':
                                     convert_validate_port_value,
                                 'is_visible': True},
            'internal_ip_address': {'allow_post': True,
                                        'allow_put': False,
                                        'validate': {
                                        'type:ip_address_or_none': None},
                                        'is_visible': True},
            'protocol': {'allow_post': True, 'allow_put': False,
                           'validate': {
                              'type:values': constants.IPTABLES_PROTOCOL_MAP.keys()},
                           'is_visible': True,
                           'convert_to': converters.convert_to_protocol},
            'internal_port_id': {'allow_post': True,
                                    'allow_put': False,
                                    'validate': {'type:int':None},
                                    'is_visible': True},
        }
    }
}

REST API Impact

The idea is to extend the Floating IP Rest API with a new extension floating_ip_port_forwarding with the below defined attributes.

Floating IP extension
Attribute Name	Type	CRUD	Default Value	Description
port_forwardings	List	R	None	The associated ‘port-forwarding’ sub resource with the particular Floating IP resource.

The Floating IP extension definition would be expanded as :

RESOURCE_ATTRIBUTE_MAP = {
     'floatingips': {
         'port_forwardings': {'allow_post': False,
                                 'allow_put': False,
                                 'is_visible': True, 'default': None}
     }
}

This new field will be exposed in the response during GET/POST/PUT requests of Floating IP resource. That means users can not change the forwarding resources through CRU FloatingIP, only can create the forwardings one by one with the new port forwarding API which will be introduced below.

For example, GET a Floating IP:

GET /v2.0/floatingips/<floatingip-uuid>

{
    "floatingip": {
         "floating_network_id": "376da547-b977-4cfe-9cba-275c80debf57",
         "router_id": "d23abc8d-2991-4a55-ba98-2aaea84cc72f",
         "fixed_ip_address": "",
         "floating_ip_address": "172.24.4.228",
         "project_id": "4969c491a3c74ee4af974e6d800c62de",
         "tenant_id": "4969c491a3c74ee4af974e6d800c62de",
         "status": "ACTIVE",
         "port_id": "",
         "id": "2f245a7b-796b-4f26-9cf9-9e82d248fda7",
         "port_forwardings": [
             {
                 "internal_ip_address": "10.0.0.3",
                 "protocol": "tcp",
                 "internal_port": "22",
                 "external_port": "7001"
             },
             {
                 "internal_ip_address": "192.168.4.32",
                 "protocol": "tcp",
                 "internal_port": "22",
                 "external_port": "7002"
             }
         ]
    }
}

For the new sub resource ‘port_forwarding’, a new url will be introduced:

/v2.0/floatingips/<floatingip-uuid>/port_forwardings

List Port Forwardings

GET /v2.0/floatingips/<floatingip-uuid>/port_forwardings

{
    "port_forwardings": [
         {
             "id": "ae34051f-aa6c-4c75-abf5-50dc9ac99ef3",
             "external_port": "7003",
             "internal_port": "22",
             "internal_ip_address": "10.0.0.10",
             "protocol": "tcp",
             "internal_port_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf"
         },
         {
             "id": "915a14a6-867b-4af7-83d1-70efceb146f9",
             "external_port": "7004",
             "internal_port": "22",
             "internal_ip_address": "10.0.0.11",
             "protocol": "tcp",
             "internal_port_id": "0c56df5d-ace5-46c8-8f4c-45fa4e334d18"
         }
    ]
}

Response Parameters
Parameter	Style	Type	Description
port_forwardings	plain	xsd:list	A list of port_forwarding objects

More parameters see Show Port Forwarding

Show Port Forwarding

GET /v2.0/floatingips/<floatingip-uuid>/port_forwardings/<port-forwarding-id>

{
    "port_forwarding": {
         "id": "ae34051f-aa6c-4c75-abf5-50dc9ac99ef3",
         "external_port": "7003",
         "internal_port": "22",
         "internal_ip_address": "10.0.0.10",
         "protocol": "tcp",
         "internal_port_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf"
     }
}

Response Parameters
Parameter	Style	Type	Description
port_forwarding	plain	xsd:dict	A port_forwarding object
id	plain	xsd:string	The ID of port_forwarding object
external_port	plain	xsd:string	The exposed external protocol port number
internal_port	plain	xsd:string	The port forwarding mapped internal protocol port number
internal_ip_address	plain	xsd:string	The IP Address from the fixed ips of a particular port
protocol	plain	xsd:string	The traffic protocol type, such as TCP or UDP. Default value is ‘TCP’.
internal_port_id	plain	xsd:string	The Neutron internal Port ID.

Create Port Forwarding

POST /v2.0/floatingips/<floatingip-uuid>/port_forwardings

{
    "port_forwarding": {
         "external_port": "7233",
         "internal_port": "22",
         "internal_port_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf"
     }
}

Response parameters

see Show Port Forwarding

{
    "port_forwarding": {
         "id": "f8a44de0-fc8e-45df-93c7-f79bf3b01c95",
         "external_port": "7233",
         "internal_port": "22",
         "internal_ip_address": "192.168.43.33",
         "protocol": "tcp",
         "internal_port_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf"
     }
}

Delete Port Forwarding

DELETE /v2.0/floatingips/<floatingip-uuid>/port_forwardings/<port-forwarding-id>

This operation does not accept a request body and does not return a response body.

Effects on Existing Floating IP APIs

Slight adjustment to existing Floating IP APIs:

Create a Floating IP just the same with current behavior.
Associate a Neutron internal port with a Floating IP, if the requested url is the same as previous, the Floating IP will work as 1:1 DNAT like current behavior. If the request with the new url, that means the request needs a port-forwarding function towards the Floating IP.
Get a Floating IP resource will be the same as before if the Floating IP resource had already associated with a neutron port for 1:1 DNAT. If a Floating IP resource contains more than 1 port-forwarding sub resource, it is better to show the port-forwardings summary in the Floating IP response body to distinguish which Floating IP resource is available for different requirements, such as 1:1 DNAT, port-forwarding.

Command Line Client Impact

Openstack Client would have additional options portforwarding for Floating IP CLI, which would define the port-forwarding characteristics.

Security Impact

Port forwarding is similar in nature to centralized DNAT, so should not pose additional security implications. But if users actually want to use Port forwarding, they must make sure to allow the associated ingress Security Group towards the internal ip which is used by the neutron port of their VMs.

Notifications Impact

Depends on the implementation spec

Other End User Impact

None

Performance Impact

Performance testing must be conducted to see what is the overhead of enabling this feature, of course that if the feature is disabled no performance impact should be noticed.

IPv6 Impact

IPv6 is not supported

Other Deployer Impact

Deployer will be able to leverage port forwarding for a unique way to reach a private VM/Container without wasting new public IPs

Developer Impact

Future SNAT distribution plans should take port forwarding into consideration. Kuryr can leverage port forwarding for feature compatibility with Docker port mapping.

Alternatives

Users can use an external VM that provide this NAT capability or assign a new Floating IP for each VM.

Implementation

Assignee(s)

Primary assignees:

reedip <reedip.banerjee@nectechnologies.in>

Other contributors:

gal-sagie <gal.sagie@gmail.com>

tian-mingming <tian.mingming@h3c.com>

zhaobo <zhaobo6@huawei.com>

Work Items

API Implementation
DB Implementation
Reference implementation
Tests
Documentation

Dependencies

None

Testing

Tempest Tests

Need to add tempest tests

Functional Tests

Need to add functional tests

API Tests

Need to add API tests

Fullstack Tests

Need to add Fullstack tests.

Documentation Impact

User Documentation

Needs user documentation

Developer Documentation

Needs devref documentation

References

Neutron-Neutron Interconnections

Wed, 20 Nov 2024 00:00:00

Launchpad RFE: https://bugs.launchpad.net/neutron/+bug/1750368

Problem Description

Today, to realize connectivity between two OpenStack clouds or more (e.g. between distinct OpenStack deployments, or between OpenStack regions, for instance) some options are available, such as floating IPs, VPNaaS (IPSec-based), and BGPVPNs.

However, none of these options are appropriate to address use cases where all the following properties are desired:

interconnection consumable on-demand, without admin intervention [1]
have network isolation and allow the use of private IP addressing end-to-end [2]
avoid the overhead of packet encryption [3]

An additional design requirement to the solution is to avoid introducing a component that would require having admin rights on all the OpenStack clouds involved.

The goal of this spec is a solution to provide network connectivity between two or more OpenStack deployments or regions, respecting these requirements.

Use cases

Example 1

User Foo has credentials for OpenStack cloud A and OpenStack cloud B, one Router X in OpenStack A, one Router Y in OpenStack B, both using a distinct subnet in the private IPv4 address space.

User Foo would like to consume via an API a service that will result in establishing IP connectivity between Router A and Router B.

Example 2

Same as example 1, but this time L2 connectivity is desired between Network X in OpenStack A, and Network Y in OpenStack B.

Example 3

Same as example 1 or 2, but with 3 OpenStack deployments involved.

Proposed Change

Overview

The proposition consists in introducing a service plugin and a corresponding API extension involving a new ‘interconnection’ resource. The ‘interconnection’ resource on a Neutron instance will refer to both a local resource (e.g. Router A) and a remote resource (OpenStack B:Router B), and will have the semantic that connectivity is desired between the two.

This resource will be part of two sorts of API calls:

calls between the user having the connectivity need and each Neutron instance involved; the role of these calls is to let the need for connectivity be known by all Neutron instances
API calls between a Neutron instance and another Neutron instance; the role of these calls is to:
- let each Neutron instance validate the consistency between the “interconnection” resources defined locally and the “interconnection” resources made in other Neutron instances
- after this validation, let two Neutron instances identify the mechanism to use and the per-interconnection parameters (depending on the mechanism; could be a pair of VLANs on a interconnection box, a BGPVPN RT identifiers, VXLAN ids, etc.)

                               .-------------.
                               | tenant user |
                               '----+---+----'
                                    |   |
   .--------------------------------'   '--------------------.
   | A1. create "interconnection"                            |
   |     between local net X,                                |
   |     and "Neutron B: net Y"                              |
   |                                          A2. create "interconnection"
   |                                              between local net Y,
   |                                              and "Neutron A: net X"
   |                                                         |
   |                                                         |
   |                                                         |
   |                                                         |
   V                         B1. check symmetric inter.      V
.-------------------------.                      (fail) .--------------------.
|                         +---------------------------> |                    |
|   Neutron A             |                             |   Neutron B        |
|                         | <---------------------------+                    |
|                         |  B2. check symmetric inter. |                    |
|                         |                       (ok!) |                    |
|                         |                             |                    |
|                         +---------------------------> |                    |
|                         | <---------------------------+                    |
'-------------------------'  B3. exchange info to build '--------------------'
                   net X        interconnection           net Y
                     |                                      |
                     |                                      |
                   --+                                      +--
                     |      C. interconnection is built     |
                     +--    - - - - - - - - - - - - - -   --+
                     |                                      |
                     |                                      |

Note that the order between A1/A2/B1/B2 can vary, but the result is unchanged: at least one of the two Neutron instances will eventually confirm that the interconnection has been defined on both side symmetrically, and the interconnection setup phase will ultimately proceed on both sides (see Details on operations).

When more than two OpenStack deployments, or more than two OpenStack regions, are involved, these API calls will happen for each pair of region/deployments.

Base assumptions, trust model

The base assumption underlying the trust model in this proposal is that the end user requesting connectivity delegates trust to each OpenStack deployment to provide only the interconnection requested, i.e. to not create connectivity between resources unless requested.

To respect this contract each OpenStack deployment needs, by definition, a trust relationship with the other(s) OpenStack deployment(s) involved in these interconnections; practically speaking it cannot do better, when receiving a packet from another deployment, identified as intended for own of its local network A (VLAN, VXLAN ID, MPLS label, etc.) to trust that this identifier was pushed by the OpenStack deployment by mechanisms ultimately respecting the contract of these specifications.

Another aspect, obvious but better made explicit, is that the choice and definition of the network identifiers that will be used for an interconnection and to keep interconnections isolated from one another, are not controlled by consumers of this API. In this proposal these consumers do not and cannot write, or even read, these identifiers.

Note that only the API calls to the ‘interconnection’ resources at steps A1/A2 require write access to the “interconnection” resources by tenant users (but not to the attributes related to the network mechanism to use).

The calls at steps B1/B2/B3, only require read-only access to these resources; this can be achieved by introducing an “interconnection” role with read-only access to all “interconnection” resources, and having each OpenStack deployment having credentials for a user with this role in other OpenStack deployments.

With the above in mind, Keystone federation is not required for the calls at steps A1/A2, nor for the calls at step B1/B2. However, using Keystone Federation for the user(s) used at step B1/B2 will certainly be useful and will avoid requiring the management in each Neutron instance of the credentials to use to each other OpenStack.

Interconnection mechanisms

Although these specifications try to be agnostic to the network technique ultimately used to realize an interconnection, the assumption is made that for each ‘interconnection’, there is a technique common to the two OpenStack deployments involved.

The approach proposed is a simple approach where each OpenStack deployment determines based on a configuration file, which technique to use when establishing an interconnection with a given OpenStack deployment.

Note that only the parameters that do not differ between two interconnection would sit in a configuration file. The API exchange between two Neutron instances is used to exchange parameters that are specific to each interconnection.

Example interconnection techniques

The following techniques can be considered for realizing interconnections:

BGP-based VPNs: already supported via the Neutron BGPVPN Interconnection service (see networking-bgpvpn), it allows to create L2 (with EVPN) or L3 connectivity (with BGP/MPLS IP VPNs)
VXLAN stitching with networking-l2gw (details remain to be investigated)
VLAN stitching (details remain to be investigated)

It is expected that the first implementation will provide at least support for the BGPVPN interconnection technique, which is already supported across an interesting range of Neutron backends (Neutron reference drivers, OpenDaylight, OpenContrail, Nuage Networks), and hence would allow this API extension to be implemented on day one with a support for all these controllers without any further per-controller driver development.

Details on operations

When an “interconnection” resource is created, the Neutron instance will check that the symmetric interconnection exists on the remote Neutron instance designated in the interconnection, and will not proceed further until this becomes true.

This check is what establishes the end-to-end trust, that on both sides the connectivity has been requested.

Once a Neutron instance determines that an interconnection is symmetrically defined, further exchanges happen to determine the network parameters to use to realize the interconnection:

the Neutron (e.g. Neutron B) that just confirmed the symmetricity allocates the required network identifiers, and asks the remote Neutron instance (A) to refresh its state
Neutron A refreshes its state: checks symmetricity again (which now succeeds) retrieves at the same time the network identifiers allocated by B, and asks Neutron B to refresh
Neutron B refreshes again, this time retrieves at the same time the network identifiers allocated by A

In the above, “ask the remote Neutron instance to refresh its state” is done with a PUT on a specific refresh action on the interconnection.

Interconnection lifecycle

In the previous section, it is implicit that an interconnection is along its life in different states before it is ultimately realized. When an interconnection resource is deleted on one side, the other side need also to ultimately be able to update its own state (if only for cleanup purpose or giving proper feedback to end users).

Additionally, the interaction between a Neutron instance with another Neutron instance needs to happen out of the API call processing path, because it is not desirable that the success of a local API call would depend on the success of an operation with an external component which possibly would not be available at the moment.

For all these reasons, a state machine will be introduced to handle the lifecycle of an interconnection resource, with triggered and periodic operation being done out-of-band of the API calls, to handle the operations for each state.

Exposing this state in the API will allow:

end users to have feedback on how close they are to having something working
each Neutron instance to possibly identify that the remote state is inconsistent with the local state

State machine summary:

TO_VALIDATE: interconnection resource has been created, but the existence of the symmetric interconnection hasn’t been validated yet
VALIDATED: the existence of the symmetric interconnection has been validated and local interconnection parameters have been allocated (remote parameters are still unknown)
ACTIVE: both local parameters and remote parameters are known, interconnection has been setup, it should work
TEARDOWN: local action taken to delete this interconnection, action is being taken to have the remote state get in sync
(DELETED): implicit state corresponding to the resource not existing anymore

REST API Impact

The proposal is to introduce an API extension interconnection, exposing a new interconnection resource.

Interconnection resource

The new interconnection API resource will be introduced under the interconnection API prefix, and having the following attributes:

Attribute Name	Type	Access	Comment
id	uuid	RO
project_id	uuid	RO
type	enum	RO	`router`, `network_l2`, `network_l3`
state	enum	RO will be updated by Neutron along the life of the resource	see states in Interconnection lifecycle
name	string	RW
local_resource_id	uuid	RO	router or network UUID
remote_resource_id	uuid	RO	router or network UUID
remote_keystone	string	RO	AUTH_URL of remote keystone
remote_region	string	RO	region in remote keystone
remote_interconnection_id	uuid	RO will be updated by Neutron along the life of the resource	uuid of remote interconnection
local_parameters	dict
remote_parameters	dict

This resource will be used with typical CRUD operations:

POST /v2.0/interconnection/interconnections
GET /v2.0/interconnection/interconnections
GET /v2.0/interconnection/interconnections/<uuid>
PUT /v2.0/interconnection/interconnections/<uuid>
DELETE /v2.0/interconnection/interconnections/<uuid>

Additionally, an additional REST operation is introduced to trigger a refresh action on an interconnection resource:

PUT /v2.0/interconnection/interconnections/<uuid>/refresh

When this action is triggered the neutron instance on which the call is made will try to retrieve (GET) an interconnection resource on the remote neutron instance having same type as the local resource, as local_resource_id the local resource remote_resource_id, and as remote_resource_id the local resource local_resource_id. Depending on the current local state, and depending on success or failure to find such a resource, the local state machine will transition.

Example

This shows an example of API exchanges for a Neutron-Neutron interconnection between two Networks.

API Call A1, from tenant user to Neutron A:

POST /v2.0/interconnection/interconnections
     {'interconnection': {
         'type': 'network_l3',
         'local_resource_id': <uuid of network X>,
         'remote_keystone': 'http//<keystone-B>/identity',
         'remote_region': 'RegionOne',
         'remote_resource_id': <uuid of network Y>
      }}

Response: 200 OK

{'interconnection': {
     'id': <uuid 1>,
     ...
 }}

API Call B1, from Neutron A to Neutron B:

GET /v2.0/interconnection/interconnections?local_resource_id=<uuid of network Y>&remote_resource_id=<uuid of network X>

Response: 404 Not Found

API Call A2, from tenant user to Neutron B:

POST /v2.0/interconnection/interconnections
     {'interconnection': {
         'type': 'network_l3',
         'local_resource_id': <uuid of network Y>,
         'remote_keystone': 'http//<keystone-A>/identity',
         'remote_region': 'RegionOne',
         'remote_resource_id': <uuid of network X>
      }}

Response: 200 OK

{'interconnection': {
     'id': <uuid 2>,
     ...
 }}

API Call B2, from Neutron B to Neutron A:

GET /v2.0/interconnection/interconnections/local_resource_id=<uuid of network X>&remote_resource_id=<uuid of network Y>

Response: 200 OK

{'interconnection': {
     'id': <uuid 1>,
     ...,
     'local_parameters': {}
 }}

API Call B3’ from Neutron B to Neutron A:

PUT /v2.0/interconnection/interconnections/<uuid 1>/refresh

Response: 200 OK

API Call B3’’, from Neutron A to Neutron B

GET /v2.0/interconnection/interconnections/local_resource_id=<uuid of network Y>&remote_resource_id=<uuid of network X>

Response: 200 OK

{'interconnection': {
     'id': <uuid 2>,
     ...,
     'local_parameters': {
         'foo': '42'
     }
 }}

API Call B3’’’, from Neutron A to Neutron B

PUT /v2.0/interconnection/interconnections/<uuid 2>/refresh

Response: 200 OK

API Call B3’’’’, from Neutron B to Neutron A

GET /v2.0/interconnection/interconnections/<uuid 1>

Response: 200 OK

{'interconnection': {
     'id': <uuid 1>,
     ...,
     'remote_interconnection_id': <uuid 2>,
     'remote_parameters': {
         'foo': '42'
     },
     'local_parameters': {
         'bar': '43'
     }
 }}

Command Line Client Impact

python-neutronclient will be updated to introduce an OSC extension to create/remove/update/delete interconnection resources.

Client Libraries Impact

python-neutronclient and openstacksdk will need to be updated to support create/remove/update/delete operations on interconnection resources.

Credits

Przemyslaw Jasek contributed to exploring ideas that lead to this proposal during a six-months internship at Orange.

References

OpenStack Summit Sydney, lightning talk https://www.openstack.org/videos/sydney-2017/neutron-neutron-interconnections

Introduce distributed locks to ipam module

Wed, 20 Nov 2024 00:00:00

https://blueprints.launchpad.net/neutron/+spec/introduce-distributed-locks-to-ipam RFE:https://bugs.launchpad.net/neutron/+bug/1836834

Introduce the OpenStack tooz distributed lock to the ipam module. In the scenario of large-scale port creation, avoid ip allocation exceeding the maximum retry limit failure and improve ip allocation efficiency.

Problem Description

Current port creation has a probability of failure.

When the virtual machines are created in batches, nova will call the neutron API to create the ports concurrently. Port creation will fail if there is an ip allocation conflict, see [1].

And bulk create port has the similiar problem, when multiple “create_port_bulk” APIs are called simultaneously on the same subnet, although this scenario is used less frequently.
When creating ports concurrently, the utilization ratio of neutron server CPU increases and the allocation efficiency decreases.

When an ip allocation conflict fails to submit a database, a DB ERROR exception is thrown. Create_port will catch the above exception and rest after retry_interval=0.1 and re-call create_port until it exceeds max_retries=10. When it exceeds max_retries=10 times, “Create_port” will fail. When concurrency is large and conflict intensifies, repeated call to create_port increases CPU’s burden and reduces allocation efficiency. Adjusting retry_interval and max_retries can only reduce the probability of problems, but can not solve them thoroughly.

Proposed Change

This solution implements a new ipam driver by introducing a distributed lock to completely solve the problem of ip address allocation conflict leading to failure.

For distributed locks we will use OpenStack tooz [2] , which supports many backend drivers, such as Zookeeper, Memcached, Redis, Mysql, etc., and it is an OpenStack native project. We will support the configuration of tooz backend drivers in neutron.conf, such as adding [tooz] configuration items.

The new IPAM allocate ip seqdiag.

Alternatives

We can modify the current ipam driver to introduce distributed locks to solve the above mentioned problems when creating a new ipam driver is not feasible.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

Operator can configure the backend driver for tooz using the [tooz] configuration block in neutron.conf.

[tooz]
# Tooz backend connection string.
backend_url = file://$state_path

# Number of seconds between heartbeats for distributed coordination.
heartbeat = 1.0

# Number of seconds to wait after failed reconnection to Tooz backend.
initial_reconnect_backoff = 0.1

# Maximum number of seconds between sequential reconnection retries to Tooz backend.
max_reconnect_backoff = 60.0

Operator can switch to our new ipam driver by setting “ipam_driver” in neutron.conf.

[default]
# Neutron IPAM (IP address management) driver to use. By default, the reference
# implementation of the Neutron IPAM driver is used. (string value)
ipam_driver = ipam_with_dlm

Performance Impact

When using rally to test concurrently to create vms or ports or the similar scenes.

The good aspects:

Solve the failure of creating vms or ports due to ip allocation conflicts, and improve the success rate.
Reduced average time to create vms or ports.
Create vms or ports with a smoother distribution of time.

Minor impact:

The minimum time to create vms or ports has increased slightly, and creating a port time in a non-concurrent scenario will also increase slightly.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: qinhaizhong
Other contributors:: zhouhenglc

Work Items

Create a new ipam driver.
Support for parsing [tooz] backend drivers, encapsulating distributed lock modules, and implementing distributed lock initialization, locking, unlocking, etc.
Make “create_port” to support the new ipam driver.
Make “bulk_create_port” to support the new ipam driver.
Documentation work.

Dependencies

None

Testing

Unit tests, functional tests.

Documentation Impact

None

References

ML2/OVN - Coexistence Support for OVN externally managed resources

Mon, 06 May 2024 00:00:00

https://bugs.launchpad.net/neutron/+bug/2027742

Currently two individual Neutron deployments using ML2/OVN are separate and may only communicate using normal provider networks. However OVN supports the feature OVN interconnect (OVN-IC), that allows multiple separate OVN deployments to be connected together. The interconnection is implemented using a normal overlay (just like the one between compute nodes) and can therefore be created easily in a large scale (something that is not necessarily the case for provider networks).

Nowadays the Neutron OVN db sync command is actively interfering with OVN interconnect by removing non-Neutron resources from the Northbound database. The goal of this spec is to remove this interference and allow operators to use OVN interconnect and other features externally managed by OVN. It is out of the scope of this spec to integrate OVN interconnect directly to Neutron (however this might be part of a future spec).

Background: OVN Interconnect use case

The OVN Interconnect allows multiple clusters to be interconnected at Layer 3 level. This can be useful for deployments with multiple Availability Zones (AZs) that needs to allow connectivity between workloads in separate zones.

In the case of layer 3 interconnection, the logical routers on each cluster / availability zone can be connected via transit overlay networks. The transit network is an abstract representation of the interconnect layer, and the element responsible for this intercluster visibility is the Transit Switch (TS) . These interconnection switches are created in a global database and replicated to each AZ via OVN-IC daemon. So, basically, a logical router in one AZ is connected to a Logical router in another AZ via Transit switch. More details can be found in the ovn-architecture manpage [1].

The reference design using two separated OVN clusters (north and south) is described below:

+---------------------------------------------------------------------+
|OVN north cluster                                                    |
|                        +------------------------------------------+ |
|                        |OVN central node                          | |
| +---------------+      |  +-------------+        +-------------+  | |
| |network node   |      |  |ovsdb-server |        |ovsdb-server |  | |
| |ovn-controller +---------+southbound   |        |northbound   |  | |
| |               |      |  |             |        |             |  | |
| +---------------+      |  |             |        |             |  | |
|                        |  |             |        |             |  | |
| +---------------+      |  |             |        |             |  | |
| |compute node   |      |  |             |        |             |  | |
| |ovn-controller +---------+             |        |             |  | |
| |               |      |  |             |        |             |  | |
| +---------------+      |  +------+------+        +-------+-----+  | |
|                        |         |                       |        | |
|                        |         +---+---------------+---+        | |
|                        |             |               |            | |
|                        |  +----------+-----+   +-----+---------+  | |
|                        |  |ovn-ic          |   |ovn-northd     |  | |
|                        |  |                |   |               |  | |
|                        |  |                |   |               |  | |
|                        |  +-------+--------+   +---------------+  | |
|                        |          |                               | |
|                        +----------|-------------------------------+ |
|                                   |                                 |
+-----------------------------------|---------------------------------+
                                    |
            +-----------------------+---------------------+
            |Global DB - Transit switches                 |
            |ovsdb-server: northbound IC, southbound IC   |
            |                                             |
            +-----------------------+---------------------+
                                    |
+-----------------------------------|---------------------------------+
|OVN south cluster                  |                                 |
|                        +----------|-------------------------------+ |
|                        |          |                               | |
|                        |  +-------+--------+   +---------------+  | |
|                        |  |ovn-ic          |   |ovn-northd     |  | |
|                        |  |                |   |               |  | |
|                        |  |                |   |               |  | |
|                        |  +----------+-----|   +-----+---------+  | |
|                        |             |               |            | |
|                        |         +---+---------------+---+        | |
|                        |         |                       |        | |
| +---------------+      |  +------+------+        +-------+-----+  | |
| |network nodeer |      |  |ovsdb-server |        |ovsdb-server |  | |
| |ovn-controller +---------+southbound   |        |northbound   |  | |
| |               |      |  |             |        |             |  | |
| +---------------+      |  |             |        |             |  | |
|                        |  |             |        |             |  | |
| +---------------+      |  |             |        |             |  | |
| |compute node   |      |  |             |        |             |  | |
| |ovn-controller +---------+             |        |             |  | |
| |               |      |  |             |        |             |  | |
| +---------------+      |  +-------------+        +-------------+  | |
|                        |OVN central node                          | |
|                        +------------------------------------------+ |
+---------------------------------------------------------------------+

The following example will outline how OVN Interconnect works within the scope of OVN. It therefore follows the naming of OVN and not that of Neutron (should the two disagree).

Example setup:

+-------------------------------------------------------------+
|                     Logical_Switch                          |
|                     ts1                                     |
+----------+----------------------------------+---------------+
           |                                  |
   +-------+-----------+              +-------+-----------+
   |Logical_Switch_Port|              |Logical_Switch_Port|
   |lsp_ts1_lr1        |              |lsp_ts1_lr2        |
   +-------+-----------+              +-------+-----------+
           |                                  |
   +-------+------------+             +-------+------------+
   |Logical_Router_Port |             |Logical_Router_Port |
   |lrp_lr1_ts1         |             |lrp_lr2_ts1         |
   |172.24.0.10/24      |             |172.24.0.20/24      |
   +-------+------------+             +-------+------------+
           |                                  |
   +-------+------------+             +-------+------------+
   |Logical_Router      |             |Logical_Router      |
   |lr1                 |             |lr2                 |
   +-------+------------+             +-------+------------+
           |                                  |
   +-------+------------+             +-------+------------+
   |Logical_Router_Port |             |Logical_Router_Port |
   |lrp_lr1_ls1         |             |lrp_lr2_ls2         |
   |192.168.0.1/24      |             |192.168.1.1/24      |
   +-------+------------+             +-------+------------+
           |                                  |
   +-------+------------+             +-------+------------+
   |Logical_Switch_Port |             |Logical_Switch_Port |
   |lsp_ls1_lr1         |             |lsp_ls2_lr2         |
   +-------+------------+             +-------+------------+
           |                                  |
   +-------+------------+             +-------+------------+
   |Logical_Switch      |             |Logical_Switch      |
   |ls1                 |             |ls2                 |
   +-------+------------+             +-------+------------+
           |                                  |
   +-------+------------+             +-------+------------+
   |Logical_Switch_Port |             |Logical_Switch_Port |
   |lsp_ls1_vm1         |             |lsp_ls2_vm2         |
   +--------------------+             +--------------------+

The example above is a logical representation of the elements involved in the interconnection process. On each side we have an OVN cluster/AZ with its local managed resources: LSP for VMs, LS, LSP connecting the VM to the router and the LR. What does OVN interconnect add to a standard topology to make this work? A connection between the Tenant logical router and a Transit Switch.

The global database of the OVN IC dynamically replicates the TS between all members of the interconnect domain (clusters/AZs), and what needs to be done is basically to add this dynamically created TS in the OVN Northbound database with the Tenant logical router.

Ownership of resources

With the usage of OVN Interconnect Neutron is no longer the only owner of resources in each OVN deployment.

We therefore need to first define which component owns which kind of resources. The resources will be listed below for the left OVN deployment from the example above.

Resources owned by Neutron

lr1
lrp_lr1_ls1
lsp_ls1_lr1
ls1
lsp_ls1_vm1

All of these represent the: * Tenant Network (ls1) * Port of VMs or other things (lsp_ls1_vm1) * Router of the Tenant (lr1, lrp_lr1_ls1, lsp_ls1_lr1)

Resources owned by OVN-IC

ts1
lsp_ts1_lr2 (created in the left OVN deployment)
potentially Logical_Router_Static_Routes attached to lr1 (if route learning is enabled)

The resources owned by OVN-IC are dynamically created and removed by the OVN-IC daemon when the process synchronizes the Northbound database between interconnect domain elements.

resources owned by the operator

lsp_ts1_lr1
lrp_lr1_ts1

The resources to connect the Transit Switch to the router of the user need to be created by the operator (manually or with some kind of automation). Managing these resources in Neutron is out of scope of this spec.

Problem Description

The above setup can already be created by an operator. However the neutron-ovn-db-sync-util tool will remove the resources owned by OVN-IC and the operator, as Neutron does not know about them.

Example of resources created by OVN-IC:

Logical_Switch other_config:interconn-ts with any value
Logical_Router_Static_Route external_ids:ic-learned-route with any value
Logical_Switch_Port type is set to remote

These fields are automatically set by OVN-IC, so they do not have a Neutron key in the external_ids or other_config registers.

Example of resources owned by the operator:

Logical_Switch_Port external_ids or other_config with any value
Logical_Router_Port external_ids or other_config with any value

For resources created by operators such predefined options for external_ids or other_config do not exist.

Proposed Change

To solve the problem described above, the proposal is to introduce a new filter rule to check for the Neutron key during the neutron-ovn-db-sync-util method and not remove resources externally managed by OVN.

This implementation is being named as to coexistence support for OVN externally managed resources because it is out of scope any type of OVN externally managed resources integration as part of Neutron. The proposal of this implementation is the creation of filters in the checking of the resources created by Neutron, and it is the basis for any future implementation that intends to integrate the OVN interconnect or other OVN features to Neutron.

To implement the coexistence support the neutron-ovn-db-sync-util tool only needs to check the resources managed by Neutron. The main idea of this proposal is described below.

For resources created by Neutron the proposed solution implements the support by checking the specific Neutron signature on these resources. Neutron creates resources in the OVN NB database with the neutron: key in the external_ids register:

Logical_Switch external_ids:”neutron:” with any value

_uuid               : 5bf82c8e-fa49-4b46-bc4f-737311359f44
acls                : []
copp                : []
dns_records         : []
external_ids        : {"neutron:availability_zone_hints"="",
                       "neutron:mtu"="1442",
                       "neutron:network_name"=self_network_az1_tenant1,
                       "neutron:revision_number"="2"}
forwarding_groups   : []
load_balancer       : []
load_balancer_group : []
name                : neutron-2979fcc1-9540-4012-88a2-5f83738b5b6f
other_config        : {mcast_flood_unregistered="false", mcast_snoop="false",
                       vlan-passthru="false"}
ports               : [8e128001-5d9a-41eb-a84b-052dc28a74bc,
                       98555f1c-1a12-4703-8dcb-67f44900f6b8,
                       abeca18e-5706-4a2d-871d-5514ba20f554,
                       ba0e5d5e-5571-4f8c-b15c-dfa5115ba61c]
qos_rules           : []

Logical_Switch_Port, Logical_Router, Logical_Router_Port, etc. external_ids:”neutron:” with any value

These neutron:… keys in the external_ids are automatically set by Neutron, so we can rely on them being there. We need to ensure that all methods called by neutron-ovn-db-sync-util check the Neutron signature in the external_ids, and filter/ignore all the other externally managed resources when synchronizing between Neutron and the OVN NB database.

DB Impact

None

Rest API Changes

None

OVN driver changes

Update neutron-ovn-db-sync-util as described above.

Out of Scope

Integrating ovn-interconnect or other OVN externally managed feature into Neutron in any way;

Implementation

Assignee(s)

Primary assignees: Felix Huettner <felix.huettner@mail.schwarz> Roberto Bartzen Acosta <rbartzen@gmail.com>

Work Items

Add exclusion to neutron-ovn-db-sync-util
Implement relevant unit and functional tests using the existing facilities in Neutron.
Write documentation.

Documentation Impact

Administrator Documentation

Administrator documentation will need be included to describe to operators how to use the OVN interconnect when building their interconnected OpenStack clusters.

Testing

Unit/functional tests [2].

References

Apic ML2 driver enhancements

Mon, 06 May 2024 00:00:00

https://blueprints.launchpad.net/neutron/+spec/apic-driver-enhancements

This blueprint keeps track of some improvements needed by the APIC ML2 Driver. Specifically, the target enhancements will be:

Refactoring of the apic-manager: Making it more robust and it’s operations idempotent, being also able to “fix” any incoherence that may happen between the APIC and the Neutron world;

Name mapping: Defining configurable naming strategy for APIC. This allows the end-user to see names instead of uuids for APIC resources in APIC GUI;

External Gateway: Enable router external gateway feature for the APIC plugin;

Dynamic Topology Discovery: Enable the ability for the APIC plugin to automatically discover the physical topology (eg. compute host attached interfaces). This allows for less manual configuration from the user.

Support for VPC/LACP (bonding): Enable the use of openstack servers connected in a redundant topology with VPC/LACP (port bonding).

Problem description

This blueprint is targeting a number of small enhancements to solve different problems.

High Availability: Current APIC ML2 driver does not support a High Availability and Synchronization strategy for APIC, making it hard to recover from failures of the Neutron server. Also, the driver doesn’t leverage APIC management address redundancy/fallback mechanism, which translates into plugin failures every time the single APIC controller address in it’s configuration is not reachable.

Name Mapping: At the current state, creating objects in the neutron model is translated into an APIC model always named by the object UUID. A alternative name mapping strategy should be provided to the user.

L3 External Gateway Support: Today, the APIC L3 driver doesn’t support the external gateway extension for routers.

Dynamic Topology Discovery: The APIC drivers discovers the topology by leveraging some static Neutron configuration. Ideally we want to be able to automatically discover the topology by the use of a new agent.

Support for VPC/LACP (bonding): The current APIC driver does not support VPC/LACP or bonding of ports. This is a desirable configuration for HA deployments that we want to support.

Proposed change

High Availability: The changes for this point can be summarized into two major items:

First, modifying the current APIC driver configuration so that the user can specify as many APIC controllers as he likes. The APIC client will then rotate among these addresses every time there’s a connection failure.

To improve the manager’s operations idempotence and reliability, a transactional model is introduced which leverages the ability of the APIC to run operations on a Model’s full subtree transactionally.

Name Mapping: A new DB table will be added to keep track of the “APIC Name” used by a specific neutron object’s ID. Every time the driver is called, the mapper will be used to retrieve the actual names in the backend.

L3 External Gateway Support: Every time a port with device_owner “router:gateway” is created, the mechanism driver will run all the needed operations in order to guarantee external connectivity to the proper networks. Some configuration may be needed to describe the external router.

Dynamic Topology Discovery: An APIC specific agent is proposed. It will be installed as a 3rd party agent on all openstack nodes that need to be connected to the APIC fabric. The agent will listen to LLDP advertisements, and on discovering a topology change, it will update the APIC with the appropriate interface mapping.

Support for VPC/LACP (bonding): VPC pairs are defined in the plugin configuration. When an interface from a switch that is in a VPC pair is detected, the appropriate fabric configuration to enable that interface to work as a part of VPC/LACP bond is created on the APIC.

Alternatives

None

Data model impact

Following new models are created by this driver. Note that these are specific to this driver:

ApicName: Tracks the name mapping.
ApicKey: Tracks persistent config on APIC that needs to be in sync with driver
HostLink: Tracks connectivity of host links.

There are also changes on the existing model. Again, everything is APIC specific:

NetworkEPG: Deleted;
PortProfile: Deleted;
TenantContract: Becomes RouterContract, since the contract will now be considered by Router and no by Tenant.

A database migration is included to create the tables for these models.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

There is potentially a new configuration (for VPC pairs), if that feature is used. Also, if dynamic topology discovery is used, the agent should be added to all the compute node, while the service will be likely installed on the cloud controller.

Performance Impact

The transactional model for the APIC manager does a lot of work “locally”, and only 1 rest call at the end of each transaction. This will improve the responsiveness of the plugin.

Other deployer impact

The configuration of host connectivity is no longer required in the configuration file.

Developer impact

None

Implementation

Assignee(s)

Ivar Lazzaro (mmaleckk)

Mandeep Dhami (mandeep-dhami)

Work Items

APIC Mechanism driver package;
APIC L3 driver package;
Unit Tests Code (APIC Specific).

Dependencies

None

Testing

Unit Test

Documentation Impact

Configuration Reference guide will be updated.

References

Adding support for host-routes and dns_nameservers options for Nuage Plugin

Mon, 06 May 2024 00:00:00

https://blueprints.launchpad.net/neutron/+spec/dhcp-host-routes-and-dns-support-for-nuage-plugin

Adding support for host-routes and dns_nameservers options via DHCP options for the Nuage Plugin

Problem description

The current the Nuage Plugin does not support adding host routes or DNS nameservers via DHCP options for a Neutron subnet.

Proposed change

Currently the Nuage Plugin does not support Neutron’s adding host routes or DNS nameservers via DHCP options on a subnet.

The Nuage’s VSP supports this feature and the support needs to be added in the plugin code.

The following DHCP options will be supported :

DNS nameserver
Host routes

The proposed change is to support the creation of DNS nameserver and/or Host routes using the Neutron DNS and Host routes

For example:

neutron subnet-create test 192.168.10.0/24: –dns_nameservers list=true 8.8.4.4 8.8.8.8

This action will create a subnet name test with a CIDR 192.168.10.0/24. The nameservers 8.8.4.4 8.8.8.8 will added to this subnet, this translate to a subnet in the Nuage’s VSP subnet with the same nameservers

The CRUD operations will be supported by the Nuage’s VSP plugin for both DNS nameserver and Host routes.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

The modification are required on the plugin.py

The new method __create_port_gateway will be created.
This method will create a port of type network:dhcp for the current subnet and tenant
The __validate_create_subnet method will be modified to allow the host_routes key to be a valid options of the subnet dictionary
The update_subnet method will be added to the plugin
The _create_nuage_subnet method will be merged into the create_subnet method. This will newly updated method will also create a DHCP port using the method _create_port_gateway described above.

Assignee(s)

Franck Yelles

Primary assignee:: fyelles

Other contributors:

Work Items

Extension code in Nuage plugin
Nuage CI coverage addition
Nuage Unit tests additions, the following test units will be updated/added : * test_create_subnet_bad_hostroutes * test_update_subnet_adding_additional_host_routes_and_dns * test_create_subnet_with_one_host_route * test_create_subnet_with_two_host_routes * test_create_subnet_with_too_many_routes * test_update_subnet_route * test_update_subnet_route_to_None * test_update_subnet_route_with_too_many_entries * test_delete_subnet_with_route * test_delete_subnet_with_dns_and_route * test_validate_subnet_host_routes_exhausted * test_validate_subnet_dns_nameservers_exhausted

Dependencies

None

Testing

Unit Test coverage for the DHCP options within Nuage unit test Nuage CI will be modified to start supporting this extension tests

Documentation Impact

None

References

None

Enhancement to Neutron BGPaaS

Mon, 06 May 2024 00:00:00

https://bugs.launchpad.net/neutron/+bug/1921461

Enhancement to Neutron BGPaaS to directly support Neutron Routers and bgp-peering from such routers over internal & external Neutron Networks

Problem Description

There are very good foundation APIs in Neutron BGPaaS that brought in BGP service functionality (see [1]) into Neutron through Neutron Dynamic Routing. (see [3] )

Note

In this spec, Neutron-BGPaaS, BGPaaS and neutron-dynamic-routing/bgp are used as synonymns. These refers to “Neutron Dynamic Routing using BGPaaS” function already available in Neutron. Similarly, ‘ISP network’ and ‘Neutron External Network’ are used as synonymns.

In a Telco Service Provider environment, VNFs (Virtual Network Functions [4]) & CNFs (Containerized-Network-Functions) are run that provides simple-to-fairly complex services to subscribers. These VNFs use many Non-Neutron IP-Addresses which need to be exposed to subscribers. The subscribers access these services through these Non-Neutron IP-Addresses, referred as ‘Service-Address’. It can belong to either IPv4 (or) IPv6 address-families.

Service-Address are wholly managed inside a VNF transparent to Neutron. But these Service-Address are reachable on the dataplane by subscribers, through VNICs (or Ports) holding Neutron-managed Addresses. In short, the Neutron managed primary-addresses on VNFs VNICs act as NextHops to the Service-Address of VNF.

In order to enable L3 reachability to these Service-Addresses from an ISP-network (or) make the Service-addresses reachable from another neutron-tenant-network entity, these Service-Addresses need to be brought into a Neutron Router’s Routing Table.

The number of VNFs are high and hence the number of service addresses to be exposed are higher. “BGP between the VNFs and their hosting Neutron Routers” provides a very elegant and scalable automated approach for a control-plane driven route-learning of Service-Address by the Neutron Router from the VNFs. This enables L3 reachability of such Service-Addresses from other neutron-tenant-network-VNFs that is attached to the same Neutron Router.

These service addresses also need to be advertised to the ISP-PE-Router. This will enable subscribers present on ISP-network be able to use the services exported by applications in the VNFs.

The existing Neutron Dynamic BGPaaS falls a little short of supporting the above requirements due to below reasons:

Existing BGPaaS lacks the functionality (both API and implementations) to do BGP Peering over Neutron Internal Networks. The addition of service-addresses to Neutron Router’s Routing Table requires BGP control plane to bring them in via BGP peering (see [2]) towards the VNFs over Neutron Tenant Networks from Neutron Router. It also requires BGP speaker to learn routes from the BGP peers. Current implementation advertises routes to peers but doesn’t learn from peers.
Existing Neutron BGPaaS supports BGP Peering only over Non-Neutron Networks. Please refer the current implementation and documentation as in [8] . Advertisment of service-addresses to the ISP-PE-Router requires BGP Peering over Neutron External Networks. This will enable the ISP-PE-Router to learn the Service-Addresses over the BGP control plane. Thereby, enabling L3-reachability for subscribers on the ISP-side to be able to take advantage of services offered by VNFs (and CNFs).
Exising BGPaaS doesn’t provide API/implementation to associate a BGP speaker to a router. It currently supports a’loose-association’of network to BGP speaker. This ‘loose-association’ is concept where the BGP speaker derives the routes of the Neutron Router through APIs and not due to direct connectivity of BGP Speaker to the Neutron Router. Association of BGP speaker to router enables access to all the neutron-router-interfaces ie. both internal/external interfaces. This will enable a single bgpspeaker associated with the Neutron Router, be able to peer simultaneously towards VNFs on neutron-tenant-network as well as the ISP-PE-Router on neutron-external-network.
It also doesn’t support BGP peer group feature. BGP Peer Group feature enables automatic-bgp-peering towards VNFs by Neutron Router whenever the VNFs are scaled-out in a VNF cluster or whenever the VNFs are scaled in from a VNF cluster. The automatic-bgp-peering refers to the fact that the current BGP Speaker will be enhanced to automatically accept peering-requests from VNF if the VNFs are within a pre-configured ip-address-range (aka listen-range) that openstack tenant can configure for the BGP speaker.
BGPaaS capability will have varying implementations from different Neutron backends. This makes it important to have some feedback ingrained in the Neutron BGPaaS API. It will allow various Neutron backends (including reference BGPaaS backend) to drive realization status inside Openstack Infra for the BGP resources exposed by Neutron BGPaaS. To provide that, the proposal in the spec is to add a new resource type ‘Association’ that can work with current BGPSpeaker object. More specifically, the plan is to support router_association, peer_association as resources under a BGPSpeaker object. A router_association resource will be used to create a binding between a BGPSpeaker and a Neutron Router, that will internally realizes BGP Control Plane over that Neutron Router. Similarly, a peer_association resource will be used to create a binding between a BGPSpeaker and a BGPPeer that will internally realize BGP Peering between the BGPSpeaker and the BGPPeer resource.

Proposed Change

To enable L3 reachability to the Service-Addresses from ISP-network or from another-neutron-tenant-network entity, the below usecases are introduced :

Enable direct BGP-Peering (see [2]_) from a Neutron Router towards the VNFs (or CNFs) on its connected networks. This provides L3 reachability to the service-addresses hosted on such VNFs, as such addresses will be installed in the Neutron Router’s routing-table by the BGPaaS control plane. The proposal is to provide facility to directly associate a BGP Speaker to a neutron-router, that will in turn enable the router to reach BGP-Control plane learned routes from the VNF.. A single router can be associated to a BGP speaker and the speaker will be running inside the L3 router namespace.
Enable BGP-Peering over Neutron External Networks from a Neutron Router to the ISP-PE-Router. This allows the ISP-PE-Router to learn the Service-Addresses over the BGP control plane from Neutron Router. Thereby, enabling L3-reachability for subscribers on the ISP-side to leverage services offered by VNFs (and CNFs).
Support BGP peer group feature This will allow to transparently scale bgp peering to a number of VNFs, whenever VNF clusters are scaled-out or scaled-in. It supports BGP peering to a group of remote neighbors that are configured using a range of IP addresses. This can also be used in k8s over openstack deployment where CNF scale-out/scale-in is a pure k8s operation. It would be undesirable to call neutron APIs in this case.

New resources (Network Association, Router Association, Peer Association) will be added as part of this spec. Association resources provides the below benefits

Enables provisioning of certain BGP control plane parameters specific to BGP-Speaker-to-BGP-Peer binding and also specific to BGP-Speaker-to-Router binding. For eg: some attributes (status, advertise-extra-routes etc) are useful when linked to associations rather than directly to parent resources (speaker, peer). A BGP Peer can be associated to different BGP Speakers, so having a Status field in bgp-peer won’t be beneficial. It is useful only when associated to Peer Association resource.
Enables the Neutron BGPaaS APIs to provide feedback on the realization status of BGP Control Plane towards Openstack Tenant. For eg: In this spec, a new Status field is planned to be introduced. This will allow a Neutron-BGPaaS backend driver to provide realization-status for different BGP resources inside Openstack Infra.

+-----------------------------------------------+
|                                               |
|                ISP- PE Router                 |---------|
|                                               |         |
+-----------------------------------------------+        EBGP
                        |                                 |
                        |                                 |
+-----------------------------------------------+         |
|                                               |---------|
|                Neutron Router                 |
|                                               |---------|
+-----------------------------------------------+         |
                      |                                   |
                      |                                 EBGP
+-----------------------------------------------+         |
|                 VNF Cluster                   |         |
|  +-----------------+   +-----------------+    |---------|
|  |       VM1       |   |       VM2       |    |
|  +-----------------+   +-----------------+    |
+-----------------------------------------------+

To advertise the multiple service addresses hosted by VNFs to ISP-PE routers, 2 BGP sessions are created. One BGP session created directly towards the VNFs from a Neutron Router hosting the neutron-tenant-networks of the VNF. And the next BGP-Peering towards the ISP-PE-Routers from Neutron Router directly over the Neutron External Networks.

Proposal is to enhance existing BGPaas (see [6]), allow neutron router to be associated to a BGP Speaker and allow BGP Speaker to peer with both the internal-Networks and External-Networks present on that Neutron Router. This will be implemented using enhancements to the neutron-service and neutron-dynamic-routing.

Implementation approaches

Option 1

A BGP speaker is spawned inside router namespace, when a neutron-router is associated to BGP Speaker. BGP speaker will be running inside the L3 router namespace which enables access to all the neutron-router-interfaces ie. both internal/external interfaces. BGP functionality provided by OS-Ken will be reused to excite BGP speaker functionality to run only within the neutron router namespace.

BGP Service Plugin will be enhanced to accept the request to associate a BGP speaker to the router. BGP L3 agent extension will implement the L3 agent side of BGP enhancements. It will be responsible for realizing bgpspeaker inside the router-namespace and now bgpspeaker can peer over networks on the router from inside the router-namespace. BGP speaker can listen on 0.0.0.0:179 which enables it to listen on all the neutron-router-interfaces and can peer with the neighbour based on the configuration.

+---------------+              +-----------------------------------+
|DHCP namespace |              |Router namespace                   |
|     qdhcp     |              |     qrouter                       |
|   +------+    |              |  +---+   +---+   +-----------+    |
|   | tap  |    |              |  | qr|   | qg|   |BGP Speaker|    |
|   +------+    |              |  +---+   +---+   +-----------+    |
|       |       |              |    |       |                      |
+---------------+              +-----------------------------------+
        |                           |       |
+--------------------------------------------------------------+
|       |                           |       |                  |
|     +--------+               +-------+  +-------+            |
|     |Port tap|               |Port qr|  |Port qg|            |
|     +--------+               +-------+  +-------+            |
|                      br-int                                  |
+--------------------------------------------------------------+

Option 2

In Option 1, since BGP speaker is spawned inside the router namespace, the speaker is tightly coupled with that router namespace. For a different BGP Speaker on a different router, wherein that router in turn is realized on the same network-node hosting the original router, a new instantiation of the BGP Speaker is required. This could potentially bring in scalability and performance problems with Option 1.

In Option 2, the proposal is to create and use VRF for BGP-Peering. VRF device (see [9]) can be used instead of namespace for BGP peering. VRF is a layer 3 master network device with its own associated routing table. It also provides added benefit of VRF any socket which allows a single process instance to efficiently provide service across all VRFs. So for N router associated BGP speakers, N vrfs will be created but a single BGP speaker will serve all of them.

+---------------+    +-----------------+    +-----------------+
|DHCP namespace |    |Router namespace |    |  VRF device     |     +-------+
|        qdhcp  |    |       qrouter   |    |                 |     |  BGP  |
|   +------+    |    |  +---+   +---+  |    |  +---+   +---+  |N---1|speaker|
|   | tap  |    |    |  | qr|   | qg|  |    |  | vr|   | vg|  |     +-------+
|   +------+    |    |  +---+   +---+  |    |  +---+   +---+  |
|       |       |    |    |       |    |    |    |       |    |
+-------------- +    +-----------------+    +-----------------+
|                 |       |       |              |       |
+-----------------------------------------------------------------+
|       |                 |       |              |       |        |
|     +--------+      +-------+  +-------+  +-------+  +-------+  |
|     |Port tap|      |Port qr|  |Port qg|  |Port vr|  |Port vg|  |
|     +--------+      +-------+  +-------+  +-------+  +-------+  |
|                                                                 |
|                      br-int                                     |
+-----------------------------------------------------------------+

Current L3Plugin and L3Agent will continue to provide Neutron-Router functionality. VRF will be realized by BGP-Dr-Agent and a new VRF is created when a neutron-router is associated to BGP Speaker. VRF will be used only for BGP peering and the learnt routes are installed inside router namespace.

Extra IPs has to be allocated from internal and external subnet pools for the vr, vg interfaces respectively. These IPs will be used for BGP peering and the allocation of new IPs can be considered as a short coming with this option.

Option 3

VRF device can provide most of the L3 functionalities provided by router namespace. In this option, the idea is to completely replace the namespace with VRF device. A new VRF will be created when a neutron-router is created.

Both L3-Routing as well as BGP can be realized through VRFs. Linux-VRF will provide neutron router functionality which includes dataplane L3-forwarding for east-west, north-south and NATing.

This provides the same advantages as in option 2 and eliminates its disadvantages. But the main concern with this approach is that it will make a huge deviation from the existing L3 implementation which will require a lot of effort and time. This also requires changes to upgrade mechanisms.

Note

The current plan is to implement Option 1 which is simpler and doesn’t deviate much from the existing implementation. Incase the implementation option 3 is going to be used, it will be only a gradual replacement (from namespace to vrf) with proper upgrade path.

HA-Capable Neutron Router with BGPaaS

When HA-Capable neutron router is associated to a BGPSpeaker, two BGPSpeaker instances will be run, one on active router namespace and other on the standby router namespace. Whenever the failover happens from Active to Standby Namespace, the BGPSpeaker on Standby will be able to peer with BGP-Peers and become an Active-BGP-Speaker managing the Router.

The implementation planned in this spec supports only Centralized Neutron Routers (both HA and non-HA) and not Distributed Routers.

REST API & Neutron client command Impact

Use-case a)

Associate neutron router to BGP Speaker API

POST /v2.0/bgp-speakers/<bgp-speaker-id>/router_associations

{
 "router_association":
     {
      "router_id": "c930d7f6-ceb7-40a0-8b81-a425dd994ccf",
      "advertise_extra_routes": True
     }
}

router_id represents the UUID of the neutron router to which the BGP speaker has to be associated.
advertise_extra_routes decides whether neutron extra routes on the neutron router will be redistributed to bgp-peers by the bgpspeaker. Default is True and so all neutron extra routes will be redistributed to every bgp-peer that is bound to the BGP Speaker. Turning OFF advertise_extra_routes will disable advertisement of neutron router’s extra-routes by the bgpspeaker.

Note

The enhancement will support a router to be associated to only a single BGP speaker. The association of router to speaker and network to speaker will be a mutually exclusive operation.

New neutron Client Command:

openstack bgp speaker router association create
              [--advertise-extra-routes]
              [--no-advertise-extra-routes]
              <bgpspeaker>
              <router>

Disassociate neutron router from BGP Speaker API

DELETE  /v2.0/bgp-speakers/<bgp-speaker-id>/router_associations/<router
    -association-uuid>

New neutron Client Command:

openstack bgp speaker router association delete
              <bgpspeaker>
              <router>

Note

Deleting a BGP SPeaker will not be permitted, if the speaker already has multiple associations on itself like peer-associations and router-associations. Similarly, deletion of a router that is asoociated to a BGP speaker is also not allowed.

Update BGP Speaker Router Association

PUT /v2.0/bgp-speakers/<bgp-speaker-id>/router_associations/<router
    -association-uuid>

{
 "router_association":
     {
      "router_id": "c930d7f6-ceb7-40a0-8b81-a425dd994ccf",
      "advertise_extra_routes": True
     }
}

advertise_extra_routes field can be set to True or False when updating the router association.

New neutron Client Command:

openstack bgp speaker router association update
              [--advertise-extra-routes]
              [--no-advertise-extra-routes]
              <bgpspeaker>
              <router>

List Router associations for a given BGP speaker

GET /v2.0/bgp-speakers/<bgp-speaker-id>/router_associations

{
 "router_associations": [
     {
      "router_id": "c930d7f6-ceb7-40a0-8b81-a425dd994ccf",
      "advertise_extra_routes": True
      "status": 'ACTIVE'
     },
     {
      "router_id": "a330d7f6-ceb7-40a0-8b81-a425dd994bbe",
      "advertise_extra_routes": True
      "status": 'ACTIVE'
     }
 ]
}

status attribute helps to know whether Neutron-BGPaaS backend software has realized the router association successfully on the openstack infrastructure. Status field can be either DOWN or ACTIVE.

Show details for a BGP speaker Router Association

GET /v2.0/bgp-speakers/<bgp-speaker-id>/router_associations/<router
    -association-id>

{
 "router_association":
     {
      "router_id": "c930d7f6-ceb7-40a0-8b81-a425dd994ccf",
      "advertise_extra_routes": True
      "status": 'ACTIVE'
     }
}

Create BGP speaker Peer Association

POST /v2.0/bgp-speakers/<bgp-speaker-id>/peer_associations

{
 "peer_association":
     {
      "peer_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf",
     }
}

peer_id represents the UUID of the BGP peer to which the BGP speaker has to be associated.

New neutron Client Command:

openstack bgp speaker peer association create
              <bgpspeaker>
              <peer>

Delete BGP speaker Peer Association

DELETE  /v2.0/bgp-speakers/<bgp-speaker-id>/peer_associations/<peer
    -association-uuid>

Note

Deleting a BGP speaker which has peer-associated will not be allowed.

New neutron Client Command:

openstack bgp speaker peer association delete
              <bgpspeaker>
              <peer>

List Peer associations for a given BGP speaker

GET /v2.0/bgp-speakers/<bgp-speaker-id>/peer_associations

{
 "peer_associations": [
     {
      "peer_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf",
      "status": 'ACTIVE'
     },
     {
      "peer_id": "a640d7f6-ceb7-40a0-8b81-a425dd994dde",
      "status": 'ACTIVE'
     }
 ]
}

status attribute helps to know whether Neutron-BGPaaS backend software has realized the peer association successfully on the openstack infrastructure. Status field can be either DOWN or ACTIVE.

Show details for a BGP speaker Peer Association

GET /v2.0/bgp-speakers/<bgp-speaker-id>/peer_associations/<peer
    -association-id>

{
 "peer_association":
     {
      "peer_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf",
      "status": 'ACTIVE'
     }
}

Show routes managed by BGP Speaker API

GET /v2.0/bgp-speakers/<bgp-speaker-id>/get_routes

{
     "routes": [
         {
              "cidr": "192.168.10.0/24",
              "nexthop": ["10.0.0.1"],
              "route-type": "local"
         },
         {
              "cidr": "192.168.11.0/24",
              "nexthop": ["10.0.0.5", "10.0.0.6"],
              "route-type": "peer"
         }
       ]
}

nexthop is a list which contains ip addresses that is going to be used to reach a certain destination cidr.
cidr represents the CIDR prefix.
route-type can be local or peer based on whether the routes are local or learnt from the peer respectively.

The routes can be obtained from the backend. For example, in case of os_ken backend, rib_get method (see [10]) can be used to obtain the routes.

New Neutron Client Command:

openstack bgp speaker list routes <bgpspeaker>

Use-case b)

Create BGP Peer Group API

POST /v2.0/bgp-peer-groups/
{
   "bgp_peer_groups":{
      "name":"bgppeergroup1",
      "project_id":"",
      "remote_asn":"4566",
      "next_hop_self" : True,
      "update_source_ip": "10.20.1.5"
      "auth_type": "md5",
      "password": "<passwd>"
    }
}

remote_asn represents the remote AS number of a BGP peer group
next_hop_self decides whether to modify the nexthop attribute to its own during BGP advertisement.
update_source_ip forces BGP to use the IP address specified while talking to a BGP neighbor.
auth_type determines the authentication algorithm. Supported algorithms are none and md5, none by default.
password represents the authentication password for the specified authentication type.

New neutron Client Commands:

openstack bgp peer group create
              [--next-hop-self]
              [--no-next-hop-self]
              [--auth-type <auth-type>]
              [--password <password>]
              [--update_source-ip <ip-address>]
              --remote-asn <asn-number>
              <bgp-peer-group-name>

Delete BGP Peer Group API

DELETE /v2.0/bgp-peer-groups/<bgp-peer-group-id>

Deletion of peer-group will succeed only if there are no BGP-Peers referring to this peer-group.

New Neutron Client Commands:

openstack bgp peer group delete <bgp-peer-group-id>

Create a bgp-peer using a pre-created peer-group API

POST /v2.0/bgp-peers/
{
   "bgp_peer":{
      "peer_group_id":"a930d7f6-ceb7-40a0-8b81-a425dd994ccf",
      "name":"bgppeer1",
      "listen_range": "10.2.1.0/24"
      "listen_limit": 10
      "auth_type": "md5",
      "password": "<passwd>"
      }
}

peer_group_id represents the UUID of the BGP peer group
listen_range defines a prefix range to be associated with the peer group.
listen_limit defines the maximum number of BGP peers that can be created automatically.

These are new fields to the existing BGP peer API and are optional parameters. A BGP-Peer-Group can be re-used by multiple BGP Peers.

Changed Neutron Client Commands ((see [7]):

openstack bgp peer create
              [--listen-range <listen-network-range>]
              [--listen-limit <number>]
              [--auth-type <auth-type>]
              [--password <password>]
              [--peer-group <peer-group-id>]
              <bgp-peer>

openstack bgp speaker add peer mybgpspeaker bgpppeer1

peer-group attribute is introduced in the already existing bgp peer neutron client command. It specifies the name/UUID of the BGP peer group that has to be used by BGP Peer.

Create BGP peers with update-source and next-hop-self parameters API

As part of the enhancement, new attributes are introduced for the BGP peer model. These are next_hop_self and update_source_ip. next_hop_self decides whether to modify the nexthop attribute to its own during BGP advertisement. And update_source_ip forces BGP to use the IP address specified while talking to a BGP neighbor.

POST /v2.0/bgp-peers/
{
   "bgp_peer":{
      "auth_type":"none",
      "remote_as":"1001",
      "name":"bgppeer1",
      "peer_ip":"10.0.0.3",
      "next_hop_self":True,
      "update_source_ip": "10.2.0.15"
   }
}

Changed Neutron Client Commands:

openstack bgp peer create
              [--peer-ip <peer-ip>]
              [--next-hop-self True]
              [--auth-type <auth-type>]
              [--update-source-ip <ip>]
              --remote-as <as>
              <bgppeer>

update-source-ip and next-hop-self are introduced in the existing bgp peer neutron client command.

Backend for reference implementation

os-ken can be used to handle BGP (see [5] ). os-ken is able to start bgp speaker using BGPSpeaker class and it is possible to receive event notifications (BGP_BEST_PATH_CHANGED) which can be then used to populate routes inside the namespace.

Data Model Impact

bgp_peers table will be updated to incorporate the new fields.

Field	Type	Description
update_source_ip next_hop_self	String Boolean	Forces BGP to use the IP address specified while talking to a BGP neighbor Setting to True enables all published routes to carry BGPSpeaker IP address over the BGP Control Plane towards this BGP Peer.

Field

Type

Description

update_source_ip

next_hop_self

String

Boolean

Forces BGP to use the IP address specified while talking to a BGP neighbor Setting to True enables all published routes to carry BGPSpeaker IP address over the BGP Control Plane towards this BGP Peer.

New table bgp_speaker_router_bindings will be created to manage the speaker to router association.

Field	Type	Description
id bgp_speaker_id router_id advertise_extra_routes status	uuid-str uuid-str uuid-str Boolean String	UUID of the BGP speaker router association UUID of the BGP speaker UUID of the router to which the BGP speaker has to be associated Decides whether neutron extra routes on the neutron router will be advertised to bgp-peers Shows whether Neutron-BGPaaS backend software has realized the router association successfully on the openstack infrastructure. Status field can be either DOWN or ACTIVE.

Field

Type

Description

bgp_speaker_id router_id

advertise_extra_routes

status

uuid-str

uuid-str uuid-str

Boolean

String

UUID of the BGP speaker router association UUID of the BGP speaker UUID of the router to which the BGP speaker has to be associated Decides whether neutron extra routes on the neutron router will be advertised to bgp-peers Shows whether Neutron-BGPaaS backend software has realized the router association successfully on the openstack infrastructure. Status field can be either DOWN or ACTIVE.

Similarly, id and status field will be introduced in the existing bgp_speaker_peer_bindings tables.

New table bgp_peer_group will be created to manage bgp peer group.

Field	Type	Description
id name project_id remote_asn update_source_ip next_hop_self	uuid-str String String Integer String Boolean	UUID of the BGP peer group Human readable name of the BGP peer group Owner of the BGP peer group Remote AS number of a BGP peer group Forces BGP to use the IP address specified while talking to a BGP neighbor decides whether to modify the nexthop attribute to its own during BGP advertisement

Field

Type

Description

id name project_id remote_asn update_source_ip

next_hop_self

uuid-str String String Integer String

Boolean

UUID of the BGP peer group Human readable name of the BGP peer group Owner of the BGP peer group Remote AS number of a BGP peer group Forces BGP to use the IP address specified while talking to a BGP neighbor decides whether to modify the nexthop attribute to its own during BGP advertisement

Security Impact

There can be security impacts with the introuduction of peer groups which support automatic peering requests from peers within a pre-configured listen-range. This is of major concern if we support this option for external networks.

And this problem can be resolved using authentication which will be supported in peer-groups.

Performance Impact

There can be performance impact based on the selection of the implementation approaches.

Accurate testing is however needed to understand the overhead.

Implementation

Assignee(s)

Primary assignee:

Manu B <manubk2020@gmail.com> (IRC: manubk)

Work Items

REST API update.
BGP plugin, L3 agent extension to handle BGP:
- Associate router to bgp speaker
- Disassociate router from bgp speaker.
- HA support for BGP speaker
- Support new peer group APIs
- New attributes for BGP peer APIs
CLI update.
Documentation.
Tests and CI related changes.

Testing

Unit Test
Functional test
API test Perhaps this is something that can be easily tested end-to-end with fullstack tests. Need more investigation.

Documentation Impact

User Documentation

New API and changes to legacy APIs like neutron-router and neutron-bgpaas must be documented.

References

Active-active L3 Gateway with Multihoming

Wed, 06 Mar 2024 00:00:00

https://bugs.launchpad.net/neutron/+bug/2002687

Currently Neutron routers only support one external gateway port and ECMP default routes can only be added manually as extra static routes. Likewise, BFD is not configurable for ECMP routes. This specification provides an extension to the existing Neutron API for configuring multiple external gateways with automatic addition of ECMP default routes and BFD for those routes. It also discusses the problem of scheduling multiple gateway ports per router on different chassis. OVN is chosen as a primary backend.

Problem Description

Some network designs include multiple L3 gateways to:

Share the load across different gateways: both in terms of different OVN chassis hosting different gateway ports and sharing the processing load and upstream gateways handling parts of the north-south flows;
Provide independent network paths for the north-south direction (e.g. via different ISPs) for resiliency without relying on the same L2.

Having multi-homing implemented at the instance level imposes additional burden on the end user of a cloud and support requirements for the guest OS, whereas utilizing ECMP and BFD at the router side alleviates the need for instance-side awareness of a more complex routing setup.

Adding more than one gateway port implies extending the existing data model which was described in the multiple external gateways spec. However, it left adding additional gateway routes out of scope leaving this to future improvements around dynamic routing. Also the focus of neutron-dynamic-routing has so far been around advertising routes, not accepting new ones from the external peers - so dynamic routing support like this is a very different subject. However, manual addition of extra routes does not utilize the default gateway IP information available from subnets in Neutron while this could be addressed by implementing an extra conditional behavior when adding more than one gateway port to a router.

ECMP routes can result in black-holing of traffic if the next-hop of a route becomes unreachable. BFD is a standard protocol adopted by IETF for next-hop failure detection which can be used for route eviction. OVN supports BFD as of v21.03.0 with a data model that allows enabling BFD on a per next-hop basis by associating BFD session information with routes, however, it is not modeled at the Neutron level even if a backend supports it.

Maintaining too many BFD sessions can have a performance impact as periodic protocol messages are going to consume CPU cycles of a host. The implementation will aim to minimize the amount of BFD sessions necessary per destination. One way to do it is to reuse the BFD session information across routers’ default routes and extra routes if the peer endpoint and the rest of the session configuration matches. However, care should be taken by operators when it comes to the amount of routers they would like to use with BFD configured.

From the Neutron data model perspective, ECMP for routes is already a supported concept since ECMP support spec got implemented in Wallaby (albeit the spec focused on the L3-agent based implementation).

As for OVN and BFD, the OVN database state needs to be populated by Neutron based on the data from the Neutron database, therefore, data model changes to the Neutron DB is needed to represent the BFD session parameters.

Proposed Change

DB Impact

Core Model

Currently there are two ways in which router to gateway port relationship is expressed in Neutron:

The gw_port_id foreign key in the routers table which is set to a UUID of the router port that has a type of network:router_gateway;
The routerports table which was added for referential integrity purposes and stores router_id to port_id mappings along with a redundant port_type.

In terms of the representation of multiple gateway ports in the Neutron DB the proposal follows the multiple external gateways spec:

Keep the routers, routerports and ports tables as they are now but start storing multiple network:router_gateway ports per router in the routerports table;
For backwards-compatibility store a single gateway port id in routers.gw_port_id. The compatibility gateway port will be stored along with the other gateway ports in the routerports table.
Extend the neutron.db.models.l3.Router class with a new attribute gw_ports that will map to all relevant network:router_gateway ports stored in the routerports table.

BFD and ECMP Route Behavior Modeling

Each external gateway info dictionary of a router should contain additional information about the policy around handling of default routes derived from the subnets associated with the gateway ports of a router. This is needed so that Neutron as a CMS has the required state to specify to OVN whether ECMP routes are wanted and whether BFD needs to be enabled for them instead of just passing those options through at the time of addition of an external gateway to a router. Therefore, this information needs to be stored along the router.

The following additional columns are proposed for the router_extra_attributes table:

enable_default_route_ecmp is a router-level policy on whether ECMP default routes should be used or not (if the L3 service plugin supports them).

enable_default_route_bfd is a router-level policy on whether BFD should be used for checking whether the next-hop of a default route is reachable (if the L3 service plugin supports BFD).

OVN modeling of BFD will be used to implement the support for BFD sessions for default routes.

OVN allows making a BFD session for a particular static route to use a different destination IP for checking reachability rather than the next hop of the static route itself (by storing the BFD peer IP address in the dst_ip field in the BFD table). This is a useful semantic for separating the control plane and data plane and can be used for static routes of Neutron routers too. However, when it comes to default routes inferred from gateway port to subnet associations, Neutron’s behavior should be to use the next hop of the default route as a dst_ip.

OVN models the BFD table as follows (see ovn-nb docs for more information):

dst_ip - a BFD peer IP address;
min_tx - an integer specifying the minimum interval (milliseconds, >=1) that OVN would use when transmitting the BFD control packet (minus jitter);
min_rx - an integer specifying the minimum interval (milliseconds) between the received BFD control packets that OVN is capable of supporting (minus the jitter applied by the sender). Can be set to 0 to state that BFD control packet transmission from the BFD peer is not desired;
detect_mult - an integer (>= 1) specifying the detection time multiplier. The negotiated transmit interval, multiplied by this value, provides the Detection Time for the receiving system in Asynchronous mode.
OVN includes the options field that includes a map of string-string pairs which is reserved for future use. A similar field or extra columns can be added later to the Neutron model to account for additional extensions.

The proposed approach for this spec is limited to optionally enabling BFD for inferred default routes only leaving a more advanced BFD model with per extra static route BFD sessions for the future iterations (e.g. during the implementation of the BFD support spec spec).

Rest API Changes

Router API

The API changes augment the changes from the multiple external gateways spec but also include additional changes behavioral changes, thus, a different name for an API extension is used in this specification: external-gateway-multihoming and API changes are listed here in full.

New attributes are also added as extra router attributes and the API is extended to handle those in the router API resource (a separate extension is added per attribute):

enable_default_route_ecmp is a router-level policy on whether ECMP default routes should be used or not (if the L3 service plugin supports them).
enable_default_route_bfd is a router-level policy on whether BFD should be used for checking whether the next-hop of a default route is reachable (if the L3 service plugin supports BFD).

With the external-gateway-multihoming extension a new router API resource attribute is added called external_gateways which is a list of external_gateway_info structures.

The first element of the external_gateways list is special for compatibility purposes as it contains the same information as the external_gateway_info does. When enable_default_route_ecmp is set on a router to false it also defines the default gateway placed into the routers routing table (while the OVN driver currently does not support routing for the multi-segment network case, the placement of a gateway port would matter for inferring the default gateway based on the subnet used on a network segment).

The order of the rest of the list is ignored.

Duplicates in the list (that is multiple external gateways with the same network_id) are allowed: in that case multiple gateway ports will be attached to the same network (this can be used to have the active-active setup when external gateways are available on the same network). However, attaching multiple gateway ports to different networks with overlapping subnet ranges will cause routing issues, so that kind of overlap is not allowed.

Updating external_gateway_info also updates the first element of external_gateways and it leaves the rest of external_gateways unchanged. Setting external_gateway_info to an empty value removes a single (compatibility) gateway from the set of gateway ports of a router and chooses an existing extra gateway as a replacement for the compatibility gateway instead.

The external_gateways attribute cannot be set in POST /v2.0/routers or PUT /v2.0/routers/{router_id} requests, instead it can be managed via sub-methods:

PUT /v2.0/routers/{router_id}/add_external_gateways

Accepts a list of external_gateway_info structures. Adding gateways to the same network is allowed provided that fixed IPs (if specified) are not used yet. If one or more gateways are present for a router already then the first item in the list for addition will become an extra gateway. If none are present, the first item will be treated as a compatibility gateway.
PUT /v2.0/routers/{router_id}/update_external_gateways

Accepts a list of external_gateway_info structures. The external gateways to be updated are identified by the network_id field and external_fixed_ips found in the PUT request. Updating enable_snat is only possible at the per-router basis on the first item specified. Updating external_fixed_ips is possible without recreating a port.
PUT /v2.0/routers/{router_id}/remove_external_gateways

Accepts a list of potentially partial external_gateway_info structures. A combination of network_id and external_fixed_ips fields from external_gateway_info structure is used to identify a particular gateway to be removed. Other keys can be present but their values are ignored.

The add/update/remove PUT sub-methods respond with the whole router object just as POST/PUT/GET /v2.0/routers.

Extra routes API: ECMP

As the ECMP support spec notes, there are no API changes to make to support ECMP routes per se: multiple routes to the same destination and different next-hops can already be specified when adding extra routes. However, that spec focused on the agent-based implementation - part of the work to implement this spec is to check whether the same is true for the OVN-based L3 implementation.

Extra routes API: BFD

In the absence of a full BFD API, users will have an option to specify a policy on the routers (enable_default_route_bfd).

OVN driver changes

In general, we will update the existing OVN driver to handle the presence of multiple gateway ports wherever gateway ports are currently handled in the existing code. A few areas of interest are highlighted below.

Router level External IDs

There are a couple of router level external IDs in the existing implementation which do not work with multiple gateway ports:

ovn_const.OVN_GW_PORT_EXT_ID_KEY
ovn_const.OVN_GW_NETWORK_EXT_ID_KEY

These will be deprecated and replaced by methods that look up the required information at runtime.

L3 Scheduler Changes

One of the main use cases for routers with multiple gateway ports is resiliency. Whenever there are multiple gateway ports present for a single router, we want to ensure diverse placement of these ports across chassis to minimize impact of chassis failure.

This will be implemented by updating the leastloaded scheduler to apply soft anti-affinity when scheduling gateway ports for routers with multiple gateway ports.

No changes will be made to the chance scheduler.

Out of Scope

BFD session data model and API;
BFD authentication as it is not implemented in the OVN BFD implementation while it is present in the protocol RFC itself. Therefore, the data model should be extensible to support this in the future;
Enabling BFD for extra routes. For now the spec will only address the inferred routes leaving this for future iterations;
Solving the distributed SNAT problem. One direction is to use conntrack state synchronization between the gateway ports. Other ideas involve making smarter control plane choices about where this conntrack state should exist instead of distributing it everywhere - this can be done by ensuring that processing of flows is done locally to the instance but there are downsides to that as well which needs to be considered more carefully
Dealing with asymmetric routing:
- Conntrack can be utilized to avoid responses generated by instances to go via the route different from the one the request came in on in presence of ECMP routes. OVN has support for making the reply traffic take the symmetric path. This can be configured by utilizing the options column in the logical router static routes table in OVN which allows configuring ECMP symmetric reply by setting ecmp_symmetric_reply option to true (it is modeled at the route level in OVN as well).
- Routes in Neutron could have an ecmp_symmetric_reply option to specify a policy on whether to enable ECMP symmetric reply depending on whether the L3 service plugin supports it or not.
- However, the commit introducing the feature in OVN notes a limitation on its use: it can only be used on gateway routers, not distributed routers that have a gateway port due to the dependency of the ingress pipeline logic of the logical router on the hypervisor-local CT state.
Accepting ECMP routes via dynamic routing protocols. The current aim is to utilize the default gateway information available in Neutron subnets to configure default gateway ECMP routes or to use the extra routes extension. This specification is a building block for the future support of dynamic routing.
Modeling of route metrics. While there are cases where one default route could be preferred over the other for the same destination, neither Neutron nor OVN model this concept today;
Implementation of BFD for the non-OVN L3 implementation based on Linux namespaces.

Implementation

Assignee(s)

Frode Nordahl <frode.nordahl@canonical.com> (~fnordahl)
Dmitrii Shcherbakov <dmitrii.shcherbakov@canonical.com> (~dmitriis)

Work Items

Add the new REST API by making neutron-lib and Neutron changes to the API, core Neutron and OVN integration code;
Change the DB schema to add new attributes and create relevant DB migrations;
Implement support for external-gateway-multihoming extension in the OVN driver.
Update the OVN L3 leastloaded scheduler to apply soft anti-affinity when scheduling gateway ports for routers with multiple gateway ports.
Update the CLI in order to utilize the newly added rest API;
Update the relevant documentation;
Implement relevant unit and functional tests using the existing facilities in Neutron.

Expose backend hints in the port API, hint ovs-tx-steering

Tue, 24 Oct 2023 00:00:00

RFE: https://bugs.launchpad.net/neutron/+bug/1990842

Introduce a port attribute that allows passing in backend specific hints mainly to allow backend specific performance tuning.

Problem Description

Some of our performance sensitive users would like to tweak Open vSwitch’s Tx packet steering option [1] under OpenStack.

This is available since Open vSwitch v2.17.0. [2], [3]

Proposed Change

We propose to introduce two new extension: port-hints and port-hint-ovs-tx-steering.

The purpose of decomposing this feature to two extensions is extensibility: to allow other hints in the future. port-hints signals the presence of a new port attribute, for details see below. port-hint-foobar signals what values of the new port attribute (or which is the same: what hints) are known to a Neutron instance.

port-hints adds a new port attribute: hints. Its default policy is admin_only. Its value is a dict (by default empty). The dict is keyed by the standard mechanism driver aliases as in [4]. This spec allows only a single key: openvswitch. Other keys for other mechanism drivers may be introduced by later specs. The value for a mechanism_driver is possibly a complex structure, but at least a dict on the top level. In this spec we only partially define its format - introducing one hint. This is marked by the second extension port-hint-ovs-tx-steering. Definition of other hints is left to future specs. Here consider this partial body for a create port request:

{
    "port": {
        "hints": {
            "openvswitch": {"other_config": {"tx-steering": "thread"|"hash"}}}
        ...
}

Everything in hints is interpreted not as a demand, but as a suggestion. That is, neutron is free to ignore some or all of the requested hints without returning an error response or putting the port into an error status. Particularly neutron is free to ignore the requested hints when the port is bound by a different mechanism driver.

hints can be set at port create or update, but there’s no guarantee that an update to the hints after the port was bound will have any effect.

hints is intentionally not named binding:hints and it should not be affecting the binding process and decision.

In the openstack CLI we propose to expose the above API feature as:

openstack port create/set --hint HINT-ALIAS=HINT-VALUE [--hint ...] ...
openstack port unset --hints ...

For example:

openstack port create --hint ovs-tx-steering=thread ...
openstack port create --hint ovs-tx-steering=hash ...

The above CLI syntax allows key:value type hints, and can be extended to allow boolean hints if needed. There is an arbitrary but documented mapping between a HINT-ALIAS=HINT-VALUE and the fully fledged data structure passed to the API. The openstack CLI also allows an arbitrary JSON value in the –hint parameter which it passes to the API as-is.

Given the ovs-tx-steering hint passed in, ovs-agent can set the corresponding OVS interface’s other_config (using the python native interface of course, not ovs-vsctl):

sudo ovs-vsctl set Interface ovs-interface-of-the-port other_config:tx-steering=thread
sudo ovs-vsctl set Interface ovs-interface-of-the-port other_config:tx-steering=hash

The default value of hint ovs-tx-steering is thread.

API Impact

Add a new admin_only field to the port resource called hints. This field can be present in GET, POST and PUT requests. This field cannot be longer than 4095 characters. For its semantics, please see above.

DB Impact

Introduce a new table porthints and autojoin it with the ports table.

Client Impact

Relevant changes in osc and openstacksdk.

Testing

Unit tests.
Tempest tests in neutron-tempest-plugin.

Assignee(s)

Bence Romsics <bence.romsics@gmail.com>

References

L3/BGP - Allow to configure the BGP peer connect mode

Tue, 24 Oct 2023 00:00:00

https://bugs.launchpad.net/neutron/+bug/2040001

This RFE intends to implement a configuration option to allow the Operator to change the Neutron Dynamic Routing (n-d-r) speaker’s default BGP connect mode to all supported modes: ACTIVE, PASSIVE or BOTH.

Problem description

When using n-d-r, the speaker’s current behaviour is to work in ACTIVE mode which will not create a local TCP socket on TCP 179 port. Therefore, in the switch’s BGP session, it needs to be configured as PASSIVE and not initiate TCP connections.

Operators could have a switch brand that does not follow correctly the RFC 4271 [1] regarding with the BGP FSM (8.2).

When the n-d-r and the switch has a health BGP session, the Operator could run a maintenance window in the switch or in the n-d-r agent and the session could flap and go down for while and on the switch side, the BGP session state can change to IDLE.

According to RFC 4271(8.2.2.), when a BGP session is in IDLE state, it refuses all inbound BGP connection attempts, and starts a TCP connection to the peer(n-d-r) and n-d-r speaker’s will not answer because n-d-r is in ACTIVE mode and has no listen port for this. If the switch correctly follow the RFC regarding BGP PASSIVE mode when enabled, it will ignore it and accept the incoming connection from the n-d-r and the connection will be restablished, regardless of the IDLE state.

That is definitely a switch-side issue but when having this situation, we could allow the Operator to change the default behaviour of n-d-r speaker’s from ACTIVE and set it as PASSIVE or BOTH using a n-d-r configuration option and then open the local TCP socket on the TCP 179 port in order to anwser incoming TCP connections.

Proposed change

This proposal implements a configuration option to allow the Operator to change the n-d-r speaker’s default BGP connect mode from ACTIVE to PASSIVE or BOTH. For Operators who do not have the problem described above, it is not necessary to change anything as the BGP connection mode will remain as ACTIVE.

To change the n-d-r speaker’s default BGP connect mode, a new configuration option should be enabled via [BGP] section in the n-d-r agent configuration file.

* ``bgp_connect_mode = both``

The bgp_connect_mode values supported would be:

active initiate the TCP connection to establish the BGP session and do not require to open a local TCP socket on port 179. The default value for bgp_connect_mode must be active to keep the current behaviour.
passive open a local TCP socket on port 179 and wait for an incoming TCP connection from the BGP peer.
both open a local TCP socket on port 179 and wait for an incoming TCP connection from the BGP peer. Additionally, a connection could be initiated to establish the BGP session with the peer.

Neutron Dynamic Routing Impact

When the n-d-r agent is started:

if the option bgp_connect_mode is ommited or has the value active the BGP connection mode will remain as ACTIVE and it will not create a local TCP socket on port 179.
if the option bgp_connect_mode has the value passive the BGP connection mode will be used as PASSIVE and a local TCP socket on port 179 will be opened.
if the option bgp_connect_mode has the value both the BGP connection mode will be used as BOTH and a local TCP socket on port 179 will be opened.

Because n-d-r could start the agent as an unprivileged user, systemd will require for the n-d-r agent service to have net bind capability to allow for creation of the local TCP socket on port 179.

Security Impact

The Operator will have to ensure the safety of the n-d-r agent when using bgp_connect_mode as passive or both, and create local firewall rules to only allow incoming TCP connection from BGP peers.

Implementation

Assignee(s)

Primary assignees: * Tiago Pires <tiagohp@gmail.com> * Roberto Bartzen Acosta <rbartzen@gmail.com>

Work Items

Implement a new configuration option.
Implement relevant unit and functional tests using the existing facilities in Neutron Dynamic Routing.
Write documentation.

Testing

Unit/functional tests.

Documentation Impact

User Documentation

Information about the bgp_connect_mode configuration option.

References

L3/BGP - Make BGP Speaker peer sessions resilient to infrastructure outage

Tue, 24 Oct 2023 00:00:00

https://bugs.launchpad.net/neutron/+bug/2006145

This RFE intends to implement resilient support for BGP peer sessions established by the BGP speaker. Nowadays, the Neutron Dynamic Routing service architecture depends on some type of message between the DRAgent service and the Neutron server. However, these communications are strongly dependent on the messaging service availability (such as RabbitMQ), and any transient/permanent failures in OpenStack infrastructure nodes may affect prefix advertising via BGP.

Problem Description

When we are using dynamic routing with BGP to advertise network address prefixes in a cloud infrastructure, all the network traffic depends on the correct operation by the BGP peer elements. The DRAgent has a crucial role in that scenario because this service is responsible for advirtise all L3 prefixes in a North/South communication, being a single point of vulnerability for the cloud operation.

The current DRAgent reference design [1] using two separated periodic tasks as described below:

State Report periodic task (DrAgentWithStateReport class):

+------------------------------------------------+
|DrAgentWithStateReport class                    |
|                                                |
|     +------------------------------------+     |
|     |heartbeat = _report_state           |     |
|     |interval=CONF.AGENT.report_interval |     |
|     |  call _report_state                |     |
|     +------------------+-----------------+     |
|                        |                       |
|     +------------------+-----------------+     |
|     |_report_state polling task          |     |
|     |                                    |     |
|     | RPC processing                     |     |
|     |   if agent_status == revived       |     |
|     |     call schedule_full_resync      |     |
|     +------------------+-----------------+     |
|                        |                       |
|     +------------------+-----------------+     |
|     |schedule_full_resync                |     |
|     +------------------------------------+     |
|                                                |
+------------------------------------------------+

BGP DRAgent periodic task (BgpDrAgent class):

+---------------------------------------+
|BgpDrAgent class                       |
|                                       |
|   +-------------------------------+   |
|   |periodic_resync polling task   |   |
|   |interval=CONF.periodic_interval|   |
|   |  call _periodic_resync_helper |   |
|   +---------------+---------------+   |
|                   |                   |
|   +---------------+---------------+   |
|   |_periodic_resync_helper        |   |
|   | if full_sync or resync/reason |   |
|   |   call sync_state             |   |
|   +---------------+---------------+   |
|                   |                   |
|   +---------------+---------------+   |
|   |sync_state                     |   |
|   |                               |   |
|   | if bgp_speaker_id == None     |   |
|   |   remove speaker from DRAgent |   |
|   |                               |   |
|   | Exception (MessagingTimeout)  |   |
|   |   call schedule_full_resync   |   |
|   +--------------+----------------+   |
|                  |                    |
|   +--------------+----------------+   |
|   |schedule_full_resync           |   |
|   +-------------------------------+   |
|                                       |
+---------------------------------------+

These two periodic tasks have independently configured interval ranges, the periodic_interval for BGP DRAgent task, and the AGENT.report_interval for the heartbeat report state. However, a full resync request can come from both periodic tasks, and the sync_state method will be processed. The case of the RPC heartbeat response with agent_status revived activates this entire flow described above until the speaker is removed from the DRAgent. For the case where an exception occurs in the sync_state method, a full resync is scheduled (according to the flow described above) until the speaker is removed from the DRAgent.

Regardless of the origin of the full resync request, the caching mechanism apparently is not working, as the agent was previously configured and after receiving a revived status via RPC it simply removes the speaker from the DRAgent, and reschedules a future resync to later be re-included (depending on the configured periodic intervals). When the speaker is removed from DRAgent the BGP peer sessions are closed and will only be reestablished if the speaker is re-added.

This problem can be manifested in two different ways:

When the messaging service/RabbitMQ is offline and does not respond for a long interval AMQP server is unreachable (note the timeouts configured for Neutron agent_down); and then RMQ/RPC becomes active again.
When the messaging service/RabbitMQ queues are under a lot of pressure and/or experiencing exception timeouts, such as: Timed out waiting for a reply; and then RMQ/RPC becomes active again.

Proposed Change

To solve the problem described above, the proposal is to introduce a new speaker cache logic for the DRAgent can keep the speaker settings and the BGP peer sessions in case of RPC Exceptions, and/or reestablishment of communication via RPC. In addition, to not remove the BGP speaker configuration from DRAgent due to transient failures in RPC communication, it is required to change the get_bgp_speakers empty returns handling logic to first schedule a full sync, and then allow the BGP speaker to be removed in the next periodic sync.

To enable the new DRAgent speaker cache mechanism, a new config option should be enabled via [BGP] section in bgp_dragent.ini file.

* ``speaker_cache_timeout = 300``

This configuration option enable the DRAgent to keep the speaker settings and related BGP peer sessions until the configured timeout (in seconds). The purpose of this timeout cache is prevent errors in the resync checking logic so that no speaker removed until the timeout condition is satisfied.

The default value for speaker_cache_timeout must be zero and thus maintain the current DRAgent behavior. Any non-zero value should affect DRAgent behavior as described below:

State Report periodic task: with the revived RPC status, the DRAgent must be start the cache timeout timer, and set the cache_out_of_sync flag as True. If the DRAgent performs the sync_state method before the cache_out_of_sync becomes to False, then the full sync process must be ignored, and wait for the next periodic check.
DRAgent periodic task: if the sync_state method throws any exception during operation, the DRAgent must be start the cache timeout timer, and set the cache_out_of_sync flag as True. Similar to the case described above, if the DRAgent performs the sync_state method before the cache_out_of_sync becomes to False, then the full sync process must be ignored, and wait for the next periodic check.
Cache timeout task: if another revived or Exception event occurs during the timer count, the cache task must reset the timeout interval count and start again. Otherwise, the cache task must terminate at the expiry of the configured timeout, and set the cache_out_of_sync flag to False.
Default workflow: If the speaker_cache_timeout is empty or set to zero; or if the configured cache timeout timer has expired - cache_out_of_sync flag is False; any periodic task should be run a full resync.

DB Impact

None

Rest API Changes

None

Implementation

Assignee(s)

Primary assignees: Roberto Bartzen Acosta <rbartzen@gmail.com>

Work Items

Implement a new cache logic in DRAgent speaker resync.
Implement relevant unit and functional tests using the existing facilities in Neutron Dynamic Routing.
Write documentation.

Documentation Impact

User Documentation

Information about the DRAgent speaker caching support.

Testing

Unit/functional tests.

References

Flavor/service provider support for routers in the ML2/OVN L3 plugin

Mon, 05 Jun 2023 00:00:00

https://bugs.launchpad.net/neutron/+bug/2020823

In Multiple L3 backends, functionality was added to the ML2/OVS mechanism driver to support L3 routers implemented by different back-end drivers. Recently, OpenStack deployers in the Telco / 5G industry have expressed a strong interest to have the same support with the ML2/OVN mechanism driver. This specification proposes to implement that functionality.

Problem Description

Since early in the development of Neutron, the community recognized the need to provide frameworks to enable operators and users of a cloud deployment to configure and select from a set of options different implementations of services such as VPNs, firewalls, routers, etc. This has led over the years to proposals and additions to Neutron such as Neutron/Service Type Framework, Service Flavor Framework and the aforementioned Multiple L3 backends. All this is represented in the API by Networking Flavors Framework, Service Providers and the L3 flavors extension.

As of the writing of this specification, operators using ML2/OVN cannot take advantage of all these existing frameworks to offer different implementations (flavors from here on) of vrouters to their users. As a consequence, Neutron cannot serve a use case where Telco operators want to provide at the same time different grades of vrouters, such as:

OVN vrouters for their IT / back-office workloads.
Specialized vrouters outside OVN for their 5G mobile networks.
Specialized vrouters outside OVN for their 4G mobile networks.
Other types of specialized vrouters.

Proposed Change

The proposal is to add to the ML2/OVN L3 plugin the ability to process different flavors of vrouters. The default flavor will be the native OVN vrouter.

To achieve this, all the OVN specific functionality in the ML2/OVN L3 plugin related to vrouters and floating IPs will be refactored to a separate driver. Once this is done, the L3 plugin will be responsible for performing only the Neutron DB processing steps related to vrouters and floating IPs, while letting separate drivers to take care of all the backend processing. These drivers will listen and act on events notifications sent by the refactored L3 plugin for the creation, update and deletion of vrouters and floating IPs. Each driver will be responsible to act only on those events that pertain to its flavor.

Cloud administrators will configure, load and associate with flavor names drivers for backends different to OVN. The following diagram illustrates the proposed scenario:

+--------+          +------------+               +------------+    +------+    +------+    +-----------------+
|        |          |            |     Event     |            |--->| NBDB |--->| SBDB |--->| OVN Controllers |
|        |          |            | Notifications | Driver for |    +------+    +------+    +-----------------+
|        |          |            |-------------->| flavor OVN |
|        |          |            |               |            |
|        |          |            |               +------------+
|        |          |            |
|        |          |            |               +------------+    +-----------+
|        |          |            |     Event     |            |--->| Backend A |
|  ReST  | Requests |     L3     | Notifications | Driver for |    +-----------+
|  API   |--------->|   plugin   |-------------->|  flavor A  |
|        |          |            |               |            |
|        |          |            |               +------------+
|        |          |            |
|        |          |            |               +------------+    +-----------+
|        |          |            |     Event     |            |--->| Backend B |
|        |          |            | Notifications | Driver for |    +-----------+
|        |          |            |-------------->|  flavor B  |
|        |          |            |               |            |
+--------+          +------------+               +------------+
                          |                             |
                          |       +-------------+       |
                          |       |             |       |
                  Updates +------>|   Neutron   |<------+ Reads
                                  |   database  |
                                  |             |
                                  +-------------+

In this diagram, backend means any combinations of agents and databases (outside the purview of Neutron) that a driver needs to instantiate routers and floating IPs in the data plane. This specification implementation will include a driver for OVN and the necessary scaffolding to enable other drivers to be configured in a Neutron deployment, but the drivers themselves and their associated databases and agents will be provided by third parties.

Once drivers are configured, users will specify the required flavor when creating vrouters. The OVN flavor will be the default and will be denoted by specifying no flavor in the create request. The following commands illustrate how to create a thrid party flavor vrouter:

$ openstack network flavor list
+--------------------------------------+----------------------------+---------+---------------+------------------------------------------------------+
| ID                                   | Name                       | Enabled | Service Type  | Description                                          |
+--------------------------------------+----------------------------+---------+---------------+------------------------------------------------------+
| e47c1c5c-629b-4c48-b49a-78abe6ac7696 | user-defined-router-flavor | True    | L3_ROUTER_NAT | User defined flavor for routers in the L3 OVN plugin |
+--------------------------------------+----------------------------+---------+---------------+------------------------------------------------------+

$ openstack router create router-of-user-defined-flavor --flavor e47c1c5c-629b-4c48-b49a-78abe6ac7696
+-------------------------+--------------------------------------+
| Field                   | Value                                |
+-------------------------+--------------------------------------+
| admin_state_up          | UP                                   |
| availability_zone_hints |                                      |
| availability_zones      |                                      |
| created_at              | 2023-06-08T20:26:01Z                 |
| description             |                                      |
| enable_ndp_proxy        | None                                 |
| external_gateway_info   | null                                 |
| flavor_id               | e47c1c5c-629b-4c48-b49a-78abe6ac7696 |
| id                      | fe110d50-f994-4593-86a4-fc2ecca34c38 |
| name                    | router-of-user-defined-flavor        |
| project_id              | b807321af03f44dc808ff06bbc845804     |
| revision_number         | 1                                    |
| routes                  |                                      |
| status                  | ACTIVE                               |
| tags                    |                                      |
| tenant_id               | b807321af03f44dc808ff06bbc845804     |
| updated_at              | 2023-06-08T20:26:01Z                 |
+-------------------------+--------------------------------------+

As of the writing of this specification, the openstack client doesn’t allow the specification of a flavor ID when creating a vrouter. Adding this functionality to the client will be part of this specification’s implementation.

The OVN flavor driver will be loaded by default when the L3 plugin starts. This is an example of the steps a cloud administrator will follow to configure and load drivers for other flavors:

Add the service provider to neutron.conf:

[service_providers]
service_provider = L3_ROUTER_NAT:user-defined:neutron.services.ovn_l3.service_providers.user_defined.UserDefined

Re-start the neutron server and verify the user defined provider has been loaded:

$ openstack network service provider list
+---------------+--------------+---------+
| Service Type  | Name         | Default |
+---------------+--------------+---------+
| L3_ROUTER_NAT | user-defined | False   |
| L3_ROUTER_NAT | ovn          | True    |
+---------------+--------------+---------+

Create a service profile for the router flavor:

$ openstack network flavor profile create --description "User defined router flavor profile" --enable --driver neutron.services.ovn_l3.service_providers.user_defined.UserDefined
+-------------+--------------------------------------------------------------------+
| Field       | Value                                                              |
+-------------+--------------------------------------------------------------------+
| description | User defined router flavor profile                                 |
| driver      | neutron.services.ovn_l3.service_providers.user_defined.UserDefined |
| enabled     | True                                                               |
| id          | a717c92c-63f7-47e8-9efb-6ad0d61c4875                               |
| meta_info   |                                                                    |
| project_id  | None                                                               |
+-------------+--------------------------------------------------------------------+

Create the router flavor:

$ openstack network flavor create --service-type L3_ROUTER_NAT --description "User defined flavor for routers in the L3 OVN plugin" user-defined-router-flavor
+---------------------+------------------------------------------------------+
| Field               | Value                                                |
+---------------------+------------------------------------------------------+
| description         | User defined flavor for routers in the L3 OVN plugin |
| enabled             | True                                                 |
| id                  | e47c1c5c-629b-4c48-b49a-78abe6ac7696                 |
| name                | user-defined-router-flavor                           |
| service_profile_ids | []                                                   |
| service_type        | L3_ROUTER_NAT                                        |
+---------------------+------------------------------------------------------+

Add service profile to router flavor:

$ openstack network flavor add profile user-defined-router-flavor a717c92c-63f7-47e8-9efb-6ad0d61c4875

REST API impact

No REST API impact is expected.

DB Impact

No DB impact is expected.

Implementation

Assignee(s)

Miguel Lavalle irc: mlavalle, email: mlavalle@redhat.com

Work Items

Re-factor the OVN vrouters and floating IPs (NAT) functionality embedded in the L3 plugin to a separate driver.
Implement a drivers controller for the L3 plugin that will load automatically the driver for OVN routers and floating IPs.
Update the the ML2/OVN maintenance task so vrouters of third party flavors are not synched with the NBDB.
Add functionality to the openstack client to enable users to specify a flavor when creating a vrouter.
Implement tests.
Write documentation.

References

Provided inline in the text above.

Configurable Security Group rules added by default to the new Security Groups

Thu, 18 May 2023 00:00:00

https://bugs.launchpad.net/neutron/+bug/1983053

Problem Description

For every project, Neutron always creates security group called default. It is created always when first time security groups are going to be used or requested to be listed via API. For each of such default security group Neutron automatically creates 4 hardcoded rules. Those rules allow:

all IPv4 egress traffic from the port,
all IPv6 egress traffic from the port,
all IPv4 ingress traffic to the port incoming from other ports which are using the same security group,
all IPv6 ingress traffic to the port incoming from other ports which are using the same security group.

For any other security group created by user, Neutron automatically creates 2 rules which allows:

all IPv4 egress traffic from the port,
all IPv6 egress traffic from the port,

There is at least couple of issues with such approach:

it is a known fact that the security group rules with remote_group_id (rule 3. and 4. above) don’t scale well e.g. with neutron-openvswitch-agent [1],
some operators would like to define different rules to be created by default for each new project.

Of course, those rules created by default by Neutron can be easily removed by the security group owner, but having possibility to define different set of rules added automatically to each security group could make it easier for users and administrators of the cloud.

Proposed Change

To solve the problem described above, proposal is to introduce new API to create security group rules template used to create new security group rules for each new security group. To keep backward compatibility with the existing hardcoded rules, by default Neutron will have configured the same 4 rules as are hardcoded today (see above for defails). Cloud admin will be able to delete those rules and create new ones which will be then used by Neutron for every new security group. In case when list of the default_security_group_rule is empty, new security groups will be created without any rules created by default.

REST API Impact

New API resource called default_security_group_rule will be introduced. Details of the API are below:

GET /v2.0/default-security-group-rules

List default security group rules used to create rules for every new Security Group

Response:

{
  "default_security_group_rules": [
  {
      "direction": "egress",
      "ethertype": "IPv6",
      "id": "3c0e45ff-adaf-4124-b083-bf390e5482ff",
      "port_range_max": null,
      "port_range_min": null,
      "protocol": null,
      "remote_group_id": null,
      "remote_address_group_id": null,
      "remote_ip_prefix": null,
      "used_in_default_security_group": True,
      "used_in_non_default_security_group": True,
      "description": ""
  },
  {
      "direction": "egress",
      "ethertype": "IPv4",
      "id": "93aa42e5-80db-4581-9391-3a608bd0e448",
      "port_range_max": null,
      "port_range_min": null,
      "protocol": null,
      "remote_group_id": null,
      "remote_address_group_id": null,
      "remote_ip_prefix": null,
      "used_in_default_security_group": True,
      "used_in_non_default_security_group": True,
      "description": ""
  },
  {
      "direction": "ingress",
      "ethertype": "IPv6",
      "id": "3c0e45ff-adaf-4124-b083-bf390e5482ff",
      "port_range_max": null,
      "port_range_min": null,
      "protocol": null,
      "remote_group_id": "PARENT",
      "remote_address_group_id": null,
      "remote_ip_prefix": null,
      "used_in_default_security_group": True,
      "used_in_non_default_security_group": False,
      "description": ""
  },
  {
      "direction": "ingress",
      "ethertype": "IPv4",
      "id": "93aa42e5-80db-4581-9391-3a608bd0e448",
      "port_range_max": null,
      "port_range_min": null,
      "protocol": null,
      "remote_group_id": "PARENT",
      "remote_address_group_id": null,
      "remote_ip_prefix": null,
      "used_in_default_security_group": True,
      "used_in_non_default_security_group": False,
      "description": ""
  },
  {
      "direction": "ingress",
      "ethertype": "IPv6",
      "id": "3c0e45ff-adaf-4124-b083-bf390e5482ff",
      "port_range_max": 22,
      "port_range_min": 22,
      "protocol": null,
      "remote_group_id": null,
      "remote_address_group_id": null,
      "remote_ip_prefix": null,
      "used_in_default_security_group": False,
      "used_in_non_default_security_group": True,
      "description": "Allow SSH connections over IPv6"
  },
  {
      "direction": "ingress",
      "ethertype": "IPv4",
      "id": "3c0e45ff-adaf-4124-b083-bf390e5482ff",
      "port_range_max": 22,
      "port_range_min": 22,
      "protocol": null,
      "remote_group_id": null,
      "remote_address_group_id": null,
      "remote_ip_prefix": null,
      "used_in_default_security_group": False,
      "used_in_non_default_security_group": True,
      "description": "Allow SSH connections over IPv4"
  }]
}

POST /v2.0/default-security-group-rules

Create default security group rule used to create rules for every new Security Group

Request:

{
  "default_security_group_rule": {
    "direction": "ingress",
    "port_range_min": "80",
    "ethertype": "IPv4",
    "port_range_max": "80",
    "protocol": "tcp",
  }
}

Response:

{
  "default_security_group_rule": {
    "direction": "ingress",
    "ethertype": "IPv4",
    "id": "2bc0accf-312e-429a-956e-e4407625eb62",
    "port_range_max": 80,
    "port_range_min": 80,
    "protocol": "tcp",
    "remote_group_id": null,
    "remote_address_group_id": null,
    "remote_ip_prefix": null,
    "used_in_default_security_group": False,
    "used_in_non_default_security_group": True,
    "description": ""
  }
}

GET /v2.0/default-security-group-rules/{rule_id}

Show default security group rule used to create rules for every new Security Group

Response:

{
  "security_group_rule": {
    "direction": "egress",
    "ethertype": "IPv6",
    "id": "3c0e45ff-adaf-4124-b083-bf390e5482ff",
    "port_range_max": null,
    "port_range_min": null,
    "protocol": null,
    "remote_group_id": null,
    "remote_address_group_id": null,
    "remote_ip_prefix": null,
    "used_in_default_security_group": False,
    "used_in_non_default_security_group": True,
    "description": ""
  }
}

DELETE /v2.0/default-security-group-rules/{rule_id}

Delete default security group rule used to create rules for every new Security Group

DB Impact

Default security group rule DB table:

Attribute	Type	Req	CRUD	Description
id	uuid-str	No	R	Id of default security group rule.
direction	String	Yes	CR	Direction in which the security group rule is applied.
ethertype	String	No	CR	Must be IPv4 or IPv6.
remote_group_id	String	No	CR	The remote group UUID to associate with this security group rule. Special value `PARENT` can be also used and it means to always use id of the security group in which will be created with such rule.
remote_address_group_id	String	No	CR	The remote address group UUID to associate with this security group rule.
protocol	String	No	CR	The IP protocol can be represented by a string, an integer, or null. Valid strings or integers are the same as for the `security group rule`.
port_range_min	String	No	CR	The minimum port number in the range that is matched by the security group rule.
port_range_max	Integer	No	CR	The maximum port number in the range that is matched by the security group rule.
remote_ip_prefix	String	No	CR	The remote IP prefix that is matched by this security group rule.
standard_attr_id	Ingeger	Yes	R	Id of the associated standard attribute record.
used_in_default_sg	Boolean	No	CR	If it is set to `True` such rule will be used in a template for the `default` security group which is created automatically for every project. Default value is `False`
used_in_non_default_sg	Boolean	No	CR	If it is set to `True` such rule will be used in a template for the every `non default` security group. Default value is `False`

Security Impact

New API will be by default available only for the admin users.

Performance Impact

None

Implementation

Assignee(s)

Primary assignee:: Slawek Kaplonski <skaplons@redhat.com> (IRC: slaweq)

Work Items

REST API update.
DB schema update.
Security Group DB code update.
CLI update.
Documentation.
Tests and CI related changes.

Testing

Unit Test
API test

Documentation Impact

User Documentation

New API must be documented in the Neutron API reference document.

References

Port extension to create hardware offloaded ports

Thu, 04 May 2023 00:00:00

Launchpad bug: https://bugs.launchpad.net/neutron/+bug/2013228

The aim of the RFE is to create a new port extension to allow to create ports with hardware offloaded capabilities.

Problem Description

The RFE [1] introduced the capability of creating a port that could use the hardware offload support implemented in Open vSwitch [2]. This feature was implemented in the following set of patches (initially for ML2/OVS): [3], [4] and [5].

In order to create a port with hardware offload capabilities, it is needed to define the VNIC type and set a specific value in the port binding profile:

openstack port create --vnic-type direct \
    --binding-profile '{"capabilities": ["switchdev"]}' port_hwol

This method to create a port has several drawbacks:

The port binding profile field is an admin only parameter by default. The values contained in this field can have references to PCI addresses of the host and should not be modified by a non-admin user.
Actually this parameter should not be written by Neutron, only by Nova when the port is bound.

Proposed Change

This RFE proposes to create a new port extension, called “port-hardware-offload”, that will replace the need of manually defining the port binding profile information when creating the port; please note that this is referring only to the port creation process. This API extension consists of a string parameter that will be “None” by default. This string will be the hardware offload type; OpenStack Neutron and Nova are currently supporting only “switchdev” [6]. The strings allowed by the API will be limited to a set of defined constants.

This new string parameter will be passed to Nova along with the port information dictionary. Nova will use this parameter instead of the port binding profile information to command os-vif to create the corresponding layer 1 port (a devlink port [3] [7]).

Because of this Nova-Neutron communication, this RFE is going to be implemented in two steps:

The first one will implement only the Neutron and OSC (plus the OpenstackSDK bits) code. The port dictionary passed to Nova will contain both the new parameter added in this RFE and the port binding profile, that will be added internally by Neutron when the new paramter is not “None”.
The second step involves the Nova implementation. This change implies that Nova can read the port new parameter and create the corresponding port. The port binding profile information passed by Neutron will be irrelevant.

Note

The scope of this RFE, as discussed during the presentation of this new feature in the Neutron drivers meeting, involves only the first step. The Nova code change will be covered in other RFE and spec.

Client Impact

The OSC and OpenstackSDK projects will be updated in order to be able to create a port using this new extension. Port creation example:

openstack port create --vnic-type direct \
    --hardware-offload-type switchdev port_hwol

The new string field will be visible when showing the port resource.

This spec is not covering the Nova migration from reading the port binding profile to using the new parameter provider in the port definition. As commented in the first section, Neutron will still populate the port binding profile as long as the “hardware-offload-type” is provided in the port creation command. In order to stop users to create a port with the old method, the OSC will check if the “capabilities” key is provided inside the port binding profile and a warning will be printed in the command line, encouraging the user to create the port using the “hardware-offload-type” parameter.

REST API Impact

Proposed attribute:

RESOURCE_ATTRIBUTE_MAP = {
    port.COLLECTION_NAME: {
        "hardware_offload_type": {
            'allow_post': True,
            'allow_put': False,
            'convert_to': converters.convert_to_string,
            'default': None,
            'is_visible': True,
            'is_filter': True,
            'validate': {
                'type:values': constants.VALID_HWOL_TYPES}
        }
    }
}

Sample REST calls:

POST /v2.0/ports
{
    "port": {
       ...
       "hardware_offload_type": "switchdev"
}

Response:
{
    "port": {
        "id": "<id>",
        ...
        "hardware_offload_type": "switchdev"
    }
}

Data Model Impact

This RFE proposes to create a child table to ports, called port_hardware_offload.

**Port hardware offload**
Attribute	Type	CRUD	Description
port_id	uuid-str	CR	Unique identifier for the port object
hardware_offload_type	str	CR	String to indicate the hardware offload type

Security Impact

By default, this new field will be writable by the admin only. However, the admin can consider changing the rule owner and allow any project user to create a port defining this parameter:

policy.DocumentedRuleDefault(
    name='create_port:hardware_offload_type',
    check_str=base.ADMIN,
    scope_types=['project'],
    operations=ACTION_POST
)

This rule change was completely discouraged for the rule ‘create_port:binding:profile’ for the reasons provided in the problem description.

Performance Impact

The port resource view will now require a new “JOIN” operation between the ports table and the ports_hardware_offload table when building the port OVO. However there will be a 1:1 relationship between both tables and the data retrieved from the child table is minimal (two columns).

Other Impact

None.

Implementation

Assignee(s)

Primary assignees:: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> (IRC: ralonsoh)

Work Items

API implementation (neutron-lib and Neutron).
Database migration (Neutron)
CLI implementation (OpenstackSDK and OSC)
Documentation.
Tests and CI related changes.

Testing

Unit/functional tests.
Fullstack API tests.

Documentation Impact

User Documentation

Document the new way to create hardware offload ports and deprecate the older method.

References

Tunnel based mirroring (ERSPAN, GRE) for Tap-as-a-service

Wed, 26 Apr 2023 00:00:00

https://bugs.launchpad.net/neutron/+bug/2015471

Mirroring is a widely used tool to analyse traffic of switch ports. The Tap-as-a-service project was created to allow admins to mirror traffic of one Neutron port to another Neutron port.

Mirroring can also be done by encapsulating the traffic into a tunnel, like GRE (Generic Routing Encapsulation) or ERSPAN (Encapsulated Remote Switch Port Analyzer). ERSPAN first was used widely in Cisco switches, and GRE is widely used as tunneling protocol.

ERSPAN protocol has 3 versions of which the last two is adapted, these are version 1 and version 2 (the other versioning uses TYPE I, II and III, and TYPE II is version 1 and TYPE III is version 2) For more details see the ERSPAN draft from Cisco.

Since OVS 2.10 it is possible to use ERSPAN with OVS (see OVS basic configuration, and OVS protocol header fields) both ERSPAN v1 and v2, see erspan NEWS update commit.

Since OVN v22.12.0 it is possible to create mirrors with OVN (see OVN 22.12 nbctl man page, and OVN commit that introduced mirroring).

Note

OVN only supports ERSPAN v1, and with OVN it is also possible to create a clean GRE type mirror.

This specification proposes an extension to the current tap-as-a-service (TAAS) API to allow the users to create ERSPAN or GRE mirrors from Neutron ports to a remote IP, and proposes the necessary backend changes to the current OVS driver of TAAS and proposes a new driver for OVN to use ERSPAN or GRE mirroring with OVN.

Problem Description

Mirroring traffic can be useful in many situations for operators, for example to debug network issues.

Tap-as-a-service provided a solution for traffic mirroring by allowing to create tap-flows and mirror the traffic of them to the related tap-service. Each tap-flow and tap-service can be attached to a Neutron port, so the port attached to tap-flow is the source of the mirrored traffic and the port of tap-service is the destination of the mirroring. There is a N-1 relation between tap-flows and tap-services. This mirroring model is mirroring traffic from one Neutron port to another Neutron port over a Neutron network.

An ERSPAN or GRE mirror is a good solution when an operator needs to mirror traffic from the cloud (from Neutron ports) to an analyser outside the cloud.

Use Cases

As an operator I want to mirror the traffic from a Neutron port to a network analyser that can be outside of my cloud.
As an operator I want to mirror the traffic from a Neutron port to a Floating IP.
As an operator I want to mirror the traffic of a Neutron port to a dedicated infra network, to avoid overloading the tenant networks.
As an operator I want to use the extra headers ERSPAN provides (i.e.: original VLAN, original CoS in case of version 1 ERSPAN).

Proposed Change

The proposal is to use OVS and OVN builtin ERSPAN version 1 and GRE mirroring features. For using ERSPAN or GRE with OVS a port is added to an OVS bridge with type=erspan or type=gre in case of GRE.

As ERSPAN is a modification of GRE, with some extra ERSPAN specific headers (see ERSPAN draft from Cisco), OVS creates the tunnel from the previously created OVS port to the destination IP. This means that the mirrored traffic is encapsulated to an ERSPAN/GRE tunnel.

Example wireshark dump of an ICMP echo reply:

Frame 1: 148 bytes on wire (1184 bits), 148 bytes captured (1184 bits)
Ethernet II, Src: RealtekU_79:ff:db (52:54:00:79:ff:db), Dst: RealtekU_91:2f:52 (52:54:00:91:2f:52)
Internet Protocol Version 4, Src: 100.109.0.84, Dst: 100.109.0.142
Generic Routing Encapsulation (ERSPAN)
Encapsulated Remote Switch Packet ANalysis Type II
Ethernet II, Src: fa:16:3e:d5:4b:c1 (fa:16:3e:d5:4b:c1), Dst: fa:16:3e:80:ed:09 (fa:16:3e:80:ed:09)
Internet Protocol Version 4, Src: 10.0.0.39, Dst: 10.0.0.47
Internet Control Message Protocol

This means that the source IP of the tunnel is the IP of the host on which the ERSPAN port is created (in my virtual env it is 100.109.0.84). In the above example the 2 inner IPs (10.0.0.47 and 10.0.0.39) are the fixed IPs of 2 Openstack ports (VMs).

The outer protocol source IP (100.109.0.84) is the IP of the host on which the mirror port is created, thus outside of the cloud, and in the control of the admin.

Note

This spec doesn’t specify the source address (mac or IP) of the tunnel. So this is the task of the admin and out of scope for this spec.

The outer protocol destination IP (100.109.0.142) in this case is also outside of the cloud, and Openstack Neutron control, and another host on which I can run tcpdump.

REST API impact

The current API model of TAAS uses two high level objects: tap-services and tap-flows. The tap-flow represents the source of the mirrored traffic, and a tap-service represents the destination of the mirrored traffic. For one tap-service multiple tap-flows can be attached. For details please check the tap-as-a-service API reference. Both a tap-flow and a tap-service are referencing a Neutron port, and the traffic on that port will be the source of the mirror (in case of a tap-flow), or the destination of the mirror (in case of a tap-service).

Warning

Mirroring only works for traffic which is allowed by security-groups.

The above API model is not useful when the intent is to implement mirroring using ERSPAN or GRE tunnels, because the traffic source is a bridge port (in the case of OVS) or a logical switch port (in the case of OVN) and the destination is represented only by an IP address.

The proposal is to introduce a new high level API for ERSPAN or GRE mirroring: tap_mirror.

This solution keeps the current API clean and not overloaded, and makes it easier for operators to expect the right behaviour after API operations.

The proposed API is admin only, to avoid the overloading of infrastructure networks by tenants.

The suggested API request:

POST /v2.0/taas/tap_mirrors

Create a tap mirror that mirrors traffic from a Neutron port to an external IP:

{
    "tap_mirror": {
        "name": "mirror-traffic-of-server-a0",
        "description": "Mirror the traffic from server-a0",
        "directions": {"IN": 99, "OUT": 100},
        "port_id": "1a1a5a96-e8cb-11ed-9678-9b663820b519",
        "remote_ip": "172.31.1.1",
        "mirror_type": "erspanv1"|"gre"
    }
}

port_id is the source of the mirroring, this is a Neutron port. One port can be attached to one tap_mirror only.

Note

Only VM ports can be used as the source of the mirroring.

remote_ip: The IP of the remote end of the tunnel.

Note

This API proposal keeps the current TAAS API’s N-1 relationship between source and destination. Multiple source ports’ traffic can be mirrored to one destination IP.

mirror_type field is to select between ERSPAN and GRE.
directions is a dictionary with direction:tunnel_id pairs. The current tap-as-a-service API allows the operator to select the direction when the tap-flow is created, it can be: IN, OUT, BOTH. This specification proposes to keep IN and OUT as the keys of the dictionary. Meaning of the directions:
- IN: the traffic towards the port, and into the VM attached to it (ingress traffic).
- OUT: traffic from the port, out of the VM attached to the port (egress traffic).
The tunnel_id will be the value of the directions dictionary, this is the identifier of the ERSPAN or GRE session between the source and destination.

Note

There is a big difference in the GRE and ERSPAN id size: GRE has 32 bits key size but ERSPAN has only 10 bits for ERSPAN session ID. This must be documented and validated on the API.

Warning

It is not possible to create tunnel (GRE or ERSPAN) with the same tunnel_id to the same remote_ip from the same port with OVN. This means that the tunnel_id can’t be the same for the 2 directions.

Note

Both GRE and ERSPAN handle the fragmentation, so if the mirrored traffic’s packet size with the extra headers is bigger than the MTU on the interface, the packet in the tunnel will be sent fragmented.

Note

For the GRE type mirroring 8 octet extra header is added over IP headers.
For ERSPAN 8 octet is added for GRE, 8 octet is added for ERSPAN and an extra trailing 4 byte CRC is added, so in summary 20 octets extra header is added in this case.

The proposed API definition:

mirror_types_list = ['erspan', 'gre']

RESOURCE_ATTRIBUTE_MAP = {
    'tap_mirror': {
        'id': {
           'allow_post': False, 'allow_put': False,
           'validate': {'type:uuid': None}, 'is_visible': True,
           'primary_key': True},
        'name': {
            'allow_post': True, 'allow_put': True,
            'validate': {'type:string': None},
            'is_visible': True, 'default': ''},
        'description': {
            'allow_post': True, 'allow_put': True,
            'validate': {'type:string': None},
            'is_visible': True, 'default': ''},
        'port_id': {
            'allow_post': True, 'allow_put': False,
            'validate': {'type:uuid': None},
            'enforce_policy': True, 'is_visible': True},
        'directions': {
            'allow_post': True, 'allow_put': False,
            'validate': 'type:dict': {
                'IN': {'type:integer': None, 'default': None, 'required': False},
                'OUT': {'type:integer': None, 'default': None, 'required': False}},
            'is_visible': True},
        'remote_ip': {
            'allow_post': True, 'allow_put': False,
            'validate': {'type:ip_address': None},
            'is_visible': True},
        'mirror_type': {
            'allow_post': True, 'allow_put': False,
            'validate': {'type:values': mirror_types_list},
            'is_visible': True,},
    }
}

DB Impact

To persist the new tap_mirror in the DB, a new table tapmirrors is needed:

op.create_table(
    'tapmirrors',
    sa.Column('id', sa.String(length=db_const.UUID_FIELD_SIZE),
              primary_key=True, nullable=False),
    sa.Column('project_id', sa.String(
              length=db_const.PROJECT_ID_FIELD_SIZE), nullable=False),
    sa.Column('name', sa.String(length=db_const.NAME_FIELD_SIZE),
              nullable=True),
    sa.Column('description', sa.String(
              length=db_const.DESCRIPTION_FIELD_SIZE), nullable=True),
    sa.Column('port_id', sa.String(db_const.UUID_FIELD_SIZE),
              nullable=False, unique=True),
    sa.Column('directions', sa.String(255), nullable=False),
    sa.Column('remote_ip', sa.String(db_const.IP_ADDR_FIELD_SIZE)),
    sa.Column('mirror_type', mirror_type_enum, nullable=False)
)

This also means that a new TapMirror DB model will be added.

OVN driver for mirroring

The following shows the API call to create a tap_mirror based on version 1 ERSPAN and the corresponding backend changes:

$ # REST API operation
$ curl -g -i -X POST http://<host_ip:9696>/networking/v2.0/taas/tap_mirrors \
  -d '{"tap_mirror": {"name": "mirror1", "port_id": "54c4b09f-8b3d-4685-b66d-ce22c67956a9",
                      "directions": {"OUT": 42}, "remote_ip": "100.109.0.142",
                      "mirror_type": "erspan"}}'

$ # backend changes
$ sudo ovn-nbctl mirror-list
mirror_out_297b12c0-e9a5-11ed-9f90-07946c615270:
  Type     :  erspan
  Sink     :  100.109.0.142
  Filter   :  from-lport
  Index/Key:  42

$ sudo ovs-vsctl show
    Bridge br-int
        ...
        Port ovn-my_mirror2
            Interface ovn-mirror_out_297b12c0-e9a5-11ed-9f90-07946c615270
                type: erspan
                options: {erspan_idx="42", erspan_ver="1", key="42", remote_ip="100.109.0.142"}

Note

Using GRE is very similar: the OVN mirror type will be GRE and the OVS port type will be GRE.

Description of the above port options:

erspan_idx is a in hex, and it is the index field in ERSPAN header.
erspan_ver is the version, for version 1 erspan it is 1.
key is SpanID or Session ID field in the ERSPAN header.

Note

ERSPAN index field is not specified as OVN doesn’t allow to set it separately from the key-tunnel_id field. When OVN allows the setting of it, a future TAAS change can handle it.

Example packet headers from Wireshark:

Generic Routing Encapsulation (ERSPAN)
    Flags and Version: 0x1000
    Protocol Type: ERSPAN (0x88be)
    Sequence Number: 18
Encapsulated Remote Switch Packet ANalysis Type II
    0001 .... .... .... = Version: Type II (1)
    .... 0000 0000 0000 = Vlan: 0
    000. .... .... .... = COS: 0
    ...0 0... .... .... = Encap: Originally without VLAN tag (0)
    .... .0.. .... .... = Truncated: Not truncated (0)
    .... ..00 0000 0010 = SpanID: 42
    0000 0000 0000 .... .... .... .... .... = Reserved: 0
    .... .... .... 0000 0000 0000 0010 0000 = Index: 66

Index: 66: this is from ``erspan_idx=42``. Note that OVN's ``mirror-add``
command (and the OVN ovn-nb.ovsschema) accepts one `index` parameter and
uses that for OVS' ``key`` and ``erspan_idx`` parameters, but erspan_idx
is a hex value, so this is probably a bug or not fully considered thing
in OVN.

SpanId: 42: this is from ``key=42``

With OVN to mirror both ingress and egress traffic of the source port 2 mirrors must be created (as the OVN mirror can have only from-lport or to-lport as direction), and attached to the port (logical-switch-port), one with filter=from-lport and one with filter=to-lport.

Note

OVN implementation of mirroring allows the user to create a mirror by selecting a direction as filter with ovn-nbctl, or via ovsdb (see ovn-nb.ovsschema mirror table)

So if the user creates a tap_mirror with direction IN the filter will be to-lport, if OUT the filter will be from-lport and in case of both IN and OUT in the directions dictionary 2 mirrors will be created one with to-lport and one with from-lport.

The above means that in case of mirroring both ingress and egress traffic tap-as-a-service will create 2 ERSPAN or GRE ports on br-int for each tap_mirror.

OVS driver changes

To keep consistency between the 2 drivers, this specification proposes to use GRE and ERSPAN version 1 for OVS drive also.

The end-to-end call will look like this:

$ # REST API operation
$ curl -g -i -X POST http://<host_ip:9696>/networking/v2.0/taas/tap_mirrors \
  -d '{"tap_mirror": {"name": "mirror1", "port_id": "54c4b09f-8b3d-4685-b66d-ce22c67956a9",
                      "direction": "IN", "remote_ip": "100.109.0.142", "tunnel_id": "42",
                      "mirror_type": "erspan"}}'

$ # Backend changes
$ sudo ovs-vsctl show
    Bridge br-tap
        ...
        Port mirror_in_ed6046d
        Interface mirror_in_ed6046d
            type: erspan
            options: {erspan_idx="42", erspan_ver="1", key="2", remote_ip="100.109.0.84"}

$ sudo ovs-ofctl dump-flows  br-tap
   ...
   ... priority=20,dl_dst=fa:16:3e:d3:3a:d1 actions=output:"mirror_in_ed6046d"

For details on what the port properties key, erspan_idx mean, see OVN driver for mirroring .

For the details on how the case will be handled when both IN and OUT direction will be selected by the user see OVN driver for mirroring. OVS driver will create 2 OVS ports (type=erspan or type=gre) for the 2 directions like OVN does.

For the two directions 2 different flows will be installed on br-tap with different output port in the action field:

$ # Direction IN
$ sudo ovs-ofctl dump-flows  br-tap
...
... priority=20,dl_dst=fa:16:3e:d3:3a:d1 actions=output:"mirror_in_ed6046d"

$ # Direction OUT
$ sudo ovs-ofctl dump-flows  br-tap
...
... priority=20,dl_src=fa:16:3e:d3:3a:d1 actions=output:"mirror_out_ed6046d"

Out of Scope

This specification is not proposing to make the OVN driver fully compatible with the current OVS or SRIOV driver. So the proposed OVN driver will implement only ERSPAN.

To make OVN driver fully feature compatible with the current OVS or SRIOV driver can be part of a future specification.

Implementation

Assignee(s)

Lajos Katona (~lajoskatona) <lajos.katona@est.tech>, <katonalala@gmail.com>

Work Items

Add new REST API extension for tap-as-a-service, neutron-lib and tap-as-a-service changes.
Change tap-as-a-service db schema accordingly.
Adapt ovsdbapp to make it possible to manipulate both ovsdb and ovn-northd and create mirrors.
Change OVS driver.
Create a new ERSPAN only OVN tap-as-a-service driver.
Adapt the documentation.
Implement the necessary tests.
- end-to-end test in tempest can be done using Floating IPs.
Adapt OpenstackSDK and the necessary CLI code.
Adapt Heat to make it possible to create ERSPAN mirrors.

References

[OVN] - IPv6 Distributed routing

Fri, 16 Dec 2022 00:00:00

RFE: https://bugs.launchpad.net/neutron/+bug/1998609

This RFE intends to implement distributed routing support for IPv6 only or dual-stack usage scenarios. Distributed routing is already a reality for IPv4 FIP addresses and the benefits of implementing DVR for IPv6 are the same as for IPv4 FIP addresses.

Problem Description

When we are using an IPv6 only or dual-stack deployment, the traffic for IPv6 addresses remains centralized by the gateway port of the router. DVR support for IPv4 FIP addresses has already implemented in neutron using the enable_distributed_floating_ip flag in the [ovn] section - ml2_conf.ini file.

This issue affects both End Users and Deployers:

For End Users - (Network performance) Since traffic is centralized, all network applications need to go through the gateway router port chassis resident <-> compute node (via tunneling - e.g. Geneve) before reaching the endpoint. In case the gw router port is on the same chassis as the VM compute node, this is not applicable.
For Deployers - (Additional settings for dynamic routing) As the IPv6 GUA subnet is associated behind the router’s gw port, it is necessary to advertise this subnet dynamically to the border network element (which routes the external/internal traffic of IPv6 prefixes). It could be statically, but it is not feasible when we talk about large scale deployments. So, we need to enable additional settings like BGP with the neutron-dynamic-routing, for example.

If the compute node where the VM is running knows the path to send packets to external networks (with the help of routing protocols configured on each compute node, e.g. using FRR), outgoing traffic may work directly to the border. However, the incoming traffic will always be forwarded to the router’s gw port, as it is the only one that knows how to respond the Neighbor Solicitation to the IPv6 GUA address of the VM.

DVR Use case:

The provider networks must be stretched over the Underlay Network and each Compute Node would have the bridge for external traffic. In an L3 Leaf-Spine Underlay, for example, the network to be reached is for the Underlay Network be able to stretch an L2 domain(VLAN) via VXLAN as dataplane and BGP EVPN as Control Plane. In this solution, the Leaf switches would need to work as HW VTEP Gateways to initiate and terminate the VXLANs tunnels and use BGP EVPN to learn and advertise the MAC Addresses from the Compute Node’s provider network.

- E/W PATH:
  Compute Node VM <-> OVS br-int <-> br-provider <-> external-bridge-mapping
  <-> FRR <-> BGP EVPN type2 <-> E/W spine/leaf <-> [reverse flow to VM]
- N/S PATH:
  Compute Node VM <-> OVS br-int <-> br-provider <-> external-bridge-mapping
  <-> FRR <-> BGP EVPN type2 <-> N/S Border Leaf <-> external network

Proposed Change

To solve the problem described above, the proposal is to introduce a new NAT rule for the IPv6 GUA addresses that are allocated to VMs (OVN understands IPv6 GUA like a FIP). Even though it is a global address, the ovn-controller running on the chassis needs this rule to start responding to Neighbor Advertisements for IPv6 just like it does with GARPs for IPv4 FIP’s.

To enable the IPv6 DVR NAT rule management, a new config option should be enabled via [ovn] section in ml2_conf.ini file.

* ``enable_distributed_ipv6 = True``

This option is similar to a configuration option to enable DVR for FIP’s: enable_distributed_floating_ip.

OVN Impact

OVN support of distributed routing for an IPv6 GUA follows the same idea of the IPv4 case [1]. For the ovn-controller, the external_ip and local_ip can contain IPv4 or IPv6 addresses. Therefore, the contract rule for creating the IPv6 NAT rule remains the same used for FIP IPv4 addresses.

NAT rule fields (OVN):

type        : dnat_and_snat
logical_ip  : The same VM IPv6 GUA address
external_ip : The same VM IPv6 GUA address
logical_port: VM logical port
external_ids: Managed by Neutron
external_mac: VM MAC address

The external_ip and logical_ip used in the NAT rule are the same (e.g. VM GUA). With this entry, the OVN should add logical flows to respond to IPv6 Neighbor Solicitation requests for the IPv6 GUA directly by ovn-controller on the chassis where the VM resides.

We know that there is no NAT for IPv6, so this special rule that translates the GUA address to itself is only used to create the flows in the chassis where the VM resides. Without this rule, the compute node does not know how to respond to that IPv6 GUA and centralize the communication through the router’s GW port (via Geneve tunnel if the router resides on another host).

Neutron Impact

The NAT rules for FIP IPv4 are managed by Neutron floating ips database. In the case of the NAT rule for IPv6 GUA addresses, we need to create a “fake translation rule” for the address associated with a VM. IPv6 addresses are allocated to VMs during port creation and managed via ipv6_address_mode (static, SLAAC, DHCPv6 Stateful, or DHCPv6 Stateless).

This means that port creation and deletion events manage IPv6 addresses and MAC addresses allocated to a VM. In this case, the creation of the NAT rule for IPv6 GUA can follow the port creation and deletion process provided by ovn_client in the OVN driver.

The port structure provides the IPv6 address (subnet), the VM MAC address, and the logical_port but to create a NAT rule it is necessary to allocate this rule to the router, being necessary to search for the id of the router associated with the port of the VM.

IPv6 GUA NAT rule creation:

{
  args = {'type': 'dnat_and_snat',
          'logical_ip': IPv6_ADDRESS,
          'external_ip': IPv6_ADDRESS,
          'logical_port': PORT_ID,
          'external_ids': EXTERNAL_IDS,
          'external_mac': EXTERNAL_MAC}
  _nb_idl.add_nat_rule_in_lrouter(gw_lrouter_name, args)
}

IPv6 GUA NAT rule deletion:

{
  args = {'type': 'dnat_and_snat',
          'logical_ip': IPv6_ADDRESS,
          'external_ip': IPv6_ADDRESS}
  _nb_idl.delete_nat_rule_in_lrouter(gw_lrouter_name, args)
}

For IPv4 FIP NAT rules, the external_ids has important fields such as the fip_key, for example, but for IPv6 the external_ids information is not relevant. So, we can set common port information in that field to keep tracking. The neutron dnat_and_snat rule is composed with the following information in the external_ids:

{ external_ids = OVN_PORT_EXT_ID_KEY, OVN_DEVID_EXT_ID_KEY,
            OVN_NETWORK_NAME_EXT_ID_KEY, OVN_ROUTER_NAME_EXT_ID_KEY}
}

The common port information is stored in the OVN “NAT:external_ids” dictionary:

$ ovn-nbctl list NAT
 _uuid               : 4ab3ef03-c832-4635-bce9-779d1092fc89
 allowed_ext_ips     : []
 exempted_ext_ips    : []
 external_ids        : {"neutron:device_id"=
                                 "a467f63b-8159-4e87-83ce-d6b55ed4401f",
                        "neutron:network_name"=
                               neutron-b3110178-9798-4951-a681-2e12301001f8,
                        "neutron:port_id"=
                                 "9910cdd5-d764-403c-ae35-565930107c7a",
                        "neutron:router_name"=
                               neutron-a00e21fa-4332-4e26-8e34-73a24cbf46d1}
 external_ip         : "2001:db8:1234:1::33"
 external_mac        : "fa:16:3e:1d:31:57"
 external_port_range : ""
 logical_ip          : "2001:db8:1234:1::33"
 logical_port        : "9910cdd5-d764-403c-ae35-565930107c7a"
 options             : {}
 type                : dnat_and_snat

IPv6 DVR events

Neutron will configure the NAT rule for the IPv6 distributed routing reacting to the following events:

VM port creation: with this event, neutron create a NAT rule for the IPv6 GUA address allocated to the VM. If the VM does not have a router associated with the VM subnet, do not create the IPv6 NAT rule.
VM port deletion: same behavior as the VM port creation, this event is received and neutron must delete the NAT rule attached to the router. If the VM does not have a router associated with the VM subnet, do not delete the IPv6 NAT rule.
Router port creation: if this event is received, it is necessary to check if the added port is in the same subnet associated with the previously created VM port. This means neutron must create the IPv6 NAT rule if the router port is associated with the VM subnet.
Router port deletion: same behavior as the router port creation, this means neutron must delete the IPv6 NAT rule if the router port is associated with the VM subnet.

FIP IPv4 Impact

Until now, the only NAT rule of type ‘dnat_and_snat’ was used exclusively for FIP’s, therefore, the functions that search the OVN database for NAT rules of type ‘dnat_and_snat’, need to filter for exclusive FIP rules. One idea would be to create a new constant in external_ids to set the IP version, but it might be enough just to look for rules that have the OVN_FIP_EXT_ID_KEY fieldset.

The change in neutron OVN NAT lookup for FIP’s may require changes in existing tests for floating ips.

DB Impact

No new column entries or tables are needed to implement this RFE. IPv6 and MAC address information is already recorded on ports associated with VMs.

However, to ensure consistency of information between neutron and OVN, it is necessary to implement a check for IPv6 NAT rules during ovn_db_sync. For VMs already created before enabling ‘enable_distributed_ipv6’ flag, neutron will notify in the log about the config differences when it is restarted. The NAT rules for IPv6 GUA will be automatically added or removed when executing a SYNC_MODE_REPAIR.

Implementation

Assignee(s)

Primary assignees:: Roberto Bartzen Acosta <rbartzen@gmail.com>

Testing

Unit/functional tests.

Documentation Impact

User Documentation

Information about the IPv6 distributed routing support.

References

OVN Neutron Agent and hardware offloaded QoS extension

Sat, 10 Dec 2022 00:00:00

Launchpad bug: https://bugs.launchpad.net/neutron/+bug/1998608

The aim of the RFE is to define the architecture of a new generic OVN agent that will execute any needed functionality, being extensible as any other ML2 agent. The first functionality to be implemented will be the hardware offloaded QoS extension.

Problem Description

ML2/OVN is a mechanism driver that doesn’t have backend agents. This mechanism driver exposes two agent types: the OVN controller agent and the metadata agent. The OVN controller agent is a representation of the status of the ovn-controller service on the node; the metadata agent is the service that provides the metadata to the virtual machines.

However the ML2/OVN mechanism driver does not have an agent to perform the local Open vSwitch service configuration as in ML2/OVS, with extension drivers (like QoS or Trunk). The ovn-controller reads the OVN Southbound database to configure the local OVS database.

The lack of an agent running on the local node, owned by Neutron, prevents us from implementing some functionalities currently not supported by OVN nor the drivers. For example, and this will be the first feature that the OVN agent will implement, hardware offloaded ports cannot apply the OVS QoS rules due to limitations in the drivers, in particular the ones for the NVIDIA Mellanox ConnectX-5 network cards [0].

Proposed Change

This RFE proposes to create a generic agent running on the compute node. This agent will be called “OVN Neutron Agent”. The execution of this agent is discretionary and will be needed only if the specific features implemented on it are requested in a particular compute node. In other words, initially this service will be needed if a compute node has hardware offloaded ports and QoS is needed (new features could be implemented in the future).

Unlike other mechanism driver agents (like in ML2/OVS or ML2/SRIOV), this agent does not implement an RPC client. The information required to work will be retrieved from the local OVS database, the OVN Northbound database and the OVN Southbound database. It will be discussed in a future RFE to include RPC capabilities to have a direct connection to the Neutron server and the Neutron database.

Extensible functionalities

The new agent functionalities will be defined via extensions. It will use the neutron.agent.agent_extensions_manager.AgentExtensionsManager class to load the configured OVN Neutron Agent extensions. It will use the same configuration variable “agent.extensions” used by other ML2 agents. The OVN Neutron Agent extensions will inherit from neutron_lib.agent.extensions.AgentExtension and will be loaded during the agent startup by the agent extension manager.

As commented before, the OVN agent will have active OVN Northbound, OVN Southbound and local OVS database connections, using the corresponding IDL classes. These classes will monitor a certain set of tables and will register a set of events. Each extension must report what tables and events are needed during the startup process. The extensions will inherit from a specific OVN Neutron Agent extension class (derived from the mentioned AgentExtension) that will provide API methods to retrieve this information.

OVN agent status report

This new OVN Neutron Agent needs to be tracked in the Neutron server. There are currently two types of agents: the “OVN Controller agent”/”OVN Controller Gateway agent” and the “OVN metadata agent”. This RFE will introduce the “OVN Neutron Agent” type. This is a generic name matching the goal of this spec that is to create a generic OVN agent to implement all needed functionalities that the OVN controller cannot provide (metadata will be the next feature to be implemented there, migrated from the current “OVN metadata agent”).

The reporting mechanism will be the same as the one implemented in the OVN metadata agent. When the Southbound table “SB_Global” is updated, the agent will update a key in the “external_ids” dictionary. Please check neutron.agent.ovn.metadata.agent.SbGlobalUpdateEvent for an implementation example. This variable will be read by the server that will check the timestamp and determine the agent status.

QoS for hardware offloaded ports

The first feature (extension) that will be implemented within this agent is the QoS extension for hardware offloaded ports. Because of the current drivers limitations, these ports cannot be properly configured with the current QoS rules.

This RFE proposes to use the corresponding “ip-link” [1] commands (or their “pyroute2” equivalents) to configure the port representor virtual function rates (maximum and minimum). This is similar to what is done in ML2/SRIOV for a virtual function (VF).

A port representor (or a devlink port [2]) is an API to expose the device information; it is not a physical port but a representative of a hardware port. The devlink port is connected to the local OVS switch; using the “devlink” tool (or the “pyroute2” equivalent), it is possible to retrieve the port information, including the physical function (PF) PCI address. E.g.:

$ devlink port show enp4s0f1_5 -jp
 "port": {
     "pci/0000:04:00.1/65542": {
         "type": "eth",
         "netdev": "enp4s0f1_5",
         "flavour": "pcivf",
         "pfnum": 1,
         "vfnum": 5,
         "splittable": false,
         "function": {
             "hw_addr": "fa:16:3e:f8:7b:10"
         }
     }
 }

With the PF PCI address, it is possible to retrieve the PF name, that will be stored in:

$ cat /sys/bus/pci/devices/$pf_pci_address/net

Using the ML2/SRIOV class PciDeviceIPWrapper, it is possible to set the maximum and minimum rates of a VF, knowing the VF index and the PF name.

OVN QoS information location

The OVN QoS information is stored in two different places (due to how QoS is currently implemented in ML2/OVN and core OVN):

The maximum bandwidth (“rate”) and maximum burst (“burst”) limits are stored in the “Qos:bandwidth” dictionary:

$ ovn-nbctl list Qos
 _uuid               : 376303c4-5290-4c4e-a489-d35894315199
 action              : {}
 bandwidth           : {rate=1000, burst=800}
 direction           : to-lport
 external_ids        : {"neutron:port_id"="89a81cc0-7b3e-473d-8c01-2539cf2a9a6a"}
 match               : "outport == \"89a81cc0-7b3e-473d-8c01-2539cf2a9a6a\""
 priority            : 2002

The minimum bandwidth rate is stored in the “Logical_Switch_Port:options” dictionary:

$ ovn-nbctl list Logical_Switch_Port
 _uuid               : 502b3852-4a46-4fcb-9b49-03063cfc0b34
 addresses           : ["fa:16:3e:c4:b8:29 10.0.0.8"]
 dhcpv4_options      : a7e4cfb1-ef22-490a-9ffe-fea04de3fe1c
 dhcpv6_options      : []
 dynamic_addresses   : []
 enabled             : true
 external_ids        : {...(skipped)}
 ha_chassis_group    : []
 name                : "e6808371-c9ac-4015-94a3-7f16ac3fbb2d"
 options             : {mcast_flood_reports="true", qos_min_rate="1000",
                        requested-chassis=u20ovn}
 parent_name         : []
 port_security       : ["fa:16:3e:c4:b8:29 10.0.0.8"]
 tag                 : []
 tag_request         : []
 type                : ""
 up                  : true

OVN QoS events

The hardware offloaded QoS extension will configure the QoS setting on a port reacting to the following events (please check the working POC [3] for more information):

The local OVS interface creation: with this event, the OVN monitor will store what ports are bound to the local instance. It will store the Neutron port ID (stored in the “Interface.external_ids:iface-id” string) and the OVS port name.
The local OVS interface deletion: this event will trigger the QoS reset and the interface local cache deletion.
The OVN Southbound “Port_Binding” creation event: this event is received after a port is created in a local OVS. If this event is received, that will trigger the QoS update of the local port. It’s worth mentioning that this event happens always after the local OVS interface creation; that means the OVN monitor has already registered that this port is bound locally.
The OVN Northbound “Logical_Switch_Port” register change: if minimum bandwidth of a locally bound LSP changes, this event triggers the QoS update.
The OVN Northbound “QoS” register change: similar to the previous one but affecting the maximum bandwidth limit.

REST API Impact

This RFE does not introduce any API change.

Data Model Impact

This RFE does not introduce any model change.

Security Impact

None.

Performance Impact

Each monitor will have a connection to the local OVS database and the remotes OVN Northbound and Southbound databases. The connection to the remote OVN databases can have a severe impact on the load of the OVN database node (that are usually the OpenStack controllers). This initial implementation will subscribe to the following tables:

Northbound: QoS, Logical_Switch_Port and Logical_Switch
Southbound: Chassis, Encap, Port_Binding, Datapath_Binding and SB_Global

This performance impact should be reduced by:

Reducing the number of agents on the node. The next step to be implemented (out of scope in this RFE) is to move the OVN metadata agent functionality to this new OVN Neutron agent. That will reduce the Southbound database connections to one single agent.
Find a way to store the minimum bandwidth information outside the Logical Switch Port. By not subscribing to this table, the Northbound connection will reduce the load notably. The Logical Switch Port is one of the most populated and active ones; not locally caching nor receiving its updates will reduce the impact on the OVN database.

Other Impact

None.

Implementation

Assignee(s)

Primary assignees:: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> (IRC: ralonsoh)

Work Items

OVN monitor implementation and the hardware offloaded QoS extension.
Tests and CI related changes.

Testing

Unit/functional tests.

Note

The hardware offloaded QoS extension requires specific hardware to test this feature. Currently is not possible to implement any tempest test on the CI.

Documentation Impact

User Documentation

Information about how to configure and spawn the OVN monitor.
Information about the hardware offloaded QoS extension.

References

Strict minimum bandwidth support for tunnelled networks

Mon, 03 Oct 2022 00:00:00

Launchpad bug: https://bugs.launchpad.net/neutron/+bug/1991965

The aim of the RFE is to improve the previous implemented RFE Strict minimum bandwidth support [1].

Problem Description

Since [1], Neutron has the ability to model the available bandwidth of the physical interfaces (ingress and egress) connected to the physical networks (flat and VLAN networks). This information is collected by Placement and used to spawn virtual machines with network ports on compute nodes with enough bandwidth. That guarantees a minimum port throughput.

This feature is currently implemented in three backends: ML2/SR-IOV, ML2/OVS and ML2/OVN. The full list of related patches can be reviewed at [2].

However, most of the current deployments do not use physical backed networks (flat or VLAN) but overlay networs (VXLAN and Geneve). Of course those deployments use ML2/OVS and ML2/OVN; ML2/SR-IOV does not support tunnelled networks. That leads to an existing gap in the currently implemented feature: there is no way to model tunnelled networks.

Proposed Change

This RFE proposes a way to model the available bandwidth for tunnelled networks in compute nodes. This implementation will be limited to ML2/OVS and ML2/OVN backends.

The referred backends handle the overlay traffic sending and receiving this traffic from a host interface, that acts as a VTEP [3]. This host interface is identified by an IP address, known as “local_ip” in the ML2 plugin configuration file [4].

This RFE proposes to use the same configuration options provided in [1], adding a static string constant to define a resource provider in Placement that could be configurable by the administrator. This string will be the suffix of the resource provider name, same as Neutron uses the physical bridge names to build their resource provider names. For example “u20ovn:OVN Controller agent:rp_tunnelled”, being “rp_tunnelled” the provided string. The default value will be “rp_tunnelled”. This configuration variable will be stored in the [ml2] section and will be accesible from the Neutron server and the OVS agent:

[ml2]
tunnelled_network_rp_name = custom_rp_name  # "rp_tunnelled" by default.

Example of ML2/OVS configuration section:

[ovs]
resource_provider_bandwidths = br0:EGRESS:INGRESS,rp_tunnelled:EGRESS:INGRESS

Example of ML2/OVN configuration, stored in the local database of the OVS service:

root@u20ovn:~# ovs-vsctl list open_vswitch .
...
external_ids : {hostname=u20ovn,
                ovn-bridge=br-int,
                ovn-bridge-mappings="public:br-ex",
                ovn-cms-options="enable-chassis-as-gw,
                                 resource_provider_bandwidths=br-ex:4001:5001;rp_tunnelled:4001:5001,
                                 resource_provider_inventory_defaults=allocation_ratio:1.0;min_unit:5,
                                 resource_provider_hypervisors=br-ex:u20ovn;rp_tunnelled:u20ovn",
                ovn-encap-ip="192.168.10.40",
                ovn-encap-type=geneve,
                ovn-remote="tcp:192.168.10.40:6642",
                system-id="a55c8d85-2071-4452-92cb-95d15c29bde7"}

Note that in ML2/OVN, it is mandatory to define the tunnelled resource provider assignation to the host in the “resource_provider_hypervisors” list.

This new string constant cannot be used as a physical bridge name. To avoid any possible clash, there will be a new check when parsing the physical network bridge mappings.

A host with ML2/OVN backend with a physical network (mapped to the physical bridge “br-ex”) and a tunnelled network will report the following resource provider tree:

$ openstack resource provider list
+--------------------------------------+------------------------------------------+------------+--------------------------------------+--------------------------------------+
| uuid                                 | name                                     | generation | root_provider_uuid                   | parent_provider_uuid                 |
+--------------------------------------+------------------------------------------+------------+--------------------------------------+--------------------------------------+
| 8f0e060d-bf63-42a1-85e6-710c8b2fccc8 | u20ovn                                   |         10 | 8f0e060d-bf63-42a1-85e6-710c8b2fccc8 | None                                 |
| cb101b60-527b-5264-8e7f-213c7b88e9e1 | u20ovn:OVN Controller agent              |          1 | 8f0e060d-bf63-42a1-85e6-710c8b2fccc8 | 8f0e060d-bf63-42a1-85e6-710c8b2fccc8 |
| 521f53a6-c8c0-583c-98da-7a47f39ff887 | u20ovn:OVN Controller agent:br-ex        |         20 | 8f0e060d-bf63-42a1-85e6-710c8b2fccc8 | cb101b60-527b-5264-8e7f-213c7b88e9e1 |
| dfdbf43f-f60b-577c-bae8-3dcea448c735 | u20ovn:OVN Controller agent:rp_tunnelled |          6 | 8f0e060d-bf63-42a1-85e6-710c8b2fccc8 | cb101b60-527b-5264-8e7f-213c7b88e9e1 |
+--------------------------------------+------------------------------------------+------------+--------------------------------------+--------------------------------------+

A new static trait will be added to represent this resource provider: “CUSTOM_NETWORK_TUNNEL_PROVIDER”. This is what identify that this resource provider is for tunnelled networks. E.g.:

$ openstack resource provider trait list $rp_tun
+----------------------------------+
| name                             |
+----------------------------------+
| CUSTOM_VNIC_TYPE_NORMAL          |
| CUSTOM_VNIC_TYPE_DIRECT          |
| CUSTOM_VNIC_TYPE_DIRECT_PHYSICAL |
| CUSTOM_VNIC_TYPE_MACVTAP         |
| CUSTOM_VNIC_TYPE_VDPA            |
| CUSTOM_VNIC_TYPE_REMOTE_MANAGED  |
| CUSTOM_VNIC_TYPE_BAREMETAL       |
| CUSTOM_NETWORK_TUNNEL_PROVIDER   |
+----------------------------------+

This is an example of a port resource request, sent to Nova when creating a virtual machine. The port has a minimum bandwidth rule of 500 kbps, egress direction:

port['resource_request'] = {
    'request_groups': [
        {'id': 'c51c6f07-8e01-548c-9756-d5e54a780bb6',
         'required': ['CUSTOM_NETWORK_TUNNEL_PROVIDER', 'CUSTOM_VNIC_TYPE_NORMAL'],
         'resources': {'NET_BW_EGR_KILOBIT_PER_SEC': 500}
        }
    ],
    'same_subtree': ['c51c6f07-8e01-548c-9756-d5e54a780bb6']
}

Note

This spec is not considering the case of shared resource providers. For example when the same interface is shared between a VLAN/flat network and and overlay network. What this spec is proposing is to provide the scheduling functionality to ports in overlay networks. In case of having shared resources, the administrator will need to split bandwidth assignation between resource providers. Currently Placement API nor Neutron cannot provide a way to model a shared resource.

REST API Impact

This RFE does not introduce any API change.

Data Model Impact

This RFE does not introduce any model change.

Security Impact

None.

Performance Impact

None.

Other Impact

Currently there is no support for minimum bandwidth QoS rules for tunnelled networks, neither in Placement nor in the ML2 backend (OVS, OVN). However, it is possible to have ports with those type of QoS rules (maybe inherited from the network QoS policy). With this feature, the minimum bandwidth QoS rules won’t be discarded, like now, when the port resource request is built (that is the Placement blob to request a specific bandwidth in a specific network).

A new check will be added to inform about those ports located on tunnelled networks with minimum bandwidth QoS rules. The output of this check will be a log with the list of ports, their networks and QoS policies. This spec is considering the current Neutron implementation:

ML2/OVS rejects the assignation of a QoS policy with minimum bandwidth rules and prevents from binding a port with them.
The ML2/OVN mechanism driver implemented the minimum bandwidth rule support recently and does not prevent this scenario. However this functionality was implemented in Zed release; it is unlikely that many deployments are in this state (with ports located in overlay networks with QoS policies and minimum bandwidth rules).

This spec does not consider the rebuild of the current allocations. Any port already present in a host that creates a new resource provider for tunnelled networks, won’t be allocated. Once there is standard a procedure to perform this action, a new spec/bug will be created to track this improvement, but this is out of scope in this spec.

Part of this RFE will be to document the alternatives the user has to, in case of having a port with minimum bandwidth rules before enabling this feature, create the needed allocations:

Live migrate the VM with the port. That will trigger the Placement scheduling and the allocation creation.
Detach and attach again the port to the VM. That will have the same effect. This functionality was added to Nova in Wallaby.

Implementation

Assignee(s)

Primary assignees:: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> (IRC: ralonsoh)

Work Items

ML2 plugin update.
Migration script to log those existing ports with minimum bandwidth rules in tunnelled networks.
Documentation, including the methods to re-created the allocations for pre- created ports.
Tests and CI related changes.

Testing

Unit/functional tests.
Fullstack tests: increase the current fullstack tests coverage to check this new feature.
Tempest tests: create a VM with a minimum bandwidth port, update the QoS policy and minimum bandwidth rule limits, unset the QoS policy, migrate the VM.

Documentation Impact

User Documentation

Ammend the “Strict minimum bandwidth support” [1] documentation, adding this new improvement.

References

Metadata rate limit

Fri, 09 Sep 2022 00:00:00

Launchpad Bug: https://bugs.launchpad.net/neutron/+bug/1989199

The metadata agents currently do not limit the number of requests they try to process. Mis-behaved instances repetitively querying the metadata endpoint can incur a high load on services above the metadata-agents (Nova and Neutron).

Problem Description

Platform administrators would benefit from being able to rate-limit requests handled by metadata in order to protect other OpenStack components from DoS.

Rate-limiting should be configurable via the config files
Requests should be rate-limited by source IP
The default settings for the rate-limiting should not impact the normal provisioning of an instance (eg. through cloud-init)

Proposed Change

The metadata agent (both in the case of OVN and OVS/linuxbridge) receive instances requests through an haproxy process it configures.

We could leverage haproxy’s stick-tables to implement rate-limiting, similar to what is done in this haproxy configuration example [1]. Concretely, requests coming-in from IPs that have a request rate above a configured threshold would receive 429s from haproxy, without hitting the metadata-agent.

As far as configuration is concerned, we would introduce some new directives to enable the rate-limiting and to configure the thresholds and time-window sizes.

In order to accommodate some events that normally occur during the life of an instance (for example: cloud-init, periodic refresh of the network-metadata), we could make the rate-limit burstable, for example: limit the request rate to 30 over 60s and to 10 over 5s. This could be implemented by creating two stick-tables holding the request rates over a long base window and over a short burst window and by denying requests coming from an IP when its request rate crosses either the base or the burst threshold.

Impact

Configuration impact

Add new configuration directives to enable and control the rate-limiting:

rate_limit_enabled (bool): Whether or not to enable request rate-limiting.
base_window_duration (seconds): Duration of the base window
burst_window_duration (seconds): Duration of the burst window
base_query_rate_limit (short): Limit of the query rate expressed over the duration of the base window.
burst_query_rate_limit (short): Limit of the query rate expressed over the duration of the burst window.

Metadata’s haproxy impact

Add two backends meant to hold the stick-tables storing http_req_rate per ip for the base window and for the burst window.
In the listener, deny requests with 429 if src_http_req_rate is greater than the configured threshold of either the base or burst limits.
In the listener, track requests in both the base and burst stick-tables.

Implementation

Assignee(s)

Guillaume Espanel <guillaume.espanel@gmail.com>

Testing

Unit tests to ensure haproxy’s configuration is built according to the config parameters.

References

Firewall Group Ordering on Port Association

Fri, 29 Jul 2022 00:00:00

https://bugs.launchpad.net/neutron/+bug/1979816

Currently, packets will sometimes be passed, and other times be blocked, depending on the ordering of groups applied to a port. This is contrary to the existing FWaaS spec, which states that a packet will be allowed so long as any group on the port would allow the packet.

Problem Description

When multiple firewall groups are applied to a port, the order in which the groups are evaluated can change whenever one of the groups is modified. Therefore, the combined firewall ruleset that results from multiple firewall groups is rearranged unintentionally. This can result in certain traffic being allowed or denied when the opposite behavior would be intended.

Proposed Change

Similar to firewall_policy_rule_associations_v2, the firewall_group_port_associations_v2 table should have a required position column to maintain the order in which firewall groups are applied to ports.

In addition, modification of this ordering should be limited by user role. For example, an openstack administrator may want a particular group to always be applied first or last, regardless of which groups are added to a port by a tenant. In iptables, this is typically referred to as HEAD and TAIL rules. All HEAD groups should be applied first, in order. All TAIL groups should be applied last, in order. All other groups would be applied in between, again, in order. Only openstack users with the admin role should have access to the HEAD and TAIL tiers by default.

Ex.

firewall_group_id	port_id	position	tier
da4be831-907b-43d9-86e0-b14a3bd391fc	efb7d60e-d3fc-4f97-91ed-ca71d930bb7c	1	HEAD
0814e179-d2be-464a-a9d4-e13c94451532	efb7d60e-d3fc-4f97-91ed-ca71d930bb7c	2	HEAD
33ce9937-d9db-48b8-a65d-05fa3a75844a	efb7d60e-d3fc-4f97-91ed-ca71d930bb7c	1	null
6b3172af-9ae0-40e4-b455-c70de7c80c24	efb7d60e-d3fc-4f97-91ed-ca71d930bb7c	2	null
70a7087e-c6ae-4cef-9b30-35e702746b68	efb7d60e-d3fc-4f97-91ed-ca71d930bb7c	1	TAIL
ff1e5eda-c285-4ec2-80f8-49f1a6d77347	efb7d60e-d3fc-4f97-91ed-ca71d930bb7c	2	TAIL

Position should auto-increment if the position keyword is not specified. If the position keyword is specified, and that number is available, that number is used. If the number is already used, the existing groups are shifted downward from that point, and the new group is applied in its place. For example, if positions 1-5 are in use, and position 2 is added, the table would be updated as follows:

position	new position
1	1
2	New Group (2)
2	3
3	4
4	5
5	6

REST API Impact

PUT and POST types for /v2.0/fw/firewall_groups will be updated to support the addition of position and tier.

Response bodies should include the new fields.

# Create (POST)
{
    "firewall_rule": {
        "ports":[
             "8722e0e0-9cc9-4490-9660-8c9a5732fbb0"
         ],
        "name": "FW_GROUP_1",
        "position": 2,
        "tier": "HEAD"
    }
}

# Update (PUT)
{
    "firewall_rule": {
        "ports":[
             "8722e0e0-9cc9-4490-9660-8c9a5732fbb0"
         ],
        "name": "FW_GROUP_1",
        "position": 3,
        "tier": "TAIL"
    }
}

2. GET requests for both list and show methods should include the new values in their responses.

# List/Show (GET) Response
{
    "firewall_groups": [
        {
            "description": "",
            "ingress_firewall_policy_id": null,
            "egress_firewall_policy_id": null,
            "id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
            "name": "FW_GROUP_1",
            "project_id": "45977fa2dbd7482098dd68d0d8970117",
            "ports":[
                 "8722e0e0-9cc9-4490-9660-8c9a5732fbb0"
             ],
            "position": 3,
            "tier": "TAIL"
        }
    ]
}

Data Model Impact

position and tier are to be added to the firewall_group_port_associations_v2 table.

Existing entries should be assigned consecutive position numbers starting at 1, and the default tier value of null.

Firewall Group Port associations

Attribute	Type	Req	CRUD	Description
position	integer	Y	CRU	Position at which this firewall group is evaluated
tier	String	Y	CRU	Tier at which this firewall group exists (HEAD, TAIL, null) Default: null

References

https://etherpad.opendev.org/p/fwaas-api-evolution-spec https://specs.openstack.org/openstack/neutron-specs/specs/newton/fwaas-api-2.0.html

Floating IP distributed

Mon, 27 Jun 2022 00:00:00

Launchpad Bug: https://bugs.launchpad.net/neutron/+bug/1978039

Neutron adds distributed attributes to each Floating IP. Users can set this attribute according to their actual environment and use requirements.

Problem Description

Neutron supports setting the floating IP to distributed. External traffic can go directly to the compute node without passing through the network node, and increases the network performance. This is very useful for the demand for high-performance networks.

However, we can decide whether floating ips are distributed or centralized by setting configuration option(ovn backend) or setting router’s “distributed” attribute(ovs dvr mode). This is global. After configuration, all floating ips are centralized or distributed.

The actual use may be complicated:

Not all vms with floating ip may require high-performance networks, and may only be exposed to the outside world to provide services.
Due to equipment conditions, some compute nodes are not equipped with external network card. If enable distributed, the vm of this compute node cannot allocate floating ip.

Proposed Change

Add new API extension to extend floatingip resource in Neutron with distributed attribute. If not set this attribute, the distributed attribute of the router will be used for the floating ip [1].

Server side changes

A new API extension of Neutron will be added with new attribute for the floatingip resource. This new attribute will be called distributed:

RESOURCE_ATTRIBUTE_MAP = {
    l3.FLOATINGIPS: {
        "distributed": {
            'allow_post': True,
            'allow_put': True,
            'convert_to': converters.convert_to_boolean_if_not_none,
            'default': constants.ATTR_NOT_SPECIFIED,
            'is_visible': True,
            'is_filter': True
        }
    }
}

DB Impact

Extend the floatingips table with boolean column distributed.

REST API Impact

New API extension: floatingip-distributed introducing new floatingip attribute: distributed.

Client Impact

Relevant changes in OSC and openstacksdk to add support for new floatingip’s attribute. To enable it for floatingip, it should be something like:

openstack floating ip create --distributed

and to disable it:

openstack floating ip create --centralized

and update it:

openstack floating ip update [--distributed | --centralized ]

Testing

Unit tests.
Functional test.
Tempest tests in neutron-tempest-plugin.

Assignee(s)

ZhouHeng <zhouhenglc@inspur.com>

References

Multiple routed provider segment per host

Thu, 17 Mar 2022 00:00:00

https://bugs.launchpad.net/neutron/+bug/1764738

The proposed spec is to extend the current feature routed provider networks [0] to allow provisioning more than one segment per physical network.

There is currently a limitation for a compute node to only have an interface on one segment in a multisegment network. Operators that want to extend IP range will have to provision new networks which degrade user experience, or increase the broadcast domain which does not scale.

Problem Description

As an operator I want to extend IP pool without creating multiple networks. I also want to limit the brodcast domain.

Proposed Change

The change is limited to OVS agent, it will be still possible to implement it for linux-bridge. That is said, this implementation is out of the scope of the proposed spec.

The main purpose is to add the support for an interface of a compute node to be attached to more than one segment of a given network.

This implies changes on different components. The DHCP agent currently only supports one interface per network. It will be necessary to extend it to support more domains per network. The VLAN manager that maintains details of VLAN ports for OVS by using network id will have to be updated to use segment id instead. The agent will have to be updated to support the feature and finally the limitation that rejects such case will have to be removed.

VLAN Manager

The current implementation for VLAN manager that maintains internal VLAN mapping to external network segmentation ids is using the network ids as principal identifier [1]. This will have to be updated to use a combination of network ids, segment ids. This should not implies upgrade impact as they are in-memory and recalculated after restarts.

DHCP Agent

Using multiple segments per network implies more than one network domain. The current DHCP agent supports one domain per network [2].

A DHCP port is plugged to the internal bridge using the local VLAN provided by the VLAN manager for a given network. This means that the DHPC Process running can address reauest for one domain. To reflect the current desire of multiple domains per network, a DHCP port should be plugged to br-int per combination of network id / segmentation id.

The proposed implementation is to avoid changing DHCP Process logic to make it supporting more than one interface per VLAN as it will add complexity and may imply upgrade impact. Instead, it has been considered to keep the well trusted logic of the DHCP Process to use one per segmentation id. One may in future consider to revisit this implementation.

The benefit of having one DHCP Process running per local VLAN will limit the changes and avoid upgrade impact. For a deployment already running routed network provider, two DHCP process will be running. One considered as legacy running under /net-id/pid, and an other running at /seg-id/net-id/pid. It will involve the operator to remove the legacy port using opentack API.

Limitations

The proposed spec only implies OVS support but it should be possible to provide the same kind of support for linux-bridge. A change has been proposed with [3] with related spec [5] It implies an upgrade impact which can be mitigated like for this proposed OVS implementation. Also that the linux-bridge implementation can take benefit of the change in DHCP agent which is missed in the proposed change for linux-bridge.

For a given fixed ip, during scheduling Nova will not be able creating a port a the segment related to the subnet of the fixed ip. It’s an already known limitation. Bug 1979959 [9] has been reported to address the issue.

A regression has been noted since the bug 1952730 [8], with routed provider segments created are not taken into account. Bug 1979958 [7].

Notes

In its process Nova needs to retrieve network details [4]. In case of a given network with multiple segments the process will use the first segment that it discovers with a physcal_network. This is a design issue that is already known. The process change is not expected to fix it.

To have Nova schedule instances on hosts based on segments where ports are attached to. Aggregates are created per segment [6]. The process is using segment ids as names. It is still expected to continue creating aggregates with the new segment created for a given network. This should not bring any issue.

Implementation

Primary Assignees

Sahid Orentino Ferdjaoui

Distributed Router Advertisement on Openvswitch Agent

Tue, 08 Mar 2022 00:00:00

RFE: https://bugs.launchpad.net/neutron/+bug/1961011

Neutron L3 router will run radvd to send out RA (Router Advertisement) packets to notify the guests about the subnet ManagedFlag, LinkMTU and prefix of IPv6 subnet. This is fine to work for small scale clouds, or long stand clouds with no changes. But for large scale cloud environments, management complexity increases.

Neutron openvswitch agent can be considered as a distributed SDN controller, this spec describes how to make RA work on each compute node in distributed mode without any extra processes.

Problem Description

radvd is spawned in L3 router namespaces, which will multicast a RA on the subnet from the router interface, reaching all attached VMs. Current radvd config looks like this:

interface qr-8fd65326-c4
{
   AdvSendAdvert on;
   MinRtrAdvInterval 30;
   MaxRtrAdvInterval 100;
   AdvLinkMTU 1500;
   AdvManagedFlag on;
   prefix fda7:a5cc:3460:3::/64
   {
        AdvOnLink on;
        AdvAutonomous off;
   };
};

In order to do such work, L3 agent needs to do: * start a process manager for radvd * render the config for radvd * monitor the extra process in case of unexpected exit * respawn the process if something changes

For a DVR router, radvd will be run in every local router namespace on the compute node as well.

In certain cases, this increases the number of server processes, increases the resource consumption of the server, increases the failure point and reduces the processing performance of the L3 agent.

There is a defect of current RA mechanism. It forces the subnet to have association with a Neutron router, if you want Neutron to manage the IPv6 related configurations [2]. Then if a subnet does not connect to a router, no radvd answers the RS request, no RA back. VMs under isolated networks will get failed to find out how to do with IPv6 address, how to configure IPv6 address, how to do with DHCPv6.

So, this agent extension will make isolated networks can work for IPv6. VM under it will get RA about the IPv6 subnets prefixes, autonomous address configuration flag, managed address configuration flag and other configuration flag and so on. VMs under the isolated networks are going to work with IPv6 based on these flags.

Proposed Change

Adds an agent extension for Neutron openvswitch agent to send RA out to local guests in a more natural and graceful style.

Note

This extension is only for openvswitch agent, other mechanism drivers will not be considered, because this new extension will rely on the openflow protocol and principle.

Solution Proposed

The new agent extension will use os-ken(ryu) [1] to assemble the RA packets, and then directly do packet-out to the ofport (VM port). Mission accomplished, very simple! Some main code procedure will be:

new extension initialized with a subnet cache list
start a periodic loop to send RA to each port from those subnets
in handle_port method each VM port’s subnet will be added to the cache
in delete_port remove the VM port from subnet’s port cache
get subnet information by the resource cache RPC
looping of sending RA to each port

Openvswitch agent will pull ports, subnets and networks information by “resource cache” related RPC.

Note

For some cases, neutron router will interact with the upstream physical router. This extension will not handle such configurations and deployment architecture.

The attribute ipv6_ra_mode of subnet is None, then Neutron will not generate RA to VMs. ipv6_ra_mode = None will be used to indicate if there are external (physical) routers.

Implementation

Assignee(s)

LIU Yulong <i@liuyulong.me>

Work Items

Adding ovs-agent extension to do the RA work.
Make L3-agent radvd configurable.
Testing.
Documentation.

Dependencies

None

Testing

Test cases to verify the RA can work for ports.

References

Off-path SmartNIC DPU Port Binding with OVN

Thu, 17 Feb 2022 00:00:00

https://blueprints.launchpad.net/neutron/+spec/off-path-smartnic-dpu-port-binding-with-ovn

Off-path SmartNIC DPUs introduce an architecture change where network agents responsible for NIC switch configuration and representor interface plugging run on a separate SoC with its own CPU, memory and that runs a separate OS kernel. The side-effect of that is that hypervisor hostnames no longer match SmartNIC DPU hostnames which are seen by ovs-vswitchd and OVN [3] agents while the existing port binding code relies on that. The goal of this specification is to introduce changes necessary to extend the existing hardware offload code to cope with the hostname mismatch and related design challenges while reusing the rest of the code. To do that, PCI(e) add-in card tracking is introduced for cards with unique serial numbers so that it can be used to determine the correct hostname of a SmartNIC DPU which is responsible for a particular VF. Additionally, more information is suggested to be passed in the “binding:profile” during a port update to facilitate representor port plugging.

Nova spec: https://review.opendev.org/c/openstack/nova-specs/+/787458

Problem Description

Terminology

Data Processing Unit (DPU) - an embedded system that includes a CPU, a NIC and possibly other components on its board which integrates with the main board using some I/O interconnect (e.g. PCIe);
Off-path SmartNIC DPU architecture [1] [2] - an architecture where NIC cores are responsible for programming a NIC Switch and are bypassed when rules programmed into the NIC Switch are enough to make a decision on where to deliver packets. Normally, NIC cores only participate in packet forwarding for the “slow path” only and the “fast path” is handled in hardware like an ASIC;
On-path SmartNIC DPU architecture [1] [2] - an architecture where NIC cores participate in processing of every packet going through the NIC as a whole. In other words, NIC cores are always on the “fast path” of all packets;
NIC Switch (or eSwitch) - a programmable embedded switch present in various types of NICs (SR-IOV-capable NICs, off-path SmartNICs). Typically relies on ASICs for packet processing;
switchdev [4] - in-kernel driver model for switch devices which offload the forwarding (data) plane from the kernel.
Representor ports [5] - a concept introduced in the switchdev model which models netdevs representing switch ports. This applies to NIC switch ports (which can be physical uplink ports, PFs or VFs);
devlink [6] - a kernel API to expose device information and resources not directly related to any device class, such as chip-wide/switch-ASIC-wide configuration;
PCI/PCIe Vital Product Data (VPD) - a standard capability exposed by PCI(e) endpoints which, among other information, includes a unique serial number (read-only, persistent, factory-generated) of a card shared by all functions exposed by it. Present in PCI local bus 2.1+ and PCIe 4.0+ specifications.

Detailed overview

The related Nova specification [7] is an integral part of the cross-project effort and its problem description is assumed to be a part of this specification.

There are also relevant changes to OVS [8] and OVN itself [9] [10] that are needed in order to facilitate the implementation of this specification in Neutron. The result of upstream OVN discussions has been that a separate ovn-vif [11] component is going to be introduced and the corresponding change [10] is in the late stages of the review process at the time of writing. Support in other ML2 mechanism drivers is possible, but we would like to target OVN first and leave other ML2 drivers out of scope for Yoga cycle.

In the context of off-path SmartNIC DPUs, the main problems that need to be addressed in this change are:

SmartNIC DPU hostname selection such that the correct host is selected for representor port plugging. This corresponds to selecting the right chassis (transport node) in OVN terms;
VF representor selection corresponding the VF and a PF it is associated with selected at the hypervisor host side by Nova.

Some practical challenges are as follows:

If PCIe is used to access the NIC at a SmartNIC DPU host, PCI addresses seen by the hypervisor host and the SmartNIC DPU host differ as they have their own root complexes and see isolated PCIe topologies (while communicating with the same NIC). Therefore, PCI addresses seen from the SmartNIC DPU host side have no meaning at the hypervisor host side;
A NIC Switch is not exposed to PFs at the hypervisor host side. Meanwhile, PF representor numbers are tied to a particular NIC Switch (to a controller number) at the SmartNIC DPU host side. There may be multiple controllers per SmartNIC DPU host in a general case so simply passing PF logical numbers from the host side is not enough to identify a PF representor at the transport node side;
VF logical numbering seen via devlink port attributes depends on a device driver implementation at the hypervisor host and does not always match what is seen at the SmartNIC DPU host via the NIC Switch. It is possible to retrieve VF logical numbers tied to PFs at the hypervisor side based on the VF PCI address assignment process governed by the PCI SIG SR-IOV specification;
During port binding, the OVN mechanism driver needs to handle the vnic type VNIC_REMOTE_MANAGED which it currently does not.

The following diagram illustrates various components that play a role in the context of this specification:

                       ┌────────────────────────────────────┐
                       │  Hypervisor                        │    LoM Ports
                       │  ┌───────────┐       ┌───────────┐ │   (on-board,
                       │  │ Instance  │       │  Nova     │ ├──┐ optional)
                       │  │ (QEMU)    │       │ Compute   │ │  ├─────────┐
                       │  │           │       │           │ ├──┘         │
                       │  └───────────┘       └───────────┘ │            │
                       │                                    │            │
                       └────────────────┬─┬───────┬─┬──┬────┘            │
                                        │ │       │ │  │                 │
                                        │ │       │ │  │ Control Traffic │
                           Instance VF  │ │       │ │  │ PF associated   │
                                        │ │       │ │  │ with an uplink  │
                                        │ │       │ │  │ port or a VF.   │
                                        │ │       │ │  │ (used to replace│
                                        │ │       │ │  │  LoM)           │
   ┌────────────────────────────────────┼─┼───────┼─┼──┼─┐               │
   │   SmartNIC DPU Board               │ │       │ │  │ │               │
   │   (transport node)                 │ │       │ │  │ │               │
   │  ┌──────────────┐ Control traffic  │ │       │ │  │ │               │
   │  │   App. CPU   │ via PFs or VFs  ┌┴─┴───────┴─┴┐ │ │               │
   │  ├──────────────┤  (DC Fabric)    │             │ │ │               │
   │  │ovn-controller├─────────────────┼─┐           │ │ │               │
   │  ├──────────────┤                 │ │           │ │ │               │
   │  │ovs-vswitchd  │     Port        │ │NIC Switch │ │ │               │
   │  ├──────────────┤   Representors  │ │  ASIC     │ │ │               │
   │  │    br-int    ├─────────────────┤ │           │ │ │               │
   │  │              ├─────────────────┤ │           │ │ │               │
   │  └──────────────┘                 │ │           │ │ │               │
   │                                   │ │           │ │ │               │
   │                                   └─┼───┬─┬─────┘ │ │               │
 ┌─┴──────┐Initial NIC Switch            │   │ │       │ │               │
─┤OOB Port│configuration is done via     │   │ │uplink │ │               │
 └─┬──────┘the OOB port to create        │   │ │       │ │               │
   │       ports for control traffic.    │   │ │       │ │               │
   └─────────────────────────────────────┼───┼─┼───────┼─┘               │
                                         │   │ │       │                 │
                                      ┌──┼───┴─┴───────┼────────┐        │
                                      │  │             │        │        │
                                      │  │   DC Fabric ├────────┼────────┘
                                      │  │             │        │
                                      └──┼─────────────┼────────┘
                                         │             │
                                         │         ┌───┴──────┐
                                         │         │          │
                                     ┌───▼──┐  ┌───▼───┐ ┌────▼────┐
                                     │OVN SB│  │Neutron│ │Placement│
                                     └──────┘  │Server │ │         │
                                               └───────┘ └─────────┘

Proposed Change

Identifying a SmartNIC DPU Host and VF Representor

The related Nova specification [7] introduces add-in-card serial number collection via PCIe VPD and forwarding of that information in port updates to Neutron, therefore, it becomes possible to match that against what is seen at the SmartNIC DPU host side. Those serial numbers are unique, read-only, and assigned at the manufacturing time (per the PCI and PCIe specs). The board serial number can be stored in the OVN SB database as an external-id for a particular OVN chassis:

external_ids:ovn-cms-options='card-serial-number=UNIQUEBOARDSERIAL'

Note that there can be other means of accessing the NIC from the SmartNIC DPU host side (e.g. platform devices or other I/O types) so querying the serial number can be done either by extracting PCIe VPD via sysfs or by using devlink-info API and getting board.serial_number (which does not depend on a particular I/O interconnect type). However, this is a concern for OVN itself since Neutron does not have any agents running at the SmartNIC DPU side - it only needs to look up a chassis by a card-serial-number regardless of how it got collected at the SmartNIC DPU host.

The responsibility of placing it there can be given to:

A deployer who will be responsible of configuring the ovn-controller agent to be responsible for a particular transport node;
ovn-controller itself based on some default behavior of looking up a card serial number based on the switchdev-capable devices available on the SmartNIC DPU host.

Regardless of the chosen method of populating this value, Neutron will then be able to lookup which OVN chassis should handle representor plugging and flow programming for a particular port update that has a card serial included.

VF logical numbers are relevant in the context of a particular PF. In turn, PF logical numbers are tied to a particular controller. Since a NIC Switch is not exposed to the hypervisor host, it cannot determine the controller logical number as visible by the SmartNIC DPU host and while PF numbers could be inferred indirectly from their PCI address function numbers, this information is not enough to identify a PF representor at the SmartNIC DPU host side. To address that, a PF MAC seen by the hypervisor host can be passed to the SmartNIC DPU host. While port representors have a different MAC address from the port they represent, it is possible to use generic in-kernel API to retrieve a MAC address of a function via its representer (as of kernel 5.9 [12], subject to the switchdev-aware device driver support, for example [13]). While Neutron will not be responsible for doing this lookup (OVN will), it needs to accept and forward this information to OVN.

As a result, the binding:profile attribute for a port updated by Nova is expected to contain the following information:

binding:profile={
    "pci_vendor_info",
    "pci_slot",
    "physical_network",
    "card_serial_number": "UNIQUEBOARDSERIAL",
    "pf_mac_address": "de:ad:be:ef:ca:fe",
    "vf_num": 42
}

With this information both the right OVN chassis hostname and port representor at the SmartNIC DPU host side can be identified.

The OVN mechanism driver also needs to be changed (where relevant) to use the hostname looked up based on a serial number instead of relying on the hypervisor hostname passed in binding_host_id.

After receiving a port update from Nova, Neutron needs to create relevant Logical Switch Ports in the OVN database (the final approach to communicating the necessary information to OVN is to be determined based on the outcome of ovn-vif discussions in upstream OVN):

Logical_Switch_Port
  options:requested-chassis=fqdn-of-a-smartnic-dpu-host
  options:plug-type=representor
  options:plug-mtu-request=1500
  options:plug:representor:pf-mac=de:ad:be:ef:ca:fe
  options:plug:representor:vf-num=0

If vf-num is not specified, then a PF representor should be plugged instead of a VF representor.

Port Binding and Port Capabilities

Currently, to support hardware offload, the bind_port method of the OVN mechanism driver is able to bind ports of type direct if they also have the “switchdev” capability supplied in their binding:profile attribute:

binding:profile='{"capabilities": ["switchdev"]}'

This capability is discovered by Nova from PCI(e) endpoints via devlink - when an NIC Switch is visible via a PF, the “switchdev” capability is added to both PFs and VFs tracked in Nova. SmartNIC DPUs do not expose the NIC Switch to the hypervisor host, therefore, this capability is not discovered. To address that, the related Nova specification [7] relies on a new port type for the purposes of working with SmartNIC DPUs called VNIC_REMOTE_MANAGED. The VNIC_SMARTNIC VNIC type already present in neutron-lib has been considered but it was found later that the fact that scheduling happens after resource request creation makes it to run into a conflict with the Ironic use-case).

The OVN mechanism driver needs to be extended to also handle VNIC_REMOTE_MANAGED ports. It will allow the port binding code to pick the right code-path and trigger the representor port plugging logic in OVN.

Implementation

Primary Assignees

Dmitrii Shcherbakov (lp: ~dmitriis, oftc || libera: dmitriis)
Frode Nordahl (lp: ~fnordahl, oftc || libera: fnordahl)

Dependencies

OVS [8] and OVN [9] [10] [11] changes.
A related Nova specification [7] depends on this spec;

References

Allowed Address Pair: support matching ANY MAC address

Fri, 29 Oct 2021 00:00:00

Include the URL of your launchpad RFE:

https://bugs.launchpad.net/neutron/+bug/1946251

Allowed address pairs API allows to match against a subset of IP addresses, or even all of them, by virtue of netmasks being part of ‘ip_address’ field format. For example, using ‘0.0.0.0/0’ means allowing any IP address used with the corresponding MAC address. This proposal suggests extending the API so that a user can allow any MAC address used with the corresponding IP address.

Problem Description

An end user would like to configure their topology so that a port enforces security group rules but allows any MAC addresses. To achieve this, the user:

creates a security group with appropriate rules;
creates a port and assigns it to the security group;
adds allowed address pair for ANY MAC address and some IP address(es).

Right now, the only way to disable MAC address enforcement is to set port_security_enabled attribute to false. But this attribute also disables security group rule enforcement. In this scenario, only anti-spoofing enforcement should be disabled.

Proposed Change

The proposal is to extend ‘allowed_address_pairs’ attribute for ports so that its ‘mac_address’ field accepts a special value “ANY” that would disable MAC anti-spoofing mechanism. IP address enforcement may still continue, based on the corresponding ‘ip_address’ field value of the pair.

Having multiple address pairs with “ANY” ‘mac_address’ field value is supported. Having a list of pairs where some but not all ‘mac_address’ field values are “ANY” is supported.

A new Networking API extension will be introduced to indicate support for the new special value. This support is backend specific, which means that ML2 may need to offload decision to enable the extension to underlying drivers.

Database model for allowed address pairs will not need adjustment to accept the new special value because it already accepts all strings.

Client library may need to be adjusted to support the new special value “ANY” for the ‘mac_address’ field in the API validator module.

For agent-based setups, there are two cases to handle where the firewall driver chosen implements anti-spoofing / allowed_address_pair handling itself, and where this task is left to the agent. The firewall driver indicates the former by setting provides_arp_spoofing_protection attribute to True. In this scenario, the driver will have to implement handling of the new “ANY” value set for a MAC-IP pair. A new firewall driver flag (support_any_mac_anti_spoofing) will be added to indicate support for the new feature, defaults to False. If the flag is not set, the agent will filter out ANY address pairs before passing them to the firewall driver. It will also log a warning about the issue. Each driver that handles anti-spoofing itself will need to opt in receiving “ANY” address pairs by setting the support_any_mac_anti_spoofing flag to True.

Alternatives

Instead of using a new special value “ANY”, we could introduce a masking scheme for ‘mac_address’ field. For example, “de:ad:/32” would indicate “any MAC address that starts with de:ad: prefix”. But it’s debatable whether using any mask but /0 is generally useful, since MAC addresses are opaque values with no inherent ordering or grouping (that said, traditionally vendors reuse the same prefixes for their devices).

Instead of reusing the existing allowed address pairs API attribute, we could introduce a new port level attribute that would disable MAC address anti-spoofing without disabling security group rules. Since SG API still allows to match against particular IP addresses, this approach would be functionally identical to this. That said, I think it’s generally better to avoid introducing new attributes to base resources, and it’s better to keep anti-spoofing related functionality under the same attribute.

References

Add Possibility for Cascade Deletion of Neutron Networks

Fri, 24 Sep 2021 00:00:00

https://bugs.launchpad.net/neutron/+bug/1870319

When creating a network, a user needs to create other resources such as subnets and ports depending on their needs. They would need to make numerous API calls to the Neutron server to achieve this, and that’s okay. Deleting the network doesn’t need to be that much of a hassle, though. This spec explores a way to extend the delete API command to include an argument to cascade delete all corresponding ports and subports upon the deletion of a network.

Problem Description

Currently, deleting a network with many ports requires many API calls to the Neutron server as the user needs to list all ports and delete them before requesting the deletion of the network itself. Even more requests are required when the network is plugged to a router(s) or has trunk ports as those also need to be removed before network can be deleted. A simple openstack network delete <network name or ID> as prescribed in the OpenStackClient (OSC) to delete a given network does not work if there is one or more ports still in use on the network. In other cases, e.g. when Kuryr [1] is used, it puts a lot of load on Neutron’s client-side. This is what Kuryr currently needs to do to delete “namespace”:

Get the ACTIVE ports in the subnet.
Get the trunks.
For each ACTIVE port, obtain what trunk it is attached to and call Neutron to detach it.
Remove the ports (one by one; there is no bulk deletion).
Get the DOWN ports in the given subnet.
Remove the ports (one by one; there is no bulk deletion).
Detach the subnet from the router.
Remove the network (which will also remove the subnet).

Proposed Change

The goal of this spec is to add a new API to allow for the cascade deletion of a given network and all the resources that belong to that network, similarly to Octavia’s loadbalancer delete --cascade option [2]. When the proposed network cascade deletion API call is sent to the Neutron server, it is expected to find and clean:

All router interfaces that belong to the subnets in that network,
All trunk ports whose parent ports are in that network. It will also need to find and delete all the subports in these trunks,
Remove all ports in that network, this includes especially ports owned by nova compute, attached to VMs; ports owned by DEVICE_OWNER_DHCP, DEVICE_OWNER_DISTRIBUTED or DEVICE_OWNER_AGENT_GW can be omitted as they will have been already automatically removed.

After all this cleaning is done, the network will be finally deleted.

REST API Impact

The new option will be added to the DELETE /v2.0/networks/{network_id} API [3].

The new option will be called cascade, and will be a boolean value.

If this new option is set to True, Neutron will remove all the resources that need to be cleaned to successfully remove the given network.

The main concern regarding the proposed API is the return code in case the deletion of some resources fails and thus the request is only partially completed. There are a few possible options in such a case:

Return HTTP 207 Multi-status [4] and in the response body, return a list of actions on various resources and their statuses, for example:

{
    "data": [
        {
            "message": "Success",
            "resource": {
                "name": "port",
                "id": "40ac1143-1488-42df-bf85-b0fde7598c97"
            },
            "status": 200
        },
        {
            "message": "Forbidden",
            "resource": {
                "name": "trunk",
                "id": "cb4acbfa-190a-4ad2-9227-ea8595b59b19",
            },
            "status": 403
        },
        {
            "message": "Network in use.",
            "resource": {
                "name": "network",
                "id": "a7ebad6d-fd63-4024-b7db-8c30f5f9dea3"
            },
            "status": 409
        }
    ]
}

In this case, Neutron can always stop on first failure, so the response body will always contain only one or two resources which failed during the request: the network and one of the other resources that had to be cleaned in cascade but, for some reason, failed.

Stop processing the cascade request on first failure and always return HTTP 409 Conflict with detailed information about what went wrong, e.g. “Port <port_id> could not be deleted. Reason: <message>”.
Make this Neutron API asynchronous. In this case, the return code would only tell if the cascade delete request was accepted and will be processed by the server, or not. Later, the user would be able to periodically check if the network is already deleted or not yet. This is how Octavia implemented the cascade deletion of the loadbalancer [2]. But it would break the documented Neutron API behaviour [4].

The proposal of this spec is to implement third solution from the ones mentioned above and make this Neutron API asynchronous.

Data Model Impact

Network’s status field will have new possible value DELETING which will indicate that network is in the middle of the asynchronous deletion.

To avoid execution of several DB transactions at once, e.g. when one worker will proceed with asynchronous cascade deletion of the network and all its resources and other one would try to update one of the ports/subnets it will be needed to add check of the network’s status for all CREATE/UPDATE/DELETE operations for ports and subnets as well as for attach/detach subnets from the routers.

Security Impact

None

Performance Impact

Finding and removing all the resources to be cascade deleted will be slow, so the cascade deletion of a network with many resources may take a long time. To minimize that time, we can think of a couple of optimizations like:

Implementing bulk port deletion,
Removing a port from a trunk automatically if it is the trunk’s subport.

Implementation

Assignee(s)

Primary assignees:: Sharon Koech <skoech@protonmail.ch> (IRC: skoech)

Work Items

REST API update.
ML2 plugin update.
CLI update.
Documentation.
Tests and CI related changes.

Testing

Unit Tests.
API tests.

Documentation Impact

User Documentation

The new API option must be documented in the Neutron api-ref document.

References

QoS minimum guaranteed packet rate

Wed, 15 Sep 2021 00:00:00

https://bugs.launchpad.net/neutron/+bug/1922237

Similarly to how bandwidth can be a limiting factor of a network interface, packet processing capacity tend to be a limiting factor of the soft switching solutions like OVS. In the same time certain applications are dependent on not just guaranteed bandwidth but also on guaranteed packet rate to function properly. OpenStack already supports bandwidth guarantees via the minimum bandwidth QoS policy rules. This specification is aiming for adding support for a similar minimum packet rate QoS policy rule.

To add support for the new QoS rule type both Neutron and Nova needs to be extended. This specification only focuses on the Neutron impact. For the high level view and for the Nova specific impact please read the Nova specification.

Problem Description

Currently there are several parameters, quantitative and qualitative, that define a VM and are used to select the correct host and network backend devices to run it. The packet processing capacity of the soft switching solution (like OVS) could also be such a parameter that affects the placement of new workload.

This spec focuses on managing the packet processing capacity of the soft switch (like OVS) running on the hypervisor host serving the VM. Managing packet processing capacity in other parts of the networking backend (like in Top-Of-Rack switches) are out of scope.

Guaranteeing packet processing capacity generally involves enforcement of constraints on two levels.

placement: Avoiding over-subscription when placing (scheduling) VMs and their ports.
data plane: Enforcing the guarantee on the soft switch

This spec addresses placement enforcement only. Data plane enforcement can be developed later. When the packet rate limit policy rule feature is implemented then a basic data plane enforcement can be applied by adding both minimum and maximum packet rate QoS rules to the same QoS policy where maximum limit is set to be equal to the minimum guarantee.

Use cases

I as an administrator want to define the maximum packet rate, in kilo packet per second (kpps), my OVS soft switch capable of handle per compute node, so that I can avoid overload on OVS.

I as an end user want to define the minimum packet rate, in kilo packet per second (kpps) a Neutron port needs to provide to my VM, so that my application using the port can work as expected.

I as an administrator want to get a VM with such ports placed on a compute node that can still provide the requested minimum packet rate for the Neutron port so that the application will get what it requested.

I as an administrator want that the VM lifecycle operations are rejected in case the requested minimum packet rate guarantee of the Neutron ports of the server cannot be fulfilled on any otherwise eligible compute nodes, so that the OVS overload is avoided and application guarantees are kept.

Proposed Change

Note

For the high level solution including the impact and interactions between Nova, Neutron and Placement see the Nova specification.

This feature is similar to the already supported minimum bandwidth QoS policy rules. So this spec describes the impact in relation to the already introduced concepts and implementation while also pointing out key differences.

The solution needs to differentiate between two deployment scenarios.

The packet processing functionality is implemented on the compute host CPUs and therefore packets processed from both ingress and egress directions are handled by the same set of CPU cores. This is the case in the non-hardware-offloaded OVS deployments. In this scenario OVS represents a single packet processing resource pool. Which can be represented with a single new resource class, NET_PACKET_RATE_KILOPACKET_PER_SEC.
The packet processing functionality is implemented in a specialized hardware where the incoming and outgoing packets are handled by independent hardware resources. This is the case for hardware-offloaded OVS. In this scenario a single OVS has two independent resource pools one for the incoming packets and one for the outgoing packets. Therefore these needs to be represented with two new resource classes NET_PACKET_RATE_EGR_KILOPACKET_PER_SEC and NET_PACKET_RATE_IGR_KILOPACKET_PER_SEC.

OVS packet processing capacity

Neutron OVS agent needs to provide a configuration options for the administrator to define the maximum packet processing capability of the OVS per compute node. This means the following new configuration options are added to OVS agent configuration:

[ovs]
# Comma-separated list of <hypervisor>:<packet_rate> tuples, defining the
# minimum packet rate the OVS backend can guarantee in kilo (1000) packet
# per second. The hypervisor name is used to locate the parent of the
# resource provider tree. Only needs to be set in the rare case when the
# hypervisor name is different from the DEFAULT.host config option value as
# known by the nova-compute managing that hypervisor or if multiple
# hypervisors are served by the same OVS backend.
# The default is :0 which means no packet processing capacity is guaranteed
# on the hypervisor named according to DEFAULT.host.
# resource_provider_packet_processing_without_direction = :0
#
# Similar to the resource_provider_packet_processing_without_direction but
# used in case the OVS backend has hardware offload capabilities. In this a
# case the format is
# <hypervisor>:<egress_packet_rate>:<ingress_packet_rate> which allows
# defining packet processing capacity per traffic direction. The direction
# is meant from the VM perspective. Note that the
# resource_provider_packet_processing_without_direction and the
# resource_provider_packet_processing_with_direction
# are mutually exclusive options.
# resource_provider_packet_processing_with_direction = :0:0
#
# Key:value pairs to specify defaults used while reporting packet rate
# inventories. Possible keys with their types: allocation_ratio:float,
# max_unit:int, min_unit:int, reserved:int, step_size:int
# resource_provider_packet_processing_inventory_defaults = {
#   'allocation_ratio': 1.0, 'min_unit': 1, 'step_size': 1, 'reserved': 0}

Note

Note that while bandwidth inventory is defined per OVS bridge in the [ovs]resource_provider_bandwidths configuration option, the packet processing capacity is applied globally to the whole OVS instance.

Note

Note that to support both OVS deployment scenarios there are two mutually exclusive configuration options. One to handle the normal OVS deployments with directionless resource inventories and one to handle hardware-offloaded OVS deployments with direction aware resource inventories.

The configurations field of the OVS agent heartbeat RPC message is extended to report the packet processing capacity configuration to the Neutron server. If the hypervisor name is omitted from the configuration it is resolved to the value of [DEFAULT]host in the RPC message.

The Neutron server reports the packet processing capacity as the new NET_PACKET_RATE_KILOPACKET_PER_SEC or NET_PACKET_RATE_[E|I]GR_KILOPACKET_PER_SEC resource inventory on the OVS agent resource provider (RP) to Placement in a similarly way how the bandwidth resource is reported today. Now that the OVS agent RP has resource inventory the Neutron server also needs to report the same CUSTOM_VNIC_TYPE_ traits on the OVS agent RP as reported on the bridge RPs. These are the vnic types this agent configured to support. Note that CUSTOM_PHYSNET_ traits are not needed for the packet rate scheduling as this resource is not split between the available physnets.

Note

Regarding the alternative of reporting the resources on the OVS bridge level instead, please see the Nova specification

Minimum packet rate QoS policy rule

A new Neutron API extension is needed to extend the QoS API with the new minimum_packet_rate rule type. This rule has a similar structure and semantic as the minimum_bandwidth rule:

qos_apidef.SUB_RESOURCE_ATTRIBUTE_MAP = {
    'minimum_packet_rate_rules': {
        'parent': qos_apidef._PARENT,
        'parameters': {
            qos_apidef._QOS_RULE_COMMON_FIELDS,
            'min_kpps': {
                'allow_post': True,
                'allow_put': True,
                'convert_to': converters.convert_to_int,
                'is_visible': True,
                'is_filter': True,
                'is_sort_key': True,
                'validate': {
                    'type:range': [0, db_const.DB_INTEGER_MAX_VALUE]}
            },
            'direction': {
                'allow_post': True,
                'allow_put': True,
                'is_visible': True,
                'default': constants.EGRESS_DIRECTION,
                'validate': {
                    'type:values': (
                        constants.ANY_DIRECTION,
                        constants.INGRESS_DIRECTION,
                        constants.EGRESS_DIRECTION
                    )
                }
            }
        }
    }
}

The direction, ingress or egress of the rule is considered from the Nova server’s perspective. So an 1 kpps ingress guarantee means the system ensures that at least 1000 packets can enter the VM via the given port per second.

To support the two different OVS deployment scenarios we need to allow that the direction field of the new QoS policy rule to be set to any to indicate that this QoS rule is valid in the normal OVS case where the resource accounting is directionless.

Mixing direction aware and directionless minimum packet rate rules in the same QoS policy will always result in a NoValidHost error during scheduling as no single OVS instance can have both direction aware and directionless inventory at the same time. Therefore such rule creation will be rejected by the Neutron server with HTTP 400.

Note

For the alternative about having only direction aware QoS rule types see the Nova specification

The above definition allows the min_kpps value to be updated with a PUT request. This request will be rejected with HTTP 501 if the rule is already used in a bound port similarly to the minimum bandwidth rule behavior.

Neutron provides the necessary information to the admin clients via the resource_request of the port to allow the client to keep the placement resource allocation consistent and therefore implement the schedule time guarantee. Nova will use this information to do so. However neither Neutron nor Nova can enforce that another client, which can bound ports also properly allocates resources for those ports. This is out of scope.

This results in the following new API resources and operations:

GET /v2.0/qos/policies/{policy_id}/minimum_packet_rate_rules

List minimum packet rate rules for QoS policy

Response:

{
  "minimum_packet_rate_rules": [
      {
          "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c793",
          "min_kpps": 1000,
          "direction": "egress"
      }
  ]
}

POST /v2.0/qos/policies/{policy_id}/minimum_packet_rate_rules

Create minimum packet rate rule

Request:

{
  "minimum_packet_rate_rule": {
      "min_kpps": 1000,
      "direction": "any",
  }
}

Response:

{
  "minimum_packet_rate_rule": {
      "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c793",
      "min_kpps": 1000,
      "direction": "any"
  }
}

GET /v2.0/qos/policies/{policy_id}/minimum_packet_rate_rules/{rule_id}

Show minimum packet rate rule details

Response:

{
  "minimum_packet_rate_limit_rule": {
      "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c793",
      "min_kpps": 1000,
      "direction": "egress"
  }
}

PUT /v2.0/qos/policies/{policy_id}/minimum_packet_rate_rules/{rule_id}

Update minimum packet rate rule

Request:

{
  "minimum_packet_rate_rule": {
      "min_kpps": 2000,
      "direction": "any",
  }
}

Response:

{
  "minimum_packet_rate_rule": {
      "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c794",
      "min_kpps": 2000,
      "direction": "any",
  }
}

DELETE /v2.0/qos/policies/{policy_id}/minimum_packet_rate_rules/{rule_id}

Delete minimum packet rate rule

Note

This spec intentionally does not propose the addition of the old style /v2.0/qos/policies/{policy_id}/minimum_packet_rate_rules/{rule_id} APIs as Neutron team prefers the /v2.0/qos/alias_bandwidth_limit_rules/{rule_id} style APIs in the future. The old style APIs only kept for backwards compatibility for already existing QoS rules. However as this new API will not have an old style counterpart the ‘alias’ prefix is removed from the resource name.

To persist the new QoS rule type a new DB table qos_minimum_packet_rate_rules is needed:

op.create_table(
    'qos_minimum_packet_rate_rules',
    sa.Column('id', sa.String(36), nullable=False,
              index=True),
    sa.Column('qos_policy_id', sa.String(36),
              nullable=False, index=True),
    sa.Column('min_kpps', sa.Integer()),
    sa.Column('direction', sa.Enum(constants.ANY_DIRECTION,
                                   constants.EGRESS_DIRECTION,
                                   constants.INGRESS_DIRECTION,
                                   name="directions"),
              nullable=False,
              server_default=constants.EGRESS_DIRECTION),
    sa.PrimaryKeyConstraint('id'),
    sa.ForeignKeyConstraint(['qos_policy_id'], ['qos_policies.id'],
                            ondelete='CASCADE')
)

This also means a new QosMinimumPacketRateRule DB model and OVO are added.

Request packet rate resources

Today the resource_request field of the Neutron port is used to express the resource needs of the port. The information in this field is calculated from the QoS policy rules attached to the port. So far only the minimum bandwidth rule is used as a source of the requested resources. However both the structure and the actual logic calculating the value of this field needs to be changed when the new minimum packet rate rule is added as an additional source of the requested resources.

Currently resource_request is structured like:

{
    "required": [<CUSTOM_PHYSNET_ traits>, <CUSTOM_VNIC_TYPE traits>],
    "resources":
    {
        <NET_BW_[E|I]GR_KILOBIT_PER_SEC resource class name>:
        <requested bandwidth amount from the QoS policy>
    }
},

The current structure only allows describing one group of resources and traits. However as described above the packet processing resource inventory is reported on the OVS agent RP while the bandwidth resources are reported on the OVS bridge RP. This also means that requesting these resources needs to be separated as one group of resources always allocated from a single RP in placement.

Therefore the following structure is proposed for the resource_request field:

{
    "request_groups":
    [
        {
            "id": <some unique identifier string of the group>
            "required": [<CUSTOM_VNIC_TYPE traits>],
            "resources":
            {
                NET_PACKET_RATE_[E|I]GR_KILOPACKET_PER_SEC:
                <amount requested via the QoS policy>
            }
        },
        {
            "id": <some unique identifier string of the group>
            "required": [<CUSTOM_PHYSNET_ traits>,
                         <CUSTOM_VNIC_TYPE traits>],
            "resources":
            {
                <NET_BW_[E|I]GR_KILOBIT_PER_SEC resource class name>:
                <requested bandwidth amount from the QoS policy>
            }
        },
    ]
}

Each dict in the list represents one group of resources and traits that needs to be fulfilled from a single resource provider. E.g. either from the bridge RP in case of bandwidth, or the OVS agent RP in case of packet rate. This solves the problem of the RP separation. However in the other hand a port still needs to allocate resources from the same overall entity, e.g. from OVS, by allocating packet processing capacity from the OVS agent RP and bandwidth from one of the OVS bridge RPs. In other words it is invalid to fulfill the packet processing request from OVS and the bandwidth request from an SRIOV PF managed by the SRIOV agent. In the placement model the OVS bridge RPs are the children of the OVS agent RP while the PF RPs are not. So placement already aware of the structural connections between the resource inventories. Still by default, when two groups of resources are requested Placement only enforces that they are fulfilled from two RPs in the same RP tree but it does not enforce any subtree relationship between those RPs. To express that the two groups of resources should be fulfilled from two RPs in the same subtree (in our case from the subtree rooted by the OVS agent RP) placement needs extra information in the request. Placement supports a same_subtree parameter that can express what we need. Neutron needs to add a new top level key same_subtree to the resource_request dict. I.e.:

{
    "request_groups":
    [
        {
            "id": "port-request-group-due-to-min-pps",
            # ...
        },
        {
            "id": "port-request-group-due-to-min-bw",
            # ...
        },
    ],
    "same_subtree":
    [
        "port-request-group-due-to-min-pps",
        "port-request-group-due-to-min-bw"
    ]
}

The id field is a string that is assumed to be unique for each group of each port in a neutron deployment. To achieve this a new UUID will be generated for each group by combining the port_id and UUIDs of the QoS rules contributing to the group via the UUID5 method. This way we get a unique and deterministic id for the group so we don’t need to persist the group id in Neutron.

We discussed and rejected another two alternatives:

Ignore the same subtree problem for now. The QoS configuration already requires the admin to create a setup where the different mechanism drivers support disjunct set of vnic_type s via the vnic_type_prohibit_list config option. A port always requests a specific vnic_type and the supported vnic_type s are disjunct therefore the port’s resource request always selects a specific networking backend via the CUSTOM_VNIC_TYPE_ traits. Support for packet rate resource is only added to the OVS backend, therefore if the scheduling of a port’s resource request, containing both packet rate and bandwidth resource, succeeds then we know that the packet rate resource is fulfilled by the OVS agent RP. Therefore the vnic_type matched the OVS backend. The bandwidth request also fulfilled from a backend with the same vnic_type so it is also fulfilled from the OVS backend. If we go with this direction then special care needs to be taken to document the above assumption carefully so that future developers can check if any of the statements become invalid due to new feature additions.

Make an assumption in Nova that every request group from a single port always needs to be fulfilled from the same subtree. Then the same_subtree key is not needed in the resource_request but Nova will implicitly assume that it is there and generate the placement request accordingly.

The selected alternative is more future proof. When the IP allocation pool handling are transformed to use the resource request, that resource will come from a sharing resource provider and therefore Nova cannot implicitly assume same_subtree for the whole resource_request.

Note that the current neutron API does not define the exact structure of the field, it is just a dict, so from Neutron perspective there is no need for a new API extension to change the structure. However Nova needs to know which format will be used by Neutron. We have alternatives:

A new Neutron API extension: It can signal the change of the API. Nova already use to check the extension list to see if some feature is enabled in Neutron or not.
A new top level resource_request_version field in the resource_request dict can signal the current and future versions. Nova would need to check if the field exists in resource_request and conditionally parse the rest of the dict.
Let Nova guess based on the structure. The new top level request_groups key can be used in Nova to detect the new format.

The API extension is the selected alternative as that feels like the standard way in Neutron to signal API change.

Port binding

Today Nova sends the UUID of the resource provider the port’s resource_request is fulfilled from in the allocation key of the binding:profile. The port binding logic uses this information to bind the port to the same physical device or OVS bridge the port’s resources are allocated from. This is necessary as it is allowed to have multiple such devices that are otherwise equivalent from Neutron perspective, i.e. they are connected to the same physnet and supporting the same vnic_type. When the port has two groups of resource requests (one for bandwidth and the other for packet rate) the resource allocation is fulfilled from more than one RPs. To support that we need to change the structure of the allocation key. As every group of resources in the resource_request now has a uniq identifier Nova can send a mapping of <group.id>: <RP.uuid> back in the allocaton key of binding:profile so that Neutron gets informed about which RP provided which group of resources. This means the following structure in the allocation key:

{
    <uniq id of the group requesting min_pps>:
        <OVS agent RP UUID>,
    <uniq id of the group requesting min_bw>:
        <OVS bridge RP UUID or SRIOV PF RP UUID>,
}

Only those group id keys are present in this dict that are present in the resource_request field as id.

Alternatively we could ignore the problem for now. Only OVS supports packet rate inventory for now and the packet rate inventory is global for the whole OVS instance. A single binding:host_id always maps to a single OVS instance, so Neutron can always assume that the minimum packet rate resource are allocated from the OVS agent resource provider that belongs to the compute node the port is requested to bound to. So the UUID of the packet rate resource provider is not needed of the port binding logic. Therefore the allocation key can be kept to only communicate the UUID of the bandwidth resource provider if any.

QoS policy change on bound port

Neutron supports changing the QoS policy on a bound port even if this means that resource allocation change is needed due to changes in the resource request indicated by the minimum_bandwidth QoS rule. This implementation needs to be extended to handle changes not just in minimum_bandwidth but also in minimum_packet_rate rule.

The current support matrix is:

Current scenarios
Scenario	Result
Update the port.qos_policy_id from None to valid QoS policy with minimum bandwidth rule	Accepted but the resource allocation is not adjusted as that would require a full scheduling. If the VM later scheduled to another host (i.e. migrated, resize, evacuated) then that scheduling will consider the new resource request.
Update the port.qos_policy_id from a QoS policy without minimum bandwidth rule to a QoS policy with minimum bandwidth rule	Accepted but the resource allocation is not adjusted. See above.
Update the port.qos_policy_id from a QoS policy with minimum bandwidth rule to a QoS policy with minimum bandwidth rule but with different min_kbps value or different direction.	Supported. Accepted if the bandwidth allocation on the current resource provider can be updated with the new QoS value and direction.
Update the port.qos_policy_id from a QoS policy with minimum bandwidth rule to a QoS policy without minimum bandwidth rule.	Supported. The bandwidth resource allocation for this port in placement is deleted.

After the minimum packet rate rule is added Neutron will have two QoS policy rules that requires placement resources allocation. This adds more possible cases to handle. The above scenarios still apply for minimum bandwidth and can be applied similarly to QoS policy with a single minimum packet rate rule. The more complex cases are described below:

New scenarios
Scenario	Result
The old QoS policy has both minimum bandwidth and packet rate rules, the new policy also has both rules but with different min_kbps and / or min_kpps values.	Supported. Accepted if the bandwidth allocation on the current resource providers can be updated with the new QoS values. Changing the direction of the minimum bandwidth rule or the direction of the direction aware minimum packet rate rule is also supported. However changing from a direction less minimum packet rate rule to a direction aware packet rate rule, or vice versa, is not supported and rejected.
The old QoS policy has both minimum bandwidth and packet rate rules, the new policy has less rules. Either just minimum bandwidth or just minimum packet rate or none of them.	Supported. The allocation related to the removed rule(s) are deleted in placement.
The new QoS policy adds either minimum bandwidth or packet rate rule or both compared to the old policy.	Accepted but the resource allocation is not adjusted as that would require a full scheduling. If the VM later scheduled to another host (i.e. migrated, resize, evacuated) then that scheduling will consider the resource request of the new rule(s).

Upgrade

A database schema upgrade is needed to add the new qos_minimum_packet_rate_rules table.

The changes in the Neutron server - OVS agent communication means that during rolling upgrade upgraded OVS agents might send the new packet processing capacity related keys in the hearthbeat while old agents will not send it. So Neutron server needs to consider this new key as optional.

The manipulation of the new minimum_packet_rate QoS policy rule and changes in the resource_request and allocation fields of the port requires a new API extension. We need to support upgrade scenarios where the Neutron is already upgraded to Yoga but Nova still on Xena version. However this does not require any additional logic in Neutron as Nova already landed support this API extension in Xena.

Testing

Unit tests.
Functional tests: agent-server interactions.
Tempest scenario tests: End-to-end feature test.

Documentation

Update the generic QoS admin guide
A new admin guide, similar to the one for minimum bandwidth

References

The Nova specification
The already supported minimum bandwidth QoS policy rules serving as a pattern for the new minimum packet rate QoS policy rule.

BFD as a Service for Neutron

Mon, 02 Aug 2021 00:00:00

https://bugs.launchpad.net/neutron/+bug/1907089

Add possibility to define BFD monitors in Neutron and associate it with routes.

Problem Description

BFD (Bidirectional Forwarding Detection) can be used to detect link failures between a Neutron router and an arbitrary destination, like an external non-Neutron router.

BFD can be used to provide quick, low-overhead link failure detection between adjacent forwarding engines ([1]) to check for example if an extra route (a nexthop-destination pair) is alive or not, and change routes accordingly.

Note

This spec is not covering other use cases like tunnel monitoring and HA or DVR router monitoring, or BGP or other routing protocol monitoring.

Proposed Change

The goal of this spec is to add APIs to manage bfd_monitor instances, fetch their status, associate a monitor to an extra route. The goal is to cover the use case when Neutron managed bfd_monitor instances take the ACTIVE role, thus send BFD control packets, (see [2]).

This gives possibility to the operators to monitor the routes (destination - nexthop pairs) added for a router.

Monitoring scenarios

Two different nexthops for the same destination, like ecmp route legs

dst1	nexthop1	bfd_dst1	bfd1
dst1	nexthop2	bfd_dst1	bfd2

____nexthop1 ____
                 \ ____dst1
____nexthop2____ /

monitoring 2 different destinations through the same nexthop, the case when bfd3 and bfd4 dst_ip is the same:

dst3	nexthop3	bfd_dst3	bfd3
dst4	nexthop3	bfd_dst4	bfd4

                   ____dst3
_____nexthop3_____/
                  \____dst4

REST API Impact

New bfd monitor API

Create bfd_monitor:

POST /v2.0/bfd_monitors
{
    "bfd_monitor": {
        "name": "bfd1",
        "mode": "asynchronous",
        "min_rx": 1000,
        "min_tx": 100,
        "multiplier": 3,
        "dst_ip": "10.0.0.13",
        "auth_type": "",
        "auth_keys": {1: "secret_1", 2: "secret_2"},
    }
}

List bfd_monitors:
```
GET /v2.0/bfd_monitors
```

Show bfd_monitor:

GET /v2.0/bfd_monitors/{monitor_id}
{
    "bfd_monitor": {
        "name": "bfd1",
        "project_id": "{project_id}",
        "description": "",
        "mode": "asynchronous",
        "min_rx": 1000,
        "min_tx": 100,
        "multiplier": 3,
        "dst_ip": "10.0.0.13",
        "src_ip": "169.254.112.144",
        "auth_type": "",
        "auth_keys": {1: "secret_1"},
    }
}

Update a bfd_monitor:

PUT /v2.0/bfd_monitors/{monitor_id}
{
    "bfd_monitor": {
        "name": "bfd_monitor1"
        "description": "foo",
        "min_rx": 2000,
        "min_tx": 200,
        "multiplier": 4
    }
}

Delete bfd_monitor:
```
DELETE /v2.0/bfd_monitors/{monitor_id}
```

Note

Only an unassociated bfd_monitor can be deleted.

API fields and descriptions for bfd_monitor:

Attribute	Type	Req	CRUD	Description
id	uuid-str	No	R	id of bfd_monitor
name	String	No	CRU	Human readable name for the bfd_monitor (255 characters limit). Does not have to be unique.
description	String	No	CRU	Human readable description for the bfd_monitor (255 characters limit).
project_id	String	No	R	Owner of the bfd_monitor.
mode	String	No	CR	Can be `asynchronous` (default common echo mode of BFD) or `demand` (some other mechanism is used to detect link state) and can accept future modes like `one-arm-echo` see [4] .
dst_ip	String	Yes	CR	The destination IP address to be monitored. In case of singlehop bfd this is the nexthop ip of the route, for the general case (like multihop bfd) this is an arbitrary IP (IPv4 or Ipv6) that can serve as BFD neighbor.
src_ip	String	No	CR	IP address used as source for transmitted BFD packets. An optional field, if not specified, the IP set on the interface, like on qg-xyz. This can be set on the remote end to be configured.
min_rx	Integer	No	CRU	The shortest interval, in millisecs, at which this BFD session offers to receive BFD control messages. At least 1. Defaults is 1000.
min_tx	Integer	No	CRU	The shortest interval, in millisecs, at which this BFD session is willing to transmit BFD control messages. At least 1. Default is 100.
multiplier	Integer	No	CRU	The BFD detection multiplier, An endpoint signals a connectivity fault if the given number of consecutive BFD control messages fail to arrive. Default is 3.
status	String	N/A	R	Shows if the BFD monitor was succesfully created in the backend, but nothing about the session status, for that the session_status API endpoint can be used.
auth_type	String	No	CR	The Authentication Type, which can be `password`, `MD5`, `MeticulousMD5`, `SHA1`, `MeticulousSHA1`, if empty no authentication is used.
auth_key	Dict	No	CR	A dictionary of authentication key chain in which key is an integer of `Auth Key ID` and value is a string of `Password` or `Auth Key`.

Note

For using authentication with BFD, please check [5]

API extension proposal for bfd_monitors:

ALIAS = 'bfd-monitor'
IS_SHIM_EXTENSION = False
IS_STANDARD_ATTR_EXTENSION = False
NAME = 'BFD monitors for Neutron'
DESCRIPTION = "Provides support for BFD monitors"
UPDATED_TIMESTAMP = "2021-02-12T11:00:00-00:00"
BFD_MONITOR = 'bfd_monitor'
BFD_MONITORS = 'bfd_monitors'
BFD_SESSION_STATUS = 'bfd_session_status'

BFD_MODE_ASYNC = 'asynchronous'
BFD_MODE_DEMAND = 'demand'
BFD_MODE_ONE_ARM = 'one_arm_echo'

RESOURCE_ATTRIBUTE_MAP = {
    BFD_MONITORS: {
        'id': {'allow_post': False, 'allow_put': False,
               'validate': {'type:uuid': None},
               'is_visible': True,
               'primary_key': True,
               'enforce_policy': True},
        'name': {'allow_post': True, 'allow_put': True,
                 'validate': {'type:string': db_const.NAME_FIELD_SIZE},
                 'default': '', 'is_filter': True, 'is_sort_key': True,
                 'is_visible': True},
        'description': {'allow_post': True, 'allow_put': True,
                        'is_visible': True, 'default': '',
                        'validate': {
                            'type:string': db_const.DESCRIPTION_FIELD_SIZE}},
        'project_id': {'allow_post': True, 'allow_put': False,
                       'validate': {
                           'type:string': db_const.PROJECT_ID_FIELD_SIZE},
                       'required_by_policy': True,
                       'is_visible': True, 'enforce_policy': True},
        'mode': {'allow_post': True, 'allow_put': False,
                 'validate': {'type:string': db_const.STATUS_FIELD_SIZE},
                 'default': BFD_MODE_ASYNC, 'is_filter': True,
                 'is_sort_key': True, 'is_visible': True},
        'dst_ip': {'allow_post': True, 'allow_put': False,
                   'validate': {'type:ip_address': None},
                   'is_sort_key': True, 'is_filter': True,
                   'is_visible': True, 'default': None,
                   'enforce_policy': True},
        'src_ip': {'allow_post': True, 'allow_put': False,
               'validate': {'type:ip_address_or_none': None},
               'is_sort_key': True, 'is_filter': True,
               'is_visible': True, 'default': None,
               'enforce_policy': True},
        'min_rx': {'allow_post': True, 'allow_put': True,
                   'validate': {'type:non_negative': None},
                   'convert_to': converters.convert_to_int,
                   'default': 1000,
                   'is_visible': True, 'enforce_policy': True},
        'min_tx': {'allow_post': True, 'allow_put': True,
                   'validate': {'type:non_negative': None},
                   'convert_to': converters.convert_to_int,
                   'default': 100,
                   'is_visible': True, 'enforce_policy': True},
        'multiplier': {'allow_post': True, 'allow_put': True,
                       'validate': {'type:non_negative': None},
                       'convert_to': converters.convert_to_int,
                       'default': 3,
                       'is_visible': True, 'enforce_policy': True},
        'status': {'allow_post': False, 'allow_put': False,
                   'is_filter': True, 'is_sort_key': True,
                   'is_visible': True},
        'auth_type': {'allow_post': True, 'allow_put': False,
                      'validate': {'type:string_or_none':
                                   db_const.NAME_FIELD_SIZE},
                      'default': constants.ATTR_NOT_SPECIFIED,
                      'is_visible': True},
        'auth_key': {'allow_post': True, 'allow_put': False,
                     'validate': {'type:dict_or_none': None},
                     'default': constants.ATTR_NOT_SPECIFIED,
                     'is_visible': True},
    },
}
SUB_RESOURCE_ATTRIBUTE_MAP = {}
ACTION_MAP = {
    BFD_MONITOR: {
        'bfd_session_status': 'GET',
        'bfd_monitor_associations': 'GET',
    }
}
ACTION_STATUS = {}
REQUIRED_EXTENSIONS = [l3.ALIAS]
OPTIONAL_EXTENSIONS = []

Show bfd session status (For details see rfc5880, State Variables section):

GET /v2.0/bfd_monitors/{monitor_id}/session_status
{
    "bfd_session_status": {
        "remotes": [{
            "type": "extra_route",
            "router": {router_id},
            "extra_route": {
                "destination": "10.0.3.0/24",
                "nexthop": "10.0.0.1"
            },
            "src_ip": "169.254.112.144",
            "status": {
                "SessionState": "Up",
                "RemoteSessionState": "Up",
                "LocalDiagnostic": "",
                "RemoteDiagnostic": "",
                "LocalDiscriminator": "",
                "RemoteDiscriminator": "",
                "Forwarding": "",
            }
        },
        {
            "type": "extra_route",
            ...
        }]
    }
}

Show bfd_monitor associations:

GET /v2.0/bfd_monitors/{monitor_id}/bfd_monitor_associations
{
     "bfd_monitor_associations": {
         "remotes": [{
             "type": "extra_route",
             "router": {router_id},
             "extra_route": {
                 "destination": "10.0.3.0/24",
                 "nexthop": "10.0.0.1"
             },
             "src_ip": "169.254.112.144",
         },
         {
             "type": "extra_route",
             ...
         }]
     }
}

Note

The current proposal is only for “type”: “extra_route”, so the extra_route’s details (nexthop, destination as of now) will appear.

Note

One bfd_monitor instance can be associated to several extra_routes.

Note

src_ip is the IP address used as source for transmitted BFD packets. A read only field that practically shows the IP set on the interface, like on qg-xyz. This can be set on the remote end.

Short description of the status fields:

Attribute	Description
SessionState	The state of the BFD session, it can be `admin_down`, `down`, `init`, or `up`.
RemoteSessionState	The state of the remote endpoint’s BFD session, it can be `admin_down`, `down`, `init`, or `up`.
LocalDiagnostic	A diagnostic code specifying the local system’s reason for the last change in session state. The error messages are defined in section 4.1 of the RFC 5880 (see [3]).
RemoteDiagnostic	A diagnostic code specifying the remote system’s reason for the last change in session state. The error messages are defined in section 4.1 of the RFC 5880 (see [3]).
LocalDiscriminator	The local discriminator for this BFD session, used to uniquely identify it.
RemoteDiscriminator	The remote discriminator for this BFD session. This is the discriminator chosen by the remote system.
Forwarding	Reports whether the BFD session believes this Interface may be used to forward traffic. It can be `true` or `false`.

Note

The above fields are optional, so based on the backend not all will be filled, and in worst case a cumulative Forwarding result will only present.

Changes to Router extra routes API

Changes to add or remove extra routes to router or Update router API:

PUT /v2.0/routers/{router_id}
{
    "router": {
        "routes": [
            {
                "destination": "179.24.1.0/24",
                "nexthop": "172.24.3.99"
                "bfd_monitor": {bfd_monitor_uuid}
            },
        ]
    }
}

PUT /v2.0/routers/{router_id}/add_extraroutes
{
    "router" : {
        "routes" : [
           { "destination" : "10.0.3.0/24", "nexthop" : "10.0.0.13", "bfd_monitor": {bfd_monitor_uuid} },
           { "destination" : "10.0.4.0/24", "nexthop" : "10.0.0.14" }
        ]
    }
}

PUT /v2.0/routers/{router_id}/remove_extraroutes
{
    "router" : {
        "routes" : [
           { "destination" : "10.0.3.0/24", "nexthop" : "10.0.0.13", "bfd_monitor": {bfd_monitor_uuid} }
        ]
    }
}

Note

The API will allow to remove bfd_monitor association from a route without deleting and recreating the whole nexthop-destination tuple.

Get routes status:

GET /v2.0/routers/{router_id}/routes_status
{
    "router": {
        "routes": [
            { "destination" : "10.0.3.0/24",
              "nexthop" : "10.0.0.13",
              "status": "UP",
              "bfd_monitor": {bfd_monitor_uuid}
            },
        ]
    }
}

API extension proposal for allowing bfd_monitor association to extra_routes:

ALIAS = 'bfd-for-extraroutes'
IS_SHIM_EXTENSION = False
IS_STANDARD_ATTR_EXTENSION = False
NAME = 'BFD monitors for extraroutes'
DESCRIPTION = ('Provides the possibility to associate a bfd_monitor to'
               'extra_routes')
UPDATED_TIMESTAMP = '2021-01-29T00:00:00+00:00'
RESOURCE_NAME = l3.ROUTER
COLLECTION_NAME = l3.ROUTERS
ROUTES = 'routes'
RESOURCE_ATTRIBUTE_MAP = {
    COLLECTION_NAME: {
        ROUTES: {
            'allow_post': False, 'allow_put': True,
            'validate': {'type:hostroutes': ['destination', 'nexthop',
                                             'bfd_monitor_id']},
            'convert_to': converters.convert_none_to_empty_list,
            'is_visible': True,
            'default': constants.ATTR_NOT_SPECIFIED},
    }
}
SUB_RESOURCE_ATTRIBUTE_MAP = None
ACTION_MAP = {}
REQUIRED_EXTENSIONS = [l3.ALIAS, extraroute.ALIAS]
OPTIONAL_EXTENSIONS = []
ACTION_STATUS = {}

Note

[6] proposes the change of the hostroutes validator to accept a list of possible fields to validate, like: destination, nexthop and bfd_monitor_id.

BFD itself and fetching bfd_monitor session status information can be quite resource intensive operation (the information must be fetched from the backend), so new API endpoint is proposed to fetch the status of monitoring to not overload or affect ordinary router API operations.

The routes_status API endpoint gives feedback to the user if the monitored link is up.

If the bfd_monitor’s session_status is DOWN (BFD detects the given link dead) the given route should be removed in the backend (on the command line: ip r delete….), and set back if the monitor is UP again.

The bfd_monitor instance in the backend is created when the bfd_monitor is associated with any routes, the status of the bfd_monitor is set to ACTIVE from DOWN. The bfd_monitor’s status set to DOWN when the extra route updated and bfd_monitor value is set to empty, so the bfd_monitor is deleted in the backend.

As a bfd_monitor can be associated to many routes, if it is associated to at least one route and the creation in the backend is successful the status field of the bfd_monitor is ACTIVE, and it will change to DOWN when the last route association is deleted (the bfd_monitor is deleted from the last extra_route).

Backend

OVS can handle BFD on an interface and check the status of it. The problem with it is that ovs can have 1 BFD session per port, and to manage it ovsdb is the simplest way, but as LIU Yulong stated in [7] touching ovsdb from l3-agent can be weird.

A working solution with OVS is to create a BFD bridge, like br-bfd, and add veth ports to it, and enable BFD on those interfaces.

 --------
| br-int |
 --------
   | veth-xy-br-int
   |
   | veth-xy
 --------
| br-bfd |
 --------

Another possibility is to use os-ken, but during my experiments the BFD implementation in os-ken is not mature enough, and it needs ovsdb connection anyway, so I do not suggest os-ken to be used for BFD.

Data Model Impact

New table is created for bfd_monitors, routers table is changed to make it possible to associate a bfd_monitor to an extra route (a nexthop - destination pair).

Security Impact

None

Performance Impact

BFD and fetching bfd_monitor status information can be quite resource intensive operation as it can be fetched from the backend, it must be carefully documented.

Implementation

Assignee(s)

Primary assignee:: Lajos Katona <katonalala@gmail.com> (IRC: lajoskatona)

Work Items

REST API update.
DB schema update.
L3 agent update to handle BFD:
- RPC change to send BFD data to l3 agent.
- RPC change to fetch BFD status information from l3 agent.
CLI update.
Documentation.
Tests and CI related changes.

Testing

Unit Test
Functional test
API test

Perhaps this is something that can be easily tested end-to-end with fullstack tests. Need more investigation.

Documentation Impact

User Documentation

New API and changes to legacy APIs like routers must be documented.

References

Distributed metadata datapath for Openvswitch Agent

Thu, 29 Jul 2021 00:00:00

RFE: https://bugs.launchpad.net/neutron/+bug/1933222

When instances are booting, they will try to retrieve metadata from Nova by the path of Neutron virtual switches(bridges), virtual devices, namespaces and metadata-agents. After that, metadata agent has no other functionalities. In large-scale scenarios, a large number of deployed metadata agents will cause a waste of resources for hosts and message queue. Because metadata agents will start tons of external processes based on the number of users’ resources, report state to Neutron server for keepalive.

We are going to overcome this. This spec describes how to implement an agent extension for Neutron openvswitch agent to make the metadata datapath distributed.

Problem Description

How many metadata-agent should run for a large scale cloud deployment? There is no exact answer to this question. Cloud operators may setup metadata agent to all hosts, something like DHCP agent.

Config drive [1] can be an alternative for clouds to supply metadata for VMs. But there are some limitations and security problem for config drive. For instance, if you set config drive to use local disk, the live migration may not be available. Config drive uses extra storage device and mounting it inside VMs. Can not change userdata online for users specific scripts. The security issue is that because the mounting FS can be access by all users, if the metadata includes root password or key, the password and the key can be accessed by none root users.

If you are running Neutron agents, there is no alternative to replace metadata agent for cloud deployments.

As we can see, the metadata datapath is very long via many devices, namespaces and agents. One metadata path goes down, such as agent down or external process dead, will not only influence the host, but also all related hosts that will boot new VMs on.

Proposed Change

Adds an agent extension for Neutron openvswitch agent to make the metadata datapath distributed.

Note

This extension is only for openvswitch agent, other mechanism drivers will not be considered, because this new extension will rely on the openflow protocol and principle.

Solution Proposed

Metadata Special purpose IP and MAC

Each VM in one host will generate a Meta IP + MAC pair which is unique among VM on a host and which will be used to identify the metadata requestor vm/port. The agent extension will do the generation work when handle port in the first time, and store the Meta IP + MAC to ovsdb. When ovs-agent is restarting, handle_port funtion will try to read the ovsdb first to reload the Meta IP + MAC, if not exist, re-generate.

During the agent extension initialization, it will add a internal port named “tap-meta” to the ovs-bridge br-meta. This tap device will have a Meta Gateway MAC + IP.

Potential Configrations

Config options for Neutron openvswitch agent with this extension:

[METADATA]
provider_cidr = 100.100.0.0/16
provider_vlan_id = 998
provider_base_mac = "fa:16:ee:00:00:00"

provider_cidr will be used as the IP range to generate the VM’s metadata IP.
provider_vlan_id will be a fixed vlan for data from local tap-meta dev.
provider_base_mac will be the MAC prefix to generate the VM’s META MAC.

We can reuse the existing config option from metadata agent:

METADATA_PROXY_HANDLER_OPTS = [
    cfg.StrOpt('auth_ca_cert',
               help=_("Certificate Authority public key (CA cert) "
                      "file for ssl")),
    cfg.HostAddressOpt('nova_metadata_host',
                       default='127.0.0.1',
                       help=_("IP address or DNS name of Nova metadata "
                              "server.")),
    cfg.PortOpt('nova_metadata_port',
                default=8775,
                help=_("TCP Port used by Nova metadata server.")),
    cfg.StrOpt('metadata_proxy_shared_secret',
               default='',
               help=_('When proxying metadata requests, Neutron signs the '
                      'Instance-ID header with a shared secret to prevent '
                      'spoofing. You may select any string for a secret, '
                      'but it must match here and in the configuration used '
                      'by the Nova Metadata Server. NOTE: Nova uses the same '
                      'config key, but in [neutron] section.'),
               secret=True),
    cfg.StrOpt('nova_metadata_protocol',
               default='http',
               choices=['http', 'https'],
               help=_("Protocol to access nova metadata, http or https")),
    cfg.BoolOpt('nova_metadata_insecure', default=False,
                help=_("Allow to perform insecure SSL (https) requests to "
                       "nova metadata")),
    cfg.StrOpt('nova_client_cert',
               default='',
               help=_("Client certificate for nova metadata api server.")),
    cfg.StrOpt('nova_client_priv_key',
               default='',
               help=_("Private key of client certificate."))
]

Basic backend config with http for haproxy will be:

backend backend_{{ instance_x.uuid }}_{{ metadata_ip_x }}
server metasrv http://nova_metadata_host:nova_metadata_port

In haproxy side if ssl verify is enabled, the backend will be:

backend backend_{{ instance_x.uuid }}_{{ metadata_ip_x }}
server metasrv https://nova_metadata_host:nova_metadata_port ssl verify required ca-file /path/to/auth_ca_cert

Or ssl with client certificate:

backend backend_{{ instance_x.uuid }}_{{ metadata_ip_x }}
server metasrv https://nova_metadata_host:nova_metadata_port ssl verify required ca-file /path/to/nova_client_cert crt /path/to/nova_client_priv_key

Note

If the cloud has its own client certificate, the crt parameter can point to your client certificate file. But this is supported since haproxy version >= 2.4.

Metadata data pipelines

Sample Resource Datas

VM1 - network1 local_vlan_id=1, fixed_ip 192.168.1.10, port mac fa:16:3e:4a:fd:c1, Meta_IP 100.100.0.10, Meta_MAC fa:16:ee:00:00:11
VM2 - network2 local_vlan_id=2, fixed_ip 192.168.2.10, port mac fa:16:3e:4a:fd:c2, Meta_IP 100.100.0.11, Meta_MAC fa:16:ee:00:00:22
VM3 - network1 local_vlan_id=1, fixed_ip 192.168.1.20, port mac fa:16:3e:4a:fd:c3, Meta_IP 100.100.0.12, Meta_MAC fa:16:ee:00:00:33
VM4 - network3 local_vlan_id=3, fixed_ip 192.168.3.10, port mac fa:16:3e:4a:fd:c4, Meta_IP 100.100.0.13, Meta_MAC fa:16:ee:00:00:44
META Gateway IP 100.100.0.1, META Gateway MAC: fa:16:ee:00:00:01

TCP Egress

HTTP request packets from VM direct to br-meta, and change IP headers to tap-meta, add HTTP headers in host haproxy then go to nova-metadata API. Datapath:

+----+ TCP +-----------------------------------+ TCP +---------------------------------------------+ TCP +------------------------+ TCP +-----------------------+
|    +----->             Br-int                +----->                   Br-meta                   +----->        tap-Meta        +----->        Haproxy        |
| VM |     | From VM port + 169.254.169.254:80 |     |   Source (VM MAC + IP --> Meta MAC + IP)    |     |  Meta Gateway MAC + IP |     |   Match Meta IP       |
|    |     |          add local vlan           |     |  Dest (MAC + IP --> Meta Gateway MAC + IP)  |     |       Listened by      |     |     Add Http header   |
|    |     |           to Br-meta              |     |                 to tap-Meta                 |     |        Haproxy         |     |  to Nova-Metadata-API |
+----+     +-----------------------------------+     +---------------------------------------------+     +------------------------+     +-----------------------+

Flows (some keywords are pseudo code) on br-int:

Table=0
Match: ip,in_port=<of_vm1>,nw_dst=169.254.169.254 actions=mod_local_vlan:1,output:"To_br_meta"
Match: ip,in_port=<of_vm2>,nw_dst=169.254.169.254 actions=mod_local_vlan:2,output:"To_br_meta"
Match: ip,in_port=<of_vm3>,nw_dst=169.254.169.254 actions=mod_local_vlan:1,output:"To_br_meta"
Match: ip,in_port=<of_vm4>,nw_dst=169.254.169.254 actions=mod_local_vlan:3,output:"To_br_meta"

When your VM trying to access 169.254.169.254:80, what should the dest MAC + IP be? The dest IP is clear, it is 169.254.169.254. The complicated case is the dest MAC. We have three scenarios:

a. if your VM has only one default route which point to gateway, so the request dest MAC should be gateway MAC.

b. if your VM has a route which directly point to 169.254.169.254 (for instance, to 169.254.169.254 via 192.168.1.2 <the DHCP port IP>, normally, this is set by original DHCP-agent and metadata mechanism), so some ARP responder(s) will be added for such DHCP port IPs, in case of upgrading. A fake mac will be responded for these DHCP port IPs.

c. if your VM has a link route which is telling guest OS 169.254.169.254 is directly reachable. So an ARP responder for 169.254.169.254 will be added. So the dest MAC will be a fake one as well.

Flows on br-meta:

Table=0
Match: ip,in_port=<patch_br_int>,nw_dst=169.254.169.254 Action: resubmit(,80)

Table=80
Match: dl_vlan=<local_vlan_1>,dl_src=fa:16:3e:4a:fd:c1,nw_src=192.168.1.10 Action: strip_vlan,mod_dl_src:fa:16:ee:00:00:11,mod_nw_src:100.100.0.10, resubmit(,87)
Match: dl_vlan=<local_vlan_2>,dl_src=fa:16:3e:4a:fd:c2,nw_src=192.168.2.10 Action: strip_vlan,mod_dl_src:fa:16:ee:00:00:22,mod_nw_src:100.100.0.11, resubmit(,87)
Match: dl_vlan=<local_vlan_1>,dl_src=fa:16:3e:4a:fd:c3,nw_src=192.168.1.20 Action: strip_vlan,mod_dl_src:fa:16:ee:00:00:33,mod_nw_src:100.100.0.12, resubmit(,87)
Match: dl_vlan=<local_vlan_3>,dl_src=fa:16:3e:4a:fd:c4,nw_src=192.168.3.10 Action: strip_vlan,mod_dl_src:fa:16:ee:00:00:44,mod_nw_src:100.100.0.13, resubmit(,87)

Table=87
Match: tcp,nw_dst=169.254.169.254,tp_dst=80 Action: mod_nw_dst:100.100.0.1, mod_dl_dst:fa:16:ee:00:00:01,output:"tap-meta"

TCP Ingress

HTTP packets come from tap-meta to br-meta directly, then go to br-int and finnaly direct to VM. Datapath:

+----+     +---------------------+     +---------------------------------------+     +---------------+     +------------------+
|    |     |      Br-int         |     |                Br-meta                |     |    tap-Meta   |     |      Haproxy     |
| VM |     | From patch_br_meta  |     |    Source (IP ---> 169.254.169.254)   |     |               |     |  Http Response   |
|    | TCP |   mac_dst is VM     | TCP |  Dest(Meta MAC + IP ---> VM MAC + IP) | TCP |      To       | TCP |   To Client IP   |
|    <-----+   output to VM      <-----+               to  Br-int              <-----+ Meta MAC + IP <-----+  (Meta MAC + IP) |
+----+     +---------------------+     +---------------------------------------+     +---------------+     +------------------+

Flows on br-meta:

Table=0
Match: ip,in_port="tap-meta" actions=push_vlan,goto_table:91

Table=91
Match: dl_vlan=<998>,ip,nw_dst=100.100.0.10 Action: mod_vlan_vid:1,mod_dl_dst:fa:16:3e:4a:fd:c1,mod_nw_dst:192.168.1.10,mod_nw_src:169.254.169.254,output:"to-br-int"
Match: dl_vlan=<998>,ip,nw_dst=100.100.0.11 Action: mod_vlan_vid:2,mod_dl_dst:fa:16:3e:4a:fd:c2,mod_nw_dst:192.168.2.10,mod_nw_src:169.254.169.254,output:"to-br-int"
Match: dl_vlan=<998>,ip,nw_dst=100.100.0.12 Action: mod_vlan_vid:3,mod_dl_dst:fa:16:3e:4a:fd:c3,mod_nw_dst:192.168.1.20,mod_nw_src:169.254.169.254,output:"to-br-int"
Match: dl_vlan=<998>,ip,nw_dst=100.100.0.13 Action: mod_vlan_vid:4,mod_dl_dst:fa:16:3e:4a:fd:c4,mod_nw_dst:192.168.3.10,mod_nw_src:169.254.169.254,output:"to-br-int"

Flows on br-int:

Table=0
Match: ip,in_port=<patch_br_meta>,dl_vlan=1,dl_dst=<vm1_mac_fa:16:3e:4a:fd:c1>,nw_src=169.254.169.254 actions=strip_vlan,output:<of_vm1>
Match: ip,in_port=<patch_br_meta>,dl_vlan=2,dl_dst=<vm2_mac_fa:16:3e:4a:fd:c2>,nw_src=169.254.169.254 actions=strip_vlan,output:<of_vm2>
Match: ip,in_port=<patch_br_meta>,dl_vlan=3,dl_dst=<vm3_mac_fa:16:3e:4a:fd:c3>,nw_src=169.254.169.254 actions=strip_vlan,output:<of_vm3>
Match: ip,in_port=<patch_br_meta>,dl_vlan=4,dl_dst=<vm4_mac_fa:16:3e:4a:fd:c4>,nw_src=169.254.169.254 actions=strip_vlan,output:<of_vm4>

ARP for Metadata IPs

Tap-meta device will be resident on host kernel IP stack, before the first response of TCP, the host (protocol stack) needs to know the META_IP’s MAC address. So ARP reqeust is broadcast. ARP will be sent from tap-meta device to br-meta responder. The ARP responder datapath:

+---------------------------+      +---------------+
|         Br-meta           |      |    tap-Meta   |
| ARP Responder for Meta IP |      |     Learn     |
|           to              | ARP  | Meta IP's MAC |
|         INPORT            <------+     (ARP)     |
+---------------------------+      +---------------+

The flows on br-meta will be:

Ingress:
Table=0
Match: arp,in_port="tap-meta" Action: resubmit(,90)

Table=90
Match: arp,arp_tpa=100.100.0.10 Action: ARP Responder with Meta_MAC fa:16:ee:00:00:11,IN_PORT
Match: arp,arp_tpa=100.100.0.11 Action: ARP Responder with Meta_MAC fa:16:ee:00:00:22,IN_PORT
Match: arp,arp_tpa=100.100.0.12 Action: ARP Responder with Meta_MAC fa:16:ee:00:00:33,IN_PORT
Match: arp,arp_tpa=100.100.0.13 Action: ARP Responder with Meta_MAC fa:16:ee:00:00:44,IN_PORT

Host haproxy configurations

The host haproxy is one only process which is used for all VMs. The host haproxy will add HTTP headers to the metadata request which is needed for metadata API. The headers have a fixed algorithm which is easy to assemble. For each VM’s request, haproxy will add an independent backend and a match rule of checking the source IP (aka Meta_IP). While the request from one VM’s (Meta_IP) it will be send to the matched backend, which add HTTP headers and then send to real nova-metadata-api.

Configurations:

global
    log         /dev/log local0 {{ log_level }}
    user        {{ user }}
    group       {{ group }}
    maxconn     {{ maxconn }}
    daemon

frontend public
    bind            *:80 name clear
    mode            http
    log             global
    option          httplog
    option          dontlognull
    maxconn         {{ maxconn }}
    timeout client  30s

    acl instance_{{ instance_1.uuid }}_{{ metadata_ip_1 }} src {{ metadata_ip_1 }}
    acl instance_{{ instance_2.uuid }}_{{ metadata_ip_2 }} src {{ metadata_ip_2 }}
    acl instance_{{ instance_3.uuid }}_{{ metadata_ip_3 }} src {{ metadata_ip_3 }}
    acl instance_{{ instance_4.uuid }}_{{ metadata_ip_4 }} src {{ metadata_ip_4 }}

    use_backend backend_{{ instance_1.uuid }}_{{ metadata_ip_1 }} if instance_{{ instance_1.uuid }}_{{ metadata_ip_1 }}
    use_backend backend_{{ instance_2.uuid }}_{{ metadata_ip_2 }} if instance_{{ instance_2.uuid }}_{{ metadata_ip_2 }}
    use_backend backend_{{ instance_3.uuid }}_{{ metadata_ip_3 }} if instance_{{ instance_3.uuid }}_{{ metadata_ip_3 }}
    use_backend backend_{{ instance_4.uuid }}_{{ metadata_ip_4 }} if instance_{{ instance_4.uuid }}_{{ metadata_ip_4 }}

backend backend_{{ instance_1.uuid }}_{{ metadata_ip_1 }}
    balance         roundrobin
    retries         3
    option redispatch
    timeout http-request    30s
    timeout connect         30s
    timeout server          30s

    http-request set-header X-Instance-ID {{ instance_1.uuid }}
    http-request set-header X-Tenant-ID {{ instance_1.project_id }}
    http-request set-header X-Instance-ID-Signature {{ instance_1.signature }}

    server metasrv ...

backend backend_{{ instance_2.uuid }}_{{ metadata_ip_2 }}
    balance         roundrobin
    retries         3
    option redispatch
    timeout http-request    30s
    timeout connect         30s
    timeout server          30s

    http-request set-header X-Instance-ID {{ instance_2.uuid }}
    http-request set-header X-Tenant-ID {{ instance_2.project_id }}
    http-request set-header X-Instance-ID-Signature {{ instance_2.signature }}

    server metasrv ...

backend backend_{{ instance_3.uuid }}_{{ metadata_ip_3 }}
    ...

backend backend_{{ instance_4.uuid }}_{{ metadata_ip_4 }}
    ...

IPv6 metadata

The metadata for IPv6 [2] only network has similar address fe80::a9fe:a9fe, so all these works can be mirrored for IPv6. For IPv6 the generator will use the range fe80:ffff:a9fe:a9fe::/64 to allocate Meta_IPv6 address. The Meta_gateway_IPv6 address will be fe80:ffff:a9fe:a9fe::1, Gateway MAC is still the same one fa:16:ee:00:00:01. NA responder for Meta_IPv6 address fe80:ffff:a9fe:a9fe::abcd and Meta_MAC fa:16:ee:00:00:11 will be:

Table=0
Match: icmp6,icmp_type=135,icmp_code=0,nd_sll=fa:16:ee:00:00:01,
actions=set_field:136->icmpv6_type,set_field:0->icmpv6_code,set_field:2->icmpv6_options_type,goto_table:90

Table=90
Match: icmp6,icmp_type=136,icmp_code=0,nd_target=fe80:ffff:a9fe:a9fe::abcd
actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],set_field:fa:16:ee:00:00:11->eth_src,
move:NXM_NX_IPV6_SRC[]->NXM_NX_IPV6_DST[],set_field:fe80:ffff:a9fe:a9fe::abcd->ipv6_src,
set_field:fa:16:ee:00:00:11->nd-tll,set_field:0xE000->OFPXMT_OFB_ICMPV6_ND_RESERVED,IN_PORT

Implementation

Assignee(s)

LIU Yulong <i@liuyulong.me>

Work Items

Adding flows action for ovs bridges
Adding host haproxy manager for ovs-agent
Adding host metadata IP and Mac generator with ovsdb settings
Adding ovs-agent extension to set up flows for VM ports
Testing.
Documentation.

Dependencies

None

Testing

Test cases to verify the metadata can be set properly, this can be done by using existing jobs with new new metadata driver.

References

OVS auxiliary bridge to reduce live migration networking disruption in OVN

Fri, 02 Jul 2021 00:00:00

Launchpad bug: https://bugs.launchpad.net/neutron/+bug/1933517

The goal of this spec is to propose a change in the VM - backend (OVN) connectivity to improve the live migration process.

Problem Description

The live migration is a very sensitive process that implies to configure a destination host to execute a virtual machine that is already running. When the destination hypervisor is prepared, the source virtual machine is paused and immediately unpaused in the destination host.

Since [1] and [2], Nova is capable of plugging the port and create the interface in the destination host during the pre-live migration. But the TAP device is created by libvirt when the VM is unpaused. This is when the port interface is assigned with an OpenFlow port ID.

When the port is created, the network backend detects the new port attached and informs to the OVN Northbound and this event is received by Neutron. The network backend is commanded to configure the needed rules for this new port.

The problem with this sequence of actions is that when the virtual machine is unpaused, the network backend is not ready to continue any network communication until the OVN controller (in the compute node) has set all needed OpenFlow rules and chassis configuration, using the OpenFlow port ID (interface.ofport) assigned when the TAP interface is created.

Proposed Change

The spec scope is limited to the OVN backend. This spec does not consider the case of HW offloaded ports, that have other plug process. This spec is only considering os_vif.objects.vif.VIFOpenVSwitch VIF types within the related network backend.

This spec proposes to create an intermediate OVS bridge between the integration bridge and the virtual machine tap interface. This OVS bridge will be connected to the integration bridge with a patch port. The TAP interface will be plugged into the intermediate bridge when the virtual machine is created or unpaused (that happens during the live migration). This architecture is very similar to OVS hybrid plugging, but with an OVS bridge in between.

The advantage of this approach is that the integration bridge port, in this case the patch port, that is created during the pre-live migration process, has a valid OpenFlow port ID (interface.ofport), needed to provide the correct OpenFlow rules in the integration bridge.

After the port is plugged, Nova commands to libvirt to copy the guest memory to the destination host. If “live_migration_permit_post_copy” is used [3], the virtual machine on the destination host will be activated before all its memory has been copied. However there is a period of time where the port is bound to the destination host and the virtual machine is still running on the source host. During this period of time, the virtual machine won’t be able to communicate, same as without this feature.

When the virtual machine is unpaused in the destination host, the OVN backend is ready to immediately continue transmitting packets from the guess.

Implementation

The VIF plug and unplug process is executed by Nova, using the different backend implementations provided by os-vif library. The bridge creation and deletion will be done during the plug and unplug processes [4], as in OVS hybrid plug strategy.

The proposed architecture and naming is the following one, implemented in [5]:

      ┌────────┐
      │tap-xxx │           TAP interface port
┌─────┴────────┴─────┐
│      pbr-xxx       │     Port bridge
└─────┬────────┬─────┘
      │pbp-xxx │           Port bridge patch port (port bridge side)
      └────┬───┘
      ┌────┴───┐
      │ipb-xxx │           Port bridge patch port (integration bridge side)
┌─────┴────────┴─────┐
│    integration     │
│      bridge        │
└────────────────────┘

For each new VM TAP interface, an OVS bridge is created along with the patch port to connect this bridge with the integration bridge. This port bridge will have a default OpenFlow rule, allowing all traffic from the TAP interface port to the patch port:

cookie=0x0, duration=84162.020s, table=0, n_packets=444223, n_bytes=832399534, priority=0 actions=NORMAL

OVN links the OVS ports (Open vSwitch DB) with the Logical Switch Ports (OVN NB database) using the Neutron DB port ID. The OVN NB Logical Switch Port, created by Neutron, will have this port ID in the lsp.name field. The OVS port ipb-xxx (the integration bridge side patch port) is created with interface.external_ids={iface-id: port_id}. The ovn-controller will detect this new port and assign to OVN SB Port Binding the corresponding chassis. The interface.external_ids information is added by os-vif during the port plug process [5], called by Nova.

Note: OVS and OVN ports are represented in os-vif with the same VIF type, objects.vif.VIFOpenVSwitch. The os-vif implementation could be used both for OVS and OVN backends. However, the scope of this spec is limited to OVN backend only.

Nova - Neutron events

Nova and Neutron have a mechanism to communicate the state of the VIFs [6]. This is a unidirectional API that allows Neutron to inform Nova when a VIF has been plugged or unplugged. Nova expects from Neutron a network-vif-plugged event when the port has been bound or plugged. Depending on the backend, Nova will expect a network-vif-plugged when the port has been bound or when the port has been plugged [7]. For example, in OVS with hybrid plug, Nova expects the event when the port has been plugged to the OVS. In OVN, Nova is expecting this event when the port has been bound during the live migration process [8].

Once Nova creates the intermediate bridge [1] [2], Neutron can send the network-vif-plugged event when the port is plugged and the network backend is configured.

This spec proposes to use the existing OVN event infrastructure to capture the patch port creation event, using LogicalSwitchPortCreateUpEvent, and in the handler method push the event to Nova. Along with this change, Neutron will inform Nova about the backend used in the port adding a field in port.vif_details called backend. The value will be “ovn”. This port update will be done in [9], in the mech driver port binding method.

Neutron configuration

As explained in the previous section, this spec proposes to change the event emission from Neutron. That will depend on if os-vif is configured to create this intermediate bridge between the integration bridge and the TAP device.

Because Neutron cannot read the os-vif configuration, this spec proposes to add the same config flag in the ML2 OVN plugin section: “per_port_bridge”.

If enabled, Neutron will send the network-vif-plugged when the port is plugged, not when the port binding is updated.

QoS

With this feature, the Logical Switch Port is now the patch port between the port bridge and the TAP device, not the TAP device port. In OVS, QoS rules can only be applied to external ports. However, in OVN the QoS rules are applied using the “match” field in the QoS register [10]. This is a string in the same expression language used in the Logical_Flow table. The traffic shaping applied to the TAP device is the same when using the patch port reference because the traffic flow is the same in both ports.

Performance

The intermediate bridge will have one single rule, with action NORMAL. That means all traffic coming from the TAP port will be sent to the patch port and vice versa.

The datapath will collapse this bridge resulting in an identical flow as without the bridge. In other words, the new intermediate bridge won’t have any effect on packet processing performance.

OVN DPDK

Same as with other bridges (physical bridges, tunnel bridges, external bridges), each port bridge will be connected to the integration bridge with a patch port. That keeps one single datapath (“netdev” in DPDK case), as commented in the previous section.

Upgrade Impact

This feature could be enabled in os-vif using the configuration variable “per_port_bridge” [11]. If this option is enabled, any OVN port will be plugged and unplugged using the implementation described in this spec. Currently this implementation does not handle mixing both plugging methods (connecting the TAP port directly to the integration bridge or creating the port bridge as described in this spec). If this option is enabled, the compute node should not host any virtual machine. At this point, as described in the proposed change, any virtual machine created or migrated to this host, will be plugged using the intermediate port bridge. In case of migration, that will solve or mitigate the connectivity problems when the virtual machine is unpaused in the destination host.

This spec does not cover any scenario mixing TAP ports using both plug strategies on the same host (same as with OVS hybrid plugging strategy).

Testing

Tempest Tests

Live migration tempest tests are covered in the Nova CI. In order to test this feature, a set of tests from tempest.api.compute.admin.test_live_migration should be executed on neutron-ovn-tempest-slow CI job.

Documentation Impact

User Documentation

Document the new architecture and the configuration flag to change the event emission from Neutron.

Implementation

Assignee(s)

Sean K Mooney (smooney@redhat.com, IRC: sean-k-mooney) Rodolfo Alonso Hernandez (ralonsoh@redhat.com, IRC: ralonsoh)

References

Neutron VLAN networks with QinQ support

Tue, 29 Jun 2021 00:00:00

Launchpad Bug: https://bugs.launchpad.net/neutron/+bug/1915151

Neutron supports networks with vlan transparency but in case where it is used with VLAN networks, there is no way to use it as QinQ which is different ethertype (e.g. 0x8a88) than regular vlan ethertype (0x8100).

Problem Description

Networks with enabled vlan transparency can be of any type available in Neutron (vxlan, vlan, flat, etc.). In most cases it is fine when packets sent from the node and have ethertype 0x8100 (802.1q). But in such case as we have 2 vlans already (S-Tag and C-Tag) there are some use cases where it should be sent as QinQ which is slightly different standard and have different ethertype (0x8a88) [1].

Proposed Change

Add new API extension to extend network resource in Neutron with qinq attribute. This new attribute would be valid only for vlan networks and in case of that networks would work similary to what current VLAN transparency extension [2]. The difference between vlan transparency and new qinq extensions would be in the ethertype configured for packets in such network send from the node. In case of qinq=True S-Tag and C-Tag will work in same way like it is now for the networks with enabled vlan transparency. C-Tag is configured directly in the instance by user and S-Tag is configured and controlled by Neutron. The difference will be that ethertype of the packets sent from the node will be 0x8a88. Attributes vlan_transparent and qinq will be mutually exclusive. Only one of them can be set to True for the network.

Server side changes

A new API extension of Neutron will be added with new attribute for the network resource. This new attribute will be called qinq:

RESOURCE_ATTRIBUTE_MAP = {
    network.COLLECTION_NAME: {
        "qinq": {
            'allow_post': True,
            'allow_put': False,
            'convert_to': converters.convert_to_boolean,
            'default': constants.ATTR_NOT_SPECIFIED,
            'is_visible': True,
            'is_filter': True
        }
    }
}

Note

Attribute qinq will have effect only for vlan networks.

Note

Attributes vlan_transparent and qinq would be mutually exclusive. Only one of them can be set to True.

Agent side changes

Currently linuxbridge, ovn and SR-IOV backends supports VLAN transparency and QinQ will be supported by the same backends. In case of linuxbridge and ovn backends correct ethertype will need to be set on the nodes so correct ethertype will be set for the packets. In case of the SR-IOV backend support for that feature will depend on the hardware and its driver and that will need to be tested separately.

DB Impact

Extend the networks table with boolean column qinq.

REST API Impact

None.

Client Impact

Relevant changes in osc and openstacksdk to add support for new network’s attribute. To enable it for network, it should be something like:

openstack network create --qinq

and to disable it:

openstack network create --no-qinq

Testing

Unit tests.
Fullstack test.
Tempest tests in neutron-tempest-plugin.

Assignee(s)

Slawek Kaplonski <skaplons@redhat.com>

References

Node-Local virtual IP address (Local IP)

Thu, 24 Jun 2021 00:00:00

RFE: https://bugs.launchpad.net/neutron/+bug/1930200

This spec proposes a new type of shared virtual IP address that can be used to access distributed and/or multi-node services and applications running in the cloud. The feature is primarily focused on high efficiency and performance of the networking data plane for very large scale clouds and/or clouds with high network throughput demands.

Problem Description

In a topology where several workloads access a shared resource over network obvious bottlenecks are the network and the shared resource itself (or a loadbalancer):

Proposed feature suggests to solve this bottleneck issue by deploying local replicas/caches of the shared service and re-translating local workloads` requests to these replicas transparently. Transparency is achieved by assigning same IP address to local replicas as on main shared resource.

Primary use-cases for this feature are listed in the RFE description. While the proposed virtual IP may be used on its own for simple use cases, it is mainly intended to serve as a basic building block and should be used in conjunction with other services to provide full-fledged service discovery, load balancing, proxying and other advanced features.

Proposed Change

Add a new Node Local IP (or just Local IP) resource - a virtual IP that can be shared across multiple ports/VMs (similar to anycast IP) and is guaranteed to only be reachable within the same physical server/node boundaries. The mechanism is similar to Floating IP, with the difference that Local IP could be assigned to several fixed IPs (on different nodes) simultaneously and that all address translation happens early/locally on the node’s integration bridge.

The scope of this feature is limited to:

API and CLI to provide fine grained control over allocation and assignment of the Local IP addresses
Networking data plane mechanism to efficiently redirect traffic to a port within the same physical node (if present) or to optionally forward it to a fallback destination
Local IP will only be accessible within the same L2 segment

Local IP object will be composed of the following objects & attributes:

Underlying Neutron port representing Local IP
Collection of associated Neutron ports that will serve as packets’ destination when the packet source port is located on the same physical node
IP mode for local ports:
- ‘translate’ (default) - Local IP will behave as DNAT for redirected packets
- ‘passthrough’ - packets will be redirected without modification to IP headers; destination VM’s guest OS will be responsible for configuring Local IP’s IP address on a corresponding port interface. This is similar to current allowed address pair mechanism with the benefit of fine grained control over IP allocation and assignment
Fallback policy:
- ‘forward’ (default) - to redirect packets to an underlying Neutron port IP address if no Local IP assigned port is available on the physical node
- ‘drop’ - packets will be dropped if no Local IP assigned port is available on the physical node - out of scope for now
- ‘reject’ - similar to iptables’ ‘reject’ policy to help clients fail fast if no Local IP assigned port is available on the physical node - out of scope for now

REST API Impact

New Local IP API resource

Attribute	Type	Req	CRUD	Description
id	uuid-str	Yes	R	Unique identifier for the Local IP object.
name	String	No	CRU	Human readable name for the Local IP (255 characters limit). Does not have to be unique.
description	String	No	CRU	Human readable description for the Local IP (255 characters limit).
project_id	uuid-str	No	CR	Owner of the Local IP.
local_ip	String	No	CR	Local IP CIDR (virtual) that will be reachable within the same physical server/node. Required when provided local_port_id’s Port has several fixed IPs
local_port_id	uuid-str	Yes	CR	Underlying (backup) Neutron port ID used by Local IP object to get actual IP address for local translation. If port has several fixed IPs - local_ip should be provided as well.
network_id	uuid-str	Yes	CR	ID of network from which to allocate Local IPs underlying Neutron port. Required when no local_port_id was provided upon creation.
ip_mode	String	No	CRU	One of ‘translate’ (for DNAT) or ‘passthrough’ (no NAT) modes described above. Default: ‘translate’

New Local IP Association API resource

Attribute	Type	Req	CRUD	Description
local_ip_id	uuid-str	Yes	CR	Local IP id
fixed_port_id	uuid-str	Yes	CR	ID of associated Neutron fixed port
fixed_ip	String	No	CR	Exact port’s fixed IP address that is associated with Local IP
host	List	No	R	Host where associated port is bound

Basically new API will be similar to existing Floating IP API with some differences:

Local IP could be associated with multiple fixed ports at the same time.
Local IP may be created by providing an existing Neutron port which will be used to obtain the IP address and may serve as a fallback destination
Creating Local IP without explicitly specified port will trigger creation of an underlying Neutron port to handle IPAM, network boundaries and permission checks. In this case network ID should be specified upon creation
Deletion of the Local IP underlying fallback port will be prohibited by the Neutron server as long as a Local IP associated with such port exists
Manual underlying Neutron port updates will be ignored by parent Local IP (similar to current Floating IP behavior)
Deleting Local IP should be prohibited if Local IP has associated local ports; ‘force’ option may be provided to trigger local port disassociation on deletion; deleting Local IP should also delete the underlying Neutron port representing Local IP if this port was created specifically for Local IP and did not exist before
IPAM and IP validation will be handled automatically by underlying Port
Local IP will behave similar to Port and Floating IP API in terms of user/admin access rights
Local IP may have quotas similar to Floating IP API (out of scope for now)

Data Model Impact

The following are the backend database tables for the REST API proposed above.

Local IPs

Attribute	Type	Req	Description
id (PK)	uuid-str	Yes	Unique identifier for the Local IP object.
name	string	No	Human readable name for the Local IP (255 characters limit). Does not have to be unique.
project_id	uuid-str	No	Owner of the Local IP. Only admin users can specify a project identifier other than their own.
local_ip	string	Yes	Local IP CIDR (virtual) that will be reachable within the same physical server/node.
local_port_id (FK)	uuid-str	Yes	Underlying (backup) Neutron port ID used by Local IP object to get actual IP address for local NATting
ip_mode	string	No	One of ‘translate’ (for DNAT) or ‘passthrough’ (no NAT) modes described above. Default: ‘translate’

Local IP associations

Attribute	Type	Req	Description
local_ip_id (PK)	uuid-str	Yes	UUID of a Local IP
fixed_port_id (PK)	uuid-str	Yes	UUID of a port which will serve Local IP requests on this port’s host
fixed_ip	String	Yes	Exact port’s fixed IP address that is associated with Local IP

Client impact

New CLI and OpenStack client commands to create, update, delete, list and show Local IPs and to associate/disassociate fixed IP ports

OVS, ML2 Drivers impact

RPC interface between server and OVS agent will be updated to include info about Local IPs associated with agent’s ports. Server will update port_details with this info.

OVS agent will perform following flow updates based on Local IPs info:

update ARP spoofing flow for associated port (like done for allowed address pairs)
identify network/local vlan of associated port
for each port from this network - redirect packets from table 0 or 24(ARP_SPOOF)/25(MAC_SPOOF) to a new table - 50 (LOCAL_EGRESS)

Below flow examples are for the case when fixed port (id=56fbc1e7-2c.., MAC=fa:16:3e:9a:1a:de, IP=10.0.0.51) is associated with Local IP 10.0.0.10

table=25, n_packets=0, n_bytes=0, priority=2,in_port="tap56fbc1e7-2c",
dl_src=fa:16:3e:9a:1a:de actions=resubmit(,50)

table 50(LOCAL_EGRESS) serves for memorizing local vlans to later distinguish Local IP traffic from different nets and redirect packets further to table 51(LOCAL_IP_HANDLE)

table=50, n_packets=0, n_bytes=0, priority=9,in_port="tap56fbc1e7-2c"
actions=load:0x4->NXM_NX_REG6[],resubmit(,51)

table 51(LOCAL_IP_HANDLE) has actual flows for Local IP handling

ARP responder flow to handle Local IP ARP requests from same subnet ports (fixed port and local IP associated with it are from the same IP subnet)

table=51, n_packets=0, n_bytes=0, priority=1,arp,reg6=0x4,
arp_tpa=10.0.0.10
actions=load:0x2->NXM_OF_ARP_OP[],
move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],
move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],
load:0xfa163e9a1ade->NXM_NX_ARP_SHA[],
load:0x1010102->NXM_OF_ARP_SPA[],
move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],
mod_dl_src:fa:16:3e:9a:1a:de,IN_PORT

NAT flows to do actual Local IP address translation

table=51, n_packets=0, n_bytes=0, priority=10,ip,reg6=0x4,
nw_dst=10.0.0.10
actions=ct(commit,table=60,zone=4,nat(dst=10.0.0.51))

table=51, n_packets=0, n_bytes=0, priority=10,ct_state=-trk,ip,reg6=0x4,
dl_src=fa:16:3e:9a:1a:de,nw_src=10.0.0.51 actions=ct(table=60,zone=4,nat)

After translation packets are resubmitted further to TRANSIENT_TABLE (60)

Alternatively static NAT could be used to avoid kernel conntrack: learn back flows with 7 tuple (src_mac, dest_mac, src_ip, dest_ip, protocol, src_protocol_port, dest_protocol_port)

table=51, priority=30,tcp,dl_vlan=1,dl_dst=fa:16:3e:9a:1a:de,
dl_src=fa:16:3e:4f:26:fd,nw_src=10.0.0.100,nw_dst=10.0.0.10,
tp_dst=0x8000/0x8000
actions=learn(table=62,idle_timeout=30,hard_timeout=1800,priority=90,
eth_type=0x800,nw_proto=6,NXM_OF_ETH_SRC[]=NXM_OF_ETH_DST[],
NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],NXM_OF_IP_SRC[]=NXM_OF_IP_DST[],
NXM_OF_IP_DST[]=NXM_OF_IP_SRC[],NXM_OF_TCP_DST[]=NXM_OF_TCP_SRC[],
NXM_OF_TCP_SRC[]=NXM_OF_TCP_DST[],NXM_OF_VLAN_TCI[0..11],
load:NXM_NX_REG0[0..11]->NXM_OF_VLAN_TCI[0..11],
output:NXM_OF_IN_PORT[]),set_field:10.0.0.10->ip_dst,goto_table:60

This is usable in case of OVS offloading (e.g. SmartNICs). This will be made configurable on the ovs agent side.

Not matched (not related to Local IP) packets are transmitted further

table=51, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,60)

Packet processing example:

VM1 <10.0.0.100> tries to access Local IP 10.0.0.10
Local IP is assigned to port with MAC_2/10.0.0.51:

ARP:
VM1 -> ARP -> 10.0.0.10 and response <10.0.0.10 has MAC_2>

Egress:
<VM1_MAC_1 + 10.0.0.100> to <MAC_2 + 10.0.0.10>
<<ct NAT>>
<VM1_MAC_1 + 10.0.0.100> to <MAC_2 + 10.0.0.51>

Ingress (back):
<MAC_2 + 10.0.0.51> to <VM1_MAC_1 + 10.0.0.100>
<<ct NAT>>
<MAC_2 + 10.0.0.10> to <VM1_MAC_1 + 10.0.0.100>

Scheduling

Local IP will have no effect on VM scheduling/placement. Local IP operates under assumption that at any given time any physical node in the cloud should have no more than one active port/VM associated with a given Local IP. Users are responsible to utilize OpenStack placement/scheduling features (e.g. anti-affinity rules) to avoid assigning multiple ports from the same physical node the the same Local IP.

However there might be valid cases when multiple Local IP’s local ports may be colocated on the same physycal node (e.g. as a result of temporary live migration during other node’s maintenance). Local IP should provide a deterministic way of handling such situations (e.g. in case of multiple local ports, only the oldest port shall be used).

Initial release limitations

Only IPv4 will be supported. IPv6 support will be considered in future releases
Only ‘openvswitch’ ML2 mechanism driver will support the feature
Only tunnel (VxLAN, GRE) networks will be supported. ‘vlan’ will be considered if require minimum overhead
No deterministic handling of packets if a node contains multiple local ports from same L2 segment associated with the same Local IP
no ‘reject’ and ‘drop’ fallback policies; only ‘forward’ will be supported

References

Manage default route(s) explicitly

Tue, 16 Mar 2021 00:00:00

RFE: https://launchpad.net/bugs/1921126

This spec proposes to allow explicit management of the default route(s) of a Neutron router. This is mostly useful for a user to install multiple default routes for Equal Cost Multipath (ECMP) and treat all these routes uniformly.

Problem Description

Today the extraroute API allows explicit addition of default route(s), where by default route we mean a route with destination=0.0.0.0/0 and an arbitrary (connected) nexthop. But the router already has an implicit default route derived from its external gateway settings (with its nexthop being the gateway port’s first subnet’s gateway_ip). To the best of my knowledge the interaction of the default routes out of these two sources is problematic. For example our api-ref has warnings about it [1]. Other times people not familiar with these warnings consider this behavior outright buggy [2].

We also already have a spec merged for ECMP [3]. ECMP makes as much sense for the default route as for any other route. But today if you want to manage multiple default route entries in order to have ECMP over the default route, then you have to manage a mishmash of the implicit default route and some other default routes via the extraroute API.

Proposed Change

This spec proposes to separate the implicit management of a router’s default route from the explicit. A router either has the one traditional implicit default route or all of its default route(s) are explicitly managed via the extraroute API. But never a mixture of these two.

We propose to have a new API extension: default-route-management. Which introduces a new router attribute: explicit_default_routes. Which is a boolean and defaults to False. It can be set at router create and update.

When explicit_default_routes is set to False (implicit mode) the default route of a router is derived from its external gateway as before. This default route is not visible through the extraroute API (again as before). For the sake of backward compatibility we should not reject default routes on the extraroute API. However we could document that in implicit mode managing default routes over the extraroute API is not recommended.

Updating a router’s explicit_default_routes (in both directions) is rejected except when a router’s routes attribute contains:

either no default routes,
or exactly one default route with the same nexthop as in implicit mode.

When explicit_default_routes is set to True (explicit mode) the Neutron router is not going to have any default routes installed implicitly. However a router can be updated from implicit to explicit mode without traffic loss if the one extra route allowed above (that is: exactly one default route with the same nexthop as in implicit mode) is present before switching to explicit mode. In this case the default route is preserved through the update.

In explicit mode default routes can be managed via the extraroute API. And they can be managed properly - for example all of the default routes set are visible via the extraroute API.

DB Impact

Extend the routers table with boolean column explicit_default_routes defaulting to False.

REST API Impact

New API extension: default-route-management introducing new router attribute: explicit_default_routes.

Client Impact

Relevant changes in osc and openstacksdk.

Testing

Unit tests.
Tempest tests in neutron-tempest-plugin.

Assignee(s)

Bence Romsics <bence.romsics@gmail.com>

References

QoS Rule Type Packet per Second

Thu, 11 Mar 2021 00:00:00

RFE: https://bugs.launchpad.net/neutron/+bug/1912460

Neutron supports bandwidth rate limit for ports and L3 IPs. But packet rate limit (packet per second) is not available, although it is a common measurement.

So, this spec describes adding a new QoS rule type packet per second (pps).

Problem Description

Packet per second is a very general network performance metric. Like bandwidth, it is usually used to evaluate the packet forwarding performance of a device.

For cloud providers, to limit the packet per second (pps) of VM NIC is popular and sometimes essential. Transit large set of packets for VM in physical compute hosts will consume the CPU and physical NIC I/O performance. For small packets, even if the bandwidth is low, the pps can still be higher. Without the limitation, it can be an attack point inside the cloud while some VMs are becoming hacked.

For L2 drivers like ovs and ovn, it may get extremly high usage of CPU when user send small packet (typically 64B small) from the VM to the others, even if the device has lower QoS bandwidth limitation. Then your host services and other users’ VMs will be under a higher failure point.

For network quality assurance, the resource consumption of the system is determined according to the user’s VM specifications. With the pps limitation, the VM will not consume more CPU with smaller bandwidth.

Proposed Change

Adding new API extension to QoS service plugin to allow CURD actions for packet rate limit (packet per second) rule.

Note

We will not elaborate the real limitation in L2/L3 backend, this spec only shows how we add a new QoS rule type.

Server side changes

A new API extension of Neutron will be added with new resource PacketRateLimitRule:

qos_apidef.SUB_RESOURCE_ATTRIBUTE_MAP = {
    'packet_rate_limit_rules': {
        'parent': qos_apidef._PARENT,
        'parameters': {
            qos_apidef._QOS_RULE_COMMON_FIELDS,
            'max_kpps': {
                'allow_post': True, 'allow_put': True,
                'convert_to': converters.convert_to_int,
                'is_visible': True,
                'is_filter': True,
                'is_sort_key': True,
                'validate': {
                    'type:range': [0, db_const.DB_INTEGER_MAX_VALUE]}
            },
            'max_burst_kpps': {
                    'allow_post': True, 'allow_put': True,
                    'is_visible': True, 'default': 0,
                    'is_filter': True,
                    'is_sort_key': True,
                    'convert_to': converters.convert_to_int,
                    'validate': {
                        'type:range': [0, db_const.DB_INTEGER_MAX_VALUE]}
            },
            'direction': {
                'allow_post': True,
                'allow_put': True,
                'is_visible': True,
                'is_filter': True,
                'is_sort_key': True,
                'default': constants.EGRESS_DIRECTION,
                'validate': {
                    'type:values': constants.VALID_DIRECTIONS}
            }
        }
    }
}

Note

The unit for the rate and burst is kilo (1000) packets per second, so the value range will be 1 kpps to 2147 gpps.

Potential agent side enhancements

Each of the following will be an independent and huge proposal, we will not describe the detail here.

apply pps rule to L3 IPs in agent side by iptables rule [1].
apply pps rule to VM port by ovs meter [2] [3] [4].
apply pps rule to router ports (gateway port, router interface) by iptables rule.
apply pps rule to L3 IPs and ports to OVN related devices by ovs meter.

User use cases

After the L2/L3 backends have the real abilities to limit the pps, users can use the pps rule in the following scenarios:

“pps” rule for floating IP, using the L3 agent
“pps” rule for floating IP, using OVN L3 agent
“pps” rule for gateway IP, using the L3 agent
“pps” rule for gateway IP, using OVN L3 agent
“pps” rule for ML2 OVS VM ports
“pps” rule for ML2 OVN VM ports
“pps” rule for router gateway or interface ports

Data Model Impact

Add table qos_packet_rate_limit_rules:

op.create_table(
    'qos_packet_rate_limit_rules',
    sa.Column('id', sa.String(36), nullable=False,
              index=True),
    sa.Column('qos_policy_id', sa.String(36),
              nullable=False, index=True),
    sa.Column('max_kpps', sa.Integer()),
    sa.Column('max_burst_kpps', sa.Integer()),
    sa.Column('direction', sa.Enum(constants.EGRESS_DIRECTION,
                                   constants.INGRESS_DIRECTION,
                                   name="directions"),
              nullable=False,
              server_default=constants.EGRESS_DIRECTION),
    sa.PrimaryKeyConstraint('id'),
    sa.ForeignKeyConstraint(['qos_policy_id'], ['qos_policies.id'],
                            ondelete='CASCADE')
)

Add DB model QosPacketRateLimitRule:

class QosPacketRateLimitRule(model_base.HasId, model_base.BASEV2):
    __tablename__ = 'qos_packet_rate_limit_rules'
    qos_policy_id = sa.Column(sa.String(36),
                              sa.ForeignKey('qos_policies.id',
                                            ondelete='CASCADE'),
                              nullable=False)
    max_kpps = sa.Column(sa.Integer)
    max_burst_kpps = sa.Column(sa.Integer)
    revises_on_change = ('qos_policy',)
    qos_policy = sa.orm.relationship(QosPolicy, load_on_pending=True)
    direction = sa.Column(sa.Enum(constants.EGRESS_DIRECTION,
                                  constants.INGRESS_DIRECTION,
                                  name="directions"),
                          default=constants.EGRESS_DIRECTION,
                          server_default=constants.EGRESS_DIRECTION,
                          nullable=False)
    __table_args__ = (
        sa.UniqueConstraint(
            qos_policy_id, direction,
            name="qos_packet_rate_limit_rules0qos_policy_id0direction"),
        model_base.BASEV2.__table_args__
    )

With OVO object:

@base.NeutronObjectRegistry.register
class QosPacketRateLimitRule(QosRule):

    db_model = qos_db_model.QosPacketRateLimitRule

    fields = {
        'max_kpps': obj_fields.IntegerField(nullable=True),
        'max_burst_kpps': obj_fields.IntegerField(nullable=True),
        'direction': common_types.FlowDirectionEnumField(
            default=constants.EGRESS_DIRECTION)
    }

    duplicates_compare_fields = ['direction']

    rule_type = constants.RULE_TYPE_PACKET_RATE_LIMIT

REST API Impact

GET: List packet rate limit rules for QoS policy

/v2.0/qos/policies/{policy_id}/packet_rate_limit_rules

Response:

{
  "packet_rate_limit_rules": [
      {
          "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c793",
          "max_kpps": 10000,
          "max_burst_kpps": 0,
          "direction": "egress"
      }
  ]
}

POST: Create packet rate limit rule

/v2.0/qos/policies/{policy_id}/packet_rate_limit_rules

Request:

{
  "packet_rate_limit_rule": {
      "max_kpps": "10000"
  }
}

Response:

{
  "packet_rate_limit_rule": {
      "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c793",
      "max_kpps": 10000,
      "max_burst_kpps": 0,
      "direction": "egress"
  }
}

GET: Show packet rate limit rule details

/v2.0/qos/policies/{policy_id}/packet_rate_limit_rules/{rule_id}

Response:

{
  "packet_rate_limit_rule": {
      "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c793",
      "max_kpps": 10000,
      "max_burst_kpps": 0,
      "direction": "egress"
  }
}

PUT: Update packet rate limit rule

/v2.0/qos/policies/{policy_id}/packet_rate_limit_rules/{rule_id}

Request:

{
  "packet_rate_limit_rule": {
      "max_kpps": 10000
  }
}

Response:

{
  "packet_rate_limit_rule": {
      "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c794",
      "max_kpps": "10000"
  }
}

DELETE: Delete packet rate limit rule

/v2.0/qos/policies/{policy_id}/packet_rate_limit_rules/{rule_id}

And, neutron will allow attaching new PacketRateLimitRule to QoS policy.

The Neutron basic workflow

User creates QoS policy
Creates packet rate limit rules with multiple directions to this QoS policy
Attaching this QoS policy to a port
(No available) related L2 driver apply PPS limitation driver rule to the port
Attaching this QoS policy to a L3 IP (floating IP or gateway IP).
(No available) related L3 driver apply PPS limitation driver rule to the IP

Implementation

Assignee(s)

LIU Yulong <i@liuyulong.me>

Work Items

Adding API extension and DB models for neutron server.
Testing.
Documentation.

Dependencies

None

Testing

Unit test cases to verify the DB rules are created/updated/deleted.

References

Allow multiple external gateways on a router

Tue, 09 Mar 2021 00:00:00

RFE: https://bugs.launchpad.net/neutron/+bug/1905295

This spec proposes to allow a Neutron router to have multiple external gateways, so we can represent router designs where the router has multiple legs towards the outside world for reasons of higher capacity (typically load balanced between these legs) and higher availability (tolerating the failure of a router uplink or a router gateway port).

Problem Description

Below we describe one cloud design that required us to allow multiple external gateways on a single router. However the feature proposed in this spec is in no way specific to this design, this spec only strives to allow multiple external gateways for any design that needs it.

The Neutron router could be an active-active HA router. Compute hosts are connected to the Neutron router via an MLAG, the two sides of the router present a single IP to the computes. There is a single logical router one hop away from the Neutron router towards the external world (from here on: datacenter gateway) which is also an active-active HA router. The Neutron router and the datacenter gateway are interconnected by 2x2 point-to-point links (from each side to each side). The Neutron router in this setup has 4 links towards the external world.

+--+   +--+
|R3|   |R4| Datacenter gateway
+--+   +--+
 | \   / |
 |  \ /  |
 |   X   |  2x2 point-to-point links
 |  / \  |
 | /   \ |
+--+   +--+
|R1|   |R2| Neutron router
+--+   +--+
 | \   / |
 |  \ /  |
 |   X   |  MLAG
 |  / \
 | /
+--+
|C1|   ...  Compute hosts
+--+

Proposed Change

Overall Picture

The information in an external_gateway_info structure is used to create a gateway port. A gateway port is a router leg, where beyond packet forwarding, we perform SNAT (if enabled) and DNAT (for floating IPs and port forwardings) and which influences the default route of that router. Given multiple external_gateway_info structures we can straightforwardly create multiple gateway ports. We can also straightforwardly install the directly connected routes of the additional external gateways.

With only one external gateway - regarding packet forwarding - naturally only the internal-internal and internal-external directions exist. However with multiple external gateways the external-external direction also becomes possible. We don’t have a use case for external-external forwarding. However for the sake of simplicity of implementation this spec proposes to support that, that is to allow packet forwading from one external gateway to another. If required, a follow-up spec may propose finer control of external-external packet forwarding.

However the default route of a router cannot be multiplied without complex consequences (e.g. load balancing between multiple nexthops). So for the sake of simplicity for each router we retain one external gateway as special. The special one is used to set the default route of a router as before, while the other external gateways have no influence on the default route.

NATting and reservation of floating IPs can be performed for each external gateway as usual, but please see the ‘Out of Scope’ section.

The first implementation targets centralized routers on l3-agent.

For the special, first gateway port we keep adding the same routes as before.

The usual set of implicitly managed routes in a Neutron router are:

One directly connected route for each router interface’s each subnet.
One directly connected route for each gateway port’s each subnet.
One default route to the single gateway port’s subnet’s gateway_ip.

Keeping to this logic, when we add further gateway ports, we also add a directly connected route for each additional gateway port’s each subnet. Therefore the traffic destined to these subnets will be sent already via the respective gateway ports. Also traffic may reach the gateway ports from outside. However the majority of traffic will be sent via the default route.

Out of Scope

First of all, overall support for active-active HA routers by the l3-agent is not targeted here.

We intentionally only add a default route for the first, special gateway port(‘s subnet’s gateway_ip). We do not add default routes for the other gateway ports(’ subnet’s gateway_ip). Further management of routes (to make the additional external gateways actually used) is left to:

Managing additional routes via the extraroutes API.
Future improvements to neutron-dynamic-routing so a Neutron router can also receive (not just advertise) additional routes. [1]

Since the extraroutes API is already available, we believe that some use of the multiple external gateways can be realized as soon as this feature is merged.

The advertisement of any routes with the newly introduced additional external gateways as nexthops (for tenant networks or floating IPs) is also out of scope here. We believe that advertising such routes via routing protocols clearly makes sense, however:

This can be covered by a later spec in neutron-dynamic-routing and
basic use of the changes proposed here is possible without routing protocols.

Backwards Compatibility

We propose to store and expose the external gateways a bit redundantly to make backwards compatibility easier. Where possible (API, DB, RPC) keep the current scalar external gateway (or gateway port) attribute of a router as is. But also add a new router attribute for the plural: external gateways or gateway ports which contain a list of (not just the rest but) all such objects. So the object in the scalar attribute is present in this list too - preferably as the first element of the list.

This is just a high level approach that aims to leave all code unchanged where we only need to keep backwards compatible behavior.

DB Impact

The current DB schema contains the router - external gateway relations somewhat redundantly.

First in the routerports table there is no constraint on the number of ports with type network:router_gateway belonging to one router. Today we only store at most one network:router_gateway port for each router, but the DB schema allows more. [2]

Second the gw_port_id column of the routers table is a scalar. Today it stores the same port uuid as we have in the routerports table. [3]

We propose:

To keep the SQL schema as is, but start storing multiple network:router_gateway ports in the routerports table.
To also keep routers.gw_port_id scalar and there store the one special, backwards compatible external gateway (so this one is present both in the routerports table and in routers.gw_port_id).
To extend the neutron.db.models.l3.Router class with new attribute gw_ports that map to all relevant network:router_gateway ports stored in the routerports table.

REST API Impact

Introduce a new API extension called multiple-external-gateways.

This extension adds a new router attribute: external_gateways. Which is a list of external_gateway_info structures, for example:

[
  {"network_id": ...,
   "external_fixed_ips": [{"ip_address": ..., "subnet_id": ...}, ...],
   "enable_snat": ...},
  ...
]

The first element in the list is special:

It is always the same as the original external_gateway_info.
It is the one that sets the default route of the Neutron router.

The order of the the rest of the list is irrelevant and ignored. Duplicates in the list (that is multiple external gateways with the same network_id) are not allowed.

The external_gateways attribute cannot be set in POST /v2.0/routers or PUT /v2.0/routers/{router_id} requests, instead it can be managed via sub-methods:

PUT /v2.0/routers/{router_id}/add_external_gateways

Accepts a list of external_gateway_info structures. Adding an external gateway to a network that already has one raises an error.
PUT /v2.0/routers/{router_id}/update_external_gateways

Accepts a list of external_gateway_info structures. The external gateways to be updated are identified by the network_ids found in the PUT request. The external_fixed_ips and enable_snat fields can be updated. The network_id field cannot be updated.
PUT /v2.0/routers/{router_id}/remove_external_gateways

Accepts a list of potentially partial external_gateway_info structures. Only the network_id field from external_gateway_info structure is used. The external_fixed_ips and enable_snat keys can be present but their values are ignored.

The add/update/remove PUT sub-methods respond with the whole router object just as POST/PUT/GET /v2.0/routers.

RPC Impact

The sync_routers message already has a gw_port field. Extend the message to also include a gw_ports field containing all gateway ports. Bump the rpc version of this message.

Upgrade Impact

The sync_routers RPC message will have a new version.

Client Impact

Relevant changes in osc and openstacksdk.

Testing

Unit tests.
Fullstack tests for l3-agent.
Tempest tests in neutron-tempest-plugin.

Assignee(s)

Bence Romsics <bence.romsics@gmail.com>

References

[1] https://review.opendev.org/c/openstack/neutron-specs/+/783791 [2] https://opendev.org/openstack/neutron/src/commit/b7c4a11158786431c262cfcc2fc4bc46ab6bacd2/neutron/db/models/l3.py#L24 [3] https://opendev.org/openstack/neutron/src/commit/b7c4a11158786431c262cfcc2fc4bc46ab6bacd2/neutron/db/models/l3.py#L54

L3 router support ECMP

Thu, 21 Jan 2021 00:00:00

Blueprint: https://blueprints.launchpad.net/neutron/+spec/support-for-ecmp

Launchpad Bug: https://bugs.launchpad.net/neutron/+bug/1880532

ECMP is a kind of routing technology which allows traffic to reach the same destination via multiple different links. Neutron does not need to calculate the equivalent route path, but leave that part of the work to those applications using ECMP API. Neutron just receives those parameters and configures routers. Since we have “ip route” command provided by the iproute2 utility in Linux, Neutron can simply address ECMP by using pyroute2 and adding route entry into Neutron router namespace.

This feature is currently designed to support Octavia’s multi-active scheme, allowing LoadBalancer in Octavia to have multiple amphoras at the same time. By configuring the ECMP route in the router, multiple amphoras can have a virtual IP at the same time to serve a set of functions that require high concurrency support.

Note

Items marked with [P2] refer to lower priority features to be designed / implemented only after initial release.

[P2] Currently the equal cost route is a simple 5 tuple, that means if we have one <nexthop> unreachable and remove it from ECMP routes, all connections get redistributed. To avoid this, we intend to use a consistent hashing instead of the original scheme. This scheme which can support consistent hashing is based on hmark which was added in iptables-1.4.15 or later. See the history file of the iptables on [1].

Then this spec describes how to implement ECMP in Neutron.

Problem Description

Octavia has proposed an active-active load balancing design on [2].

Topology Description

                                                     Tenant Backend
                        +----------------+              Network
                        |                |                 +
Internet+-------------->+    router/gw   +----------------->
                        |                |    ECMP         |
                        +----------------+                 |
                                                           |
Management                                                 |
 Network                                                   |
    +                                                      |
    |                                                      |         +----------+
    |               +-----------------------+              |         |  Tenant  |
    |          +----+                  +---------+         <---------+Service(1)|
    |          |MGMT|  loadbalancer(1) | VIP|Back|         |         |          |
    <----------+ IP |                  |    | IP +--------->         +----------+
    |          +---------------------------------+         |
    |                                          |           |         +----------+
    |                                          |           |         |  Tenant  |
    |                                          |  ICMP     <---------+Service(2)|
    |                                          | DETECT    |         |          |
    |                                          |           |         +----------+
    |                                          |           |
    |               +-----------------------+  v           |         +----------+
    |          +----+                  +---------+         |         |  Tenant  |
    |          |MGMT|  loadbalancer(2) | VIP|Back|         <---------+service(3)|
    <----------+ IP |                  |    | IP +--------->         |          |
    |          +---------------------------------+         |         +----------+
    |                                          |           |
    |                                          |           |
    |         +-------------+                  |           |           ● ● ●
    |         |Octavia Lbaas|                  |           |
    <---------+ Controller  |   ● ● ●          |  ICMP     |
    |         +-------------+                  | DETECT    |         +----------+
    |                                          |           |         |  Tenant  |
    |                                          |           <---------+Service(M)|
    |                                          |           |         |          |
    |               +-----------------------+  v           |         +----------+
    |          +----+                  +---------+         |
    |          |MGMT|   loadbalancer(n)| VIP|Back|         |
    <----------+ IP |                  |    | IP +--------->
    |          +---------------------------------+         |
    +                                                      +

This program proposed such a scheme:

Multiple load balancing servers in a vip-subnet, sharing one virtual IP and one or more back end pools to response clients’ request, and each loadbalancer has its own IP address.
Clients send requests to VIP, then the router distributes every single request to a load balancing server which has the correct VIP configured on it.
Finally, the load balancing server distributes the request to a back end. The loadbalancers and tenant service vm can be in the same subnet or different networks.

In such a situation, Octavia needs the router to support ECMP for distributing requests. So Octavia can send a request to Neutron for creating an ECMP route, then Neutron L3 agent executes command in the Neutron router’s namespace to create an ECMP entry in it, using VIP as the destination IP of the route’s entry, and several load balancers’ IP as nexthop IP. So those requests having VIP as their destinations can be distributed to each loadbalancer.

The whole process implements two levels of load balancing, i.e. load balancing between multiple loadbalancers and load balancing between the backend real servers

[P2] Based on current public cloud operator implementations in production environments, tenants usually only see IPs in the same network, so considering the same broadcast domain, the router needs to enable proxy ARP on the corresponding interface.(Users need to disable the proxy ARP capability of vms in nexthops by themselves)

User Workflow

Generally, users can use the ECMP function for their own purposes. For putting an ECMP entry into the router namespace, user can set routes with same destination by using command:

openstack router add route \
--route destination=20.0.20.0/24,gateway=12.0.0.11 \
--route destination=20.0.20.0/24,gateway=12.0.0.12 router-ecmp

And withdraw the ECMP entry with:

openstack router add route \
--route destination=20.0.20.0/24,gateway=12.0.0.11 \
--route destination=20.0.20.0/24,gateway=12.0.0.12 router-ecmp

For more information about router related OSC, please read [3].

An integrated sequence diagram of the Octavia’s use case is here:

+------+      +--------+     +-------+   +--------+ +-------+ +------------+
|client|      |Octavia |     |Neutron|   |LB Node | |qrouter| |service pool|
+------+      +---+----+     +---+---+   +---+----+ +---+---+ +------+-----+
  |create LB      |              |           |          |            |
  +-------------> | create ecmp  |           |          |            |
  |service        +-------------->           |          |            |
  |               | LB server boot           |          |            |
  |               +--------------+---------->+          |            |
  |               |              | set ecmp route       |            |
  |               | ecmp done    +-----------+--------->+            |
  |               +<-------------|           |          |            |
  |               | LB server boot done      |          |            |
  |LB service done+<-------------+-----------+          |            |
  +<--------------+              |           |          |            |
  |               |              |           |          |            |
  |               |              |           |          |            |
  |sending request|              |           |          |            |
  +---------------------------------------------------->|            |
  |               |              |           |  pick a LB node       |
  |               |              |           +<---------|            |
  |               |              |           | pick a service node   |
  |               |              |           +---------------------->+
  |               |              |           |          |response    |
  |               |              |           +<----------------------+
  |               |  response    |           |          |            |
  +<-----------------------------------------+          |            |
  |               |              |           |          |            |
  |               |              |           |          |            |
  v               v              +           v          v            v

Suppose a user has a set of services that require a multi-active load-balancing scheme, so the user send a request to Octavia to create a loadbalancer, specifying topology as multi-active. And post a vip-subnet to Octavia to assign an IP or directly post a virtual port, which is defined by Octavia, and then users need to submit parameters such as pool, member, listener, etc., but the latter are irrelevant to Neutron, you can find them in Octavia document.

While Octavia is creating a loadbalancer, it will also send an update_router request or an add_extraroutes request to Neutron, post severval routes entries with same destination param, and load balancers’ IPs as nexthop param.

Neutron receives the request from Octavia, determines whether to add an ECMP route by calculating whether there are multiple routes with the same destination address, making sure the router will distribute those packets with vip as their destination.

Those ECMP routes will be removed when user drops the multi-active loadbalancer, and it could be modified when adding or removing a load balancing node.

Data flow

[P2] (If on a same network, use ARP proxy) A client requests mac address of the VIP and accesses the service based on this mac address. the router will use gateway MAC address to respond.
The client’s datagram will be transmitted to the router first.
The router gateway checks ECMP routing entries then forwards the client’s packets to the load balancers.
Load balancer accepts connections from clients, receives traffic, then distributes it to the back-end server pool.
The reply traffic from the back-end server pool go through load balancers and then comes to the router (directly comes back to intranet clients if on a same network), these packets are eventually forwarded back by the router.

Proposed Change

Overview

In Server Side

There are no changes that have to be made in server side.

In Agent Side

Modify the logic of processing router_update event in L3 agent to support adding ECMP routes in routers. The routes_updated function in RouterInfo will behave as below:

When more than one route is found to have the same destination, L3 agent should execute a pyroute2 code, which looks like

ip.route('replace', dst='<destination_ip>',multipath=[{"gateway":
"<nexthop1>"},{"gateway":"<nexthop2>"}])

Then there will be an ip route entry in the namespace, which looks like

<vip> proto static
    nexthop via <nexthop_ip1> dev qr-xxxxxxxx-nn weight 1
    nexthop via <nexthop_ip2> dev qr-xxxxxxxx-nn weight 1

Then router will randomly pick a <nexthop_ip> and fill its mac address into the package’s dst_mac address when it wants to get to the <destination_ip>.

[p2]For keeping connection while removing a load balancing node, use iptables instead of simply a ip route entry.

Use HMARK to mark flows in mangle table, the fwmark values determined by the source address.
Distribute flows to different tables by fwmark values.
There is a mapping between the fwmark values and the table values
For each table, give it a default nexthop ip.
Modify the mapping between fwmark values and table values when a nexthop is unreachable.

[p2]In order to let traffic from the same network to pass through the router, L3 agent will also let router to use Proxy ARP by setting command:

sysctl -w net.ipv4.conf.<NIC_1>.proxy_arp_pvlan=1

<NIC_1> is the name of the router interface to which the destination subnet is connected. For example, router R1 is connected to a subnet sub-1 whose cidr is 10.10.10.0/24, so there will be a virtual network interface device qr-abcdefgh in the router related namespace as the gateway for the subnet sub-1, then add an ECMP route with a destination like 10.10.10.5/32 which is in the scope of the subnet sub-1, at this point, the above command will be executed and <NIC_1> will be qr-abcdefgh.
For making the ARP proxy optional, add an config option in L3Agent.ini:
```
[ECMP]

router_interface_arp_proxy = True
```

Data Model Impact

None

REST API Impact

Following REST APIs wil be affected:

PUT /v2.0/routers/<router_id>/add_extraroutes

PUT /v2.0/routers/<router_id>/remove_extraroutes

PUT /v2.0/routers/<router_id>

The above three APIs are the current methods used to add/remove custom routes. See the usage of extraroutes on [4]. (The third API PUT /v2.0/routers/<router_id> is not recommended for adding routes)

Before the ECMP routing Implementation, when L3 agent receive several route entries with same destination and different nexthops, it will only keep one entry of them, or replace the existing route with a new one. But now after these changes, there will be an ECMP route in the router. So you can add an ECMP route entry like this:

PUT /v2.0/routers/{router_id}/add_extraroutes

{ "router":
  { "routes":
    [ { "destination": "192.168.1.6/32",
        "nexthop": "192.168.1.88" },
      { "destination": "192.168.1.6/32",
        "nexthop": "192.168.1.99" }
      ...
    ]
  }
}

Then you can find the ECMP route in router related namespace:

#ip route

192.168.1.6/32 proto static
  nexthop via 192.168.1.88 dev qr-9adb238b-c2 weight 1
  nexthop via 192.168.1.99 dev qr-9adb238b-c2 weight 1

To make this behavior change discoverable, a shim extension called ‘ecmp_routes’ will be added. [p2]To make ARP proxy behavior discoverable, a shim extension called ‘ecmp_arp’ will be added, it will be removed dynamically when related option router_interface_arp_proxy in config file is False.

Implementation

Assignee(s)

XiaoYu Zhu

Work Items

L3 Agent Update
Tests
Documentation

Testing

Tempest Tests

Tempest tests

Functional Tests

New tests need to be written

Documentation Impact

User Documentation

User documentation
API reference

Developer Documentation

Needs devref documentation

References

L3 router support NDP proxy

Wed, 20 Jan 2021 00:00:00

Launchpad Bug: https://bugs.launchpad.net/neutron/+bug/1877301

The IPv6’s NDP (Neighbor Discovery Protocol) is similar to IPv4’s ARP (Address Resolution Protocol) in functional, but the NDP works at L3. The NDP proxy [1] is also similar to ARP proxy [2]. But, the NDP proxy only works on IPv6, it can proxy specific IPv6 address’s NA (Neighbor Advertisement) that to response the NS (Neighbor Solicitation) which comes from the upstream router. The Linux kernel already implement NDP proxy. With these functionalities, we can implement some APIs so that users/tenants can advertise specific IPv6 addresses. It looks a little like IPv4’s floating IP, but this solution doesn’t do any NAT action.

Problem Description

As the IPv6 more and more popularize, we should provide a simple method to make the IPv6 VMs more easily and flexibly connect to external network.

In some use cases, such as a web site which has some DB services, MQ services and portal services etc, and they work on IPv6 VMs with same subnet. The administrator of the web site just would like to publish portal services to external. So the administrator needs some methods to just advertise a part of address to external.
The current solution of publishing IPv6 address which the Neutron support, such as BGP (Border Gateway Protocol) [3], PD (Prefix delegation) [4], is more complex, they require do some complex configurations in upstream router. This maybe not suit some company as their bare metal hosted in thrid-party IDC, they don’t have the privilege of the upstream router. So we should provide a solution with not depth cooperation of upstream router.

Proposed Change

Overview

Server Side

Implement a service plugin named ndp_proxy to support the feature of this spec description. Administrator can enable the feature by append the plugin to service_plugin in Neutron configuration file.
Implement an API extension, In the extension we should define a named ndp_proxy resource and its APIs.
Add a new parameter enable_ndp_proxy to router. If it is set to True, the ndp proxy will be enabled in router namespace.

Agent Side

Implement an extension of Neutron L3 agent to set relational rules for ndp proxy. The extension behavior like below:

If the router’s enable_ndp_proxy field is true, the router’s namespace should have a default ip6tables rule to drop all packets input from the router’s external gateway device. By this way, we can eliminate affection of extra route and neighbor in upstream router. So, if a router’s enable_ndp_proxy set as True, the Prefix Delegation [4] and BGP Dynamic Routing [3] solution will have no effect, we should explain this in user document.

Note

If a router’s enable_ndp_proxy set as false when the router have ndp proxies, these ndp proxes will have no effect, the Prefix Delegation [4] and BGP Dynamic Routing [3] solution will recover. In particular, the DVR router will ignore enable_ndp_proxy.
For each ndp proxy object, the extension should set a neighbor proxy entry at the router’s external gateway device and set an ip6tables rule to permit the relational packets through. By this, the router only permits those IPv6 addresses which enabled the ndp_proxy to communication with external network. This makes the ndp proxy’s behavior is more similar to IPv4’s floating ip.

Data Model Impact

The following new tables are added as part of the ndp proxy feature.

For ndp proxy resource:

CREATE TABLE ndp_proxies (
    id CHAR(36) NOT NULL PRI KEY,
    name VARCHAR(255),
    router_id VARCHAR(36) NOT NULL FOREIGN KEY,
    port_id VARCHAR(36) NOT NULL FOREIGN KEY,
    ip_address VARCHAR(64) NOT NULL,
    project_id VARCHAR(255),
    standard_attr_id bigint(20) NOT NULL,
);

The router_id column will store the id of router which the ndp proxy belong to. The port_id column will store the id of Neutron internal port which the ip_address locate in. The ip_address column will store the IPv6 address which we need to proxy.

For router’s enable_ndp_proxy parameter:

CREATE TABLE router_ndp_proxy_state (
    router_id VARCHAR(36) NOT NULL FOREIGN KEY,
    enable_ndp_proxy BOOL NOT NULL,
);

New Resource Extension

New resource extension ndp_proxy will be added. It will define the ndp_proxy entry’s CURD APIs.

For this new feature, a new service plugin will be introduced, and the following methods will be added:

‘create_ndp_proxy()’
‘delete_ndp_proxy()’
‘update_ndp_proxy()’
‘get_ndp_proxy()’
‘get_ndp_proxies()’

So the attributes map of new resource would be like:

RESOURCE_ATTRIBUTE_MAP = {
    'ndp_proxy': {
        'id': {'allow_post': False,
               'allow_put': False,
               'validate': {'type:uuid': None},
               'is_visible': True,
               'primary_key': True},
        'name': {'allow_post': True,
                 'allow_put': True,
                 'validate': {'type:string': 255},
                 'is_filter': True,
                 'is_sort_key': True,
                 'is_visible': True, 'default': ''},
        'project_id': {'allow_post': True,
                       'allow_put': False,
                       'required_by_policy': True,
                       'validate': {'type:uuid': None},
                       'is_visible': True},
        'router_id': {'allow_post': True,
                      'allow_put': False,
                      'validate': {'type:uuid': None},
                      'is_visible': True},
        'port_id': {'allow_post': True,
                    'allow_put': False,
                    'validate': {'type:uuid': None},
                    'is_visible': True},
        'ip_address': {'allow_post': True,
                       'allow_put': False,
                       'default': None,
                       'validate': {
                           'type:ip_address_or_none': None},
                       'is_visible': True}
        'description': {'allow_post': True,
                        'allow_put': True,
                        'default': '',
                        'validate': {'type:string': 1024},
                        'is_visible': True}
    }
}

Note

The ip_address parameter is optional, if not set it when user send post request, the new service plugin will select a IPv6 address from the port_id represented port.

REST API Impact

Extend router API

The idea is to extend the router Rest API with a new extension router_ndp_proxy with the below defined attribute.

Router extension
Attribute Name	Type	CRUD	Default Value	Description
enable_ndp_proxy	Boolean	CRU	False	Whether the router enable ndp proxy function.

The router extension definition would be expanded as :

RESOURCE_ATTRIBUTE_MAP = {
     'routers': {
         'enable_ndp_proxy': {
             'allow_post': True, 'allow_put': True,
             'convert_to': converters.convert_to_boolean_if_not_none,
             'is_visible': True,
             'is_filter': True,
         }
     }
}

Default, only admin user can update enable_ndp_proxy parameter. And, a new config option enable_ndp_proxy_by_default will be introduced. If it set as True, the enable_ndp_proxy will be set as True default.

For example, GET a router:

GET /v2.0/routers/<router-uuid>

{
    "router": {
        "admin_state_up": true,
        "availability_zone_hints": [],
        "availability_zones": [
            "nova"
        ],
        "created_at": "2018-03-19T19:17:04Z",
        "description": "",
        "distributed": false,
        "enable_ndp_proxy": false,
        "external_gateway_info": {
            "enable_snat": true,
            "external_fixed_ips": [
                {
                    "ip_address": "172.24.4.6",
                    "subnet_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf"
                },
                {
                    "ip_address": "2001:db8::9",
                    "subnet_id": "0c56df5d-ace5-46c8-8f4c-45fa4e334d18"
                }
            ],
            "network_id": "ae34051f-aa6c-4c75-abf5-50dc9ac99ef3"
        },
        "flavor_id": "f7b14d9a-b0dc-4fbe-bb14-a0f4970a69e0",
        "ha": false,
        "id": "f8a44de0-fc8e-45df-93c7-f79bf3b01c95",
        "name": "router1",
        "revision_number": 1,
        "routes": [
            {
                "destination": "179.24.1.0/24",
                "nexthop": "172.24.3.99"
            }
        ],
        "status": "ACTIVE",
        "updated_at": "2018-03-19T19:17:22Z",
        "project_id": "0bd18306d801447bb457a46252d82d13",
        "tenant_id": "0bd18306d801447bb457a46252d82d13",
        "service_type_id": null,
        "tags": ["tag1,tag2"],
        "conntrack_helpers": []
    }
}

Set a router’s enable_ndp_proxy parameter to True:

Post /v2.0/routers/<router-uuid>

Request body:

{
    "router": {
        "enable_ndp_proxy": true
    }
}

For the new resource ndp_proxy, some new URLs will be introduced:

List NDP Proxies

GET /v2.0/ndp_proxies

The response body:

{
    "ndp_proxies": [
         {
             "project_id": "ad239fa5-ceb7-8b81-abf5-b930d7f6ceb7",
             "id": "ae34051f-aa6c-4c75-abf5-50dc9ac99ef3",
             "name": "test01",
             "port_id": "fa450s1f-aa6c-4c75-abf5-50dc9ac9df67",
             "ip_address": "2001:217::19",
             "created_at":"2020-05-21T11:33:21Z",
             "updated_at":"2020-06-18T08:31:48Z",
             "description": ""
         },
         {
             "project_id": "ad239fa5-ceb7-8b81-abf5-b930d7f6ceb7",
             "id": "915a14a6-867b-4af7-83d1-70efceb146f9",
             "name": "test02",
             "port_id": "4323401f-aa6c-4c75-abf5-50dc9ac99ef3",
             "ip_address": "2001:218::12"
             "created_at":"2020-05-21T11:33:21Z",
             "updated_at":"2020-06-18T08:31:48Z",
             "description": ""
         }
    ]
}

Show NDP Proxy

GET /v2.0/ndp_proxies/<ndp-proxy-id>

The response body:

{
    "ndp_proxy": {
         "project_id": "ad239fa5-ceb7-8b81-abf5-b930d7f6ceb7",
         "id": "ae34051f-aa6c-4c75-abf5-50dc9ac99ef3",
         "name": "test01",
         "port_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf"
         "ip_address": "2001:217::19",
         "created_at":"2020-05-21T11:33:21Z",
         "updated_at":"2020-06-18T08:31:48Z",
         "description": "Some descriptions"
     }
}

Create NDP Proxy

POST /v2.0/ndp_proxies

The request body:

{
    "ndp_proxy": {
         "name": "test01",
         "router_id": "5823deb7-8b81-ceb7-40a0-b930d7f6ceb7",
         "port_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf",
         "ip_address": "2001:217::19",
         "description": "Some descriptions"
     }
}

There are some constraints here:

The router’s enable_ndp_proxy parameter must be set as True.
The subnet that the ip_address allocated from must be added to the router.
The network of the port_id represented port belong to must have same ipv6_address_scope [5] with the network of router’s external gateway.

The response body:

{
    "ndp_proxy": {
         "id": "ad239fv5-aa6c-4c75-abf5-50dc9ac99ef3",
         "name": "test01",
         "project_id": "ad239fa5-ceb7-8b81-abf5-b930d7f6ceb7",
         "router_id": "5823deb7-8b81-ceb7-40a0-b930d7f6ceb7",
         "port_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf",
         "ip_address": "2001:217::19",
         "created_at":"2020-05-21T11:33:21Z",
         "updated_at":"2020-06-18T08:31:48Z",
         "description": "Some descriptions"
     }
}

Update NDP Proxy

PUT /v2.0/ndp_proxies/<ndp-proxy-id>

The request body:

{
    "ndp_proxy": {
         "name": "test02",
         "description": "New descriptions"
     }
}

The response body:

{
    "ndp_proxy": {
         "id": "ad239fv5-aa6c-4c75-abf5-50dc9ac99ef3",
         "name": "test02",
         "project_id": "ad239fa5-ceb7-8b81-abf5-b930d7f6ceb7",
         "router_id": "5823deb7-8b81-ceb7-40a0-b930d7f6ceb7",
         "port_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf"
         "ip_address": "2001:217::56",
         "created_at":"2020-05-21T11:33:21Z",
         "updated_at":"2020-06-18T08:31:48Z",
         "description": "New descriptions"
     }
}

Delete NDP proxy

DELETE /v2.0/ndp_proxies/<ndp-proxy-id>

This operation does not accept a request body and does not return a response body.

Addtionally, if a router or port will be deleted, the ndp proxies which relational with them will be delete cascade.

Effects on Existing Router APIs

Before remove the subnet from a router, neutron should check whether has any ndp proxy related to the subnet (whether the ndp proxy’s ip_address was allocated from the subnet). If has any ndp proxy related to the subnet, the subnet can’t be removed.

L3 Agent Impact

Neutron needs to implement a new l3 agent extension to cooperate with the server API to set relational rules (neigh proxy and ip6tables, distributed router needs to set some extra route rules) in router’s namespace.

We assume user has a below scenario, then respectively describe the implementions of the feature about DVR router and Legacy router:

A subnet of which cidr is 2001::1:0/112
A VM belong to the subnet and it’s IPv6 address is 2001::1:8
A distributed/legacy router which set external gateway and connect to the subnet.

Legacy router Impact

Assume the router’s external gateway device is qg-733bd76b-62:

If the router’s parameter enable_ndp_proxy is true, the extension need to set the kernel parameter proxy_ndp as 1 in the router’s qrouter-namespace, create a custom chain named neutron-l3-agent-NDP and set a default iptables rule in it to drop all packets input from the router’s external gateway device. The executed commands like below:
```
sysctl -w net.ipv6.conf.qg-733bd76b-62.proxy_ndp=1
ip6tables -N neutron-l3-agent-NDP
ip6tables -A neutron-l3-agent-NDP -i qg-733bd76b-62 -j DROP
```

Note

If the router have no external gateway, the parameter enable_ndp_proxy will be ignored.

When add the IPv6 subnet to the router (the router’s enable_ndp_proxy already set as true), Before user advertise the subnet’s address with ndp proxy, the subnet should drop all external traffic. So, the following cmd should be executed:
```
ip6tables -I neutron-l3-agent-FORWARD -i qg-733bd76b-62 --destination 2001::1:0/112 -j neutron-l3-agent-NDP
```
By this, we can eliminate the effect of extra route and neighbor entries in upstream router.
For each ndp proxies, the extension should add a neigh proxy entry to the router external gateway device, and add ip6tables rule to permit the relational packets pass. The executed commands like below:
```
ip -6 neigh add proxy 2001::1:8 dev qg-733bd76b-62
ip6tables -I neutron-l3-agent-NDP -i qg-733bd76b-62 --destination 2001::1:8 -j ACCEPT
```
When remove a ndp proxy, the extension should remove related neigh proxy entry from the router external gateway device, and remove the related ip6tables rule to re-forbid relational packets pass. The executed commands like below:
```
ip -6 neigh del proxy 2001::1:8 dev qg-733bd76b-62
ip6tables -D neutron-l3-agent-NDP -i qg-733bd76b-62 --destination 2001::1:8
```
When router’s parameter enable_ndp_proxy set to false, the extension needs to set the kernel parameter proxy_ndp as 0 in the router’s qrouter-namespace namespace, delete the custom chain named neutron-l3-agent-NDP. The executed commands like below:
```
sysctl -w net.ipv6.conf.qg-733bd76b-62.proxy_ndp=0
ip6tables -X neutron-l3-agent-NDP
```

HA router Impact

The implementation of the feature for HA router is same as Legacy router, except failover. When HA router’s state has changed, the extension should refresh the ndp proxy rules, because the ndp proxy rules may be lost after multible failover.

Note

When HA router’s state has changed, the keepalived will sends unsolicited neighbor advertisement automatically. So, we don’t need to write extra code to do this. During failover, the traffic will be breaked, but it will recover soon if the failover accomplished.

DVR Router Impact

For DVR [6] router, it’s a little different from legacy and HA router. We need to consider two scenes: If the neutron-l3-agent in compute node set as dvr mode, the neutron-l3-agent will create a fip-namespace to process north-south traffic, so we need to apply related rules in qrouter-namespace and fip-namespace. If the neutron-l3-agent in compute node set as dvr_no_external mode, the north-south traffic will be processed by snat-namespace in network node, so we need to apply related rules in qrouter-namespace and snat-namespace.

For dvr mode

Assume the fip-namespace’s fg-dev port is fg-84920cf6-5e; the port connect fip-namespace to qrouter-namespace is fpr-ea902fe0-9 and it’s IPv6 address is fe80::8493:5bff:fe9b:8d93; the port connect qrouter-namespace to fip-namespace is rfp-ea902fe0-9 and it’s IPv6 address is fe80::a0a7:c5ff:fe2c:bade. The topology like below:

      +---------------+
      |               |
      |upstream router|
      |               |
      +-------+-------+
              |
+----------------------------+
|             | compute node |
|             |              |
|     +-------+--------+     |
|     | fg-84920cf6-5e |     |
|     |                |     |
|     |  fip-namespace |     |
|     |                |     |
|     | fpr-ea902fe0-9 |     |
|     +--------+-------+     |
|              |             |
|              |             |
|     +--------+--------+    |
|     |  rfp-ea902fe0-9 |    |
|     |                 |    |
|     |qrouter-namespace|    |
|     |                 |    |
|     +-----------------+    |
|                            |
+----------------------------+

Note

We don’t need to set ip6tables rules for dvr router. Because just by adding or removing the related route in fip-namespace (as description below) can be the switch to enable/disable the IPv6 traffic.

Due to the current Neutron don’t support dvr with IPv6, the qrouter-namespace has no default route about IPv6. We should add the default route firstly if the router set external gateway, the default route’s next-hop shoule be the fpr-dev device’s IPv6 address, The executed command like this (Executed in qrouter-namespace):
```
ip route add default via fe80::8493:5bff:fe9b:8d93 dev rfp-ea902fe0-9
```
Because of the device which directly connects to upstream router is located in fip-namespace, the proxy entry should be set in fip-namespace. So, the below cmd should be executed in all fip-namespace:
```
sysctl -w net.ipv6.conf.fg-84920cf6-5e.proxy_ndp=1
```
For each ndp proxies the extension add a proxy entry to the fg-dev in fip-namespace, the proxy entry just add to one namespace which hosted in the node of the ndp proxy object’s port belong to, and add a route in the namespace so that the packets of which destination is the ndp proxy’s ip_address can be forwarded to qrouter-namespace. This cmds like below (Executed in fip-namespace):
```
ip -6 neigh add proxy 2001::1:8 dev fg-84920cf6-5e
ip route add 2001::1:8 via fe80::a0a7:c5ff:fe2c:bade dev fpr-ea902fe0-9
```
When remove a ndp proxy, the extension should remove related neigh proxy entry from the fg-dev, and remove the related route, This cmds like below (Executed in fip-namespace):
```
ip -6 neigh del proxy 2001::1:8 dev fg-84920cf6-5e
ip route del 2001::1:8 via fe80::a0a7:c5ff:fe2c:bade dev fpr-ea902fe0-9
```
Because setting of ip6tables rules is not required, so for the change of enable_ndp_proxy, the agent extension needn’t do any thing.
If an instance was migrated and it’s port was related to a ndp proxy entry. The extension should delete related rules in old host and create them in new host. Additionally, the extension should send a NA (Neighbour Advertisement) to fresh the upsteam router’s neighbor entry so that the external traffic can forward to new host’s fip-namespace immediately.

For dvr_no_external mode

Assume the snat-namespace’s qg-dev is qg-87059c6c-a9, the sg-dev is sg-68bcdb7b-a2; the qrouter-namespace’s qr-dev is qr-50457f9b-98. The topology like below:

    +---------------+
    |               |
    |upstream router|
    |               |
    +-------+-------+
            |
+-----------------------+
|network    |           |
| node      |           |
|   +-------+------+    |
|   |qg-87059c6c-a9|    |
|   |              |    |
|   |snat-namespace|    |
|   |              |    |
|   |sg-68bcdb7b-a2|    |
|   +-------+------+    |
|           |           |
+-----------------------+
            |
+-----------------------+
|compute    |           |
| node      |           |
|  +--------+--------+  |
|  | qr-50457f9b-98  |  |
|  |                 |  |
|  |qrouter-namespace|  |
|  |                 |  |
|  +-----------------+  |
|                       |
+-----------------------+

For this mode, in qrouter-namespace we just need to add a default route, so that the north-south traffic can be redirected to snat-namespace (The current neutron code already completed this demand). For snat-namespace we just treat is as legacy router’s qrouter-namepsace, about it’s rules process we can refer to Legacy router Impact.

OVN backend impact

The ndp_proxy for OVN L3 backend is not covered by this proposal. If it was proved to be feasible, we should implement the feature base on OVN backend in the future. But for now, we will just implement this base on Neutron L3 agent.

Implementation

Assignee(s)

yangjianfeng

Work Items

API Implementation
DB Implementation
Reference implementation
Tests
Documentation

Testing

Tempest Tests

The functions that the spec proposed need the external hardware router’s support (need to add a direct route entry at upstream router). This is difficult for tempest to do this. So, we can skip the scenario tests firstly.

Functional Tests

Need to add functional tests

API Tests

Need to add API tests

Fullstack Tests

Need to add Fullstack tests.

Documentation Impact

User Documentation

Needs user documentation

Developer Documentation

Needs devref documentation

API reference

Needs API reference documentation

References

Distributed DHCP for Openvswitch Agent

Mon, 28 Dec 2020 00:00:00

RFE: https://bugs.launchpad.net/neutron/+bug/1900934

Neutron DHCP agents and the scheduled network instances are relatively simple in function. But the configuration is complex, and it depends on external process (dnsmasq) and namespace. When the user’s demand is merely unique, for example, they only need the DHCP response during the virtual machine booting process, then the existing DHCP agent and its configuration procedure for network and port make things complicated.

This spec describes how to implement a DHCP extension for Neutron openvswitch agent to achive a simple and efficient solution for virtual machine DHCP function by leveraging the openflow with openvswitch.

Problem Description

Response the DHCP request is the main function for the scheduled network instance of DHCP agent. Except this, it has other functions, like isolated metadata and DNS lookup. But there are alternatives for these extended functions, such as config drive for metadata and Designate for DNS.

Then, the use frequency of the DHCP agent and its scheduled instance are relatively low. It will be used only during the VM booting if no DNS lookup. If you use config drive, aslo the scheduled DHCP instance is not useful.

And we have more problems for large scale clusters:

The scheduled network instances of DHCP port increase the consuming of L2 agent’s capacity and performance.
The DHCP provisioning block sometimes causes virtual machine booting failure.
Full sync in unknown reason causes the message queue, neutron-server and DB in high load.
DVR local router creation for the scheduled DHCP port is default behavior, which increased resource load for L2 and L3 agents implicitly [1].
Down a DHCP node causes long time recovery due to it has tons of scheduled network instance.

And it is hard to find the balance between how many DHCP agents should the deployment have and how many resources could one agent handle. Too few DHCP agent will finally make each one agent has a huge mount of resources. Too many DHCP agents will increase maintenance pressure for the operators, and make the centralized components overloaded.

There is a way to schedule one network to all DHCP agents on all compute nodes. Firstly, this may work for tiny deployment with extremely few resources. For large-scale deployment, it is basically impossible. Because there will be tens of thousands of Networks in Neutron, this will directly lead to a surge of resource pressure on each node, consume too many IP in user network, the results basically are unable to operate, virtual machine startup failure and unable to obtain IP and so on.

Proposed Change

A new extension of Neutron openvswitch agent will be added to achieve the Distributed DHCP.

Note

Solution Proposed

As we know Neutron openvswitch agent has the entire information of the ports which are pluged to the ovs bridge (If the port information was not synchronized by the ovs-agent, there is a simple cache pull mechanism which will fill the information). Neutron has the all conditions to support distributed DHCP natively:

ovs-agent is based on the python SDN controller ryu/os-ken
ovs-agent is fully distributed
ovs-agent has the entire resource information

So we can assume the Neutron openvswitch agent is a local SDN controller which will try to response the VM’s DHCP request. The basical data pipeline can be described as this:

                        +---------+                +---------------------+
+-------+ DHCP Request  |         |    packet-in   |  +----------------+ |
|  VM   +--------------->  Flows  +---------------->  |   os-ken app   | |
|       <---------------+         <----------------+  |                | |
+-------+ DHCP Response |         |    packet-out  |  | DHCP Responder | |
                        |         |                |  +----------------+ |
                        | br-int  |                |      OVS-agent      |
                        +---------+                +---------------------+

After this we will have:

Higher level availability than DHCP agent with it’s schedlued dnsmasq, DHCP requests are directly processed in the computing nodes, it is completely distributed.
No DHCP agent and its scheduling mechanism anymore
No extra external process for DHCP anymore
The (Neutron openvswith) agent downtime will no longer affect the address acquisition and virtual machine startup in other nodes.
Virtual machine startup will no longer be affected by port’s DHCP configuration, which reduces the probability of VM spawning failure.
DHCP request and reponse for VM will achieve a high success rate.

DHCP(v4/v6) protocol options

We will not repeat the DHCPv4 [2] and DHCPv6 [3] in details. But for this spec, we will ensure the following features of the related protocol:

DHCPv4 all message types
DHCPv4 host Configurations
DHCPv6 types Solicit, Advertise, Confirm, Renew, Rebind, Release and Reply
DHCPv6 Options

For Neutron, we have some Port attributes like dns_domain, dns_name and extra_dhcp_opts, and Subnet dns_nameservers, host_routes and gateway_ip and so on, all these options will be added to the final DHCP response like dnsmasq do.

Server side changes

Based on the new added config options, some DHCP related DB options, APIs, notifications and RPCs will be changed to no operation or be skipped. The config options only controls Neutron itself, the DHCP protocol will have no effect on. The final goal is to support the full features of DHCP protocol defined. The changes are:

disable DHCP scheduling mechanism and its failover forever
disable DHCP provisioning block
disable DHCP related RPCs and notificatons

OpenvSwitch Agent side changes

For Neutron openvswitch agent, we will add a new agent extension which will process the basical flow installation for each port’s further DHCP request.

There will be two basic flows which will direct DHCPv4 and DHCPv6 to independent tables. table 77 is for DHCPv4, table 78 is for DHCPv6. The flows are:

table=60, priority=101,udp,nw_dst=255.255.255.255,tp_src=68,tp_dst=67 actions=resubmit(,77)
table=60, priority=101,udp6,ipv6_dst=ff02::1:2,tp_src=546,tp_dst=547 actions=resubmit(,78)

For table 77, each DHCP request will be checked to verify the source mac and in_port in order to avoid the DHCP spoofing. If the DHCP request is matched, then submit it to the controller, aka the Neutron openvswitch agent. Any unmatched packets will be dropped. One example for a VM’s port is:

table=77, priority=100,udp,in_port="tapcc4f2da4-c5",dl_src=fa:16:3e:46:58:fe,tp_src=68,tp_dst=67 actions=CONTROLLER:0
table=77, priority=0 actions=drop

For table 78, DHCPv6 match and drop flows structure are basically same to DHCPv4:

table=78, priority=100,udp6,in_port="tapcc4f2da4-c5",dl_src=fa:16:3e:46:58:fe,tp_src=546,tp_dst=547 actions=CONTROLLER:0
table=78, priority=0 actions=drop

For the new extension of openvswitch agent, it will add a local packet_in_handler which will do the following works:

Listen on the EventOFPPacketIn event
Verify each packet to be DHCPv4 or DHCPv6
According to the openflow inport number to retrieve the port’s information
Assemble the DHCP(v4/v6) response and packet_out to in_port.

The response DHCP packet structure will be:

+------------------------------------------+
|           *Source Mac Address            |
|The gateway Port's MAC or A fake fixed MAC|
+------------------------------------------+
|       *Destination Mac Address           |
|        Neutron Port Mac Address          |
+------------------------------------------+
|           *Source IP Address             |
|      Gateway IP address from Subnet      |
+------------------------------------------+
|         *Destination IP address          |
|             Neutron Port IP              |
+------------------------------------------+

Source Mac Address will be the internal subnet gateway port’s Mac address. But actually this is not necessary for the DHCP protocol, we can use a fake fixed mac address to avoid some DB/RPC query.
Destination Mac Address will be the port’s MAC.
Source IP Address will be the internal subnet’s gateway IP.
Destination IP address will be the port’s first IP(v4/v6) address, the secondary IPs will be ignored.

Potential configurations

Config option disable_traditional_dhcp for neutron server side will be added which is aiming to control:

to disable DHCP scheduling for networks
to disable DHCP provisioning block
to disable DHCP RPC/notification
to disable all DHCP related API/attibutes network, subnet and port.

A new extension alias name dhcp will be added for neutron openvswitch agent:

[agent]
extensions = ...,dhcp

Config section [dhcp] will be added for neutron openvswitch agent and register some common options to determine DHCP protocol related parameters, the final [dhcp] section for openvswitch agent will be:

dhcp_opts = [
  cfg.BoolOpt('enable_dhcp_ipv6', default=False,
              help=_("Whether enable DHCP for IPv6")),
  cfg.IntOpt('dhcp_renewal_time', default=0,
             help=_("DHCP renewal time T1 (in seconds). If set to 0, it "
                    "will default to half of the lease time.")),
  cfg.IntOpt('dhcp_rebinding_time', default=0,
             help=_("DHCP rebinding time T2 (in seconds). If set to 0, it "
                    "will default to 7/8 of the lease time.")),
]

The Neutron basic workflow

User creates a VM in a network
Nova plug the VM’s NIC port to ovs-bridge
Ovs-agent process the port and install the DHCP related flows
L2 provisioning block released (No DHCP provisioning block)
VM booting and send DHCP request out
Match the flows and packet_in to ovs-agent
Ovs-agent directly send DHCP(v4/v6) response to VM’s port
VM booting success

Data Model Impact

None

REST API Impact

With the new config options, the following APIs will be disabled [4]:

add_network_to_dhcp_agent
remove_network_from_dhcp_agent
list_networks_on_dhcp_agent
list_dhcp_agents_hosting_network

For the option enable_dhcp of Subnet, this agent extension will set the flows based on that. If it is False, ports under this subnet will have no flows installed in table 77 and 78. The DHCP request will hit the final DROP action.

Upgrading

For native ml2/ovs deployment, this feature will be easily to upgrade to enforce. A simple way is to run all agents as they are. But disable the DHCP provisioning block. After enable the dhcp extension for ovs-agent, the DHCP request will be handled by it earlier than dnsmasq.

If you need a pure deployment without DHCP agents, the following is an overview about how to migrate to use this new feature:

Upgrading the Neutron code and restart neutron-server processes.
Setup the ovs-agent with dhcp extension.
Disable all DHCP agents to make sure no more scheduled network are created.

openstack network agent set --disable <dhcp_agent_id>

Note

This action will remove all scheduled network instances from the admin state DOWN DHCP agent.

Set the disable_traditional_dhcp = True option for neutron-server to disable the scheduling related API/RPCs.
(optional, just in case) Remove all scheduled network from all DHCP agents, this step is to pure all DHCP namespace and DHCP woker process dnsmasq.
After no more scheduled network, stop all DHCP agents and remove it from DB.

Note

This feature does not support DNS lookup. If your running deployments are using the DNS lookup function from dnsmasq, consider use designate as an alternative.

Implementation

Assignee(s)

LIU Yulong <i@liuyulong.me>

Work Items

Config options for neutron server to control DHCP related codes.
Create agent extension.
Testing.
Documentation.

Dependencies

None

Testing

Functionality

We will add fullstack test case to verify this new agent extension:

Create two fake fullstack VMs in two test namespaces
Use DHCP(v4 and v6) to config the fake VM ports
Ping (-4/6) from one fake VM to another

References

VPNaaS for OVN Networking

Wed, 16 Dec 2020 00:00:00

This specification covers the support for VPNaaS with OVN networking by adding a stand-alone VPN agent. This new agent will create a namespace on the node, connect it with the OVN distributed logical router and run the Swan process in the namespace.

Problem Description

The existing VPNaaS service plugin only supports the reference Neutron software routers, such as neutron L3 router using the L3 agent. This approach does not work if there is no L3 agent. For OVN a different solution is required.

Proposed Change

Add a new stand-alone VPN agent to support OVN+VPN. Add OVN-specific service and device drivers that support this new VPN agent. This will have no impact on the existing VPN solution, the existing L3 agent and its VPN extension will still work.

Changes on neutron server

VPN service driver and Agent scheduler

The VPN service driver has different implementations for different VPNaaS solutions. The existing implementation relies on the Neutron L3 router scheduler to decide where the VPN service process is hosted and the VPN device driver is part of the L3 agent. In OVN the L3 router scheduler and L3 agents are not used anymore, so a replacement is required.

For the newly-introduced agent type “VPN agent” we also add a corresponding VPN agent scheduler. Using the OVN-related scheduler for the router gateway port was ruled out because the VPN agent scheduler should also keep track whether the VPN agent(s) are still alive.

The new VPN agent scheduler will work on a per-router basis to have all IPSec site connections of a router processed by the same agent.

VPN gateway and transit network

In the existing implementation for ML2/OVS the VPNaaS shares the router gateway IP address with router SNAT, but for OVN the gateway public IP address (SNAT) can’t be shared with the VPN because SNAT is not in the namespace context. A new public IP address is needed for the VPNaaS namespace, so a VPN gateway port is created in the external network of the router. This address will be visible as the external_v4_ip/external_v6_ip of the VPN service.

To connect the router with the namespace a “Transit network” (169.254.0.0/30) is created and attached to the router. It is used to route traffic between virtual machine network(s) and the VPN process namespace. The respective routes added to the router are (per peer CIDR):

destination CIDR: peer CIDR
next hop: IP address of a port in the transit network (169.254.0.2)

The new ports and the transit network and subnet are managed using the API of the Neutron core plugin and OVN L3 router service plugin, not “under the hood” with OVN functions, to avoid inconsistencies between the Neutron database and OVN Northbound. This makes sure that scripts that sync Neutron DB and OVN Northbound (like neutron-ovn-db-sync-util) don’t delete lswitches that were created for the VPNaaS transit network but have no corresponding Neutron DB entries.

The IDs of the newly added ports, transit network and subnet are stored in a new database table “vpn_ext_gws” (for VPN external gateways). The gateway port and transit network are created automatically when the first VPN service of a router is created and are deleted after the last VPN service of the router is removed.

Naming of Neutron objects

External gateway port for VPN: “vpn-gw-{router_id}” (with device owner “network:vpn_router_gateway”, device ID is the router ID)
VPN transit network: “vpn-transit-network-{router_id}”
VPN transit subnet: “vpn-transit-subnet-{router_id}”
Port in the VPN transit network, to be plugged by the agent in the VPN namespace: “vpn-ns-{router_id}” (with device owner “network:vpn_namespace”, device ID is the transit subnet ID)

Changes on VPN agent

For OVN the VPN functionality is implemented in a new dedicated VPN agent using new OVN specific device drivers.

Configuration synchronization

The existing VPNaaS implementation uses rabbitmq RPCs to synchronize the VPN configuration between the database and the IKE daemon (strongswan etc) controlled by the agent.

Ideally a VPNaaS solution for OVN would avoid rabbitmq. But currently there is no support for configuring VPNs in OVN (via north/southbound databases). While it is possible to configure IPSec for tunnel ports, it’s not possible to set up and configure IPSec site connections to external peers.

To still be able to offer VPNaaS some other means than configuration via OVN north/southbound tables is necessary. Possible solutions could be:

Use rabbitmq RPCs as in the existing VPNaaS plugin.
Instead of rabbitmq, implement REST APIs on both ends (neutron server and VPN agent). This includes API endpoints that let the agent know about configuration changes (server -> agent), that allow agents to fetch the current configuration (agent -> server) and that allow agents to report state changes (agent -> server)

The least complex implementation approach is (1) to rely on existing code even if it is still using rabbitmq instead of rewriting the server/agent communication with new REST APIs.

VPN namespace management

The agent creates a VPN namespace and plugs two ports:

the external gateway port
a port in the transit network bound to 169.254.0.2

There will be one VPN namespace per router and it’s scheduled on exactly one VPN agent. The VPN agent shall make sure that only those VPN namespaces exist on the node which are scheduled there. I.e. when the agent starts it will synchronize with the controller and potentially delete superfluous VPN namespaces and/or create missing ones.

VPN namespaces are created by the VPN agent / device driver when the first IPSec site connection of a router is created and are deleted if all IPSec site connections of the router were deleted.

Routes are configured in the namespace to route incoming traffic to the local subnets (destination: local CIDR, next hop: router port of the transit network, i.e. 169.254.0.1). These routes are updated whenever the IPSec site connection configuration changes.

The agent will

create a namespace called “qvpn-{router_id}”
create two interfaces in the namespace:
- vgxxxxxxxx-xxx, where xxxxxxxx-xxx is the prefix of the port UUID of the VPN external gateway port (IP address in the external network)
- vrxxxxxxxx-xxx, where xxxxxxxx-xxx is the prefix of the port UUID of the VPN transit network port (169.254.0.2)
plug the two interfaces
add routes

Liveness of the VPN agent

There are two ways how liveness checks could be implemented:

Traditional way using rabbitmq. The VPN agent periodically reports its state (“report_state” RPC). The server side plugin utilizes the AgentSchedulerDbMixin functionality to keep track of alive agents and will potentially reschedule VPN services if an agent is down.
Liveness check using OVN similar to OVN Metadata agent. The server sets a new value of the “neutron:liveness_check_at” external_id in NB_Global, the agent monitors SB_Global and will set an external_id in its chassis row (external_id “neutron:ovn-vpnagent-sb-cfg”)

The second approach is more in line with the management of agents in the Neutron OVN mech driver.

VPN plugin configuration for OVN

The OVN VPN plugin is an extension of the existing one in order to add the VPN agent scheduler and status checks.

The service plugin to be added to neutron.conf is neutron_vpnaas.services.vpn.ovn_plugin.VPNOVNDriverPlugin

For OVN there is a new dedicated service driver, so the service_provider setting in neutron_vpnaas.conf will be: VPN:openswan:neutron_vpnaas.services.vpn.service_drivers.ovn_ipsec.IPsecOvnVPNDriver:default

There is a new agent type “VPN agent”

The agent binary is neutron-vpn-agent
The agent configuration file is /etc/neutron/vpn_agent.ini
The agent uses the VPN device driver configured in vpnagent/vpn_device_driver. An example device driver entry is vpn_device_driver = neutron_vpnaas.services.vpn.device_drivers.ovn_ipsec.OvnStrongSwanDriver

Database impact

Two new tables are added to the neutron database:

vpn_ext_gws: keeps the IDs of the additional network items needed for VPN services of a router (gateway port, transit network, subnet, port)
routervpnagentbindings: keeps the VPN agent ID per router

REST API and CLI impact

There are no changes to the REST API or CLI of VPNaaS or Neutron.

The external IP address of the VPN service is visible in its external_v4_ip/external_v6_ip and will be different than the one of the router.

References

https://bugs.launchpad.net/neutron/+bug/1905391

Source and destination filtering for metering label rules

Tue, 04 Aug 2020 00:00:00

This spec adds source and destination filtering options to Neutron metering label rules.

Problem Description

Neutron metering label rules have a parameter called “remote-ip-prefix”, which would allow operators to filter traffic based on the remote IP address. However, since [1], its meaning was changed to the exact opposite, which makes a bit of confusion. Instead of matching on the remote prefix (towards the external interface), it matches the local prefix (towards the OpenStack tenant networks).

Ideally, to satisfy the use case presented in [1] (which was achieved by inverting the use of “remote-ip-prefix”), operators should be able to create rules based on local-ip-prefix and remote-ip-prefix.

Proposed Change

As discussed in the Neutron drivers metering that approved the RFE [2] of this spec, we will deprecate the parameter remote-ip-prefix of Neutron metering API.

Therefore, we will be introducing two new parameters with this spec in the Neutron metering rule API. These new parameters will be called “source_ip_prefix”, and “destination_ip_prefix” (like in IPtables). The behavior of “remote_ip_prefix” will be maintained, but we will fix its documentation and mark it for removal in future releases [1].

The “source_ip_prefix” and “destination_ip_prefix” could be used together, or only one of them can be defined. However, a metering rule must always have at least one of them (source_ip_prefix or destination_ip_prefix) defined. On the other hand, these two new parameters will not be allowed to be used in conjunction with “remote_ip_prefix”.

API JSON

Current JSON for “v2.0/metering/metering-label-rules” endpoint:

{
  "remote_ip_prefix": "192.168.0.14/32",
  "direction": "egress",
  "metering_label_id": "9ffd6512-9d2a-4dd2-9657-6a605126264d",
  "id": "f1694467-d866-4d8e-a8dc-18da516caedc",
  "excluded": false
}

Adding new attributes:

{
  "source_ip_prefix": "192.168.0.14/32",
  "destination_ip_prefix": "0.0.0.0/0",
  "direction": "egress",
  "metering_label_id": "9ffd6512-9d2a-4dd2-9657-6a605126264d",
  "id": "f1694467-d866-4d8e-a8dc-18da516caedc",
  "excluded": false
}

Database table changes

Currently, the table “meteringlabelrules” is defined as:

+-------------------+--------------------------+------+-----+---------+-------+
| Field             | Type                     | Null | Key | Default | Extra |
+-------------------+--------------------------+------+-----+---------+-------+
| id                | varchar(36)              | NO   | PRI | NULL    |       |
| direction         | enum('ingress','egress') | YES  |     | NULL    |       |
| remote_ip_prefix  | varchar(64)              | YES  |     | NULL    |       |
| metering_label_id | varchar(36)              | NO   | MUL | NULL    |       |
| excluded          | tinyint(1)               | YES  |     | 0       |       |
+-------------------+--------------------------+------+-----+---------+-------+

We would add two new fields to it. Therefore, it would look like:

+-----------------------+--------------------------+------+-----+---------+-------+
| Field                 | Type                     | Null | Key | Default | Extra |
+-----------------------+--------------------------+------+-----+---------+-------+
| id                    | varchar(36)              | NO   | PRI | NULL    |       |
| direction             | enum('ingress','egress') | YES  |     | NULL    |       |
| remote_ip_prefix      | varchar(64)              | YES  |     | NULL    |       |
| source_ip_prefix      | varchar(64)              | YES  |     | NULL    |       |
| destination_ip_prefix | varchar(64)              | YES  |     | NULL    |       |
| metering_label_id     | varchar(36)              | NO   | MUL | NULL    |       |
| excluded              | tinyint(1)               | YES  |     | 0       |       |
+-----------------------+--------------------------+------+-----+---------+-------+

Neutron Metering agent changes

The IPtables driver in the metering agent will need to handle the new parameters “source_ip_prefix” and “destination_ip_prefix” properly. When building the IPtable rules the parameter “destination_ip_prefix” (if defined) will be used with the option “-d” (IPtables option). On the other hand, the parameter “source_ip_prefix” (if defined) will be used with option “-s”(IPtables option).

Validations

To simplify validations, we propose to remove the overlapping IP_prefixes validations for the new fields (for the remote IP prefix, it will be maintained). The rationally behind this removal is that if the operator wants to somehow create rules that overlap, we should not be the ones blocking it (there might be some business logic that needs it).

We will implement the following validation: * The source IP prefix must be a valid IPv4 CIDR * The destination IP prefix must be a valid IPv4 CIDR * Each metering label rule requires at least source, destination, or remote IP prefix to be informed. The remote IP prefix is marked as deprecated. Therefore, once it is removed, the users will have to enter at least source or destination IP prefixes. One can also use both (source and destination IP prefixes) to build rule. * source and destination IP prefixes cannot be used in conjunction with remote IP prefix.

API impacts

Two new parameters will be introduced, but they are not required. Therefore, people using it would not suffer an immediate impact. However, when the “remote_ip_prefix” is removed, people might have a problem. therefore, as soon as the new method of building rules is available, people will be encouraged to use it, instead of the “remote_ip_prefix” metering rule base.

Assignee(s)

Primary assignees:

Rafael <rafael@apache.org>

Other contributors:

Work Items

The following are the work items for the planned release.

Deprecate remote IP prefix (Neutron-lib)

Deprecate remote IP prefix

Fix documentation

Add source and destination attributes (Neutron-lib) – executed via [3]

Add new attributes in api/definitions/metering.py

Fix JSON of examples and documentation

Deprecate remote IP prefix (Neutron)

Deprecate remote IP prefix

Fix documentation

Log a warning when people use it

Change execution flow in Neutron and Neutron metering agent to use the new fields. (Neutron)

Add the new DB fields in objects.metering.MeteringLabelRule and neutron/db/models/metering.MeteringLabelRule

DB script in neutron/db/migration/alembic_migrations/versions/victoria

Actual implementation to use the new attributes, and unit tests

Update the documentation of the API with the new fields

Deprecate remote IP prefix (OpenStack SDK)

Deprecate remote IP prefix

Fix documentation

Log a warning when people use it

Deprecate remote IP prefix (OpenStack python client)

Deprecate remote IP prefix

Fix documentation

Log a warning when people use it

Add the new fields (OpenStack SDK)

Add the new fields

Fix documentation

Add the new fields (OpenStack python client)

Add the new fields

Fix documentation

After we finish all of these items, in a future release, we will need to execute the following removal items.

Remove remote IP prefix (Neutron-lib)

Fix JSON of examples and documentation

Remove remote IP prefix (Neutron)

Fix documentation

Remove remote IP prefix (OpenStack SDK)

Fix documentation

Remove remote IP prefix (OpenStack python client)

Fix documentation

Dependencies

None

References

Port NUMA affinity policy

Fri, 31 Jul 2020 00:00:00

https://bugs.launchpad.net/neutron/+bug/1886798

Add a new parameter to “port” object setting the NUMA affinity policy.

Problem Description

Currently Nova allows to define the “numa_affinity_policy” for PCI devices. Those PCI devices could be, for example, network cards. When a port is created using this physical device (an SR-IOV port with VNIC type “direct”, “direct-physical”, “macvtap” or “virtio-forwarder”), Nova will select the host to spawn the virtual machine depending on the host NUMA topology availability and the NUMA nodes associated to this PCI network device [1].

However this filtering cannot be done currently for other backends than SR-IOV. For example, Open vSwitch with DPDK will run the PMD threads attached to specific CPU cores (and to specific NUMA nodes) [2]. Nova represents this information assigning a set of NUMA nodes to each defined physical or tunneled network [3]. If the port NUMA affinity policy is provided, Nova will enforce it during the scheduling.

Proposed Change

The goal of this spec is to create a “numa_affinity_policy” parameter applicable to all ports. This information will be provided to Nova that will use it while scheluding the virtual machine, independently of the network backend.

Of course, Nova should have the network backend NUMA information topology. For example, since [3], Nova can be statically configured with the physical networks NUMA location. With the port “numa_affinity_policy” parameter, the Nova scheduler will filter those hosts matching the required policy.

This spec covers the missing cases defined in [4]. Defining the NUMA affinity policy per port (interface), includes all type of ports, not only the SR-IOV interfaces. It also avoids defining the NUMA affinity in the Nova flavor; instead of this, a generic flavor can be used with ports with different policies.

This spec proposes to add a new parameter to the “port” object. This parameter, “numa_affinity_policy”, will be an string defining the NUMA affinity policy, based on the PCINUMAAffinityPolicy [5] enum defined in Nova. That field could have different values: “required”, “legacy” and “preferred”.

This information will be populated in the “port” dictionary when informing to Nova. This parameter will be added as a new extension in the port dictionary, in a new field called “numa_affinity_policy”. By default, the value of this new parameter will be “None”, to keep backwards compatibility.

Data Model Impact

A new table, “portnumaaffinitypolicy”, will be created:

Attribute Name	Type	CRUD	Default Value	Validation/Conversion
port_id	uuid-str	R
numa_affinity_policy	str	CRU	None	enum (including None)

This child table will depend on the “ports” table. Each row will have a 1:1 relationship with a “port” row and will be deleted when the “port” row is deleted too.

REST API Impact

The parameter “numa_affinity_policy” in the “port” API:

NUMA_AFFINITY_POLICY_VALUES = (None, 'required', 'preferred', 'legacy')

RESOURCE_ATTRIBUTE_MAP = {
    'port': {
        'numa_affinity_policy': {
            'allow_post': True,
            'allow_put': True,
            'validate': {'type:values': NUMA_AFFINITY_POLICY_VALUES}
            'default': None,
            'is_visible': True}
    }
}

The parameter can be updated only if the port is not bound. That check does not depend on the API but on the server.

Security Impact

None

Performance Impact

None

Operators CLI Impact

An additional parameter will be added to the OSC “port” CLI interface, in the create, set and unset commands.

For logging resource:

openstack port create [--numa-policy-required | --numa-policy-preferred |
                       --numa-policy-legacy]

openstack port set [--numa-policy-required | --numa-policy-preferred |
                    --numa-policy-legacy]

openstack port unset --numa-policy

Implementation

Assignee(s)

Primary assignee:: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> (IRC: ralonsoh)

Testing

Unit Test
Functional test
API test

Documentation Impact

User Documentation

Add CLI usage into the networking guide for operator.

References

Floating IP port forwarding to support port ranges

Mon, 06 Jul 2020 00:00:00

https://blueprints.launchpad.net/neutron/+spec/floatingips-portforwarding-ranges

The use of port ranges in NAT eases the rules creation process as it replaces the use of many entries in the iptables by just one. Also, some users, when requesting virtual machines that have some ports exposed to the Internet, prefer to request a range of ports to be exposed to avoid doing many requests for operators regarding the networking configurations. Therefore, they usually request a slice of external ports to deploy their applications, so they will be able to choose and change the ports they desire on the fly without needing to contact anyone else or configuring it in OpenStack.

Moreover, there are applications such as FTP, video conference systems, streaming, and others that use more than one port (ports in sequence).

Therefore, it is easier to enable all of them (the requested/needed port range) in a single configuration.

Problem Description

Currently, if a user wants to create NAT rules that cover multiple ports, he/she needs to create them one by one, which is cumbersome in some use cases.
When configuring NAT via iptables [1] (and other networking systems/applications), operators can use port ranges while configuring a rule; therefore, it is natural for network engineers to handle/use port ranges in their NAT configurations.

Proposed Change

Change the Floating IP port forwarding API to allow the use of port ranges to create NAT rules. The changes are presented as follows. Today, when a user is creating a port forwarding rule, he/she creates a rule via API, or CLI and sends a JSON like:

{
  "port_forwarding": {
    "protocol": "tcp",
    "internal_ip_address": "172.16.0.7",
    "internal_port": 80,
    "internal_port_id": "b67a7746-dc69-45b4-9b84-bb229fe198a0",
    "external_port": 8080,
    "description": "desc"
  }
}

The external/internal ports are integers and if the user desires to create a port forwarding in a floating ip for many ports, he/she will need to resend this JSON many times to fit the number of ports he/she desires.

The proposal is to change the API/CLI to accept a string in external/internal ports range. This will allow users to create a rule with port ranges (instead of executing multiple calls). Therefore, a user would not need anymore to create 100 rules to reach 100 port forwards in his/her virtual machine, he/she could just send a JSON like that:

{
  "port_forwarding": {
    "protocol": "tcp",
    "internal_ip_address": "172.16.0.7",
    "internal_port_range": "8000:8100",
    "internal_port_id": "b67a7746-dc69-45b4-9b84-bb229fe198a0",
    "external_port_range": "9000:9100",
    "description": "desc"
  }
}

The API will be still accepting the internal_port and external_port attributes to maintain backward compatibility.

The new expected response will be a JSON like:

{
  "port_forwarding": {
    "protocol": "tcp",
    "internal_ip_address": "172.16.0.7",
    "internal_port_range": "8000:8100",
    "internal_port_id": "b67a7746-dc69-45b4-9b84-bb229fe198a0",
    "external_port_range": "9000:9100",
    "description": "desc",
    "id": "825ade3c-9760-4880-8080-8fc2dbab9acc"
  }
}

New validations will be created to avoid users to enter invalid ranges in ports. Here follows a summary of all of the validations executed by the system:

Only N(external port[s]) to N (internal port[s]) are allowed or N(external port[s]) to 1 (internal port). Therefore, all of the other combinations (1:N, N:M, M:N) are not allowed and will generate an error.
When defining a port(s) range, the ports in this range cannot be already in use by any other port forwarding rule for the same floating IP and protocol.
A valid port is a number between 1 and 65535. All other cases will generate an error. Even though “0” is a valid port, we will not allow its use as it is reserved.
The notation to define a port range is the following: <port_range_begin>[:<port_range_end>]; anything that does not match this definition will generate an error.
When doing the request, the user can choose between sending a JSON with internal_port_range (string) or internal_port (integer), the same to the external ports, sending both internal_port_range and internal_port will raise a validation error.

Data Model Impact

Today, the port_forwarding table schema is something like:

id	floatingip_id	external_port	internal_neutron_port_id	protocol	socket
A1	C2	80	A2	tcp	172.16.0.7:8080
B1	C2	81	B2	tcp	172.16.0.7:8081

To make the port_forwarding table more like the PortForwarding object and the JSON, the port_forwarding table will be updated splitting the socket column into three new columns (internal_ip_address, internal_port_start, internal_port_end), also, the external_port column will be split into two new columns (external_port_start and external_port_end). The new table would be like below:

id	floatingip_id	external_port	external_port_start	external_port_end	internal_neutron_port_id	protocol	internal_ip_address	internal_port_start	internal_port_end	socket
A1	C2	80	80	80	A2	tcp	172.16.0.7	8080	8080	172.16.0.7:8080
B1	C2	81	81	81	B2	tcp	172.16.0.7	8081	8081	172.16.0.7:8081

The legacy data migration would be done with an alembic migration upgrade script.

We have 5 steps in the data migration:

Create the internal_ip_address, internal_port_start, external_port_start, internal_port_end, external_port_end columns with NULL values;
Select id, socket and external_port columns for every table entry;
For each entry, split the socket value by ‘:’ and put the result in the new internal_ip_address and internal_port_start columns from same id as the socket. Migrate the value of the internal_port_start column in the internal_port_end and migrate the value of the external_port column in the external_port_start and external_port_end columns;
Change the socket column to nullable;
Change the external_port column to nullable;

Sub Resource Extension

It will be created as an extension that overrides the parameters (external_port and internal_port) to also accept port ranges in their validations. The attributes map of new sub resource would be like:

SUB_RESOURCE_ATTRIBUTE_MAP = {
    'port_forwarding': {
        'parameters': {
            'external_port_range': {
                'allow_post': True, 'allow_put': True,
                'validate': {'type:port_range': None},
                'is_visible': True,
                'is_sort_key': True,
                'is_filter': True},
            'internal_port_range': {
                'allow_post': True, 'allow_put': True,
                'validate': {'type:port_range': None},
                'is_visible': True},
        }
    }
}

REST API Impact

None.

Implementation

Assignee(s)

Primary assignees:

Pedro <phpm13@gmail.com>
Rafael <rafael@apache.org>

Other contributors:

Work Items

API extension (neutron-lib)
DB extension/data migration (neutron)
Extend Neutron API with new validations for port ranges (neutron)
Extend l3 agent to apply the ranges in the iptable rule (neutron)
Extend the floating ip port forwarding command to accept and validate ranges in the CLI (python-openstackclient)
Change the port types from int to string in the PortForwarding object (openstacksdk)
Tests
Documentation

To be done in future releases

1) Remove the columns “socket” and “external_port” from the port_forwarding table in the neutron’s database, as these columns are no more used after the release wallaby.

Dependencies

None

References

Fix the dns-assignment on neutron port

Thu, 02 Jul 2020 00:00:00

https://bugs.launchpad.net/neutron/+bug/1873091

The dns-domain part of the dns-assignment for the neutron port is always taken from the dns-domain defined in the neutron configuration

Problem Description

When the user enable the dns or dns_domain_ports extension there are multiple ways to define Designate dns_domain (Designate zone) in Neutron side as follows:

Neutron configuration file which contain the default value for dns_domain
Neutron network definition which define the dns_domain for network and enable the dns extension for that network so every port created under the network will have a dns record in that dns_domain and it override the default value in configuration file
Neutron port definition when the dns_domain_ports extension is enabled which is used to override the Neutron network dns_domain and create the dns record under another dns_domain

The dns-domain can be set on network level using openstack network set –dns-domain and on port level using dns_domain_ports extension openstack port create –dns-domain and both of those values (network level, port level dns-domians) are ignored when forming the port dns-assignment which always take the dns-domain in neutron config

Proposed Change

The port level dns-domain should take precedence over network level dns-domain which take precedence over the neutron config dns-domain which will be the default if the above two levels dont have dns-domain assigned

Testing

Unit Test

Unit test should be added to make sure that the right dns-domain is used in Neutron port dns assignment

Documentation Impact

User Documentation

The documemtation is showing the dns assignment with the dns domain defined in the Neutron configurations or defined in the Neutron network (which is usually the same) but not the dns_domain comming from Neutron port creare (use case 3) which should be updated

References

https://docs.openstack.org/neutron/latest/admin/config-dns-int-ext-serv.html#config-dns-int-ext-serv

Allow sharing security groups as read-only

Fri, 15 May 2020 00:00:00

https://bugs.launchpad.net/neutron/+bug/1875516

Allow sharing security groups as read-only.

Problem Description

Currently, security groups can be shared with the rbac system, but the only valid action is access_as_shared, which allows the target tenant to create/delete (only) new rules on the security group. This works fine for use-cases where the group should be shared in a nearly equal way.

However, some users/services may want a security group to be visible, but read-only. A prime example of this would be to enable ProjectB to add a security group owned by ProjectA as a remotely trusted group on their own security group.

The immediate need for this is found in an existing Octavia patch [1]. Octavia would like to share the security group it creates for each load-balancer with the load-balancer’s owner, so they can open access to their backend members for only a specific load-balancer.

Proposed Change

Add a new action type for security group RBAC: access_as_readonly. This action would allow the target tenant to see the shared security group with show/list, but not create/delete new rules for it or change it in any way.

Documentation Impact

Neutron documentation about sharing security groups will need to be modified to add the action type access_as_readonly.

Implementation

Assignee(s)

Adam Harwell

Work Items

Add new action type access_as_readonly
Documentation update in config-rbac.rst [2] as seen in [3]
Create additional tempest tests in RbacSharedSecurityGroupTest class [4]

References

Address Groups Support in Security Group Rules

Mon, 20 Apr 2020 00:00:00

https://bugs.launchpad.net/neutron/+bug/1592028

This specification describes how to support address groups (groups of IP address blocks) in Neutron security group rules. The concept of address groups was introduced in FWaaS v2.0 but not implemented [1].

Problem Description

Neutron security group rules currently support using an IP address block or a security group as the remote end of the network access rule. In actual usage, an OpenStack cloud may require connectivity between instances and external services which are not provisioned by OpenStack. And each service may also have multiple endpoint addresses which are not contiguous. To allow the connectivity, one rule per external address needs to be created which can be very cumbersome and difficult to maintain as the number of external addresses may be substantial.

Proposed Change

Add an API to aggregate IP address blocks into an address group object which could be later referenced when creating a security group rule. Thus the number of security group rules can be effectively reduced when there is the need to allow connectivity to a number of external service addresses. When IP addresses are updated in the address group, changes will also be reflected in the associated security group rules.

REST API Impact

New address groups API

Attribute	Type	Req	CRUD	Description
id	uuid-str	N/A	R	Unique identifier for the address_group object.
name	String	No	CRU	Human readable name for the address group (255 characters limit). Does not have to be unique.
description	String	No	CRU	Human readable description for the address group (255 characters limit).
project_id	uuid-str	No	CR	Owner of the address group. Only admin users can specify a project identifier other than their own.
addresses	List	Yes	CRU	Array of address. It supports both CIDR and IP range objects. An example of addresses: [“132.168.4.12/24”, “132.168.5.12-132.168.5.24”, “2001::db8::f00/64”]

List address groups

Lists address groups.

Request Type

GET

Endpoint

/v2.0/address-groups

Response Codes

Success

200

Error

Unauthorized(401)

Example List address groups: JSON request

GET /v2.0/address-groups
User-Agent: python-neutronclient
Accept: application/json

Example List address groups: JSON response

{
    "address_groups": [
        {
            "description": "",
            "id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
            "name": "ADDR_GP_1",
            "project_id": "45977fa2dbd7482098dd68d0d8970117",
            "addresses": [
               "132.168.4.12/24",
               "132.168.5.12-132.168.5.24",
               "2001::db8::f00/64"
            ]
        }
    ]
}

Show address group details

Shows address group details.

Request Type

GET

Endpoint

/v2.0/address-groups/<address_group_id>

Response Codes

Success

200

Error

Unauthorized(401), Not Found (404)

Example Show address group: JSON request

GET /v2.0/address-groups/8722e0e0-9cc9-4490-9660-8c9a5732fbb0
User-Agent: python-neutronclient
Accept: application/json

Example Show address group: JSON response

{
   "address_group": {
        "description": "",
        "id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
        "name": "ADDR_GP_1",
        "project_id": "45977fa2dbd7482098dd68d0d8970117",
        "addresses": [
           "132.168.4.12/24",
           "132.168.5.12-132.168.5.24",
           "2001::db8::f00/64"
        ]
    }
}

Create address group

Creates an address group.

Request Type

POST

Endpoint

/v2.0/address-groups/

Response Codes

Success

201

Error

Unauthorized(401), Bad Request(400)

Example Create address group: JSON request

POST /v2.0/address-groups
User-Agent: python-neutronclient
Accept: application/json

{
    "address_group": {
        "name": "ADDR_GP_1",
        "addresses": [
           "132.168.4.12/24",
           "132.168.5.12-132.168.5.24",
           "2001::db8::f00/64"
        ]
    }
}

Example Create address group: JSON response

HTTP/1.1 201 Created
Content-Type: application/json; charset=UTF-8

{
   "address_group": {
        "description": "",
        "id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
        "name": "ADDR_GP_1",
        "project_id": "45977fa2dbd7482098dd68d0d8970117",
        "addresses": [
           "132.168.4.12/24",
           "132.168.5.12-132.168.5.24",
           "2001::db8::f00/64"
        ]
    }
}

Update address group

Updates an address group. To update addresses, use the add addresses and remove addresses operations.

Request Type

PUT

Endpoint

/v2.0/address-groups/<address_group_id>

Response Codes

Success

200

Error

Unauthorized(401), Bad Request(400) Not Found(404)

Example Update address group: JSON request

PUT /v2.0/address-groups/8722e0e0-9cc9-4490-9660-8c9a5732fbb0
User-Agent: python-neutronclient
Accept: application/json

{
    "address_group": {
        "description": "new description",
        "name": "new name"
    }
}

Example Update address group: JSON response

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
   "address_group": {
        "description": "new description",
        "id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
        "name": "new name",
        "project_id": "45977fa2dbd7482098dd68d0d8970117",
        "addresses": [
           "132.168.4.12/24",
           "132.168.5.12-132.168.5.24",
           "2001::db8::f00/64"
        ]
    }
}

Add addresses to address group

Add addresses to an existing address group.

Request Type

PUT

Endpoint

/v2.0/address-groups/<address_group_id>/add_addresses

Response Codes

Success

200

Error

Unauthorized(401), Bad Request(400), Not Found(404)

Example Update addresses: JSON request

PUT /v2.0/address-groups/8722e0e0-9cc9-4490-9660-8c9a5732fbb0/add_addresses
User-Agent: python-neutronclient
Accept: application/json

{
    "addresses": [
       "10.0.0.1/32",
       "2001:3889:120:fe42::/64"
    ]
}

Example Update addresses: JSON response

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
   "address_group": {
        "description": "",
        "id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
        "name": "ADDR_GP_1",
        "project_id": "45977fa2dbd7482098dd68d0d8970117",
        "addresses": [
           "132.168.4.12/24",
           "132.168.5.12-132.168.5.24",
           "10.0.0.1/32",
           "2001::db8::f00/64",
           "2001:3889:120:fe42::/64"
        ]
    }
}

Remove addresses from address group

Remove addresses from an existing address group.

Request Type

PUT

Endpoint

/v2.0/address-groups/<address_group_id>/remove_addresses

Response Codes

Success

200

Error

Unauthorized(401), Bad Request(400), Not Found(404)

Example Remove addresses: JSON request

PUT /v2.0/address-groups/8722e0e0-9cc9-4490-9660-8c9a5732fbb0/remove_addresses
User-Agent: python-neutronclient
Accept: application/json

{
    "addresses": [
       "132.168.4.12/24",
       "2001::db8::f00/64"
    ]
}

Example Remove addresses: JSON response

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
   "address_group": {
        "description": "",
        "id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
        "name": "ADDR_GP_1",
        "project_id": "45977fa2dbd7482098dd68d0d8970117",
        "addresses": [
           "132.168.5.12-132.168.5.24"
        ]
    }
}

Delete address group

Deletes an address group.

This operation does not return a response body.

Request Type

DELETE

Endpoint

/v2.0/address-groups/<address_group_id>

Response Codes

Success

204

Error

Unauthorized(401), Not Found(404) Conflict(409) The Conflict error response is returned when an operation is performed while address group is in use.

Example Delete address group: JSON request

DELETE /v2.0/address-groups/8722e0e0-9cc9-4490-9660-8c9a5732fbb0
User-Agent: python-neutronclient
Accept: application/json

Example Delete address group: JSON response

HTTP/1.1 204 No Content
Content-Length: 0

Changes to the existing security group rules API

List security group rules

Lists security group rules.

Remote address group id field will be added to response.

GET /v2.0/security-group-rules
User-Agent: python-neutronclient
Accept: application/json

Example List security group rules: JSON response

{
    "security_group_rules": [
        {
            "direction": "ingress",
            "ethertype": "IPv4",
            "id": "f7d45c89-008e-4bab-88ad-d6811724c51c",
            "port_range_max": null,
            "port_range_min": null,
            "protocol": null,
            "remote_group_id": null,
            "remote_ip_prefix": null,
            "remote_address_group_id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
            "security_group_id": "85cc3048-abc3-43cc-89b3-377341426ac5",
            "project_id": "e4f50856753b4dc6afee5fa6b9b6c550",
            "revision_number": 1,
            "created_at": "2018-03-19T19:16:56Z",
            "updated_at": "2018-03-19T19:16:56Z",
            "tenant_id": "e4f50856753b4dc6afee5fa6b9b6c550",
            "description": ""
        }
    ]
}

Show security group rule details

Shows security group rule details.

Remote address group id field will be added to response.

Example Show security group rule: JSON request

GET /v2.0/security-group-rules/f7d45c89-008e-4bab-88ad-d6811724c51c
User-Agent: python-neutronclient
Accept: application/json

Example Show security group rule: JSON response

{
    "security_group_rule": {
        "direction": "ingress",
        "ethertype": "IPv4",
        "id": "f7d45c89-008e-4bab-88ad-d6811724c51c",
        "port_range_max": null,
        "port_range_min": null,
        "protocol": null,
        "remote_group_id": null,
        "remote_ip_prefix": null,
        "remote_address_group_id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
        "security_group_id": "85cc3048-abc3-43cc-89b3-377341426ac5",
        "project_id": "e4f50856753b4dc6afee5fa6b9b6c550",
        "revision_number": 1,
        "created_at": "2018-03-19T19:16:56Z",
        "updated_at": "2018-03-19T19:16:56Z",
        "tenant_id": "e4f50856753b4dc6afee5fa6b9b6c550",
        "description": ""
    }
}

Create security group rule

Creates a security group rule with a remote address group.

Note: At most one of remote-group, remote-ip and remote-address-group can be specified. The rule is matched when the remote IP address in the packet matches any one of: remote_ip_address, one of the IP addresses in the address group, or an IP address of one of the ports in the remote security group.

Example Create security group rule: JSON request

POST /v2.0/security-group-rules
User-Agent: python-neutronclient
Accept: application/json

{
    "security_group_rule": {
        "direction": "ingress",
        "port_range_min": "80",
        "ethertype": "IPv4",
        "port_range_max": "80",
        "protocol": "tcp",
        "remote_address_group_id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
        "security_group_id": "a7734e61-b545-452d-a3cd-0189cbd9747a"
    }
}

Example Create security group rule: JSON response

HTTP/1.1 201 Created
Content-Type: application/json; charset=UTF-8

{
    "security_group_rule": {
        "direction": "ingress",
        "ethertype": "IPv4",
        "id": "2bc0accf-312e-429a-956e-e4407625eb62",
        "port_range_max": 80,
        "port_range_min": 80,
        "protocol": "tcp",
        "remote_ip_prefix": null,
        "remote_group_id": null,
        "remote_address_group_id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
        "security_group_id": "a7734e61-b545-452d-a3cd-0189cbd9747a",
        "project_id": "e4f50856753b4dc6afee5fa6b9b6c550",
        "revision_number": 1,
        "tenant_id": "e4f50856753b4dc6afee5fa6b9b6c550",
        "created_at": "2018-03-19T19:16:56Z",
        "updated_at": "2018-03-19T19:16:56Z",
        "description": ""
    }
}

Data Model Impact

The following are the backend database tables for the REST API proposed above.

Address Groups

Attribute	Type	Req	CRUD	Description
id	uuid-str	N/A	R	Unique identifier for the address_group object.
name	String	No	CRU	Human readable name for the address group (255 characters limit). Does not have to be unique.
description	String	No	CRU	Human readable description for the address group (255 characters limit).
project_id	uuid-str	Yes	CR	Owner of the address group. Only admin users can specify a project identifier other than their own.

Address Group Address Associations

Attribute	Type	Req	CRUD	Description
address_group_id	uuid-str	No	CRU	UUID of address group.
address	String	No	CRU	Address that has to be associated to the address group.

Security Group Rules

Attribute remote_address_group_id will be added to security group rules table

Attribute	Type	Req	CRUD	Description
remote_address _group_id	String	No	CRU	When a remote_address_group is specified, it is matched when the remote IP address in the packet matches one of the IP addresses in the address group. This is exclusive with remote_ip_prefix and remote_group_id.

Implementation

Assignee(s)

Hang Yang

Work Items

REST API update
DB schema update
CLI update
Open vSwitch and iptables firewall drivers update
RBAC support for address groups
Documentation update

Testing

Tempest Tests

DB mixin and schema tests
Tempest tests
CLI tests

Functional Tests

New tests need to be written

API Tests

REST API and attributes validation tests

Documentation Impact

User Documentation

Neutron CLI and API documentation have to be modified.

Developer Documentation

Neutron API devref and documentation need to be updated.

References

[1] https://specs.openstack.org/openstack/neutron-specs/specs/rocky/fwaas-2.0-address-groups-support.html

[2] https://docs.openstack.org/api-ref/network/v2/#security-group-rules-security-group-rules

Integration of Neutron Classifier and QoS DSCP

Mon, 20 Jan 2020 00:00:00

https://bugs.launchpad.net/neutron/+bug/1476527

Neutron Classifier(NC) is a service plugin which provides an API to define traffic classes for consumption by other Neutron services.

Quality of service(QoS) is an OpenStack Neutron extension which provides administrators with a service to specify and enforce SLAs based on drop priority, bandwidth policing and bandwidth guarantees at a port level.

By introducing the consumption of NC’s Classification Group resources into QoS’s Rule resources we can allow for a more targeted form of SLA allowing admins to limit, prioritize or guarantee traffic based on traffic classes rather than on a Neutron Port level. This spec will focus on drop priority based on traffic class per port.

Problem Description

Neutron’s Quality of Service extension allows for SLAs to be enforced at a Neutron port level, however it does not provide a resource to allow administrators to define SLAs for the different traffic classes entering or leaving the port.

This results in an all or nothing approach to traffic class SLAs, either all the traffic leaving a port recieves the same DSCP mark or the traffic retains its default DSCP mark assigned by the workload.

Example Use Cases:

Assigning a higher drop priority for traffic leaving TCP port YY than other traffic leaving the Neutron port.
Overriding an existing DSCP mark assigned by the instance destined for IPv6 address XX with UDP port YY specifically.

Proposed Change

In order to consume the classification resources of the Neutron Classifier in a way that does not involve the introduction of external dependencies in Neutron, the Neutron Classifier service plugin will require it to be migrated into the core Neutron repository.

Details of the resource models and API can be found here ‘NC API spec’_

The modifications required to Neutron’s QoS extension is minimal, it involves the introduction of a two variables to the DSCP Marking rule. The first variable takes an ID for a classification group (a collection of individual classifications) which the QoS backends can consume to target DSCP Marks to traffic classes as opposed to marking all traffic leaving the port.

The other, a priority value, will be a positive integer ranging from 0-1000 in ascending priority. This will allow admins to decide which DSCP marking rules should be applied in a case where 2 DSPC rule classifications overlap. eg:

openstack network qos policy create example_policy

openstack network qos rule create example_policy --type dscp-marking \
    --dscp-mark 22

openstack network classification create ethernet --ethertype 0x0800 \
    eth_ipv4
openstack network classification create ipv4 --protocol 6 \
    ipv4_tcp
openstack network classification create tcp --dst-port-max 80 \
    tcp_80
openstack network classification group create class_1 \
    --classification tcp_80 ipv4_tcp eth_ipv4

openstack network qos rule create example_policy --type dscp-marking \
    --dscp-mark 28 --classification-group-id class_1 \
    --classification-priority 100

openstack network classification create ipv4 --dst-addr 10.0.0.5 \
    ipv4_dst
openstack network classification group create class_2 \
    --classification ipv4_dst eth_ipv4

openstack network qos rule create example_policy --type dscp-marking \
    --dscp-mark 24 --classification-group-id class_2 \
    --classification-priority 600

openstack network classification create ethernet --ethertype 0x86DD \
    eth_ipv6
openstack network classification create ipv6 --next-header 17 \
    --dst-addr 0:0:0:0:0:ffff:a00:5 ipv6_dst
openstack network classification create udp --dst-port-max 21 \
    udp_21
openstack network classification group create class_3 \
    --classification eth_ipv6 ipv6_dst udp_21

openstack network qos rule create example_policy --type dscp-marking \
    --dscp-mark 8 --classification-group-id class_3 \
    --classification-priority 158

openstack network classification create ethernet --ethertype 0x0806 \
    eth_arp
openstack network classification group create class_4 \
    --classification eth_arp

openstack network qos rule create example_policy --type dscp-marking \
    --dscp-mark 40 --classification-group-id class_4 \
    --classification-priority 999

openstack network qos rule list example_policy
+--------------------------------------+--------------------------------------+--------------+----------+-----------------+----------+-----------+-----------+--------------------------------------+----------+
| ID                                   | QoS Policy ID                        | Type         | Max Kbps | Max Burst Kbits | Min Kbps | DSCP mark | Direction | Classification Group                 | Priority |
+--------------------------------------+--------------------------------------+--------------+----------+-----------------+----------+-----------+-----------+--------------------------------------+----------+
| 153e4957-5c50-47f5-b934-c30eb3ada180 | 812c3e2e-cc86-4842-b41b-6b69419a123b | dscp_marking |          |                 |          |        40 |           | ae0de79d-c26a-4090-8ec2-0d1ae62797f2 |      999 |
| 184d7156-ab93-444e-96eb-f91cf8c9160e | 812c3e2e-cc86-4842-b41b-6b69419a123b | dscp_marking |          |                 |          |         8 |           | b0fd0ca7-8596-4053-b69b-630016acfd28 |      158 |
| 385970f8-9a6c-4fe2-a328-95630138c9ca | 812c3e2e-cc86-4842-b41b-6b69419a123b | dscp_marking |          |                 |          |        28 |           | 6adb4199-e42f-4fa9-8ab6-2bf64e2ea68d |      100 |
| 7fa581ac-8b82-4d64-9032-52b1ce9f3509 | 812c3e2e-cc86-4842-b41b-6b69419a123b | dscp_marking |          |                 |          |        22 |           |                                      |        0 |
| d1c5268b-302c-498b-bfd7-484e19fd28af | 812c3e2e-cc86-4842-b41b-6b69419a123b | dscp_marking |          |                 |          |        24 |           | cd0a1ea2-0cdf-42b0-a4fc-7f581d4a4be2 |      600 |
+--------------------------------------+--------------------------------------+--------------+----------+-----------------+----------+-----------+-----------+--------------------------------------+----------+

Without a clear priority between the above classes there are instances where classes class_1, class_3, class_4 and class_5 will also match class_2. Depending on the backend this can result in unpredictible behavior without a clear prioritization mechanism between traffic classes.

To accomodate these changes, modifications will need to be made in the backend to allow drivers to retrieve the classification definitions from the service plugin. This will be achieved through the extension of the agent extensions API. By extending this API all extensions will have access to classifications without having to introduce any additional resources.

Data Model Impact

In order to provide consistent functionality on upgrade the classification_group_id defaults to None, meaning a DSCP marking rule will behave the same way it did before the introduction of this feature.

The proposed change would result in the model looking like this:

Attribute Name	Type	Access	Default Value	Validation/ Conversion	Description
id	string (UUID)	RO, all	N/A	uuid	QoSRule identifier
qos_policy_id	string (UUID)	RO, all	N/A	uuid	QoSPolicy reference UUID
dscp_mark	integer (enum)	RW, tenant	N/A	0 and 56, except 2-6, 42, 44, and 50-54	DSCP Mark
classification_group_id	string (UUID)	RW, tenant	None	uuid	Classification Group reference UUID
classification_priority	integer	RW, tenant	0	A positive integer	Positive integer to denote prioity of this rule

Another modification is the removal of the of the unique contraint on the qos_policy_id. This has been replaced by a unique constraint on the qos_policy_id and classification_group_id as a tuple. This is to allow multiple DSCP rules with different classifications within the same policy, this includes when the classification_group_id is None. As a result, a QosPolicy can only have a single DSCPMarkingRule with classification_group_id set to None.

Neutron DB table: QosDSCPMarkingRule

id: primary key
qos_policy_id: foreign key to QosPolicy.id
dscp_mark: Integer within a defined set of values.
classification_group_id: foreign key to ClassificationGroup.id
classification_priority: positive integer between 0 and 1000

Before delving into further detail in regards to the data model, API and how classifications can be used, a few points need to be clarified:

1 Classification is of a single type, e.g. either Ethernet, IP, HTTP, or another supported at the time of a specific CCF release. The definition, i.e. fields to match on, depends on the type specified.
To clarify, Classification Types define the set of possible fields and values for a Classification (essentially, an instance of that Classification Type). Classification Types are defined in code, where Classifications are created via the REST API as instances of those types.
Not all supported fields need to be defined - only the ones required by the Consuming Service - which it should validate on consumption.
There are also Classification Groups, which allow Classifications or other Classification Groups to be grouped together using boolean operators. CGs are the resources that will end up being consumed by Consuming Services.
From the Consuming Service’s point of view, Classifications can only be read, not created or deleted. They need to have been previously created using the User-facing Classifications API.
As Neutron Classifier is unable to force an update action to it’s consumers and as users may be unaware of all the consumers of a classification, the classification “definition” and “type” fields and the classification group “classifications”, “classification_groups” and “operator” fields cannot be updated after creation. ie. A classification is consumed by both QoS and Security Groups with a user only knowing about QoS consuming the classification.

The initial model of the CCF will include the following Classification Types: Ethernet, IPv4, IPv6, TCP and UDP, which when combined are sufficient to provide any 5-tuple classification.

The following table presents the attributes of a Classification Group (asterisk on RW means that the attribute is non-updatable):

Attribute Name

Type

Access CRUD

Default Value

Validation/ Conversion

Description

id

string (UUID)

RO, all

generated

uuid

Identity

project_id

string (UUID)

RO, project

from auth token

uuid

Project ID

name

string

RW, project

None

string

Name of Classification Group

description

string

RW, project

None

string

Human-readable description

shared

bool

RW, project

False

boolean

Shared with other projects

operator

string (values)

RW*, project

“and”

[“and”,
“or”]

Boolean connective: AND/OR

classification_groups

list

RW*, project

[]

List of Classification Groups included

classifications

list

RW* project

[]

List of Classifications included

Consuming Services will consume Classification Groups, and not atomic Classifications, any Classification needs to be grouped in a Classification Group to be consumed individually. As such, the “operator” field is to be ignored for Classification Groups that only contain 1 Classification inside.

The following table presents the attributes of Classifications of any of the types stated in this spec (asterisk on RW means that the attribute is non-updatable):

Attribute Name

Type

Access

Default Value

Validation/ Conversion

Description

id

string (UUID)

RO, all

generated

uuid

Identity

project_id

string (UUID)

RO, project

from auth token

uuid

Project ID

name

string

RW, project

None

string

Name of Classification

description

string

RW, project

None

string

Human-readable description

type

string

RW*, project

from enum of types

The type of the Classification

definition

type-specific attributes will go here, given their volume I won’t detail them unless requested.

Classification Groups and Classifications of every type will be stored as the following tables and relationships (with table name prefix ccf_):

                          +---------------------+
                          |classification_groups|
                          +---------------------+
                          |id                   |*
                          |cg_id                +--------+
                          |name                 |        |
                          |description          |        |
                          |project_id           |        |
                          |shared               +--------+
                          |operator             |1
                          +---------------------+
                                      |1
                                      |
                                      |*
                      +------------------------------+
                      |classification_groups_mapping |
                      +------------------------------+
                      |cg_id                         |
                      |classification_id             |
                      +------------------------------+
                                      |1
+--------------------+                |                +--------------------+
|ipv4_classifications|                |                |ipv6_classifications|
+--------------------+                |                +--------------------+
|classification_id   |                |                |classification_id   |
|ihl                 |1               |               1|traffic_class       |
|diffserv            +--------+       |       +--------+traffic_class_mask  |
|diffserv_mask       |        |       |       |        |length              |
|length              |        |       |       |        |next_header         |
|flags               |        |       |       |        |hops                |
|flags_mask          |        |       |       |        |src_addr            |
|ttl                 |        |1      |1     1|        |dst_addr            |
|protocol            |     +---------------------+     +--------------------+
|src_addr            |     |classifications      |
|dst_addr            |     +---------------------+
|options             |     |id                   |
|options_mask        |     |name                 |
+--------------------+     |description          |
                           |project_id           |
                           |shared               |     +-------------------+
                           |type                 |     |tcp_classifications|
                           +---------------------+     +-------------------+
                             1|      1|       |1       |classification_id  |
+-------------------+         |       |       |        |src_port           |
|udp_classifications|         |       |       |        |dst_port           |
+-------------------+         |       |       |        |flags              |
|classification_id  |1        |       |       |       1|flags_mask         |
|src_port           +---------+       |       +--------+window             |
|dst_port           |                1|                |data_offset        |
|length             |     +------------------------+   |option_kind        |
|window_size        |     |ethernet_classifications|   +-------------------+
+-------------------+     +------------------------+
                          |classification_id       |
                          |preamble                |
                          |src_addr                |
                          |dst_addr                |
                          |ethertype               |
                          +------------------------+

Masking fields allow the user to specify which individual bits of the respective main field should be looked up during classification.

Classification Types are used to select the appropriate model of the Classification and consequently what table it will be stored in.

Classification Groups get stored in a single table and can point to other Classification Groups, to allow mixing boolean operators.

operator in Classification Group: specifies the boolean operator used to connect all the child Classifications and Classification Groups of that group. This can be either AND or OR.

OR: Logical OR, the classifications contained in these groups are inter-changable within the traffic class ie. ipv4 dst_addr y.y.y.y or ipv6 dst_addr Z::ZZZZ.

AND: logical AND, the classifications or classification groups are used to create a traffic class, the composition of this classification can be made up of Classifications or OR operand ClassificationGroups.

REST API Impact

Proposed attribute:

SUB_RESOURCE_ATTRIBUTE_MAP = {
   'dscp_marking_rules':{
        'parent': {'collection_name': 'policies',
                   'member_name': 'policy'},
        'parameters': dict(QOS_RULE_COMMON_FIELDS,
            **{'dscp_mark': {
                'allow_post': True, 'allow_put': True,
                'convert_to': attr.convert_to_int,
                'is_visible': True, 'default': None,
                'validate': {'type:values': common_constants.
                            VALID_DSCP_MARKS}
              },
               'classification_group_id': {
                'allow_post': True,
                'allow_put': True,
                'is_visible': True,
                'default': None,
                'validate': {'type:uuid_or_none': None}
              },
               'classification_priority': {
                'allow_post': True,
                'allow_put': True,
                'convert_to': attr.convert_to_int,
                'is_visible': True,
                'default': 0,
                'validate': {'type:values': range(1000)}
        }
}

Sample REST calls:

GET /v2.0/qos/policies

Response:
{
    "policy": {
        "name": "AF32",
        "project_id": "<project-id>",
        "id": "<id>",
        "description": "This policy marks DSCP outgoing AF32 traffic for DTV Control",
        "shared": "False"
     }
}

GET /v2.0/qos/policies/<policy-uuid>

Response:
{
    "policy": {
        "tenant_id": "<tenant-id>",
        "id": "<id>",
        "name": "AF32",
        "description": "This policy marks DSCP outgoing AF32 traffic for DTV Control",
        "shared": False,
        "dscp_marking_rules": [{
            "id": "<id>",
            "policy_id": "<policy-uuid>",
            "dscp_mark": 16,
            "classification_group_id": "<classification_group_id>",
            "classification_priority": 800
        }]
     }
}

POST /v2.0/qos/policies/<policy-uuid>/dscp-marking-rules/
{
    "dscp_marking_rule": {
        "dscp_mark": 16
    }
}

Response:
{
    "dscp_marking_rule":{
        "id": "<id>",
        "policy_id": "<policy-uuid>",
        "dscp_mark": 16,
        "classification_group_id": None,
        "classification_priority": 0
    }
}

GET /v2.0/classification_groups/<classification_group_id>
{
    "classification_group":{
        "id": "<classification_group_id>",
        "project_id": "<project_id>",
        "name": "ipv4-group"
        "description": "",
        "classification": [
            {
                "id":"<classification_id>",
                "name":"ipv4-dscp"
                "project_id":"<project_id>",
                "c_type": "ipv4",
                "description": "",
                "shared":false,
            }
        ],
        "classification_group": [],
        "operator": "AND",
        "shared": false,
    }
}

PUT /v2.0/qos/policies/<policy-uuid>/dscp-marking-rules/<rule-uuid>
{
    "dscp_marking_rule": {
        "dscp_mark": 8,
        "classification_group_id": "<classification_group_id>"
    }
}

Response:
{
    "dscp_marking_rule":{
        "id": "<id>",
        "policy_id": "<policy-uuid>",
        "dscp_mark": 8,
        "classification_group_id": "<classification_group_id>",
        "classification_priority": 0

    }
}

Documentation Impact

User Documentation

Existing Networking Guide will be updated for this feature.

Existing CLI guide will be updated for this feature.

Developer Documentation

Existing QoS devref document will be updated for this feature. A New Document will be drafted for the Neutron Classifier API also.

API Documentation

Existing QoS API documentation will be updated for this feature. A New Document will be drafted for the Neutron Classifier API also.

References

Neutron OVS Agent Support for Baremetal with Smart NIC

Tue, 19 Nov 2019 00:00:00

https://bugs.launchpad.net/neutron/+bug/1785608

This spec describes proposed changes to the Neutron OVS mechanism driver and the Neutron OVS agent to enable a generic, vendor-agnostic, baremetal networking service running on smart NICs, enabling baremetal networking with feature parity to the virtualization use-case.

Problem Description

While Ironic today supports Neutron provisioned network connectivity for baremetal servers through an ML2 mechanism driver, the existing support is based largely on configuration of TORs through vendor-specific mechanism drivers, with limited capabilities.

Proposed Change

There is a wide range of smart/intelligent NICs emerging on the market. These NICs generally incorporate one or more general purpose CPU cores along with data-plane packet processing acceleration, and can efficiently run virtual switches such as OVS, while maintaining the existing interfaces to the SDN layer.

The proposal is to extend the Neutron OVS mechanism driver and Neutron OVS Agent to bind the Neutron port for the baremetal host with smart NIC. This will allow the Neutron OVS Agent to configure the pipeline of the OVS running on the smart NIC and leverage the pipeline features such as: VXLAN, Security Groups and ARP Responder.

This spec is complementary to the Ironic spec [1].

In this proposal, we address two use-cases:

Neutron OVS L2 agent runs locally on the smart NIC.
Neutron OVS L2 agent(s) run remotely and manages the OVS bridges for all the baremetal smart NICs.

Example of smart NIC model:

+---------------------+
|      baremetal      |
| +-----------------+ |
| |  OS Server    | | |
| |               | | |
| |      +A       | | |
| +------|--------+ | |
|        |          | |
| +------|--------+ | |
| |  OS SmartNIC  | | |
| |    +-+B-+     | | |
| |    |OVS |     | | |
| |    +-+C-+     | | |
| +------|--------+ | |
+--------|------------+
         |

A - port on the baremetal host.
B - port that represents the baremetal port in the smart NIC.
C - port that represents to the physical port in the smart NIC.

Ironic creation of Neutron Port:
1. Create Neutron port with new vnic_type called smart-nic.
2. Add local_link_information with the following attributes:
1. smart NIC hostname - the hostname of server/smart NIC where the Neutron OVS agent is running. (required)
2. smart NIC port id - the port name that needs to be plugged to the integration bridge. B in the diagram above(required)
3. smart NIC SSH public key - ssh public key of the smart NIC (required only for remote)
4. smart NIC OVSDB SSL certificate - OVSDB SSL of the OVS in smart NIC (required only remote)
Neutron OVS ML2 Mechanism Driver:

The OVS ML2 will allow binding the smart-nic vnic_type. The rationale for creating new vnic_type and not using the barmetal one is that there is a wide range of mechanism drivers that use hierarchical port binding for configuring TOR switches and we want to allow this to work with smart NICs. for example mechanism_drivers=cisco_nexus,openvswitch

The OVS ML2 mechanism driver will determine if the Neutron OVS Agent runs locally or remotely based on smart NIC configuration passed from ironic as described above.

In case the Neutron OVS L2 agent runs locally on the smart NIC the OVS mechanism driver will locate the Neutron OVS agent by the smart NIC hostname attribute. For the remote case the changes are captured in this neutron spec [2].

Neutron OVS Agent:

Extend the port_update rpc method as following:

def port_update(self, context, **kwargs):
  port = kwargs.get('port')
  # get the port data from cache
  port_data = self.plugin_rpc.remote_resource_cache. \
          get_resource_by_id(resources.PORT, port['id'])
  # if smart-nic port add it to updated_smart_nic_ports with
  # the required information for adding the port to OVSDB
  port_binding = port_data['bindings']
  if port_binding['vnic_type'] == 'smart-nic':
          ifname = smart_nic_port_data['port_binding']['profile'] \
              ['local_link_information'][0]['port_id']
          mac = port_data['mac_address']
          node_uuid = port_data['device_id']
          port_binding['vif_type'] = vif_type
          self.updated_smart_nic_ports.append({
                          'mac': mac,
                          'node_uuid': node_uuid,
                          'iface_id': port['id'],
                          'iface_name': ifname,
                          'vif_type': vif_type})
  self.updated_ports.add(port['id'])

When Neutron processes the ports the Neutron OVS agent will add the smart NIC port(s) to the OVSDB by ovs plugin in os-vif.

Because RPC is not reliable we need to extend the full sync to do the following:

when sync is True we will retrieve all the smart-nic ports for this agent. This requires to add another RPC call.
We will compare the retrieved smart-nic ports for the Neutron server to the existing smart-nic port on the integration bridge.
if the smart-nic port is only on the Neutron server we will add it to the added list in the port_info.
if the smart-nic port is only on the integration bridge we will add it to the removed list in the port_info.

References

Allow user to create default record on port creation from shared network

Wed, 09 Oct 2019 00:00:00

https://bugs.launchpad.net/neutron/+bug/1843218

As discussed in the bug thread above, we could add the feature to allow a user to have default zone configured on a shared network.

Problem Description

On Neutron, when the DNS ML2 plugin is enabled, each user can manage their own network, and configure a dns_domain (i.e. example.com) for that network. The dns_domain (called zone) can be hosted in one and only one tenant. To create a record in a zone, the record’s tenant must be the same as the zone’s tenant. If a user creates a port in that network, it has to be the same tenant as zone to get the record created.

Network (A project) <~~> Port (B project) <==> Zone (B project) <==> Record (B project)

So we need to have all these resources hosted on the same project (port, zone and record) to get this feature working.

As ports are able to be created from several different projects in the case of a shared network, this feature doesn’t work for that case.

Proposed Change

As described in the RFE, the idea is to let the network’s admin user configure keywords in the default dns_domain on their network to allow zone to be different per user or per project.

Here are the accepted keywords :

<project_id>
<project_name>
<user_name>
<user_id>

For instance, configuring <user_name>.<project_id>.example.com. as dns_domain on any network will allow users to have one default zone per user and per project, and then be able to create records.

$ source openrc_admin
$ openstack network set --dns-domain "<user_name>.<project_id>.example.com." shared
$ source openrc_demo
$ openstack zone create --email dnsmaster@defaultzone.com UserName.HisProjectId.example.com.
$ openstack port create --dns-name myport port

Further changes included:

Documentation: api-ref.
neutron-lib adaptation
neutron dns_integration ML2 driver adaptation
dns_integration unit tests
dns_integration integration tests

Planned Impact

No impact expected on upgrades.
No impact expected on configuration.
No breaking changes

References

RFE bug report of this spec: https://bugs.launchpad.net/neutron/+bug/1843218
Neutron Drivers Meeting discussion about the feature validation : http://eavesdrop.openstack.org/meetings/neutron_drivers/2019/neutron_drivers.2019-10-04-14.00.log.html#l-171

Improve Extraroute API

Wed, 14 Aug 2019 00:00:00

https://bugs.launchpad.net/neutron/+bug/1826396

As discussed in an openstack-discuss thread we could improve the extraroute API to better support Neutron API clients, especially Heat.

Problem Description

The first problem is that the current extraroute API does not allow atomic additions/deletions of particular routing table entries. In the current API the routes attribute of a router (containing all routing table entries) must be updated at once. Therefore additions and deletions must be performed on the client side. Therefore multiple clients race to update the routes attribute and updates may get lost.

This problem can be easily (but nondeterministically) reproduced by a simple HOT template as included in the References section below.

$ openstack stack create --wait -t extraroute-concurrent.yaml stack0
$ openstack stack resource list stack0 --filter name=router0 -f value -c physical_resource_id \
  | xargs -r openstack router show -f value -c routes

The second problem perceived by programmatic users of the Neutron API is that the ownership of extra routes cannot be expressed because they don’t have unique identifiers. For example consider two Heat stacks (or any other user of Neutron’s API) needing and therefore creating the same extra route. When one of them is deleted the extra route gets deleted for both unless we have a way to track multiple needs for the same extra route. Since addressing the second problem would involve changing the conceptual abstraction of an extra route this problem is not addressed in this specification.

Proposed Change

This RFE proposes to expose the extraroute functionality of Neutron in a new way beyond the current ‘routes’ attribute of routers.

Introduce a new API extension: extraroutes-atomic

PUT /v2.0/routers/{router_id}/add_extraroutes

{ "router":
  { "routes":
    [ { "destination": "179.24.1.0/24",
        "nexthop": "172.24.3.99" },
      ...
    ]
  }
}

200 OK

{ "router":
  { "id": "1ecae6b8-be64-11e9-98ba-733d5460217b",
    "name": "router1",
    "routes":
    [ { "destination": "179.24.1.0/24",
        "nexthop": "172.24.3.99" },
      ...
    ],
    ...
  }
}

PUT /v2.0/routers/{router_id}/remove_extraroutes

{ "router":
  { "routes":
    [ { "destination": "179.24.1.0/24",
        "nexthop": "172.24.3.99" },
      ...
    ]
  }
}

200 OK

{ "router":
  { "id": "1ecae6b8-be64-11e9-98ba-733d5460217b",
    "name": "router1",
    "routes":
    [ remaining routes ],
    ...
  }
}

Partial failures are not allowed. If the addition or removal of any routing table entry fails then the whole update is reverted.

If a routing table entry to be added (removed) is already present (missing) that is not an error, but considered a successful update.

Handling of duplicate and overlapping routes is not changed from the current behavior. That is exact duplicates are rejected, but overlapping routes are allowed.

Deleting a router cascades to deleting all extra routes of that router.

The DB and OVO layer in Neutron is ready for this change, the routes attribute is is already de-composed to its own table:

mysql> describe routerroutes;
+-------------+-------------+------+-----+---------+-------+
| Field       | Type        | Null | Key | Default | Extra |
+-------------+-------------+------+-----+---------+-------+
| destination | varchar(64) | NO   | PRI | NULL    |       |
| nexthop     | varchar(64) | NO   | PRI | NULL    |       |
| router_id   | varchar(36) | NO   | PRI | NULL    |       |
+-------------+-------------+------+-----+---------+-------+

Further changes included:

Documentation: api-ref.
Unit tests.
Tempest test in neutron-tempest-plugin.
Adapt python-neutronclient.
Adapt openstackclient.
Adapt openstacksdk.
I also hope to improve Heat’s OS::Neutron::Extraroute resource building on this work, but that will be described in its own Heat blueprint.

Deprecations

As soon as the new API is merged direct updates to the ‘routes’ attribute of a router are deprecated, but keeping in line with the long standing Neutron tradition of not making backward incompatible API changes the old extraroutes extension is not removed.

Other Impact

No impact expected on upgrades. No impact expected on configuration. No impact expected on RPC.

Alternatives

Compare and Swap

Neutron has compare-and-swap API update logic. However using that to solve this problem is awkward on the client side (retry until success if the routes attibute change while the client edited it). Plus as the number of racing clients grow use of the compare-and-swap API is going to generate unnecessary load on API as update requests must be thrown away and retried.

Additionally current client code bases do not have support built in for compare-and-swap operations.

For other alternatives please see the review discussion of this specification.

References

RFE bug report of this spec: https://bugs.launchpad.net/neutron/+bug/1826396
Heat story to consume the API proposed here: https://storyboard.openstack.org/#!/story/2005522

extraroute-concurrent.yaml

description: test of extraroute concurrency
heat_template_version: 2015-04-30

resources:

  net0:
    type: OS::Neutron::Net

  subnet0:
    type: OS::Neutron::Subnet
    properties:
      network: { get_resource: net0 }
      cidr: 10.0.0.0/24

  router0:
    type: OS::Neutron::Router

  routerinterface0:
    type: OS::Neutron::RouterInterface
    properties:
      router: { get_resource: router0 }
      subnet: { get_resource: subnet0 }

  extraroute0:
    type: OS::Neutron::ExtraRoute
    properties:
      destination: 10.1.0.0/24
      nexthop: 10.0.0.10
      router_id: { get_resource: router0 }

  extraroute1:
    type: OS::Neutron::ExtraRoute
    properties:
      destination: 10.1.1.0/24
      nexthop: 10.0.0.11
      router_id: { get_resource: router0 }

  extraroute2:
    type: OS::Neutron::ExtraRoute
    properties:
      destination: 10.1.2.0/24
      nexthop: 10.0.0.12
      router_id: { get_resource: router0 }

  extraroute3:
    type: OS::Neutron::ExtraRoute
    properties:
      destination: 10.1.3.0/24
      nexthop: 10.0.0.13
      router_id: { get_resource: router0 }

  extraroute4:
    type: OS::Neutron::ExtraRoute
    properties:
      destination: 10.1.4.0/24
      nexthop: 10.0.0.14
      router_id: { get_resource: router0 }

  extraroute5:
    type: OS::Neutron::ExtraRoute
    properties:
      destination: 10.1.5.0/24
      nexthop: 10.0.0.15
      router_id: { get_resource: router0 }

  extraroute6:
    type: OS::Neutron::ExtraRoute
    properties:
      destination: 10.1.6.0/24
      nexthop: 10.0.0.16
      router_id: { get_resource: router0 }

  extraroute7:
    type: OS::Neutron::ExtraRoute
    properties:
      destination: 10.1.7.0/24
      nexthop: 10.0.0.17
      router_id: { get_resource: router0 }

  extraroute8:
    type: OS::Neutron::ExtraRoute
    properties:
      destination: 10.1.8.0/24
      nexthop: 10.0.0.18
      router_id: { get_resource: router0 }

  extraroute9:
    type: OS::Neutron::ExtraRoute
    properties:
      destination: 10.1.9.0/24
      nexthop: 10.0.0.19
      router_id: { get_resource: router0 }

API Support for Managing Custom Ethertypes

Fri, 28 Jun 2019 00:00:00

Some operators need to allow/deny custom Ethertypes for applications which use their own non-IP traffic (such as for clustering applications). The Security Group API only handles specifying behavior within the IP protocol. With the firewall reference implementation (OVS Firewall) anything other than IPv4 and IPv6 is subject to the default deny. This means OpenStack customers have no options to use OpenStack to permit protocols that use separate ethertypes like InfiniBand and FCoE.

This specification proposes adding to the Security Group API the capability to specify standard security group behaviors (allow, deny) for custom ethertypes, with the aim of implementing these controls in the OVS and OVN firewalls.

Problem Description

The first implementation of security groups in Openstack was based on iptables, which only controls traffic for the IPv4 or IPv6 ethertypes. Since all subsequent implementations have been built on the conceptual foundation of that initial implementation, the behavior of Neutron security groups for traffic that has other ethertypes has been undefined.

In the case of the OVS Firewall driver, the behavior is for traffic with non-IP ethertypes to fall through to the default deny action. A flow could be added by an operator to permit this traffic, but it would be removed the next time the neutron-openvswitch-agent restarted. This is a runtime requirement, so deployment time tools like TripleO or Ansible are unable to handle it.

Use case: Customer is running an application using InfiniBand (ethertype 0x4008) in OpenStack, and that OpenStack transitions from iptables_hybrid to OVS firewall. The Infiniband traffic is blocked by the OVS firewall, and at present the Neutron API offers no methodology to unblock it.

Proposed Change

The best solution is a new API extension that makes Neutron aware of permitted ethertypes and adjust OVS firewall rules appropriately. The current security group APIs take an ethertype option, which is constrained to IPv4 and IPv6; implementing this spec will lift that restriction, so that additional security types can be specified by hexadecimal number. If a non-IP ethertype is specified then the following arguments are ignored: protocol, port_range_max, port_range_min, remote_group_id, remote_ip_prefix.

Control for traffic with a custom ethertype will be all or nothing, a given protocol will be either entirely blocked or entirely allowed.

The ‘ethertype’ field of a SecurityGroupRule object is of type EtherTypeEnumField which is an enum of the VALID_ETHERTYPES constant; this would need to be altered to be a two octet hexadecimal number between 0x0000 and 0xffff. The security group rule database field for ethertypes is a String(40), so no modification is needed. A validator validate_ethertype is suggested, which would also replace use of sg_supported_ethertypes in the security group extension.

References

RFE bug report of this spec: https://bugs.launchpad.net/neutron/+bug/1832758

Toward Convergence of ML2+OVS+DVR and OVN

Fri, 10 May 2019 00:00:00

Launchpad Bug: https://bugs.launchpad.net/neutron/+bug/1828607

In its current state, DVR lacks some functionality that would be of benefit to operators. When looking at ways to address these items, it has become apparent there is substantial overlap in functionality between the desired state of DVR on ML2+OVS and OVN. This spec is meant to outline the areas of overlap and the areas where either OVN or ML2+OVS+DVR are lacking functionality, then describe a plan for converging these 2 solutions.

Problem Description

In its current state, ML2+OVS+DVR has the following gaps:

Control plane scalability when handling both large numbers of ports and when large numbers of nodes participate in the neutron deployment.
Support for distributed ingress/egress when using IPv6 in conjunction with BGP and/or neutron-dynamic-routing.
DVR cannot be cleanly offloaded with smart-NIC technology or user-space tools due to its use of linux namespaces.

In addition to the above shortcomings, DVR would also benefit from the following enhancements:

Support for running without network nodes if an operator chooses
Support for distributed DHCP (local DHCP responder) [1]
Support for distributed SNAT

When looking at enhancing control plane scalability, distributed DHCP, and pushing all forwarding policy into OVS for potential hardware offload, it has been noted that OVN already implements many of these things. OVN also lacks some functionality that ML2+OVS+DVR has such as fully-distributed ingress and egress which makes it possible to route all north-south traffic directly to the appropriate compute, bypassing a central network node.

Proposed Change

To address the issues previously discussed, the ML2+OVS+DVR solution will be merged with the networking-ovn solution. As a desired end state, neutron will have OVN as its reference ML2 plugin for OVS-based environments. This has the following benefits:

Neutron can take advantage of the control plane performance optimizations OVN provides.
Removes the duplication of effort for both core neutron developers and networking-ovn developers, and allows for cross-pollination of ideas for addressing scale, data plane performance, and usability issues in neutron.
Aligns the community of developers supporting neutron and OVN, broadening the potential contributor base for neutron.

For this effort to be successful and minimally disruptive to users, the following things should be considered:

Will users actually observe better control plane scale?
Will users lose support for any use cases?
What is the migration path? How disruptive will migration be? How can it be simplified?

The following sections attempt to answer these questions and outline the procedural and technical aspects of the convergence of ML2+OVS+DVR and networking-ovn.

Feature Gap Analysis

This is a list of some of the currently known gaps between ML2/OVS and OVN. It is not a complete list, but is enough to be used as a starting point for implementors working on closing these gaps. A TODO list for OVN is located at [2].

Fragmentation and Jumbo frames support
- Upstream work to support this in the OVS/OVN code has been completed, additional work is required in the networking-ovn repository to enable it. Change at [3].
Port forwarding
- Currently ML2/OVS supports Port Forwarding in the North/South plane. Specific L4 Ports of the Floating IP can be directed to a specific FixedIP:PortNumber of a VM, so that different services running in a VM can be isolated, and can communicate with external networks easily.
- This is a relatively new extension, support would need to be added to OVN.
  
  One possible way would be to use the OVN native load balancing feature. An OVN load balancer is expressed in the OVN northbound load_balancer table. Normally the VIP and its members are expressed as [4]:
  
  VIP:PORT = MEMBER1:MPORT1, MEMBER2:MPORT2
  
  The same could be extended for port forwarding as:
  
  FIP:PORT = PRIVATE_IP:PRIV_PORT
Security Groups logging API
- ML2/OVS supports a log file where security groups events are logged to be consumed by a security entity. This allows users to have a way to check if an instance is trying to execute restricted operations, or access restricted ports in remote servers.
- This is a relatively new extension, support would need to be added to OVN.
QoS DSCP support
- Currently ML2/OVS supports QoS DSCP tagging and egress bandwidth limiting. Those are basic QoS features that while integrated in the OVS/OVN C core are not integrated (or fully tested) in the neutron OVN mechanism driver.
QoS for Layer 3 IPs
- Currently ML2/OVS supports floating IP and gateway IP bandwidth limiting based on Linux TC. Networking-ovn has a prototype implementation [5] based on the meter of openvswitch [6] utility, which is supported in user space datapath only, or kernel versions 4.15+ [7].
QoS Minimum Bandwidth support
- Currently ML2/OVS supports QoS Minimum Bandwidth limiting, but it is not supported in OVN.
BGP support
- Currently ML2/OVS supports making a tenant subnet routable via BGP, and can announce host routes for both floating and fixed IP addresses.

Performance Impact

Performance tests carried out in late 2018 showed OVN outperforming ML2/OVS in most operations [8]. Only creating networks and listing ports were slower, which is mostly due to the fact that OVN creates an extra port (for metadata) upon network creation, so the amount of ports listed for the same rally task is 2x for the OVN case.

Also, the resources utilization is lower in OVN [9] vs ML2/OVS [10] mainly due to the lack of agents and from not using RPC.

One question that has been raised is OVNs use of Geneve instead of VXLAN, and if that introduces any performance issues. Research showed that this is not a problem. Normally, NICs that support VXLAN also support Geneve hardware offload. Interestingly, even in the cases where they do not, performance was found to be better using Geneve due to other optimizations that Geneve benefits from. More information can be found in Russell Bryant’s blog [11], who did extensive work in this space.

Other Deployer Impact

Migration from ML2/OVS to OVN is specific to the deployment tool being used. Currently there is documentation on migration using TripleO at [12], some of which might be applicable to other deployment tools. Re-writing the information into different sections - one being a generic, high-level description of what steps are required, and the other being deployment-specific notes, would help others when adding more tools.

Developer Impact

Move networking-ovn code to neutron repo and treat OVN as an in-tree driver
- Moving networking-ovn into the neutron repository is the first step toward supporting convergence.
- An etherpad to track information and work items is located at [13].
Distributed ingress/egress for IPv6
- In its current state, DVR supports distributed ingress and egress when using IPv4. When a floating IP is in use, it can either be routed with a /32 host route directly to the fip-xxxxx namespace on the compute node (when using neutron-dynamic-routing) or gratutitous ARP can be sent from the fip-xxxxx namespace for the floating IP.
- When IPv6 is used, all cloud ingress and egress traffic is sent through the centralized router. This means that traffic cannot be steered directly to compute nodes.
- To address this, distributed ingress/egress (AKA “fast-exit”) would be implemented for IPv6. This will allow IPv6 packets arriving in the fip-xxxxx namespace to be routed through the appropriate qrouter-xxxxx namespace on the compute node, bypassing the centralized router hosted on a network node.
- There is an existing RFE for this, see [14].
Support for smart-NIC or user-space offloads
- This involves pushing all DVR forwarding policy into OVS and implementing it via OpenFlow.
- As all forwarding/security policy is pushed directly to OVS, an evaluation of what flow rules are currently being offloaded to hardware should be done if it is possible. If possible, any rules that should be tweaked for optimizing offload potential should be re-written to support offloading.
- There is an existing blueprint for openflow-based DVR in neutron, see [15], but the proof of concept patch has been abandoned [16].
Allow for completely de-centralized operation
- This involves making it possible for operators to run their environments without introducing the concept of network nodes.
- This implies support for distributed SNAT, distributed DHCP, and distributed ingress/egress for IPv4 and IPv6.
- Compatibility with neutron-dynamic-routing for routing directly to tenant networks [17].
- In a completely de-centralized deployment, a router as a logical construct doesn’t need to be thought of as “HA” or “distributed” as it is inherently both by virtue of being instantiated on each compute node.
Distributed SNAT
- This involves allowing SNAT to happen directly on the compute node instead of centralizing it on a network node.
- To provide proper tenant isolation when using the namespace-based implementation of DVR, SNAT would be performed twice (Insert diagram and explain in detail here).
- Open questions:
  - Should SNAT traffic use the FIP gateway IP as the source address, or should it use an IP address specific to the tenant router? Using the FIP gateway IP seems simpler and does cut down on address usage (although subnet service types does make this less of an issue). On the flip side, an address associated with the router does allow for better scrutiny of network traffic as traffic flows can be tied to a specific project and router.
  - Allowing operators to choose whether to use distributed SNAT or not is the best way forward.
- See [18] for some historical context and details.

CLI Impact

No CLI impact is anticipated. All existing CLI commands will continue to be supported.

API Impact and Required Extensions

The extensions supported by OVN plugins vs. ML2+OVS should provide equivalent functionality in terms of data plane features such trunks, routed networks, address scopes, etc. Matching support for every specific extension is not necessary in all cases. For example, OVN provides DVR functionality in its implementation without implementing the API extensions DVR introduces.

Implementation

Assignee(s)

Brian Haley <haleyb.dev@gmail.com>

Work Items

Move networking-ovn into the neutron tree.
Ensure RFE’s have been created for all items in feature gaps and developer impact sections.
Rally jobs for capturing control plane performance with OVN in place.

References

Port binding records, capabilities and events

Thu, 21 Mar 2019 00:00:00

https://bugs.launchpad.net/neutron/+bug/1821058

There are several cases in the Nova/Neutron/os-vif interaction where the knowledge of the neutron core plugin or ML2 driver would be useful to facilitate a more robust handling of guest networking. For example, [3].

This spec aims to enhance the port binding API via the introduction of a new extension to provide the additional information required to enable Nova and os-vif to make more intelligent use of port binding info.

Please remember os-vif was only intended to remove the VIF plugging process from Nova. The hypervisor configuration (e.g.: libvirt XML generation) is still Nova’s responsibility. Although os-vif removes the networking back-end logic related to the plug and unplug process, Nova still needs to handle the information proposed in this spec.

Problem Description

This spec aims to resolve 3 related problems.

Network connectivity

[3] blueprint seeks to enable the use of instances with ip_allocation=none. The port parameter “ip_allocation” was introduced in the [1], to “distinguish the unaddressed port case from the deferred IP allocation case where routed networks is involved” (extracted from [2]).

To do this safely and guarantee the guest will have network connectivity, Nova must ensure that the network backend that bound the port provides L2 connectivity.

For example, in a mixed Calico/SR-IOV deployment, it is valid to use ip_allocation=none for any port that is bound by the SR-IOV ML2 driver, but not for the Calico backend. In this case, the only connectivity Calico provides to a guest is L3, therefore is not possible to spawn a VM without an IP address.

Network isolation

In resolving [4], a new config option was required in os-vif to allow enabling VIF isolation for the ML2/ovs backend. As Nova cannot differentiate between VIF_TYPE=”ovs” that is bound by ML2/odl or ML2/ovs it cannot automatically instruct os-vif to enable or disable VIF isolation for ML2/ovs hosts. If the ML2 driver(s) that bound the port are recorded in the port binding details, Nova and os-vif can use that information to more intelligently enable backend specific code paths without requiring additional network backend specific config options.

Bandwidth scheduling

The network driver information can be used too for the bandwidth scheduling featured implemented in Stein, as pointed out in this [6]. Currently Nova cannot handle the case where the PF is used by one ML2 driver while the VF is managed by the SR-IOV ML2 driver.

Proposed Change

To address the 3 problems stated above, the following additions to the binding:vif_details dictionary in the port object are proposed:

A new connectivity field will be introduced with allowed values of “l2”, “l3” and “legacy”. The “legacy” connectivity value will be the default and is a sentinel to indicate the driver does not support this extension yet and no connectivity info is available. “l2” drivers provide layer 2 connectivity, like for example Linux Bridge, Open vSwitch or SR-IOV. “l3” drivers provide only layer 3 connectivity, like for example Calico [8].
A new bound_drivers field will be added that is a dictionary mapping binding_level to driver name. The driver name will be the stevedore entry point name for the driver which is already used in config files and intended to be stable. A POC is available in [7].

To enable declaration of the connectivity, a new property will be added to the ML2 driver base class which will default to “legacy”. ML2 drivers that inherit from this will override the property.

The bound_drivers field will be used by the OVS os-vif plugin to remove the need for [5]. We can enable this option only when needed.

Examples

ML2/ovs:

binding_details: {
    ...
    "connectivity": "l2",
    "bound_drivers": {"0": "openvswtich"}
}

ML2/odl v1:

binding_details: {
    ...
    "connectivity": "legacy",
    "bound_drivers": {"0": "opendaylight"}
 }

ML2/odl v2:

binding_details: {
    ...
    "connectivity": "l2",
    "bound_drivers": {"0": "opendaylight_v2"}
}

ML2/calico

binding_details: {
    ...
    "connectivity": "l3",
    "bound_drivers": {"0": "calico"}
}

References

Change the segment ID of a VLAN provider network

Mon, 28 Jan 2019 00:00:00

Launchpad bug: https://bugs.launchpad.net/neutron/+bug/1806052

This spec proposes a method to change the segmentation ID of a VLAN-type network, which is the external VLAN ID of the network. The implementation described is limited to only the OVS back-end and to single-segment provider networks.

Problem Description

Currently, Neutron does not allow the segmentation ID parameter of a provider network to change, regardless of the status of the network (active or not) or if ports exist on the network (or not). In order to change the segmentation ID, the administrator needs to delete the network, first un-binding any bound ports, then create a new network with the required segmentation ID, and finally bind the ports back to the network.

Proposed Change

The proposed solution is to be able to change the segmentation ID of a VLAN- type network, using one single command (“openstack network set”) in an atomic operation.

In order to implement this, several changes are needed:

The CLI, to allow to update the segmentation ID in an existing network.
The Neutron server, to allow writing those changes in the database.
The networking back-end. In this spec only OVS is covered and the implementation will be limited to it.

OpenStack Client

Currently, the OpenStack Client allows updating the segmentation ID of an existing network. No changes are needed.

Neutron server

Currently, when a network update message is sent to the Neutron server, if any provider attribute is present in this message, for example ‘segmentation ID’, this command is rejected. The new implementation will allow updating the segmentation ID:

If the segmentation ID is the only provider parameter to be updated. No other provider parameters (network type, physical network) are allowed.
If the network has only one segment. In multi provider networks the server doesn’t know which one of the segments should be updated.
If the provider segment type is ‘vlan’. No other types are allowed.
If all ports bound to this network belong to a compatible network driver. A compatible network driver is one which has implemented this change. In this spec, the OVS agent implementation is described. For example, with OVS, ports with VIF type ‘vhostuser’ and ‘ovs’ are accepted, and ‘unbound’ or ‘binding_failed’ are also allowed because they are not yet bound to any network driver.

If the above conditions are all met, the Neutron server will execute the following actions (in order):

Validate the new segment, to know if the segmentation ID falls inside the minimum and maximum VLAN tags.
Reserve a new provider segment. This will ensure the segmentation ID is not used in this network.
Create a new provider segment in the database.
Release the old provider segment.
Delete the old provider segment from the database.

All these operations will be executed inside a write context, to provide concurrency and enable rollback in case of failure. If any of these database operations fail, including subsequent ones, the context will prevent the old segment from being deleted and the new one will not be created.

Mechanism driver

The agent mechanism driver interface will be extended with a new function. This function will return a list of provider network attributes that can be modified in a network with ports bound to this networking back-end.

By default, this function will return an empty list.

Neutron OVS agent

In the OVS agent, VLAN provider networks can access the provider network outside the host through the physical bridges. Each provider network has a single physical bridge assigned. A physical bridge can host the traffic of multiple provider networks, but each provider network can only send and receive traffic through one physical bridge. There is a N:1 (network:physical bridge) relationship.

Inside the integration bridge, the tenant networks are isolated using virtual networks. These virtual networks have their own VLAN ID mapping, different from the provider networks VLAN ID mapping. Both VLAN maps are related through the “LocalVlanManager”, which is in charge of correlating both mappings. When a packet from a tenant exits the integration bridge to the corresponding physical bridge, a flow changes the VLAN tag from the tenant VLAN ID to the provider segmentation ID. The opposite process happens in the physical bridge:

br_int:
cookie=0xb46613524bd0de42, duration=23646.484s, table=0, n_packets=0,
  n_bytes=0, priority=3,in_port="int-br-phy",dl_vlan=1074
  actions=mod_vlan_vid:1,resubmit(,60)

br_phy:
cookie=0x43209f9280b7fc88, duration=23666.003s, table=0, n_packets=10,
  n_bytes=788, priority=4,in_port="phy-br-phy",dl_vlan=1
  actions=mod_vlan_vid:1074,NORMAL

When a network segmentation ID is updated, the only change needed is to update the new segmentation ID in both flows. This could be done by deleting both flows and then creating them again with the new segmentation ID. For example:

phys_br.reclaim_local_vlan(port=phys_port, lvid=lvid)
phys_br.provision_local_vlan(
    port=phys_port, lvid=lvid,
    segmentation_id=new_segmentation_id,
    distributed=self.enable_distributed_routing)
self.int_br.reclaim_local_vlan(port=int_port,
    segmentation_id=old_segmentation_id)
self.int_br.provision_local_vlan(
    port=int_port, lvid=lvid,
    segmentation_id=new_segmentation_id)

Limitations

As commented, this solution will be available only for:

Single segment networks.
VLAN type networks.
Only OVS (kernel OVS or DPDK OVS) bound ports to this network.

Data Model

No changes.

REST API

A new ML2 RPC call, requested from the agent side, is needed. This new call will provide information about the updated network. With this information the agent will know if the network has changed the segmentation ID (as commented, the “LocalVlanManager” singleton stores this information).

CLI impact

No changes.

Operation Impact

The segmentation ID change implies two actions:

To change the segmentation ID of a Neutron network without unbinding the attached ports. Once this process is done in Neutron, the traffic from those ports will be tagged with the new VLAN ID and transmitted through the provider network.
To change the provider network segmentation. This process is external to Neutron (and OpenStack).

Because those processes cannot be done at the same time, the administrator should decide, depending on the site network infrastructure, the order of execution. Until both the segmentation ID of the Neutron network and the VLAN ID of the external provider network are synchronized, there will be a networking disruption.

References

None.

Network Segment Range Management

Wed, 09 Jan 2019 00:00:00

Launchpad blueprint: https://blueprints.launchpad.net/neutron/+spec/network-segment-range-management

Currently, network segment ranges are configured as an entry in ML2 config file [1] that is statically defined for tenant network allocation and therefore must be managed as part of the host deployment and management. When a normal tenant user creates a network, Neutron assigns the next free segmentation ID (VLAN ID, VNI etc.) from the configured segment ranges. Only an administrator can assign a specific segment ID via the provider extension.

This spec introduces an extension which exposes the segment range management to be administered via the Neutron API. In addition, it introduces the ability for the administrator to control the segment ranges globally or on a per-tenant basis.

Problem Description

Self-service networks, aka tenant networks primarily enable general (non-privileged) projects to manage networks without involving administrators. These networks are entirely virtual and require virtual routers to interact with provider and external networks such as the Internet. In most cases, self-service networks use overlay protocols such as VXLAN or GRE because they can support many more networks than layer-2 segmentation using VLAN tagging (802.1q) which typically requires additional configuration of physical network infrastructure. Nevertheless, in some other cases, OpenStack Neutron also supports the self-service network isolation using VLAN etc. [2].

The current Neutron implementation sets entries in a config file to statically define network segment ranges for tenant network allocation. It does not support another dynamic way (e.g. at the API level) after initialization. Neutron services have to be restarted in order to make any change to the network segment ranges take effect. However, cloud network infrastructure deployments can be complex and non-homogeneous. This rigidity brings about complexity and difficulty to cloud administrators when dealing with varied requirements.

Network segment range management provides an additional level of flexibility at the API level which enables full network orchestration of all the types of tenant networks using standard REST API rather than needing to interact with the host configuration directly. Apart from this, for VLAN tenant networks in particular, network segment range management enables a control over the management of the cloud/network infrastructure for tenant networks. It also facilitates per-tenant assignments (including shared). This enables a cloud administrator to directly control the VLAN tenant segment mapping, and ultimately the underlying L2/L3 path of the tenant traffic without exposing specific network segment range information to the tenant user. The segment allocations for tenant networks will refer to the network segment ranges which are assigned to that tenant by an administrator.

Several sample use cases are demonstrated below.

Use Case I

Enable cloud administrators to create and assign per-tenant network segment ranges. This gives an administrator the privilege to manage the underlying network segment ranges. When a tenant creates a network, it will allocate a segment ID from the segment ranges assigned to the tenant or shared if no tenant specific range available. It helps map the VMs created by that tenant with an explicit set of existing networks, for privacy or dedicated business connection needs. One example illustration is shown below:

                            +------------------+
                            | Physical Network |
     +------------+---------+ Infrastructures  +-------------------+
     |            |         +---------+--------+                   |
     |            |                   |                            |
     |            |                   |                            |
+----+----+  +----+----+         +----+----+                 +----+----+
| Tenant 0|  | Tenant 1|.........| Tenant k|.................| Tenant n|
+----+----+  +----+----+         +----+----+                 +----+----+
     |            |                   |                           |
     |            |                   |                           |
     + range-0    + range-1           + range-k1                  + range-n
                                        range-k2

One cloud is connecting and mapping with a large number of existing l2 physical network infrastructures.
n+1 tenants: Tenant 0, …Tenant k…, Tenant n are available in this cloud.
Each tenant needs to be assigned with one (or more) network segment range(s) respectively (range-0, …range-k1…, …range-k2…, range-n) by the cloud administrator, so that the networks it creates can reach the corresponding dedicated physical network infrastructure it would like to connect to according to different business requirements.

A possible workflow is presented as follows:

Cloud administrator lists all network segment ranges existing:
```
openstack network segment range list
```

If no network segment range created for one target tenant, then cloud administrator could create one for it:

openstack network segment range create
            --name <range_name>
            --shared <shared>
            --project <project_id>
            --network_type <network_type>
            --physical_network <physical_network_name>
            --minimum <range_minimum>
            --maximum <range_maximum>

Now a normal tenant user can create networks in regular way [3]. The network created will automatically allocate a segment ID from the segment ranges assigned to the tenant (per step 3) or shared if no tenant specific range available.

This helps deciding the correct VLAN mapping when multiple tenants exist. It also provides the possibility to dynamically create a network segment range for each tenant, even if it’s not pre-deployed in the host configuration.

Use Case II

In some VLAN tenant network scenarios, it is not uncommon that the existing physical network infrastructures’ segment configurations have been changed. Thus, it is a must for cloud administrators to update the VLAN allocation range for tenant networks at the same time in order to keep aligned with the connection or mapping. Hence offering the ability to dynamically manage segment ranges of self-service networks, as this spec proposes, is imperative for VLAN tenant networks and is definitely a plus for other overlay tenant networks.

A possible workflow is presented as follows:

Cloud administrator lists all network segment ranges existing and identifies the one needs to update:
```
openstack network segment range list
```

Cloud administrator updates a network segment range based on the actual requirement changing:

openstack network segment range set
            --name <range_name>
            --minimum <range_minimum>
            --maximum <range_maximum>
            <network_segment_range-id>

Now a normal tenant user can create networks in regular way [3]. The network created will automatically allocate free segment ID from the updated network segment ranges available.

Proposed Change

To address the above use cases, this spec introduces a new resource called network_segment_ranges together with its implementation.

Currently by default, all pre-configured segment information (e.g. network_vlan_ranges, vni_ranges etc. defined in [1]) is loaded into “ml2_xxx_allocations” DB tables by ML2 type drivers once the Neutron services are up. For all self-service networks’ segmentation ID sync, allocations and releases, they will be based on this information.

All this information will be maintained. The network segment ranges introduced in this spec will augment this initial allocation that is loaded from the configuration file to become part of the shared ranges. It proposes an API way that user with administrative privileges can create and manage various network segment ranges for all network types supported by ML2.

Data Model Impact

The following new table is added as part of the network network management feature:

CREATE TABLE network_segment_ranges (
  id CHAR(36) NOT NULL PRI KEY,
  name VARCHAR(255),
  default BOOL NOT NULL,
  shared BOOL NOT NULL,
  project_id VARCHAR(255) NOT NULL,
  network_type ENUM('vlan', 'vxlan', 'gre', 'geneve') NOT NULL,
  physical_network VARCHAR(64),
  minimum INT,
  maximum INT
);

For different network types, the validation strategies and responses should have the following variants:

VLAN: minimum = 1, maximum = 4094.
VXLAN: minimum = 1, maximum = 2 ** 24 - 1.
GRE: minimum = 1, maximum = 2 ** 32 - 1.
Geneve: minimum = 1, maximum = 2 ** 24 - 1.

Notes: Other validation rules like minimum <= maximum are always applicable and should be paid attention to. Most of the cited above have been supported in neutron_lib.plugins.utils.

Mixin classes to add the network segment range management extension should be provided. The DB operation logic should be handled by the ML2 type manager and the type drivers. For the values present in the existing ML2 configuration options [1] (e.g. ml2_type_vlan, ml2_type_vxlan etc.), they will be loaded as shared and default segment ranges into network_segment_ranges DB in order to provide backward compatibility for initial deployment when this extension is present.

Resource Extension

The following new resource is being introduced and its attributes maps would be like:

NETWORK_TYPE_LIST = [TYPE_VLAN, TYPE_VXLAN, TYPE_GRE, TYPE_GENEVE]
RESOURCE_ATTRIBUTE_MAPS = {
  'network_segment_ranges': {
    'id': {'allow_post': False, 'allow_put': False,
           'validate': {'type:uuid': None},
           'is_visible': True,
           'is_filter': True,
           'is_sort_key': True,
           'primary_key': True},
    'name': {'allow_post': True, 'allow_put': True,
             'validate': {
               'type:string': db_const.NAME_FIELD_SIZE},
             'default': '', 'is_visible': True, 'is_filter': True,
             'is_sort_key': True},
    'default': {'allow_post': False, 'allow_put': False,
                'convert_to': converters.convert_to_boolean,
                'default': False,
                'is_visible': True},
    'shared': {'allow_post': True, 'allow_put': False,
               'convert_to': converters.convert_to_boolean,
               'is_visible': True, 'default': True},
    'project_id': {'allow_post': True, 'allow_put': False,
                   'validate': {
                   'type:string': db_const.PROJECT_ID_FIELD_SIZE},
                   'required_by_policy': True,
                   'is_filter': True,
                   'is_sort_key': True,
                   'is_visible': True},
    'network_type': {'allow_post': True, 'allow_put': False,
                     'validate': {'type:values': NETWORK_TYPE_LIST},
                     'default': constants.ATTR_NOT_SPECIFIED,
                     'is_filter': True,
                     'is_visible': True},
    'physical_network': {'allow_post': True, 'allow_put': False,
                         'validate': {
                           'type:string': PHYSICAL_NETWORK_MAX_LEN},
                         'default': constants.ATTR_NOT_SPECIFIED,
                         'is_filter': True,
                         'is_visible': True},
    'minimum': {'allow_post': True, 'allow_put': True,
                'convert_to': converters.convert_to_int, 'is_visible': True},
    'maximum': {'allow_post': True, 'allow_put': True,
                'convert_to': converters.convert_to_int, 'is_visible': True},
    'used': {'allow_post': False, 'allow_put': False,
             'is_visible': True},
    'available': {'allow_post': False, 'allow_put': False,
                  'convert_to': attr.convert_none_to_empty_list,
                  'is_visible': True},
  },
}

REST API Impact

The idea is to add a new resource extension with the below defined attributes. Resource extension network_segment_ranges:

Attribute Name	Type	Req	CRUD	Default Value	Description
id	String	N/A	R	Generated	Identifier of network segment range
name	String	No	CRU	‘’	Name of network segment range
default	Bool	No	R	False	Default network segment range that is loaded from the host ML2 config file [1]
shared	Bool	Yes	CR	True	Shared with other projects
project_id	String	No	CR	Current project_id	Owner of network range. Optional when shared is True.
network_type	Enum	Yes	CR	None	VLAN, VxLAN, GRE Geneve
physical_network	String	No	CR	None	Optional. Only applicable for VLAN.
minimum	INT	Yes	CRU	None	Floor integer of the segment range
maximum	INT	Yes	CRU	None	Ceiling integer of the segment range
used	Dict	No	R	{}	Mapping of which segmentation ID in the range is used by which tenant
available	List	No	R	[]	List of available segmentation IDs in this segment range

To specify a range with single item, min equals to max can do the trick. For discrete segment ranges of one given network type, they are represented as several ones, each with a min and a max.

The following network segment range management Rest APIs will be provided in line with the new resources previously introduced:

List all network segment ranges. GET /v2.0/network_segment_ranges

GET /v2.0/network_segment_ranges
Accept: application/json
{
  "network_segment_ranges": [
    {
      "id": "d23abc8d-2991-4a55-ba98-2aaea84cc72f",
      "name": "network_segment_range_physnet1",
      "default": False,
      "shared": False,
      "project_id": "45977fa2dbd7482098dd68d0d8970117",
      "network_type": "vlan",
      "physical_network": "physnet1",
      "minimum": 100,
      "maximum": 105,
      "used": {"100": "07ac1127ee9647d48ce2626867104a13",
               "101": "d4fa62aa47d340d98d076801aa7e6ec4"},
      "available": [102, 103, 104, 105],
    }
  ]
}

List a network segment range information. GET /v2.0/network_segment_ranges/<network_segment_range-id>

GET /v2.0/network_segment_ranges/d23abc8d-2991-4a55-ba98-2aaea84cc72f
Accept: application/json
{
  "network_segment_range": {
    "id": "d23abc8d-2991-4a55-ba98-2aaea84cc72f",
    "name": "network_segment_range_physnet1",
    "default": False,
    "shared": False,
    "project_id": "45977fa2dbd7482098dd68d0d8970117",
    "network_type": "vlan",
    "physical_network": "physnet1",
    "minimum": 100,
    "maximum": 105,
    "used": {"100": "07ac1127ee9647d48ce2626867104a13",
             "101": "d4fa62aa47d340d98d076801aa7e6ec4"},
    "available": [102, 103, 104, 105],
  }
}

Create a network segment range for a given tenant. POST /v2.0/network_segment_ranges

POST /v2.0/network_segment_ranges
Accept: application/json
{
  "network_segment_range": {
    "name": "network_segment_range_physnet1",
    "shared": False,
    "project_id": "45977fa2dbd7482098dd68d0d8970117",
    "network_type": "vlan",
    "physical_network": "physnet1",
    "minimum": 100,
    "maximum": 200,
  }
}

Delete a network segment range by id. DELETE /v2.0/network_segment_ranges/<network_segment_range-id>

Normal Response Code: 204

Error Response Codes: Unauthorized (401), Not Found (404), Conflict (409). The Conflict error response is returned when an operation is performed while the network segment range has resource allocated within it.

This operation does not require a request body.

This operation does not return a response body.

DELETE /v2.0/network_segment_ranges/d23abc8d-2991-4a55-ba98-2aaea84cc72f
Accept: application/json

Update a network segment range with given data. PUT /v2.0/network_segment_ranges/<network_segment_range-id>

PUT /v2.0/network_segment_ranges/d23abc8d-2991-4a55-ba98-2aaea84cc72f
Accept: application/json
{
  "network_segment_range": {
    "minimum": 200,
    "maximum": 300,
  }
}

Command Line Client Impact

Openstack Client would add network segment range management related CLIs. They should be admin only CLI commands. For example:

openstack network segment range list
            [--used | --not-used]
            [--available | --not-available]

openstack network segment range show <network_segment_range-id>

openstack network segment range create
            --shared <shared>
            --network_type <network_type>
            --minimum <range_minimum>
            --maximum <range_maximum>
            [--name <range_name>]
            [--project_id <project_id>]
            [--physical_network <physical_network>]

Notes: Arguments --name, --project_id are optional, the project_id can be
       taken from the context if not given; --physical_network is optional
       and only applicable for VLAN. All the other parameters should be
       required.

openstack network segment range set
            [--name <range_name>]
            [--minimum <range_minimum>]
            [--maximum <range_maximum>]
            <network_segment_range-id>

openstack network segment range delete <network_segment_range-id>

Other Impact

ML2 plugin, plugin manager and type drivers will need to be refined and added with several new methods correspondingly in order to support this feature.
When this extension is loaded, the Neutron server will populate the proposed network_segment_ranges DB table with the ranges defined within the existing ML2 configuration as default and shared ranges. These default ranges are read-only and an administrator can make per-tenant segment range assignment based on this information. When a Neutron server starts/restarts, the default segment ranges will be reloaded and be visible to all servers. Interactions from the REST APIs will always operate based on the segment ranges defined within the database.
Validation work is needed for quite a few cases, including but not limited to:
- Admin privilege should always be checked before performing any network segment range operation cited previously.
- When deleting one network segment range, the operation should be rejected if one of the network segments is in use.
- It is allowed to update one network segment range if it does not impact the in-use segment allocated within the range (e.g. enlarging that range). Otherwise, we should fail the updating.
- We’re maintaining the consistency by “ml2_xxx_allocations” DB tables and relying on them to do the eventual validation and the underlying segment allocation. This means network segment ranges can be configured before agents are actually mapped to specific physical network mappings.
Theoretically, multiple network segment ranges can be created for one tenant (while one network segment range cannot be owned by several tenants). If a tenant has more than one segment range configured, it would pick up the next free segmentation ID (VLAN ID, VNI etc.) from all its owned network segment ranges.

Backwards compatibility comes from having the default behavior of segment ranges being assigned as a shared resource to tenants. If both of shared and specified segment range resources are exposed to a tenant, the specified should override the shared.
The extension will be optional (an option in [4] with functionality disabled by default should be added), in which case the existing ML2 configuration options will be applicable.

Other End User Impact

Users with admin privileges will be able to dynamically manage network segment ranges for which supports segmentation, all tenants and tenant networks. If no dynamic network segment range is created for a given tenant, or the feature is disabled due to the backwards compatibility consideration, there will be no impact to end users.

Other Deployer Impact

Deployers should have an option available to enable or disable this functionality so that they can continue to use the configuration file as before. They also need to be strongly warned to update their operational documentation to ensure that the new network segment information is managed using the correct facility. If the feature is disabled, nothing at the deployment level would be impacted.

Performance Impact

Performance testing must be conducted to see what is the overhead of enabling this feature, of course if the feature is disabled no performance impact should be noticed.

Implementation

Assignee(s)

Kailun Qin <kailun.qin@intel.com>

Work Items

Adjust the DB model and add the new table.
Extend current API.
Modify type drivers and all related references.
Add related tests.
Add CLI openstackclient.
Documentation.

Dependencies

None

Testing

Unit tests, functional tests, API tests and scenario tests are necessary.

Documentation Impact

The Neutron API reference will need to be updated.

References

Registering RouterInfo by L3 extension API

Tue, 18 Dec 2018 00:00:00

Launchpad blueprint: https://blueprints.launchpad.net/neutron/+spec/router-factory-with-l3-extension

Currently, most plugin implementations related to L3 override the L3NATAgent class itself for their own logic since there is no proper interface to extend the RouterInfo class. This adds unnecessary complexity for developers who just want to extend the agent mechanism instead of the whole RPC related to L3 functionalities.

This spec introduces the RouterFactory class which acts on the factory for creating the RouterInfo class, and adds a new parameter to the L3 agent extension API which enables it to dynamically register RouterInfo to the factory. Now plugin developers can use the new extension API for their own specific router.

Problem Description

Current L3 agent implementation in Neutron consists of two parts in general. One is to implement an RPC Plugin API from Neutron server, and the other is to create ports, namespaces, and iptables rules using the data obtained from the RPC API on the server. To be more specific, the former is the L3NATAgent class and the latter is the RouterInfo class.

The problem is that the two parts mentioned are now tightly coupled which means there is no clear way to extend each part individually. A lot of projects related to Neutron called networking-* [1], [2], [3] are making new L3 agent classes on their own by extending the L3NATAgent class even though they did not modify the RPC mechanism but only changed the RouterInfo mechanism running on their server.

Also, the current RouterInfo class does not have an abstract interface which makes it harder to extend the class for plugin developers. They have to find what functions and variables in RouterInfo are externally used in the L3 agent to extend the RouterInfo behaviors.

Proposed Change

Today, RouterInfo is extended in several ways according to specific router features such as distributed, ha, and distributed + ha. This document proposes changing the L3 agent to have a new class called RouterFactory which has several pre-registered classes to extend the RouterInfo class with certain features. When it comes to creating an actual RouterInfo instance, the L3 agent create a new instance from the RouterFactory following the features of the router. There is no functional change for existing code.

L3AgentExtensionAPI now has a new parameter router_factory and a new function register_router. A new abstract class called BaseRouterInfo will be added. It will declare interfaces that are currently used externally.

An L3 extension can register their own RouterInfo class which implements BaseRouterInfo using the register_router API which has two parameters.

router: RouterInfo declared in extension which overrides the one pre-registered at RouterFactory
features: features of RouterInfo. Currently it should be one of the below.

Features should be a list of strings describing router characteristics, and the ordering does not matter since it is interpreted as a set internally. (['ha', 'distributed'] and ['distributed', 'ha'] are the same)
- []: No feature. (e.g. LegacyRouter)
- ['distributed']: Distributed router. (e.g. DvrEdgeRouter,
  DvrLocalRouter)
- ['ha']: HA router. (e.g. HaRouter)
- ['ha', 'distributed']: Distributed HA router. (e.g.
  DvrEdgeHaRouter, DvrLocalRouter)
Note that a router with the feature of ['ha', 'distributed'] can be DvrLocalRouter when L3 agent mode is not dvr_snat [4].

L3 extensions can override the RouterInfo class implemented in the Neutron codebase when it is initialized using the initialize function.

Implementation

Assignee(s)

Yang Youseok <ileixe@gmail.com>

Work Items

Add RouterInfo registration in L3AgentExensionAPI and pre-register existing RouterInfo to L3NATAgent. [5]
Add new exception named RouterNotFoundInRouterFactory to neutron_lib. [6]

References

QoS minimum bandwidth allocation in Placement API

Tue, 13 Nov 2018 00:00:00

https://bugs.launchpad.net/neutron/+bug/1578989

This spec describes how to model, from Neutron, new resource providers in Placement API to describe bandwidth allocation.

Problem Description

Currently there are several parameters, quantitative and qualitative, that define a Nova server and are used to select the correct host and network backend devices to run it. Network bandwidth is not yet among these parameters. This allows situations where a physical network device could be oversubscribed.

This spec addresses managing the bandwidth on the first physical device ie. the pyhsical interface closest to the nova server. Managing bandwidth further away, for example on the backplane of a Top-Of-Rack switch or end-to-end, is out of scope here.

Guaranteeing bandwidth generally involves enforcement of constraints on two levels.

placement: Avoiding oversubscription when placing (scheduling) nova servers and their ports.
data plane: Enforcing the guarantee on the physical network devices.

This spec addresses placement enforcement only. (Data plane enforcement is covered by [4].) However the design must respect that users are interested in the joint use of these enforcements.

Since the placement enforcement itself is a Nova-Neutron cross-project feature this spec is meant to be read, commented and maintained together with its Nova counterpart: Network bandwidth resource provider [2].

This spec is based on the approved Neutron spec Add a spec for strict minimum bandwidth support [3]. The aim of the current spec is not to redefine what is already approved in [3], but to specify how it is going to be implemented in Neutron.

Use Cases

The most straightforward use case is when a user, who has paid for a premium service that guarantees a minimum network bandwidth, wants to spawn a Nova server. The scheduler needs to know how much bandwidth is already in use in each physical network device in each compute host and how much bandwidth the user is requesting.

Data plane only enforcement was merged in Newton for SR-IOV egress (see Newton Release Notes [6]).

Placement only enforcement may be a viable feature for users able to control all traffic (e.g. in a single tenant private cloud). Such placement only enforcement can be also used together with the bandwidth limit rule. The admin can set two rules in a QoS policy, both with the same bandwidth values and then each server on such chosen compute host will be able to use at most as much bandwidth as it has guaranteed.

Proposed Change

The user must be able to express the resource needs of a port.
1. Extend qos_minimum_bandwidth_rule with ingress direction.
  
  Unlike enforcement in the data plane, Placement can handle both directions by the same effort.
2. Mark qos_minimum_bandwidth_rule as supported QoS policy rule for each existing QoS driver.
  
  Placement enforcement is orthogonal to backend mechanisms. A user can have placement enforcement for drivers not having data plane enforcement (yet).
Due to the fact that we exposed (and likely want to expose further) partial results of this development effort to end users, the meaning of a qos_minimum_bandwidth_rule depends on OpenStack version, Neutron backend driver and the rule’s direction. A rule may be enforced by placement and/or on the data plane. Therefore we must document, next to the already existing support matrix in the QoS devref [10], which combinations of versions, drivers, rule directions and (placement and/or data plane) enforcements are supported.

Since Neutron’s choice of backend is hidden from the cloud user, the deployer must also clearly document which subset of the above support matrix is applicable for a cloud user in a particular deployment.
Neutron must convey the resource needs of a port to Nova.

Extend port with attribute resource_request according to section ‘How required bandwidth for a Neutron port is modeled’ below. This attribute is computed, read-only and admin-only.

Information available at port create time (ie. before the port is bound) must be sufficient to generate the resource_request attribute.

The port extension must be decoupled from ML2 and kept in the QoS service plugin. One way to do that is to use neutron.db._resource_extend like trunk_details uses it.
Neutron must populate the Placement DB with the available resources.

Report information on available resources to the Placement service using the Placement API [1]. That is information about the physical network devices, their physnets, available bandwidth and supported VNIC types.

The cloud admin must be able to control (by configuration) what is reported to Placement. To ease the configuration work autodiscovery of networking devices may be employed, but the admin must be able to override its results.

Which devices and parameters will be tracked

Even inside a compute host many networking topologies are possible. For example:

OVS agent: physical network - OVS bridge - single physical NIC (or a bond): 1-to-1 mapping between physical network and physical interface
SR-IOV agent: physical network - one or more PFs: 1-to-n mapping between physical network and physical interface(s) (See Networking Guide: SR-IOV [7].)

Each Neutron agent (Open vSwitch, Linux Bridge, SR-IOV) has a configuration parameter to map a physical network with one or more provider interfaces (SR-IOV) or a bridge connected to a provider interface (Open vSwitch or Linux Bridge).

OVS agent configuration:

[ovs]
# bridge_mappings as it exists already.
bridge_mappings = physnet0:br0,physnet1:br1

# Each right hand side value in bridge_mappings:
#   * will have a corresponding resource provider created in Placement
#   * must be listed as a key in resource_provider_bandwidths

resource_provider_bandwidths = br0:EGRESS:INGRESS,br1:EGRESS:INGRESS

# Examples:

# Resource provider created, no inventory reported.
resource_provider_bandwidths = br0
resource_provider_bandwidths = br0::

# Report only egress inventory in kbps (same unit as in the QoS rule API).
resource_provider_bandwidths = br0:1000000:

# Report egress and ingress inventories in kbps.
resource_provider_bandwidths = br0:1000000:1000000

# Later we may introduce auto-discovery (for example via ethtool).
# We reserve the option to make auto-discovery the default behavior
# when it is implemented.
resource_provider_bandwidths = br0:auto:auto

SR-IOV agent configuration:

[sriov_nic]
physical_device_mappings = physnet0:eth0,physnet0:eth1,physnet1:eth2

resource_provider_bandwidths = eth0:EGRESS:INGRESS,eth1:EGRESS:INGRESS

How required bandwidth for a Neutron port is modeled

The required minimum network bandwidth needed for a port is modeled defining a QoS policy along with one or more QoS minimum bandwidth rules [4]. However neither Nova nor Placement know about any QoS policy rule directly. Neutron translates the resource needs of a port into a standard port attribute describing the needed resource classes, amounts and traits.

In this spec we assume that a single port requests resources from a single RP. Later we may allow a port to request resources from multiple RPs.

The resources needed by a port are expressed via the new attribute resource_request extending the port as follows.

Figure: resource_request in the port

{"port": {
    "status": "ACTIVE",
    "name": "port0",
    ...
    "device_id": "5e3898d7-11be-483e-9732-b2f5eccd2b2e",
    "resource_request": {
        "resources": {
            "NET_BW_IGR_KILOBIT_PER_SEC": 1000,
            "NET_BW_EGR_KILOBIT_PER_SEC": 1000 },
        "required": ["CUSTOM_PHYSNET_NET0", "CUSTOM_VNIC_TYPE_NORMAL"]}
}}

The resource_request port attribute will be implemented by a new API extension named port-resource-request.

If a nova server boot request has a port defined and this port has a resource_request attribute, that means the Placement Service must enforce the minimum bandwidth requirements.

A host will satisfy the requirements if it has a physical network interface RP with the following properties. First, inventory of the new NET_BANDWIDTH_* resource classes and there is enough bandwidth available as shown in the ‘Networking RP model’ section. If a host doesn’t have an inventory of the requested network bandwidth resource class(es), it won’t be a candidate for the scheduler. Second, the physical network interface RP must have all the traits associated with it as listed in the required field of the resource_request attribute.

We propose two kinds of custom traits. First to express and request support for certain vnic_types. This trait uses prefix CUSTOM_VNIC_TYPE_. The vnic_type is then appended in all upper case. For example:

CUSTOM_VNIC_TYPE_NORMAL
CUSTOM_VNIC_TYPE_DIRECT

Second we’ll use traits to decide if a segment of a network (identified by its physnet name) is connected on the compute host considered in scheduling. This trait uses prefix CUSTOM_PHYSNET_. The physnet name is then appended in all upper case, any characters prohibited in traits must be replaced with underscores.

For example:

CUSTOM_PHYSNET_PUBLIC
CUSTOM_PHYSNET_NET1

If a nova server boot request has a network defined and this network has a qos_minimum_bandwidth_rule, that boot request is going to fail as documented in the ‘Scoping’ section of [2] until Nova is refactored to create the port earlier (that is before scheduling). See also SPEC: Prep work for Network aware scheduling (Pike) [11].

For multi-segment Neutron networks each static segment’s physnet trait must be included in the resource_request attribute in a format that we can only specify after Placement supports request matching logic of any(traits). See any-traits-in-allocation_candidates-query [9].

Reporting Available Resources

Some details of reporting are described in the following sections of [2]:

Neutron agent first start
Neutron agent restart
Finding the compute RP

Details internal to Neutron are the following:

Networking RP model

We made the following assumptions:

Neutron supports the multi-provider extension therefore a single logical network might map to more than one physnet. Physnets of non-dynamic segments are known before port binding. For the sake of simplicity in this spec we assume each segment directly connected to a physical interface with a mimimum bandwidth guarantee is a non-dynamic segment. Therefore those physnets can be included in the port’s resource_request as traits.
Multiple SRIOV physical functions (PFs) can give access to the same physnet on a given compute but those PFs always implement the same vnic_type. This means that using only physnet traits in Placement and in the port’s resource request does not select one PF unambiguously but it is not a problem as both PFs are equivalent from resource allocation perspective.
Two different backends (e.g. SRIOV and OVS) can give access to the same physnet on the same compute host. In this case Neutron selects the backend based on vnic_type of the Neutron port specified by the end user during port create. Therefore physical device selection during scheduling should consider the vnic_type of the port as well. This can be done via the vnic_type based traits previously described.
Two different backends (e.g. OVS and LinuxBridge) can give access to the same physnet on the same compute host while they are also implementing the same vnic_type (e.g. normal). In this case the backend selection in Neutron is done according to the order of mechanism_drivers configured by the admin in neutron.conf. Therefore physical device selection during scheduling should consider the same preference order. As the backend order is just a preference but not a hard rule supporting this behavior is out of scope in this spec but in theory it can be done by a new weigher in nova-scheduler.

Based on these assumptions, Neutron will construct in Placement a RP tree as follows:

Figure: networking RP model

Compute RP (name=hostname)
 +
 |
 +-------+Network agent RP (for OVS agent), uuid = agent_uuid
 |          inventory: # later, model number of OVS ports here
 |             +
 |             |
 |             +------+Physical network interface RP,
 |             |       uuid = uuid5(hostname:br0)
 |             |         traits: CUSTOM_PHYSNET_1, CUSTOM_VNIC_TYPE_NORMAL
 |             |         inventory:
 |             |         {NET_BW_IGR_KILOBIT_PER_SEC: 10000,
 |             |          NET_BW_EGR_KILOBIT_PER_SEC: 10000}
 |             |
 |             +------+Physical network interface RP,
 |                     uuid = uuid5(hostname:br1)
 |                       traits: CUSTOM_PHYSNET_2, CUSTOM_VNIC_TYPE_NORMAL
 |                       inventory:
 |                       {NET_BW_IGR_KILOBIT_PER_SEC: 10000,
 |                        NET_BW_EGR_KILOBIT_PER_SEC: 10000}
 |
 +-------+Network agent RP (for LinuxBridge agent), uuid = agent_uuid
 |             +
 |             |
 |             +------+Physical network interface RP,
 |                     uuid = uuid5(hostname:virbr0)
 |                       traits: CUSTOM_PHYSNET_1, CUSTOM_VNIC_TYPE_NORMAL
 |                       inventory:
 |                       {NET_BW_IGR_KILOBIT_PER_SEC: 10000,
 |                        NET_BW_EGR_KILOBIT_PER_SEC: 10000}
 |
 +-------+Network agent RP (for SRIOV agent), uuid = agent_uuid
               +
               |
               +------+Physical network interface RP,
               |       uuid = uuid5(hostname:eth0)
               |         traits: CUSTOM_PHYSNET_2, CUSTOM_VNIC_TYPE_DIRECT
               |         inventory:
               |         {VF: 8, # VF resource is out of scope
               |          NET_BW_IGR_KILOBIT_PER_SEC: 10000,
               |          NET_BW_EGR_KILOBIT_PER_SEC: 10000}
               |
               +------+Physical network interface RP,
               |       uuid = uuid5(hostname:eth1)
               |         traits: CUSTOM_PHYSNET_2, CUSTOM_VNIC_TYPE_DIRECT
               |         inventory:
               |         {VF: 8, # VF resource is out of scope
               |          NET_BW_IGR_KILOBIT_PER_SEC: 10000,
               |          NET_BW_EGR_KILOBIT_PER_SEC: 10000}
               |
               +------+Physical network interface RP,
                       uuid = uuid5(hostname:eth2)
                         traits: CUSTOM_PHYSNET_3, CUSTOM_VNIC_TYPE_DIRECT
                         inventory:
                         {VF: 8, # VF resource is out of scope
                          NET_BW_IGR_KILOBIT_PER_SEC: 10000,
                          NET_BW_EGR_KILOBIT_PER_SEC: 10000}

Custom traits will be used to indicate which physical network a given Physical network interface RP is connected to, as previously described.

Custom traits will be used to indicate which vnic_type a backend supports so different backend technologies can be distinguished, as previously described.

The current purpose of agent RPs is to allow us detecting the deletion of an RP. Later we may also start to model agent-level resources and capabilities.

Report directly or indirectly

Considering only agent-based MechanismDrivers we have two options:

direct: The agent reports resource providers, traits and inventories directly to the Placement API.
indirect: The agent reports resource providers, traits and inventories to Neutron-server which in turn reports the information to the Placement API.

Both have pros and cons. Direct reporting involves fewer components therefore it’s more efficient and more reliable. On the other hand freshness of the resource information may be important information in itself. Nova has the compute heartbeat mechanism to ensure scheduler considers the live Placement records only. In case freshness of Neutron resource information is needed the only practical way is to build on the Neutron-agent heartbeat mechanism. Otherwise the reporting and heartbeat mechanism would take different paths. If resource information is reported through the agent heartbeat mechanism then freshness of resource information is known by Neutron-server and other components (for example a nova scheduler filter) could query it from Neutron-server.

When Placement and nova-scheduler choose to allocate the requested bandwidth on a particular network resource provider (that represents a physical network interface) that choice has its implications on:

Neutron-server’s choice of a Neutron backend for a port. (vif_type, vif_details)
Neutron-agent’s choice of a physical network interface. (Only in some cases like when multiple SR-IOV PFs back one physnet.)

The later choices (of neutron-server and neutron-agent) must respect the first (in the allocation), otherwise resources could be used somewhere else than allocated.

The choice in the allocation can be easily communicated to Neutron using the chosen network resource provider UUID if this UUID is known to both Neutron-server and Neutron-agent. If available resources are reported directly from Neutron-agent to Placement then Neutron-server may not know about resource provider UUIDs. Therefore indirect reporting is recommended.

Even when reporting indirectly we must keep the (Neutron reported part of the) content of the Placement DB under the as direct as possible control of Neutron agents. It is best to keep Neutron-server in a basically proxy-like role.

Content and format of resource information reported (from agent to server)

We propose to extend the configurations field of the agent heartbeat RPC message.

Beyond the agent’s hardcoded set of supported vnic_types the following agent configuration options are the input to extend the heartbeat message:

bridge_mappings or physical_device_mappings
resource_provider_bandwidths
If needed further options controlling inventory attributes like: allocation_ratio, min_unit, max_unit, step_size, reserved

Based on the input above the configurations dictionary of the heartbeat message shall be extended with the following keys:

(custom) traits
resource_providers
resource_provider_inventories
resource_provider_traits

The values must be (re-)evaluated after the agent configuration is (re-)read. Each heartbeat message shall contain all items known by the agent at that time. The extension of the configurations fields intentionally mirrors the structure of the placement API (and does not directly mirror the agent configuration format, though can be derived from it). The values of these fields shall be formatted so they can be readily pasted into requests sent to the Placement API.

Agent resource providers shall be identified by their already existing Neutron agent UUIDs as shown in the ‘Networking RP model’ section above.

Neutron-agents shall generate UUIDs for physical network interface resource providers. Version 5 (name-based) UUIDs should be used by hashing names like HOSTNAME:OVS-BRIDGE-NAME for ovs-agent and HOSTNAME:PF-NAME for sriov-agent since this way the UUIDs will be stable through an agent restart.

Please note that the agent heartbeat message contains traits and their associations with resource providers, but there’s no traits directly listed in the agent configurations. This is possible because both physnet and vnic_type traits we’ll use can be inferred from already known pieces of information.

Synchronization of resource information reported

Ideally Neutron-agent, Neutron-server and Placement must have the same view of resources. We propose the following synchronization mechanism between Neutron-server and Placement:

Each time Neutron-server learns of a new agent it diffs the heartbeat message (for traits, providers, inventories and trait associations) with all objects found in Placement under the agent RP. It creates the objects missing from Placement. It deletes those missing from the heartbeat. It updates the objects whose attributes are different in Placement and the heartbeat.

At subsequent heartbeats received Neutron-server diffs the new and the previous heartbeats. If nothing changed no Placement request is sent. If a change in heartbeats is detected Neutron sends the appropriate Placement request based on the diff of heartbeats using the last seen Placement generation number. If the Placement request is successful Neutron stores the new generation number. If the request fails with generation conflict Neutron falls back to diffing between Placement and the heartbeat.

Progress or block until the Compute host RP is created

Neutron-server cannot progress to report resource information until the relevant Nova-compute host RP is created. (The reason being the Nova-compute host RP UUID is unpredictable to Neutron.) We believe that while waiting for the Nova-compute host RP a Neutron-server can progress with its other functions.

Port binding changes

The order of relevant operations are the following:

Placement DB is populated with both compute and network resource information.
Triggered by a nova server boot Placement selects a list of candidates.
Scheduler chooses exactly one candidate and allocates it in a single transaction. (In some complex nova server move cases the conductor may allocate but that’s unimportant here.)
Neutron binds the port.

In steps (2) and (3) the selection includes the choice of RPs representing network backend objects (beyond the obvious choice of compute host). This naturally conflicts with Neutron’s current port binding mechanism.

To solve the conflict we must make sure:

Placement must produce candidates whose ports later can be bound by Neutron. (At least with roughly the same probability as scheduler is able today.)
The choices made by Placement and made by Neutron port binding must be the same. Therefore the selection must be coordinated.

If more than one Neutron backend can satisfy the resource requirements of a port on the same host then it cannot happen that Placement chooses one, but Neutron binds another.

One way to do that is for Neutron-server to read (from Placement) the allocation of the port currently being bound and let it influence the binding. However this introduces a slow remote call in the middle of port binding therefore it is not recommended.

Another way is to pass down part of the allocation record in the call/message chain leading to the port binding PUT request. In the port binding PUT request use the binding_profile attribute. That way we would not need a new remote call, just to add an argument/payload to already existing calls/messages.

The Nova spec ([2]) proposes that the resources requested for a port are included in a numbered request group (see Granular Resource Request Syntax [8]). A numbered request group is always matched by a single resource. In general Neutron needs to know which resource provider matched the numbered request group of the port.

To express the choice made by placement and nova-scheduler we propose to add an allocation entry to binding_profile:
```
{
  "name": "port with minimum bw being bound",
  "id": ...,
  "network_id": ...,
  "binding_profile": { "allocation": RP_UUID }
}
```
If a port has the resource_request attribute then it must be bound with binding_profile.allocation supplied. Otherwise binding_profile.allocation must not be present.

Usually ML2 port binding tries the mechanism drivers in their configuration order until one succeeds to set the binding. However if a port being bound has binding_profile.allocation then only a single mechanism driver can be tried - the one implicitly identified by RP_UUID.

In case of hierarchical port binding binding_profile.allocation is meant to drive the binding only on the binding level that represents the closest physical interface to the nova server.

Out of Scope

Minimum bandwidth rule update: When a minimum bandwidth rule is updated, the ML2 plugin will list the bound ports with this QoS policy and rule attached and will update the Allocation value. The consumer_id of each Allocation is the device_id of the port. This is out of scope in this spec and should be done during the work related to os-vif migration tasks [5].

Trunk port: Subports of a trunk port are unknown to Nova. Allocating resources for subports is a task for Neutron only. This is out of scope too.

Testing

Unit tests.
Functional tests.
- Agent-server interactions.
Fullstack.
- Handling agent failure cases.
Tempest API tests.
- Port API extended with resource_request.
- Extensions of binding_profile.
Tempest scenario tests.
- End-to-end feature test.

In test frameworks where we cannot depend on Nova we can mock it away by:

Creating and binding the port as Nova would have done it, including.
- Setting its binding_profile.
- Setting its binding_host_id as if Placement and Scheduler would have chosen the host.

Upgrade

When upgrading a system with minimum_bandwidth rules to support both data plane and placement enforcement we see two options:
1. It is the responsibility of the admin to create the allocations in Placement for all ports using minimum_bandwidth rules. Please note: this assumes that bandwidth is not overallocated at the time of upgrade.
2. Add tooling for 1. as described in the ‘Upgrade impact’ section of [2].
The desired upgrade order of components is the following: Placement, Nova, Neutron

If for some reason the reverse Neutron-Nova order is desired, then the Neutron port API extension of resource_request must not be turned on until both components are upgraded.
Neutron-server must be able to handle agent heartbeats both with and without resource information in the configurations.

Work Items

These work items are designed so Neutron end-to-end behavior can be prototyped and tested independently of the progress of related work in Nova. But part of it depends on already available Placement features.

Extend agent heartbeat configuration with resource provider information.
(We already have it): Persist extended agent configuration reported via heartbeat.
(We already have it): Placement client in neutron-lib for the use of neutron-server.
Neutron-server initially diffs resource info reported by agent against Placement.
Neutron-server diffs consequent agent heartbeat configurations.
Neutron-server turns the diffs into Placement requests (with generation handling).
Extend rule qos_minimum_bandwidth_rule with direction ingress.
Extend port with resource_request based on QoS rule mimimum-bandwidth-placement.
Make the reported agent configuration queriable so neutron-server can infer which backend is implied in the RP allocated (as in binding_profile.allocation).
In binding a port with binding_profile.allocation replace the list of tried mechanism drivers with the one-element list of the inferred backend.
(We already have it): Send binding_profile to all agents.
In sriov-agent force the choice of PF as implied by the RP allocated.

For each of the above:

Tests.
Documentation: api-ref, devref, networking guide.
Release notes.

References

QoS DSCP marking support

Mon, 16 Jul 2018 00:00:00

RFE: https://bugs.launchpad.net/neutron/+bug/1468353

Problem Description

The current QoS API does not provide functionality to mark outgoing network traffic with a DSCP value. This proposal talks about enhancing the existing QoS API’s by adding DSCP marking support. The functionality will make use of the Open vSwitch support for adding DSCP marks to the outbound network traffic.

Proposed Change

We propose an update to the QoS API and OVS driver to support DSCP marks. Valid DSCP mark values are even numbers between 0 and 56, except 2-6, 42, 44, and 50-54. The full list of valid DSCP marks is:

0, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 46, 48, 56

This will serve as the reference implementation for how to use the QoS API to manage DSCP marks using the OVS driver.

Data Model Impact

The model follows the way that the QoS Bandwidth Limiting functionality applies to QoS policies by adding a QosDscpMarkingRule table.

The QosDscpMarkingRule model would look like:

Attribute Name	Type	Access	Default Value	Validation/ Conversion	Description
qos_policy_id	string (UUID)	RO, all	N/A	uuid	QoSPolicy reference
dscp_mark	integer	RW, tenant	N/A	0 and 56, except 2-6, 42, 44, and 50-54	DSCP Mark

REST API Impact

Proposed attribute:

SUB_RESOURCE_ATTRIBUTE_MAP = {
   'dscp_marking_rules':{
        'parent': {'collection_name': 'policies',
                   'member_name': 'policy'},
        'parameters': dict(QOS_RULE_COMMON_FIELDS,
            **{'dscp_mark': {
                'allow_post': True, 'allow_put': True,
                'convert_to': attr.convert_to_int,
                'is_visible': True, 'default': None,
                'validate': {'type:values': common_constants.
                            VALID_DSCP_MARKS}}})
              })
        }
}

Sample REST calls:

GET /v2.0/qos/policies

Response:
{
    "policy": {
        "name": "AF32",
        "description": "This policy marks DSCP outgoing AF32 traffic for DTV Control",
        "shared": "False"
     }
}

GET /v2.0/qos/policies/<policy-uuid>

Response:
{
    "policy": {
        "tenant_id": "<tenant-id>",
        "id": "<id>",
        "name": "AF32",
        "description": "This policy marks DSCP outgoing AF32 traffic for DTV Control",
        "shared": False,
        "dscp_marking_rules": [{
            "id": "<id>",
            "policy_id": "<policy-uuid>",
            "dscp_mark": 16
        }]
     }
}

POST /v2.0/qos/policies/<policy-uuid>/dscp-marking-rules/
{
    "dscp_marking_rule": {
        "dscp_mark": 16
    }
}

Response:
{
    "dscp_marking_rule":{
        "id": "<id>",
        "policy_id": "<policy-uuid>",
        "dscp_mark": 16
    }
}

PUT /v2.0/qos/policies/<policy-uuid>/dscp-marking-rules/<rule-uuid>
{
    "dscp_marking_rule": {
        "dscp_mark": 8
    }
}

Response:
{
    "dscp_marking_rule":{
        "id": "<id>",
        "policy_id": "<policy-uuid>",
        "dscp_mark": 8
    }
}

Command Line Client Impact

qos-dscp-marking-rule-create <policy-id> –dscp_mark <value>
qos-dscp-marking-rule-show <mark-rule-id> <policy-id>
qos-dscp-marking-rule-list <policy-id>
qos-dscp-marking-rule-update <mark-rule-id> <policy-id> –dscp_mark <value>
qos-dscp-marking-rule-delete <mark-rule-id> <policy-id>

Security Impact

None

Notifications Impact

None

Performance Impact

None

IPv6 Impact

None

Other Deployer Impact

Deployers may need to configure the specific QoS driver / ML2 agent extension.

Developer Impact

None

Community Impact

The ability to set DSCP marks on QoS policies on ports or networks using OVS.

Implementation

Assignee(s)

victor-r-howard
nate-johnston
james-reeves5546
margaret-frances

Work Items

Versioned DB objects for the new rule type
API changes to allow for DSCP API modifications
Client changes to allow for DSCP values being set
Openflow integration within OVS driver to add qos_dscp marking functionality

Dependencies

API-tests

Creating DSCP values
Updating DSCP values
Deleting DSCP values
Listing DSCP values
Showing a DSCP Value

Functional Tests

Functional tests will be used to verify system interactions:

Setting DSCP values
Updating DSCP values
Deleting DSCP values
Listing DSCP values
Ensure traffic is using DSCP marks outbound

Fullstack Tests

Setting a QoS policy for marking on a port from API, inspecting that the low-level system bits are set to do DSCP correctly
Updating QoS policy, and checking the low bits (DSCP mark bits)
Deleting QoS policy and verifying all the DSCP rules are deleted properly or not

Documentation Impact

User Documentation

Existing Networking Guide will be updated for this feature.

Existing CLI guide will be updated for this feature.

Developer Documentation

Existing QoS devref document will be updated for this feature.

API Documentation

Existing QoS API documentation will be updated for this feature.

References

Neutron Stadium Evolution

Tue, 12 Jun 2018 00:00:00

The objective of this specification is three-fold:

Define what the Neutron Stadium is.
Define the meaning of the networking community within OpenStack.
Define guidelines on how to maintain the Stadium over time.

Problem Description

Neutron grew to become a big monolithic codebase, and its core team had a tough time making progress on a number of fronts, like adding new features, ensuring stability, etc. During the Kilo timeframe, a decomposition effort started, where the codebase got disaggregated into separate efforts, like the high level services, and the various third-party solutions for L2 and L3 services.

The outcome of that initiative has yielded mixed results, nonetheless giving the various individual teams that came out of it the opportunity to iterate faster and reduce the time to feature. This has been due to the increased autonomy and implicit trust model that made the lack of oversight of the PTL and the Neutron drivers/core team acceptable for a small number of initiatives. However, the proposed arrangement made it possible for projects to be automatically enlisted as a Neutron project based simply on description, and desire for affiliation. With the ever growing number of inclusions, the mission of the PTL/drivers team of ensuring consistency in the APIs, architecture, design, implementation and testing of the project, as well as a level of quality in all aspects, like documentation, inter-project communication, release management, stable backports etc are dealt with have become harder and harder. The point about uniform APIs is particularly important, because the Neutron platform is so flexible that a project can take a totally different turn in the way it exposes functionality, that it is virtually impossible for the PTL and the drivers team to ensure that good API design principles are being followed over time. In a situation where each project is on its own, that might be acceptable, but allowing independent API evolution while still under the Neutron umbrella is counterproductive.

In a nutshell, the existing arrangement of conceding autonomy, and striving for consistency is clearly challenging because of the two conflicting forces at play. A better balance must be struck in a way that Neutron development can continue on its trajectory of agile development without compromising consistency, and without making the PTL, and the Neutron team a bottleneck in the day to day management duties of the project.

Proposed Change

The Neutron Stadium is the list of projects that show up in the following document:

https://governance.openstack.org/tc/reference/projects/neutron.html

The list includes projects that the Neutron PTL and core team are directly involved in, and manage on a day to day basis. To do so, the PTL and team ensure that common practices and guidelines are followed throughout the Stadium, for all aspects that pertain software development, from inception, to coding, testing, documentation and more.

The Stadium is not to be intended as a VIP club for OpenStack networking projects, or an upper tier within OpenStack. It is simply the list of projects the Neutron team and PTL claim responsibility for when producing Neutron deliverables throughout the release cycles.

This spec proposes how the current list (as of May 2016) needs to be revised in order to maintain the integrity of the Stadium, and how to ensure this integrity be maintained over time when modifications to the list are required.

When is a project part of the Stadium?

In order to be considered part of the Stadium, a project must show a track record of alignment with the Neutron core project. This means showing proof of adoption of practices as led by the Neutron core team. Some of these practices are typically already followed by the most mature OpenStack projects and should not be perceived as particularly stringent:

exhaustive documentation: it is expected that each project will have a developer, user/operator and API guide, reachable from docs.o.o;

exhaustive OpenStack CI coverage (unit, functional, tempest), using OpenStack CI (upstream) resources (this means using infra CI, not just downstream CI, as Grafana support will be required); access to CI resources and historical data by the team is key to ensuring stability and robustness of a project;

good release footprint, according to the chosen release model;

adherence to deprecation and stable backports policies;

demonstrated ability to do upgrades and/or rolling upgrades, where applicable;

adoption of neutron-lib (with related hacking rules applied), and proof of good decoupling from Neutron core internals; having unit tests executed against neutron-lib changes is also beneficial in order to show stability of the project;

develop client bindings and CLI according to OSC transition plan (where applicable). This means that Neutron Stadium projects will have their client side hosted in python-neutronclient. The problem stems from the fact that some service extensions do not release on pypi (like fw, vpn or lb) thus it is difficult to get hold of client extensions where they need to be used. SFC and L2GW, for example, do release on pypi so that is not so much a problem for them. In theory their client extensions can stay colocated within their tree but that would make more difficult the alignment to the OSC plugin model. Since the recommendation is to keep extensions in the python-neutronclient tree, it is beneficial to have a consistent approach where all Stadium client extensions are part of the python-neutronclient tree to avoid confusion when dealing with exceptions to the rule;

adoption of neutron-api: submission of any API change must follow the Neutron RFE submission process, where specifications to neutron-specs may be required;

adoption of modular interfaces to provide networking services: this means that L2/7 services are provided in the form of ML2 mech drivers and service plugins respectively. A service plugin can expose a driver interface to support multiple backend technologies, and/or adopt the flavor framework as necessary.

The last two criteria are particularly important for the following reasons:

To provide a consistent API experience to Neutron users/operators and assist Neutron developers more closely. This can only be achieved when API proposals and changes are funneled through a single initiative.

To provide composable networking solutions: the ML2/Service plugin framework was introduced many cycles ago to enable users with freedom of choice. Many solutions have switched to using ML2/Service plugins for high level services over the years. Although some plugins still use the core plugin interface to provide end-to-end solutions, the criterion to enforce the adoption of ML2 and service plugins for Neutron Stadium projects does not invalidate, nor does make monolithic solutions deprecated. It is simply a reflection of the fact that the Neutron team stands behind composability as one of the promise of open networking solutions. During code review the Neutron team will continue to ensure that changes and design implications do not have a negative impact on out of tree code irrespective of whether it is part of the Stadium project or not.

When a project is to be considered part of the Stadium, proof of compliance to the aforementioned practices will have to be demonstrated typically for at least two OpenStack releases. Application for inclusion is to be considered only within the first milestone of each OpenStack cycle, which is the time when the PTL and Neutron team do release planning, and have the most time available to discuss governance issues.

Projects part of the Neutron Stadium have typically the first milestone to get their house in order, during which time reassessment happens; once removed, a project cannot reapply within the same release cycle it has been evicted.

What if a project cannot be considered part of the Stadium?

The aforementioned criteria imply that the Neutron PTL is unable to favorably consider a project for inclusion, unless all the criteria are met. This in turn imposes that new initiatives start as standalone projects hosted on the openstack.org namespace, as outlined in the creator guide. Furthermore, projects that interact with Neutron purely via its REST API should also be incentivized to seek other forms of governance, such as applying as top-level OpenStack project and follow the new project requirements as outlined by the OpenStack Governance, rather than going for Stadium inclusion. Drivers that leverage proprietary software and/or hardware will also not be considered for inclusion into the Neutron Stadium, due to the barrier on access to the technology for dev/test and CI integration, as well as the inability for the Neutron team to provide effective feedback during review for any aspect of software development: from code to infrastructure and testing. This does not mean that the Neutron team will stop collaborating with the individual teams supporting these initiatives (as past experiences demonstrate), and in fact any team is encouraged to seek collaboration in order to address integration issues with the core Neutron platform. Teams for projects outside the Stadium are simply fully responsible of dealing with the end-to-end SDLC of their solutions, and thus empowered. The integration code to external systems can still be hosted on openstack.org irrespective of their nature and get access to the same resources any other OpenStack project uses. It is noteworthy that it may not be useful for them to seek inclusion on governance.openstack.org in order to achieve ‘recognition’ in that they are typically one layer underneath the projects that are considered for inclusion by the TC. Besides, other programs, like the marketplace driver certification program are more effective tools to reflect the level of quality of a specific driver solution, as they also track the level of supportability across the various OpenStack releases.

What if a project within the Neutron Stadium decides to include drivers for proprietary systems?

Today’s reality is that some Neutron projects still host drivers to proprietary systems, like vpnaas, fwaas, and lbaas/octavia; even though the decision of accepting new drivers lies with the respective teams, it is strongly encouraged that these be exceptions rather than the norm. This point becomes less relevant if, in the long run some of these efforts will no longer be considered responsibility of the Neutron team.

How do we build a Networking Community?

Neutron has been, since its inception, the networking project within OpenStack. It has been the place where APIs and networking abstractions are proposed and agreed on. There have been deadlocks and stalemates as in any other project, and one way to resolve these stalemates is to allow individual initiatives to experiment in isolation: ultimately code wins. Even though this might seem okay at first, Neutron still lack the ability to reconcile all these initiatives back together in a coherent fashion.

In order to resolve this problem, the Neutron team should consider the adoption of a neutron-api initiative where API proposals are consolidated. The neutron-api project can be a component to be consumed by Neutron Stadium projects, and to be started off as a spinout of the neutron api codebase such as neutron/api, neutron/extensions, and any other project (e.g. bgp, l2gw, sfc, etc) that is considered for Stadium inclusion. Contributions to this module will follow the Neutron RFE process with submission of specification documents to neutron-specs, if deemed required. The resulting code will end up being located in the api module itself, whilst specs will be hosted on neutron-specs. Each core team of projects belonging to the Stadium will have +2 rights on neutron-specs. The neutron drivers will continue to hold +W rights on both repos. It is noteworthy that members of the individual core teams are and will be considered for membership of the drivers team on a regular basis. Level of participation in reviewing RFE submissions, experience with the Neutron codebase, and commitment to the project are factors taken into account when identifying members of this team. From an implementation perspective, the api module can either exist on its own, as yet another repo within the Stadium list, or be incorporated as a /api module under neutron-lib. Whilst the former approach allows finer control of the release model, the latter has the obvious benefit of minimizing complexity in landing end-to-end code changes.

The reason behind the existence of this effort and the requirement to go through neutron-specs for API proposals is two-fold: a) ensure continuous alignment across all APIs exposed from Neutron Stadium projects; b) provide a place in the OpenStack networking community to discuss and achieve consensus on networking abstractions; and c) as a robust foundation to build interoperable APIs.

What happens to the current list of Neutron Stadium projects?

At the time of writing the Stadium contains the projects as outlined below. Based on all the aforementioned considerations, all of individual teams will have to work during the Newton timeframe to make the current proposal a reality. More precisely, to be considered for inclusion (continue to be part of the Stadium list), the following plan must be put in place:

neutron: the core team will have to stand up neutron-api, and transition the api models/definitions out of the neutron repo. Developer documentation on the Neutron Stadium will be revised according to the guidelines outlined in this proposal.
neutron-fwaas: the fwaas core team will have to:
- complete the transition to fwaas v2;
- adopt neutron-api, and neutron-lib;
- complete docs, end-to-end testing, and demonstrate ability to upgrade;
neutron-lbaas: the existence of this project is now questioned by the presence of Octavia. This is elaborated more in spinout spec. In a nutshell, this project will be phased out in favor of Octavia being an OpenStack top-level project.
neutron-vpnaas: this project has seen the least amount of traction during the last few cycle. Testing and documentation is not adequate therefore the vpnaas core team will have to:
- adopt neutron-api, and neutron-lib;
- complete docs, end-to-end testing, and demonstrate ability to upgrade;
- switch to OSC plugin;
neutron-dynamic-routing: this is a spinout of BGP code that landed in the Mitaka timeframe. Currently in progress.
networking-l2gw: this needs to:
- adopt neutron-api, and neutron-lib;
- complete docs, end-to-end testing, and demonstrate ability to upgrade;
- move client side code to python-neutronclient and switch to OSC plugin;
networking-bagpipe: the core team will have to:
- adopt neutron-api, and neutron-lib;
- complete docs, end-to-end testing, and demonstrate ability to upgrade;
networking-bgpvpn: the core team will have to:
- adopt neutron-api, and neutron-lib;
- complete docs, end-to-end testing, and demonstrate ability to upgrade;
- move client side code to python-neutronclient and switch to OSC plugin;
networking-calico: the core team will have to:
- adopt neutron-api, and neutron-lib;
- complete docs, end-to-end testing, and demonstrate ability to upgrade;
networking-midonet: the core team will have to:
- adopt neutron-api, and neutron-lib;
- complete docs, end-to-end testing, and demonstrate ability to upgrade;
- move client side code to python-neutronclient and switch to OSC plugin;
networking-odl: the core team will have to:
- adopt neutron-api, and neutron-lib;
- complete docs, end-to-end testing, and demonstrate ability to upgrade;
networking-ofagent: this is deprecated and marked for removal.
networking-onos: the core team will have to:
- adopt neutron-api, and neutron-lib;
- complete docs, end-to-end testing, and demonstrate ability to upgrade;
networking-ovn: the OVN project will need to consider switching back to an ML2 driver. This is beneficial for the following reasons:
- considered as a solution for multi-hypervisor deployments: this is currently only true if a hypervisor runs Open vSwitch.
- present itself as a viable alternative to supplant the ovs mech_driver in the OpenStack gate: this can only be done so if we are swapping like for like.
- do bare-metal and virtual provisioning in the context of the same Neutron deployment; hardware vtep support only is limiting a number of potential bare metal scenarios that leverage third party solutions.
- getting access to a number of performance/reliability enhancements that the Neutron team develops against ML2.
- Composability of services across the entire L2-L7 stack.
networking-sfc: the core team will have to:
- adopt neutron-api, and neutron-lib;
- complete docs, end-to-end testing, and demonstrate ability to upgrade;
- move client side code to python-neutronclient and switch to OSC plugin;
octavia: this is being tackled in the context of neutron-lbaas. In a nutshell, this will eventually be considered as top-level OpenStack project.
python-neutron-pd-driver: this is a project that depends on Neutron IPv6 support and introduces another prefix delegation driver besides the one available in tree. With the in-tree implementation of dibbler this is not as necessary any more. Dropping it in O-1 is an acceptable outcome if no-one picks it up by then.

Contributing the respective API extensions of these individual projects to neutron-api does not mean reviewing the APIs from scratch as these APIs already have working solutions behind them. Having said that, any revision required, as potentially identified through review, will have to be followed up according to RFE submission process. These projects and their respective core teams will be given time to comply until Ocata Milestone-1, after which, if not compliant, they will be removed from the Neutron Stadium. All things considered, a team can decide not to adopt the aforementioned plan and ask to be removed during the Netwon timeframe.

This deadline is considered to be firm, but will be revised according to how fast we can consolidate APIs and neutron-lib mature to a point of effective consumption by the neutron project and once thorough documentation is provided to help the various parties through the transition.

How can an existing OpenStack project apply for inclusion?

There are projects currently hosted on OpenStack that are not part of the Neutron governance list, for example tap-as-a-service, and there may be more that may be created in the future. Such initiatives should consider being part of the Neutron Stadium if and only if there is a mutual and beneficial collaboration already established across the teams involved. As mentioned above, the Stadium is not an elite club but simply the projects that the PTL and the Neutron team take responsibility for. The more involvement and mutual collaboration exist over time, the more likely it is that there is good alignement and thus Stadium inclusion becomes a a little more than a formality. In order to succeed in an application, a project should first and foremost make sure that these criteria are met:

documentation, testing and upgrade strategy are in good order as outlined earlier;
client extensions available as OSC plugins;
adoption of neutron-lib;

The request for application would then need to be submitted with an RFE and subsequent spec submission where the following information is provided:

pointers to user and developer documentations;
pointers to server and client side internals;
pointers to CI jobs (e.g. unit, tempest, grenade, etc) as well as Grafana dashboards;
pointers to stable backports and release deliverables;
pointers to API specification and definitions;

Once evaluated and successfully approved via RFE process, the teams will work together to transition API and client components respectively to api module and python-neutronclient repos. The newly added project pledge to adopt the Neutron RFE submission process for API changes (and API changes alone) and spec submission to neutron-specs where deemed necessary from that point forward.

Why is this Stadium arrangement beneficial?

The Stadium creates a place for like-minded people to collaborate, innovate, and iterate on networking solutions, and provides a degree of guarantee that a project belonging to the Stadium behaves similarly to any other project within the Stadium, thus inducing a more sustainable maintenance effort on the Neutron team. On the other hand, the Neutron team takes responsibility of the deliverables produced by Stadium project (docs, release notes, etc), and therefore giving these initiatives higher visibility to downstream consumers. Contributors of Neutron Stadium projects participate in the Neutron PTL election process and have the same status of any other OpenStack Active Technical Contributor.

What are the drawbacks of this proposal?

The cost of sanitizing the Stadium and keeping it sane over time is non negligible, though the criteria have been designed to be lightweight and impose minimal burden on the teams once the Stadium is up and running. This is the trade-off to keep all the networking related efforts working in symbiosis. Hopefully, it should be clear by now that associating a bad connotation to projects that are not part of the Stadium is simply a misconception: some projects can indeed apply to be top-level projects in OpenStack, and others are simply better off by being independent initiatives.

Are there any alternatives worth considering?

Open to suggestions.

References

Port Mirroring API for VF Mirroring

Mon, 11 Jun 2018 00:00:00

https://blueprints.launchpad.net/neutron/+spec/port-mirroring-sriov-vf

Port mirroring is a common feature where specific traffic can be mirrored to a traffic analyzer by configuring rules to identify required flows to be mirrored and by specifying the analyzer where the traffic is mirrored to. In addition, mirroring can be configured on VM interfaces to get all the traffic to and from the interface to the specified analyzer.

A common use case for this feature is a client requesting an analyzer or probe for traffic that needs to be monitored for a particular VNF using SR-IOV.

This spec will focus on port mirroring based on SR-IOV VF to VF. Currently this document concentrates on tapping traffic coming in and out of VMs using SR-IOV. This spec is based on enhancements done by Intel on i40e driver, see [9]

The current TAPaaS is implemented with OVS ports, see [1] and [3], this spec will implement the same for the SR-IOV ports.

Problem Description

In virtual environments there are NICs that have capabilities to send a copy of network packets seen on one VF port attached to VNF to another VF port attached to a VM analyzer.

This spec helps leverage the NIC capability to do the mirroring and using Tap as Service enhancements these features can be configured on hosts and leveraged by tenants.

The tenant today can use Neutron APIs to create an SRIOV port using provider network with VLAN 0 and physical network that is used in nova.conf passthrough_whitelist. This VF will not have any VLAN filters configured on it. See [2]. The guest can be configured for multiple VLANs on the same interface. Inside the guest, sub-interfaces can be configured and can communicate on multiple VLANs.

There are two approaches that can be used for implementing this.

1. The tenant can use the binding_profile value_spec e.g guest_vlans to define which VLANs will be configured inside guest.

Since the parameter is configured for a VF in the same manner as other PCI details of a VF, this can be exposed at the same place in the API as other SRIOV related fields in the binding profile.

Since this spec only describes features available for SRIOV based implementation and the NIC(i40e) only allows VLAN filter for filtering of the packets that need to be mirrored on the NIC, this approach is sufficient until we have a spec for defining multiple VLAN filters for a direct port or on a VF.

This will also help operators to look at VF config at one place in system rather than running multiple APIs to figure out VFs configuration on host.

2. The second option is to change the TapFlow to include the new field, vlan_mirror. The TapFlow is the flow classifier for the TapService, so updating it with the new field will provide the agent with the data it needs to configure the port.

Use Case I. Mirroring of specific VLANs to VF

This use case will handle sending the mirror packets to tap service for a specific VLAN configured on the monitored VF. There can be multiple VLANs configured on the VF.

Proposed Change

Update Tap as a Service to support VF to VF mirroring.

The proposed service will allow the tenants to create a tap service instance with SR-IOV port to which the user can add Neutron ports that need to be mirrored by creating tap flows from other SR-IOV port/s.

The user can define various port mirroring flows for the tap service, so one tap service should be able to handle multiple tap flows.

Use Case I. Mirroring of specific VLANs to VF

This spec will also take care of developing a new driver for Tap for NicSwitchTapDriver that will perform the sysfs implementation for Intel’s custom i40e driver.

The VFs will be derived from source port and the probe port attached to TapService. Example of how a user could mirror traffic based upon VLANs 2,4,6,18-22 to VF 3 of PF p1p1

# echo add 2,4,6,18-22>/sys/class/net/p1p1/device/sriov/3/vlan_mirror

Example on how a user could remove VLAN 4, 15-17 from traffic mirroring at destination VF

# echo rem 4,15-17>/sys/class/net/p1p1/device/sriov/3/vlan_mirror

Example on how a user could remove all VLANs from mirroring at VF 3. # echo rem 0 - 4095>/sys/class/net/p1p1/device/sriov/3/vlan_mirror

The limitation of this mirror is that the vlan/s monitored get traffic in BOTH direction on probe or service port.

The mirrored packets captured on the VF associated with Tap service will have vlan tags that are specified in field vlan_filter.

Data Model Impact

This approach will change data model for TapFlow object to include new field vlan_filter.

REST API Impact

The implementation will be based on option 2. in section Problem description above. The change will be made to the TapFlow object to add the vlan_filter field in order for the ports to be able to filter which traffic to mirror. See [4] The vlan_filter can be used to monitor a single vlan or a vlan range so the values can be specified as a string like “171” or “161-164” or “161-164,171”.

Tap Agent and Driver Impact

Tap Agent and driver for direct ports will be developed as per usecases above to configure NIC via sysfs.

Effects on Existing Tap As Service APIs

vlan_filter will be added to TapFlow API to filter the traffic that need to be mirrored.

Command Line Client Impact

Openstack/Neutron client need to be updated to support TAPaaS. See [7]

Heat support Impact

The Tap as service resources do not exist in Openstack. We need to create new heat resources to support that. See [8]

Implementation

The Tap As a Service Agents need to add support for SR-IOV ports. Need to develop a framework to support multiple agents. Also define the taas agent for sriov ports, taas-sriov-agent. Currently there is agent only for ovs. As part of the implementation a new driver for Tap as a service will be developed NicSwitchTapDriver as mentioned above in proposal. See [6] Earlier an attempt to keep SRIOV params together was done in implementation by updating the binding profile, the change was abandoned. See [5]

Additional work to integrate with Neutron Stadium

Tempest tests for the new agent and driver.

References

Neutron-vpnaas Scorecard

Wed, 23 May 2018 00:00:00

Neutron integration

N0. Does the project use the Neutron REST API or relies on proprietary backends?

Yes. Neutron-vpnaas implements its own set of Neutron API extensions on top of the Neutron core framework and it does so by using the service plugin model. The API exposed has open source implementations, and it provides a pluggable mechanism for proprietary backends.

N1. Does the project integrate/use neutron-lib?

Yes. The migration report shows that there are currently 621 total imports. Neutron is imported 119 times and Neutron-lib 81 times, for a migration percentage of 40.5000%,

N2. Do project members actively contribute to help neutron-lib achieve its goal?

No. None of the project core members have merged anything meaningful into neutron-lib (source: https://review.openstack.org/#/q/project:openstack/neutron-lib+status:merged,75).

N3. Do project members collaborate with the core team to enable subprojects to loosely integrate with the Neutron core platform by helping with the definition of modular interfaces?

Yes.
- https://github.com/openstack/neutron-vpnaas/commit/7c10c41376f10898347cbc644d8d504b8258c99c

N4. How does the project provide networking services? Does it use modular interfaces as provided by the core platform?

Yes, the VPN agent has been recently re-implemented using the L3 agent extension API. Please look at https://bugs.launchpad.net/neutron/+bug/1692128 and https://review.openstack.org/#/c/488247/. This approach makes the VPN agent easier to deploy along with the L3 agent.

N5. If the project provides new API extensions, have API extensions been discussed and accepted by the Neutron drivers team? Please provide links to API specs, if required.

VPNaaS API was initially created by the neutron core team, and so implicitly had been agreed upon by previous core team.

Documentation

D1. Does the project have a doc tox target, functional and continuously working? Provide proof (links to logs.openstack.org).

Yes.
- https://github.com/openstack/neutron-vpnaas/blob/master/tox.ini
- http://logs.openstack.org/99/502699/125/check/build-openstack-sphinx-docs/a1c179a/html/

D2. If the project provide API extensions, does the project have an api-ref tox target, functional and continously working? Provide proof (links to logs.openstack.org).

Yes. There is an API reference and is being confirmed its accuracy, see:

D3. Does the project have a releasenotes tox target, functional and continously working? Provide proof.

Yes.

D4. Describe the types of documentation available: developer, end user, administrator, deployer.

Yes. These documentations are going to be published, see:

Continuous Integration

C1. Does the project have a Grafana dashboard showing historical trends of all the jobs available? Provide proof (links to grafana.openstack.org).

Yes.
- http://grafana.openstack.org/dashboard/db/neutron-vpnaas-failure-rates

C2. Does the project have CI for unit coverage? Provide proof (links to logs.openstack.org)

Yes.
- http://status.openstack.org/openstack-health/#/g/project/openstack~2Fneutron-vpnaas

C3. Does the project have CI for functional coverage? If so, does it include DB migration and sync validation?

Yes. We have gate-neutron-vpnaas-dsvm-functional-sswan-ubuntu-xenial for functional coverage and DB migration tests are running as a part of it.
- http://logs.openstack.org/91/496991/23/check/legacy-neutron-vpnaas-dsvm-functional-sswan/3c51750/

C4. Does the project have CI for fullstack coverage?

No. We consider it as lower priority and it has none at the moment.

C5. Does the project have CI for Tempest coverage? If so, specify nature (API and/or Scenario).

Yes.
- http://status.openstack.org/openstack-health/#/job/neutron-dsvm-tempest-vpnaas

C6. Does the project require CI for Grenade coverage?

Yes. But it has none.

C7. Does the project provide multinode CI?

No. But it is needed to support L3-HA (and/or DVR) and unnecessary until then.

C8. Does the project support Python 3.x? Provide proof.

Yes.
- http://status.openstack.org/openstack-health/#/job/openstack-tox-py35-neutron-vpnaas

Release footprint

R1. Does the project adopt semver?

Yes.

R2. Does the project have release deliverables? Provide proof as available in the release repo.

Yes.
- https://tarballs.openstack.org/neutron-vpnaas/neutron-vpnaas-11.0.0.tar.gz
- https://pypi.org/project/neutron-vpnaas/

R3. Does the project use upper-constraints?

Yes.
- https://github.com/openstack/neutron-vpnaas/blob/master/tox.ini#L10

R4. Does the project integrate with OpenStack Proposal Bot for requirements updates?
- Yes.
- https://github.com/openstack/requirements/commit/1d545edbebfff2e8983d6cab24a92c32636dd6bf

Stable backports

S1. Does the project have stable branches and/or tags? Provide history of backports.

Yes. For example: https://git.openstack.org/cgit/openstack/neutron-vpnaas/log/?h=stable/pike

Client library

L1. If the project requires a client library, how does it implement CLI and API bindings?

Yes. There are Neutron CLI and API bindings. OSC is going to be done, see:
- https://review.openstack.org/#/c/439978/

Scorecard

Scorecard
N0 \| Y
N1 \| Y
N2 \| N
N3 \| Y
N4 \| Y
N5 \| Y
D1 \| Y
D2 \| Y
D3 \| Y
D4 \| Y
C1 \| Y
C2 \| Y
C3 \| Y
C4 \| N
C5 \| Y
C6 \| N
C7 \| N
C8 \| Y
R1 \| Y
R2 \| Y
R3 \| Y
R4 \| Y
S1 \| Y
L1 \| Y

Final remarks

At the time of writing the project scores changed to positively if compared with the last assessment [1] [2] for following 7 criteria: N3, N4, D2, D4, C1, C5, L1. It makes the project score positively in 20 out of 24 criteria. The subproject does not seem to lack the resources recently and the remaining gaps can be focused to make timely progress when required.

References

Decoupling database API & Utilities imports/access for neutron-lib

Thu, 17 May 2018 00:00:00

This work is not related to an enhancement request and therefore doesn’t have a RFE. Rather the intent herein is to discuss how we decouple neutron.db and related database imports/access as part of the overall neutron-lib effort.

Current neutron database access can be broken into the following high-level categories:

Database API & Utilities
Core & Extension Database Mixins
Database Resource Models
Database Migration

As the database access patterns span a wide range of logic/code, a set of specs will be proposed each focusing on a single access pattern.

This spec specifically addresses the Database API & Utilities access.

For current neutron-lib related blueprints, see [1] and [2].

Problem Description

As part of our neutron-lib effort, we need to decouple out-of-tree networking projects that depend on (e.g. import) neutron. However, today a large number of neutron consumers import database related neutron modules [3] [4] [5]. While some of these dependencies are rehomeable into neutron-lib without significant effort, others create a large dependency chain with neutron internals, coupling them to neutron itself.

The intent of this spec is to propose how we can rehome the neutron database API and utilities into neutron-lib breaking consumer dependencies on these neutron modules. The access patterns herein were gathered by inspecting usages of the respective modules using [3], [4], [5], [6].

Proposed Change

This spec proposes we rehome the generic database API/utility functionality into neutron-lib. The following considerations must be taken into account for all applicable code:

If the functionality is private and not used outside of neutron today, it can likely stay in neutron as private to reduce public API surface area.
If the functionality is private, but used outside of neutron, it’s a candidate for rehoming.
For code applicable to rehoming, determine the best form for the implementation. If used by non-OVO consumers then likely it’ll be necessary to rehome as “generic database logic”. However if applicable to OVO, it may make sense to incorporate the logic into neutron objects that will be rehomed in subsequent work.

When the work for this spec is done, we should no longer have any consumers using neutron’s database API or utils [3], [4], [5], [6], but rather using them from neutron-lib.

Subsequent subsections describe each module in greater detail.

Note that provisioning_blocks will be addressed in subsequent specs as it requires versioned objects and thus will need access to them via neutron-lib in order to implement.

Database API

The neutron.db.api module is used broadly today [3] and has already begun rehoming into neutron-lib [7]. Therefore this spec proposes we finish the database API rehoming work by:

Rehoming the remaining externally used functions in the API. This includes generic functions such as is_retriable as well as decorators like retry_db_errors and retry_if_session_inactive.
Removing deprecated and unused functions such as load_one_to_manys and get_session.
The osprofiler setup/initialization will need to be addressed to ensure proper profiling functionality.
Unit tests will need to be in place for the rehomed functionality; rehomed and/or written as needed.
Consuming the rehomed changes once released.

Generic Utils

The neutron.db._utils module contains generic logic and is thus used by some consumers today [4]. This spec therefore proposes we rehome the used functionality into neutron-lib by:

Rehoming the used functions such as resource_fields and get_marker_obj
Rehoming any functions use in the model query utils (see subsequent section) into neutron-lib.
Ensuring the proper unit test coverage is in place for the rehomed code.
Consuming the rehomed changes once released.

Model Query Utils

While today neutron.db._model_query is private, some consumers use it [5]. As this module contains generic database logic, it’s a candidate for rehoming to neutron-lib as a generic model hook module by:

Rehoming the logic in the _model_query module itself to neutron-lib.
Ensuring the generic utils are in lib that are required for this module. See the previous section on generic utils.
Rehoming the few used functions from neutron.common.utils used in the module query module. These are generic in nature and thus don’t raise any red flags for rehoming.
Rehoming neutron.objects.utils to neutron-lib as its used in the model query logic.
Ensuring the proper unit test coverage is in place for the rehomed code.
Consuming the rehomed changes once released.

Resource Extend

The neutron.db._resource_extend module is used by consumers today [6] and as it contains generic functionality, can also move into neutron-lib by:

Rehoming the resource extend module into neutron-lib.
Rehoming the single generic neutron.common.util function required by the module into neutron-lib.
Ensuring the proper unit test coverage is in place for the rehomed code.
Consuming the rehomed changes once released.

Decoupling database Resource Model imports/access for neutron-lib

Thu, 17 May 2018 00:00:00

Current neutron database access can be broken into the following high-level categories:

Database API & Utilities
Core & Extension Database Mixins
Database Resource Models
Database Migration

As the database access patterns span a wide range of logic/code, a set of specs will be proposed each focusing on a single access pattern.

This spec specifically addresses Database Resource Model access.

For current neutron-lib related blueprints, see [1] and [2].

Problem Description

Access to database models by consumers is typically used for:

Defining relationships between neutron and other project models. Note that while string names can be used when defining a relationship, this is not a viable solution for all cases due to [8].
Building queries using models and/or their fields. With the introduction of versioned objects, consumers can effectively query using methods on objects. Therefore the deliverables of this spec enable such an approach by making versioned objects accessible via neutron-lib.
Importing models to have them available for database tools. This includes database migration tools, so while this spec doesn’t directly solve the migration access, it needs to lay a foundation for it (covered in a separate spec).

The intent of this spec is to propose how we can provide access to database models without direct neutron imports by means of a bridge in neutron-lib thereby breaking consumer’s dependencies on neutron’s internal database models.

Proposed Change

While one foreseeable solution would be to publish models using discoverable entry points (stevedore) or register them using a factory, these models must be versioned to allow consumers to determine compatibility. Without versioning the underlying model can change without the consumers knowledge, thereby breaking them.

Implementing versioned models is simple enough with some new facility, however we already have such a versioning scheme; neutron versioned objects. Neutron objects already contain a reference to the underlying model as well as a version number that’s incremented as the model changes [5]. Therefore if we can provide a way for consumers to get at neutron objects, they have access to a version, corresponding model, etc..

This spec proposes we use a simple entry point scheme whereby all versioned objects are defined as entry points that can then be looked up and handed out by some bridging logic in neutron-lib. More specifically:

Neutron versioned objects can be “published” via an entry point in setup.cfg where each entry point is a versioned object class. This exposes the objects to neutron-lib that can be discovered and loaded with stevedore.
A simple API in neutron-lib allowing consumers to retrieve versioned objects at runtime. In it’s basic form a consumer asks for an object of type X, whereupon neutron-lib looks it up from the entry point and returns the concrete object class to the consumer (e.g. load_class('Port') looks up and returns the versioned object class for Port).
If consumers need to create instances of the versioned object(s) returned from neutron-lib, they can invoke it’s constructor directly to create a new instance. Since versioned object constructor’s are based on the object’s fields and the fields are tied to the model, consumers can query the object’s VERSION to determine compatibility with the constructor.

The snippet below illustrates the API from a consumers point of view:

from neutron_lib.objects import registry
# .. other imports

# Loading the object from neutron-lib uses stevedore to find the requested
# object. Also note that under the covers the model is imported since the
# concrete versioned object must import the model to ref it.
port_ovo = registry.load_class(constants.PORT)

if port_ovo.VERSION > '1.1':
    raise UnsupportedException("Can't use port objects greater than 1.1")

# It's just an OVO class, so we can use it's static/class methods directly
a_port = port_ovo.get_object(...)

# Create a new versioned object; it's VERSION determines the constructor's
# supported kwargs so consumers can detect compatibility
new_port = port_ovo(context, project_id=...)

As mentioned earlier, while this solution doesn’t completely solve database migration access, it paves the way as we now have a way to find and load versioned objects which must import the respective model.

Using this scheme, we should be able to eliminate consumers imports for:

Existing neutron.objects imports [6]. Consumers now get their object from the neutron-lib API provided herein.
Building queries; version objects have methods allowing consumers to effectively query and there should be no need to build direct queries in consumers otherwise.

In addition, the rollout of this functionality should have minimal impact to the main code paths; the only real change in neutron is to expose the objects via setup.cfg. The neutron-lib logic can be implemented, tested and rolled out independently thereby reducing risk.

For a sample proof of concept, see the patches on [7] that use this approach to remove/use versioned objects in a few of the vmware-nsx modules.

References

Firewall as a Service API 2.0 Address Groups Support

Wed, 28 Mar 2018 00:00:00

Launchpad blueprint:

https://blueprints.launchpad.net/neutron/+spec/fwaas-2.0-address-groups

This bp introduces a enhancement to Firewall as a Service(FWaaS) API 2.0 for supporting address groups. This feature has been proposed in fwaas-api-2.0 but still not implemented.

Problem Description

In actual use of firewall groups, each IP or subnet requires a corresponding firewall rule. When there are a large number of instances, a large number of firewall rules are generated and it is difficult to maintain and manage them.

Proposed Change

Add address group functions to a firewall group. By aggregating multiple address objects into address groups and using address groups instead of the original cidr to generate firewall rules, the number of firewall rules can be effectively reduced.

REST API Impact

Firewall Address Groups

Attribute	Type	Req	CRUD	Description
id	uuid-str	N/A	R	Unique identifier for the address_group object.
name	String	No	CRU	Human readable name for the address group (255 characters limit). Does not have to be unique.
description	String	No	CRU	Human readable description for the address group (255 characters limit).
project_id	uuid-str	No	CR	Owner of the address group. Only admin users can specify a project identifier other than their own.
addresses	List	Yes	CRU	Array of key-value pairs of address and ip version. It supports both CIDR and IP range objects. Attributes of CIDR and IP range objects: “address”: <CIDR or IP range> “ip_version”: 4 or 6(Integer value) An example of addresses: [{“address”: “132.168.4.12/24”, “ip_version”: 4}]

Firewall Rules

Note that as with FWaaS 1.0, in FWaaS 2.0 firewall rules always use stateful connection tracking.

Attribute	Type	Req	CRUD	Description
id	uuid-str	N/A	R	Unique identifier for the firewall rule object.
project_id	uuid-str	No	CR	Owner of the firewall rule. Only admin users can specify a project identifier other than their own.
name	String	No	CRU	Human readable name for the firewall rule (255 characters limit). Does not have to be unique.
description	String	No	CRU	Human readable description for the firewall Rule (255 characters limit).
shared	Bool	No	CRU	When set to True makes this firewall rule visible to projects other than its owner, and can be used in firewall policies not owned by its project.
protocol	String	No	CRU	IP Protocol.
source_port	port-range	No	CRU	Source port number or a range (an int in [1, 65535] or range in a:b).
destination_port	port-range	No	CRU	Destination port number or a range ( an int in [1, 65535] or range in a:b).
ip_version	Integer	No	CRU	IP Protocol Version.
source_ip_address	String	No	CRU	Source IP address or CIDR.
destination_ip_address	String	No	CRU	Destination IP address or CIDR.
source_address _group_ids	List	No	CRU	This is a list of source address groups. When they are specified, they are matched when the source IP address in the packet matches one of the IP addresses in one of the address groups.
destination_address _group_ids	List	No	CRU	This is a list of destination address groups. When they are specified, they are matched when the destination IP address in the packet matches one of the IP addresses in one of the address groups.
action	String	No	CRU	Action to be performed on the traffic matching the rule (ALLOW, DENY, REJECT). Default: DENY.
enabled	Bool	No	CRU	When set to False will disable this rule in the firewall policy. Facilitates selectively turning off rules without having to disassociate the rule from the firewall policy. Default: True.

Note: At most one of source_ip_address, source_address_group_ids and source_firewall_group_id can be specified. The rule is matched when the source IP address in the packet matches any one of: source_ip_address, one of the IP addresses in the address group, or an IP address of one of the ports in the firewall group. If you want it to match any packet, set the source or destination to 0.0.0.0/0 or ::/0. The same applies to destination_ip_address, destination_address_group_ids, and destination _firewall_group_id, with respect to the destination IP address in the packet.

List address groups

Lists address groups.

Request Type

GET

Endpoint

/v2.0/fwaas/address_groups

Response Codes

Success

200

Error

Unauthorized(401)

Example List address groups: JSON request

GET /v2.0/fwaas/address_groups.json
User-Agent: python-neutronclient
Accept: application/json

Example List address groups: JSON response

{
    "address_groups": [
        {
            "description": "",
            "id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
            "name": "ADDR_GP_1",
            "project_id": "45977fa2dbd7482098dd68d0d8970117",
            "addresses": [
               {"address": "132.168.4.12/24", "ip_version": 4},
               {"address": "132.168.5.12-132.168.5.24", "ip_version": 4},
               {"address": "2001::db8::f00/64", "ip_version": 6}
            ]
        }
    ]
}

Show address group details

Shows address group details.

Request Type

GET

Endpoint

/v2.0/fwaas/address_groups/<address_group_id>

Response Codes

Success

200

Error

Unauthorized(401), Not Found (404)

Example Show address group: JSON request

GET /v2.0/fwaas/address_groups/8722e0e0-9cc9-4490-9660-8c9a5732fbb0.json
User-Agent: python-neutronclient
Accept: application/json

Example Show address group: JSON response

{
   "address_group": {
        "description": "",
        "id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
        "name": "ADDR_GP_1",
        "project_id": "45977fa2dbd7482098dd68d0d8970117",
        "addresses": [
           {"address": "132.168.4.12/24", "ip_version": 4},
           {"address": "132.168.5.12-132.168.5.24", "ip_version": 4},
           {"address": "2001::db8::f00/64", "ip_version": 6}
        ]
    }
}

Create address group

Creates an address group.

Request Type

POST

Endpoint

/v2.0/fwaas/address_groups/

Response Codes

Success

201

Error

Unauthorized(401), Bad Request(400)

Example Create address group: JSON request

POST /v2.0/fwaas/address_groups.json
User-Agent: python-neutronclient
Accept: application/json

{
    "address_group": {
        "name": "ADDR_GP_1",
        "addresses": [
           {"address": "132.168.4.12/24", "ip_version": 4},
           {"address": "132.168.5.12-132.168.5.24", "ip_version": 4},
           {"address": "2001::db8::f00/64", "ip_version": 6}
        ]
    }
}

Example Create address group: JSON response

HTTP/1.1 201 Created
Content-Type: application/json; charset=UTF-8

{
   "address_group": {
        "description": "",
        "id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
        "name": "ADDR_GP_1",
        "project_id": "45977fa2dbd7482098dd68d0d8970117",
        "addresses": [
           {"address": "132.168.4.12/24", "ip_version": 4},
           {"address": "132.168.5.12-132.168.5.24", "ip_version": 4},
           {"address": "2001::db8::f00/64", "ip_version": 6}
        ]
    }
}

Update address group

Updates an address group.

Request Type

PUT

Endpoint

/v2.0/fwaas/address_groups/<address_group_id>

Response Codes

Success

200

Error

Unauthorized(401), Bad Request(400) Not Found(404)

Example Update address group: JSON request

PUT /v2.0/fwaas/address_groups/8722e0e0-9cc9-4490-9660-8c9a5732fbb0.json
User-Agent: python-neutronclient
Accept: application/json

{
    "address_group": {
        "addresses": [
           {"address": "132.168.4.12/24", "ip_version": 4},
           {"address": "132.168.5.12-132.168.5.24", "ip_version": 4},
           {"address": "2001::db8::f00/64", "ip_version": 6}
        ]
    }
}

Example Update address group: JSON response

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
   "address_group": {
        "description": "",
        "id": "8722e0e0-9cc9-4490-9660-8c9a5732fbb0",
        "name": "ADDR_GP_1",
        "project_id": "45977fa2dbd7482098dd68d0d8970117",
        "addresses": [
           {"address": "132.168.4.12/24", "ip_version": 4},
           {"address": "132.168.5.12-132.168.5.24", "ip_version": 4},
           {"address": "2001::db8::f00/64", "ip_version": 6}
        ]
    }
}

Delete address group

Deletes an address group.

This operation does not return a response body.

Request Type

DELETE

Endpoint

/v2.0/fwaas/address_groups/<address_group_id>

Response Codes

Success

204

Error

Unauthorized(401), Not Found(404) Conflict(409) The Conflict error response is returned when an operation is performed while address group is in use.

Example Delete address group: JSON request

DELETE /v2.0/fwaas/address_groups/8722e0e0-9cc9-4490-9660-8c9a5732fbb0.json
User-Agent: python-neutronclient
Accept: application/json

Example Delete address group: JSON response

HTTP/1.1 204 No Content
Content-Length: 0

List firewall rules

Lists firewall rules.

Request Type

GET

Endpoint

/v2.0/fwaas/firewall_rules

Response Codes

Success

200

Error

Unauthorized(401)

Example List firewall rules: JSON request

GET /v2.0/fwaas/firewall_rules.json
User-Agent: python-neutronclient
Accept: application/json

Example List firewall rules: JSON response

{
    "firewall_rules": [
        {
            "action": "ALLOW",
            "description": "",
            "enabled": true,
            "firewall_policy_id": "56632e51-d2aa-4b79-9fd4-45f51088c4ed",
            "id": "9faaf49f-dd89-4e39-a8c6-101839aa49bc",
            "name": "ALLOW_HTTP",
            "position": 1,
            "shared": false,
            "protocol": "tcp",
            "source_port": null,
            "destination_port": "80",
            "ip_version": 4,
            "source_ip_address": null,
            "destination_ip_address": null
            "source_address_group_ids": [],
            "destination_address_group_ids": ["8315762a-f0ae-4f6b-981a-a16a6c3103c2"],
            "project_id": "45977fa2dbd7482098dd68d0d8970117"
        }
    ]
}

Show firewall rule details

Shows firewall rule details.

Request Type

GET

Endpoint

/v2.0/fwaas/firewall_rules/<firewall_rule_id>

Response Codes

Success

200

Error

Unauthorized(401), Not Found (404)

Example Show firewall rule: JSON request

GET /v2.0/fwaas/firewall_rules/9faaf49f-dd89-4e39-a8c6-101839aa49bc.json
User-Agent: python-neutronclient
Accept: application/json

Example Show firewall rule: JSON response

{
    "firewall_rule": {
        "action": "ALLOW",
        "description": "",
        "enabled": true,
        "firewall_policy_id": "56632e51-d2aa-4b79-9fd4-45f51088c4ed",
        "id": "9faaf49f-dd89-4e39-a8c6-101839aa49bc",
        "name": "ALLOW_HTTP",
        "position": 1,
        "shared": false,
        "protocol": "tcp",
        "source_port": null,
        "destination_port": "80",
        "ip_version": 4,
        "source_ip_address": null,
        "destination_ip_address": null,
        "source_address_group_ids": [],
        "destination_address_group_ids": ["8315762a-f0ae-4f6b-981a-a16a6c3103c2"],
        "project_id": "45977fa2dbd7482098dd68d0d8970117"
    }
}

Create firewall rule

Creates a firewall rule.

Request Type

POST

Endpoint

/v2.0/fwaas/firewall_rules/

Response Codes

Success

201

Error

Unauthorized(401), Bad Request(400)

Example Create firewall rule: JSON request

POST /v2.0/fwaas/firewall_rules.json
User-Agent: python-neutronclient
Accept: application/json

{
    "firewall_rule": {
        "action": "ALLOW",
        "enabled": true,
        "name": "ALLOW_HTTP",
        "protocol": "tcp",
        "source_port": null,
        "destination_port": "80",
        "source_ip_address": null,
        "destination_ip_address": null,
        "source_address_group_ids": [],
        "destination_address_group_ids": ["8315762a-f0ae-4f6b-981a-a16a6c3103c2"]
    }
}

Example Create firewall rule: JSON response

HTTP/1.1 201 Created
Content-Type: application/json; charset=UTF-8

{
    "firewall_rule": {
        "action": "ALLOW",
        "description": "",
        "enabled": true,
        "firewall_policy_id": null,
        "id": "9faaf49f-dd89-4e39-a8c6-101839aa49bc",
        "name": "ALLOW_HTTP",
        "position": 1,
        "shared": false,
        "protocol": "tcp",
        "source_port": null,
        "destination_port": "80",
        "ip_version": 4,
        "source_ip_address": null,
        "destination_ip_address": null,
        "source_address_group_ids": [],
        "destination_address_group_ids": ["8315762a-f0ae-4f6b-981a-a16a6c3103c2"],
        "project_id": "45977fa2dbd7482098dd68d0d8970117"
    }
}

Update firewall rule

Updates a firewall rule.

Request Type

PUT

Endpoint

/v2.0/fwaas/firewall_rules/<firewall_rule_id>

Response Codes

Success

200

Error

Unauthorized(401), Bad Request(400) Not Found(404)

Example Update firewall rule: JSON request

PUT /v2.0/fwaas/firewall_rules/9faaf49f-dd89-4e39-a8c6-101839aa49bc.json
User-Agent: python-neutronclient
Accept: application/json

{
    "firewall_rule": {
        "shared": "true"
    }
}

Example Update firewall rule: JSON response

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8

{
    "firewall_rule": {
        "action": "ALLOW",
        "description": "",
        "enabled": true,
        "firewall_policy_id": null,
        "id": "9faaf49f-dd89-4e39-a8c6-101839aa49bc",
        "name": "ALLOW_HTTP",
        "position": 1,
        "shared": true,
        "protocol": "tcp",
        "source_port": null,
        "destination_port": "80",
        "ip_version": 4,
        "source_ip_address": null,
        "destination_ip_address": null,
        "source_address_group_ids": [],
        "destination_address_group_ids": ["8315762a-f0ae-4f6b-981a-a16a6c3103c2"],
        "project_id": "45977fa2dbd7482098dd68d0d8970117"
    }
}

Delete firewall rule

Deletes a firewall rule.

This operation does not return a response body.

Request Type

DELETE

Endpoint

/v2.0/fwaas/firewall_rules/<firewall_rule_id>

Response Codes

Success

204

Error

Unauthorized(401), Not Found(404) Conflict(409) The Conflict error response is returned when an operation is performed while firewall rule is in use.

Example Delete firewall rule: JSON request

DELETE /v2.0/fwaas/firewall_rules/9faaf49f-dd89-4e39-a8c6-101839aa49bc.json
User-Agent: python-neutronclient
Accept: application/json

Example Delete firewall rule: JSON response

HTTP/1.1 204 No Content
Content-Length: 0

Data Model Impact

The following are the backend database tables for the REST API proposed above.

Firewall Address Groups

Attribute	Type	Req	CRUD	Description
id	uuid-str	N/A	R	Unique identifier for the address_group object.
name	String	No	CRU	Human readable name for the address group (255 characters limit). Does not have to be unique.
description	String	No	CRU	Human readable description for the address group (255 characters limit).
project_id	uuid-str	Yes	CR	Owner of the address group. Only admin users can specify a project identifier other than their own.

Firewall Address Group Address associations

Attribute	Type	Req	CRUD	Description
id	uuid-str	N/A	R	Unique identifier for the address_group object.
firewall_address _group_id	uuid-str	No	CRU	UUID of firewall address group.
address	String	No	CRU	Address that has to be associated to the firewall address group.
ip_version	Integer	No	CRU	IP Protocol Version of the address.

Firewall Rules

Attribute	Type	Req	CRUD	Description
id	uuid-str	N/A	R	Unique identifier for the firewall rule object.
project_id	uuid-str	Yes	CR	Owner of the firewall rule. Only admin users can specify a project identifier other than their own.
name	String	No	CRU	Human readable name for the firewall rule (255 characters limit). Does not have to be unique.
description	String	No	CRU	Human readable description for the firewall Rule (255 characters limit).
shared	Bool	No	CRU	When set to True makes this firewall rule visible to projects other than its owner, and can be used in firewall policies not owned by its project.
protocol	String	No	CRU	IP Protocol.
source_port	port-range	No	CRU	Source port number or a range (an int in [1, 65535] or range in a:b).
destination_port	port-range	No	CRU	Destination port number or a range ( an int in [1, 65535] or range in a:b).
ip_version	Integer	No	CRU	IP Protocol Version.
source_ip_address	String	No	CRU	Source IP address or CIDR.
destination_ip_address	String	No	CRU	Destination IP address or CIDR.
source_address _group_ids	List	No	CRU	When a source_address_group is specified, it is matched when the source IP address in the packet matches one of the IP addresses in the address group.
destination_address _group_ids	List	No	CRU	When a destination_address_group is specified, it is matched when the destination IP address in the packet matches one of the IP addresses in the address group.
action	String	No	CRU	Action to be performed on the traffic matching the rule (ALLOW, DENY, REJECT). Default: DENY.
enabled	Bool	No	CRU	When set to False will disable this rule in the firewall policy. Facilitates selectively turning off rules without having to disassociate the rule from the firewall policy. Default: True.

Firewall Rules Source Address Group associations

Attribute	Type	Req	CRUD	Description
id	uuid-str	N/A	R	Unique identifier for the address_group object.
firewall_rule_id	uuid-str	No	CRU	UUID of firewall rule.
address_group_id	String	No	CRU	UUID of source address group.

Firewall Rules Destination Address Group associations

Attribute	Type	Req	CRUD	Description
id	uuid-str	N/A	R	Unique identifier for the address_group object.
firewall_rule_id	uuid-str	No	CRU	UUID of firewall rule.
address_group_id	String	No	CRU	UUID of destination address group.

Security Impact

None.

Notifications Impact

None.

Other End User Impact

None.

Performance Impact

None.

IPv6 Impact

None.

Other Deployer Impact

None.

Developer Impact

None.

Community Impact

None.

Alternatives

None.

Implementation

Assignee(s)

Wang Tao

Work Items

REST API
DB Schema
FWaaS plugin update
CLI update
L3 agent iptables driver
L2 agent ovs driver
FWaaS dashboard

Dependencies

Testing

Tempest Tests

DB mixin and schema tests
FWaaS Plugin with mocked driver end-to-end tests
Tempest tests
CLI tests

Functional Tests

New tests need to be written

API Tests

REST API and attributes validation tests

Documentation Impact

User Documentation

Neutron CLI and FWaaS API documentation have to be modified.

Developer Documentation

neutron-fwaas repo will have a devref and documentation will be written.

References

[1] https://specs.openstack.org/openstack/neutron-specs/specs/newton/fwaas-api-2.0.html

[2] https://developer.openstack.org/api-ref/network/v2/#fwaas-v2-0-current-fwaas-firewall-groups-firewall-policies-firewall-rules

Mitaka Postmortem documentation

Thu, 01 Mar 2018 00:00:00

Release Metrics

Release Metrics
Blueprints submitted	33
Blueprints implemented	21 (63%)
Bug reports submitted	1104
Bug reports closed	678 (61%)
Fixes released	538
Incomplete reports	59
RFEs submitted	33
RFEs approved	27 (81%)

Bug reports submitted: reports filed since Sept-23-2015 (Mitaka starts)
Bug reports closed: reports marked released, committed, invalid or wontfix
Fixes released: marked released/committed
Incomplete reports: marked incomplete

Note

Metrics accurate at the time of writing.

Blueprints

External DNS Resolution

Status: Complete
Assignee: minsel
Link: https://blueprints.launchpad.net/neutron/+spec/external-dns-resolution
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: Complete (Nova/Designate)
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Optional

Get Me a Network

Status: Complete
Assignee: gessau
Link: https://blueprints.launchpad.net/neutron/+spec/get-me-a-network
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: Incomplete (283206)
- OpenStack Infra support: N/A
- DevStack/Grenade support: Complete
- Horizon Support: Optional

API for l2 agent extensions

Status: Complete
Assignee: ihar-hrachyshka
Link: https://blueprints.launchpad.net/neutron/+spec/l2-api-extensions
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: N/A

Modular L2 Agent

Status: Complete
Assignee: andreas-scheuring
Link: https://blueprints.launchpad.net/neutron/+spec/modular-l2-agent
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: N/A
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: N/A

Support multiple L3 backends

Status: Deferred
Assignee: kevinbenton
Link: https://blueprints.launchpad.net/neutron/+spec/multi-l3-backends
FFE Status: Denied (Some experimental code may still make the release, but nothing production worthy)
- CLI support: Incomplete
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: Incomplete
- DevStack/Grenade support: Incomplete
- Horizon Support: N/A

Provide better user-facing mechanism to choose service capabilities

Status: Incomplete (Undocumented)
Assignee: dougwig
Link: https://blueprints.launchpad.net/neutron/+spec/neutron-flavor-framework
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: Complete
- Horizon Support: Optional

Split neutron into base library and servers/agents

Status: Ongoing effort
Assignee: dougwig
Link: https://blueprints.launchpad.net/neutron/+spec/neutron-lib
FFE Status: Denied (Goal for Mitaka was for lbaas to be fully severed, with fw/vpn to follow. We are not there. Work will be ongoing throughout Newton as well. The limited goal in Mitaka was completely severing lbaas. At present, the library exists and is plumbed throughout the infra. The first rev is being used by neutron and neutron-lbaas. Patches exist for bumping both to the second version of the lib. More patches exist to delete a lot of cruft from lbaas that will mean less to migrate, and plans are in place to stop the dependency on test code. The remaining items that were aimed at Mitaka but will miss are base db model/migration foo, and data model foo, both of which are ongoing, but neither of which needs to land in the critical end of mitaka timeframe (they can iterate in gerrit for now.) As soon as the Mitaka branch is baked, we can:
- turn on deprecation warnings
- nuke lbaas v1 and v2 agent
- start mass import renames in neutron
- merge the above db/model items
The goal of completely severing lbaas is realistically about 3-4 weeks away from Feature Freeze. The separation of the remaining subprojects are next after that. Attempting to get patches in gerrit for the lbaas separation goal before the summit.
- CLI support: N/A
- Server/Agent support: See FFE notes
- Testing coverage: See FFE notes
- Documentation: Incomplete
- Advanced/Sub-project support: See FFE notes
- Other Projects support: See FFE notes
- OpenStack Infra support: Complete
- DevStack/Grenade support: Complete
- Horizon Support: N/A

Use push style notifications for all server->agent information

Status: Deferred
Assignee: kevinbenton
Link: https://blueprints.launchpad.net/neutron/+spec/push-notifications
FFE Status: Denied (minor enhancemetns may be allowed as RC bugs, e.g. 280595).
- CLI support: N/A
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: N/A
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: N/A

Restructure L2 agent

Status: Complete
Assignee: rossella-o
Link: https://blueprints.launchpad.net/neutron/+spec/restructure-l2-agent
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: N/A
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: N/A

Clean up resources when a tenant is deleted

Status: Complete
Assignee: john-davidge
Link: https://blueprints.launchpad.net/neutron/+spec/tenant-delete
FFE Status: Granted
- CLI support: Complete
- Server/Agent support: N/A
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: Optional
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Optional

Add availability zones for agents

Status: Complete
Assignee: ichihara-hirofumi
Link: https://blueprints.launchpad.net/neutron/+spec/add-availability-zone
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Optional

Add tags to core resources

Status: Complete
Assignee: ichihara-hirofumi
Link: https://blueprints.launchpad.net/neutron/+spec/add-tags-to-core-resources
FFE Status: Granted (docs pending)
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Optional

Add timestamp to neutron resources

Status: Complete
Assignee: zhaobo6
Link: https://blueprints.launchpad.net/neutron/+spec/add-timestamp-attr
FFE Status: Granted
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: N/A
- Advanced/Sub-project support: Optional
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: N/A

Introduce Address Scopes

Status: Complete
Assignee: carl-baldwin
Link: https://blueprints.launchpad.net/neutron/+spec/address-scopes
FFE Status: Granted (docs pending)
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Optional
References
- https://review.openstack.org/#/c/286294/

Automatically generate etc/neutron.conf file

Status: Complete
Assignee: martin-hickey
Link: https://blueprints.launchpad.net/neutron/+spec/autogen-neutron-conf-file
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: Complete
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: Complete
- Horizon Support: N/A
References:
- http://docs.openstack.org/draft/config-reference/networking/samples/
- http://docs.openstack.org/developer/neutron/devref/contribute.html#configuration-files

Router Extension for Dynamic Routing Using BGP

Status: Complete
Assignee: ryan-tidwell
Link: https://blueprints.launchpad.net/neutron/+spec/bgp-dynamic-routing
FFE Status: Granted
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: Complete
- Horizon Support: Optional
References:

Allow for per-subnet dhcp options

Status: Deferred
Assignee: sambetts
Link: https://blueprints.launchpad.net/neutron/+spec/dhcp-options-per-subnet
FFE Status: Denied (Unable to determine status)
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack/Grenade support:
- Horizon Support:

Add Guru Meditation Report Functionality to Neutron

Status: Incomplete
Assignee: ihar-hrachyshka
Link: https://blueprints.launchpad.net/neutron/+spec/guru-meditation-report
FFE Status: Granted (to provide guru support to vpn/lb/fwass)
- CLI support: N/A
- Server/Agent support: N/A
- Testing coverage: Complete
- Documentation: Incomplete
- Advanced/Sub-project support: Incomplete
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: Incomplete (279035 needs review)
- Horizon Support: N/A
References:
- https://review.openstack.org/#/c/287795/
- https://review.openstack.org/#/c/287801/

Improve DVR router sheduling mechanism for better performance/scalability

Status: Complete
Assignee: obondarev
Link: https://blueprints.launchpad.net/neutron/+spec/improve-dvr-l3-agent-binding
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: N/A
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: N/A

Moving to Keystone v3

Status: Deferred
Assignee: smigiel-dariusz
Link: https://blueprints.launchpad.net/neutron/+spec/keystone-v3
FFE Status: Denied (Neutron Server supports v3, but schema and API migration is still undergoing).
- CLI support: Incomplete
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: Incomplete (281357 needs work).
- Advanced/Sub-project support: Incomplete
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: Optional
- Horizon Support: N/A

LBaaS Layer 7 rules

Status: Incomplete (Undocumented)
Assignee: avishayb
Link: https://blueprints.launchpad.net/neutron/+spec/lbaas-l7-rules
FFE Status: Granted (docs pending)
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Incomplete
- Advanced/Sub-project support: Complete
- Other Projects support: Complete (Octavia)
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Incomplete

ML2/LinuxBridge QoS support with bandwidth limiting

Status: Complete
Assignee: slaweq
Link: https://blueprints.launchpad.net/neutron/+spec/ml2-lb-ratelimit-support
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: Optional
- DevStack/Grenade support: Complete
- Horizon Support: N/A

Adds a network-ip-usage api to fetch network and subnet IP usage counts

Status: Complete
Assignee: mdorman-m
Link: https://blueprints.launchpad.net/neutron/+spec/network-ip-usage-api
FFE Status: Granted
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Incomplete
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Optional

Role-based Access Control for QoS policies

Status: Complete
Assignee: hdaniel
Link: https://blueprints.launchpad.net/neutron/+spec/rbac-qos
FFE Status: Granted
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Optional

enable vhost-user support with ovs agent

Status: Complete
Assignee: sean-k-mooney
Link: https://blueprints.launchpad.net/neutron/+spec/vhost-ovs
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete (user guide desirable)
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: N/A
References
- http://docs.openstack.org/developer/neutron/devref/ovs_vhostuser.html

Allow vm to boot without l3 address(subnet)

Status: Deferred
Assignee: yalei-wang
Link: https://blueprints.launchpad.net/neutron/+spec/vm-without-l3-address
FFE Status: Denied (vm-without-l3-address shows a post-nuclear landscape).
- CLI support: Incomplete
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: Incomplete (Nova)
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Optional
References:
- https://review.openstack.org/#/c/239276/
- https://review.openstack.org/#/q/topic:bp/vm-without-l3-address

allow multiple subnets to connect to vpn

Status: Complete
Assignee: pcm
Link: https://blueprints.launchpad.net/neutron/+spec/vpn-multiple-subnet
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Optional

RFE

Openstack services should support SIGHUP signal

Status: Incomplete
Assignee: eezhova
Link: https://bugs.launchpad.net/neutron/+bug/1276694
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack/Grenade support:
- Horizon Support:

[RFE] Neutron support for OSprofiler

Status: Deferred
Assignee: salvatore-orlando
Link: https://bugs.launchpad.net/neutron/+bug/1335640
- CLI support: N/A
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project support: Incomplete
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: N/A
References
- https://review.openstack.org/#/c/273951/

[RFE] Unable to create a router thats both HA and distributed

Status: Complete
Assignee: carl-baldwin
Link: https://bugs.launchpad.net/neutron/+bug/1365473
FFE Status: Granted (needs docs)
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: Incomplete
- DevStack/Grenade support: N/A
- Horizon Support: Optional
References
- https://review.openstack.org/#/c/296711/
- https://review.openstack.org/#/c/296836/

[RFE] [LBaaS] ssh connection timeout

Status: Deferred
Assignee: reedip-banerjee
Link: https://bugs.launchpad.net/neutron/+bug/1457556
FFE Status: Denied (no tangible progress)
- CLI support: Incomplete
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: Incomplete
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Optional

[rfe] openvswitch based firewall driver

Status: Complete (partially documented)
Assignee: libosvar
Link: https://bugs.launchpad.net/neutron/+bug/1461000
FFE Status: Granted
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Incomplete (user guide docs desirable)
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: Optional
- Horizon Support: N/A
References:

[RFE] Create a full load balancing configuration with one API call

Status: Deferred
Assignee: trevor-vardeman
Link: https://bugs.launchpad.net/neutron/+bug/1463202
FFE Status: Denied (too many missing pieces).
- CLI support: Incomplete
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Incomplete

RFE: Security Rules should support VRRP protocol

Status: Complete
Assignee: sreesiv
Link: https://bugs.launchpad.net/neutron/+bug/1475717
- CLI support: Optional
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: N/A
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Optional

[RFE] Allow annotations on Neutron resources

Status: Complete
Assignee: kevinbenton
Link: https://bugs.launchpad.net/neutron/+bug/1483480
FFE Status: Granted
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: N/A
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Optional

[RFE] DHCP agent should provide ipv6 RAs for isolated networks with ipv6 subnets

Status: Deferred
Assignee: ihar-hrachyshka
Link: https://bugs.launchpad.net/neutron/+bug/1498987
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack/Grenade support:
- Horizon Support:

RFE Replace the existing default subnetpool configuration options with an admin-only API

Status: Complete
Assignee: john-davidge
Link: https://bugs.launchpad.net/neutron/+bug/1501328
FFE Status: Granted (docs pending)
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Optional
References
- https://review.openstack.org/#/c/286293/

[RFE] Improve SG performance as VMs/containers scale on compute node

Status: Complete
Assignee: kevinbenton
Link: https://bugs.launchpad.net/neutron/+bug/1502297
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: N/A
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: N/A

RFE There is no facility to name LBaaS v2 Members and Health Monitors

Status: Incomplete (undocumented)
Assignee: reedip-banerjee
Link: https://bugs.launchpad.net/neutron/+bug/1515506
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Optional

[RFE] IPAM migration from non-pluggable to pluggable

Status: Deferred
Assignee: pasha117
Link: https://bugs.launchpad.net/neutron/+bug/1516156
- CLI support: N/A (server side entry point)
- Server/Agent support: N/A
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: Incomplete
- DevStack/Grenade support: Incomplete
- Horizon Support: Incomplete
References:

https://review.openstack.org/#/c/277767

[RFE] Transition neutron CLI from python-neutronclient to python-openstackclient

Status: Deferred
Assignee: rtheis
Link: https://bugs.launchpad.net/neutron/+bug/1521291
FFE Status: Denied (even though Devref patch will likely merge, the actual work will need to move to Newton. OSC in Mitaka will have a significant increase in support for core neutron resources Detailed status available at https://etherpad.openstack.org/p/osc-neutron-support.
- CLI support: Incomplete
- Server/Agent support: N/A
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project support: Incomplete
- Other Projects support: Incomplete
- OpenStack Infra support: N/A
- DevStack/Grenade support: Incomplete
- Horizon Support: N/A

RfE: Cascading delete for LBaaS Objects

Status: Deferred
Assignee: german-eichberger
Link: https://bugs.launchpad.net/neutron/+bug/1521783
FFE Status: Denied
- CLI support: Incomplete
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: Complete
References
- https://review.openstack.org/#/c/287593/
- https://review.openstack.org/#/c/288187/

[RFE] use oslo-versioned-objects to help with dealing with upgrades

Status: Deferred
Assignee: justin-hammond
Link: https://bugs.launchpad.net/neutron/+bug/1522102
FFE Status: Denied (Wont happen in Mitaka. Should be moved to Newton. Was not expected to land in Mitaka, the bug is just a placeholder for a large effort).
- CLI support: N/A
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project support: Incomplete
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: N/A

RFE Add support for external vxlan encapsulation to neutron router

Status: Deferred
Assignee: ruansx
Link: https://bugs.launchpad.net/neutron/+bug/1525059
FFE Status: Denied (too many parts lacking exhaustive review)
- CLI support: Incomplete
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project supprt: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: Optional
- Horizon Support: Optional
References:
- https://review.openstack.org/#/q/topic:bug/1525059

[RFE] Security groups resources are not extendable

Status: Complete
Assignee: roeyc
Link: https://bugs.launchpad.net/neutron/+bug/1529109
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: N/A
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack/Grenade support: N/A
- Horizon Support: N/A

RFE Add F5 plugin driver to neutron-lbaas

Status: Deferred
Assignee: j-longstaff
Link: https://bugs.launchpad.net/neutron/+bug/1539717
FFE Status: Denied (no met requirements for inclusion)
- CLI support: N/A
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: N/A
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: Incomplete
- DevStack/Grenade support: N/A
- Horizon Support: N/A

Newton Postmortem documentation

Thu, 01 Mar 2018 00:00:00

Release Metrics

Release Metrics
Blueprints targeted	25
Blueprints/RFE implemented	14 (56%)
Bug reports submitted	789
Bug reports closed	428 (54%)
Fixes released	315 (40%)
Incomplete reports	51
RFEs submitted	49

Bug reports submitted: reports filed since Mar-16-2016 (Newton starts)
Bug reports closed: reports marked released, committed, invalid or wontfix
Fixes released: marked released/committed
Incomplete reports: marked incomplete

Note

Metrics accurate at the time of writing.

Blueprints

Firewall as a Service API 2.0

Status: Incomplete
Assignee: skandasw
Link: https://blueprints.launchpad.net/neutron/+spec/fwaas-api-2.0
FFE: Granted (testing and documentation lacking)
- CLI support: Incomplete
- Server/Agent support: Complete (see release notes for more details)
- Testing coverage: Incomplete (unit testing only)
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: Incomplete
- DevStack support: Complete
- Horizon Support: Optional
References

API for l2 agent extensions

Status: Incomplete
Assignee: mangelajo
Link: https://blueprints.launchpad.net/neutron/+spec/l2-api-extensions
FFE: Denied (still at specification stage)
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:
References
- Spec: https://review.openstack.org/#/c/320439/

Support multiple L3 backends

Status: Complete (pending documentation)
Assignee: kevinbenton
Link: https://blueprints.launchpad.net/neutron/+spec/multi-l3-backends
FFE: Granted (small testing/devref gaps are being filled)
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete (Init, API)
- Documentation: In progress
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: Optional
References
- https://review.openstack.org/#/c/364001/
- https://review.openstack.org/#/c/358866/

Split neutron into base library and servers/agents

Status: Ongoing
Assignee: dougwig
Link: https://blueprints.launchpad.net/neutron/+spec/neutron-lib
FFE: N/A
- CLI support: N/A
- Server/Agent support: N/A
- Testing coverage: N/A
- Documentation: Complete
- Advanced/Sub-project support: Ongoing
- Other Projects support: N/A
- OpenStack Infra support: Complete
- DevStack support: Complete
- Horizon Support: N/A
References
- http://docs.openstack.org/developer/neutron-lib/

Upgrade controllers with no API downtime

Status: Incomplete
Assignee: ihar-hrachyshka
Link: https://blueprints.launchpad.net/neutron/+spec/online-upgrades
FFE: Denied: no work happened, will be more active in Ocata; has a dependency on adopt-oslo-versioned-objects-for-db.
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

Use push style notifications for all server->agent information

Status: Incomplete (>50% complete - to land early in Ocata-1)
Assignee: kevinbenton
Link: https://blueprints.launchpad.net/neutron/+spec/push-notifications
FFE: Denied (due to incomplete OVO refactoring).
- CLI support: N/A
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: N/A

Support Routed Networks in Neutron

Status: Incomplete (pending client and Nova support).
Assignee: carl-baldwin
Link: https://blueprints.launchpad.net/neutron/+spec/routed-networks
FFE: Granted
- CLI support: OSC bindings incomplete
- Server/Agent support: Complete
- Testing coverage: (unit, more in progress)
- Documentation: In progress
- Advanced/Sub-project support: N/A
- Other Projects support: Nova scheduler support is incomplete.
- OpenStack Infra support: N/A
- DevStack support: Complete
- Horizon Support: Optional
References

Diagnostics of Neutron components

Status: Incomplete
Assignee: hmlnarik-s
Link: https://blueprints.launchpad.net/neutron/+spec/troubleshooting
FFE: Denied (still at specification stage)
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:
References
- https://review.openstack.org/#/c/308973/

VLAN aware VMs

Status: Complete (pending documentation)
Assignee: rossella-o
Link: https://blueprints.launchpad.net/neutron/+spec/vlan-aware-vms
FFE: Granted (OVS and Linuxbridge agent-side patches need merging but are moving at fast pace, and the bulk has already merged in a while; small gaps to fill after that).
- CLI support: Complete
- Server/Agent support: Complete (pending LB+OVS agent patches)
- Testing coverage: Complete (unit, functional, API)
- Documentation: In progress
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: Complete
- Horizon Support: Optional
References

Adopt oslo.versionedobjects for database interactions

Status: Incomplete
Assignee: ihar-hrachyshka
Link: https://blueprints.launchpad.net/neutron/+spec/adopt-oslo-versioned-objects-for-db
FFE: Granted (Slipping into Ocata)
- CLI support: N/A
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project support: Incomplete
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: N/A

BGP Dynamic Routing spin out

Status: Complete
Assignee: vikschw
Link: https://blueprints.launchpad.net/neutron/+spec/bgp-spinout
FFE: Granted
- CLI support: Complete (OSC bindings incomplete)
- Server/Agent support: Complete
- Testing coverage: Complete (unit, API, functional)
- Documentation: Complete
- Advanced/Sub-project support: Complete
- Other Projects support: N/A
- OpenStack Infra support: Complete
- DevStack support: Complete
- Horizon Support: Optional
References
- https://review.openstack.org/#/c/340763/

Allow for per-subnet dhcp options

Status: Incomplete
Assignee: sambetts
Link: https://blueprints.launchpad.net/neutron/+spec/dhcp-options-per-subnet
FFE: Denied
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

Use the new enginefacade from oslo_db

Status: Incomplete (>50% complete)
Assignee: akamyshnikova
Link: https://blueprints.launchpad.net/neutron/+spec/enginefacade-switch
FFE: Granted (bulk of the code to enable adoption of new engine facade merged. There are more follow ups to go in Ocata).
- CLI support: N/A
- Server/Agent support: N/A
- Testing coverage: Complete (unit, functional)
- Documentation: Incomplete
- Advanced/Sub-project support: Incomplete
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: N/A

Allow instance-ingress bandwidth limiting

Status: Incomplete
Assignee: slaweq
Link: https://blueprints.launchpad.net/neutron/+spec/instance-ingress-bw-limit
FFE: Denied (a few patches in conflict/stale).
- CLI support: Incomplete
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: N/A
References

Moving to Keystone v3

Status: Complete (pending documentation and codebase cleanup)
Assignee: smigiel-dariusz
Link: https://blueprints.launchpad.net/neutron/+spec/keystone-v3
FFE: Granted (to provide API support).
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete (Unit, functional, API)
- Documentation: Incomplete (api-ref to be updated)
- Advanced/Sub-project support: Complete (deprecation warnings emitted)
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: N/A
References
- https://review.openstack.org/#/c/357977/
- https://review.openstack.org/#/c/372857/

Add agent extension framework for L3 agent

Status: Complete
Assignee: njohnston
Link: https://blueprints.launchpad.net/neutron/+spec/l3-agent-extensions
FFE: Granted (bulk of functionality is merged, increased coverage, or more documentation should be allowed to go in).
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete (unit)
- Documentation: Complete
- Advanced/Sub-project support: Complete
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: Complete
- Horizon Support: N/A
References
- http://docs.openstack.org/developer/neutron/devref/agent_extensions.html

QoS support with dscp marking

Status: Complete
Assignee: victor-r-howard
Link: https://blueprints.launchpad.net/neutron/+spec/ml2-ovs-qos-with-dscp
- CLI support: Complete (from python-neutronclient 4.2.0)
- Server/Agent support: Complete
- Testing coverage: Complete (unit, API, functional, fullstack)
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: Complete
- DevStack support: Complete
- Horizon Support: Optional
References

Neutron in-tree API reference

Status: Ongoing
Assignee: amotoki
Link: https://blueprints.launchpad.net/neutron/+spec/neutron-in-tree-api-ref
- CLI support: N/A
- Server/Agent support: N/A
- Testing coverage: N/A
- Documentation: In progress
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: Complete
- DevStack support: N/A
- Horizon Support: N/A
References
- http://developer.openstack.org/api-ref/networking/

QoS minimum egrees bandwidth

Status: Complete (pending documentation)
Assignee: rodolfo-alonso-hernandez
Link: https://blueprints.launchpad.net/neutron/+spec/qos-min-egress-bw
FFE: Granted (close to being complete, OVS and Linuxbridge missing the implementation).
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete (no fullstack testing)
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: Optional
Reference

Improved validation mechanism for QoS rules with port types

Status: Incomplete
Assignee: slaweq
Link: https://blueprints.launchpad.net/neutron/+spec/qos-rules-validation
FFE: Denied (requires more work that will slip into Ocata)
- CLI support: N/A
- Server/Agent support: Incomplete
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: N/A
Rerefences
- https://review.openstack.org/#/c/319694/
- https://review.openstack.org/#/c/351858/

Security-group Logging

Status: Incomplete
Assignee: y-furukawa-2
Link: https://blueprints.launchpad.net/neutron/+spec/security-group-logging
FFE: Denied (still at specification stage)
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:
References
- https://review.openstack.org/#/c/203509/

Differentiate between service and floating subnets

Status: Complete (pending client support)
Assignee: john-davidge
Link: https://blueprints.launchpad.net/neutron/+spec/service-subnets
FFE: Granted (nearly complete, pending CLI)
- CLI support: Incomplete
- Server/Agent support: Complete
- Testing coverage: Complete (unit)
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: Optional
References
- http://docs.openstack.org/draft/networking-guide/config-service-subnets.html
- https://review.openstack.org/#/c/360526/

Enable adoption of an existing subnet into a subnetpool

Status: Incomplete
Assignee: ryan-tidwell
Link: https://blueprints.launchpad.net/neutron/+spec/subnet-onboard
FFE: Denied (server side patch needs some love).
- CLI support: Incomplete
- Server/Agent support: In progress
- Testing coverage: In progress
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: Optional
References
- https://review.openstack.org/#/c/348080/

Allow vm to boot without l3 address(subnet)

Status: Incomplete (>50% complete)
Assignee: carl-baldwin
Link: https://blueprints.launchpad.net/neutron/+spec/vm-without-l3-address
FFE: Granted (patch actively under review)
- CLI support: Incomplete (allow creation of ports with no fixed IPs)
- Server/Agent support: Complete
- Testing coverage: Incomplete (test how security groups behave with an unaddressed ports)
- Documentation: In progress
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: Optional
References
- https://review.openstack.org/#/c/361455

add-neutron-extension-resource-timestamp

Status: Complete
Assignee: zhaobo
Link: https://blueprints.launchpad.net/neutron/+spec/add-neutron-extension-resource-timestamp
FFE: Granted (cleanup/refactoring patches from kevinbenton)
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: N/A
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: Optional

RFEs

Openstack services should support SIGHUP signal

Status: Incomplete
Assignee: eezhova
Link: https://bugs.launchpad.net/neutron/+bug/1276694
FFE: Denied
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

[RFE] Neutron support for OSprofiler

Status: Complete
Assignee: dbelova
Link: https://bugs.launchpad.net/neutron/+bug/1335640
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete (release notes)
- Advanced/Sub-project support: Complete
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: N/A

[RFE] [LBaaS] ssh connection timeout

Status: Incomplete
Assignee: reedip-banerjee
Link: https://bugs.launchpad.net/neutron/+bug/1457556
FFE: Denied (not progress, not worked on)
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

[RFE] Add API to set ipv6 gateway

Status: Incomplete
Assignee: scollins
Link: https://bugs.launchpad.net/neutron/+bug/1460720
FFE: Denied (not progress, not worked on)
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

[RFE] Create a full load balancing configuration with one API call

Status: Incomplete
Assignee: trevor-vardeman
Link: https://bugs.launchpad.net/neutron/+bug/1463202
FFE: Granted (to fill testing/documentation gaps).
- CLI support: Incomplete
- Server/Agent support: Complete
- Testing coverage: Complete (unit)
- Documentation: Incomplete
- Advanced/Sub-project support: Complete
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: Optional

[RFE] Add the ability to create lb vip and member with network_id

Status: Incomplete
Assignee: dougwig
Link: https://bugs.launchpad.net/neutron/+bug/1465758
FFE: Granted
- CLI support: Incomplete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: Optional
References
- https://review.openstack.org/#/c/363302/

RFE: Pure Python driven Linux network configuration

Status: Incomplete
Assignee: gus
Link: https://bugs.launchpad.net/neutron/+bug/1492714
FFE: Denied (not progress, not worked on).
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

[RFE] DHCP agent should provide ipv6 RAs for isolated networks with ipv6 subnets

Status: Incomplete
Assignee: None
Link: https://bugs.launchpad.net/neutron/+bug/1498987
FFE: Denied (not progress, not worked on).
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

[RFE] IPAM migration from non-pluggable to pluggable

Status: Complete (pending documentation)
Assignee: carl-baldwin
Link: https://bugs.launchpad.net/neutron/+bug/1516156
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: N/A
References
- https://bugs.launchpad.net/neutron/+bug/1588984

[RFE] Transition neutron CLI from python-neutronclient to python-openstackclient

Status: Ongoing
Assignee: rtheis
Link: https://bugs.launchpad.net/neutron/+bug/1521291
FFE: N/A (OSC in Mitaka has a significant increase in support for core Neutron resources. In addition, there is now an OSC plugin for some advanced neutron features and services. The overall status is available in the docs with a detailed status available on etherpad.
- CLI support: Incomplete
- Server/Agent support: N/A
- Testing coverage: Incomplete
- Documentation: Incomplete
- Advanced/Sub-project support: Incomplete
- Other Projects support: Incomplete
- OpenStack Infra support: Complete
- DevStack support: Incomplete
- Horizon Support: N/A

[RFE] Cascading delete for LBaaS Objects

Status: Incomplete
Assignee: brandon-logan
Link: https://bugs.launchpad.net/neutron/+bug/1521783
FFE: Denied (not currently worked on).
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

[RFE] Add support for external vxlan encapsulation to neutron router

Status: Incomplete
Assignee: ruansx
Link: https://bugs.launchpad.net/neutron/+bug/1525059
FFE: Denied
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

ML2 OpenvSwitch Agent GRE/VXLAN tunnel code does not support IPv6 addresses as tunnel endpoints

Status: Complete
Assignee: brian-haley
Link: https://bugs.launchpad.net/neutron/+bug/1525895
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: Complete (dsvm-neutron-serviceipv6 experimental job)
- DevStack support: Incomplete
- Horizon Support: N/A
References
- http://docs.openstack.org/draft/networking-guide/config-ipv6.html
- https://bugs.launchpad.net/devstack/+bug/1619476

[RFE] Add F5 plugin driver to neutron-lbaas

Status: Incomplete
Assignee: None
Link: https://bugs.launchpad.net/neutron/+bug/1539717
FFE: Denied (no active progress)
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

Miscellaneous

Switch to OVSDB and OpenFlow native interfaces.
- Status: Complete
- Assignee: Terry Wilson, IWAMOTO Toshihiro
- References:
  - https://review.openstack.org/#/c/299655/ (ovsdb)
  - https://review.openstack.org/#/c/319770/ (openflow)
- Agent support: Complete
- Testing coverage: Complete (Functional tests show parity, Tempest API and scenario tests use the new default)
- Documentation: config options being documented.

Ocata Postmortem documentation

Thu, 01 Mar 2018 00:00:00

Release Metrics

Release Metrics
Blueprints targeted	15
Blueprints/RFE implemented	4
Bug reports submitted	556
Bug reports closed	334
Fixes released	225
Incomplete reports	29
RFEs submitted	22

Bug reports submitted: reports filed since Sep-16-2016 (Ocata starts)
Bug reports closed: reports marked released, committed, invalid or wontfix
Fixes released: marked released/committed
Incomplete reports: marked incomplete

Note

Metrics accurate at the time of writing.

Blueprints

Adopt oslo.versionedobjects for database interactions

Status: Incomplete
Assignee: ihar-hrachyshka
Link: https://blueprints.launchpad.net/neutron/+spec/adopt-oslo-versioned-objects-for-db
- CLI support: N/A
- Server/Agent support: Partially implemented
- Testing coverage: unit, API, functional coverage
- Documentation: developer documentation (https://review.openstack.org/#/c/336518/)
- Advanced/Sub-project support: out of scope
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: N/A

Split neutron into base library and servers/agents

Status: Complete (the process to develop and consume neutron-lib).
Assignee: boden
Link: https://blueprints.launchpad.net/neutron/+spec/neutron-lib
- CLI support: N/A
- Server/Agent support: Complete
- Testing coverage: N/A
- Documentation: Complete
- Advanced/Sub-project support: Complete
- Other Projects support:
- OpenStack Infra support: Complete
- DevStack support: N/A
- Horizon Support: N/A

Upgrade controllers with no API downtime

Status: Deferred
Assignee: ihar-hrachyshka
Link: https://blueprints.launchpad.net/neutron/+spec/online-upgrades
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

Use push style notifications for all server->agent information

Status: Incomplete
Assignee: kevinbenton
Link: https://blueprints.launchpad.net/neutron/+spec/push-notifications
FFE Status: Granted
- CLI support: N/A
- Server/Agent support: Server support partially merged. L2 agent support pending reviews. In particular, server generates notifications for all L2 components (port, networks, etc) and security groups. The only remaining server component waiting to merge is a way to query for multiple OVO objects.
- Testing coverage: code has UTs and is executed as part of tempest runs.
- Documentation: N/A
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: N/A

Support Routed Networks in Neutron

Status: Complete
Assignee: minsel
Link: https://blueprints.launchpad.net/neutron/+spec/routed-networks
FFE status: Granted
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Coverage, except scenario coverage.
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: We got all the functionality we needed from Nova to implement the updating of routed networks segments IPv4 addresses inventories in the placement API. The Nova scheduler will not be able to use that information yet to place instances. Nova will implement that during Pike.
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: N/A

Support agentless driver in neutron-dynamic-routing

Status: Deferred
Assignee: yuyangbj
Link: https://blueprints.launchpad.net/neutron/+spec/agentless-driver
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

Use the new enginefacade from oslo_db

Status: Ongoing
Assignee: akamyshnikova
Link: https://blueprints.launchpad.net/neutron/+spec/enginefacade-switch
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

Firewall as a Service API 2.0

Status: Deferred
Assignee: nate-johnston
Link: https://blueprints.launchpad.net/neutron/+spec/fwaas-api-2.0
- CLI support: Complete
- Server/Agent support: missing two neutron server side patches
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: Complete
- DevStack support: Complete
- Horizon Support: Incomplete

Moving to Keystone v3

Status: Complete
Assignee: smigiel-dariusz
Link: https://blueprints.launchpad.net/neutron/+spec/keystone-v3
- CLI support: Complete
- Server/Agent support: Complete
- Testing coverage: Complete
- Documentation: Complete
- Advanced/Sub-project support: Complete
- Other Projects support: N/A
- OpenStack Infra support: Complete
- DevStack support: Complete
- Horizon Support: N/A

API for l2 agent extensions (+ovs flow management)

Status: Deferred (spec under discussion)
Assignee: david-shaughnessy
Link: https://blueprints.launchpad.net/neutron/+spec/l2-api-extensions
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

nova portbinding information for live migration

Status: Deferred (spec merged).
Assignee: anindita-das
Link: https://blueprints.launchpad.net/neutron/+spec/live-migration-portbinding
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

Neutron in-tree API reference

Status: Ongoing
Assignee: amotoki
Link: https://blueprints.launchpad.net/neutron/+spec/neutron-in-tree-api-ref
- CLI support: N/A
- Server/Agent support: N/A
- Testing coverage: N/A
- Documentation: Complete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: Complete
- DevStack support:
- Horizon Support:

Port data plane status

Status: Deferred (spec approved).
Assignee: cgoncalves
Link: https://blueprints.launchpad.net/neutron/+spec/port-data-plane-status
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

Security-group Logging

Status: Deferred (spec in progress).
Assignee: y-furukawa-2
Link: https://blueprints.launchpad.net/neutron/+spec/security-group-logging
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

Diagnostics of Neutron components

Status: Deferred
Assignee: boden
Link: https://blueprints.launchpad.net/neutron/+spec/troubleshooting
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

RFEs

RFE: Pure Python driven Linux network configuration

Status: Incomplete
Assignee: omer-anson
Link: https://bugs.launchpad.net/neutron/+bug/1492714
- CLI support: N/A
- Server/Agent support: Only partially complete
- Testing coverage:
- Documentation:
- Advanced/Sub-project support: Incomplete
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: N/A

[RFE] Transition neutron CLI from python-neutronclient to python-openstackclient

Status: Incomplete
Assignee: amotoki
Link: https://bugs.launchpad.net/neutron/+bug/1521291
- CLI support: Incomplete
- Server/Agent support: N/A
- Testing coverage: N/A
- Documentation: Incomplete
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: Complete
- Horizon Support: Incomplete

[RFE] No notification on floating ip status change

Status: Incomplete
Assignee: boden
Link: https://bugs.launchpad.net/neutron/+bug/1593793
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

relocate model definitions

Status: Complete
Assignee: manjeet-s-bhatia
Link: https://bugs.launchpad.net/neutron/+bug/1597913
- CLI support: N/A
- Server/Agent support: N/A
- Testing coverage: N/A
- Documentation: https://github.com/openstack/neutron/blob/master/doc/source/devref/db_models.rst
- Advanced/Sub-project support: N/A
- Other Projects support: N/A
- OpenStack Infra support: N/A
- DevStack support: N/A
- Horizon Support: N/A

Support for DSCP marking in Linuxbridge L2 agent

Status:
Assignee: slaweq
Link: https://bugs.launchpad.net/neutron/+bug/1644369
- CLI support:
- Server/Agent support:
- Testing coverage:
- Documentation:
- Advanced/Sub-project support:
- Other Projects support:
- OpenStack Infra support:
- DevStack support:
- Horizon Support:

Floating IP rate limit

Thu, 01 Mar 2018 00:00:00

RFE: https://bugs.launchpad.net/neutron/+bug/1596611

Floating IP bandwidth is unrestricted now. But the NIC and the export of data center may have a limitation for a cloud deployment. Then floating IP rate limit is needed.

This spec describes how to add a built-in extension to limit the floating IP bandwidth.

Problem Description

Neutron now has floating IP whose bandwidth is not restricted, then there are several reasons for adding rate limit to floating IP:

a) Currently, neutron QoS implementation only affects neutron ports, more detail is that the bandwidth restriction is based on the neutron port, so all the VM traffic will be limited under that restriction, including L3 traffic. L3 needs its IP based implementation.
1. North/South traffic always rely on infrastructure networking capacity.
c) In case of Floating IP Traffic, cloud deployment do not have enough bandwidth to meet the total bandwidth requirement of all tenants at the same time. Due to this, the SLA of user’s network traffic cannot be guaranteed. For example, if the VM NICs are given a high QoS value, then to meet the requirement of each tenant, the DC network would have a heavy load, thus it may be unable to facilitate additional requests and affect not only the tenant network bandwidth from North-South, but also East-West traffic.
d) SNAT traffic in centralized network node will not meet the needs of all tenant bandwidth either. Because a NIC has limited bandwidth.

Proposed Change

Overview

We want to limit the bandwidth of floating IP. At the same time, the east/west traffic should have no effect.

Some agreements:

Allow binding QoS policy to floating IP.
Each floating IP should have only one QoS policy.
L3 agent needs a QoS extension to process L3 IP related QoS rules.
QoS policy parameter is nullable. Floating IP can have no QoS policy assigned.

How the change works:

Linux TC (Traffic Control) [1] will be used to implement such functionality. And for egress traffic a HTB [2] qdisc will be used to limit the floating IP outgoing bandwidth.

Solution Proposed

Server side changes

New floating IP extension for QoS attribute (qos_policy_id).
Adding a new relationship table between floating IP and QoS policy QoSFIPPolicyBinding.
L3 plugin will return the floating IP list with its bonding QoS policy during router sync RPC.

L3 agent side TC rules

Where to install the TC rules:

For HA/legacy routers, rules for floating IP will be installed into network node qrouter namespace, and the traffic control device will be qg-device.
For DVR routers, it’s compute node qrouter namespace, but the floating IP traffic control device will be rfp-device (one of qrouter-namespace to fip-namespace pair).

For all IPs, in the corresponding namespace, the egress and ingress qdisc rules are basically the same. The {qdisc_id} will be used to create the IPs’ tc filter. Some tips:

Remember all the following commands will be executed in a namespace.
[a-device] represents the qg-device or rfp-device in a namespace.
{egress/ingress_qdisc_id} is the qdisc handle id.

Create qdisc commands:

# egress
tc qdisc add dev [a-device] root htb

# ingress
tc qdisc add dev [a-device] ingress

Actual qdisc rules:

$ tc qdisc show dev [a-device]
# egress
qdisc htb {egress_qdisc_id} root refcnt 2 r2q 10 default 0 direct_packets_stat 400

# ingress
qdisc ingress {ingress_qdisc_id} parent ffff:fff1 ----------------

Then we can use that {ingress_qdisc_id} and {egress_qdisc_id} to create the L3 IPs filters. Assuming we have a L3 IP 172.16.6.161, and it has a QoS policy with rule {qos_rate_value: 1000Kbit, qos_burst_value: 1Mb}. The ingress and egress tc filter create commands will be:

# ingress
$ tc filter add dev [a-device] parent {ingress_qdisc_id} protocol ip prio 1 \
  u32 match ip dst 172.16.6.161 police \
  rate 1000Kbit burst 1Mb drop flowid :1

# egress
$ tc filter add dev [a-device] parent {egress_qdisc_id} protocol ip prio 1 \
  u32 match ip src 172.16.6.161 police \
  rate 1000Kbit burst 1Mb drop flowid :1

Actual filter rules:

# ingress
$ tc -s -d -p filter show dev [a-device] parent {ingress_qdisc_id}
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP dst 172.16.6.161/32 (success 0 )
 police 0x67 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
...

# egress
$ tc -s -d -p filter show dev [a-device] parent {egress_qdisc_id}
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP src 172.16.6.161/32 (success 0 )
 police 0x68 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
...

The neutron basic workflow

Floating IP

1. User create a floating IP.
1. Floating IP was associated with a port.
1. Create/find QoS policy and rule for floating IP.
1. Bind QoS policy to floating IP (floating IP update API).
1. L3 agent processes the router and its floating IP.
1. L3 agent sets the TC rules to the qrouter-namespace relevant device.

After this the floating IP will be set under a bandwidth restriction.

Example L3 agent side TC rules

Assuming that we have a legacy router: cf6951cb-b050-4543-9742-c63a4989edae, gateway ip: 172.16.6.167, 1Mbps, floating IP: 172.16.10.69, 1Mbps, then in its scheduled network node you can get the following rules:

# ip netns exec qrouter-cf6951cb-b050-4543-9742-c63a4989edae ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    inet 127.0.0.1/8 scope host lo
140: qr-4aba9b05-36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
    inet 192.168.233.1/24 brd 192.168.233.255 scope global qr-4aba9b05-36
141: qr-aef0d42d-f9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
    inet 192.168.232.1/24 brd 192.168.232.255 scope global qr-aef0d42d-f9
143: qg-c99a5832-7f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UNKNOWN
    inet 172.16.6.167/16 brd 172.16.255.255 scope global qg-c99a5832-7f
    inet 172.16.10.69/32 brd 172.16.10.69 scope global qg-c99a5832-7f
       valid_lft forever preferred_lft forever

# ip netns exec qrouter-cf6951cb-b050-4543-9742-c63a4989edae tc qdisc show dev qg-c99a5832-7f
qdisc htb 801b: root refcnt 2 r2q 10 default 0 direct_packets_stat 400
qdisc ingress ffff: parent ffff:fff1 ----------------

# ip netns exec qrouter-cf6951cb-b050-4543-9742-c63a4989edae tc -s -d -p filter show dev qg-c99a5832-7f parent 801b:
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP src 172.16.6.167/32 (success 0 )
 police 0xa74 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1

 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
filter protocol ip pref 1 u32 fh 800::801 order 2049 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP src 172.16.10.69/32 (success 0 )
 police 0xa76 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1

 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)

# ip netns exec qrouter-cf6951cb-b050-4543-9742-c63a4989edae tc -s -d -p filter show dev qg-c99a5832-7f parent ffff:
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP dst 172.16.6.167/32 (success 0 )
 police 0xa73 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1

 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
filter protocol ip pref 1 u32 fh 800::801 order 2049 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP dst 172.16.10.69/32 (success 0 )
 police 0xa75 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1

 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)

And DVR router 0580788c-c919-447c-aea1-87d415aa173a with floating IP 172.16.6.161 1Mbps in a compute node:

# ip netns exec qrouter-0580788c-c919-447c-aea1-87d415aa173a ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    inet 127.0.0.1/8 scope host lo
24: rfp-0580788c-c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP qlen 1000
    inet 169.254.106.114/31 scope global rfp-0580788c-c
    inet 172.16.6.161/32 brd 172.16.6.161 scope global rfp-0580788c-c
102: qr-43185e93-af: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
    inet 192.168.199.1/24 brd 192.168.199.255 scope global qr-43185e93-af
104: qr-51635a61-46: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
    inet 192.168.198.1/24 brd 192.168.198.255 scope global qr-51635a61-46

# ip netns exec qrouter-0580788c-c919-447c-aea1-87d415aa173a tc qdisc show dev rfp-0580788c-c
qdisc htb 801e: root refcnt 2 r2q 10 default 0 direct_packets_stat 5
qdisc ingress ffff: parent ffff:fff1 ----------------

# ip netns exec qrouter-0580788c-c919-447c-aea1-87d415aa173a tc -s -d -p filter show dev rfp-0580788c-c parent 801e:
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP src 172.16.6.161/32 (success 0 )
 police 0x68 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1

 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)

# ip netns exec qrouter-0580788c-c919-447c-aea1-87d415aa173a tc -s -d -p filter show dev rfp-0580788c-c parent ffff:
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP dst 172.16.6.161/32 (success 0 )
 police 0x67 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1

 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)

IPv6 Impact

Only IPv4 has the concept of floating IP.

Data Model Impact

For floating IP: A new table, QoSFIPPolicyBinding, will be created to represent the 1-1 association between floating IP and QoS policy, each policy has rules in both ingress and egress direction.

REST API Impact

New floating IP API extension:

floatingip: {
       ...
       'qos_policy_id': {'allow_post': True, 'allow_put': True,
                         'validate': {'type:uuid_or_none': None},
                         'is_visible': True,
                         'default': None},}

New floating IP commands for creation, deletion and update elements with QoS property:

openstack floating ip create --qos-policy policy_id <network>
openstack floating ip set --qos-policy policy_id floating_ip_id
openstack floating ip unset --qos-policy floating_ip_id

Implementation

Assignee(s)

LIU Yulong <i@liuyulong.me>

Work Items

Create model tables and API extensions.
New TC command wrapper for floating IP rate limit.
L3 agent extension for QoS rule installation.
CLI support.
Testing.
Documentation.

Dependencies

None

Testing

Functionality

The bandwidth of floating IP should be restricted properly. Adding new fullstack and tempest scenarios to test the bandwidth of ingress and egress.

References

Logging API for security-group-rules

Thu, 01 Mar 2018 00:00:00

https://bugs.launchpad.net/neutron/+bug/1468366

This spec implements a way to capture and store events related to security groups.

Problem Description

Operators (including cloud admin and developers) or projects want to store logs of network traffic of:

North-south network traffic travels between an instance and external network.
East-west network traffic travels between instances.

To detect illegal communication or attack patterns, alert abnormal activities for compliance reason.

In addition, these logs can also be used for debugging security groups (help project make sure security group rules work as expected or not). However this is not the main goal since other approaches present (e.g TaaS, tcpdump) to fix the same problem.

In Neutron, all traffic allowed/dropped on instances are managed by security groups. However, logging is currently a missing feature in security groups.

Proposed Change

To address this problem, we’d like to propose a logging API with objective:

Define format and ways to collect events related to security groups.

Logging API will collect ACCEPT or DROP or both events related to security groups configured on port(s). Please refer to Expected API behavior for more details.

This spec implements logging API for security groups for OVS native firewall (OVS) and Iptables-based (Linux Brigde) for operator-only. It also lays out a model which can be extended to other resources (e.g Firewall logging) easily.

Initially, we have both PoC for OVS native firewall and Iptables-based logging. The OVS native firewall logging will be completely supported first as it is now becoming the default driver, Iptables-based logging will be finished later. This API is considered to work consistently between OVS native firewall and Iptables-based logging.

Regarding multiple driver environment, this feature will have a similar validation like QoS [1] as a first approach. [2]

Logging implementation

Logging API is designed as a service plugin. It is defined as a generic logging API for resources such as security groups and firewall.

In server side, the class design for LoggingApiPlugin would look like:

+-----------------------+
| LoggingApiPluginBase  |
+----------^------------+
           |
+-----------------------+
|  LoggingApiPlugin     |
+-----------------------+
|  _init_()             |
|  get_logs()           |
|  get_log()            |
|  create_log()         |
|  update_log()         |
|  delete_log()         |
+-----------------------+

The reference implementation can be found in [3].

The LoggingExtension is an agent extension. It is common for all logging resources like security groups and firewall. The LoggingExtension will receive CREATED/UPDATED/DELETED events of the logging resource and pass these events to logging drivers. Each logging driver defines the resources it supports. In case of security group, we can have a security group logging driver implemented as OVSFirewallLogging or IptablesLoggingDriver.

Class design of LoggingExtension would look like:

+-----------------------+
|     AgentExtension    |
+------------^----------+
             |
+------------+----------+
|    LoggingExtension   |
+-----------------------+
| initialize()          |
| consume_api()         |
| _handle_notification()|
+-----------------------+

Class design for LoggingDriver would look like:

                 +-----------------------------+
                 |    LoggingDriver            |
                 +-----------------------------+
                 |SUPPORTED_LOGGING_TYPES=set()|
                 |initialize()                 |
                 |start_logging(log_resource)  |
                 |stop_logging(log_resource)   |
                 +--------------^--------------+
                                |
               +----------------+-----------------+
               |                                  |
+--------------+---------------+   +--------------+---------------+
| IptablesLoggingDriver        |   | OVSFirewallLoggingDriver     |
+------------------------------+   +------------------------------+
| SUPPORTED_LOGGING_TYPES=[...]|   | SUPPORTED_LOGGING_TYPES=[...]|
| initialize()                 |   | initialize()                 |
| start_logging(log_resource)  |   | start_logging(log_resource)  |
| stop_logging(log_resource)   |   | stop_logging(log_resource)   |
| _handle_logging()            |   |                              |
| _create_security_group_log   |   +------------------------------+
| _delete_security_group_log   |
| ...                          |
+------------------------------+

For OVS firewall: OVSFirewallLoggingDriver will act as a controller program. It runs in each compute node to handle packet_in to produce a log line per packet to a file. To generate a packet in message for ACCEPT event: The OVSFirewallLoggingDriver will insert “actions=controller” to flows matched with security group rules whose conntrack state is NEW (first packet). It will also insert flows with ‘actions=controller’ before flows has ‘actions=drop’ and conntrack state is INVALID or NOT ESTABLISHED to generate packet_in messages for DROP event via port-number for project-isolation. For a flow example would look like:

For ingress tables:

table=82, priority=73,ct_state=+new-est,icmp,reg5=0xa,dl_dst=fa:16:3e:1d:d0:8b actions=ct(commit,zone=NXM_NX_REG6[0..15]),strip_vlan,output:10,CONTROLLER:65535
table=82, priority=70,ct_state=+new-est,icmp,reg5=0xa,dl_dst=fa:16:3e:1d:d0:8b actions=ct(commit,zone=NXM_NX_REG6[0..15]),strip_vlan,output:10
table=82, priority=70,ct_state=+est-rel-rpl,icmp,reg5=0xa,dl_dst=fa:16:3e:1d:d0:8b actions=strip_vlan,output:10
table=82, priority=53,ct_mark=0x1,reg5=0xa actions=CONTROLLER:65535
table=82, priority=50,ct_mark=0x1,reg5=0xa actions=drop
table=82, priority=50,ct_state=+inv+trk actions=drop
table=82, priority=50,ct_state=+est-rel+rpl,ct_zone=1,ct_mark=0,reg5=0xa,dl_dst=fa:16:3e:1d:d0:8b actions=strip_vlan,output:10
table=82, priority=50,ct_state=-new-est+rel-inv,ct_zone=1,ct_mark=0,reg5=0xa,dl_dst=fa:16:3e:1d:d0:8b actions=strip_vlan,output:10
table=82, priority=43,ct_state=-est,reg5=0xa actions=CONTROLLER:65535
table=82, priority=40,ct_state=-est,reg5=0xa actions=drop
table=82, priority=40,ct_state=+est,ip,reg5=0xa actions=ct(commit,zone=NXM_NX_REG6[0..15],exec(load:0x1->NXM_NX_CT_MARK[]))
table=82, priority=40,ct_state=+est,ipv6,reg5=0xa actions=ct(commit,zone=NXM_NX_REG6[0..15],exec(load:0x1->NXM_NX_CT_MARK[]))
table=82, priority=0 actions=drop

For egress tables:

table=72, priority=73,ct_state=+new-est,icmp,reg5=0xa,dl_src=fa:16:3e:1d:d0:8b actions=resubmit(,73),CONTROLLER:65535
table=72, priority=70,ct_state=+new-est,icmp,reg5=0xa,dl_src=fa:16:3e:1d:d0:8b actions=resubmit(,73)
table=72, priority=70,ct_state=+est-rel-rpl,icmp,reg5=0xa,dl_src=fa:16:3e:1d:d0:8b actions=resubmit(,73)
table=72, priority=53,ct_mark=0x1,reg5=0xa actions=CONTROLLER:65535
table=72, priority=50,ct_mark=0x1,reg5=0xa actions=drop
table=72, priority=50,ct_state=+inv+trk actions=drop
table=72, priority=50,ct_state=+est-rel+rpl,ct_zone=1,ct_mark=0,reg5=0xa actions=NORMAL
table=72, priority=50,ct_state=-new-est+rel-inv,ct_zone=1,ct_mark=0,reg5=0xa actions=NORMAL
table=72, priority=43,ct_state=-est,reg5=0xa actions=CONTROLLER:65535
table=72, priority=40,ct_state=-est,reg5=0xa actions=drop
table=72, priority=40,ct_state=+est,ip,reg5=0xa actions=ct(commit,zone=NXM_NX_REG6[0..15],exec(load:0x1->NXM_NX_CT_MARK[]))
table=72, priority=40,ct_state=+est,ipv6,reg5=0xa actions=ct(commit,zone=NXM_NX_REG6[0..15],exec(load:0x1->NXM_NX_CT_MARK[]))
table=72, priority=0 actions=drop

The reference implementation can be found in [5].

For Iptables-based: IptablesLoggingDriver implementation is quite clear. IptablesLoggingDriver just adds NFLOG rules to iptables [4].

Cloud deployers can choose log_driver for OVS native firewall driver or iptables driver by specifying ovs_fw_log or iptables_log. They are also able to configure rate_limit and burst_limit [8] to limit maximum packets logging per second. Log output file can be specified by option local_output_log_base. These options are defined in /etc/neutron/plugins/ml2/ml2_conf.ini:

[log]
# Driver for security group logging in the L2 agent (string value)
# 'ovs_fw_log' or 'iptables_log'
log_driver = ovs_fw_log

# Maximum packets logging per second. (integer value, at least 100)
rate_limit = 100

# Maximum number of packets per rate_limit(integer value, at least 25)
burst_limit = 25

# Output logfile
local_output_log_base = /var/log/syslog

To consume log-data, operators can:

use third-party services such as Monasca service to consume log-data.
access directly to ‘local_output_log_base’ to take log-data.

Expected API behavior

This spec takes security groups logging for an example:

Operators can collect security events (ACCEPT/DROP or ALL (both ACCEPT and DROP)) for some cases:

collect events related to a specific security group applied to all VMs by passing its security group ID to ‘resource_id’.

collect events related to a specific security group applied to a specific VM by passing its security group ID to ‘resource_id’ and its bound Neutron port ID to ‘target_id’.

collect events related to all security groups being applied to a specific VM by passing its Neutron port ID to ‘target_id’.

collect events related to security groups in a project: in this case operators do not pass any value to ‘resource_id’ or ‘target_id’.

More details, this API will log first packet which is matched with security group rule for ACCEPT events. It will also log all packets which is unmatched with any security group rules for DROP events.

Please note that, depends on above use cases, if a new VM or a new security group or a new security group rule is launched, its related security events will be collected, too. On the other hands, if a VM or a security group or a security group rule is deleted, its related security events will be not logged. However, security events related to the remains will still be logged.

Future work beyond this spec

Extending logging for FWaaS v2.
Exposing the logging API for projects (e.g by using RBAC).

API operation sample

Take below scenario as an example:

Operator boots a number of VMs, the VMs’ traffic is restricted by security-group-rules.
Check supported logging capabilities
GET /v2.0/logging/loggable-resources
{ "loggable_resources":[{"type": "security_group"}] }

Create a logging-resource and define events related to security groups need collecting for this project. For example, operator wants to collect all ACCEPT & DROP events related to all security groups applied in project demo (Read Expected API behavior section for more details)

POST /v2.0/logging/logs

{
    "log": {
        "name": "create_log_test1",
        "description": "Collecting all security events in project demo",
        "project_id": "8d4c70a21fed4aeba121a1a429ba0d04",
        "resource_type": "security_group",
        "event": "ALL"}
}

Response

{
    "log": {
        "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c793",
        "project_id": "8d4c70a21fed4aeba121a1a429ba0d04",
        "name": "create_log_test1",
        "description": "Collecting all security events in project demo",
        "enabled": True,
        "resource_type": "security_group",
        "event": "ALL",
        "resource_id": None,
        "target_id": None}
}

Operator use external services (e.g Monasca) to consume log-data.

Example logging format

Logging output should include timestamp, action(ACCEPT/DROP) project ID, logging-resource ID, VM port ID, source MAC, destination MAC, source IP address, destination IP address, protocol, source L4 port and destination L4 port. Making the format configurable is out of the scope though and the format can be defined in implementation phase.

For initial implementation would look like:

log-data of an ACCEPT event:

May 5 09:05:07 action=ACCEPT project_id=736672c700cd43e1bd321aeaf940365c
log_resource_ids=['4522efdf-8d44-4e19-b237-64cafc49469b', '42332d89-df42-4588-a2bb-3ce50829ac51']
vm_port=e0259ade-86de-482e-a717-f58258f7173f
ethernet(dst='fa:16:3e:ec:36:32',ethertype=2048,src='fa:16:3e:50:aa:b5'),
ipv4(csum=62071,dst='10.0.0.4',flags=2,header_length=5,identification=36638,offset=0,
option=None,proto=6,src='172.24.4.10',tos=0,total_length=60,ttl=63,version=4),
tcp(ack=0,bits=2,csum=15097,dst_port=80,offset=10,option=[TCPOptionMaximumSegmentSize(kind=2,length=4,max_seg_size=1460),
TCPOptionSACKPermitted(kind=4,length=2), TCPOptionTimestamps(kind=8,length=10,ts_ecr=0,ts_val=196418896),
TCPOptionNoOperation(kind=1,length=1), TCPOptionWindowScale(kind=3,length=3,shift_cnt=3)],
seq=3284890090,src_port=47825,urgent=0,window_size=14600)

log-data of a DROP event:

May 5 09:05:07 action=DROP project_id=736672c700cd43e1bd321aeaf940365c
log_resource_ids=['4522efdf-8d44-4e19-b237-64cafc49469b'] vm_port=e0259ade-86de-482e-a717-f58258f7173f
ethernet(dst='fa:16:3e:ec:36:32',ethertype=2048,src='fa:16:3e:50:aa:b5'),
ipv4(csum=62071,dst='10.0.0.4',flags=2,header_length=5,identification=36638,offset=0,
option=None,proto=6,src='172.24.4.10',tos=0,total_length=60,ttl=63,version=4),
tcp(ack=0,bits=2,csum=15097,dst_port=80,offset=10,option=[TCPOptionMaximumSegmentSize(kind=2,length=4,max_seg_size=1460),
TCPOptionSACKPermitted(kind=4,length=2), TCPOptionTimestamps(kind=8,length=10,ts_ecr=0,ts_val=196418896),
TCPOptionNoOperation(kind=1,length=1), TCPOptionWindowScale(kind=3,length=3,shift_cnt=3)],
seq=3284890090,src_port=47825,urgent=0,window_size=14600)

Data Model Impact

This model defines a generic model for all logging resources like security groups and firewall.

Log model would look like:

Attribute Name	Type	Access	Default Value	Validation/ Conversion	Description
id	string (UUID)	RO	generated	uuid	Identity
project_id	string (UUID)	RW(No update)	N/A	uuid	Project ID of Logging object owner
name	string	RW	N/A	none	Logging object name
description	string	RW	N/A	none	Logging object description
enabled	bool	RW	True	Boolean	Enable/disable log
resource_type	string	RW(No update)	N/A	none	Logging resource type, it could be ‘security_group’ in the initial implemenation
resource_id	string (UUID)	RW(No update)	N/A	uuid	ID of resource which is enabled to be logged. When a resource_id (which could be a security group ID in the initial implementation) is specified, its related events will be collected. For detail usage refer to Expected API behavior
event	string	RW(No update)	N/A	none	ACCEPT/DROP or ALL can be specified. ALL is set as default.
target_id	string (UUID)	RW(No update)	N/A	uuid	ID of resource (which could be a port ID in the initial implementation), where we want to collect log. When a target_id is specified, events related to it will be colleted. For detail usage refer to Expected API behavior

REST API Impact

The new resources will be added.

LOGGING_PREFIX = '/logging'
EVENTS = ['ACCEPT', 'DROP', 'ALL']
RESOURCE_ATTRIBUTE_MAP = {
    'log': {
        'id': {'allow_post': False, 'allow_put': False,
               'validate': {'type:uuid': None},
               'is_visible': True, 'primary_key': True},
        'project_id': {'allow_post': True, 'allow_put': False,
                       'required_by_policy': True,
                       'validate': {'type:string': attr.TENANT_ID_MAX_LEN},
                       'is_visible': True},
        'name': {'allow_post': True, 'allow_put': True,
                 'validate': {'type:string': attr.NAME_MAX_LEN},
                 'default': '', 'is_visible': True},
        'description': {'allow_post': True, 'allow_put': True,
                        'validate': {'type:string': attr.DESCRIPTION_MAX_LEN},
                        'default': '', 'is_visible': True},
        'resource_type': {'allow_post': True, 'allow_put': False,
                          'required_by_policy': True,
                          'validate': {'type:validate_log_resource_type': None},
                          'is_visible': True},
        'resource_id': {'allow_post': True, 'allow_put': False,
                        'is_visible': True, 'default': None,
                        'validate': {'type:uuid_or_none': None}},
        'event': {'allow_post': True, 'allow_put': False,
                  'validate': {'type:values': EVENTS},
                  'is_visible': True, 'default': 'ALL'},
        'target_id': {'allow_post': True, 'allow_put': False,
                      'is_visible': True, 'default': None,
                      'validate': {'type:uuid_or_none': None}},
        'enabled': {'allow_post': True, 'allow_put': True,
                    'is_visible': True, 'default': True,
                    'convert_to': attr.convert_to_boolean},
    },

    'loggable_resources':{
        'type': {'allow_post': False, 'allow_put': False,
                 'is_visible': True}
    }
}

Object	URI	Type
log	/logging/logs	POST
log	/logging/logs	GET
log	/logging/logs/{logging-resource-id}	GET
log	/logging/logs/{logging-resource-id}	DELETE
log	/logging/logs/{logging-resource-id}	PUT
loggable-resource	/logging/loggable-resources	GET

Security Impact

This REST API will only be accessible to admin_only which is controlled by policy.json.

The volume of log highly depends on user traffic. It affects performance and it is a potential security impact (a kind of DoS).

Notifications Impact

None

Operators CLI Impact

Additional methods will be added to neutron OSC plugin to create, update, delete, list and get logging resource.

For logging resource:

openstack network logging create
            --resource-type <resource-type>
            [--description <description>]
            [--enable | --disable]
            [--resource-id <resource-id>]
            [--event=<ACCEPT/DROP/ALL>]
            [--target-id=<port-id>]
            [--project <project> [--project-domain <project-domain>]]
            name

openstack network logging list

openstack network logging set
            [--name <new-name>]
            [--description <description>]
            [--enabled |--disabled]
            <logging-resource-id>

openstack network logging show <logging-resource-id>
openstack network logging delete <logging-resource-id>

Check supported logging capabilities:

openstack network loggable resources list

Performance Impact

Currently, logged data will be stored into files. So this may increase a little bit performance overhead.

IPv6 Impact

Support both IPv4 and IPv6.

Other Deployer Impact

Configure to enable logging API service.

Developer Impact

None

Community Impact

None

Alternatives

Changing the security-group API by adding ‘logging’ attribute to Security Group API resource. It would break the compatibility with the EC2 API [6]. And we should avoid changing FWaaS API by extend it [7].

Hence, the logging API allow enable logging is necessary.

Implementation

Assignee(s)

Primary assignee:: y-furukawa-2
Other contributors:: hoangcx, annp, tuhv

Work Items

Finalize a way to log data
Implement
- REST API + API tests
- Database model & database migrations
- Oslo versioned object database implementation
- Iptables based reference implementation
- OSC support

Dependencies

None

Testing

Unit Test
Functional test
Fullstack test

Tempest Tests

None, since this is covered by the in-tree API tests.

Functional Tests

Regression test with the logging feature enabled. Test if the logging feature actually works.

API Tests

The new API interface will be tested via API tests, to ensure all the operations work as expected.

Documentation Impact

User Documentation

Add some usages into the networking guide for operator.
Write how to enable the logging API plugin.

References

VPN Services Support QoS

Thu, 04 Jan 2018 00:00:00

https://bugs.launchpad.net/neutron/+bug/1727578

Currently, there is no way to set VPN services’ bandwidth. This specification proposed a way to set the VPN services’ bandwidth limit.

Problem Description

Currently, Neutron VPNaaS provides site to site VPN services, but its bandwidth consumption is not regulated, as the VPN tunnel will cost the bandwidth from the outside public bandwidth provided by the ISP or other organizations. That means it is not free. The OpenStack provider or users should pay for the limited bandwidth. So it is necessary for saving the resources.

Currently, physical device configurations outside OpenStack environment cannot be modified by OpenStack, but we can shape the outgoing traffic. VPN services provide the security guarantee, but it shouldn’t abuse the bandwidth of underlay network that it will affects other traffic.

Use Case

Consider that an OpenStack public cloud has been deployed in a real data center. A cloud enterprise user has a requirement in which one of the applications should connect to its private network 20.0.0.0/24, but for other traffic flows, still use the original network functionality provided by OpenStack. The network topology diagram is shown below:

                                                                                                         +
                       +--------------------+   +                                                        |DataCenter
                       |VM1                 |   |                                                        |external network
                       |nic IP: 10.0.0.11   |   |                 +-------------------------------+      |        +----+
                       |default via 10.0.0.1+---+                 |Default Router                 |      |        |    |
                       |                    |   +-----------------+Interface: 10.0.0.1            +------+        |    |
                       +--------------------+   |                 |gw: 172.24.4.11                |      |        |    |
OpenStack                                       |                 |route: 20.0.0.0/24 via 10.0.0.5|      |        |    |
private                                .        |                 |                               |      |        |    |
network                                .        |                 +-------------------------------+      |        |    |
                                                |                                                        |        |    |
subnet: 10.0.0.0/24                             |                                                        |        |    |
                       +--------------------+   |                                                        |        |    |
                       |VM2                 |   |                                                        +--------+    |Internet
                       |nic IP: 10.0.0.12   |   |                                                        |        |    |
                       |default via 10.0.0.1+---+                                                        |        |    |
                       |                    |   |                                                        |        |    |
                       +--------------------+   |                                                        |        |    |
                                                |                                                        |        |    |
                                       .        |                 +--------------------------------+     |        |    |       +-------------------------------+
                                       .        |                 |VPN Router                      |     |        |    |       |Vendor GW Router               |
                                                |                 |Interface: 10.0.0.5             |     |        |    |       |peer vpn service running       |
                                                +-----------------+gw: 172.24.4.12                 +-----+        |    +-------+hold private subnet 20.0.0.0/24|
                       +--------------------+   |                 |VPN service running             |     |        |    |       |                               |
                       |VMX                 |   |                 |                                |     |        |    |       |                               |
                       |nic IP: 10.0.0.13   |   |                 +--------------------------------+     |        +----+       +-------------------------------+
                       |default via 10.0.0.1+---+                                                        |
                       |                    |   |                                                        |
                       +--------------------+   |                                                        |
                                                |                                                        |
                                                +                                                        +

As this diagram shows, the enterprise user owned many VMs in the private network and the allocated IPs are from the private subnet 10.0.0.0/24. There are two routers connected, Default Router is used for the normal network functions, such as NAT, Floating IP, Routing. VPN Router is mainly used for VPN traffic process, also contains NAT and route for other traffic. Both of the routers are attached to the external network which is the physical underlay network in the real data center. The VPN Router in OpenStack and Vendor GW Router in other site maintain a VPN peer relationship across the Internet.

There are two scenarios towards the VM outgoing traffic:

VMs in the OpenStack private network access the normal websites, first send the network packets to its gateway which is located on the Default Router. Then send to the Internet by the data center devices.
VMs in the OpenStack private network access the other site private network 20.0.0.0/24, still first send the network packets to its gateway 10.0.0.1, then check the route tables, the nexthop of 20.0.0.0/24 is 10.0.0.5 which is located on VPN Router. The network traffic will be sent based on the existing VPN tunnel to Vendor private site.

Like the diagram shows, the QoS Policy should be set on the qg-XX port of the VPN router for limiting the outgoing VPN traffic.

This spec focuses on the QoS of VPN outgoing traffic, so for neutron-vpnaas, this spec will focus on the Router related with VPN services. And for the general use cases which is that VPN service usually setup across the Internet in tunnel mode, we will only introduce the QoS support on tunnel type VPN services.

Proposed Change

We propose the VPN Service resource accepts the Neutron QoS Policy. Once the ipsec site connection is created, the QoS Policy will be applied on the VPN router’s qg-XX port, as the ESP encapsulation will use the qg-XX port’s IP to access other sites.

So there are three parts that require to work:

DB related changes, including new table qos_vpnservice_policy_bindings addition and data model change.
API changes, including extend the API to accept the Neutron QoS Policy.
Introduce a new l3 agent extension to extend the ability to process the QoS policy installation on the router.

Alternatives

Accept the QoS parameters and implement the QoS function on our own.
Apply QoS Policy on the Router interface directly, but this would affect the west-east traffic.

Data model impact

In this spec, the QoS data model and function will be provided by Neutron, so vpnservices table need to maintain the relationship with Neutron QoS Policy.

The following new table is added as part of the VPN QoS feature:

CREATE TABLE `qos_vpnservice_policy_bindings` (
  `vpn_service_id` varchar(36) NOT NULL,
  `qos_policy_id` varchar(36) NOT NULL,
  UNIQUE KEY `vpn_service_id` (`vpn_service_id`),
  KEY `qos_policy_id` (`qos_policy_id`),
  CONSTRAINT `qos_vpn_service_policy_bindings_ibfk_1` FOREIGN KEY (
  `qos_policy_id`) REFERENCES `qos_policies` (`id`) ON DELETE CASCADE,
  CONSTRAINT `qos_vpn_service_policy_bindings_ibfk_2` FOREIGN KEY (
  `vpn_service_id`) REFERENCES `vpnservices` (`id`) ON DELETE CASCADE
);

REST API impact

Proposed attribute:

EXTEND_FIELDS = {
    'qos_policy_id':{'allow_post': True, 'allow_put': True,
                     'validate': {'type:uuid': None},
                     'is_visible': True,
                     'default': None}
}

Some samples in VPN service create/update. Users are allowed to pass qos_policy_id.

Create/Update VPN service Request:

POST /v2.0/vpn/vpnservices
{
    "vpnservice": {
        "subnet_id": null,
        "qos_policy_id": "a36c20d0-18e9-42ce-88fd-82a35977ee8c",
        "router_id": "66e3b16c-8ce5-40fb-bb49-ab6d8dc3f2aa",
        "name": "myservice",
        "admin_state_up": true
    }
}

Response:
{
    "vpnservice": {
        "router_id": "66e3b16c-8ce5-40fb-bb49-ab6d8dc3f2aa",
        "status": "PENDING_CREATE",
        "name": "myservice",
        "external_v6_ip": "2001:db8::1",
        "admin_state_up": true,
        "subnet_id": null,
        "project_id": "10039663455a446d8ba2cbb058b0f578",
        "tenant_id": "10039663455a446d8ba2cbb058b0f578",
        "external_v4_ip": "172.32.1.11",
        "id": "5c561d9d-eaea-45f6-ae3e-08d1a7080828",
        "description": "",
        "qos_policy_id": "a36c20d0-18e9-42ce-88fd-82a35977ee8c"
    }
}

PUT /v2.0/vpn/vpnservices/{service_id}
{
    "vpnservice": {
        "name": "NEW VPN SERVICE NAME",
        "description": "Updated description",
        "qos_policy_id": "a36c20d0-18e9-42ce-88fd-82a35977ee8c"
    }
}

Response:
{
    "vpnservice": {
        "router_id": "881b7b30-4efb-407e-a162-5630a7af3595",
        "status": "ACTIVE",
        "name": "NEW VPN SERVICE NAME",
        "admin_state_up": true,
        "subnet_id": null,
        "project_id": "26de9cd6cae94c8cb9f79d660d628e1f",
        "tenant_id": "26de9cd6cae94c8cb9f79d660d628e1f",
        "id": "41bfef97-af4e-4f6b-a5d3-4678859d2485",
        "description": "Updated description",
        "qos_policy_id": "a36c20d0-18e9-42ce-88fd-82a35977ee8c"
    }
}

QoS Policy Application Details

The reason for introducing this, for example, we change the use case below, we deploy the vpn service on the Default Router, delete the VPN Router. That means the general traffic and VPN traffic will pass through the Default Router, then we apply the Neutron QoS policy on the qg-XX port of the Default Router, it will limit all the bandwidth, so the VPN’s bandwidth may have a lower performance, or we can say it is not consistent with expectations.

Currently, Neutron provides the QoS function but not for some interest streams. Here we will focus on the VPN traffic. For this function, we will combine the iptables and tc together. The reason for choosing them is that, iptables could mark the VPN interest stream by the ipsec VPN transform protocols(such as esp, ah-esp protocols), the interface that the packets will go out and the local encapsulated IP if running in tunnel mode. Also we need to shape the vpn traffic before send out to the underlay network, so some new iptables rules will be installed on mangle table in the router’s namespace. Also the fwmark is eaiser to extend, such as ipchains.

And we will introduce a new tc wrapper which will use htb and it will provides classification algorithm. Then developers can easily implement other complex traffic control. That means we will extend the current tc_lib in Neutron repo. And vpn_qos will based on this.

Just like above description, a new L3 agent extension will be introduced like fip_qos done. We suggest to name it vip_qos, it will install the appropriate iptables rules in the router’s namespace which binds with VPN service. Then users or managers could use the QoS function to the Default Router and not affect other network streams.

Security impact

None

Notifications impact

No expected change.

Other end user impact

Users will be able to specify qos_policy during create/update VPN service.

Performance Impact

It will save the bandwidth of the underlay network in data center.

Other deployer impact

None

Developer impact

Developer may use the new tc wrapper to do other things, as it is powerful to support other stream control functionality. But there may be a conflict with openstack/neutron-classifier, as it provides defining the traffic. So we may reconsider that if possible.

Implementation

Assignee(s)

zhaobo

Work Items

Add the DB model and extend the table column.
Extend VPN API to accept QoS policy.
Extend new tc wrapper which support classification algorithm based on traffic classifier feature.
Extend new L3 agent extension vip_qos.
Add API validation code to validate access/existence of the qos_policy which created in Neutron.
Add UTs to Neutron-vpnaas.
Add API tests.
Update CLI to accept QoS fields.
Documentation work.

Dependencies

None

Testing

Unit tests, functional tests, API tests and scenario tests are necessary.

Documentation Impact

The Neutron API reference will need to be updated.

References

Neutron Common Classification Framework

Tue, 19 Dec 2017 00:00:00

https://blueprints.launchpad.net/neutron/+spec/common-classification-framework https://wiki.openstack.org/wiki/Neutron/CommonClassificationFramework

Problem Description

Different Neutron projects, services, features and related projects have different traffic classifiers or classification APIs. They differ on both syntax and semantics. A few examples are:

Security group rules [6]
openstack/neutron-fwaas [7]
openstack/networking-sfc (Flow Classifier) [8]
openstack/neutron-classifier [2]
openstack/networking-bgpvpn [12]
openstack/tap-as-a-service [13]
Neutron QoS [10]

Furthermore, there are other projects with classification APIs, such as openstack/group-based-policy [9] and it’s possible that more will want to support classifications in their own APIs, further reinventing the wheel and fragmenting the language used in the OpenStack ecosystem when it comes to defining traffic classifications.

This spec introduces the Neutron Common Classification Framework (CCF), a culmination of previous efforts: Common Classifier [1], openstack/neutron-classifier [2] and Common Flow Classifier [11].

The previous spec for this RFE is archived at [1].

Proposed Change

Terminology:

Common Classification Framework (CCF): the name of this project, which includes the base models and resources introduced below, the API, logic and versioned objects to allow Consuming Services to consume from the framework.
Classification: the Neutron resource being introduced for the purpose of defining an atomic traffic classification.
Classification Group (CG): the Neutron resource being introduced for the purpose of grouping Classifications, or other Classification Groups, together, specifying a common relation across them. This is the resource that ultimately will be consumed by a Consuming Service.
Classification Type (CT): every Classification has a type, defining the format, supported fields and syntax validators of that classification.
Classifications API: the User-facing REST API that is provided by the Common Classification Framework to instantiate the resources above.
Consuming Service (CS): Neutron itself or a Neutron project, service, extension or feature that is capable of consuming Classification Groups (acquiring them as resources), examples of potential Consuming Services are the ones listed in Problem Description. These services do not talk directly to the Classifications API.
User: the end-user, human or machine, tenant or admin, typically cloud consumer, calling into the Classifications API and any Consuming Services’ REST APIs.

This spec defines the new kinds of resources to be introduced, an initial set of Classification Types and their models, and the Classifications API in order to define such traffic classifications, which the User will call.

The dedicated Classifications API provides a way to define and create Classifications and CGs, storing them like any other resource. CSs are then able to fetch those traffic classification definitions as oslo.versionedobjects (OVO) and feed them to their respective internal network classifiers (i.e. to the mechanism capable of matching on traffic, such as Open vSwitch via the respective ML2 mechanism driver, or iptables - it really depends on what service, and its configuration, is consuming the Classification Groups). What Consuming Services do with the resources consumed is entirely out of scope from the CCF.

Updating Classification or CG resources is disallowed in the API, except for changes not related to the traffic classification semantics, such as name and description. With this, having to deal with the complexities of advertising resource changes to CSs is also avoided which, at least for a first version of the CCF, is beneficial as it keeps the project simpler.

Consuming Services are not required to keep a local database copy of the consumed classifications, beyond mapping CG UUIDs with Consuming Services’ own resources. Consequently, the Common Classification Framework should prevent Classifications from being deleted if they are in use by at least 1 resource of any Consuming Service.

Summary of changes necessary to create the Common Classification Framework:

Creation of a Neutron extension to expose the new, first version of the Classifications API, as additional modifications will require new extensions.
Creation of models for the different supported protocols’ types/definitions.
Creation of a Neutron service plugin and DB layer to validate the Classifications defined by the API according to the types defined and respective supported models, and store them in the database.
Adoption of OVO so that every Classification defined becomes a versioned and serializable object, which will be transferred from the Common Classification Framework to Consuming Services.
Create CLI using OSC so Users can define their own Classifications with ease.

Though out of scope, this is a summary of changes needed in Consuming Services:

Add a dependency on the CCF’s own extension.
Add (or reuse) fields to/in certain API resources with the intent of receiving Classification Group UUIDs (e.g. networking-sfc port-chain’s flow_classifiers field could expect a common Classification Group UUID instead of local Flow Classifier UUIDs).
Evaluate the Classifications provided to ascertain whether they are compatible with the specific extension’s classifier (mechanism).
Add logic to fetch the CGs provided by UUID, making use of OVO in the process, potentially via neutron-lib (xgerman).
Translate the classifications’ definitions to the specific classifier’s mechanism or language (e.g. create Open vSwitch flows matching on traffic).
Typically Consuming Services will already have their own CLI, so changes must be made consistently with the changes made to their own APIs and/or classification-fetching code.

The diagram at [5] shows the relationship between Users, the Common Common Classification Framework and the Consuming Services.

Data Model Impact

Grouping traffic classifications by individual types, called Classification Types, will allow future types to be added and agreed in the future, while keeping the remaining ones intact. Existing types can of course be updated, thanks to versioning. Thus, this approach follows a modular, rather than a monolithic way of defining classifications. Looking at the existing neutron-classifier project [2], which was decided as the repo to keep the Common Classifier [3] (during the Tokyo Summit 2015), there are hints of an architecture like the one proposed here, as can be seen in [4]. As such, this framework is partially based on the neutron-classifier with the major difference that it presents a REST API for Users to define classifications.

Before delving into further detail in regards to the data model, API and how classifications can be used by interested Neutron subprojects, a few points need to be clarified:

Classification Types can be introduced or extended (with new fields e.g.) in every release of the CCF. API extensions will be added to reflect these additions in the REST API and maintain backwards compatibility.
1 Classification is of a single type, e.g. either Ethernet, IP, HTTP, or another supported at the time of a specific CCF release. The definition, i.e. fields to match on, depends on the type specified.
To clarify, Classification Types define the set of possible fields and values for a Classification (essentially, an instance of that Classification Type). Classification Types are defined in code, where Classifications are created via the REST API as instances of those types.
Not all supported fields need to be defined - only the ones required by the Consuming Service - which it should validate on consumption.
There are also Classification Groups, which allow Classifications or other Classification Groups to be grouped together using boolean operators. CGs are the resources that will end up being consumed by Consuming Services.
The CCF has to be able to check if a Classification Group is currently being used, and prevent it from getting deleted if so.
From the Consuming Service’s point of view, Classifications can only be read, not created or deleted. They need to have been previously created using the User-facing Classifications API. Figure [5] attempts to illustrate this.

The initial model of the CCF will includes the following Classification Types: Ethernet, IPv4, IPv6, TCP and UDP, which when combined are sufficient to provide any 5-tuple classification.

The following table presents the attributes of a Classification Group (asterisk on RW means that the attribute is non-updatable):

Attribute Name

Type

Access CRUD

Default Value

Validation/ Conversion

Description

id

string (UUID)

RO, all

generated

uuid

Identity

project_id

string (UUID)

RO, project

from auth token

uuid

Project ID

name

string

RW, project

None

string

Name of Classification Group

description

string

RW, project

None

string

Human-readable description

shared

bool

RW, project

False

boolean

Shared with other projects

operator

string (values)

RW*, project

“and”

[“and”,
“or”]

Boolean connective: AND/OR

classification_groups

list

RW*, project

[]

List of Classification Groups included

classifications

list

RW* project

[]

List of Classifications included

Consuming Services will consume Classification Groups, and not atomic Classifications (that would create more difficulties in terms of the relationships between CCF and CSs databases), any Classification needs to be grouped in a Classification Group to be consumed individually. As such, the “operator” field is to be ignored for Classification Groups that only contain 1 Classification inside.

The following table presents the attributes of Classifications of any of the types stated in this spec (asterisk on RW means that the attribute is non-updatable):

Attribute Name

Type

Access

Default Value

Validation/ Conversion

Description

id

string (UUID)

RO, all

generated

uuid

Identity

project_id

string (UUID)

RO, project

from auth token

uuid

Project ID

name

string

RW, project

None

string

Name of Classification

description

string

RW, project

None

string

Human-readable description

type

string

RW*, project

from enum of types

The type of the Classification

negated

bool

RW*, project

False

boolean

Whether to negate classification (boolean NOT)

definition

type-specific attributes will go here, given their volume I won’t detail them unless requested.

Classification Groups and Classifications of every type will be stored as the following tables and relationships (with table name prefix ccf_):

                          +---------------------+
                          |classification_groups|
                          +---------------------+
                          |id                   |*
                          |cg_id                +--------+
                          |name                 |        |
                          |description          |        |
                          |project_id           |        |
                          |shared               +--------+
                          |operator             |1
                          +---------------------+
                                      |1
                                      |
                                      |*
                      +------------------------------+
                      |classification_groups_mapping |
                      +------------------------------+
                      |cg_id                         |
                      |classification_id             |
                      +------------------------------+
                                      |1
+--------------------+                |                +--------------------+
|ipv4_classifications|                |                |ipv6_classifications|
+--------------------+                |                +--------------------+
|classification_id   |                |                |classification_id   |
|ihl                 |1               |               1|traffic_class       |
|diffserv            +--------+       |       +--------+traffic_class_mask  |
|diffserv_mask       |        |       |       |        |length              |
|length              |        |       |       |        |next_header         |
|flags               |        |       |       |        |hops                |
|flags_mask          |        |       |       |        |src_addr            |
|ttl                 |        |1      |1     1|        |dst_addr            |
|protocol            |     +---------------------+     +--------------------+
|src_addr            |     |classifications      |
|dst_addr            |     +---------------------+
|options             |     |id                   |
|options_mask        |     |name                 |
+--------------------+     |description          |
                           |project_id           |
                           |shared               |     +-------------------+
                           |type                 |     |tcp_classifications|
                           |negated              |     +-------------------+
                           +---------------------+     |classification_id  |
+-------------------+        1|      1|       |1       |src_port           |
|udp_classifications|         |       |       |        |dst_port           |
+-------------------+         |       |       |        |flags              |
|classification_id  |1        |       |       |       1|flags_mask         |
|src_port           +---------+       |       +--------+window             |
|dst_port           |                1|                |data_offset        |
|length             |     +------------------------+   |option_kind        |
|window_size        |     |ethernet_classifications|   +-------------------+
+-------------------+     +------------------------+
                          |classification_id       |
                          |preamble                |
                          |src_addr                |
                          |dst_addr                |
                          |ethertype               |
                          +------------------------+

Some of the fields of the Classification Types presented above in the database schema, such as length, src_addr, and others, will allow ranges or lists to be input, through the use of commas or hyphens, for example.

Masking fields allow the user to specify which individual bits of the respective main field should be looked up during classification.

Besides the Classification Types presented above, the following types are also expected to be part of the first release of the CCF: - Neutron (destination/source port, subnets, networks, at least) - ICMP - ICMPv6 - SCTP - ARP - VLAN - GRE - VXLAN - Geneve - MPLS - NSH

Classification Types are used to select the appropriate model of the Classification and consequently what table it will be stored in.

Classification Groups get stored in a single table and can point to other Classification Groups, to allow mixing boolean operators.

There are two important fields meant for boolean logic:

operator in Classification Group: specifies the boolean operator used to connect all the child Classifications and Classification Groups of that group. This can be either AND or OR.
negated per Classification “usage”: specifies whether to negate the definition of the Classification, when mapped to a Classification Group, essentially a boolean NOT. This can be True or False. Please note that Classification Groups cannot be negated using this model.

REST APIs

A new API extension is being introduced. The base URL for the Classifications API is /v2.0/.

The following table summarizes available URIs:

+----------------------+------------------------------------+-------+
|Resource              |URI                                 |Type   |
+======================+====================================+=======+
|classification_types  |/classification_types               |GET    |
+----------------------+------------------------------------+-------+
|classification_group  |/classification_groups/             |POST   |
+----------------------+------------------------------------+-------+
|classification_groups |/classification_groups              |GET    |
+----------------------+------------------------------------+-------+
|classification_group  |/classification_groups/{id}         |GET    |
+----------------------+------------------------------------+-------+
|classification_group  |/classification_groups/{id}         |PUT    |
+----------------------+------------------------------------+-------+
|classification_group  |/classification_groups/{id}         |DELETE |
+----------------------+------------------------------------+-------+
|classification        |/classifications/                   |POST   |
+----------------------+------------------------------------+-------+
|classifications       |/classifications                    |GET    |
+----------------------+------------------------------------+-------+
|classification        |/classifications/{id}               |GET    |
+----------------------+------------------------------------+-------+
|classification        |/classifications/{id}               |PUT    |
+----------------------+------------------------------------+-------+
|classification        |/classifications/{id}               |DELETE |
+----------------------+------------------------------------+-------+

The CCF should provide a way to mark a Classification Group as being in use (or increase the usage count) and a way to check for that and abort certain operations if the group is in use.

The CCF does not provide any mechanism to synchronize Classification Groups to Consuming Services.

Examples for a Classification Group with two Classifications inside.

To list available Classification Types:

GET /v2.0/classification_types

Response:
{
   "classification_types": [{"type": "ethernet"},
                            {"type": "ipv4"},
                            {"type": "ipv6"},
                            {"type": "tcp"},
                            {"type": "udp"}]
}

To create a Classification of type TCP:

POST /v2.0/classifications/
{
    "classification": {
        "name": "not_tcp_syns",
        "type": "tcp",
        "negated": true,
        "definition": {
            "control_flags": "0x2",
            "control_flags_mask: "0x2"
        }
    }
}

Response:
{
    "classification": {
        "id": "3dcc561a-1bb8-11e7-b615-23717626a4e5",
        "project_id": "0a36035e-1bb9-11e7-b8ef-e782361fd276",
        "name": "not_tcp_syns",
        "description": "",
        "type": "tcp",
        "negated": true,
        "shared": false,
        "definition": {
            "src_port": null,
            "dst_port": null,
            "control_flags": "0x2",
            "control_flags_mask: "0x2",
            "ecn": null,
            "ecn_mask": null,
            "min_window": null,
            "max_window": null,
            "min_data_offset": null,
            "max_data_offset": null,
            "option_kind": null
        }
    }
}

To create a Classification of type Ethernet:

POST /v2.0/classifications/
{
    "classification": {
        "name": "ipv4_over_eth",
        "type": "ethernet",
        "definition": {
            "ethertype": "0x800"
        }
    }
}

Response:
{
    "classification": {
        "id": "021c1ad2-1bb9-11e7-907d-937a75c8a5db",
        "project_id": "0a36035e-1bb9-11e7-b8ef-e782361fd276",
        "name": "ipv4_over_eth",
        "description": "",
        "type": "ethernet",
        "negated": false,
        "shared": false,
        "definition": {
            "negated": false,
            "preamble": null,
            "src_addr": null,
            "dst_addr": null,
            "ethertype": "0x800"
        }
    }
}

To create a Classification Group:

POST /v2.0/classification_groups/
{
    "classification_group": {
        "name": "no_syns_on_ipv4",
        "description": "Any IPv4 traffic carried over Ethernet except TCP SYNs.",
        "operator": "and",
        "classifications": [
            "3dcc561a-1bb8-11e7-b615-23717626a4e5",
            "021c1ad2-1bb9-11e7-907d-937a75c8a5db"
        ]
    }
}

Response:
{
    "classification_group": {
        "id": "387299fa-250d-11e7-8620-b38d21865984",
        "project_id": "0a36035e-1bb9-11e7-b8ef-e782361fd276",
        "name": "no_syns_on_ipv4",
        "description": "Any IPv4 traffic carried over Ethernet except TCP SYNs.",
        "shared": false,
        "operator": "and"
        "classifications": [
            "3dcc561a-1bb8-11e7-b615-23717626a4e5",
            "021c1ad2-1bb9-11e7-907d-937a75c8a5db"
        ],
        "classification_groups": []
    }
}

List Classification Groups:

GET /v2.0/classification_groups

Response:
{
    "classification_groups": [
        {
            "id": "387299fa-250d-11e7-8620-b38d21865984",
            "project_id": "0a36035e-1bb9-11e7-b8ef-e782361fd276",
            "name": "no_syns_on_ipv4",
            "description": "Any IPv4 traffic carried over Ethernet except TCP SYNs.",
            "shared": false,
            "operator": "and"
            "classifications": [
                "3dcc561a-1bb8-11e7-b615-23717626a4e5",
                "021c1ad2-1bb9-11e7-907d-937a75c8a5db"
            ],
            "classification_groups": []
        },
        {
            ...
        }
    ]
}

Community Impact

Services that intend to consume Classifications only have to:

Modify their existing REST APIs slightly in order to allow for Classification Group UUIDs to be passed, if they don’t already have an endpoint for such (e.g. networking-sfc doesn’t require port-chain API resource changes).
Implement the underlying fetching of Classification definitions, with support for OVO.

Other End User Impact

The OpenStack Client will be extended to support the Classifications API.

Possible CLI syntax of a hypothetical scenario with Neutron QoS as the Consuming Service and the Classification Group presented above in the example API calls, to illustrate the workflow:

$ openstack network classification create --type=tcp --control_flags=0x2 --control_flags_mask=0x2 --negated tcp_syns
$ openstack network classification create --type=ethernet --ethertype=0x800 ipv4_over_eth
$ openstack network classification group create --description "Any IPv4 traffic carried over Ethernet except TCP SYNs." \
      --classification tcp_syns --classification ipv4_over_eth --operator and no_syns_on_ipv4
$ openstack network qos rule create --type bandwidth-limit --max-kbps 8000 --classification-group no_syns_on_ipv4 myqospolicy

Alternatives

Possible alternatives to the data model and REST API presented are:

Neutron QoS -inspired model;
Using oslo.versionedobjects serialized into JSON for the REST API, to support the exact same resources specified in here.
With regards to the initial model, it could as well include Security Group Rule UUIDs, BGP VPN resource UUIDs, HTTP, and many more.

Implementation

Work has started as an initial Proof of Concept, available at [14]. After an initial merge on the neutron-classifier repository, work will continue towards the goals outlined in this spec.

Assignee(s)

Igor Duarte Cardoso (igor.duarte.cardoso@intel.com)

David Shaughnessy (david.shaughnessy@intel.com)

We request the Neutron team to provide access to the neutron-classifier repo.

Given the findings with the PoC code, there is no expectation that existing code in neutron-classifier will be reused, so the repository is to be wiped (but all the history will be kept).

Work Items

Prototype/PoC (Pike-1) (David) - Finished.
Finish spec (Pike) (Igor) - In Progress.
Acquire rights to merge on a repository (Pike) (Igor) - In Progress.
Implement first consumable version of this project (Pike to Queens) (David, Igor)
Bring first time support to a Neutron service (TBD) (Queens to Rocky)

Scorecard
N0 \| Y
N1 \| Y
N2 \| N
N3 \| Y
N4 \| Y
N5 \| Y
D1 \| Y
D2 \| Y
D3 \| Y
D4 \| Y
C1 \| Y
C2 \| Y
C3 \| Y
C4 \| N
C5 \| Y
C6 \| N
C7 \| N
C8 \| Y
R1 \| Y
R2 \| Y
R3 \| Y
R4 \| Y
S1 \| Y
L1 \| Y

Request Type	`GET`
Endpoint	`/v2.0/address-groups`
Response Codes	Success	200
	Error	Unauthorized(401)

Request Type	`POST`
Endpoint	`/v2.0/address-groups/`
Response Codes	Success	201
	Error	Unauthorized(401), Bad Request(400)

Request Type	`PUT`
Endpoint	`/v2.0/address-groups/<address_group_id>`
Response Codes	Success	200
	Error	Unauthorized(401), Bad Request(400) Not Found(404)

Request Type	`DELETE`
Endpoint	`/v2.0/address-groups/<address_group_id>`
Response Codes	Success	204
	Error	Unauthorized(401), Not Found(404) Conflict(409) The Conflict error response is returned when an operation is performed while address group is in use.