Floating IP rate limit

RFE: https://bugs.launchpad.net/neutron/+bug/1596611

Floating IP bandwidth is unrestricted now. But the NIC and the export of data center may have a limitation for a cloud deployment. Then floating IP rate limit is needed.

This spec describes how to add a built-in extension to limit the floating IP bandwidth.

Problem Description

Neutron now has floating IP whose bandwidth is not restricted, then there are several reasons for adding rate limit to floating IP:

  • a) Currently, neutron QoS implementation only affects neutron ports, more detail is that the bandwidth restriction is based on the neutron port, so all the VM traffic will be limited under that restriction, including L3 traffic. L3 needs its IP based implementation.

    1. North/South traffic always rely on infrastructure networking capacity.

  • c) In case of Floating IP Traffic, cloud deployment do not have enough bandwidth to meet the total bandwidth requirement of all tenants at the same time. Due to this, the SLA of user’s network traffic cannot be guaranteed. For example, if the VM NICs are given a high QoS value, then to meet the requirement of each tenant, the DC network would have a heavy load, thus it may be unable to facilitate additional requests and affect not only the tenant network bandwidth from North-South, but also East-West traffic.

  • d) SNAT traffic in centralized network node will not meet the needs of all tenant bandwidth either. Because a NIC has limited bandwidth.

Proposed Change

Overview

We want to limit the bandwidth of floating IP. At the same time, the east/west traffic should have no effect.

Some agreements:

  • Allow binding QoS policy to floating IP.

  • Each floating IP should have only one QoS policy.

  • L3 agent needs a QoS extension to process L3 IP related QoS rules.

  • QoS policy parameter is nullable. Floating IP can have no QoS policy assigned.

How the change works:

  • Linux TC (Traffic Control) [1] will be used to implement such functionality. And for egress traffic a HTB [2] qdisc will be used to limit the floating IP outgoing bandwidth.

Solution Proposed

Server side changes

  • New floating IP extension for QoS attribute (qos_policy_id).

  • Adding a new relationship table between floating IP and QoS policy QoSFIPPolicyBinding.

  • L3 plugin will return the floating IP list with its bonding QoS policy during router sync RPC.

L3 agent side TC rules

Where to install the TC rules:

  • For HA/legacy routers, rules for floating IP will be installed into network node qrouter namespace, and the traffic control device will be qg-device.

  • For DVR routers, it’s compute node qrouter namespace, but the floating IP traffic control device will be rfp-device (one of qrouter-namespace to fip-namespace pair).

For all IPs, in the corresponding namespace, the egress and ingress qdisc rules are basically the same. The {qdisc_id} will be used to create the IPs’ tc filter. Some tips:

  • Remember all the following commands will be executed in a namespace.

  • [a-device] represents the qg-device or rfp-device in a namespace.

  • {egress/ingress_qdisc_id} is the qdisc handle id.

Create qdisc commands:

# egress
tc qdisc add dev [a-device] root htb

# ingress
tc qdisc add dev [a-device] ingress

Actual qdisc rules:

$ tc qdisc show dev [a-device]
# egress
qdisc htb {egress_qdisc_id} root refcnt 2 r2q 10 default 0 direct_packets_stat 400

# ingress
qdisc ingress {ingress_qdisc_id} parent ffff:fff1 ----------------

Then we can use that {ingress_qdisc_id} and {egress_qdisc_id} to create the L3 IPs filters. Assuming we have a L3 IP 172.16.6.161, and it has a QoS policy with rule {qos_rate_value: 1000Kbit, qos_burst_value: 1Mb}. The ingress and egress tc filter create commands will be:

# ingress
$ tc filter add dev [a-device] parent {ingress_qdisc_id} protocol ip prio 1 \
  u32 match ip dst 172.16.6.161 police \
  rate 1000Kbit burst 1Mb drop flowid :1

# egress
$ tc filter add dev [a-device] parent {egress_qdisc_id} protocol ip prio 1 \
  u32 match ip src 172.16.6.161 police \
  rate 1000Kbit burst 1Mb drop flowid :1

Actual filter rules:

# ingress
$ tc -s -d -p filter show dev [a-device] parent {ingress_qdisc_id}
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP dst 172.16.6.161/32 (success 0 )
 police 0x67 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
...

# egress
$ tc -s -d -p filter show dev [a-device] parent {egress_qdisc_id}
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP src 172.16.6.161/32 (success 0 )
 police 0x68 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
...

The neutron basic workflow

Floating IP

    1. User create a floating IP.

    1. Floating IP was associated with a port.

    1. Create/find QoS policy and rule for floating IP.

    1. Bind QoS policy to floating IP (floating IP update API).

    1. L3 agent processes the router and its floating IP.

    1. L3 agent sets the TC rules to the qrouter-namespace relevant device.

After this the floating IP will be set under a bandwidth restriction.

Example L3 agent side TC rules

Assuming that we have a legacy router: cf6951cb-b050-4543-9742-c63a4989edae, gateway ip: 172.16.6.167, 1Mbps, floating IP: 172.16.10.69, 1Mbps, then in its scheduled network node you can get the following rules:

# ip netns exec qrouter-cf6951cb-b050-4543-9742-c63a4989edae ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    inet 127.0.0.1/8 scope host lo
140: qr-4aba9b05-36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
    inet 192.168.233.1/24 brd 192.168.233.255 scope global qr-4aba9b05-36
141: qr-aef0d42d-f9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
    inet 192.168.232.1/24 brd 192.168.232.255 scope global qr-aef0d42d-f9
143: qg-c99a5832-7f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UNKNOWN
    inet 172.16.6.167/16 brd 172.16.255.255 scope global qg-c99a5832-7f
    inet 172.16.10.69/32 brd 172.16.10.69 scope global qg-c99a5832-7f
       valid_lft forever preferred_lft forever

# ip netns exec qrouter-cf6951cb-b050-4543-9742-c63a4989edae tc qdisc show dev qg-c99a5832-7f
qdisc htb 801b: root refcnt 2 r2q 10 default 0 direct_packets_stat 400
qdisc ingress ffff: parent ffff:fff1 ----------------

# ip netns exec qrouter-cf6951cb-b050-4543-9742-c63a4989edae tc -s -d -p filter show dev qg-c99a5832-7f parent 801b:
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP src 172.16.6.167/32 (success 0 )
 police 0xa74 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1

 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
filter protocol ip pref 1 u32 fh 800::801 order 2049 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP src 172.16.10.69/32 (success 0 )
 police 0xa76 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1

 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)

# ip netns exec qrouter-cf6951cb-b050-4543-9742-c63a4989edae tc -s -d -p filter show dev qg-c99a5832-7f parent ffff:
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP dst 172.16.6.167/32 (success 0 )
 police 0xa73 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1

 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
filter protocol ip pref 1 u32 fh 800::801 order 2049 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP dst 172.16.10.69/32 (success 0 )
 police 0xa75 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1

 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)

And DVR router 0580788c-c919-447c-aea1-87d415aa173a with floating IP 172.16.6.161 1Mbps in a compute node:

# ip netns exec qrouter-0580788c-c919-447c-aea1-87d415aa173a ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    inet 127.0.0.1/8 scope host lo
24: rfp-0580788c-c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP qlen 1000
    inet 169.254.106.114/31 scope global rfp-0580788c-c
    inet 172.16.6.161/32 brd 172.16.6.161 scope global rfp-0580788c-c
102: qr-43185e93-af: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
    inet 192.168.199.1/24 brd 192.168.199.255 scope global qr-43185e93-af
104: qr-51635a61-46: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
    inet 192.168.198.1/24 brd 192.168.198.255 scope global qr-51635a61-46

# ip netns exec qrouter-0580788c-c919-447c-aea1-87d415aa173a tc qdisc show dev rfp-0580788c-c
qdisc htb 801e: root refcnt 2 r2q 10 default 0 direct_packets_stat 5
qdisc ingress ffff: parent ffff:fff1 ----------------

# ip netns exec qrouter-0580788c-c919-447c-aea1-87d415aa173a tc -s -d -p filter show dev rfp-0580788c-c parent 801e:
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP src 172.16.6.161/32 (success 0 )
 police 0x68 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1

 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)

# ip netns exec qrouter-0580788c-c919-447c-aea1-87d415aa173a tc -s -d -p filter show dev rfp-0580788c-c parent ffff:
...
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1  (rule hit 0 success 0)
  match IP dst 172.16.6.161/32 (success 0 )
 police 0x67 rate 1000Kbit burst 1Mb mtu 64Kb action drop overhead 0b
ref 1 bind 1

 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)

IPv6 Impact

Only IPv4 has the concept of floating IP.

Data Model Impact

For floating IP: A new table, QoSFIPPolicyBinding, will be created to represent the 1-1 association between floating IP and QoS policy, each policy has rules in both ingress and egress direction.

REST API Impact

New floating IP API extension:

floatingip: {
       ...
       'qos_policy_id': {'allow_post': True, 'allow_put': True,
                         'validate': {'type:uuid_or_none': None},
                         'is_visible': True,
                         'default': None},}

New floating IP commands for creation, deletion and update elements with QoS property:

openstack floating ip create --qos-policy policy_id <network>
openstack floating ip set --qos-policy policy_id floating_ip_id
openstack floating ip unset --qos-policy floating_ip_id

Implementation

Assignee(s)

Work Items

  • Create model tables and API extensions.

  • New TC command wrapper for floating IP rate limit.

  • L3 agent extension for QoS rule installation.

  • CLI support.

  • Testing.

  • Documentation.

Dependencies

None

Testing

Functionality

The bandwidth of floating IP should be restricted properly. Adding new fullstack and tempest scenarios to test the bandwidth of ingress and egress.

References