QoS Rule Type Packet per Second

RFE: https://bugs.launchpad.net/neutron/+bug/1912460

Neutron supports bandwidth rate limit for ports and L3 IPs. But packet rate limit (packet per second) is not available, although it is a common measurement.

So, this spec describes adding a new QoS rule type packet per second (pps).

Problem Description

Packet per second is a very general network performance metric. Like bandwidth, it is usually used to evaluate the packet forwarding performance of a device.

For cloud providers, to limit the packet per second (pps) of VM NIC is popular and sometimes essential. Transit large set of packets for VM in physical compute hosts will consume the CPU and physical NIC I/O performance. For small packets, even if the bandwidth is low, the pps can still be higher. Without the limitation, it can be an attack point inside the cloud while some VMs are becoming hacked.

For L2 drivers like ovs and ovn, it may get extremly high usage of CPU when user send small packet (typically 64B small) from the VM to the others, even if the device has lower QoS bandwidth limitation. Then your host services and other users’ VMs will be under a higher failure point.

For network quality assurance, the resource consumption of the system is determined according to the user’s VM specifications. With the pps limitation, the VM will not consume more CPU with smaller bandwidth.

Proposed Change

Adding new API extension to QoS service plugin to allow CURD actions for packet rate limit (packet per second) rule.

Note

We will not elaborate the real limitation in L2/L3 backend, this spec only shows how we add a new QoS rule type.

Server side changes

A new API extension of Neutron will be added with new resource PacketRateLimitRule:

qos_apidef.SUB_RESOURCE_ATTRIBUTE_MAP = {
    'packet_rate_limit_rules': {
        'parent': qos_apidef._PARENT,
        'parameters': {
            qos_apidef._QOS_RULE_COMMON_FIELDS,
            'max_kpps': {
                'allow_post': True, 'allow_put': True,
                'convert_to': converters.convert_to_int,
                'is_visible': True,
                'is_filter': True,
                'is_sort_key': True,
                'validate': {
                    'type:range': [0, db_const.DB_INTEGER_MAX_VALUE]}
            },
            'max_burst_kpps': {
                    'allow_post': True, 'allow_put': True,
                    'is_visible': True, 'default': 0,
                    'is_filter': True,
                    'is_sort_key': True,
                    'convert_to': converters.convert_to_int,
                    'validate': {
                        'type:range': [0, db_const.DB_INTEGER_MAX_VALUE]}
            },
            'direction': {
                'allow_post': True,
                'allow_put': True,
                'is_visible': True,
                'is_filter': True,
                'is_sort_key': True,
                'default': constants.EGRESS_DIRECTION,
                'validate': {
                    'type:values': constants.VALID_DIRECTIONS}
            }
        }
    }
}

Note

The unit for the rate and burst is kilo (1000) packets per second, so the value range will be 1 kpps to 2147 gpps.

Potential agent side enhancements

Each of the following will be an independent and huge proposal, we will not describe the detail here.

• apply pps rule to L3 IPs in agent side by iptables rule [1].

• apply pps rule to VM port by ovs meter [2] [3] [4].

• apply pps rule to router ports (gateway port, router interface) by iptables rule.

• apply pps rule to L3 IPs and ports to OVN related devices by ovs meter.

User use cases

After the L2/L3 backends have the real abilities to limit the pps, users can use the pps rule in the following scenarios:

• “pps” rule for floating IP, using the L3 agent

• “pps” rule for floating IP, using OVN L3 agent

• “pps” rule for gateway IP, using the L3 agent

• “pps” rule for gateway IP, using OVN L3 agent

• “pps” rule for ML2 OVS VM ports

• “pps” rule for ML2 OVN VM ports

• “pps” rule for router gateway or interface ports

Data Model Impact

Add table qos_packet_rate_limit_rules:

op.create_table(
    'qos_packet_rate_limit_rules',
    sa.Column('id', sa.String(36), nullable=False,
              index=True),
    sa.Column('qos_policy_id', sa.String(36),
              nullable=False, index=True),
    sa.Column('max_kpps', sa.Integer()),
    sa.Column('max_burst_kpps', sa.Integer()),
    sa.Column('direction', sa.Enum(constants.EGRESS_DIRECTION,
                                   constants.INGRESS_DIRECTION,
                                   name="directions"),
              nullable=False,
              server_default=constants.EGRESS_DIRECTION),
    sa.PrimaryKeyConstraint('id'),
    sa.ForeignKeyConstraint(['qos_policy_id'], ['qos_policies.id'],
                            ondelete='CASCADE')
)

Add DB model QosPacketRateLimitRule:

class QosPacketRateLimitRule(model_base.HasId, model_base.BASEV2):
    __tablename__ = 'qos_packet_rate_limit_rules'
    qos_policy_id = sa.Column(sa.String(36),
                              sa.ForeignKey('qos_policies.id',
                                            ondelete='CASCADE'),
                              nullable=False)
    max_kpps = sa.Column(sa.Integer)
    max_burst_kpps = sa.Column(sa.Integer)
    revises_on_change = ('qos_policy',)
    qos_policy = sa.orm.relationship(QosPolicy, load_on_pending=True)
    direction = sa.Column(sa.Enum(constants.EGRESS_DIRECTION,
                                  constants.INGRESS_DIRECTION,
                                  name="directions"),
                          default=constants.EGRESS_DIRECTION,
                          server_default=constants.EGRESS_DIRECTION,
                          nullable=False)
    __table_args__ = (
        sa.UniqueConstraint(
            qos_policy_id, direction,
            name="qos_packet_rate_limit_rules0qos_policy_id0direction"),
        model_base.BASEV2.__table_args__
    )

With OVO object:

@base.NeutronObjectRegistry.register
class QosPacketRateLimitRule(QosRule):

    db_model = qos_db_model.QosPacketRateLimitRule

    fields = {
        'max_kpps': obj_fields.IntegerField(nullable=True),
        'max_burst_kpps': obj_fields.IntegerField(nullable=True),
        'direction': common_types.FlowDirectionEnumField(
            default=constants.EGRESS_DIRECTION)
    }

    duplicates_compare_fields = ['direction']

    rule_type = constants.RULE_TYPE_PACKET_RATE_LIMIT

REST API Impact

GET: List packet rate limit rules for QoS policy

• /v2.0/qos/policies/{policy_id}/packet_rate_limit_rules

Response:

{
  "packet_rate_limit_rules": [
      {
          "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c793",
          "max_kpps": 10000,
          "max_burst_kpps": 0,
          "direction": "egress"
      }
  ]
}

POST: Create packet rate limit rule

• /v2.0/qos/policies/{policy_id}/packet_rate_limit_rules

Request:

{
  "packet_rate_limit_rule": {
      "max_kpps": "10000"
  }
}

Response:

{
  "packet_rate_limit_rule": {
      "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c793",
      "max_kpps": 10000,
      "max_burst_kpps": 0,
      "direction": "egress"
  }
}

GET: Show packet rate limit rule details

• /v2.0/qos/policies/{policy_id}/packet_rate_limit_rules/{rule_id}

Response:

{
  "packet_rate_limit_rule": {
      "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c793",
      "max_kpps": 10000,
      "max_burst_kpps": 0,
      "direction": "egress"
  }
}

PUT: Update packet rate limit rule

• /v2.0/qos/policies/{policy_id}/packet_rate_limit_rules/{rule_id}

Request:

{
  "packet_rate_limit_rule": {
      "max_kpps": 10000
  }
}

Response:

{
  "packet_rate_limit_rule": {
      "id": "5f126d84-551a-4dcf-bb01-0e9c0df0c794",
      "max_kpps": "10000"
  }
}

DELETE: Delete packet rate limit rule

• /v2.0/qos/policies/{policy_id}/packet_rate_limit_rules/{rule_id}

And, neutron will allow attaching new PacketRateLimitRule to QoS policy.

The Neutron basic workflow

1. User creates QoS policy

2. Creates packet rate limit rules with multiple directions to this QoS policy

3. Attaching this QoS policy to a port

4. (No available) related L2 driver apply PPS limitation driver rule to the port

5. Attaching this QoS policy to a L3 IP (floating IP or gateway IP).

6. (No available) related L3 driver apply PPS limitation driver rule to the IP

Implementation

Assignee(s)

• LIU Yulong <i@liuyulong.me>

Work Items

• Adding API extension and DB models for neutron server.

• Testing.

• Documentation.

Dependencies

None

Testing

Unit test cases to verify the DB rules are created/updated/deleted.

References

[1]

https://linux.die.net/man/8/iptables

[2]

http://workshop.netfilter.org/2017/wiki/images/d/db/Nfws-ovs-metering.pdf

[3]

http://www.openvswitch.org//support/dist-docs/ovs-ofctl.8.txt

[4]

https://github.com/openvswitch/ovs/blob/master/NEWS#L312