L3 router support NDP proxy¶
Launchpad Bug: https://bugs.launchpad.net/neutron/+bug/1877301
The IPv6’s NDP (Neighbor Discovery Protocol) is similar to IPv4’s ARP (Address Resolution Protocol) in functional, but the NDP works at L3. The NDP proxy [1] is also similar to ARP proxy [2]. But, the NDP proxy only works on IPv6, it can proxy specific IPv6 address’s NA (Neighbor Advertisement) that to response the NS (Neighbor Solicitation) which comes from the upstream router. The Linux kernel already implement NDP proxy. With these functionalities, we can implement some APIs so that users/tenants can advertise specific IPv6 addresses. It looks a little like IPv4’s floating IP, but this solution doesn’t do any NAT action.
Problem Description¶
As the IPv6 more and more popularize, we should provide a simple method to make the IPv6 VMs more easily and flexibly connect to external network.
In some use cases, such as a web site which has some DB services, MQ services and portal services etc, and they work on IPv6 VMs with same subnet. The administrator of the web site just would like to publish portal services to external. So the administrator needs some methods to just advertise a part of address to external.
The current solution of publishing IPv6 address which the Neutron support, such as BGP (Border Gateway Protocol) [3], PD (Prefix delegation) [4], is more complex, they require do some complex configurations in upstream router. This maybe not suit some company as their bare metal hosted in thrid-party IDC, they don’t have the privilege of the upstream router. So we should provide a solution with not depth cooperation of upstream router.
Proposed Change¶
Overview¶
Server Side¶
Implement a service plugin named
ndp_proxyto support the feature of this spec description. Administrator can enable the feature by append the plugin toservice_pluginin Neutron configuration file.Implement an API extension, In the extension we should define a named
ndp_proxyresource and its APIs.Add a new parameter
enable_ndp_proxyto router. If it is set to True, the ndp proxy will be enabled in router namespace.
Agent Side¶
Implement an extension of Neutron L3 agent to set relational rules for ndp proxy. The extension behavior like below:
If the router’s
enable_ndp_proxyfield is true, the router’s namespace should have a default ip6tables rule to drop all packets input from the router’s external gateway device. By this way, we can eliminate affection of extra route and neighbor in upstream router. So, if a router’senable_ndp_proxyset as True, the Prefix Delegation [4] and BGP Dynamic Routing [3] solution will have no effect, we should explain this in user document.For each ndp proxy object, the extension should set a neighbor proxy entry at the router’s external gateway device and set an ip6tables rule to permit the relational packets through. By this, the router only permits those IPv6 addresses which enabled the
ndp_proxyto communication with external network. This makes the ndp proxy’s behavior is more similar to IPv4’s floating ip.
Data Model Impact¶
The following new tables are added as part of the ndp proxy feature.
For ndp proxy resource:
CREATE TABLE ndp_proxies (
id CHAR(36) NOT NULL PRI KEY,
name VARCHAR(255),
router_id VARCHAR(36) NOT NULL FOREIGN KEY,
port_id VARCHAR(36) NOT NULL FOREIGN KEY,
ip_address VARCHAR(64) NOT NULL,
project_id VARCHAR(255),
standard_attr_id bigint(20) NOT NULL,
);
The router_id column will store the id of router which the ndp proxy belong
to. The port_id column will store the id of Neutron internal port which the
ip_address locate in. The ip_address column will store the IPv6 address
which we need to proxy.
For router’s enable_ndp_proxy parameter:
CREATE TABLE router_ndp_proxy_state (
router_id VARCHAR(36) NOT NULL FOREIGN KEY,
enable_ndp_proxy BOOL NOT NULL,
);
New Resource Extension¶
New resource extension ndp_proxy will be added. It will define the
ndp_proxy entry’s CURD APIs.
For this new feature, a new service plugin will be introduced, and the following methods will be added:
‘create_ndp_proxy()’
‘delete_ndp_proxy()’
‘update_ndp_proxy()’
‘get_ndp_proxy()’
‘get_ndp_proxies()’
So the attributes map of new resource would be like:
RESOURCE_ATTRIBUTE_MAP = {
'ndp_proxy': {
'id': {'allow_post': False,
'allow_put': False,
'validate': {'type:uuid': None},
'is_visible': True,
'primary_key': True},
'name': {'allow_post': True,
'allow_put': True,
'validate': {'type:string': 255},
'is_filter': True,
'is_sort_key': True,
'is_visible': True, 'default': ''},
'project_id': {'allow_post': True,
'allow_put': False,
'required_by_policy': True,
'validate': {'type:uuid': None},
'is_visible': True},
'router_id': {'allow_post': True,
'allow_put': False,
'validate': {'type:uuid': None},
'is_visible': True},
'port_id': {'allow_post': True,
'allow_put': False,
'validate': {'type:uuid': None},
'is_visible': True},
'ip_address': {'allow_post': True,
'allow_put': False,
'default': None,
'validate': {
'type:ip_address_or_none': None},
'is_visible': True}
'description': {'allow_post': True,
'allow_put': True,
'default': '',
'validate': {'type:string': 1024},
'is_visible': True}
}
}
Note
The ip_address parameter is optional, if not set it when user
send post request, the new service plugin will select a IPv6 address
from the port_id represented port.
REST API Impact¶
Extend router API¶
The idea is to extend the router Rest API with a new extension
router_ndp_proxy with the below defined attribute.
Attribute Name |
Type |
CRUD |
Default Value |
Description |
enable_ndp_proxy |
Boolean |
CRU |
False |
Whether the router enable ndp proxy function. |
The router extension definition would be expanded as :
RESOURCE_ATTRIBUTE_MAP = {
'routers': {
'enable_ndp_proxy': {
'allow_post': True, 'allow_put': True,
'convert_to': converters.convert_to_boolean_if_not_none,
'is_visible': True,
'is_filter': True,
}
}
}
Default, only admin user can update enable_ndp_proxy parameter. And, a new
config option enable_ndp_proxy_by_default will be introduced. If it set as
True, the enable_ndp_proxy will be set as True default.
For example, GET a router:
GET /v2.0/routers/<router-uuid>
{
"router": {
"admin_state_up": true,
"availability_zone_hints": [],
"availability_zones": [
"nova"
],
"created_at": "2018-03-19T19:17:04Z",
"description": "",
"distributed": false,
"enable_ndp_proxy": false,
"external_gateway_info": {
"enable_snat": true,
"external_fixed_ips": [
{
"ip_address": "172.24.4.6",
"subnet_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf"
},
{
"ip_address": "2001:db8::9",
"subnet_id": "0c56df5d-ace5-46c8-8f4c-45fa4e334d18"
}
],
"network_id": "ae34051f-aa6c-4c75-abf5-50dc9ac99ef3"
},
"flavor_id": "f7b14d9a-b0dc-4fbe-bb14-a0f4970a69e0",
"ha": false,
"id": "f8a44de0-fc8e-45df-93c7-f79bf3b01c95",
"name": "router1",
"revision_number": 1,
"routes": [
{
"destination": "179.24.1.0/24",
"nexthop": "172.24.3.99"
}
],
"status": "ACTIVE",
"updated_at": "2018-03-19T19:17:22Z",
"project_id": "0bd18306d801447bb457a46252d82d13",
"tenant_id": "0bd18306d801447bb457a46252d82d13",
"service_type_id": null,
"tags": ["tag1,tag2"],
"conntrack_helpers": []
}
}
Set a router’s enable_ndp_proxy parameter to True:
Post /v2.0/routers/<router-uuid>
Request body:
{
"router": {
"enable_ndp_proxy": true
}
}
For the new resource ndp_proxy, some new URLs will be introduced:
List NDP Proxies¶
GET /v2.0/ndp_proxies
The response body:
{
"ndp_proxies": [
{
"project_id": "ad239fa5-ceb7-8b81-abf5-b930d7f6ceb7",
"id": "ae34051f-aa6c-4c75-abf5-50dc9ac99ef3",
"name": "test01",
"port_id": "fa450s1f-aa6c-4c75-abf5-50dc9ac9df67",
"ip_address": "2001:217::19",
"created_at":"2020-05-21T11:33:21Z",
"updated_at":"2020-06-18T08:31:48Z",
"description": ""
},
{
"project_id": "ad239fa5-ceb7-8b81-abf5-b930d7f6ceb7",
"id": "915a14a6-867b-4af7-83d1-70efceb146f9",
"name": "test02",
"port_id": "4323401f-aa6c-4c75-abf5-50dc9ac99ef3",
"ip_address": "2001:218::12"
"created_at":"2020-05-21T11:33:21Z",
"updated_at":"2020-06-18T08:31:48Z",
"description": ""
}
]
}
Show NDP Proxy¶
GET /v2.0/ndp_proxies/<ndp-proxy-id>
The response body:
{
"ndp_proxy": {
"project_id": "ad239fa5-ceb7-8b81-abf5-b930d7f6ceb7",
"id": "ae34051f-aa6c-4c75-abf5-50dc9ac99ef3",
"name": "test01",
"port_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf"
"ip_address": "2001:217::19",
"created_at":"2020-05-21T11:33:21Z",
"updated_at":"2020-06-18T08:31:48Z",
"description": "Some descriptions"
}
}
Create NDP Proxy¶
POST /v2.0/ndp_proxies
The request body:
{
"ndp_proxy": {
"name": "test01",
"router_id": "5823deb7-8b81-ceb7-40a0-b930d7f6ceb7",
"port_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf",
"ip_address": "2001:217::19",
"description": "Some descriptions"
}
}
There are some constraints here:
The router’s
enable_ndp_proxyparameter must be set as True.The subnet that the
ip_addressallocated from must be added to the router.The network of the
port_idrepresented port belong to must have sameipv6_address_scope[5] with the network of router’s external gateway.
The response body:
{
"ndp_proxy": {
"id": "ad239fv5-aa6c-4c75-abf5-50dc9ac99ef3",
"name": "test01",
"project_id": "ad239fa5-ceb7-8b81-abf5-b930d7f6ceb7",
"router_id": "5823deb7-8b81-ceb7-40a0-b930d7f6ceb7",
"port_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf",
"ip_address": "2001:217::19",
"created_at":"2020-05-21T11:33:21Z",
"updated_at":"2020-06-18T08:31:48Z",
"description": "Some descriptions"
}
}
Update NDP Proxy¶
PUT /v2.0/ndp_proxies/<ndp-proxy-id>
The request body:
{
"ndp_proxy": {
"name": "test02",
"description": "New descriptions"
}
}
The response body:
{
"ndp_proxy": {
"id": "ad239fv5-aa6c-4c75-abf5-50dc9ac99ef3",
"name": "test02",
"project_id": "ad239fa5-ceb7-8b81-abf5-b930d7f6ceb7",
"router_id": "5823deb7-8b81-ceb7-40a0-b930d7f6ceb7",
"port_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf"
"ip_address": "2001:217::56",
"created_at":"2020-05-21T11:33:21Z",
"updated_at":"2020-06-18T08:31:48Z",
"description": "New descriptions"
}
}
Delete NDP proxy¶
DELETE /v2.0/ndp_proxies/<ndp-proxy-id>
This operation does not accept a request body and does not return a response body.
Addtionally, if a router or port will be deleted, the ndp proxies which relational with them will be delete cascade.
Effects on Existing Router APIs¶
Before remove the subnet from a router, neutron should check whether has any ndp proxy related to the subnet (whether the ndp proxy’s
ip_addresswas allocated from the subnet). If has any ndp proxy related to the subnet, the subnet can’t be removed.
L3 Agent Impact¶
Neutron needs to implement a new l3 agent extension to cooperate with the server API to set relational rules (neigh proxy and ip6tables, distributed router needs to set some extra route rules) in router’s namespace.
We assume user has a below scenario, then respectively describe the implementions of the feature about DVR router and Legacy router:
A subnet of which cidr is
2001::1:0/112A VM belong to the subnet and it’s IPv6 address is
2001::1:8A distributed/legacy router which set external gateway and connect to the subnet.
Legacy router Impact¶
Assume the router’s external gateway device is qg-733bd76b-62:
If the router’s parameter
enable_ndp_proxyis true, the extension need to set the kernel parameterproxy_ndpas1in the router’s qrouter-namespace, create a custom chain namedneutron-l3-agent-NDPand set a default iptables rule in it to drop all packets input from the router’s external gateway device. The executed commands like below:sysctl -w net.ipv6.conf.qg-733bd76b-62.proxy_ndp=1 ip6tables -N neutron-l3-agent-NDP ip6tables -A neutron-l3-agent-NDP -i qg-733bd76b-62 -j DROP
Note
If the router have no external gateway, the parameter
enable_ndp_proxy will be ignored.
When add the IPv6 subnet to the router (the router’s
enable_ndp_proxyalready set as true), Before user advertise the subnet’s address with ndp proxy, the subnet should drop all external traffic. So, the following cmd should be executed:ip6tables -I neutron-l3-agent-FORWARD -i qg-733bd76b-62 --destination 2001::1:0/112 -j neutron-l3-agent-NDP
By this, we can eliminate the effect of extra route and neighbor entries in upstream router.
For each ndp proxies, the extension should add a neigh proxy entry to the router external gateway device, and add ip6tables rule to permit the relational packets pass. The executed commands like below:
ip -6 neigh add proxy 2001::1:8 dev qg-733bd76b-62 ip6tables -I neutron-l3-agent-NDP -i qg-733bd76b-62 --destination 2001::1:8 -j ACCEPT
When remove a ndp proxy, the extension should remove related neigh proxy entry from the router external gateway device, and remove the related ip6tables rule to re-forbid relational packets pass. The executed commands like below:
ip -6 neigh del proxy 2001::1:8 dev qg-733bd76b-62 ip6tables -D neutron-l3-agent-NDP -i qg-733bd76b-62 --destination 2001::1:8
When router’s parameter
enable_ndp_proxyset to false, the extension needs to set the kernel parameterproxy_ndpas0in the router’s qrouter-namespace namespace, delete the custom chain namedneutron-l3-agent-NDP. The executed commands like below:sysctl -w net.ipv6.conf.qg-733bd76b-62.proxy_ndp=0 ip6tables -X neutron-l3-agent-NDP
HA router Impact¶
The implementation of the feature for HA router is same as Legacy router, except failover. When HA router’s state has changed, the extension should refresh the ndp proxy rules, because the ndp proxy rules may be lost after multible failover.
Note
When HA router’s state has changed, the keepalived will sends
unsolicited neighbor advertisement automatically. So, we don’t need
to write extra code to do this. During failover, the traffic will be
breaked, but it will recover soon if the failover accomplished.
DVR Router Impact¶
For DVR [6] router, it’s a little different from legacy and HA router. We need
to consider two scenes: If the neutron-l3-agent in compute node set as
dvr mode, the neutron-l3-agent will create a fip-namespace to process
north-south traffic, so we need to apply related rules in qrouter-namespace and
fip-namespace. If the neutron-l3-agent in compute node set as
dvr_no_external mode, the north-south traffic will be processed by
snat-namespace in network node, so we need to apply related rules in
qrouter-namespace and snat-namespace.
For dvr mode¶
Assume the fip-namespace’s fg-dev port is fg-84920cf6-5e; the port connect
fip-namespace to qrouter-namespace is fpr-ea902fe0-9 and it’s IPv6
address is fe80::8493:5bff:fe9b:8d93; the port connect qrouter-namespace
to fip-namespace is rfp-ea902fe0-9 and it’s IPv6 address is
fe80::a0a7:c5ff:fe2c:bade. The topology like below:
+---------------+
| |
|upstream router|
| |
+-------+-------+
|
+----------------------------+
| | compute node |
| | |
| +-------+--------+ |
| | fg-84920cf6-5e | |
| | | |
| | fip-namespace | |
| | | |
| | fpr-ea902fe0-9 | |
| +--------+-------+ |
| | |
| | |
| +--------+--------+ |
| | rfp-ea902fe0-9 | |
| | | |
| |qrouter-namespace| |
| | | |
| +-----------------+ |
| |
+----------------------------+
Note
We don’t need to set ip6tables rules for dvr router. Because just by adding or removing the related route in fip-namespace (as description below) can be the switch to enable/disable the IPv6 traffic.
Due to the current Neutron don’t support dvr with IPv6, the qrouter-namespace has no default route about IPv6. We should add the default route firstly if the router set external gateway, the default route’s next-hop shoule be the fpr-dev device’s IPv6 address, The executed command like this (Executed in qrouter-namespace):
ip route add default via fe80::8493:5bff:fe9b:8d93 dev rfp-ea902fe0-9
Because of the device which directly connects to upstream router is located in fip-namespace, the proxy entry should be set in fip-namespace. So, the below cmd should be executed in all fip-namespace:
sysctl -w net.ipv6.conf.fg-84920cf6-5e.proxy_ndp=1
For each ndp proxies the extension add a proxy entry to the fg-dev in fip-namespace, the proxy entry just add to one namespace which hosted in the node of the ndp proxy object’s port belong to, and add a route in the namespace so that the packets of which destination is the ndp proxy’s ip_address can be forwarded to qrouter-namespace. This cmds like below (Executed in fip-namespace):
ip -6 neigh add proxy 2001::1:8 dev fg-84920cf6-5e ip route add 2001::1:8 via fe80::a0a7:c5ff:fe2c:bade dev fpr-ea902fe0-9
When remove a ndp proxy, the extension should remove related neigh proxy entry from the fg-dev, and remove the related route, This cmds like below (Executed in fip-namespace):
ip -6 neigh del proxy 2001::1:8 dev fg-84920cf6-5e ip route del 2001::1:8 via fe80::a0a7:c5ff:fe2c:bade dev fpr-ea902fe0-9
Because setting of ip6tables rules is not required, so for the change of
enable_ndp_proxy, the agent extension needn’t do any thing.If an instance was migrated and it’s port was related to a ndp proxy entry. The extension should delete related rules in old host and create them in new host. Additionally, the extension should send a NA (Neighbour Advertisement) to fresh the upsteam router’s neighbor entry so that the external traffic can forward to new host’s fip-namespace immediately.
For dvr_no_external mode¶
Assume the snat-namespace’s qg-dev is qg-87059c6c-a9, the sg-dev is
sg-68bcdb7b-a2; the qrouter-namespace’s qr-dev is qr-50457f9b-98. The
topology like below:
+---------------+
| |
|upstream router|
| |
+-------+-------+
|
+-----------------------+
|network | |
| node | |
| +-------+------+ |
| |qg-87059c6c-a9| |
| | | |
| |snat-namespace| |
| | | |
| |sg-68bcdb7b-a2| |
| +-------+------+ |
| | |
+-----------------------+
|
+-----------------------+
|compute | |
| node | |
| +--------+--------+ |
| | qr-50457f9b-98 | |
| | | |
| |qrouter-namespace| |
| | | |
| +-----------------+ |
| |
+-----------------------+
For this mode, in qrouter-namespace we just need to add a default route, so that the north-south traffic can be redirected to snat-namespace (The current neutron code already completed this demand). For snat-namespace we just treat is as legacy router’s qrouter-namepsace, about it’s rules process we can refer to Legacy router Impact.
OVN backend impact¶
The ndp_proxy for OVN L3 backend is not covered by this proposal. If it was
proved to be feasible, we should implement the feature base on OVN backend in
the future. But for now, we will just implement this base on Neutron L3 agent.
Implementation¶
Assignee(s)¶
yangjianfeng
Work Items¶
API Implementation
DB Implementation
Reference implementation
Tests
Documentation
Testing¶
Tempest Tests¶
The functions that the spec proposed need the external hardware router’s
support (need to add a direct route entry at upstream router). This is
difficult for tempest to do this. So, we can skip the scenario tests
firstly.
Functional Tests¶
Need to add functional tests
API Tests¶
Need to add API tests
Fullstack Tests¶
Need to add Fullstack tests.
Documentation Impact¶
User Documentation¶
Needs user documentation
Developer Documentation¶
Needs devref documentation
API reference¶
Needs API reference documentation
