.. This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode ============================================ Dynamic Advertising Routes for Public Ranges ============================================ `Blueprint Link `_ The goal of this blueprint is to add dynamic routing capability to OpenStack deployment. This feature would allow Neutron to advertise its own routes to an external router using the BGP protocol. Problem Description =================== This is a new feature and this section will describe use cases where dynamic routing may be useful. **Expose External Networks Dynamically** Allow Neutron to dynamically announce to external uplink routers. An OpenStack cloud would run a routing protocol (for example, BGP) against at least one router in each uplink network provider. By announcing network prefixes to those peers, the Neutron network would be reachable by the rest of the internet via both paths. If the link to an uplink provider broke, the failure information would propagate to routers further up the stream, keeping the cloud reachable through the remaining healthy link. **Sample Topology** .. nwdiag:: nwdiag { network external { gateway_router[color = red]; tenant_router1; tenant_router2; } group { color = lightgreen; //label = "OpenStack Deployment"; gateway_router; tenant_router1; tenant_router2; } network isp_net1 { gateway_router; uplink_router1; } network isp_net2 { gateway_router; uplink_router2; } } Please note that in this diagram, the red gateway router is actually a linux machine with a software BGP speaker and a software router. This blueprint proposes to configure this BGP speaker through the Neutron API (see below). This also would be a valid scenario for an internal cloud, where the uplink router is actually one's provider router and the routes are advertised using iBGP. **Routed Model for Floating IPs on External Network** This use case will allow an external network with a large public IP space to possibly span more than one L2 network. This could allow for improved scale and isolation between AZs while maintaining the appearance of a large external network where floating IPs and networks can float freely. This use case may also include announcing public networks behind Neutron routers to an upstream router. For example, this will be useful for IPv6 networks when using the `subnet-allocation` mechanism to allocate routable IPv6 prefixes to tenant networks. It does not require learning routes from the upstream router. `Address Scopes`_ blueprint adds a new L3 context to make public routable tenant networks (among other advantages). Address Scopes may leverage this development to advertise the new routable subnets to uplink routers. **Sample Topology** .. nwdiag:: nwdiag { network external { tor_switch1; tor_switch2; } tor_switch1 -- l2_network1; tor_switch2 -- l2_network2; group { color = lightgreen; tor_switch1; l2_network1; } group { color = yellow; tor_switch2; l2_network2; } } The topology of this use case can be seen as a generalization of the previous one, with a multi-homed OpenStack installation and leverage the fact that a floating IP can be seen as a /32 network. Proposed Change =============== Overview -------- A new system that dynamically advertises routes to other peers outside the OpenStack deployment is proposed. From the Neutron API, cloud administrator should be able to define these peers as well as the way to interact with them. Routing Peer ++++++++++++ A system that supports dynamic routing must be able to advertise its own routes to its peers. It order to achieve this goal the system must know the list of its peers and be able to trust the information that it receives from them. Route Advertisement +++++++++++++++++++ Routes added manually by administrator will be advertised to external dynamic routing peers. Dynamic Routing System ++++++++++++++++++++++ In default implementation a new system will be used to manage dynamic routing information at the edge of OpenStack deployment. Dynamic peering will not be performed by each individual Neutron router due to scaling concerns. Such an approach could create an inordinately high number of peering relationships. Instead, the model proposed sets up a speaker to represent the Neutron deployment as a whole to external routers. This system will allow different implementations (develop your own BGP speaker implementation, for instance) to fit into third party requirements. IPv6 Impact ----------- The implementation must be able to exchange IPv4 and IPv6 routes. This blueprint may be even more important with IPv6 to complete routing from outside the cloud to tenant networks running IPv6. Solution Proposed ----------------- Reference implementation will employ a dynamic routing agent (``dr_agent``). This agent will have direct connectivity with configured peers and it will be responsible to update the dynamic routes of some defined routers. Another spec will be filled to define the details of this implementation. So, through the Neutron API, cloud administrator should be able to define the peer connections for each agent according to his needs. The cloud administrator will be able to execute the following new actions, through the Neutron API: 1. Create a ``BGPSpeaker`` entity with configuration options of the BGP connection. First approach will only need the ``local_as`` attribute. Future implementations, such as the `policy support for dynamic routing`_, will need to add more attributes here. The implementation driver need to read this configuration options an establish BGP sessions to its peers. 2. Create a ``RoutingPeer`` entity. 3. Associate a ``RoutingPeer`` entity to a ``BGPSpeaker``. The ``BGPSpeaker`` implementation driver will be able to connect to this ``RoutingPeer`` 4. Associate a ``Router`` to a ``BGPSpeaker``. That means the router will have dynamic routing capabilities that will let it advertise its routes 5. Associate a ``Network`` to a ``BGPSpeaker``. All the routers with external gateways attached to this network will use the same ``BGPSpeaker`` attributes. At this point, Neutron will compare the address scope of the subnets to which assigned routers (step 5 implies all the routers of the external network) have their external gateways connected to the internal subnets of the tenants to see if they belong to the same address scope. If so, the BGP speaker implementation will advertise these tenant subnets to its configured ``RoutingPeers``. Floating IPs will be included implicitly since they are allocated from the external network. Considerations -------------- Alternatives ------------ Multi-homed clouds can be handled using classic networking infrastructure, configuring manually the vendor router with BGP outside the OpenStack deployment. This is limited. It can't handle the routed floating ip model proposed above. Data Model Impact ----------------- This document proposes modifying data objects and schema in the following way. For a quick glance of the Data Object Model, check out this etherpad_. Data Object Changes +++++++++++++++++++ Three new data model classes will be added: ``BGPSpeaker``, ``RoutingPeer`` and ``AdvertiseRoute``. We will need the binding entities: * ``RoutingPeerBGPSpeakerBinding`` to associate peers to ``BGPSpeaker``. * ``RouterBGPSpeakerBinding`` to associate routers ``BGPSpeaker``. * ``NetworkBGPSpeakerBinding`` to associate networks to ``BGPSpeaker``. New ``BGPSpeaker`` class will contain the following attributes: * ``id``: UUID of the entity. * ``local_as``: Local AS value. Now we only need these values. In the future, more advanced configuration options of ``BGPSpeaker`` will be able to be added here. New ``RoutingPeer`` class that represents a peer connection will contain the following attributes: * ``id``: UUID of the entity * ``ip``: IP Address of the peer. * ``remote_as``: Remote Peer's AS value. * ``auth``: Authentication data of the connection that can be serialized as a dictionary. { ``type``: 'MD5', ``password``: '234a23d10234' } could be a simple example and first approach. Another data object called ``AdvertiseRoute`` will be created extending the ``Route`` entity and associated to a router. Will have the following attributes: * ``nexthop``: IP address of the next hop. * ``destination``: CIDR prefix. * ``router_id``: UUID of the Routing Instance. ``RoutingPeerBGPSpeakerBinding`` is an n-to-n relationship between bgp peer and bgp speaker and only will have: * ``peer_id``: UUID of the BGPPeer * ``bgpspeaker_id``: UUID of the Dynamic Routing Agent ``RouterBGPSpeakerBinding`` is an n-to-1 relationship with the attributes: * ``router_id``: UUID of the Router * ``bgpspeaker_id``: UUID of the Dynamic Routing Agent. ``NetworkBGPSpeakerBinding`` is an n-to-1 relationship with the attributes: * ``network_id``: UUID of the Network * ``bgpspeaker_id``: UUID of the Dynamic Routing Agent. ``RouterBGPSpeakerBinding`` it will be used when you want to propagate a single router's AdvertiseRoutes. This option is suitable for the ``Expose External Networks Dynamically`` use case, explained above. It can be seen as adding dynamic routing capabilities to a Router ``NetworkBGPSpeakerBinding`` is the ``Routed Model for Floating IPs on External Network`` use case: in this case you propagate tenant routes (IPv6) or Floating IPs into an upstream router. These routes are more dynamic to assign, because it is up to the tenant use to set them. You will want any router attached to a network, to automatically be added to propagate its routes. REST API Impact --------------- API endpoints should be implemented according to the Solution Proposed section. Security Impact --------------- This feature will allow an external system to manipulate routing information within Neutron network. The external system should be trusted and may be authenticated using a shared secret. Dynamic routing may only be configured by the system administrator. Notifications Impact -------------------- A notification should be provided when connectivity of control channel over which routes are exchanged is interrupted Other End User Impact --------------------- The following CLI commands will be added to manage dynamic routing specification for connecting OpenStack to outside networks: * **bgp-speaker-list**: List configured bgpspeakers. * **bgp-speaker-show**: Show detailed bgpspeaker configuration. * **bgp-speaker-create** Create new bgpspeaker connection. * **bgp-speaker-update**: Update bgpspeaker specification. * **bgp-speaker-delete**: Delete bgpspeaker specification. * **bgp-speaker-peer-add**: Associate a peer to a bgpspeaker. * **bgp-speaker-peer-list**: List Peers on bgpspeaker * **bgp-speaker-peer-remove**: Remove peers on bgpspeaker. * **bgp-speaker-network-add**: Associate a network to a bgpspeaker. * **bgp-speaker-network-list**: List networks on bgpspeaker * **bgp-speaker-network-remove**: Remove networks on bgpspeaker. * **bgp-speaker-router-add**: Associate a peer to a bgpspeaker. * **bgp-speaker-router-list**: List routers on bgpspeaker * **bgp-speaker-router-remove**: Remove router on bgpspeaker. * **bgp-peer-list**: List configured peers. * **bgp-peer-show**: Show detailed peer configuration. * **bgp-peer-create** Create new peer connection. * **bgp-peer-update**: Update peer specification. * **bgp-peer-delete**: Delete peer specification. * **router-advertiseroutes-list**: List advertise routes. Horizon Requirements ++++++++++++++++++++ A new screen will be added to configure gateway configuration for connecting OpenStack to outside networks. This screen will allow routes and peer configuration to be added to gateway configuration. An external network will have an option to be linked to a routing instance. Usage Example +++++++++++++ Configure 2 uplinks for the routing instance serving an external network to advertise its routes and update the discovered ones. Sample configuration using Neutron CLI commands: :: neutron bgp-speaker-create --local-as 12345 neutron bgp-peer-create --ip 123.23.43.4 --remote-as=1234 neutron bgp-speaker-peer-add peer1 (previous step created peer. should be an uuid, but modified for the sake of understanding) neutron bgp-speaker-network-add bgpspeaker1 network1 (same here, with uuids) Performance Impact ------------------ This feature describes an out of band mechanism to negotiate routing configuration. This feature should not have a performance impact on Neutron network. Other Deployer Impact --------------------- This feature would have to explicitly enabled and configured before it will take effect. There are no changes to configuration files. Developer Impact ---------------- This change does not affect current developments or any plugin development. Neutron API exposed is agnostic of the exchange routing protocol used. If another developer want to provide other driver than BGP with exabgp, only the ``dr_agent`` part will be affected with new code. Community Impact ---------------- This change does not impact community. Alternatives ------------ Taking into account the use case (BGP connectivity), we think that the agent approach is the only one that fits into Neutron. Although maybe the functionality would be solved using another entities and workflow. Implementation ============== Assignee(s) ----------- This is a pre-liminary contributor list Primary assignee: tidwellr vikram Other contributors: devvesa YAMAMOTO Work Items ---------- * Create the ``dr_agent``, exposing the API and implemented with the chosen BGP speaker. (`BGP Comparison`_) * Model tables and API resources. * Periodically scheduled process to communicate with ``dr_agent``. * Testing. * Devstack. * Documentation. Dependencies ============ Depending on the implementation, new system library or python library will need to be installed. Testing ======= Tempest Tests ------------- Dynamic routing testing may be performed in an isolated environment. An external autonomous system may be simulated with an instance of BGP capable software router (for example, quagga). The following dynamic routing scenarios could be tested: Verify that when BGP is enabled on the gateway and one peer is configured the agent establishes BGP session with the peer, receives a list of routes, and submits advertised routes to the peer. Verify that when BGP is disabled on the gateway and one peer is configured the dr_agent establishes no BGP sessions. Verify that when BGP is enabled on the agent and 3 BGP peer connections are configured, the agent establishes 3 BGP sessions, one to each of the configured peers. When 2 or more peers are configured, verify that BGP implementation is able to detect when the BGP session is interrupted the routes received from that BGP session are automatically removed from the routing table. Functional Tests ---------------- Full top-down Neutron API internal logic must be developed by mocking the agent. API Tests --------- All API exposed endpoints by Neutron extensions must be tested. Documentation Impact ==================== New documentation for the whole functionality. User Documentation ------------------ User documentation explaining the functionality must be provided. Developer Documentation ----------------------- Developer documentation about how to develop a new driver for the ``dr_agent`` must be provided. References ========== Previous work: * `Introducing entities `_ * `Add the dr_agent `_ Links and helpers: * `Neutron Dynamic Routing Use Cases `_ * `Pluggable External Net Blueprint `_ * `Border Gateway Protocol `_ * `Quagga `_ * `BGP/MPLS IP Virtual Private Networks (VPNs) `_ * `etherpad `_ * `VRF `_ * `BGP Comparison `_ * `Ryu's BGP speaker `_ * `policy support for dynamic routing `_ * `Address Scopes `_