Introduce distributed locks to ipam module

https://blueprints.launchpad.net/neutron/+spec/introduce-distributed-locks-to-ipam RFE:https://bugs.launchpad.net/neutron/+bug/1836834

Introduce the OpenStack tooz distributed lock to the ipam module. In the scenario of large-scale port creation, avoid ip allocation exceeding the maximum retry limit failure and improve ip allocation efficiency.

Problem Description

  1. Current port creation has a probability of failure.

    When the virtual machines are created in batches, nova will call the neutron API to create the ports concurrently. Port creation will fail if there is an ip allocation conflict, see [1].

    And bulk create port has the similiar problem, when multiple “create_port_bulk” APIs are called simultaneously on the same subnet, although this scenario is used less frequently.

  2. When creating ports concurrently, the utilization ratio of neutron server CPU increases and the allocation efficiency decreases.

    When an ip allocation conflict fails to submit a database, a DB ERROR exception is thrown. Create_port will catch the above exception and rest after retry_interval=0.1 and re-call create_port until it exceeds max_retries=10. When it exceeds max_retries=10 times, “Create_port” will fail. When concurrency is large and conflict intensifies, repeated call to create_port increases CPU’s burden and reduces allocation efficiency. Adjusting retry_interval and max_retries can only reduce the probability of problems, but can not solve them thoroughly.

Proposed Change

This solution implements a new ipam driver by introducing a distributed lock to completely solve the problem of ip address allocation conflict leading to failure.

For distributed locks we will use OpenStack tooz [2] , which supports many backend drivers, such as Zookeeper, Memcached, Redis, Mysql, etc., and it is an OpenStack native project. We will support the configuration of tooz backend drivers in neutron.conf, such as adding [tooz] configuration items.

The new IPAM allocate ip seqdiag.

../../_images/ipam-allocate.png

Alternatives

  • We can modify the current ipam driver to introduce distributed locks to solve the above mentioned problems when creating a new ipam driver is not feasible.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

  • Operator can configure the backend driver for tooz using the [tooz] configuration block in neutron.conf.

    [tooz]
# Tooz backend connection string.
backend_url = file://$state_path

# Number of seconds between heartbeats for distributed coordination.
heartbeat = 1.0

# Number of seconds to wait after failed reconnection to Tooz backend.
initial_reconnect_backoff = 0.1

# Maximum number of seconds between sequential reconnection retries to Tooz backend.
max_reconnect_backoff = 60.0

  • Operator can switch to our new ipam driver by setting “ipam_driver” in neutron.conf.

    [default]
# Neutron IPAM (IP address management) driver to use. By default, the reference
# implementation of the Neutron IPAM driver is used. (string value)
ipam_driver = ipam_with_dlm

Performance Impact

When using rally to test concurrently to create vms or ports or the similar scenes.

The good aspects:

  • Solve the failure of creating vms or ports due to ip allocation conflicts, and improve the success rate.

  • Reduced average time to create vms or ports.

  • Create vms or ports with a smoother distribution of time.

Minor impact:

  • The minimum time to create vms or ports has increased slightly, and creating a port time in a non-concurrent scenario will also increase slightly.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

qinhaizhong

Other contributors:

zhouhenglc

Work Items

  • Create a new ipam driver.

  • Support for parsing [tooz] backend drivers, encapsulating distributed lock modules, and implementing distributed lock initialization, locking, unlocking, etc.

  • Make “create_port” to support the new ipam driver.

  • Make “bulk_create_port” to support the new ipam driver.

  • Documentation work.

Dependencies

None

Testing

Unit tests, functional tests.

Documentation Impact

None

References