Introduce distributed locks to ipam module¶
Introduce the OpenStack tooz distributed lock to the ipam module. In the scenario of large-scale port creation, avoid ip allocation exceeding the maximum retry limit failure and improve ip allocation efficiency.
Current port creation has a probability of failure.
When the virtual machines are created in batches, nova will call the neutron API to create the ports concurrently. Port creation will fail if there is an ip allocation conflict, see 1.
And bulk create port has the similiar problem, when multiple “create_port_bulk” APIs are called simultaneously on the same subnet, although this scenario is used less frequently.
When creating ports concurrently, the utilization ratio of neutron server CPU increases and the allocation efficiency decreases.
When an ip allocation conflict fails to submit a database, a
DB ERRORexception is thrown.
Create_portwill catch the above exception and rest after
create_portuntil it exceeds
max_retries=10. When it exceeds
max_retries=10times, “Create_port” will fail. When concurrency is large and conflict intensifies, repeated call to create_port increases CPU’s burden and reduces allocation efficiency. Adjusting
max_retriescan only reduce the probability of problems, but can not solve them thoroughly.
This solution implements a new ipam driver by introducing a distributed lock to completely solve the problem of ip address allocation conflict leading to failure.
For distributed locks we will use OpenStack tooz 2 , which supports many backend drivers, such as Zookeeper, Memcached, Redis, Mysql, etc., and it is an OpenStack native project. We will support the configuration of tooz backend drivers in neutron.conf, such as adding [tooz] configuration items.
- The new IPAM allocate ip seqdiag.
We can modify the current ipam driver to introduce distributed locks to solve the above mentioned problems when creating a new ipam driver is not feasible.
Data model impact¶
REST API impact¶
Other end user impact¶
Operator can configure the backend driver for tooz using the [tooz] configuration block in neutron.conf.
[tooz] # Tooz backend connection string. backend_url = file://$state_path # Number of seconds between heartbeats for distributed coordination. heartbeat = 1.0 # Number of seconds to wait after failed reconnection to Tooz backend. initial_reconnect_backoff = 0.1 # Maximum number of seconds between sequential reconnection retries to Tooz backend. max_reconnect_backoff = 60.0
Operator can switch to our new ipam driver by setting “ipam_driver” in neutron.conf.
[default] # Neutron IPAM (IP address management) driver to use. By default, the reference # implementation of the Neutron IPAM driver is used. (string value) ipam_driver = ipam_with_dlm
When using rally to test concurrently to create vms or ports or the similar scenes.
The good aspects:
Solve the failure of creating vms or ports due to ip allocation conflicts, and improve the success rate.
Reduced average time to create vms or ports.
Create vms or ports with a smoother distribution of time.
The minimum time to create vms or ports has increased slightly, and creating a port time in a non-concurrent scenario will also increase slightly.
Other deployer impact¶
- Primary assignee:
- Other contributors:
Create a new ipam driver.
Support for parsing [tooz] backend drivers, encapsulating distributed lock modules, and implementing distributed lock initialization, locking, unlocking, etc.
Make “create_port” to support the new ipam driver.
Make “bulk_create_port” to support the new ipam driver.
Unit tests, functional tests.