No blueprint, this is intended as a reference document.
Eventlet is used in many of the OpenStack projects as the default concurrency model, and there are some things we’ve learned about it over the years that currently exist only as tribal knowledge. This is an attempt to codify those in a central location for everyone’s benefit.
It is worth noting that while there has been a push from some members of the community to move away from eventlet entirely, there is currently no approved plan to do so. Even if there were, it will likely take a long time to implement, so eventlet will be something we have to care about for at least the short and medium term.
In some ways eventlet behaves much differently from other concurrency models and can even change the behavior of the Python standard library. This means that scenarios exist where a bad interaction between eventlet and some other code, often code that is not eventlet-aware, can cause problems. We need some best practices that will minimize the potential for these issues to occur.
Guidelines for using eventlet:
When using eventlet.monkey_patch, do it first or not at all. In practice, this means monkey patching in a top-level __init__.py which is guaranteed to be run before any other project code. As an example, Nova monkey patches in nova/cmd/__init__.py and nova/tests/unit/__init__.py so that in both the runtime and test scenarios the monkey patching happens before any Nova code executes.
The reasoning behind this is that unpatched stdlib modules may not play nicely with eventlet monkey patched ones. For example, if thread A is started, the application monkey patches, then starts thread B, now you’ve mixed native threads and green threads and the results are undefined but most likely bad.
It is not practical to expect developers to recognize all such possible race conditions during development or review, and in fact it is impossible because the race condition could be introduced by code we consume from another library. Because of this, it is safest to simply eliminate the races by monkey patching before any other code is run.
Monkey patching should also be done in a way that allows services to run without it, such as when an API service runs under Apache. This is the reason for Nova not simply monkey patching in nova/__init__.py.
Another example is Keystone, which recommends running under Apache but also supports eventlet. They have a separate eventlet binary ‘keystone-all’ which handles monkey patching before running any other code. Note that eventlet is deprecated in Keystone as of the Kilo cycle.
Monkey patching with thread=False is likely to cause problems. This is done conditionally in many services due to problems running under a debugger with the threading module monkey patched. Unfortunately, even simple concurrency scenarios can result in deadlocks with this sort of setup. For example, the following code provided by Josh Harlow will cause hangs:
import eventlet eventlet.monkey_patch(os=False, thread=False) import threading import time thingy_lock = threading.Lock() def do_it(): with thingy_lock: time.sleep(1) threads =  for i in range(0, 5): threads.append(eventlet.spawn(do_it)) while threads: t = threads.pop() t.wait()
It is unclear at this time whether there is a way to enable debuggers and also have a sane monkey patched environment. The eventlet backdoor was mentioned as a possible alternative.
Monkey patching can cause problems running flake8 with multiple workers. If it does, the monkey patching can be made conditional based on an environment variable that can be set during flake8 test runs. This should not be a problem as monkey patching is not needed for flake8.
import os if not os.environ.get('DISABLE_EVENTLET_PATCHING'): import eventlet eventlet.monkey_patch()
Even though os is being imported before monkey patching, this should be safe as long as no other code is run before monkey patching occurs.
Primary assignee: bnemec
Additional contributors: harlowja, ihrachyshka
This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode