rootwrap daemon mode¶
Neutron is one of projects that heavily depends on executing actions on network nodes that require root priviledges on Linux system. Currently this is achieved with oslo.rootwrap that has to be run with sudo. Both sudo and rootwrap produce significant performance overhead. This blueprint covers mitigating the sudo and rootwrap introduced part of the overhead by using the new mode of rootwrap operation called ‘daemon mode’.
As Miguel Angel Ajo stated in 2:
On a database with 1 public network, 192 private networks, 192 routers, and 192 nano VMs, with OVS plugin:
Network node setup time (rootwrap): 24 minutes
Network node setup time (sudo): 10 minutes
As you can see, rootwrap presents major overhead comparing to plain sudo usage and most of this overhead is due to long Python interpreter startup time required for every request. Details of the overhead are covered in 1.
This blueprint proposes adopting oslo.rootwrap daemon model which allows to run rootwrap as a daemon. The daemon works just as a usual rootwrap but will accept commands to be run over authenticated UNIX domain socket instead of command line, running continuously in background. An overview of rootwrap daemon is at 3.
Two new configuration options should be added:
rootwrap_mode= daemon | process will make agents use daemon or the usual rootwrap process; daemon will be enabled by default. This option could be deprecated in the future if we consider the daemon mode performs as we expect.
rootwrap_configwill provide path to rootwrap config file used to run daemon, by default it will point to the neutron provider rootwrap files.
root_helper is passed around in Neutron (as opposed to use of global
config in other projects) we have to make root_helper as some object
encapsulating all necessary info to run commands with root priviledges.
This is done in the first patch 4.
The patch introduces two root helper classes:
ProcessHelper(base) simply runs command as is, with no wrappers;
WrappedProcessHelperruns commands with a wrapper just as
root_helperoption used to do.
These classes have two interface methods:
create_processstarts a process and returns Popen instance associated with it;
executestarts a process (using
create_processby default), feeds it some input and captures return code along with stdout and stderr output.
execute methods in the
neutron.agent.linux.utils module are still there but they just preprocess
arguments and pass them to appropriate methods of
The second step 5 would be to create those configuration options and
add another root helper class
RootwrapDaemonHelper. This class would employ
one rootwrap client for all instances with same rootwrap_config. Its
execute method would just call
execute method of rootwrap client that
would pass request to the daemon.
Note that we can’t currently start long-running processes using daemon so
create_process will use usual rootwrap to start the process. Also the gain
for long running processes is negligible.
Data Model Impact¶
REST API Impact¶
This change will start oslo rootwrap daemons running as root, and available to neutron via a well authenticated (OTP) unix domain socket.
neutron-rootwrap-daemon should be added to the
with the same settings used for neutron-rootwrap.
Security is handled by oslo-rootwrap which is quite well proven.
All security issues with using client+daemon instead of plain rootwrap are covered in 1.
Other End User Impact¶
This change introduces performance boost for agents that rely on calling a huge
amount of commands with rootwrap. Current state of rootwrap daemon shows over
10x speedup comparing to usual
sudo rootwrap call. Total speedup for agents
will be less impressive but should be very noticeable. The rootwrap daemon
implementation includes a benchmark which generated the speedup results.
Other Deployer Impact¶
This change introduces two new config variables:
rootwrap_configmust point to
rootwrap.confdeployed for Neutron (default
Note that by default
use_rootwrap_daemon will be turned off so to get the
speedup one will have to turn it on. With it turned on
option is ignored and
neutron-rootwrap-daemon is used to run most commands
that require root priviledges.
This change also introduces new binary
neutron-rootwrap-daemon that should
be deployed beside
neutron-rootwrap and added to
The community is broadly interested in faster agent operation.
Use sudo without rootwrap, at the cost of lower security, or by implementing specific sudo plugins to allow equivalent equivalent rootwrap filtering. That would require maintaining C sudo plugins specific for openstack.
Rewrite rootwrap into C, but that requires code auditing, and also deviates from the standard python of the openstack community.
Automatically translate rootwrap into C++, and obtain a compiled version, this option has been explored, and seems to be possible via shedskin with small modifications to rootwrap to avoid dynamic typing. By the way, shedskin is experimental, and some python modules are not implemented. Also the C++ produced code is not easily human readable, and may need a security assesment. This solution doesn’t eliminate the sudo performance impact, which is not negligible.
Other solutions are mostly related to rootwrap optimizations and are covered in 1 (some of them are covered in 6).
This is not the only option of what can be done on the Neutron side besides switching to rootwrap daemon (see 6 for details):
Avoid unnecessary locking in router processing in L3 agent
Avoid unnecessary calls
Consolidate system calls: Carl Baldwin explored this option, and proved to be very difficult and invasive to the code base.
All of these can be done with rootwrap daemon as well since every call to the daemon still impose some overhead.
- Primary assignee:
twilson (Terry Wilson, otherwiseguy @ freenode)
- Original contributor:
yorik-sar (Yuriy Taraday, YorikSar @ freenode)
This change doesn’t change APIs so it doesn’t require additional integration
tests. If tempest is happy with
use_rootwrap_daemon turned on, the feature
Existing functional tests will exercise the rootwrap client side when performing agents functional testing. Rootwrap also has it’s own functional testing for the rootwrap client/daemon pieces 7.
No API tests are required.
As we are including two new configuration settings, those need to be properly documented.
Developer documentation may be updated to describe how to use the new interface to execute system commands, if any change were made to the interface.
oslo.rootwrap blueprint: https://blueprints.launchpad.net/oslo.rootwrap/+spec/rootwrap-daemon-mode
Original mailing list thread: http://lists.openstack.org/pipermail/openstack-dev/2014-March/029017.html
Daemon design overview: https://etherpad.openstack.org/p/rootwrap-agent
Change request “Abstract out root_helper calls to classes”: https://review.openstack.org/82787
Change request “Implement rootwrap daemon support”: https://review.openstack.org/84667
Original problem statement summarized here: https://etherpad.openstack.org/p/neutron-agent-exec-performance
Rootwrap daemon functional testing https://github.com/openstack/oslo.rootwrap/blob/master/tests/test_functional.py