As it was pointed out several times on ML   different services (most notably Neutron and Nova) suffer from performance penalty of having to run new instance of rootwrap executable for each call that needs root privileges.  ended basically with “Who’s up to the task?” question.
The structure of this overhead has been analyzed in . It’s clear that the main issue here is rootwrap startup time that consists of Python interpreter startup and rootwrap config parsing.
I propose creating a new mode of operation for rootwrap - daemon mode. In this mode rootwrap would start, read config file and wait for commands to be run with root privileges. Each service’s process will have its own rootwrap daemon process.
Daemon will be started using a separate binary (like neutron-rootwrap-daemon for Neutron) pointing to oslo.rootwrap.cmd:daemon endpoint (instead of :main). The binary will receive the same options (config file) as the normal one except the command to be run in priviledged mode. For example:
The startup process is the same as in normal mode up to the point when the command is about to be run. In daemon mode a separate method is called instead that starts RPC  server and falls into infinite loop serving requests.
Upon startup daemon will push to its stdout:
These credentials can be used to connect a multiprocessing.BaseManager-based client. Because pickle and xmlrpclib are unsafe, it uses its own JSON serialization (see Under the hood section). The only exposed object is rootwrap with one method: run_one_command(userargs, env=None, stdin=None). Arguments are:
The method returns 3-tuple containing:
Here is a sketch of basic usage for rootwrap daemon:
>>> from subprocess import * >>> from multiprocessing.managers import BaseManager >>> process = Popen(["rootwrap-daemon", "rootwrap.conf"], stdout=PIPE) >>> address = process.stdout.readline()[:-1].decode('utf-8') >>> authkey = process.stdout.read(32) >>> class MyManager(BaseManager): pass ... >>> MyManager.register("rootwrap") >>> from oslo.rootwrap import client # to set up 'jsonrpc' serializer only >>> manager = MyManager(address, authkey, serializer='jsonrpc') >>> manager.connect() >>> proxy = manager.rootwrap() >>> proxy.run_one_command(["cat"], stdin="Hello, world!") [0, u'Hello, world!', u''] >>> process.kill()
Note that this requires rootwrap-daemon pointing to oslo.rootwrap.cmd:daemon to be available in PATH.
The run_one_command call returns a list here because of JSON serialization but it doesn’t change Python API usage.
To simplify daemon usage a oslo.rootwrap.client module is provided containing one class Client that wraps all necessary steps to work with rootwrap daemon.
Its constructor expects one argument - a list that can be passed to Popen to create rootwrap daemon process. For Neutron it’ll be ["sudo", "neutron-rootwrap-daemon", "/etc/neutron/rootwrap.conf"] for example.
The class provides one method execute with the same profile as run_one_command method shown above.
The class lazily creates an instance of the daemon, connects to it and passes arguments. Note that some reconnection and respawning mechanism will be in place so that if daemon process dies or hangs, Client will detect this on the next call and will simply restart it.
This laziness will allow user to kill all rootwrap daemon processes to reload config file, for example.
The biggest expected security risk in this proposal is the way that client talks to daemon so I’ll discuss the underlying protocol in details.
The credentials required to connect to daemon are passed to stdout stream that is expected to be bound through a pipe directly to calling process. They are exposed only to the kernel and calling process here.
multiprocessing involves digest authentication for every new connection made to the server. The key is never passed over the socket, so we could even use TCP socket here (no way). The key is generated using os.urandom(32) call.
Managers use threadlocal connections so there’s no need to create a pool of them. Although it might sound wasteful to create new connection for every thread, creating a connection over UNIX socket is almost nothing compared to spawning a new process time-wise.
By default multiprocessing uses pickle to serialize RPC  requests and responses but it’s very unsafe as it allows to call any method on the receiving side (see warning in ). The only other option is to use xmlrpclib as a serializer but it’s unsafe as it’s prone to resource exhaustion attacks . That’s why JSON serialization support is implemented. It’s very simple (~50 SLOC) and is safe because JSON serialization is widely regarded as being safe.
This serialization is plugged in using undocumented features of multiprocessing.managers module:
The author of multiprocessing module assured that this mechanism is not going to go away any time soon .
Although it’s risky to rely on undocumented feature, it’s mitigated by the assurance of the stdlib module author.
There are a number of alternative approaches to optimize number of rootwrap calls to mitigate the overhead (see ). There were a number of suggestions in original thread in mailing list  and in corresponding etherpad :
Scrap rootwrap, switch to sudo.
We’ll lose current fine-grained control over what can be run as root.
Use other interpretator for rootwrap.
Doesn’t fix the interpretator startup cost.
Rewrite rootwrap in other language.
This includes suggestions to rewrite entirely or partially in C or some Python dialect that would be translateable to C.
As OpenStack community is focused on Python development, bringing some other language to the field would require more developers that would be familiar with it.
Filter commands on the calling process side and use sudo.
This would provide the same security as the first option.
Consolidate calls that use rootwrap into scripts
We could create scripts that would require only one rootwrap call to do all necessary work for one request, for example. But these scripts will either become very complex (say rewriting parts of Neutron agents in shell) or there will be too many of them. Either way defeats the purpose of sudo and rootwrap - to minimize amount and complexity of code running with root privileges.
Per-host daemon process
This would require some D-Bus or MQ set up and secured. It doesn’t look feasible to set up such secure daemons. Supporting them across projects seems even less feasible since every project uses its own rootwrap configuration and such configurations might be host-dependent.
Operations in standalone mode are not affected in any way, so all existing usages will work as before.
This change require another binary to be added to sudoers for the projects that would use daemon mode.
Daemon itself will listen UNIX domain socket but every connection is passed through digest authentication.
JSON is used as a transport to avoid known vulnerabilities in other types of serialization mechanisms.
There is a benchmark results in commit message in . They show that although initial daemon startup time is slightly bigger than with usual rootwrap call, on average daemon shows over 10x better performance.
None in oslo.rootwrap itself. Projects using it might require a separate option for new behavior.
This blueprint suggests rather small addition to rootwrap.
Further integration into different projects should be covered separately.
Both mechanisms involved in daemon work and its API should be covered in docs.
|||(1, 2, 3) Original mailing list thread: http://lists.openstack.org/pipermail/openstack-dev/2014-March/029017.html|
|||(1, 2) Earlier mailing list thread http://lists.openstack.org/pipermail/openstack-dev/2013-July/012539.html|
|||(1, 2) Nowhere in this document “RPC” refer to that MQ-based RPC messaging that oslo.messaging does|
|||(1, 2) https://etherpad.openstack.org/p/neutron-agent-exec-performance|
|||(1, 2) https://review.openstack.org/81798|
This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode