The 2.x series of the C Python interpreter on which OpenStack releases through Kilo are built is reaching the end of its extended support period, defined by the upstream developers. This spec describes motivation for porting fully to Python 3 and some of the work we will need to enable testing applications as they move to Python 3.
There are a lot of small motivations for moving to Python 3, including better unicode support and new features in the language and standard library. The primary motivation, however, is that Python 2 is reaching its end-of-life for support from its developers.
Just as we expect our users to update to new versions of OpenStack in order to continue to receive support, the python-dev team expects users of the language to update to reasonably modern and supported versions of the interpreter in order to receive bug and security fixes. When Python 3 was introduced, the support period for Python 2 was extended beyond the normal length of time to allow projects plenty of time to migrate, and to allow the python-dev team to receive feedback to make changes to the language so that migration is easier. That period is coming to an end, and we need to consider migration seriously.
“Python 3.0 was released in 2008. The final 2.x version 2.7 release came out in mid-2010, with a statement of extended support for this end-of-life release. The 2.x branch will see no new major releases after that. 3.x is under active development and has already seen over five years of stable releases, including version 3.3 in 2012 and 3.4 in 2014. This means that all recent standard library improvements, for example, are only available by default in Python 3.x.” – Python2orPython3
That said, we cannot expect all of OpenStack to be ported at one time. It’s likely that we could not port everything in a single release cycle, given the other work going on. So we need a way to stage the porting work so that projects can port when they are ready, without having to wait for any other projects to finish their ports.
Our services communicate through REST APIs and the message bus. This means they are decoupled enough that we can port them one at a time, if our tools support running some services on Python 2 and some on Python 3. Our unit test tool, tox, supports multiple Python versions already, and in fact most of our library projects are testing under Python 2.6, 2.7, and 3.4 today. Our integration tests, however, do not yet support multiple Python versions, so that’s the next step to take.
Even after all applications are on 3.x, we need to maintain some python 2.7 support for client libraries and the Oslo libraries they use. We should consider the deprecation policy of Python 2 for the client libraries independently of porting the applications to 3.
We have discussed this before, and it continues to be a moving target. Version 3.4 seems to be our best goal for now.
Besides the functional and integration tests for applications, we also have functional tests for libraries. I propose that we configure the test jobs to run those only under Python 3, to avoid duplication and expose porting issues that would have an impact on applications as early as possible.
Commercial support is likely to be available from distros for longer than it is available upstream, but even that will stop at some point.
Some applications may benefit from PyPy’s JIT compiler. It currently supports 2.7.8 and 3.2.5, which means our Python 2 code would probably run but code designed for Python 3.4 will not. I’m not aware of any large deployments using PyPy to run services, so I’m not sure this is really a problem. Given the expected long time frame for porting to Python 3, it is likely that PyPy will be able to catch up to the language level needed to run OpenStack by the time we are fully moved to Python 3.
Moving from 3.4 to 3.5 should require much less work than moving from 2.7 to 3.4. We can therefore start now, and monitor adoption of 3.5 by distributions to decide whether to ultimately use 3.4 or a later version.
We have discussed installing applications into virtualenvs a couple of times. Doing that is orthogonal to these proposed changes, since we would still need to use the correct version of Python within the virtualenv.
We could run parallel test jobs configured to run the functional tests for libraries under both Python 2 and 3. This would largely duplicate effort, though it might uncover some inconsistent handling of bytes vs. strings. We shouldn’t start out trying to do this, but if we do uncover problems we can add more test jobs.
Primary assignee: Doug Hellmann
Update devstack to install pip for both Python 2 and Python 3.
Update devstack to look at the supported Python versions for a project, and choose the correct copy of pip to install it and its dependencies.
This may be as simple as:
python setup.py --classifiers | grep 'Language' | cut -f5 -d: | grep '\.'
When installing libraries from source using the LIBS_FROM_GIT feature of devstack, ensure that the libraries are installed for both Python 2 and Python 3.
Begin porting applications to Python 3.
Some functions in Oslo libraries have been identified as having incompatibilities with Python 3. As these cases are reported, we will need to decide, on a case-by-case basis whether it is feasible to create versions of those functions that work for both Python 2 and 3, or if we will need to create some new APIs for use under Python 3 (see oslo_utils.encodeutils.safe_decode, oslo_utils.strutils.mask_password, and oslo_concurrency.processutils.execute as examples).
This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode