Contributing and reviewing existing and/or new vendor code in Neutron is painful
for a number of reasons. We are proposing changes to the existing structure
of the project to address these pain points. More precisely we are promoting
changes in the following areas:
Vendor and community concerns are competing for scarce resources
because our process is currently one-size-fits-all. Holding
plugins/drivers close made sense in the early days, to build the community, but
Neutron has matured as a project. We have the opportunity to evolve the
relationship between plugin/driver maintainers and the core to ensure that as
a community we can grow ever larger without incurring the bottlenecks and
issues we are currently facing.
Currently, we have 20+ plugins and 12+ ML2 mechanism drivers. There are
plenty of new plugin/drivers in the pipeline whose specifications and
code contributions need to go through the contribution process where:
- a developer submits a blueprint specification to the neutron-specs repo;
- a developer submits the code patch(es) to the neutron repo;
- at least two members from the core team need to (iteratively) review the spec;
- at least one member of the drivers team needs to approve the spec;
- at least two core reviewers need to (iteratively) review/approve the patch(es);
This model clearly does not scale because each core can only have limited
knowledge of some parts of the codebase and the sheer amount of vendor-specific
contributions that Neutron as a project has seen so far meant that no single
core reviewer can know it all. This model has also the following issues:
- Requires an artificial oversight from the core team, as core reviewers do
not have the ability to fully understand and judge quality of vendor-specific
source, sometimes in the form of interrelated patches made of many
thousands lines of code; granted, a good static code review can help
address issues like styling, use of data structures, algorithms,
modularity, etc, but without the ability to fully understand all the moving
parts it is sometime difficult to assess possible failure modes, potential
bottlenecks, and even why certain fixes after the fact end up being necessary;
- Core review may sometimes require playing with the patch, which is made
difficult, if not impossible, by the lack of access to proprietary and/or
- Prevents core reviewer resources from being solely focused on targeting core
concerns of Neutron as a management service;
- Prevents vendors from having more control over the development and release of
their own code;
- Prevents vendors from having different levels of engagement with the Neutron
community: the existing model imposes a high acceptance bar. Vendor
developers are invited to actively participate in the day-to-day management
of the project, maintaining their code etc, even though they sometimes fail
to do so. If the code contribution is substantial, it is only customary for
these developers to participate in the cross-cutting issues that may arise
(e.g. gate failures, oslo libraries refreshes, change in style guidelines,
etc); in other words, the current process requires that a vendor generate
goodwill through contribution to non-vendor initiatives such that cores are
willing to review or address vendor’s non-core contributions in a timely
fashion. In an ideal world this would foster community development, but it
presupposes that all vendors have similar resources to devote to Neutron and
that the core team can scale non-linearly (neither of which is the case
- Does not promote clear separation of concerns, as the code is in one big
- The maintainance cost of the entire codebase is unevenly distributed across
the Neutron team;
- Refactoring effort is hindered by the difficulty of handling code that the
developer is not familiar with;
- Negatively impacts unit test feedback runtime, as a lot more tests need to run
at once to for a specific change;
All these issues ultimately affects the pace at which the project can evolve.
It is worth noting that this problem, and potential solutions, may affect
different people in the OpenStack community in different ways. For instance:
- OpenStack Infra: what tooling would the team need to provide to help the
project? Would they be impacted on a regular basis because of a change
in the way the project decides to organize itself?
- Vendor Developers: how can a developer get his/her changes merged faster
and with minimal overhead? Writing better code and participating more in
core reviews certainly helps, but this is a bar that not every contributor
can or is willing to set for himself/herself;
- Vendor Marketeers: how can a marketeer claim that his/her vendor plugin is
OpenStack friendly? In this particular regard, the fact that vendor plugins
are shipped as ‘compatible’ with OpenStack is largely independent from the
code’s origin. The perception that things in openstack.org trunk are
‘good’ is in reality a misconception, and as such it is a task for the
marketing team(s) to fix it.
- Distros: how can a distro package Neutron by capturing the code necessary,
their external depedencies, and provide configuration tools that allow for
easy consumption of the solution?
- Neutron developers: how can a developer support (non)vendor code
contributions without burning out?
- Operators: do they even care so long as they can get access to a stable,
upgradable and well documented product?
- OpenStack Technical Committee: would the TC members promote this effort,
so long as OpenStack ideals like Openness, Transparency, Commonality,
Integration and Quality are still enforced? As far as openness is concerned,
any new project arrangement must guarantee that the four opens are still
in place (open source, open design, open development, open community );
TC members also represent the Active Technical Contributor, so it is of
paramount importance for them to alleviate the pain in contributing to
- OpenStack Foundation: so long as the job of promoting the development,
distribution and adoption of the OpenStack cloud operating system is not
hindered by any attempt at making a project functioning better, they should
actually welcome the effort. The Foundation has also the duty, with the
marketplace on openstack.org, to promote drivers compatibiility and soon
there will also be RefStack ;
Having said that, the proposed change will primarily address the development
challenges that the Neutron community faces today, leaving social or marketing
perceptions and concerns thereof, a solution to be sought with guidance of
the OpenStack Foundation.
The proposal being made here is an initial step, and something it has been
deemed attainable in the span of a single release. It is not by all means
the end goal, but a good balance that will let us find the golden mean 
in iterations. It is also worth noting that this effort is about the core
plugins, rather than advanced services like Load Balancer, VPN and Firewall.
Spinning off those services will be tracked with a different blueprint .
High Level Codebase Structure
We propose that:
The monolithic plugins, ML2 MechanismDrivers, and L3 service plugins
become integration-only to code that lives outside the tree (more details
provided in the section below); the same applies for any vendor-specific
agents: the only part that will remain in the tree is the agent ‘main’ (a
small python file that imports agent code from the vendor library and
starts it). L3 being an integral part of Neutron core will go through the
same process as core plugins and ML2 drivers. The ‘outside the tree’ can be
anything which the vendor is comfortable with: it may be a stackforge repo
for instance, a tarball, a pypi package, etc. It is then important that this
vendor library be publicly accessible for a number of reasons:
- Potential licensing conflicts, that might be solved only by getting access
to the vendor library, but more importantly under no circumstances should
a key OpenStack project like Neutron be encouraging closed source
development of a critical component such a plugin or mechanism driver;
- Ease of packaging support from distros; even though in most cases, distros
will have a relationship with vendors they want to package and validate,
there may still be pure open source players, that may be interested in
getting to use the undelying technology, should that be open source itself.
A plugin/drivers maintainer team self-governs in order to promote sharing,
reuse, innovation, and release of the ‘out-of-tree’ backbone. It should not
be required for any member of the core team to be involved with this process,
although core members of the Neutron team can be in whichever capacity
necessary for out-of-tree development.
- The following elements are suggested to remain in the tree for all plugins
and drivers (called vendor integration hereinafter) - at least in the first
iteration of this effort:
- Data models;
- Extension definitions;
- Configuration files;
- Requirements file targeting vendor code;
- Things that do not remain in the tree (called vendor library hereinafter):
- Vendor specific logic;
- Associated unit tests;
The idea here would be to keep in-tree the plugin/driver code that implements
an API, but have it delegate to out-of-tree code for backend-specific
interactions; the vendor integration will then typically involve minor
passthrough/parsing of parameters, minor handling of DB objects as well as
handling of responses, whereas the vendor library will do the heavylifting and
implement the vendor-specific logic. The boundary between the in-tree layer and
the out-of-tree one should be defined by the maintainer while asking these
types of questions:
- If something changes in my backend, do I need to alter the integration
layer drastically? Clearly, the least impact there is, the better the
separation being achieved;
- If I expose vendor details (e.g. protocols, auth, e.g.), can easily swap
and replace (e.g. a hardware with a newer version being supplied) without
affecting the integration too much? Clearly, the more reusable the
integration the better separation.
Furthermore, we can observe that:
- Regarding db models, we would like to keep backward compatibility
(for already deployed systems) at the present time. There are a number of
technical points to be addressed before moving db models out of tree
completely, e.g., FK, joined query, db migration for existing deployments
and so on, as currently alembic requires extra work to support multiple db
migration paths onto a single database . This technical challenge can
be dealt with in due course, once the major parts of this proposal have
- Moving config files elsewhere (or even relying on config autogeneration
capabilities) could be explored as a next step. We have been tracking
config changes with DocImpact flags to ensure that OpenStack Manuals are
kept in sync, and we should not break this model;
- The vendor code must be publicly available via pypi, or public
source repository compatible with pip requirements (if packaging is
desired); this requirements file should not be confused with the Neutron
requirements file that lists all common dependencies); instead it is a
file ‘requirements.txt’ that is located in neutron/plugins/pluginXXX/,
whose content is something along the lines of
‘my_plugin_xxx_library>=X.Y.Z’. Vendors should be responsible
for ensuring that their library did not depend on libraries conflicting
with global requirements, but could include libraries not included
in the global requirements. If the vendor library depended on a python
package not captured in the global requirements, this is no different
by what already happens today and we are not trying to address this
in the scope of this effort.
- Versioning can be used to pin vendor code to specific versions;
- For the reference implementation: they will still follow the same model, but
the implementation will remain in tree, due to gating requirements; the
reason being that there is no way to externalize the reference implementation
until we have backwards compatible interfaces in the mix. Having to
coordinate the landing of a breaking change in core neutron repo and a fix
in the reference implementation repo so as to avoid breaking openstack
integration jobs would be an extremely tricky proposition.
The proposal documented here is intended to encourage plugins/drivers to
separate themselves into a part that integrates with Neutron and a part that
integrates with a backend. Given that most breakage is going to be in the
integration with Neutron (which remains in the tree) - and that breakage will
only be allowed when fixes can reasonably be expected to be landed before the
end of cycle (e.g. before milestone2) there should substantially less risk
for non-gating plugins/drivers. Trunk will always work: for breaking changes
that can be detected, we can ensure that at least the in-tree part of plugin
drivers are fixed before merge. Plugin/driver-specific breakage will have to
be fixed by the maintainer of that plugin/driver, but as a community we
should address the potential for that kind of breakage separately.
- No DevStack changes required: Devstack already has the right hooks [6,7,10];
moreover, the vendor is in full control of the 3rd party CI environment;
they could just install the needed dependencies out of band, while the CI
is setting up the environment and before stacking. Some of them already do
all sorts of stuff.
For instance a vendor integration module can become as simple as one that
contains only the following:
- Registering config options;
- Registering the plugin class;
- Registering the models;
- Registering the extensions.
> class DummyPlugin(db_base_plugin_v2.NeutronDbPluginV2,
> supported_extension_aliases = ["the_extension_you_implement"]
> __native_bulk_support = True
> __native_pagination_support = True
> __native_sorting_support = True
> def __init__(self):
> super(DummyPlugin, self).__init__()
The Nuage plugin [2, 3] follows a slightly similar approach. Since unit test
must be purely testing the units that make up for the vendor library, they no
longer make sense to be part of the Neutron codebase. Having said that, the
vendor integration should be tested via functional testing. The VMware plugin
was also going down that direction , where the plugin definition becomes
The testing process will change as follows:
There will be no unit tests for plugins and drivers in the tree, except the
reference implementation. The expectation is that vendors would run unit test
in their own external library (e.g. in stackforge where Jenkins setup is for
free); For unit tests that validate the vendor library, it is responsibility
of the vendor to choose what CI system they see fit to run them; there is no
need or requirement to use OpenStack CI resources if they do not want to;
ultimately these tests can run as part of the 3rd party CI system  that is
currently required, based on specific filters, if necessary. It is noteworthy
that this effort may unveil areas of the core code that turn out to be barely
tested. We should strive to identify those, and improve their coverage so
that, in turn, the improvements can be beneficial for the various
vendor integrations as well, as added/revised tests will exercise the shared
3rd Party CI will continue to validate vendor integration with Neutron via
Tempest (e.g. functional testing); 3rd Party CI is a communication mechanism.
This objective of this mechanism is threefold:
- it communicates to plugin/driver maintainers when someone has contributed
a change that is potentially breaking. It is then up to the a given
maintainer to determine whether the failure is transient or real, and
resolve the problem if it is real;
- it communicates to a patch author that they may be breaking a plugin/driver.
If they have the time/energy/relationship with the maintainer of the
plugin/driver in question, then they can (at their discretion) work to
resolve the breakage;
- it communicates to the community at large whether a given plugin/driver
is being actively maintained.
Gating on 3rd party CI is a terrible idea, though. It introduces too much
variability into the merge process, and we already have more chaos than we
can handle. The proposal in question attempts to minimize some breakage
by keeping plugin/driver interfaces in-tree. We could add sanity unit tests
of interface integration, but it appears that integration failures should
be more effectively caught at a functional and integration level rather
than unit one. Having said that, nothing outlined here would prevent to
adjust test coverage over time.
Review and Defect Management Strategies
We propose that:
- The same bug management process applies: bugs that affect vendor code can be
filed against the Neutron integration, if the integration code is at fault;
otherwise, the code maintainer may decide to fix a bug without oversight, and
issue a new version pin against the requirements file for the code in question
in case the vendor library is being pinned; it makes sense to require 3rd
party CI for a given plugin/driver to pass when changing their dependency
before merging to any branch (i.e. both master and stable branches);
- The same review process applies: vendor specific code follow the same review
guidelines as any other code in the tree, however, the code being reviewed
strictly applies to integration code or requirements’ version bumps; as for
the vendor library, it becomes an external repo; the vendor can choose anyone
to approve/merge changes in this repo;
- The same release process applies: this does not change;
- The same security vulnerability process applies: this does not change. However,
since the attack surface is a lot larger for the vendor library, it is most
likely that the vulnerability is found there, and the maintainer will need to
issue a fix according to his/her own procedures; if a vulnerability is found in
the vendor integration, the vulnerability fix will need to follow the same
procedures are defined by OpenStack. Vendor may decide to publish the
vulnerability on LP or any other tracking system: completely up to them.
Blueprint Spec Submission Strategy
We propose that:
- Provided vendors adhere to the limited development footprint laid out in the
proposal, they should not be required to follow the spec process for changes
that only affect their vendor integration and library.
- Proposal of new vendor plugins and drivers should no longer follow the
spec submission process in place for any other Neutron contribution, so long
as the vendor plugin/driver adopts the guidelines outlined in this proposal.
New contributions could simply be submitted for code review, with the proviso
that adequate documentation and 3rd CI party is supplied at the time of
the code submission. For tracking purposes, the review itself can be tagged
with a Launchpad bug report, marked as wishlist; design documents can still
be supplied in form of RST documents, within the same vendor library repo,
for documentation purposes;
- Blueprint specifications for new vendor plugin/drivers that target the
current release cyle for Neutron will therefore be evaluated with the
assumption that this proposal goes ahead, and hence marked for abandonment.
Code submission can still go ahead, and evaluated according to the
guidelines laid out in this proposal. Code inclusion will most likely
considered in time for the third milestone of the cycle to allow enough
time for the core team to adequately demonstrate this proposal in practice.
There is a sister proposal for docs referenced in . The intention going
forward is that the documentation team will fully document the reference
plugins/drivers and add short sections for vendor plugins and drivers, in
the same spirit promoted by this proposal (about code decomposition). Existing
plugins and drivers will have to add references to additional docs, just like
new ones will, once their vendor integration merges in the Neutron repo.
Adoption And Deprecation Policy
We propose that:
- All new (core or L3) plugins and drivers follow this model from Kilo;
- It is strongly recommended that existing plugins and drivers adopt
this model in the Kilo cycle, and structure the code according the
guidelines outlined above;
- No in-tree features can be added to existing plugins and drivers: this
means that the feature of existing plugins/drivers code in the main
Neutron tree will be frozen for the Kilo release; some may argue that the
freeze might not be necessary and we could still rely on some review
best effort. However, it is important to recognize that strict discipline
is required to see this proposal through;
- Only critical bug fixes will be permitted to existing plugins and drivers;
ML2 needs to be excluded, at least for now, because we gate on ML2+OVS
which precludes splitting any part of it out in advance of backwards
compatible interfaces being introduced. It is nonetheless proposed that we
refactor the OVS ML2 driver such that the drivers themselves contain the
same minimum footprint proposed for all drivers and that they are cleanly
separated from the backend implementation.
- Plugins and drivers that are still ‘as is’ (i.e. not complying with this
proposal) at the end of Kilo will be evaluated for deprecation and removal;
maintainers will need to make a positive effort to start this process in
the Kilo timeframe; process well underway but to complete in the Lxxx
will still be acceptable; however, it is paramount that maintainers
prioritize their resources so that the rewriting process is not deferred
until Lxxx, to minimize the impact of the code freeze, and to mitigate the
potential danger of deprecation and removal in time for Mxxx.
Having said that, the deprecation target and freeze will need to be assessed
at the various Kilo milestones, to establish how close to completion this
proposal is, and how far each vendor maintainer is in completing the tasks
outilined in the ‘Work Items’ section below. Should this proposal get the go
ahead, then the clock will be set and actions will be tracked for each plugin,
ML2 driver, as well as the exent of progress being made (e.g. setting up of
repos, patches addressing the relevant code, etc).
NOTE: No action or lack of engagement from the interested parties will need to
be tracked to ensure that, deprecation is considered a last resort measure.
The approach being proposed here tackles all the issues (outlined above) that
Neutron developers have grown so tired of; it does so by delegating some
control to the vendor, whilst retaining some of degree of visibility into what
is considered part of the Neutron project.
Add your question here and we will try to answer to best way we can:
- Q: If ML2 plugin refactoring is pursued by the core team. Code-changes for
the same can affect the interfaces and behaviour of existing large set of
vendor mech-drivers in the neutron tree. So, does the vendor mech-drivers
continue to remain in core tree.
- A: The proposal is not to move drivers out of the tree. Rather, the
intention is to encourage a separation of concerns such that plugins and
drivers are as lightweight as possible, and serve primarily to integrate
with libraries external to the tree that provide backend-specific
implementation. This would have the advantage of allowing vendors greater
control over their backend-specific code and and offer the possibility of
allowing changes to plugins/drivers with a lighter-weight process (no spec
process and faster merging).
- Q: Wouldn’t it be easier if the three code artifacts (Model, Extensions,
and config files) also lived in the vendor repositories? By leaving the
above three in-tree, the proposal still puts significant burden on the
Neutron cores to review vendor code artifacts.
- A: Short answer is No, as (hopefully) the artifacts are fairly small and
easy to understand. The long answer is: the effort required for full
separation is considerable. The suggested approach would allow plugin/driver
maintainers to gain control over their efforts faster and with less risk. It
will most likely take the entire cycle to get everyone on board and adjusted
to the new model, which does not require any additional work, like new hooks,
different DB timelines, etc. but just the ‘mechanical’ translation.
Once that is cleared out, we can then focus on refining the model and pursue
further delegation to vendor code.
We also need to consider the migration path: we have a couple of things
to consider on splitting db model: for data models, we need to consider the
db migration path and FK relationship to Neutron core models (networks,
ports, etc). Coming up with a concrete plan on how this can be handled ahead
of this proposal is considered a risk worth deferring. API extensions and
configuration files may be able to be moved out, but it has been considered
useful documentation material, at least for the first iteration.
- Q: The DB models for each vendors will stay in the main repo. This probably
avoid the alembic challenge for migrations, but it also confuses me a bit
when it comes to synchronizing models in master and versioning how the plugin
library. In other words, if you release a version of a library which requires
changes in the data model, then you will have to wait for data model changes
to merged before pinning Neutron to that version of the library. What should
the process be in this case?
- A: Ideally, the model changes are backward compatible, but if they were not,
then, as outlined, the process is still pretty straighforward.
- Q: What is the distro packaging strategy? For instance, do we need to define
versioning policy for a vendor plugin module? For example, vendor plugin
module for Kilo should have a version of 2015.1.N?
- A: We do not require openstack specific versioning of other python
dependencies, and we should not require it of the plugin/driver dependency
libraries either. We could rely on the per-plugin/driver requirements files
in a given Neutron release to authoritatively determine the version of the
library required for a given plugin/driver for that release, and the version
could be updated on stable branches as required.
- Q: It seems that the plugin library maintainer is expected to provide
packaging for the appropriate distros. Is this true?
- A: Just as distros are responsible for determining how to package
plugin/driver dependencies today, they would continue to do so under this
proposal. Their job would be made easier by the addition of a plugin/driver
requirements file that could be used to determine plugin/driver dependencies
that currently require manual discovery. The expectation here is that the
plugin business logic will have its own lifecycle in a library which is not
part of OpenStack. The relationship between Neutron and the vendor library
would be for instance the same that there is between Neutron and sqlalchemy.
Therefore versioning for the vendor library can be whatever one sees fit.
The plugin itself however, stays in the main neutron repository, but it is
minimised in a way that it could be just a stub. We will still have a Neutron
“vendor X” plugin for openstack 2015.1
- Q: How do we deal with Oslo integrations?
- A: When oslo integrations occur, 3rd party CI (and a plugin/driver maintainers
internal testing) should detect any breakage in the external dependencies of
the plugins/drivers. We may want to announce when these kinds of changes are
expected such that plugin/driver maintainers can respond in a timely fashion
and minimize the duration of job failure. Only if plugin/driver maintainers
were completely unresponsive to breaking changes in the Neutron tree would
the possibility exist of those changes making it all the way to production.
Were a maintainer to be that unresponsive, the inclusion of their plugin
in the tree would likely be in question. The aim here is to relieve the core
team of some of the maintainance burder in addressing these types of issues.
- Q: How can we ensure that API’s will not be changed and versioned? Say a
internal method of a mixin is updated - then a vendor overwriting or using
this may miss this. In other words, we do not have stable API’s, therefore
in the current model breaking changes can be prevented by doing a one
sweep-fix-all type of patch. With the model being proposed here, how are we
going to deal with this type of issue?
- A: The proposal is to keep as strict a separation as possible between Neutron
and vendor concerns, and continuing to develop our test suite to catch
breakage (API or otherwise) when it occurs. Provided plugin/driver maintainer
is diligent in following up on breakage, the only thing that will change
under this proposal is how fast breakage will be fixed (same patch vs one
patch to Neutron + one patch per external plugin dependency). The fact that
the cost of breakage will be increased could actually be a positive thing if
it can encourage the community to take more care in providing cleaner
separation between concerns (via APIs or otherwise). Today both the
integrated gate and non-gated plugins happen to break from time to time; so
long as there is a prompt reaction to the issue, we do not see this model
exacerbating this issue further. As a matter of fact, CI’s are getting
better and the team is getting better at catching these problems and fixing
them swiftly; all we need is to be engaged!
- Q: Can you please elaborate a little more on the governane of new repos?
There is at least 1 maintainer needed. How many minimum number of members?
If there is 1 member, he/she will be committing, reviewing and merging
his/her own code?
- A: This proposal does not mandate any kind of governance (or even distribution
via a repo) of a given plugin/driver dependency library. We would continue to
require an in-tree maintainer/point of contact for each plugin/driver, as we
do today, and that maintainer would be responsible for coordinating with
- Q: Previously, if there was any change affecting plugins/drivers, it was done
in all the code paths to-be-affected to make sure nothing breaks. However,
with split repos, how to make third party CIs pass on changes which break
vendor code and require a change across two or more repos? Looks like we
will have to override third party CI vote either in Neutron or vendor
plugin/driver repo? If the override is absolutely needed, where should it
- A: 3rd party CI votes but does not gate, and this proposal in no way changes
that. When breaking changes land in the Neutron tree, it will be up to the
plugin/driver maintainers to resolve the breakage on their own timeline. In
the short-term we are introducing coordination overhead by externalizing
backend-specific code, but in the long-term this should encourage a decrease
in coupling that minimizes such costs.
- Q: How do we define some vendor plugin is a part of OpenStack release? Do we
define it? Defined by third party testing integration? It seems related to
distro packaging policy to some extent.
- A: We can define whether a plugin is part of a given release is the answer to
the question: ‘does the plugin/driver exist in the Neutron tree?’. The
question that we do need to answer is ‘under what conditions is a
plugin/driver allowed to be in the tree’? We are going to have to rely on a
combination of relationship management (PTL <-> plugin/driver maintainer) and
3rd party CI to answer that question, and this proposal does not need to
answer this question. Some plugins already rely on a variation of the
proposed model, and there does not seem to be any issue with distribution.
- Q: Should Third party CI result be posted to Neutron core change? We need to
define third party testing requirements?
- A: They can choose to do so, if they believe the change may break their
support, but this already happens today; 3rd party CI requirements need to
be clarified regardless of this proposal.
- Q: Does this proposal make any effort into an ML2 Agent no longer feasible?
- A: No, effort into developing an ML2 agent would not be the best use of
community resources. For historic reasons Neutron has grown to be not only a
networking orchestration project but also a reference implemention that is
resembling what some might call an SDN controller. We need to move away from
this model, and for these reasons, pursuing an ML2 agent is no longer viable.
- Q: What about Linux Bridge?
- A: Linux Bridge is not currently the option we choose to focus both the
development and testing; any effort in maintaining it at par with OVS would
require resources that need to be identified and quantified. In lack of those
resources, Linux Bridge has already fallen behind and will not be considered
in the refactoring effort, unless someone else is willing to sponsor it.
- Q: How do we deal with imports of Neutron specifics (like utility functions,
constants, base artifacts, etc) in the vendor library?
- A: External libraries that implement backend-specific interaction for
plugins/drivers are expected to explicitly depend on Neutron to allow them
to reference things like db models, utility functions and constants. Since
the Neutron’s dependency on the external libraries is not explicit
(plugin/driver requirements file will not be considered by pip), a circular
dependency will not result. The model would follow loosely what any OpenStack
project already does when referencing Oslo. When it comes to pinning to a
specific ‘snapshot’ of the core codebase, git hashes could be used. The model
would need to ultimately evolve so that a better demarcartion will occur
between core and vendor specifics, which makes the dependency less of a
- Q: What if a vendor library uses an API defined in the core?
- A: Reuse of some plugin methods by an external library is possible through
dependency injection, which are fancy words for ‘pass a reference to the
plugin to the code that needs to call its methods’.
- Q: With this approach, would packagers ignore some of the plugins in tree by
not making a package for their library?
- A: Before the split, all plugins were ‘magically’ packaged by distributions,
at least their source code. The packager then explicitely needed to add
‘control’ rules to integrate the various parts together with the OS, and/or
config management tools to reflect the configuration choices in the respective
configuration files. This does not necessarily endorse validity of the
vendor solution on the platform, but packagers were left with no choice but
to follow this route in order to avoid dangling code being packaged and
distributed. With this approach, consuming the actual source code becomes
optional, and it is up to the vendor to decide to continue to preserve the
existing packaging model by pulling from the various sources. In other words,
this opens up the question whether all distributions should consider if it is
worth the effort of shipping plugins that are not explicitely covered by
relationship agreements with vendors. Every distro has different internal
processes for tracking packages and releases, and it sensible that distros
take the appropriate actions to ensure the overall quality of the Neutron
solution they sell and support. We should emphasize that according to this
proposal, long term, some plugins may loose packaging and distribution in the
commercial editions of these distros, but at the same time, open source
initiatives can still go ahead unaffected.
- Q: The fact that many vendor-specific plugins/drivers are currently in-tree,
creates the impression that a new vendor has ‘failed’ in some sense if they
do not get their new plugin/driver in tree as well - where ‘failed’ has
various possible connotations including ‘not being guaranteed to work with
OpenStack, and to continue to work as OpenStack evolves’, and ‘not having
demonstrated sufficient engagement with the OpenStack community’. Does this
proposal help in addressing this aspect of code contribution?
- A: Absolutely! First of all, let us be clear of one thing: in-tree does not
equate success, and out-of-tree does not equate failure. This is a perception
that somewhat built over time into people’s minds, because of past events
that led to the removal of code of questionable quality or because of lack
of testing (most notably Hyper-V in Nova). Today, the reality is completely
different, OpenStack has evolved and 3rd party CI plays a fundamental role
in establishing the sanity of all the components in motion in an OpenStack
based cloud. Furthermore, with this proposal, the failure for not getting
new vendor support in the tree is mitigated by the fact that it is a lot
easier for the core team to go through the checklist of steps to be taken
to consider a vendor integrated with Neutron, without being encumbered by
the need of going through multiple review cycles to get the plugin/driver
in the tree.
Other End User Impact
From an end-user standpoint (the tenant or admin that interact with the Neutron
API); this approach has no impact on them.
Other Deployer Impact
Minimal. Depending on how Neutron is consumed, deployers need to pull an
extra dependency, which is the vendor library of the plugin of their choice.
This dependency can be pulled in in various ways, with or without the actual
support from the OS/Distro the deployer is using to consume Neutron.
It is worth mentioning that this approach empowers the distros (namely the
packagers) to choose where to focus resources more effectively. This means
that they can now choose to target packaging only for specific vendor
solutions more explicitely. Although this may sound like a setback for some,
the benefit of automatic packaging was only giving false reassurance that the
overall Neutron solution was going to be free of supportability issues, or
that it was implicitely validated by either the vendor or the distro. Any
customer should do the appropriate due diligence when adopting the combination
of Neutron+Backend, should they choose to access the overall Neutron solution
via two distinct channels. This proposal does hinder this process.
As outlined in this proposal.
Alternatives could be explored with or without tooling assistance:
Increase the number of Neutron cores (1)
We could allow every vendor to have core(s) reviewers so that they can
approve only their own code (i.e. subtree delegation). This poses an
interesting set of challenges:
- What do we do with new contributors?
- What does core developer actually mean?
- How can we ensure a coherent standard of code reviews?
Rubberstamping from Neutron cores (2)
Neutron cores can become paper pushers and blindly approve changes
that pertains only to a specific vendor. This poses an interesting
set of challenges:
- How do we ensure that cross-cutting concerns are not overlooked?
- How do we ensure that vendor maintainers are still held accountable?
Adopt a different decomposition of the source tree
The existing source tree could be decomposed in separate repositories:
- One repo hosts core elements (DB, API, RPC, etc) and fully open source
implementation of L2/L3 API that is currently gated;
- One repo for each vendor, that has contributed one or more (monolithic)
- One repo for all ML2 mechanism drivers; A vendor that contributed both
a monolithic plugin and an ML2 driver should decide where the ML2 driver
needs to belong;
Ideally this is decomposition that makes the most sense, if it was not for
the political and social rules that we are constrained by. In other words,
from a pure software development standpoint, this decomposition makes the
most sense, and any objection to this approach could be overcome with
technical solutions. That said, it may be too hard to pursue this approach
in the span of a single release. It might make sense to reassess this
approach in a release from now, once the codebase has gone through this
initial transformation and other efforts (e.g. advanced service spin-off,
API refactoring, etc.) have been completed.
Using git submodules to decompose the source tree
This has been strongly discouraged by OpenStack Infra, because of the
complications of managing git submodules: not only does it add potential
complications in CI tooling, but it is nearly indistinguishable from using
separate repositories and as such does not really address any of the
concerns raised with separate repositories.
Using feature branches
This could be achieved by using long lived branches (similar to the Linux
kernel model where subteam maintainers have there own branch). Short lived
branches do not really lend themselves to representing the lifespan of a
plugin/driver. Even though one of the perks of feature branches is that
they can own different ACLs so a different group of people can get +2/+W
on them, the drawback to this approach is that it does not promote clear
separation and responsibilities between the various parts of the system.
In other words, if we had stable interfaces between these parties, using
feature branches would not provide any benefits more than having these
parties reside in different repositories.
While approaches 1 and 2 may help address change velocity, they may
ultimately lead to an increased instability of the codebase, they do
not promote cleaner interfaces, and they weight down the Neutron team
that would need to deal with a much, much bigger codebase. In other
words, even though these approaches may make sense in the short term,
they may seriously affect the project in the long run.
For further details, see section on Documentation Strategies.
After this proposal, the overall Neutron system will not change in terms of how it
functions. From a user perspective, once the system has been deployed, configured
and up and running, no difference can be perceived. However, a sibling proposal
has been documented in . Operators will be expected to follow documentation
and/or pointers to external references provided by the Upstream OpenStack docs
to understand how to install and configure the system to use specific plugins and
No API change has been proposed here, so providing guidelines for contributing
a plugin/driver will suffice. This proposal already outlined quite extensively
some of the work involved, and will be used as base for the in-tree developer