Replace home grown WSGI layer with Pecan¶
https://blueprints.launchpad.net/neutron/+spec/wsgi-pecan-switch
This document describes a plan to replace the current home-grown WSGI framework, including REST controllers, with a solution entirely based on the Pecan framework [1]
The specification discussed in this document can make use of the V3 plugin specification, though it is no longer assumed to be dependent on that specification. See [2] for more details.
Problem Description¶
This specification addresses a number of issues arising from the fact that Neutron so far has been relying on and evolving its own framework for managing web service lifecycle and dispatch API operations to plugins.
Namely:
API resource definition is performed using dictionaries, which contain information about object attribute types, default values and attribute validation. This has a number of limits, especially when it comes to performing validation and serialization of API resources, and it also encourages a behavior where everything is passed around as dictionaries.
The current API extension management framework implies that extensions can pretty much do everything they want with the API - even redefining parts of it.
There is home grown code based on fork() for managing multiple API workers. While this is generally not a problem, it still is a significant amount of code that needs to be maintained. Many REST frameworks like Pecan provide built-in support for spawning multiple API workers.
The REST controllers have become heavyweight components since they also need to take care of tasks such as enforcing quotas and authorizing API requests.
The actual response returned by the REST layer is currently built within the plugin, because there is no object-oriented interface with the plugin. Indeed, the REST controller passes the resource to the plugin as a dict, and expects a resource as a dict from the plugin. It assumes that the plugin builds a dictionary which respects the resource model (ensuring however that only valid attributes are returned to the API consumer).
Most importantly, the current WSGI/REST framework is a relatively large size component in Neutron’s codebase. Switching to a well-established framework will make the whole codebase a lot more maintainable.
Proposed Change¶
In a nutshell: replace the existing framework with Pecan, remove the current code, and ensure that operators are unaffected by the change.
This means that we expect the following for the Kilo release:
All API requests will be served by Pecan REST controllers This will have impact on in-tree attribute extensions. Such extensions indeed will be refactored as they currently define the resources they handle using the same dict-based style as attributes.py [3] There will therefore be a new process for adding extensions to the Neutron API. This process will be documented as a part of this blueprint.
Service startup will not happen anymore through Python PasteDeploy, and will be managed by Pecan. Similarly multiple API workers will be handled through Pecan as well.
The Pecan REST controller will simply take care of serializing responses and deserializing requests into appropriate transfer objects describing API resources. Until the V3 plugin layer referenced here [2] is merged, REST layer will continue to be responsible for authorization and quota enforcement. Request validation will occur in the REST API layer. The goal is to specify constraints using JSON schema. At the time of writing this spec it has not yet been analyzed whether it is possible to express all the validation constraints currently applied to the Neutron API using JSON schema. The final implementation, which is not necessarily the first iteration, might either:
Use JSON schema only
Embed validation logic in API objects (see [2] more information)
Use a mix of JSON schema and custom validation logic, and possibly encapsulate everything within API objects.
On the other hand, authentication for a request must happen before the call is dispatched to the plugin layer. Pecan hooks [4] will be used to perform authentication at the appropriate time.
The Pecan framework however only takes care of managing the REST API server. For this reason as a part of this blueprint the REST and RPC over AMQP servers will be split. Potential impacts of this change on deployers are discussed in the relevant section. This split is not expected to have any other relevant impact on operators, developers, or users. The API server and RPC server will communicate via RPC where necessary. Where async notification tasks need to fire, any notification messages will be handled by post api call hooks. The notification system will be moving from being a side-effect of the API layer to part of the plugin layer. The API can fire off RPC notifications or possibly spawn tasks that execute on RPC/task flow workers.
Data Model Impact¶
No data model change expected.
REST API Impact¶
Even if this patch has a deep impact on the Neutron management layer, the REST API itself will not change at all, and will preserve its capabilities in terms of resources, available operations, filtering, pagination and sorting.
Security Impact¶
Radical changes in the framework handling REST API requests always have a potential security impact.
In this case, since we are moving away from a home grown framework to one which is already widely adopted across OpenStack projects, the overall security level should increase.
Notifications Impact¶
The REST API layer is currently responsible for sending notifications such as those needed by the Telemetry service. With these change the notifications will not be handled in the REST API layer anymore, but moved within the plugin interface as specified in [2]
Other End User Impact¶
End users will not even notice the difference between a server running the home grown framework and one which switched to Pecan.
Performance Impact¶
No significant impact expected. We have however no measurement available to justify this claim.
For this reason performance measurements should be done as part of this blueprint implementation to ensure that switching to Pecan does not negatively impact application performance.
For the purpose of this work, Rally will be used to provide before/after benchmarks. If other tools such as OsProfiler are deemed useful, they will be used as well in the evaluation.
IPv6 Impact¶
IPv6-related APIs and IPAM capabilities will be unchanged.
Other Deployer Impact¶
We expect the deployer impact to be minimal. The main difference introduced by this change, from a deployer perspective is the fact that the HTTP server will be split from the AMQP server.
For green-field deployments this will not be a problem at all. It will also provide deployers with the desirable option of deploying the HTTP and AMQP servers on different nodes.
For existing deployments, updates should be smooth and transparent. The only difference would be that after an upgrade there would not be a single neutron server service, but two - one for the REST API, and one for RPC over AMQP.
Developer Impact¶
New extensions will need to be developed in a different way. This will be thoroughly documented in developer documentation.
Community Impact¶
Moving away from the home-grown framework will allow the community to focus exclusively on Neutron’s business logic. Moreover, members of the Neutron community will also be encouraged to contribute back to Pecan.
Alternatives¶
Other solutions such as Falcon [5] and WSME + Pecan [6] have been considered. However the adoption of Pecan appears the one that better suits Neutron.
A mailing list discussion [7] on REST API frameworks has been used to provide some guidance. For WSME, even if it is an interesting solution to increase code maintanability, and ease the development process, we struggled during some early experiments to make it work with the current extension model. Even if it might be argued that the problem in this case is the extension model, we are unable to recommend it as a part of this blueprint.
Implementation¶
Assignee(s)¶
- Primary assignee:
Kevin Benton (kevinbenton) Brandon Logan (blogan)
- Other contributors:
Sean Collins (sccal68) [developer docs] Salvatore Orlando (salv-orlando) [reserve dev] Mark McClain (markmcclain)
Work Items¶
Define framework for Pecan controllers for core and extended resources.
Re-implement controllers for base and extended resources, paying particular attention to dealing properly with ‘attribute’ extensions. The deliverable of this work item will be a new “base controller” which will leverage the v3 plugin interface proposed in [2].
Plug authorization and quota enforcement in the “plugin management” layer.
Split out RPC over AMQP server
Redefine unit tests to work with new framework
Validate new solution with integration testing, perform performance and scalability analysis.
Dependencies¶
While not a direct dependency, the V3 plugin interface [2] is listed in the case it is proposed again for Liberty.
Testing¶
Once the changes are in place and integrated with the new plugin interface discussed in [2], gate tests should run as usual. We do not expect this change to have any impact that might trigger race conditions leading to intermittent gate failures.
On the other hand, this change will have a significant impact on unit testing. Most unit tests exercise the REST API server and with this change these unit tests will be inevitably broken. Under this proposal we therefore expect significant changes in the “base classes” for unit test, such as [8].
Besides, new modules introduced as a part of this blueprint should be thoroughly unit tested, with a target level of coverage between 90% and 100%. Test coverage should be verified with tox -ecover.
Tempest Tests¶
No new tests are anticipated.
Functional Tests¶
Even if API functional testing will eventually be a relevant part of Neutron’s functional testing suite, this is outside the scope of this spec.
API Tests¶
Please see previous section.
Documentation Impact¶
As the specification discussed in this document changes the way in which the Neutron server is deployed because of the split between the HTTP and RPC over AMQP server, this will need to be appropriately documented in the admin guide.
User Documentation¶
No change.
Developer Documentation¶
The new process for developing Neutron extensions should be thoroughly documented.
Also the developer documentation for the api layer [9] needs to be updated according to the changes being made as part of this blueprint.
