Switching oslo.vmware’s backing SOAP library

We want to switch the SOAP library used in olso.vmware to another library, zeep, because the currently-used library (suds-jurko) is unmaintained, not compatible with Python 3.10 and lacks in performance.

Problem description

With Python 2 being removed from all of OpenStack, we need the library backing oslo.vmware’s SOAP calls to be compatible with Python 3. We see suds-jurko as being unmaintained upstream, so we cannot rely on any changes happening. Since we currently already see deprecation warnings generated by nova’s tests for Python 3.6, we need a fix before the release of Python 3.10 or oslo.vmware will stop working alltogether.

Additionally, the vast number of VMs handled by a single compute-node with nova’s VMware driver makes suds-jurko a performance bottleneck. This can be overcome with switching to a library more optimized for parsing XML.

Proposed change

We propose to change the backing library of oslo.vmware to zeep. The library is

  • Still maintained

  • Written for Python 2 and 3

  • Uses lxml for faster XML parsing

With the interface of zeep being similar but not the same as suds, we have to introduce some compatibility functions. This is also necessary, because some library-specific representation of objects, e.g. attribute access to ManagedObjectReference objects, leaked through to consuming projects. We therefore propose to change the library and consuming projects in multiple phases.

Phase 1

Add compatibility functions to oslo.vmware for value and type access of ManagedObjectReference and add additional functions/helpers for attribute access to move all code using suds-specificas into oslo.vmware.

Phase 2

Introduce patches to cinder and nova for using the functions introduced in phase 1.

Phase 3

Switch the code in oslo.vmware to use zeep instead of suds-jurko while keeping the interface for consuming projects the same. The core of these changes should be confined to oslo_vmware.service.

Alternatives

There’s another fork of suds based on suds-jurko called suds-community, which is still maintained, but while using it may solve the Python compatibility problem it doesn’t improve performance.

While there are other libraries around, see e.g. this list in the Python wiki, a quick glance at most of them shows them as unmaintained, not supporting Python 3 or having an interface that requires too many compatibility shims for keeping changes in consuming projects minimal.

Another alternative would be to get rid of oslo.vmware alltogether, which is mainly hindered by nova and cinder still including drivers using this library.

An alternatives to adding helper functions for hiding the differences between suds and zeep would be changing the depending project’s drivers to use the new interface instead. But this would still leak implementation details of the SOAP library backing oslo.vmare into depending projects, which can also hinder future library switches.

Impact on Existing APIs

Consuming code will be required to use the newly introduced helper-functions for compatibility in attribute access.

Security impact

None

Performance Impact

With the XML parser of the library switching to the C-based lxml, we expect a performance increase. In our (very simple) tests, we achieved up to 50 % reduction in request times and CPU load.

Configuration Impact

None

Developer Impact

Consuming code will be required to use the newly introduced helper-functions for compatibility in attribute access.

Testing Impact

Since there’s a new way to access ManagedObjectReference’s attributes, we need to update all tests - including depending project’s - to use the helper functions we are going to introduce. Additionally, we will have to update fake ManagedObjectReference objects defined in the tests of depending projects to include both old and new attributes for the transition phase. This is necessary, because the project’s tests have to work indepedently of the backing libraries helper-function implementation, so that introduces changes doesn’t break them. An alternative to this would be providing fake objects importabable via oslo.vmware. Even then there can still be cases where oslo.vmware’s backing SOAP library’s object representation might leak into depending object’s tests, because there are other objects, that are not ManagedObjectReferences, but behave exactly the same.

Implementation

Assignee(s)

Primary assignee:

jkulik

Milestones

Target Milestone for completion: unclear

Work Items

  • Implement helper functions in oslo.vmware for compatibility layer

  • Patch nova’s VMware driver to use helper functions

  • Patch cinder’s VMware driver to use helper functions

  • Add zeep to global requirements as documented

  • Implement Service object using zeep client in oslo.vmware

Documentation Impact

None

Dependencies

This adds zeep into OpenStack’s requirements, while removing suds-jurko.

References

Note

This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode