Adopting futurist

https://blueprints.launchpad.net/oslo.utils/+spec/adopt-futurist

The goal of this library would be to provide a well documented futures classes/utilities/additions that start of with the a basic set and can expand into a larger set as time goes on. This library (and the futures library/module in general) allows for providing a level of transparency in how asynchronous work gets executed.

This library currently adds on the following (going above and beyond the built-in futures library in python 3.2 or newer and the futures backport that is commonly used on python 2.6 or 2.7).

  • Statistics gathering (number of futures ran, average failure rate, average run time…)

  • An eventlet executor (aka a GreenThreadPoolExecutor) that uses a eventlet greenpool and greenthreads (making the same futures/executor interface work when running in an eventlet environment). This kind of addition makes it eas(ier) to move from eventlet to another style of executor (since the interface that is provided/used is the same as the other executor types).

  • A synchronous executor (aka a SynchronousExecutor) that does not using any pools, threads, or processes but executes the submitted function in the calling thread (or program) and returns a future with the result or exception set. This allows for blocking usage where for whatever reason the same executor/futures interface is desired but executing in threads, processes (or other) is not desired.

Some of the current/planned usages for it (that are being actively pursued):

  • Internal usage in taskflow (when no executor object is provided by a user). Also used for internal retry activation (that itself uses a synchronous executor).

  • Internal usage in oslo.messaging (replacing/augmenting/unifying the executor interface inside that codebase).

The code for this library was/has been extracted from taskflow and is desired to be used by oslo.messaging (and hopefully others)?

Library Name

futurist

Contents

  • futurist/__init__.py

  • futurist/futures.py

  • futurist/utils.py (internal usage only)

Early Adopters

  • Taskflow

  • Oslo.messaging

  • Others?

Public API

Currently the public API (in pseudo-code) is the following:

Actual code can be found here.

from concurrent import futures as _futures
from concurrent.futures import process as _process
from concurrent.futures import thread as _thread

ThreadPoolExecutor(_thread.ThreadPoolExecutor)
    """
    Executor that uses a thread pool to execute calls asynchronously.
    It gathers statistics about the submissions executed for
    post-analysis...

    See: https://docs.python.org/dev/library/concurrent.futures.html
    """
    - statistics (read-only property)
    - alive (read-only property)
    - submit (same as parent class)
    - shutdown (same as parent class)

ProcessPoolExecutor(_process.ProcessPoolExecutor)
    """
    Executor that uses a process pool to execute calls asynchronously.
    It gathers statistics about the submissions executed for
    post-analysis...

    See: https://docs.python.org/dev/library/concurrent.futures.html
    """
    - statistics (read-only property)
    - alive (read-only property)
    - submit (same as parent class)
    - shutdown (same as parent class)

SynchronousExecutor(_futures.Executor)
    """Executor that uses the caller to execute calls synchronously.
    This provides an interface to a caller that looks like an executor but
    will execute the calls inside the caller thread instead of executing it
    in a external process/thread for when this type of functionality is
    useful to provide... It gathers statistics about the submissions
    executed for post-analysis...
    """
    - statistics (read-only property)
    - alive (read-only property)
    - submit (same as parent class)
    - restart (Restarts this executor (*iff* previously shutoff/shutdown))
    - shutdown (same as parent class)

GreenFuture(Future)
    """Future that ensures internal condition is a green(ed) condition."""

GreenThreadPoolExecutor(_futures.Executor)
    """Executor that uses a green thread pool to execute calls
    asynchronously.

    See: https://docs.python.org/dev/library/concurrent.futures.html
    and http://eventlet.net/doc/modules/greenpool.html for information on
    how this works.

    It gathers statistics about the submissions executed for
    post-analysis...
    """
    - statistics (read-only property)
    - alive (read-only property)
    - submit (same as parent class)
    - shutdown (same as parent class)

ExecutorStatistics(object)
    """Holds *immutable* information about a executors executions."""
    - failures (read-only property)
    - executed (read-only property)
    - runtime (read-only property)
    - cancelled (read-only property)
    - average_runtime (read-only property)

Implementation

Assignee(s)

Primary assignee:

  • Harlowja

Other contributors:

  • Sileht

Primary maintainer

Primary maintainer:

  • Harlowja

Other contributors:

  • Sileht

Security Contact

Security Contact: harlowja

Milestones

Target Milestone for completion: liberty-1

Work Items

  • Create launchpad project

  • Change owner of Launchpad project (make it part of the Oslo project group)

  • Give openstackci Owner permissions on PyPI

  • Create Initial Repository

  • Make the library do something ✓

  • Update the README.rst

  • Publish git repo

  • Oslo team review new repository

  • Infra project configuration

  • Update Gerrit Groups and ACLs

  • openstack-infra/devstack-gate adjustments

  • openstack/requirements projects.txt adjustments

  • Update project list on docs.openstack.org

  • Tag a release

  • Profit!

Adoption Notes

N/A

Dependencies

Requirements

Note

All of the currently planned dependencies are in the requirements repository.

References

N/A

Note

This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode