Cinder task logging improvements

https://blueprints.launchpad.net/cinder/+spec/task-logging

To improve the usefulness of the usage of tasks & flows in cinder in the create_volume workflow (with more workflows to come later) we need to improve the level of awareness and usage of the interconnect between taskflow and cinder. This will help track a single workflow while it runs (allowing for introspection into what is occurring or what has occurred). It will help for operators, for developers and others to have this information available for usage.

Problem description

Current taskflow usage does not take advantage of the taskflow engine internals event/notification mechanism. This makes it hard to determine what is happening internally to taskflow and how to debug it when things start to fail (which they inevitably do). It would be better to have this information exposed in a more easier to use and understandable manner for future use as well (but we first have to get the basics working in the first place).

Use Cases

Proposed change

To start we need to hook into the notification system that taskflow engines provide/emit. This will solve the issue of having a useful set of information being emitted by the taskflow engines used in cinder and will help in debugging the state-transitions & activity of tasks and flows ran there-in. This will help solve the issue of debuggability and tracking of the internals of taskflow; making the lives of operators, developers, and users much easier.

Part of the proposed change and one that needs more feedback is the location of this log (is it a log at all?). Using the log listener that taskflow provides we can plug in any type of logging compatible object (typically a LOG from the python logging module) as the receiver of these events. In the cinder case this should be a oslo incubator LOG object (as is typically used in cinder and elsewhere in openstack). The question becomes where should this LOG output the task and flow events to.

The following suggestion seems to be agreed upon:

  • A log location for task/flow events, provided by a new configuration option task_log_file (the configuration option name can be changed as needed, this was just a example and may not be the best name for this usage) that would be set in cinder.conf that would specify where this data would be written to. If this location is not set then it will default to the main cinder log (this will allow the log to be separated out for those that desire to do this).

Alternatives

N/A

Data model impact

N/A

REST API impact

Not currently.

Some ideas that could be useful in the future:

  • Provide the same information (or a reduced set of this information) to end users of the cinder api so that they can gain insight into what is occurring with there request inside cinder. This would involve saving the same event stream into something not a log file but some more appropriate structure that can be given back to the user in a nice format (json or other). This is similar to providing a task log back to the user of cinder that other OpenStack projects are also currently exploring/implementing.

Security impact

N/A

Notifications impact

N/A

Other end user impact

N/A

Performance Impact

More log data would mean a rotation mechanism would need to be setup if it was not setup previously. Since these log files can contain a lot of information they would need to be rotated at a faster rate than log files that only contain ERROR or WARNING information.

Other deployer impact

An optional configuration setting to enable a new log file location. If a log file location is added then it would involve a new log file rotation and associated policy and this is something operators would need to be aware of and correctly configure to avoid filling up a servers hard drive. The goal is that this new configuration option would default to the existing main cinder log (so that this impact would be minimal for most operators).

Developer impact

N/A

Implementation

Assignee(s)

Primary assignee: harlowja

Work Items

  • Create new config options for cinder for event stream target location.

  • Use event stream in create_volume usage of taskflow task/flows and engines.

Dependencies

N/A

Testing

It should be feasible to replace the LOG or notification mechanism being used in cinder with taskflow with a mock object that can also gather the same event stream and allow it to be used in a test to verify that the event stream contains the desired events (likely we should be selective in the test and only verify that certain key events were emitted, since taskflow could add new events in the future).

Documentation Impact

N/A

References

Summit discussion: