Cinder task logging improvements¶
To improve the usefulness of the usage of tasks & flows in cinder in the
create_volume workflow (with more workflows to come later) we need to
improve the level of awareness and usage of the interconnect
between taskflow and cinder. This will help track a single workflow while
it runs (allowing for introspection into what is occurring or what has
occurred). It will help for operators, for developers and others to have this
information available for usage.
Current taskflow usage does not take advantage of the taskflow engine internals event/notification mechanism. This makes it hard to determine what is happening internally to taskflow and how to debug it when things start to fail (which they inevitably do). It would be better to have this information exposed in a more easier to use and understandable manner for future use as well (but we first have to get the basics working in the first place).
To start we need to hook into the notification system that taskflow engines provide/emit. This will solve the issue of having a useful set of information being emitted by the taskflow engines used in cinder and will help in debugging the state-transitions & activity of tasks and flows ran there-in. This will help solve the issue of debuggability and tracking of the internals of taskflow; making the lives of operators, developers, and users much easier.
Part of the proposed change and one that needs more feedback is the location of
this log (is it a log at all?). Using the log listener that taskflow
provides we can plug in any type of logging compatible object (typically
LOG from the python logging module) as the receiver of these events. In
the cinder case this should be a oslo incubator
LOG object (as is typically
used in cinder and elsewhere in openstack). The question becomes where should
LOG output the task and flow events to.
The following suggestion seems to be agreed upon:
A log location for task/flow events, provided by a new configuration option
task_log_file(the configuration option name can be changed as needed, this was just a example and may not be the best name for this usage) that would be set in
cinder.confthat would specify where this data would be written to. If this location is not set then it will default to the main cinder log (this will allow the log to be separated out for those that desire to do this).
Data model impact¶
REST API impact¶
Some ideas that could be useful in the future:
Provide the same information (or a reduced set of this information) to end users of the cinder api so that they can gain insight into what is occurring with there request inside cinder. This would involve saving the same event stream into something not a log file but some more appropriate structure that can be given back to the user in a nice format (json or other). This is similar to providing a task log back to the user of cinder that other OpenStack projects are also currently exploring/implementing.
Other end user impact¶
More log data would mean a rotation mechanism would need to be setup if it was
not setup previously. Since these log files can contain a lot of information
they would need to be rotated at a faster rate than log files that only
Other deployer impact¶
An optional configuration setting to enable a new log file location. If a log file location is added then it would involve a new log file rotation and associated policy and this is something operators would need to be aware of and correctly configure to avoid filling up a servers hard drive. The goal is that this new configuration option would default to the existing main cinder log (so that this impact would be minimal for most operators).
Primary assignee: harlowja
Create new config options for cinder for event stream target location.
Use event stream in
create_volumeusage of taskflow task/flows and engines.
It should be feasible to replace the
LOG or notification mechanism being
used in cinder with taskflow with a
mock object that can also gather the
same event stream and allow it to be used in a test to verify that the event
stream contains the desired events (likely we should be selective in the test
and only verify that certain key events were emitted, since taskflow could
add new events in the future).