Enable logging to stdout/journald for rsyslog¶
We can optimize the current logging implementation to take advantage of metadata that our default logging driver (journald) adds to the logs.
Currently, we put all the logs of the containers into a directory in the host (/var/log/containers/). While this approach works, it relies on mounting directories from the host itself. This makes it harder for logging forwarders, since we need to configure them to track all those files. With every service that we add, we end up having to write configuration for that service for those specific files.
Furthermore, we lose important metadata with this approach. We can figure out what service wrote what log, but we lose the container name and ID, which is very useful. These we can easily get just by using the default docker logging mechanism.
Instead of relying on the host filesystem for our logs, we can adopt a simpler solution that both preserves important metadata that is discarded by the current implementation and that will support most services without requiring per-service configuration.
The proposal is to configure containerized services to log to stdout/stderr as is common practice for containerized applications. This allows the logs to get picked up by the docker logging driver, and thus we can use “docker logs” to view the logs of a service as one would usually expect. It will also help us decouple the containers from the host, since we will no longer be relying on host filesystem mounts for log collection.
In the case of services where it’s difficult or not possible to log to
stdout or stderr, we will place log files in a docker volume, and this
volume will be shared with a sidecar container that will output the
logs to stdout so they are consumable by the logging drvier. This will
also apply for containers that log only to syslog (such as HAProxy).
We will stop mounting
/dev/log from the host, and instead add a
sidecar container that will output the logs instead.
Additionally, since our default logging driver is journald, we will
get all the container logs accessible via
journalctl and the
journald libraries. So one would be able to do
CONTAINER_NAME=<container name> to get the logs of a specific
container on the node. Furthermore, we would get extra metadata
information for each log entry . We would benefit for
getting the container name (as the
CONTAINER_NAME metadata item)
and the container ID (as the
CONTAINER_ID_FULL metadata items) from each journald log entry
without requiring extra processing. Adding extra tags to the
containers is possible , and would get reflected via the
CONTAINER_TAG metadata entry. These tags can optionally describe the
application that emitted the logs or describe the platform that it
This will also make it easier for us to forward logs, since there will be a centralized service (journald) on each host from which we can collect the logs. When we add a new service, it will be a matter of following the same logging pattern, and we will automatically be able to forward those logs without requiring specific configuration to track a new set of log files.
With this solution in place, we need to also provide tooling to integrate with centralized logging solutions. This will then cover integration to the Openshift Logging Stack  and ViaQ . We are proposing the use of rsyslog for message collection, manipulation, and log forwarding. This will also be done in a containerized fashion, where rsyslog will be a “system container” that reads from the host journal. Rsyslog will perform metadata extraction from log messages (such as extracting the user, project, and domain from standard oslo format logs), and will then finally forward the logs to a central collector.
The implementation needs to be done in a pluggable manner. This is because
end-users have already created automation based on the assumption that logs
exist in the
that we have been providing. For this reason, logging to stdout/stderr will be
optional, and we’ll keep logging to files in the host as a default for now.
This will then be optionally enabled via an environment file.
In the proposed solution, the standard nova logs will go to the nova_api container’s stdout/stderr. However, since we are also interested in the apache access logs, we will then create a docker volume where the access logs will be hosted. A sidecar container will mount this volume, create a FIFO (named pipe) and output whatever it gets from that file. Note that this sidecar container will need to be started before the actual nova_api container.
For each log file generated in the main container, we will create a sidecar container that outputs that log. This will make it easier to associate log messages with the originating service.
Keep logging to files in the hosts’ directory.
We can still use the current solution; however, it is not ideal as it violates container logging best practices, relies heavily on directories on the host (which we want to avoid) and is inconsistent in the way we can get logging from services (some in files, some in syslog).
Other End User Impact¶
Since we’re not getting rid of the previous logging solution, users won’t be impacted. They will, however, get another way of getting logs and interacting with them in the host system, and further create automation from that if needed.
TODO: Any performance considerations on getting everything to journald?
- Primary assignees:
jaosorior jbadiapa larsks
Allow services to log to stdout/stderr (if possible).
Implement pluggable logging for each service in t-h-t.
Add Rsyslog container.
TODO: Evaluate how can we log to an EFK stack in upstream CI. Do we have one available?
 https://docs.docker.com/engine/admin/logging/journald/  https://docs.docker.com/engine/admin/logging/log_tags/  https://docs.openshift.com/container-platform/3.5/install_config/aggregate_logging.html  https://github.com/ViaQ/Main/blob/master/README-install.md