Best practices for logging of containerized services¶

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/tripleo/+spec/containerized-services-logs

Containerized services shall persist its logs. There are many ways to address that. The scope of this blueprint is to suggest best practices and intermediate implementation steps for Pike release as well.

Problem Description¶

Pike will be released with a notion of hybrid deployments, which is some services may be running in containers and managed by docker daemon, and some may be managed by systemd or Pacemaker and placed on hosts directly.

The notion of composable deployments as well assumes end users and developers may want to deploy some services non-containerized and tripleo heat templates shall not prevent them from doing so.

Despite the service placement type, end users and developers shall get all logs persisted, consistent and available for future analysis.

Proposed Change¶

Overview¶

Note

As the spec transitions from Pike, some of the sections below are split into the Pike and Queens parts.

The scope of this document for Pike is limited to recommendations for developers of containerized services, bearing in mind use cases for hybrid environments. It addresses only intermediate implementation steps for Pike and smooth UX with upgrades from Ocata to Pike, and with future upgrades from Pike as well.

A 12factor is the general guideline for logging in containerized apps. Based on it, we rephrase our main design assumption as: “each running process writes its only event stream to be persisted outside of its container”. And we put an additional design constraint: “each container has its only running foreground process, nothing else requires persistent logs that may outlast the container execution time”. This assumes all streams but the main event stream are ephemeral and live no longer than the container instance does.

Note

HA statefull services may require another approach, see the alternatives section for more details.

The scope for future releases, starting from Queens, shall include best practices for collecting (shipping), storing (persisting), processing (parsing) and accessing (filtering) logs of hybrid TripleO deployments with advanced techniques like EFK (Elasticsearch, Fluentd, Kibana) or the like. Hereafter those are referred as “future steps”.

Note, this is limited to OpenStack and Linux HA stack (Pacemaker and Corosync). We can do nothing to the rest of the supporting and legacy apps like webservers, load balancing revers proxies, database and message queue clusters. Even if we could, this stays out of OpenStack specs scope.

Here is a list of suggested best practices for TripleO developers for Pike:

Host services shall keep writing logs as is, having UIDs, logging configs, rotation rules and target directories unchanged.

Note

Host services changing its control plane to systemd or pacemaker in Ocata to Pike upgrade process, may have logging configs, rules and destinations changed as well, but this is out of the scope of this spec.
Containerized services that normally log to files under the /var/log dir, shall keep logging as is inside of containers. The logs shall be persisted with hostpath mounted volumes placed under the /var/log/containers path. This is required because of the hybrid use cases. For example, containerized nova services access /var/log/nova with different UIDs than the host services would have. Given that, nova containers should have log volumes mounted as -v /var/log/nova:/var/log/containers/nova in order to not bring conflicts. Persisted log files then can be pulled by a node agent like fluentd or rsyslog and forwarded to a central logging service.
Containerized services that can only log to syslog facilities: bind mount /dev/log into all tripleo service containers as well so that the host collects the logs via journald. This should be a standard component of our container “API”: we guarantee (a) a log directory and (b) a syslog socket for every containerized service. Collected journald logs then can be pulled by a node agent like fluentd or rsyslog and forwarded to a central logging service.
Containerized services that leverage Kolla bootstrap, extended start and/or config facilities, shall be templated with Heat deployment steps as the following:
- Host prep tasks to ensure target directories pre-created for hosts.
- Kolla config’s permissions to enforce ownership for log dirs (hostpath mounted volumes).
- Init containers steps to chown log directories early otherwise. Kolla bootstrap and DB sync containers are normally invoked before the kolla_config permissions to be set. Therefore come init containers.
Containerized services that do not use Kolla and run as root in containers shall be running from a separate user namespace remapped to a non root host user, for security reasons. No such services are currently deployed by TripleO, though.

Note

Docker daemon would have to be running under that remapped non root user as well. See docker documentation for the --userns-remap option.
Containerized services that run under pacemaker (or pacemaker remote) control plane and do not fall into any of the given cases: bind mount /dev/log as well. At this stage the way services log is in line with the best practice w.r.t “dedicated log directory to avoid conflicts”. Pacemaker bundles isolate the containerized resources’ logs on the host into /var/log/pacemaker/bundles/{resource}.

Future steps TBD.

Alternatives¶

Those below come for future steps only.

Alternatively to hostpath mounted volumes, create a directory structure such that each container has a namespace for its logs somewhere under /var/log. So, a container named 12345 would have all its logs in the /var/log/container-12345 directory structure (requires clarification). This also alters the assumption that in general there is only one main log per a container, which is the case for highly available containerized statefull services bundled with pacemaker remote, with multiple logs to capture, like /var/log/pacemaker.log, logs for cluster bootstrapping events, control plane agents, helper tools like rsyncd, and the statefull service itself.

When we have control over the logging API (e.g. via oslo.log), we can forsake hostpath mounted volumes and configure containerized services to output to syslog (via bind mount /dev/log) so that the host collects the logs via journald). Or configure services to log only to stdout, so that docker daemon collects logs and ships them to the journald.

Note

The “winning” trend is switching all (including openstack services) to syslog and log nothing to the /var/log/, e.g. just bind-mount -v /dev/null:/var/log for containers.

Or use a specialized log driver like the oslo.log fluentd logging driver (instead of the default journald or json-file) to output to a fluentd log agent running on the host or containerized as well, which would then aggregate logs from all containers, annotate with node metadata, and use the fluentd secure_forward protocol to send the logs to a remote fluentd agent like common logging.

These are not doable for Pike as requiring too many changes impacting upgrade UX as well. Although, this is the only recommended best practice and end goal for future releases and future steps coming after Pike.