💢 🧒🏻 🤱🏼 Logging in Kubernetes: how to collect, store, parse and process logs 👨🏼‍🎤 😟 ™️

We will analyze the basics of logging in Docker and Kubernetes, and then consider two tools that can be safely used in production: Grafana Loki and the EFK stack (Elasticsearch + Fluent Bit + Kibana).

The material of the article is an extract from an open lecture of the Slurm school . If there is a desire, and even more so for a production need, you can complete a full training - sign up for a course on Monitoring and Logging Infrastructure in Kubernetes .

Docker logging

At the Kubernetes level, applications are running in pods, but at the level below they usually run in Docker. Therefore, you need to configure logging in such a way as to collect logs from containers. Containers are launched by Docker - so you need to figure out how logging works at the Docker level.

, : stdout/stderr, . Docker Daemon, , stdout/stderr. : ( Logrotate ), Docker Daemon .

Docker - . Docker Community Edition (CE) - , Docker Enterprise Edition (EE).

Docker EE : Southbridge Open Source , Docker EE .

- Docker CE:

local — Docker Daemon;

json-file — json-log ;

journald — journald.

Docker daemon.json.

“log-driver” , “log-opts” — . “json-file”, — “max-size”: “10m”; ( ) — “max-file”: “3”; , .

- . , -.

Docker:

: -, json-file, . (Rsyslog, Fluentd, Logagent ) Elastic, Sematext .

Kubernetes

Kubernetes : pod, , stdout/stderr. Docker , .

Kubernetes.

. . , , . Kubernetes --previous, Pod, .

. , . , .

, . (, Rsyslog), — Docker (, journal-bit - Docker journald). journal-bit — ( - Docker , journald), ( CentOS 7 systemd journald). , . , journal-bit , .

— . CentOS 7 (messages, audit, secure) var- . Docker json. , CentOS 7 Docker .

ELK Stack. : Elasticsearch, Logstash Kibana.

Elasticsearch , Logstash , Kibana , . ELK Stack , , , . , .

. , , , . , . , , , Pod , namespace . .

. , . , , . — .

, , — , «warning» «error». nginx ingress-, , 200. : - Nginx, .

, . , , . 200. — ingress-.

, : , , , .

. , Prometheus, .

: , — . , .

, Kubernetes :