Photo by Mohammad Rahmani on Unsplash
Kubernetes Logging Basics: Tracking at Container and Pod Levels
In Kubernetes, logging at the container and pod levels is essential for monitoring and debugging applications. Kubernetes captures logs from each container in a running Pod.
Container-Level Logging
Container logs are generated by individual containers running within pods. These logs typically include application-specific logs, such as error messages, access logs, and other runtime information.
Image credits: Kubernetes
In general Kubernetes handles container logs in the following way:
Containers write their logs to standard output (
stdout) and standard error (stderr) streams.The container runtime, like Docker or containerd, intercepts these streams.
Kubernetes uses a standardized format (CRI logging) to interact with the container runtime and access the logs.
By default, logs are saved in files on the node, typically under
/var/log/pods.You can view container logs using the
kubectl logs <pod_name> [container_name]command.The
container_nameis optional if the pod only has one container.You can use the
-fflag to follow the logs in real-time.
Standard Output and Error Streams:
Containers in Kubernetes are designed to log information to the standard output (stdout) and standard error (stderr) streams. Container runtimes captures these streams and stores the logs for later access. Different container runtimes implement this in different ways; however, the integration with the kubelet is standardized as the Container Runtime Interface (CRI) logging format.
Kubernetes CRI defines a standard for container runtimes to follow, which also affects log format. A typical CRI log entry includes a timestamp, the output stream type, and the log message itself.
2023-10-06T00:17:09.669794202Z stdout F Your log message here
2023-10-06T00:17:09.669794202Z stdout P Another log pt 1
2023-10-06T00:17:09.669794202Z stdout P Another log pt 2
2023-10-06T00:17:10.113242941Z stderr F Another log final
Log Storage:
The way that the kubelet and container runtime write logs depends on the operating system that the node uses (Linux, Windows).
On Linux nodes that use systemd, the kubelet and container runtime write to journald by default. You use journalctl to read the systemd journal; for example: journalctl -u kubelet.
If systemd is not present, the kubelet and container runtime write to .log files in the /var/log directory. If you want to have logs written elsewhere, you can indirectly run the kubelet via a helper tool, kube-log-runner, and use that tool to redirect kubelet logs to a directory that you choose.
By default, kubelet directs your container runtime to write logs into directories within /var/log/pods.
For more information on kube-log-runner, read System Logs.
Accessing Logs:
You can access container logs using the kubectl logs command:
kubectl logs <pod-name> -c <container-name>
If a pod has only one container, you can omit the -c <container-name> option.
You can use kubectl logs --previous to retrieve logs from a previous instantiation of a container.
Log Aggregation:
While accessing logs directly is useful for quick debugging, for large-scale applications, aggregating logs from all containers and storing them in a centralized location is more efficient. This is usually done using logging agents like Fluentd, Logstash, Loki or others, which collect logs from containers and forward them to a logging backend.
Pod-Level Logging
Pod logs encompass the logs of all containers within a pod. A pod is the smallest deployable unit in Kubernetes and can consist of one or more containers that share the same network namespace and storage volumes.
Multi-Container Pods:
When a pod contains multiple containers, each container logs independently. These logs need to be aggregated to get a complete picture of the pod’s activities.
In general Kubernetes manages pod level logging in the following way:
A pod might have multiple containers.
Kubernetes captures logs from each container within the pod.
By default,
kubectl logs <pod_name>retrieves logs only from the first container.To view logs from a specific container within a pod, use
kubectl logs <pod_name> -c <container_name>.
Sidecar Containers:
Often, a sidecar container is used within the same pod to handle logging. The sidecar can read logs from shared volumes or directly from other containers and forward them to a centralized logging system.
You can use a sidecar container in one of the following ways:
The sidecar container streams application logs to its own
stdout.The sidecar container runs a logging agent, which is configured to pick up logs from an application container.
Image credits: Kubernetes
The sidecar containers read logs from a file, a socket, or journald. Each sidecar container prints a log to its own stdout or stderr stream.
This approach allows you to separate several log streams from different parts of your application, some of which can lack support for writing to stdout or stderr. The logic behind redirecting logs is minimal, so it's not a significant overhead. Additionally, because stdout and stderr are handled by the kubelet, you can use built-in tools like kubectl logs.
For example, a pod runs a single container, and the container writes to two different log files using two different formats. Here's a manifest for the Pod:
apiVersion: v1
kind: Pod
metadata:
name: counter
spec:
containers:
- name: count
image: busybox:1.28
args:
- /bin/sh
- -c
- >
i=0;
while true;
do
echo "$i: $(date)" >> /var/log/1.log;
echo "$(date) INFO $i" >> /var/log/2.log;
i=$((i+1));
sleep 1;
done
volumeMounts:
- name: varlog
mountPath: /var/log
volumes:
- name: varlog
emptyDir: {}
It is not recommended to write log entries with different formats to the same log stream, even if you managed to redirect both components to the stdout stream of the container. Instead, you can create two sidecar containers. Each sidecar container could tail a particular log file from a shared volume and then redirect the logs to its own stdout stream.
Here's a manifest for a pod that has two sidecar containers:
apiVersion: v1
kind: Pod
metadata:
name: counter
spec:
containers:
- name: count
image: busybox:1.28
args:
- /bin/sh
- -c
- >
i=0;
while true;
do
echo "$i: $(date)" >> /var/log/1.log;
echo "$(date) INFO $i" >> /var/log/2.log;
i=$((i+1));
sleep 1;
done
volumeMounts:
- name: varlog
mountPath: /var/log
- name: count-log-1
image: busybox:1.28
args: [/bin/sh, -c, 'tail -n+1 -F /var/log/1.log']
volumeMounts:
- name: varlog
mountPath: /var/log
- name: count-log-2
image: busybox:1.28
args: [/bin/sh, -c, 'tail -n+1 -F /var/log/2.log']
volumeMounts:
- name: varlog
mountPath: /var/log
volumes:
- name: varlog
emptyDir: {}
Now when you run this pod, you can access each log stream separately by running the following commands:
kubectl logs counter count-log-1
kubectl logs counter count-log-2
Sidecar containers can also be used to rotate log files that cannot be rotated by the application itself. An example of this approach is a small container running logrotate periodically. However, it's more straightforward to use stdout and stderr directly, and leave rotation and retention policies to the kubelet.
Log Management Tools:
Using tools like Fluentd, Fluent Bit, or Promtail (from Loki), you can deploy them as DaemonSets or sidecars to collect logs from all pods across the cluster.
Example Fluentd DaemonSet configuration:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
namespace: kube-system
labels:
k8s-app: fluentd-logging
spec:
selector:
matchLabels:
name: fluentd
template:
metadata:
labels:
name: fluentd
spec:
containers:
- name: fluentd
image: fluent/fluentd:v1.17
env:
- name: FLUENTD_ARGS
value: "--no-supervisor -q"
resources:
limits:
memory: 200Mi
cpu: 100m
requests:
memory: 200Mi
cpu: 100m
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
Centralized Logging
For large Kubernetes clusters, centralized logging is crucial. It involves collecting logs from all nodes, containers, and pods and storing them in a centralized logging system for easy access, search, and analysis. Common centralized logging solutions include: