Container logs are crucial for monitoring and debugging applications running inside containers. Whether troubleshooting an issue, analyzing performance, or tracking container activity, understanding how to efficiently manage container logs is essential. This guide will cover how to view, download, and extract logs using both Docker and Podman, including specific details about configuring log paths and log rotation options.
Viewing Container Logs
Both Docker and Podman provide commands to view container logs. These commands display the stdout (standard output) and stderr (standard error) streams from the running container. Logs can be forwarded to a remote host via supported protocols. By default, all logs can be written to the host.
Basic Command to View Logs
To view logs from a container:
For Docker:
docker logs <container_name_or_id>
For Podman:
podman logs <container_name_or_id>
For example:
docker logs my_container
or
podman logs my_container
Log File Path
For Docker, logs are stored in a specific path on the host machine:
/var/lib/docker/containers/<containerID>/<containerID>-json.log
Each container has its own directory, and the log files are saved as JSON.
For Podman, the log file path can be configured manually. Here’s an example from the setup script:
--log-opt path=/var/log/cse-v<Latest Tagged Version>.log
This allows you to specify a custom log location, which is useful for centralized log storage and management.
Showing the Latest Logs
To display only the most recent logs, both Docker and Podman support the '--tail' option:
docker logs --tail 100 <container_name_or_id>
podman logs --tail 100 <container_name_or_id>
This command shows the last 100 lines of logs.
Following Logs in Real-Time
Both Docker and Podman offer the -f
or --follow
option for real-time log monitoring:
docker logs -f <container_name_or_id>
podman logs -f <container_name_or_id>
Displaying Timestamped Logs
To add timestamps for when each log entry occurred, use the '--timestamps' option in Docker or Podman:
docker logs --timestamps <container_name_or_id>
podman logs --timestamps <container_name_or_id>
Configuring and Downloading Logs
In both Docker and Podman, logs can be exported and downloaded for offline analysis or external storage.
Configuring Log Rotation
Log rotation is important to prevent container logs from consuming too much disk space. Here's how to configure log rotation in Docker and Podman.
For Docker, log rotation is configured as part of the container setup:
docker run --detach\
--env-file ./cse.env\
-p 443:9000 \
-v /var/virtru/cse/server.cert:/run/secrets/server.cert \
-v /var/virtru/cse/server.key:/run/secrets/server.key \
--restart unless-stopped \
--log-opt max-size=100m \
--log-opt max-file=10 \
--name cse-v<Latest Tagged Version> \
containers.virtru.com/cse:v<Latest Tagged Version>
-
--log-opt max-size=100m
: This limits each log file to 100MB. -
--log-opt max-file=10
: Docker will keep a maximum of 10 log files, discarding the oldest ones.
For Podman, log rotation is configured similarly, with the ability to set the log path:
podman run --detach\
--env-file ./cse.env\
-p 443:9000 \
-v /var/virtru/cse/server.cert:/run/secrets/server.cert \
-v /var/virtru/cse/server.key:/run/secrets/server.key \
--restart unless-stopped \
--name cse-v<Latest Tagged Version> \
--log-opt path=/var/log/cse-v<Latest Tagged Version>.log \
containers.virtru.com/cse:v<Latest Tagged Version>
In this setup, logs are saved to /var/log/cse-v<Latest Tagged Version>.log
on the host machine, and the file can be rotated similarly to Docker logs by configuring the 'log-opt' parameters.
Downloading Logs
To download logs, you can save them to a file and then retrieve them from the host machine.
For Docker:
docker logs <container_name_or_id> > container_logs.txt
For Podman:
podman logs <container_name_or_id> > container_logs.txt
You can then use 'scp' or other file transfer tools to download the log file from the remote host:
scp user@remote_host:/path/to/container_logs.txt /local/path/
Extracting and Analyzing Logs
Compressing Logs
Logs can grow quite large. Compressing them makes it easier to store and transfer.
For Docker:
docker logs <container_name_or_id> | gzip > container_logs.tar.gz
For Podman:
podman logs <container_name_or_id> | gzip > container_logs.tar.gz
Extracting Logs
To extract logs from a compressed .tar.gz
file, use the tar
command:
tar -xvzf container_logs.tar.gz
Analyzing Logs
Once logs are downloaded, you can use standard Linux tools for analysis:
- Grep: Search for specific keywords or error patterns.
grep 'ERROR' container_logs.txt
- AWK/Sed: Extract specific fields or manipulate log data.
awk '/ERROR/ {print $2, $3, $4}' container_logs.txt
- Log Aggregation: Send logs to centralized services like ELK (Elasticsearch, Logstash, Kibana), Fluentd, or Datadog for real-time monitoring and analysis.
Exporting Logs to a SIEM
Ubuntu Docker Host
- Container Name: cse-v<Latest Tagged Version>
- Remote Syslog Server: 192.168.10.15
- Remote Syslog Server Port: 10514
- Remote Syslog Transport: TCP
cd /var/virtru/cse/scripts
nano run.sh
Once inside the script file, modify the logging entries to match:
docker run --detach\
--env-file ./cse.env\
-p 443:9000 \
-v /var/virtru/cse/server.cert:/run/secrets/server.cert \
-v /var/virtru/cse/server.key:/run/secrets/server.key \
--restart unless-stopped \
--name cse-v<Latest Tagged Version> \
--log-driver syslog \
--log-opt syslog-address=tcp://192.168.10.15:10514 \
containers.virtru.com/cse:v<Latest Tagged Version>
Host Level Logs
Logs can be read locally and directed to a remote syslog server.
Note: There will be additional events generated beyond the container logs
Example:
- Ubuntu Docker Host
- Remote Syslog Server: 192.168.10.15
- Remote Syslog Server Port: 10514
- Remote Syslog Transport: TCP
nano /etc/rsyslog.d/50-default.conf
Add at the top:
*.* action(type="omfwd" target="192.168.10.15" port="10514" protocol="tcp" action.resumeRetryCount="100"
queue.type="linkedList" queue.size="10000")
This will forward all host syslog entries to the remote syslog and retry 100 times and queue up 10000 entries before discarding any. Due to the nature of TCP, if the remote syslog server is unavailable all entries will be blocked and discarded if the remote server is unavailable and a retry queue is not set.
Conclusion
Both Docker and Podman provide powerful tools for managing container logs. By understanding how to configure log paths, enable log rotation, and export logs for analysis, you can efficiently manage container activity and troubleshoot issues.
With Docker, logs are saved by default to /var/lib/docker/containers
, while Podman allows flexible log path configuration via the --log-opt path
option. Whether using Docker or Podman, you can control log file size, rotate logs, and analyze them using standard tools or external log management services.
Complete documentation on supported docker logging drivers can found here: https://docs.docker.com/config/containers/logging/configure/