Rotating Docker logs

I've been disturbed by having my services unavailable on my machine because of "no space left on device". Every time I was forced to do some df -h / and du -h / commands and digging down to the root cause of the lack of disk space.

Today I again encountered this issue and I'm done with it. I decided to find out what has been swallowing my disk space.

du -sh /*
du -sh /var
du -sh /var/*
du -sh /var/lib/*
du -sh /var/lib/docker/*

I ran du -ch /var/lib/docker/containers/*/*-json.log, and got an interesting output here:

$ sudo sh -c 'du -ch /var/lib/docker/containers/*/*-json.log'
13G    /var/lib/docker/containers/../..-json.log

Holy damn. That's one good log file consisting of 13 gigabytes data.

So I just ran sudo sh -c 'truncate -s 0 /var/lib/docker/containers/*/*-json.log'. Problem solved?

No. The next thing is figuring out how I can prevent this from happening again in the future, and Logrotate comes to the rescue.

Create /etc/logrotate.d/docker-logs with the following content:

/var/lib/docker/containers/*/*.log {
 rotate 7

You can check out logrotate logs to see if you've got everything configured correctly. If it doesn't complain, then you can be assured that Docker container logs will be truncated every day.