I've been disturbed by having my services unavailable on my machine because of "no space left on device". Every time I was forced to do some
df -h / and
du -h / commands and dig down to what leads to the lack of disk space.
Today I again encountered this issue and I'm done with it. I decided to find out what has been swallowing my disk space.
55 du -sh /*
56 du -sh /var
57 du -sh /var/*
58 du -sh /var/lib/*
59 du -sh /var/lib/docker/*
(be sure to
sudo su first if you want to use
du -ch /var/lib/docker/containers/*/*-json.log, and I got an interesting result:
$ sudo sh -c "du -ch /var/lib/docker/containers/*/*-json.log"
Holy crap. That's a log file of 13 gigabytes. I then moved to Google on how to clear Docker logs because there's no official implementation of so. The fastest and cleanest way is
sudo sh -c 'truncate -s 0 /var/lib/docker/containers/*/*-json.log', from https://stackoverflow.com/a/43570083/2465955.
Problem solved. The next thing is how do I prevent this from happening again in the future? And under the previous linked Stackoverflow answer, here's a quick answer to my next question: Rotate the log periodically.
/etc/logrotate.d/docker-logs, and add the following to the file:
And if everything is fine, logrotate.d will do the jobs.