Docker generates huge number of tmp files

I ran into an issue with Docker. It generates a lot of TMP files and eventually fills up all the space on the SSD. Once that happens, I cannot run most of the commands on the system, for example when trying to stop the containers via $ docker-compose stop or $ docker-compose down ubuntu does not recognize these commands.

Harware I am using a home server which has the following configuration:

• 2x Xeon E5620
• 48G of ECC ram
• 120g ssd for running ubuntu + dockerized apps (EXT4 FS)
• 4x 2TB WD RED drives for storing data that is generated by the web apps (usually do data scraping for analytics) (EXT4 FS)

Software

The issue emerges when I am running a laravel application which does data scraping. Laravel is running on docker using docker-compose. I have the following containers:

• nginx
• php-fpm - used just as workspace
• cron - for running laravel schedules
• worker - for running laravel horizon using supervisor
• redis - for queues
• mysql - used to use mongo but reverted to mysql with json fields

Worker container is running 320 scraping jobs at a time. Performance is pretty good. I am able to scrape around 300, 000 websites per day but after several hours docker is cluttering the file system with tmp files and the performance starts to degrade.

The issue:

If I run $ sudo ncdu I can see that /var/lib/docker/overlay2 has 60+ gb of tmp files and 58GB of them are in /var/lib/docker/overlay2/8dc08cc525cefb8f7934e91979bfb6d57a464cbdd18ab0d3d7191b1b99fa6636/diff/tmp folder. File names look like "phpYL7iq7" and all of them have php as first three letters. Maybe any of you know where are these files coming from and if there is any mechanism to prevent generating these files.

I could run an shell script with a cron job to remove the files every hour or so but maybe there is some better solution or docker settings to automatically remove such tmp files?

via Chebli Mohamed