I recently discovered the hard way while building a Docker Image that when you add a user with an abnormally large UID it will eat an exorbitant amount of disk space on your Docker host. The bug that lead me to this conclusion is here, and the fix is easy, just use the “-l” flag with the useradd command, But my original run left me in a pickle. In this one case, I can’t add enough disk space to my LVM logical volume to recover, I needed to delete the culprit. I tried to use docker system prune to no avail, tried deleting all unnecessary images, nothing. MBs were freed-up and I still had a giant black hole that swallowed up almost 50 GB.
This might not be the best approach and IT WILL DELETE YOUR DOCKER DATA, but for this host I don’t care about keeping docker data, All images are posted upstream in Artifactory once built, this host just builds them once and sends them upstream. In my case the culprit was a sub-folder in /var.lib/docker/overlay2 and took almost 50 GB. If you delete the specific sub-folder folder in overlay2, you’ll get an error that “no such file or directory” exists inside of the overlay2 directory when trying to build the image again. Here’s how I recovered, but be warned again, it’s a bit like killing a fly with a sledge hammer…. you’ve accomplished your goal in the end, but there’s a hole in the table when you’re done.
systemctl stop docker rm -rf /var/lib/docker/ systemctl start docker
This immediately freed up the space necessary, and Docker rebuilds the contents of /var/lib/docker when the daemon starts. From here I was able to run the build again without issues from the useradd command.