The files in /var/lib/docker/overlay2/ have grown large, causing Anthias not to start (HDD capacity is running low)

Anthias
1

The HDD capacity on my Raspberry Pi 4 is running low, and Anthias is not starting up.

Please refer to the following for details about the environment.

Raspberry Pi Model ・・・ Raspberry Pi 4 Model B Rev 1.5
Anthias Version・・・experimental@7f04d72
HDD capacity・・・32GB

As a result of the investigation, a single file within the directory below had grown to 24GB

root@raspberrypi:/var/lib/docker# du -sh /var/lib/docker/overlay2/*
20K /var/lib/docker/overlay2/028f08e57221cd8570ea375cbfa52b4a81c030b8ab919a97096f66f7f4c6e397
808M /var/lib/docker/overlay2/0503221c4f147c7ac48a2836bd93e1299c98d2213b6418a38278bf07b9629750
24G /var/lib/docker/overlay2/0e688e127544bd1e5f2b9f2fda7ef1e7cd5c1ab00efdae2929e0ee1796915da5
20K /var/lib/docker/overlay2/0e8fa204b894fa5931954985b6dffc6e1a012826bd48c1132b989b87a71a9ee8
・
・
・

After deleting this 24GB file , the issue was resolved.

While this situation seems abnormal, I would like to know the reason behind it.
Additionally, if anyone knows how to prevent this issue from occurring again, I would appreciate your advice."

I look forward to your response.Thank you.

2

I’m very sorry to hear that. Artifacts like container logs, among many others, are not cleaned up by default.

If you’re running Anthias on a device running Raspberry Pi OS Lite (which is essentially Debian) or Debian, you can SSH to the device and run the following:

docker system prune -f # Prefix `sudo` if you're having permission-related issues.

For reference, we also support cleaning up of Docker artifacts (uused containers and images, for instance) when the install/upgrade script is executed, but feel free to run the prune command whenever you need it.

I hope it helps.

3

Thank you for your prompt response.

Would running the command docker system prune -f daily via cron, for example, have any impact on Anthias?

4

I haven’t tested it yet, but it shouldn’t affect Anthias. For good measures, you can go to the web UI, click Settings, then click Get Backup, which you can recover later if data (assets and assets metadata) gets lost.

5

Thank you for your response. I tried the command, and it seems that some files were deleted. However, the files in /var/lib/docker/overlay2/ were not removed.

To delete the files in /var/lib/docker/overlay2/, do I need to stop docker, delete them with rm -rf, and then restart docker?

6

@koukkou, are the assets and the assets metadata wiped off as well?

I havent tried removing /var/lib/docker/overlay2 yet, you can backup the assets data first and try that one.

Yes. feel free to create a cron job for running docker system prune -f and sudo rm -rf /var/lib/docker/overlay2.

7

@nicomiguelino, thank you for your question. When I deleted the previous 24GB file, the assets and metadata were not removed, and there were no issues.

I’d like to delete everything under /var/lib/docker/overlay2/*, but I’m hesitant as I’m concerned about the potential impact.

To avoid any issues with Anthias, I feel it’s better to delete only when oversized files appear. However, I would like to delete everything under /var/lib/docker/overlay2/* if it’s safe, but I’m unsure.

I’m commenting here in case you might have some insights.

8

Hello, @koukkou. Thank you for sharing your thoughts. I haven’t tested that one yet, but feel free to experiment given that you have backup and such.

9

@nicomiguelino,Thank you.
It’s difficult to test because it’s a production environment, but I’ll try to do what I can. If you can confirm that deleting everything under /overlay2/ will not cause any issues, I would appreciate it if you could let me know.

10

@koukkou,

TL;DR: Please don’t remove the entire /var/lib/docker/overlay2 directory.

I tried removing the overlay2/ directory just now. After that, I cannot restart the Docker containers successfully. I keep getting error messages that /var/lib/docker/overlay2/<hex_value> cannot be found. Removing it made my Pi 4 unusable.

11

@nicomiguelino,Deleting entire directories feels way too risky. I’ll be careful! Thanks so much!

1 Like