I would like to know if Kubernetes or Docker provided some workaround for this issue (instead of restart all pods and causes interruption)?
As I am not familiar with the Docker file system, I am just notice that the /var mount point is used over 90% during the issue. Thus, I am wonder why the over 50% of disk space under “/var/” is still occupied by Docker container as I have limited the log file and ephemeral storage in Docker and Kubernetes respectively.
My Docker service options:
[Service]
Environment="DOCKER_OPTS= --iptables=false
–data-root=/var/lib/docker
–log-opt max-size=50m --log-opt max-file=5
"
All ephemeral-storage is limited in Kubernetes as shown.
resources:
limits:
ephemeral-storage: 2Gi
And the docker system df is not that much after restart all pod:
$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 25 25 2.537GB 44.97MB (1%)
Containers 48 45 45.32MB 0B (0%)
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0B
“/var” mount point
$ df -hT /var
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/cl-var ext4 98G 8.8G 85G 10% /var