No space left on device

General Discussions
1

Cluster information:

Kubernetes version: 1.26.3
Installation method: kubesprey
Host OS: Ubuntu 22.04

Hello, I can’t run the self-built image in kubernetes, I built the image in docker and it worked fine, when I run it in kubernetes I get an error:

MountVolume.SetUp failed for volume “kube-api-access-wfc5x” : write /var/lib/kubelet/pods/5855a853-25a1-4664-9e74-f8600e5c4d48/volumes/kubernetes.io~projected/kube-api-access-wfc5x/..2023_07_05_07_00_49.2076039338/token: no space left on device

kubelet Unable to attach or mount volumes: unmounted volumes=[kube-api-access-wfc5x], unattached volumes=[kube-api-access-wfc5x]: timed out waiting for the condition

My deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: minecraft
  labels:
    app: minecraft
spec:
  replicas: 1
  selector:
    matchLabels:
      app: minecraft
  template:
    metadata:
      labels:
        app: minecraft
    spec:
      containers:
      - image: www.maks100128.ru:5050/root/gitlab/minecraft:latest
        name: minecraft
        workingDir: /minecraft
        ports:
        - containerPort: 25565
        resources:
            limits:
              memory: 3000m
      imagePullSecrets:
      - name: gitlab

My dockerfile

FROM centos:7

WORKDIR /minecraft
RUN yum -y install wget nano 
RUN wget https://piston-data.mojang.com/v1/objects/84194a2f286ef7c14ed7ce0090dba59902951553/server.jar
RUN wget https://download.oracle.com/java/17/latest/jdk-17_linux-x64_bin.rpm
RUN rpm -ivh jdk-17_linux-x64_bin.rpm
RUN java -version
RUN java -Xmx1024M -Xms1024M -jar server.jar nogui
RUN echo "eula=true" > eula.txt
RUN sed -i -e 's/online-mode=true/online-mode=false/g' server.properties

What could be the problem? I have enough disk space on my drive

2

May be in the node you may have space, but did you check the space where /var mounted partion
please check the spache of the partion which mounts /var

3 
@maks1001281
 
was this resolved? am getting same error. What was the resolution?
4

Pleae ceck the /var space

5

If you’ve already checked that /var has enough disk space (df -h /var), the next thing to check is inode exhaustion:

df -i /var/lib/kubelet

If IUse% is above 90%, you’ve run out of inodes (file entries) even though you have free bytes. This is common in Kubernetes when container images accumulate many small files.

Quick cleanup if inodes are the issue:

  crictl rmi --prune
  crictl rm $(crictl ps -a -q --state exited)

@maks1001281 - can you share the output of both df -h /var and df -i /var? That would help diagnose the exact cause.

Btw I wrote a detailed article covering this exact scenario with more diagnostic commands and long-term fixes: 'No Space Left on Device' with 40% Disk Free: The Inode and OverlayFS Death Spiral | Michal Drozd