Projected volume mounts, but service account token is never written

General Discussions
1

I’m trying to start a single-node cluster without kubeadm, but can’t seem to get service account tokens mounted into any containers.

I’m using a very simple test pod spec for this test:

apiVersion: v1
kind: Pod
metadata:
  name: debug
spec:
  hostNetwork: true
  containers:
    - name: alpine
      image: docker.io/library/alpine:3.23.3
      command:
        - /bin/sh
      args:
        - '-c'
        - 'sleep 3600'
      volumeMounts:
        - name: service-account
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true
  volumes:
    - name: service-account
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 360

I’m using the default namespace and service account:

kubectl apply -f tests/k8s/debug.yaml

When I go seek out the service account token, it’s not there:

$ kubectl exec debug -- ls -la /var/run/secrets/kubernetes.io/serviceaccount
total 0
drwxrwxrwx    1 root     root             0 Apr 16 19:42 .
drwxr-xr-t    1 root     root            28 Apr 16 19:42 ..

Debugging steps

Cranked verbosity on the API server to determine whether it was issuing tokens, and it is:

I0416 19:32:14.348730       1 httplog.go:134] "HTTP" verb="POST" URI="/api/v1/namespaces/default/serviceaccounts/default/token" latency="3.061416ms" userAgent="kubelet/v1.35.4 (linux/arm64) kubernetes/7b8c6cf" audit-ID="301b8b51-1542-4407-a6c9-012b4a420799" srcIP="192.168.18.3:45340" apf_pl="system" apf_fs="system-nodes" apf_iseats=1 apf_fseats=0 apf_additionalLatency="0s" apf_execution_time="2.760375ms" resp=201

Turned up the kubelet verbosity to make sure it was setting up the projected volume as expected, and it is:

Apr 16 19:32:14 test0 kubelet[1030]: I0416 19:32:14.146224    1030 kubelet.go:2601] "SyncLoop RECONCILE" source="api" pods=["default/debug"]
Apr 16 19:32:14 test0 kubelet[1030]: I0416 19:32:14.203544    1030 desired_state_of_world.go:310] "expected volume SELinux label context" volume="service-account" label=""
Apr 16 19:32:14 test0 kubelet[1030]: I0416 19:32:14.203564    1030 desired_state_of_world.go:330] "volume does not support SELinux context mount, clearing the expected label" volume="service-account"
Apr 16 19:32:14 test0 kubelet[1030]: I0416 19:32:14.203569    1030 desired_state_of_world_populator.go:318] "Added volume to desired state" pod="default/debug" volumeName="service-account" volumeSpecName="service-account"
Apr 16 19:32:14 test0 kubelet[1030]: I0416 19:32:14.244712    1030 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-account\" (UniqueName: \"kubernetes.io/projected/d2b0fbb5-a130-4b1a-af7c-26e9368dd796-service-account\") pod \"debug\" (UID: \"d2b0fbb5-a130-4b1a-af7c-26e9368dd796\") " pod="default/debug"
Apr 16 19:32:14 test0 kubelet[1030]: I0416 19:32:14.345215    1030 reconciler_common.go:214] "Starting operationExecutor.MountVolume for volume \"service-account\" (UniqueName: \"kubernetes.io/projected/d2b0fbb5-a130-4b1a-af7c-26e9368dd796-service-account\") pod \"debug\" (UID: \"d2b0fbb5-a130-4b1a-af7c-26e9368dd796\") " pod="default/debug"
Apr 16 19:32:14 test0 kubelet[1030]: I0416 19:32:14.345239    1030 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-account\" (UniqueName: \"kubernetes.io/projected/d2b0fbb5-a130-4b1a-af7c-26e9368dd796-service-account\") pod \"debug\" (UID: \"d2b0fbb5-a130-4b1a-af7c-26e9368dd796\") " pod="default/debug"
Apr 16 19:32:14 test0 kubelet[1030]: I0416 19:32:14.345288    1030 projected.go:187] Setting up volume service-account for pod d2b0fbb5-a130-4b1a-af7c-26e9368dd796 at /var/lib/kubelet/pods/d2b0fbb5-a130-4b1a-af7c-26e9368dd796/volumes/kubernetes.io~projected/service-account
Apr 16 19:32:14 test0 kubelet[1030]: I0416 19:32:14.349140    1030 empty_dir_linux.go:99] Statfs_t of /var/lib/kubelet/pods/d2b0fbb5-a130-4b1a-af7c-26e9368dd796/volumes/kubernetes.io~projected/service-account: {Type:2435016766 Bsize:4096 Blocks:13699479 Bfree:12552315 Bavail:12458716 Files:0 Ffree:0 Fsid:{Val:[-1529449832 -899098667]} Namelen:255 Frsize:4096 Flags:4132 Spare:[0 0 0 0]}
Apr 16 19:32:14 test0 kubelet[1030]: I0416 19:32:14.349169    1030 empty_dir.go:338] pod d2b0fbb5-a130-4b1a-af7c-26e9368dd796: mounting tmpfs for volume wrapped_service-account
Apr 16 19:32:14 test0 kubelet[1030]: I0416 19:32:14.349180    1030 mount_linux.go:260] Mounting cmd (mount) with arguments (-t tmpfs -o size=2979602432,noswap tmpfs /var/lib/kubelet/pods/d2b0fbb5-a130-4b1a-af7c-26e9368dd796/volumes/kubernetes.io~projected/service-account)
Apr 16 19:32:14 test0 kubelet[1030]: I0416 19:32:14.351520    1030 atomic_writer.go:203] pod default/debug volume service-account: performed write of new data to ts data directory: /var/lib/kubelet/pods/d2b0fbb5-a130-4b1a-af7c-26e9368dd796/volumes/kubernetes.io~projected/service-account/..2026_04_16_19_32_14.91822510
Apr 16 19:32:14 test0 kubelet[1030]: I0416 19:32:14.351566    1030 operation_generator.go:614] "MountVolume.SetUp succeeded for volume \"service-account\" (UniqueName: \"kubernetes.io/projected/d2b0fbb5-a130-4b1a-af7c-26e9368dd796-service-account\") pod \"debug\" (UID: \"d2b0fbb5-a130-4b1a-af7c-26e9368dd796\") " pod="default/debug"
Apr 16 19:32:14 test0 kubelet[1030]: I0416 19:32:14.445621    1030 volume_manager.go:471] "All volumes are attached and mounted for pod" pod="default/debug"

I went to the host path where the projected volume is mounted, and it is also unsurprisingly empty:

$ sudo ls -al /var/lib/kubelet/pods/d2b0fbb5-a130-4b1a-af7c-26e9368dd796/volumes/kubernetes.io~projected/service-account
total 0
drwxrwxrwx. 1 root root  0 Apr 16 19:42 .
drwxr-xr-x. 1 root root 30 Apr 16 19:42 ..

I’m a little lost, I don’t really know where else to look. I’m happy to share the static pod spec for the API server, and the kubelet service unit file if you have a hunch. I’m really surprised this seems to be failing silently.

Updates

Adding more here as I figure things out:

I can inspect the mounts from the perspective of kubelet:

$ ps -e | grep kubelet
   1017 ? 00:00:01 kubelet
$ cat /proc/1017/mounts | grep service-account
tmpfs /var/lib/kubelet/pods/d2b0fbb5-a130-4b1a-af7c-26e9368dd796/volumes/kubernetes.io~projected/service-account tmpfs rw,seclabel,relatime,size=2909768k,inode64,noswap 0 0

The tmpfs mount for the projected volume is indeed there. Is it normal for the kubelet to have its own mount namespace though? Wouldn’t that prevent the container runtime from seeing the contents of the projected volume directory? If I use the kubelet’s mount namespace, everything is there:

$ sudo nsenter --target=1017 -m
$ ls /var/lib/kubelet/pods/d2b0fbb5-a130-4b1a-af7c-26e9368dd796/volumes/kubernetes.io~projected
token

Is there something wrong with the way systemd is managing the service, creating a mount namespace for the kubelet process when it shouldn’t? Or is this expected?

Cluster information:

Kubernetes version: 1.35.4
Cloud being used: bare-metal
Installation method: custom
Host OS: Fedora 43
CNI and version: hostNetwork
CRI and version: CRI-O v1.35.2

2

Hello,

Did you try adding serviceaccount with automountServiceAccountToken as true:

apiVersion: v1
kind: Pod
metadata:
  name: debug
spec:
  serviceAccountName: <service-account-name>
  automountServiceAccountToken: true

It automounts the token.
Let me know if this works for your use case.

3

Hi! I believe auto-mounting is opt-out rather than opt-in by default, but I tested it anyway:

apiVersion: v1
kind: Pod
metadata:
  name: debug
spec:
  hostNetwork: true
  automountServiceAccountToken: true
  serviceAccountName: default
  containers:
    - name: alpine
      image: docker.io/library/alpine:3.23.3
      command:
        - /bin/sh
      args:
        - '-c'
        - 'sleep 3600'

The token still isn’t there:

$ kubectl exec debug -- ls -l /var/run/secrets/kubernetes.io/serviceaccount
total 0

Logs are the same as far as I can tell. Thanks for your response though!

4

Hi,

Thank you for trying this.

I missed an important detail in your case, i.e. it’s a static-pod.
AFAIK, the static Pod spec cannot refer to API objects like configMap, secrets, serviceAccount, etc.

This means we cannot use

spec:
  serviceAccountName: <service-account-name>
  automountServiceAccountToken: true

Nor can we use projected volume, like you were doing, because projected volume configuration will refer to the serviceAccount for the parent pod.

  ... 
  serviceAccountName: <serviceaccount-name>
  volumes:
  - name: <volume-name>
    projected:
      sources:
      - serviceAccountToken:
        ...

If you do not provide serviceAccountName the pod is going to defer to the default service account. Which takes us back to the limitation of static-pods, they can’t have serviceAccount object.

Conclusion.

I think you should rather use a daemonSet instead of a static pod, as they are managed through and are cared for by the kubernetes API-server, unlike a static pod which relies on kubelet.

If you desire Pods to run on every node, you should probably be using a DaemonSet instead.

This will give you libertly to work with serviceAccount and tokens.

5

The “debug” pod I’m referencing is not a static pod, but a regular pod submitted using kubectl. Sorry if that was unclear. I only mentioned static pods to explain that I’m using my own static pod specs for control plane components, which include some customization that may be relevant. Given the API server is returning valid tokens without any issues though, I’ll update the initial question to avoid confusion.

I really do appreciate you taking the time though, thank you!

6

Fixed it! The culprit was my own misguided systemd service unit file.

[Unit]
...

[Service]
...
StateDirectory=kubernetes::ro
ExecStart=/usr/bin/kubelet ...

[Install]
...

Adding ::ro to make a directory path read-only makes a new mount namespace for the unit.

The container runtime interface was mounting the projected directory from the host mount namespace, but the kubelet was writing to a private tmpfs mount in its own mount namespace, therefore never propagating the mount to the host namespace. Use of nsenter made this apparent as described in the updated original post.

Removing ::ro fixed it.

7

You’re welcome! I love to see & try to help with Kubernetes challenges/issues.
Glad that you figured the root cause.
Happy that I learned something too.

See ya & happy k8s orchestration!