Can't get UserNamespacesSupport to work on kubernetes 1.28.1

General Discussions
,
1

Cluster information:

Kubernetes version: 1.28.1
Cloud being used: bare-metal
Installation method: kubeadm
Host OS: Ubuntu 23.10
CNI and version: calico 1.30.7
CRI and version: crio 1.28.1 with crun 1.9.2

I want to enable UserNameSpaceSupport on my k8s cluster but I can’t get it to work.
I’ve followed User Namespaces: Now Supports Running Stateful Pods in Alpha exactly, but still cant get the expected results

From the above link, I understood the following requirement

On Linux you need Linux 6.3 or greater. This is because the feature relies on a kernel feature named idmap mounts, and support to use idmap mounts with tmpfs was merged in Linux 6.3.

If you are using CRI-O with crun, this is supported in CRI-O 1.28.1 and crun 1.9 or greater. If you are using CRI-O with runc, this is still not supported.

None of these containerd limitations apply to CRI-O 1.28.

I’ve configured my cluster exactly the same

root@ishan-mantic-minotaur:~/ishan# /usr/local/bin/crun --version
crun version 1.9.2

root@ishan-mantic-minotaur:~/ishan# /usr/local/bin/crio --version
crio version 1.28.1
Version:        1.28.1

root@ishan-mantic-minotaur:~/ishan# uname -r
6.5.0-9-generic

root@ishan-mantic-minotaur:~/ishan# kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2

root@ishan-mantic-minotaur:~/ishan# grep UserN /etc/kubernetes/manifests/kube-apiserver.yaml
    - --feature-gates=UserNamespacesSupport=true

I’m expecting the container to be in a different user namespace, and the process inside it to not run as root on the host. But that doesnt seem to happen.

# INSIDE POD

root@ishan-mantic-minotaur:~/ishan# k get po test4-6f5bc89f5b-82znp -o yaml | grep hostUser
  hostUsers: false

root@ishan-mantic-minotaur:~/ishan# k exec -it test4-6f5bc89f5b-82znp bash
root@test4-6f5bc89f5b-82znp:/# id
uid=0(root) gid=0(root) groups=0(root)
root@test4-6f5bc89f5b-82znp:/# readlink /proc/self/ns/user
user:[4026531837]
root@test4-6f5bc89f5b-82znp:/# cat /proc/self/uid_map
         0          0 4294967295
root@test4-6f5bc89f5b-82znp:/# sleep 12345 &
[1] 8


# ON HOST

root@ishan-mantic-minotaur:~/ishan# readlink /proc/self/ns/user
user:[4026531837]
root@ishan-mantic-minotaur:~/ishan# cat /proc/self/uid_map
         0          0 4294967295
root@ishan-mantic-minotaur:~/ishan# ps aux | grep 12345
root        8257  0.0  0.0   2484  1280 ?        S    14:31   0:00 sleep 12345

Please help me understand this.