Hai All,
I/m very new using kubernetes try to develop my container application. I use ubuntu server as master, ubuntu server installed in KVM. an use centos as worker, centos is vagrant with vmware_desktop provider. I setup 4 CPUs and 2GB memori for centos. I’v right setup in centos worker running docker and kubenertes service (kubelet) correctly. and I can join with kubeadm to master node kubernetes, so centos worker can join ubuntu server as master in my kubernetes cluster. But until now the status stiil showing not ready? in another worker I can join and status running when ubuntu server as worker(installed in KVM) join to ubuntu server as master, but why my centos is still status not ready? is this bug from kubernetes or what? if i try to run kubectl describe node centos the message produce with that command is NodeHasSufficientMemory.
Is not enough 2GB memory and 4 CPUs just only run worker node? I’m curious why just only run workder node need ram greater than 2GB, I htinks this bug in kubernetes cluster worker? I’ve searched on the internet but no one I can find related problem solving with my problem.
Cluster information:
Kubernetes version:
server@ubuntu-server2:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:09:57Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}
Cloud being used: (put bare-metal if not on a public cloud)
Installation method: Install manual kubelet kubeadm and kubectl, swapoff -a net.ipv4.ip_forward=1 and edited daemon.json to use systemd in docker.
Host OS:
Master: installed in KVM as host, Ubuntu Server 20.04.3
Workder: Installed in Vagrant image with vmware provider
CNI and version
server@ubuntu-server2:~$ cat /etc/cni/net.d/10-flannel.conflist
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
CRI and version:
I’m sorry I can’t find how to look CRI version in Kubernetes.
This is status centos as workder join master and success:
[vagrant@centos ~]$ sudo kubeadm join 192.168.1.33:6443 --token v8dz3w.fm0np0v0k8u1vc2m --discovery-token-ca-cert-hash sha256:4e878f8ddb32c4468f55d00a80986c2d0a8d7ea2f748526f84774cfaf5d910d8
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
This kubelet run correctly in centos worker
[vagrant@centos ~]$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Wed 2021-12-29 02:01:16 UTC; 26min ago
Docs: https://kubernetes.io/docs/
Main PID: 5531 (kubelet)
Tasks: 17 (limit: 11398)
Memory: 44.4M
CGroup: /system.slice/kubelet.service
└─5531 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/co>
lines 1-11/11 (END)
This all node worker in master node
server@ubuntu-server2:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
centos NotReady <none> 38m v1.23.1
tumbleweed NotReady <none> 37h v1.23.1
ubuntu-cloud NotReady <none> 13d v1.23.0
ubuntu-server1 Ready <none> 13d v1.23.1
ubuntu-server2 Ready control-plane,master 13d v1.23.1
This is all pods running in master node
server@ubuntu-server2:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-example 1/1 Running 2 (45h ago) 11d
kube-system coredns-64897985d-5gls4 1/1 Running 28 (50m ago) 13d
kube-system coredns-64897985d-jz87n 1/1 Running 28 (50m ago) 13d
kube-system etcd-ubuntu-server2 1/1 Running 27 (50m ago) 13d
kube-system kube-apiserver-ubuntu-server2 1/1 Running 49 (45m ago) 13d
kube-system kube-controller-manager-ubuntu-server2 1/1 Running 60 (41m ago) 13d
kube-system kube-flannel-ds-qd6ff 0/1 Init:0/2 0 37h
kube-system kube-flannel-ds-rm8ld 1/1 Running 26 (50m ago) 13d
kube-system kube-flannel-ds-v62tl 1/1 Running 2 (12d ago) 13d
kube-system kube-flannel-ds-xjjjn 1/1 Running 4 (45h ago) 13d
kube-system kube-flannel-ds-zmghz 0/1 Init:0/2 0 38m
kube-system kube-proxy-4hkfk 1/1 Running 25 (50m ago) 13d
kube-system kube-proxy-dljq7 1/1 Running 4 (45h ago) 13d
kube-system kube-proxy-kc65w 1/1 Running 2 (12d ago) 13d
kube-system kube-proxy-lczmq 0/1 ContainerCreating 0 37h
kube-system kube-proxy-vq4p4 0/1 ContainerCreating 0 38m
kube-system kube-scheduler-ubuntu-server2 1/1 Running 40 (14h ago) 13d
This is describe node from centos worker
server@ubuntu-server2:~$ kubectl describe node centos
Name: centos
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=centos
kubernetes.io/os=linux
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 29 Dec 2021 01:55:26 +0000
Taints: node.kubernetes.io/not-ready:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: centos
AcquireTime: <unset>
RenewTime: Wed, 29 Dec 2021 02:45:21 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Wed, 29 Dec 2021 02:42:20 +0000 Wed, 29 Dec 2021 02:01:33 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 29 Dec 2021 02:42:20 +0000 Wed, 29 Dec 2021 02:01:33 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 29 Dec 2021 02:42:20 +0000 Wed, 29 Dec 2021 02:01:33 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Wed, 29 Dec 2021 02:42:20 +0000 Wed, 29 Dec 2021 02:01:33 +0000 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
InternalIP: 192.168.247.138
Hostname: centos
Capacity:
cpu: 4
ephemeral-storage: 71645Mi
hugepages-2Mi: 0
memory: 1860064Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 67612704657
hugepages-2Mi: 0
memory: 1757664Ki
pods: 110
System Info:
Machine ID: 50242dc9967e48a9a7042ba49dd07240
System UUID: 28244d56-9217-9c13-f763-66e14fcfb9fb
Boot ID: e4b44501-37d9-4218-bf45-6d532cbaa1fc
Kernel Version: 4.18.0-348.2.1.el8_5.x86_64
OS Image: CentOS Linux 8
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://20.10.12
Kubelet Version: v1.23.1
Kube-Proxy Version: v1.23.1
PodCIDR: 10.244.4.0/24
PodCIDRs: 10.244.4.0/24
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system kube-flannel-ds-zmghz 100m (2%) 100m (2%) 50Mi (2%) 50Mi (2%) 50m
kube-system kube-proxy-vq4p4 0 (0%) 0 (0%) 0 (0%) 0 (0%) 50m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 100m (2%) 100m (2%)
memory 50Mi (2%) 50Mi (2%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 14h kubelet Starting kubelet.
Normal NodeAllocatableEnforced 14h kubelet Updated Node Allocatable limit across pods
Warning CheckLimitsForResolvConf 14h kubelet open /run/systemd/resolve/resolv.conf: no such file or directory
Warning Rebooted 14h kubelet Node centos has been rebooted, boot id: 4a03d149-c27f-4958-a25f-a8a2f6f7a542
Normal NodeNotReady 14h kubelet Node centos status is now: NodeNotReady
Warning ImageGCFailed 14h (x4 over 14h) kubelet failed to get imageFs info: non-existent label "docker-images"
Normal NodeNotSchedulable 14h kubelet Node centos status is now: NodeNotSchedulable
Normal NodeHasSufficientPID 14h (x6 over 14h) kubelet Node centos status is now: NodeHasSufficientPID
Normal NodeHasNoDiskPressure 14h (x6 over 14h) kubelet Node centos status is now: NodeHasNoDiskPressure
Normal NodeSchedulable 14h kubelet Node centos status is now: NodeSchedulable
Normal NodeHasSufficientMemory 14h (x7 over 14h) kubelet Node centos status is now: NodeHasSufficientMemory
Normal Starting 50m kubelet Starting kubelet.
Normal NodeHasSufficientMemory 50m (x2 over 50m) kubelet Node centos status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 50m (x2 over 50m) kubelet Node centos status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 50m (x2 over 50m) kubelet Node centos status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 50m kubelet Updated Node Allocatable limit across pods
Warning CheckLimitsForResolvConf 50m kubelet open /run/systemd/resolve/resolv.conf: no such file or directory
Normal NodeHasSufficientMemory 44m (x8 over 44m) kubelet Node centos status is now: NodeHasSufficientMemory
Thank You in Advance
Best regards,
Afiadi