Migrated VMs that has Kubernetes from AWS to Azure, but kube-apiserver pod is getting failed

General Discussions
, , ,
1

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version:v1.30.8
Cloud being used: (put bare-metal if not on a public cloud) Azure
Installation method: migration of vm from aws to azure
Host OS: redhat8
CNI and version:“0.3.1”
CRI and version: v1 1.7.25
[root@master01 ~]# kubectl get nodes
E0123 19:00:34.243852 5093 memcache.go:265] couldn’t get current server API group list: Get “https://10.5.0.4:6443/api?timeout=32s”: dial tcp 10.5.0.4:6443: connect: connection refused
E0123 19:00:34.244360 5093 memcache.go:265] couldn’t get current server API group list: Get “https://10.5.0.4:6443/api?timeout=32s”: dial tcp 10.5.0.4:6443: connect: connection refused
E0123 19:00:34.245667 5093 memcache.go:265] couldn’t get current server API group list: Get “https://10.5.0.4:6443/api?timeout=32s”: dial tcp 10.5.0.4:6443: connect: connection refused
E0123 19:00:34.245912 5093 memcache.go:265] couldn’t get current server API group list: Get “https://10.5.0.4:6443/api?timeout=32s”: dial tcp 10.5.0.4:6443: connect: connection refused
E0123 19:00:34.247181 5093 memcache.go:265] couldn’t get current server API group list: Get “https://10.5.0.4:6443/api?timeout=32s”: dial tcp 10.5.0.4:6443: connect: connection refused
The connection to the server 10.5.0.4:6443 was refused - did you specify the right host or port?

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.

2

Dear Experts,

on the azure k8s VMs we see kubelet running but there are many errors to it.

Jan 22 14:24:18 master01 kubelet[12829]: E0122 14:24:18.988185 12829 event.go:368] “Unable to write event (may retry after sleeping)” err=“Post "https://10.5.0.4:6443/api/v1/namespaces/kube-system/events": dial tcp 10.5.0.4:6443: connect: connection refused” event=“&Event{ObjectMeta:{kube-apiserver-master01.example.com.181d060d9b5189fb kube-system 0 0001-01-01 00:00:00 +0000 UTC map map },InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-master01.example.com,UID:fe0249daaade4f842dd70878a4f06674,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{kube-apiserver},},Reason:Pulled,Message:Container image "registry.k8s.io/kube-apiserver:v1.30.8" already present on machine,Source:EventSource{Component:kubelet,Host:master01.example.com,},FirstTimestamp:2025-01-22 13:14:46.745033211 +0000 UTC m=+0.547731254,LastTimestamp:2025-01-22 13:14:46.745033211 +0000 UTC m=+0.547731254,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:master01.example.com,}”
Jan 22 14:24:26 master01 kubelet[12829]: E0122 14:24:26.342072 12829 pod_workers.go:1298] “Error syncing pod, skipping” err=“failed to "StartContainer" for "kube-apiserver" with CrashLoopBackOff: "back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master01.example.com_kube-system(fe0249daaade4f842dd70878a4f06674)"” pod=“kube-system/kube-apiserver-master01.example.com” podUID="fe0249daaade4f842dd70878a4f06674
[root@master01 ~]# kubectl config view
apiVersion: v1
clusters:

  • cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://10.5.0.4:6443
    name: kubernetes
    contexts:
  • context:
    cluster: kubernetes
    user: kubernetes-admin
    name: kubernetes-admin@kubernetes
    current-context: kubernetes-admin@kubernetes
    kind: Config
    preferences: {}
    users:
  • name: kubernetes-admin
    user:
    client-certificate-data: DATA+OMITTED
    client-key-data: DATA+OMITTED
    [root@master01 ~]#

I am a beginner to kubernetes.Could you please post your suggestions.

3

The kube-apiserver is pointed to 10.5.0.4 in azure,but still it is pointing to AWS IP as 172.16.1.146

[root@master01 manifests]# netstat -ntplu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:40719 0.0.0.0:* LISTEN 1215/containerd
tcp 0 0 127.0.0.1:10257 0.0.0.0:* LISTEN 2181/kube-controlle
tcp 0 0 127.0.0.1:10259 0.0.0.0:* LISTEN 2123/kube-scheduler
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1453/sshd
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 7342/kubelet
tcp6 0 0 :::22 :::* LISTEN 1453/sshd
tcp6 0 0 :::10250 :::* LISTEN 7342/kubelet
tcp6 19 0 :::6443 :::* LISTEN 7434/kube-apiserver
udp 0 0 127.0.0.1:323 0.0.0.0:* 980/chronyd
udp6 0 0 ::1:323 :::* 980/chronyd
[root@master01 manifests]# ps -ef |grep -i 7434/
root 7510 2402 0 18:24 pts/0 00:00:00 grep --color=auto -i 7434/
[root@master01 manifests]# ps -ef |grep -i 7434
root 7434 1941 5 18:24 ? 00:00:00 kube-apiserver --advertise-address=172.16.1.146 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root 7512 2402 0 18:24 pts/0 00:00:00 grep --color=auto -i 7434
[root@master01 manifests]#