APi Server is showing "Liveness probe failed: HTTP probe failed with statuscode: 500" while describing the pod

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: 1.26.9
Cloud being used: (put bare-metal if not on a public cloud) : bare-metal
Installation method: kubeadm
Host OS: ubuntu
CNI and version: flannel - 0.3.1
CRI and version: containerd - 1.6.21

Hi Team,
I’ve just upgraded my k8s cluster from v1.25.4 to v1.26.9 using the commands mentioned here

Everything (be it applications, pods etc) are working fine.
Except for the master node components. There pods are running fine but showing
“Startup probe failed with status code 403”
“Readiness probe failed with status code 500”
“Liveness probe failed with status code 500”

etcd also not returning any member list.

When I checked the apiserver endpoints, it is returning everything as ok

curl -k https://localhost:6443/readyz?verbose
[+]ping ok
[+]log ok
[+]etcd ok
[+]etcd-readiness ok
[+]informer-sync ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok
[+]poststarthook/priority-and-fairness-filter ok
[+]poststarthook/storage-object-count-tracker-hook ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/priority-and-fairness-config-producer ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/start-kube-apiserver-identity-lease-controller ok
[+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok
[+]poststarthook/start-legacy-token-tracking-controller ok
[+]poststarthook/aggregator-reload-proxy-client-cert ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
[+]poststarthook/apiservice-openapiv3-controller ok
[+]shutdown ok
readyz check passed

Could someone please help in fixing the issue.
Please let me know if any additional information is required from my end.

Best,
Ankit Kaushik