Validating Admission Policy enabling cluster goes to unavailable

Cluster information:

Kubernetes version: v1.29.1
Cloud being used: bare-metal
Installation method: kubeadm
Host OS: Centos 7
CNI and version: Calico
CRI and version: containerd (1.6.27)

Hi Team,

I am Trying to enable ValidatingAdmissionPolicy by adding the below parameter in kube-api-server.yaml file.

- --feature-gates=ValidatingAdmissionPolicy=true

But the cluster unavailable for few minutes getting an error as

E0130 15:01:47.062134   21663 memcache.go:265] couldn't get current server API group list: Get "https://x.x.x.x:6443/api?timeout=32s": dial tcp x.x.x.x:6443: connect: connection refused
E0130 15:01:47.065644   21663 memcache.go:265] couldn't get current server API group list: Get "https://x.x.x.x:6443/api?timeout=32s": dial tcp x.x.x.x:6443: connect: connection refused
E0130 15:01:47.069175   21663 memcache.go:265] couldn't get current server API group list: Get "https://x.x.x.x:6443/api?timeout=32s": dial tcp x.x.x.x:6443: connect: connection refused
The connection to the server x.x.x.x:6443 was refused - did you specify the right host or port?```

after few minutes cluster available. then after few minutes goes to unbailable. it was repeating. 

any solution ?