Issue with pod communication with each other

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version:1.30
Cloud being used: put bare-metal
Installation method: manual
Host OS: 9.4
CNI and version: calico 3.28.1
CRI and version: 1.31.0
Got repos for k8s and cri-o from CRI-O is moving towards pkgs.k8s.io | Kubernetes

I have setup a base setup, one controller and one worker node using the above. I have added argo-cd and a pod for curlimages/curl. I can curl argocd’s pod IP from the hosts, but I cannot do it from the curl pod. This came up because nginx was not able to reach it the first time I built it out.

kubectl exec -it test-pod-same-node -n default – curl -kL http://192.168.43.137:8080
curl: (7) Failed to connect to 192.168.43.137 port 8080 after 1 ms: Could not connect to server
command terminated with exit code

routing table on the worker node

[root@k8w-d1 ~]$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default jhl-1248-4500xb 0.0.0.0 UG 100 0 0 ens192
10.76.8.0 0.0.0.0 255.255.255.0 U 100 0 0 ens192
10.85.0.0 0.0.0.0 255.255.0.0 U 0 0 0 cni0
192.168.43.128 0.0.0.0 255.255.255.192 U 0 0 0 *
192.168.43.129 0.0.0.0 255.255.255.255 UH 0 0 0 calic138a17461b
192.168.43.130 0.0.0.0 255.255.255.255 UH 0 0 0 cali8b8bdf9b628
192.168.43.131 0.0.0.0 255.255.255.255 UH 0 0 0 cali97594500b58
192.168.43.132 0.0.0.0 255.255.255.255 UH 0 0 0 calib64380c8e12
192.168.43.133 0.0.0.0 255.255.255.255 UH 0 0 0 calic9e21022839
192.168.43.134 0.0.0.0 255.255.255.255 UH 0 0 0 cali086fb0b769e
192.168.43.135 0.0.0.0 255.255.255.255 UH 0 0 0 cali193ebe3a88d
192.168.43.136 0.0.0.0 255.255.255.255 UH 0 0 0 calib119896d2fc
192.168.43.137 0.0.0.0 255.255.255.255 UH 0 0 0 calie52c0757531
192.168.43.138 0.0.0.0 255.255.255.255 UH 0 0 0 cali61a9554ce92
192.168.43.139 0.0.0.0 255.255.255.255 UH 0 0 0 calia27d62e6b0a
192.168.82.128 k8c-d1.lib.ncsu 255.255.255.192 UG 0 0 0 ens192

argocd argocd-server-7bc7b97977-5mthc 1/1 Running 0 31m 192.168.43.137 k8w-d1
argocd test-pod-same-node 1/1 Running 0 16m 192.168.43.140 k8w-d1
NAMESPACE NAME READY STATUS RESTARTS AGE
argocd argocd-application-controller-0 1/1 Running 0 83m
argocd argocd-applicationset-controller-744b76d7fd-kqnlt 1/1 Running 0 83m
argocd argocd-dex-server-5bf5dbc64d-scv89 1/1 Running 0 83m
argocd argocd-notifications-controller-84f5bf6896-wl72f 1/1 Running 0 83m
argocd argocd-redis-74b8999f94-fdwlw 1/1 Running 0 83m
argocd argocd-repo-server-57f4899557-mxggp 1/1 Running 0 83m
argocd argocd-server-7bc7b97977-5mthc 1/1 Running 0 83m
argocd test-pod-same-node 1/1 Running 1 (7m27s ago) 67m
calico-apiserver calico-apiserver-655699488c-7pljf 1/1 Running 0 83m
calico-apiserver calico-apiserver-655699488c-nv8hk 1/1 Running 0 83m
calico-system calico-kube-controllers-6b9567d8f4-6kxhc 1/1 Running 0 84m
calico-system calico-node-gdvtg 1/1 Running 0 84m
calico-system calico-node-j2dfw 1/1 Running 0 84m
calico-system calico-typha-697b54fdc-7ngg6 1/1 Running 0 84m
calico-system csi-node-driver-85wnj 2/2 Running 0 84m
calico-system csi-node-driver-8vrmx 2/2 Running 0 84m
kube-system coredns-55cb58b774-5k4l7 1/1 Running 0 85m
kube-system coredns-55cb58b774-9sgm8 1/1 Running 0 85m
kube-system etcd-k8c-d1.lib.ncsu.edu 1/1 Running 1 86m
kube-system kube-apiserver-k8c-d1.lib.ncsu.edu 1/1 Running 3 (97m ago) 86m
kube-system kube-controller-manager-k8c-d1.lib.ncsu.edu 1/1 Running 0 86m
kube-system kube-proxy-lx8rv 1/1 Running 0 85m
kube-system kube-proxy-wsxq4 1/1 Running 0 85m
kube-system kube-scheduler-k8c-d1.lib.ncsu.edu 1/1 Running 3 86m
tigera-operator tigera-operator-77f994b5bb-dkvxx 1/1 Running 0 85m

do you mean you can get success result when you do which one from following from host?
curl -kL http://192.168.43.137:8080 or
when you do

curl -kL http://192.168.43.137