I am trying to deploy 200 springboot dynamic pods (job object) from another springboot pod (deployment object). Each dynamic has CPU and Memory limits of 500m and 1.3Gb respectively.
A set of 10 pods will be created at a time and there will be a 2 min delay before creating another 10 pods. After creating around 100 pods, some of the dynamic pods (around 3 to 5 pods) failed to resolve the service dns of another pod and fails with jave.net.unknownhostexception. Can anyone please explain the reason for this behavior? Also how to address this issue?
NOTE: We never hit the unknownhostexception when we created 5 pods at a time with the same 2 min delay
Cluster information:
Kubernetes version: 1.23
Cloud being used: Bare-Metal VMs created from VMWare
Installation method: Air Gapped RKE2 Installation
Host OS: RHEL 8.6
CNI and version: caliclo - v3.22.1
CRI and version: v1.23.0