GKE Private cluster blocking TLS handshake on port 9090 from inside pod

General Discussions
1

we just migrated from public cluster to private on GKE, we are facing a weird issue where a pod is unable to call an external API having https on port 9090. It becomes unresponsive and after 60 second throw Connection reset by peer.

We are using cloud NAT for enabling internet access for pods. We have tested it’s not a port blocking issue by temporary adding egress and ingress rules in the firewall to allow all traffic on every port.

We tested the ssl handshake using curl from inside pod by running following command:

curl -k -vvv --location 'https://103.xx.xx.xx:9090/v1/users/login' POST \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46QnJhbmR2ZXJzZUAxMjM=' \
--data-raw ''

It gets stuck on this response:

*   Trying 103.xx.xx.xx:9090...
* TCP_NODELAY set
* Connected to 103.xx.xx.xx (103.xx.xx.xx) port 9090 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):

While on any other service its working fine.

Would really appreciate any help or pointer in this regard. Thanks