Post 408 through aws elb to nginx with uwsgi

General Discussions
1
### Cluster information:
  • Kubernetes version:1.14.0
  • Cloud being used: (put bare-metal if not on a public cloud) AWS
  • Installation method: KOps v1.11.1
  • Host OS: Ubuntu 16.04
  • CNI and version: calico v3
  • CRI and version:

I'm running an application in a kops generated cluster, the app containers have an nginx reverse proxy to uwsgi. From outside the cluster, I have no issues performing any get requests, but if I send a post with data and content length header, I get intermittent failures. Inside the cluster, I can run the same post to the service endpoint with no issues. I can also run the post from a node to the exposed localhost port with no issues, but it fails from another node outside the cluster directly to that cluster's nodes exposed port.

I have tried the aws-alb-controller, the nginx-ingress-controller, and just a load-balancer service. All experience the same issue. It seems like packets are getting dropped somewhere when they enter the cluster network or somehow the request is getting messed up. The only time I can get everything working from outside the cluster is if I only have one node and pod.

Additional info, I am running KubeDNS, a private topology (entire cluster is in private subnets, but the elb is public).

Any help would be immensely appreciated. I've been stuck on this bug for 4 days now with little improvement. If more info is needed, I will gladly provide what I can.

Thanks

2

I built a cluster on EKS and deployed our app to it and did not experience the same issue. EKS uses a different networking layer, etcd, or a number of kubernetes things. At some point, I will test building a cluster with kops and other networking layers to see if there is an issue with calico and AWS.

3

Alternate networking overlays experienced the same issue