# LoadBalancer service not working **URL:** \ **Category:** General Discussions\ **Created:** [March 24, 2019, 1:43am UTC](https://discuss.kubernetes.io/t/loadbalancer-service-not-working/5614 "2019-03-24T01:43:49Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![marcelk](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/marcelk/32/2433_2.png) [@marcelk](https://discuss.kubernetes.io/u/marcelk)\ **Post date:** [March 24, 2019, 1:43am UTC](https://discuss.kubernetes.io/t/loadbalancer-service-not-working/5614/1 "2019-03-24T01:43:50Z")
I’m trying to figure out why my service (type: LoadBalancer) isn’t working. This is running on AWS. I followed the steps in [Debug Services - Kubernetes](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/) but can’t find any smoking gun. In particular, when ssh’ing into a cluster node, the following all work: `nslookup ` `curl :` `nslookup ` kube-proxy is running `curl localhost:` What doesn’t work: `curl :`: Empty reply from server `curl :`: Empty reply from server I have another service running in the system that works just fine (meaning `curl :` returns something). I looked at the iptables entries for both services, but there doesn’t appear to be anything obvious missing for the non-working service. (The one thing that didn’t agree with what [Debug Services - Kubernetes](https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/) expects is the content of /etc/resolve.conf: domain us-west-2.compute.internal search us-west-2.compute.internal nameserver 172.20.0.2 However, this doesn’t appear to impact the working service.) Can anyone suggest some next steps to get to the heart of the problem? ---
**Author:** ![rata](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/rata/32/18244_2.png) [@rata](https://discuss.kubernetes.io/u/rata)\ **Post date:** [March 24, 2019, 5:04am UTC](https://discuss.kubernetes.io/t/loadbalancer-service-not-working/5614/2 "2019-03-24T05:04:52Z")
What is the problem you see? Is a load balancer created on AWS (like browsing the AWS console)? Is there any event in the objects (svc)? If you have the load balancer on AWS created, see the listener it is using (in it’s config). Is it listening in which port? And to which port is forwards traffic to the kubernetes nodes? Is that the port listed in the service as nodePort? Do the kubernetes nodes accept traffic from the security group the load balancer is using? And if so, the security group the load balancer is using, accepts traffic from the internet in the ports it is listening? I think one of those should be causing this issue, if I had to bet. But please try them all, and report back if it works or it doesn’t 🙂 ---
**Author:** ![marcelk](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/marcelk/32/2433_2.png) [@marcelk](https://discuss.kubernetes.io/u/marcelk)\ **Post date:** [March 25, 2019, 4:23am UTC](https://discuss.kubernetes.io/t/loadbalancer-service-not-working/5614/3 "2019-03-25T04:23:22Z")
Hi Rodrigo, thanks for your response. The one thing I noticed is that all of the instances for the broken service’s load balancer are listed as ‘out of service’ (and - not surprisingly - for the healthy service they are ‘in service’), so I need to dig into that. ---
**Author:** ![rata](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/rata/32/18244_2.png) [@rata](https://discuss.kubernetes.io/u/rata)\ **Post date:** [March 25, 2019, 11:16pm UTC](https://discuss.kubernetes.io/t/loadbalancer-service-not-working/5614/4 "2019-03-25T23:16:02Z")
Great. And which port is using for the health check? Is it using the nodePort of the service? If it is, then my next bet would be that the security group the load balancer is using can’t connect to the workers on that node. So, I’d make sure to accept incoming connections from the security group the workers are running. Can you check that? 🙂 ---
**Author:** ![marcelk](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/marcelk/32/2433_2.png) [@marcelk](https://discuss.kubernetes.io/u/marcelk)\ **Post date:** [March 26, 2019, 12:52am UTC](https://discuss.kubernetes.io/t/loadbalancer-service-not-working/5614/5 "2019-03-26T00:52:38Z")
Hi Rodrigo, your guess was correct, the health check was configured incorrectly (in this case, the nodeport it was using wasn’t serving anything). When I manually reconfigured the load balancer to use a different port, one that’s actually listened on, it worked. Thanks for your help! A follow-on question: I noticed that the load balancers have _all_ ec2 instances that are port of the entire cluster registered. In my specific case that is undesirable because the (single) pod that’s backing the service can only run on a single node (via a node selector). Was it a conscious choice to register all cluster nodes with every load balancer, even when the backing pods can only run on a subset of the nodes? ---
**Author:** ![rata](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/rata/32/18244_2.png) [@rata](https://discuss.kubernetes.io/u/rata)\ **Post date:** [March 26, 2019, 1:46am UTC](https://discuss.kubernetes.io/t/loadbalancer-service-not-working/5614/6 "2019-03-26T01:46:00Z")
Glad it worked! Yes, registering all is intended. The thing is the following: The ELB does not know about pods, it knows about VMs instances. So, you can only register instances and not pods. Then, how do you distribute traffic evenly across pods? One option is to register all nodes as backends and then let kubernetes do the load balancing between pods (as kubernetes does know about pods). This is basically what is happening, and the most common setup. The nodePort, as you seen the iptables rules, load balances between pods. If you have a good reason to not want that (but I would recommend really having a good reason), you can change the behavior by changing the service external Traffic Policy setting to LocalOnly. If you do that, the health check will fail on all nodes except on the ones that are running a pod for the application. That way, you will see only one node as healthy. But, then, if the pod is scheduled to some other node you need to wait for that node to become healthy for the AWS load balancer (with it’s params for healthy checking) and might introduce downtime and those not wanted stuff. IMHO, it’s better to let kubernetes do the load balancing for pods, as it knows about pods. Amazon ELB doesn’t, so it’s not the best layer for the true load balancing. ---
**Author:** ![mohibk](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mohibk](https://discuss.kubernetes.io/u/mohibk)\ **Post date:** [August 4, 2019, 2:14pm UTC](https://discuss.kubernetes.io/t/loadbalancer-service-not-working/5614/7 "2019-08-04T14:14:29Z")
Can you please tell how to let Kubernetes do the loadbalacning for pods as you mentioned above? Does it do it by-default if you expose a service like in the tutorial [Hello Minikube - Kubernetes](https://kubernetes.io/docs/tutorials/hello-minikube/#create-a-service) using ```auto kubectl expose deployment hello-node --type=LoadBalancer --port=8080 ``` Would be helpfull if you can explain as I want loadbalancing across pods by kubernetes itself. ---
**Author:** ![rata](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/rata/32/18244_2.png) [@rata](https://discuss.kubernetes.io/u/rata)\ **Post date:** [August 4, 2019, 7:44pm UTC](https://discuss.kubernetes.io/t/loadbalancer-service-not-working/5614/8 "2019-08-04T19:44:07Z")
Yes, it is done when you expose the service like that. No need to do anything special 🙂 ---
**Author:** ![rata](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/rata/32/18244_2.png) [@rata](https://discuss.kubernetes.io/u/rata)\ **Post date:** [August 6, 2019, 8:48am UTC](https://discuss.kubernetes.io/t/loadbalancer-service-not-working/5614/10 "2019-08-06T08:48:20Z")
Yes, it happens. Load balancing is done using iptables rules (if using iptables for kube-proxy, if using something else, then something else). Basically, when a request goes to a service IP, iptables routes it to one of the pods using a probability (so, if you have 2 pods, 50% of the times, more or less, will go to one and 50% to another). Just normal iptables rules. Maybe the service documentation in the kubernetes website expands on this? Have you checked that out? ---
**Author:** ![mohibk](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mohibk](https://discuss.kubernetes.io/u/mohibk)\ **Post date:** [August 6, 2019, 2:13pm UTC](https://discuss.kubernetes.io/t/loadbalancer-service-not-working/5614/11 "2019-08-06T14:13:59Z")
@rata Thanks. I read the Services document which also describes the same. It says ‘By default, kube-proxy in iptables mode chooses a backend at random… **If kube-proxy is running in iptables mode and the first Pod that’s selected does not respond, the connection fails**. This is different from userspace mode: in that scenario, kube-proxy would detect that the connection to the first Pod had failed and would automatically retry with a different backend Pod.’ Looks like above is the cause that one of my request fails? ---
**Author:** ![rata](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/rata/32/18244_2.png) [@rata](https://discuss.kubernetes.io/u/rata)\ **Post date:** [August 6, 2019, 4:56pm UTC](https://discuss.kubernetes.io/t/loadbalancer-service-not-working/5614/12 "2019-08-06T16:56:16Z")
Not sure about which problem you are talking about. It’s another thread? Will look later 🙂 ---
**Author:** ![mohibk](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mohibk](https://discuss.kubernetes.io/u/mohibk)\ **Post date:** [August 6, 2019, 5:29pm UTC](https://discuss.kubernetes.io/t/loadbalancer-service-not-working/5614/13 "2019-08-06T17:29:04Z")
Sorry for mixing two things. Yes it is the another thread that i reffered here [Loadbalancing across pods in a service using minikube](https://discuss.kubernetes.io/t/loadbalancing-across-pods-in-a-service-using-minikube/7430/5) .I shall put this comment there for reference.