Cluster information:
Kubernetes version: 1.28.9
Cloud being used: Azure Cloud
Installation method: Kubeadm
Host OS: ubuntu-server-jammy 22.04-lts-gen2
CNI and version: Calico v3.27.3
CRI and version: cri-dockerd v0.3.13
Hello everyone,
I am training myself on the administration of a Kubernetes cluster. I specify I begin my Kubernetes adventure. For more precision on what I try to set up this is my architecture.
After initializing my Kubernetes cluster with Kubeadm I install Calico v3.27.3 then I install an Azure Cloud Controller Manager v1.29.0 out-of-tree. I install this CCM so that from my cluster we can create Azure services, an External Load Balancer for example.
After that, I install Nginx Ingress Controller with Helm like this :
helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace --set rbac.create=true --set controller.stats.enabled=true --set controller.metrics.enabled=true --set controller.service.externalTrafficPolicy="Local" --set controller.service.loadBalancerIP="Client Public IP Adress" --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-resource-group"="Resource-Group-of-the-Client-Public-IP-Adress" --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-pip-name"="Name-of-the-Client-Public-IP-Adress" --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer"="true"
At this point, installing the Nginx Ingress Controller creates a LoadBalancer service in the cluster, but that service still remains in “pending” mode for the EXTERNAL-IP. I’m going to check the LoadBalancer service logs and I have this error:
LAST SEEN TYPE REASON OBJECT MESSAGE
4m59s (x8 over 15m) Normal EnsuringLoadBalancer Service/ingress-nginx-controller Ensuring load balancer
4m59s (x8 over 15m) Warning SyncLoadBalancerFailed Service/ingress-nginx-controller Error syncing load balancer: failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {\r
"error": {\r
"code": "InvalidResourceReference",\r
"message": "Resource /subscriptions/************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_client_front_ip referenced by resource /subscriptions/************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/outboundRules/OutboundRuleForAllVMs was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.",\r
"details": []\r
}\r
}
The problem is that I have a outbound rule called OutboundRuleForAllVMs in my External Azure Load Balancer’s configuration.
Then I went to see the logs of my CCM and I have this:
I0427 17:14:53.693462 1 controller.go:398] Ensuring load balancer for service ingress-nginx/ingress-nginx-controller
I0427 17:14:53.694142 1 controller.go:954] Adding finalizer to service ingress-nginx/ingress-nginx-controller
I0427 17:14:53.695482 1 event.go:376] "Event occurred" object="ingress-nginx/ingress-nginx-controller" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0427 17:14:53.750477 1 azure_loadbalancer.go:128] reconcileService: Start reconciling Service "ingress-nginx/ingress-nginx-controller" with its resource basename "a0471feda47784f7e8ca995b940658b6"
I0427 17:14:53.750569 1 azure_loadbalancer.go:1625] reconcileLoadBalancer for service(ingress-nginx/ingress-nginx-controller) - wantLb(true): started
I0427 17:14:53.840890 1 azure_loadbalancer_repo.go:73] LoadBalancerClient.List(clou-902-kubequest-infra-rg) success
I0427 17:14:53.903396 1 azure_loadbalancer_repo.go:73] LoadBalancerClient.List(clou-902-kubequest-infra-rg) success
I0427 17:14:53.979339 1 azure_loadbalancer.go:857] get(ingress-nginx/ingress-nginx-controller): lb(kubequest_lb) - found frontend IP config, primary service: false
I0427 17:14:53.979453 1 azure_loadbalancer.go:883] getServiceLoadBalancerStatus gets ingress IP "Client Public IP Adress" from frontendIPConfiguration "kubequest_lb_client_front_ip" for service "ingress-nginx/ingress-nginx-controller"
I0427 17:14:53.979494 1 azure_loadbalancer.go:662] getServiceLoadBalancer(ingress-nginx-controller, kubernetes, true): current lb IPs: ["Client Public IP Adress"]
I0427 17:14:53.979527 1 azure_loadbalancer.go:383] shouldChangeLoadBalancer(ingress-nginx-controller, kubequest_lb, kubernetes): change the LB to another one kubernetes
I0427 17:14:53.979563 1 azure_privatelinkservice.go:69] reconcilePrivateLinkService for service(ingress-nginx/ingress-nginx-controller) - LB fipConfigID(kubequest_lb_client_front_ip) - wantPLS(false) - createPLS(false)
I0427 17:14:54.042063 1 azure_privatelinkservice_repo.go:127] No privateLinkService found for frontendIPConfig "/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_client_front_ip" in rg "clou-902-kubequest-infra-rg"
I0427 17:14:54.042108 1 azure_privatelinkservice.go:180] reconcilePrivateLinkService for service(ingress-nginx/ingress-nginx-controller) finished
I0427 17:14:54.042144 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.06252547 request="services_ensure_privatelinkservice_deleted" resource_group="clou-902-kubequest-infra-rg" subscription_id="***********************" source="ingress-nginx/ingress-nginx-controller" result_code="succeeded"
I0427 17:14:54.042164 1 azure_loadbalancer.go:483] removeFrontendIPConfigurationFromLoadBalancer(kubequest_lb, ["kubequest_lb_client_front_ip"], kubernetes, ingress-nginx-controller): updating the load balancer
W0427 17:14:54.224796 1 azure_loadbalancer_repo.go:150] LoadBalancerClient.CreateOrUpdate(kubequest_lb) failed: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {
"error": {
"code": "InvalidResourceReference",
"message": "Resource /subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_client_front_ip referenced by resource /subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/outboundRules/OutboundRuleForAllVMs was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.",
"details": []
}
}, LoadBalancer request: {"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb","location":"francecentral","properties":{"backendAddressPools":[{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/backendAddressPools/kubequest_lb_admin_backend_pool","name":"kubequest_lb_admin_backend_pool","properties":{"loadBalancerBackendAddresses":[{"name":"18a0fdee-61d2-4bc7-b993-f6195563e869","properties":{}},{"name":"05585bf2-3788-4434-8836-b24b4ba1a68a","properties":{}},{"name":"de8c119d-60af-43df-b202-13a854c254f6","properties":{}}]}},{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/backendAddressPools/kubequest_lb_client_backend_pool","name":"kubequest_lb_client_backend_pool","properties":{"loadBalancerBackendAddresses":[{"name":"be755a8c-661e-4e82-a085-fad109b60f04","properties":{}},{"name":"86792ee6-3885-49b3-bddb-72dc12a5fb5a","properties":{}}]}}],"frontendIPConfigurations":[{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_admin_front_ip","name":"kubequest_lb_admin_front_ip","properties":{"privateIPAllocationMethod":"Dynamic","publicIPAddress":{"id":"/subscriptions/***********************/resourceGroups/kubequest_rg_ip/providers/Microsoft.Network/publicIPAddresses/kubequest_ip_1"}}}],"inboundNatPools":[],"inboundNatRules":[{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/inboundNatRules/kubequest_lb_kubeapiendpoint_rule","name":"kubequest_lb_kubeapiendpoint_rule","properties":{"backendPort":6443,"enableFloatingIP":false,"enableTcpReset":true,"frontendIPConfiguration":{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_admin_front_ip"},"frontendPort":6443,"idleTimeoutInMinutes":5,"protocol":"Tcp"}},{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/inboundNatRules/kubequest_lb_ssh_rules","name":"kubequest_lb_ssh_rules","properties":{"backendAddressPool":{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/backendAddressPools/kubequest_lb_admin_backend_pool"},"backendPort":22,"enableFloatingIP":false,"enableTcpReset":true,"frontendIPConfiguration":{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_admin_front_ip"},"frontendPort":0,"frontendPortRangeEnd":510,"frontendPortRangeStart":500,"idleTimeoutInMinutes":4,"protocol":"Tcp"}}],"loadBalancingRules":[{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/loadBalancingRules/kubequest_lb_80_rule","name":"kubequest_lb_80_rule","properties":{"backendAddressPool":{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/backendAddressPools/kubequest_lb_client_backend_pool"},"backendAddressPools":[{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/backendAddressPools/kubequest_lb_client_backend_pool"}],"backendPort":80,"disableOutboundSnat":true,"enableFloatingIP":false,"enableTcpReset":true,"frontendIPConfiguration":{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_client_front_ip"},"frontendPort":80,"idleTimeoutInMinutes":4,"loadDistribution":"SourceIPProtocol","probe":{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/probes/kubequest_lb_probe_80"},"protocol":"Tcp"}},{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/loadBalancingRules/kubequest_lb_443_rule","name":"kubequest_lb_443_rule","properties":{"backendAddressPool":{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/backendAddressPools/kubequest_lb_client_backend_pool"},"backendAddressPools":[{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/backendAddressPools/kubequest_lb_client_backend_pool"}],"backendPort":443,"disableOutboundSnat":true,"enableFloatingIP":false,"enableTcpReset":true,"frontendIPConfiguration":{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_client_front_ip"},"frontendPort":443,"idleTimeoutInMinutes":4,"loadDistribution":"SourceIPProtocol","probe":{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/probes/kubequest_lb_probe_443"},"protocol":"Tcp"}}],"outboundRules":[{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/outboundRules/OutboundRuleForAllVMs","name":"OutboundRuleForAllVMs","properties":{"allocatedOutboundPorts":3192,"backendAddressPool":{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/backendAddressPools/kubequest_lb_admin_backend_pool"},"enableTcpReset":true,"frontendIPConfigurations":[{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_client_front_ip"}],"idleTimeoutInMinutes":4,"protocol":"All"}}],"probes":[{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/probes/kubequest_lb_probe_80","name":"kubequest_lb_probe_80","properties":{"intervalInSeconds":5,"numberOfProbes":2,"port":80,"probeThreshold":1,"protocol":"Tcp"}},{"id":"/subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/probes/kubequest_lb_probe_443","name":"kubequest_lb_probe_443","properties":{"intervalInSeconds":5,"numberOfProbes":2,"port":443,"probeThreshold":1,"protocol":"Tcp"}}]},"sku":{"name":"Standard","tier":"Regional"},"tags":{"ApplicationID":"calculatrice","Author":"michael.ranivo@epitech.eu","BackupPolicy":"No-Backup","Country":"FR","Environment":"POC","Project":"Kubequest","Version":"1"}}
E0427 17:14:54.224834 1 azure_loadbalancer.go:486] removeFrontendIPConfigurationFromLoadBalancer(kubequest_lb, ["kubequest_lb_client_front_ip"], kubernetes, ingress-nginx-controller): failed to CreateOrUpdateLB: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {
"error": {
"code": "InvalidResourceReference",
"message": "Resource /subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_client_front_ip referenced by resource /subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/outboundRules/OutboundRuleForAllVMs was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.",
"details": []
}
}
E0427 17:14:54.224906 1 azure_loadbalancer.go:678] getServiceLoadBalancer(ingress-nginx-controller, kubernetes, true): failed to remove frontend IP configurations ["kubequest_lb_client_front_ip"] from load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {
"error": {
"code": "InvalidResourceReference",
"message": "Resource /subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_client_front_ip referenced by resource /subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/outboundRules/OutboundRuleForAllVMs was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.",
"details": []
}
}
E0427 17:14:54.224947 1 azure_loadbalancer.go:1650] reconcileLoadBalancer: failed to get load balancer for service "ingress-nginx/ingress-nginx-controller", error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {
"error": {
"code": "InvalidResourceReference",
"message": "Resource /subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_client_front_ip referenced by resource /subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/outboundRules/OutboundRuleForAllVMs was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.",
"details": []
}
}
E0427 17:14:54.224996 1 azure_loadbalancer.go:132] reconcileLoadBalancer(ingress-nginx/ingress-nginx-controller) failed: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {
"error": {
"code": "InvalidResourceReference",
"message": "Resource /subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_client_front_ip referenced by resource /subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/outboundRules/OutboundRuleForAllVMs was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.",
"details": []
}
}
I0427 17:14:54.225069 1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.474559584 request="services_ensure_loadbalancer" resource_group="clou-902-kubequest-infra-rg" subscription_id="***********************" source="ingress-nginx/ingress-nginx-controller" result_code="failed_ensure_loadbalancer"
I0427 17:14:54.225131 1 controller.go:887] Finished syncing service "ingress-nginx/ingress-nginx-controller" (531.691696ms)
I0427 17:14:54.225468 1 event.go:376] "Event occurred" object="ingress-nginx/ingress-nginx-controller" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message=<
Error syncing load balancer: failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {
"error": {
"code": "InvalidResourceReference",
"message": "Resource /subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_client_front_ip referenced by resource /subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/outboundRules/OutboundRuleForAllVMs was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.",
"details": []
}
}
>
E0427 17:14:54.225486 1 controller.go:298] error processing service ingress-nginx/ingress-nginx-controller (retrying with exponential backoff): failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {
"error": {
"code": "InvalidResourceReference",
"message": "Resource /subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/frontendIPConfigurations/kubequest_lb_client_front_ip referenced by resource /subscriptions/***********************/resourceGroups/clou-902-kubequest-infra-rg/providers/Microsoft.Network/loadBalancers/kubequest_lb/outboundRules/OutboundRuleForAllVMs was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.",
"details": []
}
}
I tried to assign the Public IP Address Client to the External Azure Load Balancer as a front-end IP address and remove all the External Azure Load Balancer rules that concern this IP address, so that the CCM can create the rules it needs for the cluster, but when I restart the installation of the Nginx Ingress Controller the CCM removes the IP address from the External Azure Load Balancer and it creates a new External Azure Load Balancer with IP address and new rules.
I don’t know if I’m doing well for the synchronization between the LoadBalancer service of the Nginx Ingress Controller and the External Azure Load Balancer. I would like to force the CCM to use the External Azure Load Balancer I created instead of it creating a new External Azure Load Balancer.
Is it possible to synchronize a LoadBalancer service with an External Load Balancer?
Thanks to those who read me, I know it was a bit long.
Thank you in advance to those who will respond to my message.