Kubernetes version: v1.26.0
Cloud being used: bare-metal
Installation method: k8s
Host OS: RHEL9
CNI and version: Calico v3.26.1
CRI and version: containerd v1.6.20
I’m designing a bare-metal cluster+statefulset application that I want to be able to roll out to clients with minimal changes regardless of their networking setup. The statefulset needs to receive data from the outside world, so I’ve written a simple service to allow incoming traffic:
Where spec.externalIPs is the load-balanced IP of the bare-metal cluster.
However, since each client network will assign my cluster a different IP address, this service config would require me to change the IP every time. Is there a way I can specify a dynamic external location using DNS, for example “clientname.cluster.local” and have the service still work?
The application is a SIEM that will be built internally and rolled out to client sites. SIEMs collect logs from endpoints, so they need an externally accessible network socket.
I’m trying to make a build process that is reliable, regardless of what IP the SIEM has when it’s installed on the client network (because it will change client-to-client and I want to reduce the number of changes required), so it makes sense to me to use DNS instead.
Is this possible to do? Is there a better solution for this?
So you are installing a cluster into each customer environment? Someone has to allocate an IP in whatever address space each customer wants to use, right?
What ‘externalIPs’ does (poorly) is tell nodes that if they receive traffic sent to that IP, they should send it to the Service. I don’t think that does all that you need.
So you are installing a cluster into each customer environment?
Yes (though we are building the cluster in house first and then migrating it across).
Someone has to allocate an IP in whatever address space each customer wants to use, right?
Yes, the client allocates an IP & DNS for the cluster and we use the DNS name where possible (e.g. kubeadm init). The upside of using DNS means we can build the cluster in-house and migrating becomes almost trivial. Hardcoding IP addresses which will change for each client makes this harder.
What ‘externalIPs’ does (poorly) is tell nodes that if they receive traffic sent to that IP, they should send it to the Service. I don’t think that does all that you need.
Yes, this is exactly what we want, and in our testing “externalIPs” is doing exactly what we need. The service directs the incoming traffic to a statefulset application as expected. I’m just looking for a way to make it more robust using DNS if possible.
If I’m misguided in using this tool, please advise a better one.
You cannot use DNS in the externalIPs field. You can maybe deploy a controller which reads the infromation from somewhere and updates the Service (the proverbial “another level of indirection”).
Are these bare-metal clusters? What does “migrating it across” mean?
You are setting type: LoadBalancer - is there a proper load-balancer controller running? Setting status.loadBalancer.ingress would be more “normal” than externalIPs.
You can maybe deploy a controller which reads the infromation from somewhere and updates the Service
Hmm. This seems like unnecessary complexity for a change that would only happen once per client. I’d be better off sed-ing all the files which use the IP before kubectl create-ing them.
Are these bare-metal clusters?
Yes, as mentioned in my opening post, it is a bare-metal cluster. More specifically, it is multiple servers forming a cluster and using a virtual IP (via keepalived and haproxy) to distribute incoming traffic. There is no dedicated load balancing server in front of them.
What does “migrating it across” mean?
We unplug the servers from our server room and network, transport them to the client’s data centre, and plug them in.
is there a proper load-balancer controller running?
No there isn’t. What does a load-balancer controller do and why should I want one?
I will say there is not much that is “normal” about our use-case, so just because it’s convention doesn’t mean we’d do it.
it is multiple servers forming a cluster and using a virtual IP (via keepalived and haproxy) to distribute incoming traffic. There is no dedicated load balancing server in front of them.
So, IIUC, this delivers a SINGLE address to one of the nodes in the cluster. How is this IP address provisioned and how does it find its way into keepalived config?
Where is haproxy running - as a pod in the cluster? Or as a system daemon (not managed by k8s)? Or something else?
is there a proper load-balancer controller running?
No there isn’t. What does a load-balancer controller do and why should I want one?
An LB controller basically does:
watch services
for each service create/update:
if service.type == LoadBalancer:
make sure an IP address is provisioned and routed (lots of nuance in there) to pods
make sure the service status.loadBalancer.ingress shows that IP
else
make sure any previously provisioned IP is torn down and released
make sure the service status.loadBalancer.ingress is clear
This would automate the provisioning and some last-mile routing (which you are doing manually).
If you don’t need more than one IP, then this is sort of academic and not super useful excpet to know how the thing you are doing by hand is “usually” done.
You can maybe deploy a controller which reads the infromation from somewhere and updates the Service
Hmm. This seems like unnecessary complexity for a change that would only happen once per client. I’d be better off sed-ing all the files which use the IP before kubectl create-ing them.
If you are confident it will never ever change, then you’re probably right.