Hello everyone!
Need some help with my k8s deployment.
Maybe, I’m doing something totally wrong so, any advice regarding my whole setup also appreciated.
Cluster information:
Kubernetes version: v1.30.1
Cloud being used: bare-metal on VPS with public IPs
Installation method: kubeadm + Ansible
Host OS: Rocky Linux 9.4
CNI and version: flannel v0.25.1
CRI and version: containerd 1.7.16
Routing mode: IPVS
What I’m not using nor planning to:
- Cloud-based k8s provider (GKE, AWS, …)
- Cloud-based load balancer
I have 3 nodes (1 master and 2 slaves) with A DNS records at my provider’s zone as follows:
pub.name x.x.x.x (domain and almost every other subdomain for concrete service points to a master node's public IP)
mn.pub.name x.x.x.x (master)
sn1.pub.name y.y.y.y (slave 1)
sn2.pub.name z.z.z.z (slave 2)
Where pub.name - is my publicly available registered domain.
And x.x.x.x, y.y.y.y, z.z.z.z are public static IPs of respecting nodes. These IPs are not from the same subnet. Just some random public IPs assigned by provider on VPS creation.
I’m using IPv4 single stack; IPv6 is disabled on every node
Firewall is disabled. SELinux is disabled.
My goal - is to have a scalable k8s cluster based on VPS VMs with public IPs.
This cluster will be used for proof of concept purposes, so no production-grade requirements have to be met.
I want to have all the services to be available on a single IP address via corresponding DNS records.
My thought is that this single node handling all incoming requests should be master node. (Please, tell me if it’s better done otherwise)
I do not care for extra hop required to get from master node to a slave node hosting actual pod (POC).
Unfortunately for me, list of ports that supposed to be available on this public node goes beyond standart 80 & 443.
I would also need mail-related ports (25, 465, 587, …) for docker-mailserver installation
And a few others from system range (within 1000). Also, I definetely need ports 80 & 443 to be able to serve HTTP and HTTPS ingress traffic.
What I understand from documentation (correct me if I’m wrong here) - is that to have system ports publicly available, I have to use load balancer.
Across load balancers, MetalLB seemed to suite my use-case the best.
To use MetalLB, I have to deploy my k8s cluster in IPVS mode.
The problem is: once cluster works in IPVS mode, whenever I deploy any service with external IP, the node that is hosting pod corresponding to that service, goes into “NotReady” state. And ping from that node to master node fails.
It doesn’t matter if I’m using MetalLB to assign external IP to a service, or just configure externalIP in values.yaml for Helm installation of that service.
Here are some configs:
IPAddressPool resource definition for MetalLB:
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: ipap-public
namespace: lb
spec:
addresses:
- 'x.x.x.x/32'
L2Adverdisement resource definition for MetalLB:
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: l2a-public
namespace: lb
spec:
ipAddressPools:
- ipap-public
Pods after installing docker-mailserver (All pods were in “Ready” status before installation):
kubectl get po -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cert manager-cert-manager-cainjector-5996484b89-2jvw9 1/1 Terminating 2 (3h40m ago) 46h 10.244.4.38 sn2.pub.name <none> <none>
cert manager-cert-manager-cainjector-5996484b89-xsqxw 0/1 Pending 0 3h11m <none> <none> <none> <none>
cert manager-cert-manager-controller-77b697fdc8-8kj9l 0/1 Pending 0 3h11m <none> <none> <none> <none>
cert manager-cert-manager-controller-77b697fdc8-s7pcx 1/1 Terminating 2 (3h40m ago) 46h 10.244.4.34 sn2.pub.name <none> <none>
cert manager-cert-manager-webhook-b4d6fb4db-6fst7 0/1 Pending 0 3h11m <none> <none> <none> <none>
cert manager-cert-manager-webhook-b4d6fb4db-xhlhm 1/1 Terminating 2 (3h40m ago) 46h 10.244.4.35 sn2.pub.name <none> <none>
ingress nginx-ingress-nginx-controller-548c97b899-fd5rc 1/1 Terminating 2 (3h40m ago) 46h 10.244.4.37 sn2.pub.name <none> <none>
ingress nginx-ingress-nginx-controller-548c97b899-h759b 0/1 Pending 0 3h11m <none> <none> <none> <none>
kube-flannel kube-flannel-ds-h6c7p 1/1 Running 7 (3h40m ago) 8d x.x.x.x mn.pub.name <none> <none>
kube-flannel kube-flannel-ds-pf2tn 1/1 Running 120 (3h39m ago) 8d z.z.z.z sn2.pub.name <none> <none>
kube-flannel kube-flannel-ds-wqlwm 1/1 Running 248 (3h30m ago) 8d y.y.y.y sn1.pub.name <none> <none>
kube-system coredns-7db6d8ff4d-bs8sl 1/1 Running 7 (3h40m ago) 8d 10.244.0.17 mn.pub.name <none> <none>
kube-system coredns-7db6d8ff4d-cdczf 1/1 Running 7 (3h40m ago) 8d 10.244.0.16 mn.pub.name <none> <none>
kube-system etcd-mn.pub.name 1/1 Running 7 (3h40m ago) 8d x.x.x.x mn.pub.name <none> <none>
kube-system kube-apiserver-mn.pub.name 1/1 Running 7 (3h40m ago) 8d x.x.x.x mn.pub.name <none> <none>
kube-system kube-controller-manager-mn.pub.name 1/1 Running 7 (3h40m ago) 8d x.x.x.x mn.pub.name <none> <none>
kube-system kube-proxy-9gghz 1/1 Running 9 (3h40m ago) 8d y.y.y.y sn1.pub.name <none> <none>
kube-system kube-proxy-jx765 1/1 Running 7 (3h40m ago) 8d x.x.x.x mn.pub.name <none> <none>
kube-system kube-proxy-rzxhl 1/1 Running 8 (3h40m ago) 8d z.z.z.z sn2.pub.name <none> <none>
kube-system kube-scheduler-mn.pub.name 1/1 Running 7 (3h40m ago) 8d x.x.x.x mn.pub.name <none> <none>
lb metal-metallb-controller-586d8ff949-fscs5 0/1 Pending 0 3h11m <none> <none> <none> <none>
lb metal-metallb-speaker-js29q 4/4 Running 12 (3h40m ago) 3d4h x.x.x.x mn.pub.name <none> <none>
lb metal-metallb-speaker-rchjk 4/4 Running 211 (3h31m ago) 3d4h y.y.y.y sn1.pub.name <none> <none>
lb metal-metallb-speaker-wwzsh 4/4 Running 16 (3h40m ago) 3d4h z.z.z.z sn2.pub.name <none> <none>
mail server-docker-mailserver-dc6c85597-5bl8k 0/1 Pending 0 3h11m <none> <none> <none> <none>
mail server-docker-mailserver-dc6c85597-wq4qb 0/1 Terminating 0 3h17m <none> sn1.pub.name <none> <none>
Nodes after installing docker-mailserver (nodes were all ready before installation):
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
mn.pub.name Ready control-plane 8d v1.30.1 x.x.x.x <none> Rocky Linux 9.4 (Blue Onyx) 5.14.0-427.16.1.el9_4.x86_64 containerd://1.7.16
sn1.pub.name NotReady <none> 8d v1.30.1 y.y.y.y <none> Rocky Linux 9.4 (Blue Onyx) 5.14.0-427.16.1.el9_4.x86_64 containerd://1.7.16
sn2.pub.name NotReady <none> 8d v1.30.1 z.z.z.z <none> Rocky Linux 9.4 (Blue Onyx) 5.14.0-427.16.1.el9_4.x86_64 containerd://1.7.16
Services after installing docker-mailserver:
kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
AGE
cert manager-cert-manager-controller-metrics ClusterIP 10.96.2.46 <none> 9402/TCP
45h
cert manager-cert-manager-webhook ClusterIP 10.96.9.119 <none> 443/TCP
45h
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP
8d
ingress nginx-ingress-nginx-controller ClusterIP 10.96.8.89 <none> 80/TCP,443/TCP
2d
ingress nginx-ingress-nginx-controller-admission ClusterIP 10.96.15.151 <none> 443/TCP
2d
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP
8d
lb metallb-webhook-service ClusterIP 10.96.4.83 <none> 443/TCP
3d3h
mail server-docker-mailserver LoadBalancer 10.96.7.14 x.x.x.x 25:32646/TCP,465:32674/TCP,587:31297/TCP,10465:32246/TCP,10587:32643/TCP,143:31931/TCP,993:30564/TCP,10143:32139/TCP,10993:30131/TCP,11334:30402/TCP 108m
Ping & arping result from slave node 1 to master after installation:
[root@sn1 ~]# ping -c 4 mn.pub.name
PING mn.pub.name (x.x.x.x) 56(84) bytes of data.
From mn.pub.name (x.x.x.x) icmp_seq=1 Destination Port Unreachable
From mn.pub.name (x.x.x.x) icmp_seq=2 Destination Port Unreachable
From mn.pub.name (x.x.x.x) icmp_seq=3 Destination Port Unreachable
From mn.pub.name (x.x.x.x) icmp_seq=4 Destination Port Unreachable
--- mn.pub.name ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3004ms
[root@sn1 ~]# arping -c 4 -I eth0 x.x.x.x
ARPING x.x.x.x from y.y.y.y eth0
Unicast reply from x.x.x.x [FE:54:00:90:C0:0A] 44.181ms
Unicast reply from x.x.x.x [FE:54:00:90:C0:0A] 0.759ms
Unicast reply from x.x.x.x [FE:54:00:90:C0:0A] 0.679ms
Unicast reply from x.x.x.x [FE:54:00:90:C0:0A] 0.846ms
Sent 4 probes (1 broadcast(s))
Received 4 response(s)
IPVS routes:
ipvsadm -l
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP mn.pub.name:smtp rr
TCP mn.pub.name:imap rr
TCP mn.pub.name:urd rr
TCP mn.pub.name:submission rr
TCP mn.pub.name:imaps rr
TCP mn.pub.name:10143 rr
TCP mn.pub.name:10465 rr
TCP mn.pub.name:10587 rr
TCP mn.pub.name:10993 rr
TCP mn.pub.name:11334 rr
TCP mn.pub.name:30131 rr
TCP mn.pub.name:30402 rr
TCP mn.pub.name:30564 rr
TCP mn.pub.name:31297 rr
TCP mn.pub.name:31931 rr
TCP mn.pub.name:32139 rr
TCP mn.pub.name:32246 rr
TCP mn.pub.name:32643 rr
TCP mn.pub.name:32646 rr
TCP mn.pub.name:32674 rr
TCP mn.pub.name:https rr
-> mn.pub.name:sun-sr-https Masq 1 4 0
TCP mn.pub.name:domain rr
-> 10.244.0.16:domain Masq 1 0 0
-> 10.244.0.17:domain Masq 1 0 0
TCP mn.pub.name:9153 rr
-> 10.244.0.16:9153 Masq 1 0 0
-> 10.244.0.17:9153 Masq 1 0 0
TCP mn.pub.name:sec-pc2fax-srv rr
TCP mn.pub.name:https rr
TCP mn.pub.name:smtp rr
TCP mn.pub.name:imap rr
TCP mn.pub.name:urd rr
TCP mn.pub.name:submission rr
TCP mn.pub.name:imaps rr
TCP mn.pub.name:10143 rr
TCP mn.pub.name:10465 rr
TCP mn.pub.name:10587 rr
TCP mn.pub.name:10993 rr
TCP mn.pub.name:11334 rr
TCP mn.pub.name:http rr
TCP mn.pub.name:https rr
TCP mn.pub.name:https rr
TCP mn.pub.name:https rr
TCP mn.pub.name:30131 rr
TCP mn.pub.name:30402 rr
TCP mn.pub.name:30564 rr
TCP mn.pub.name:31297 rr
TCP mn.pub.name:31931 rr
TCP mn.pub.name:32139 rr
TCP mn.pub.name:32246 rr
TCP mn.pub.name:32643 rr
TCP mn.pub.name:32646 rr
TCP mn.pub.name:32674 rr
TCP mn.pub.name:30131 rr
TCP mn.pub.name:30402 rr
TCP mn.pub.name:30564 rr
TCP mn.pub.name:31297 rr
TCP mn.pub.name:31931 rr
TCP mn.pub.name:32139 rr
TCP mn.pub.name:32246 rr
TCP mn.pub.name:32643 rr
TCP mn.pub.name:32646 rr
TCP mn.pub.name:32674 rr
UDP mn.pub.name:domain rr
-> 10.244.0.16:domain Masq 1 0 0
-> 10.244.0.17:domain Masq 1 0 0
As soon as I’m uninstalling docker-mailserver, things got back to normal (nodes, pods, ping, …)
This page says I have to provide some IP addresses to MetalLB IPAddressPool other than public IPs assigned by my VPS provider. Like it is done in IPAddressPool resource manifest where spec.addresses is set to ['203.0.113.10-203.0.113.15']
But, provider’s support refuses to give me more than one IP per node.
Tried to post to Slack channels metallb and #ingress-nginx-users
No solution so far.
Any advices, suggestions?