Local kubernetes cluster network problem/config

General Discussions
1

Cluster information:

Kubernetes version: v1.24.2
Cloud being used: bare-metal
Installation method: kubeadm
Host OS: ubuntu 22.04
CNI and version: calico 0.3.1
CRI and version: containerd 1.6.6

I installed via kubeadm: 3x control panels, 3x workers, external etcd, HA via external haproxy.

My network:
ISP router (192.168.1.1) →
PfSense VLANs (10.0.10.1, 10.0.50.1, …) →
K8s Cluster (10.0.50.31…10.0.50.36) →
Cluster network (10.0.80.0)

Installed metallb as LB, traefik as ingress controller.

I deployed a test nginx:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: nginx
  name: nginx-deploy-main
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx-main
  template:
    metadata:
      labels:
        run: nginx-main
    spec:
      containers:
      - image: nginx
        name: nginx
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: nginx
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`nginx.example.com`)
      kind: Rule
      services:
        - name: nginx-deploy-main
          port: 80
  tls:
    certResolver: letsencrypt

I can curl the nginx from inside the network (the certificate is properly created).
I am pointing the domain (nginx.example.com - changed to real domain) to the ISP router’s IP via cloudflare, but I can’t access it from outside, the traffic seems to stop before reaching the service.

I should have proprely port-forwarded HTTP and HTTPS traffic from both the ISP router and the pfSense to traefik.
Should I port-forward to the single service instead?

Thank you,

Thanks,

2

How do you have metallb configured? Is the IP range allocated to it routable on your host network?

3

The network (10.0.50.0) is a vlan managed by pfsense.
PfSense has dhcp enabled on a different range from metallb’s range.
All other devices/machines on that network are visible, it’s the same network the k8s hosts are.

How do I see of that particular range is routable or not?

Traefik Loadbalancer IP is 10.0.50.100, on the same network the cluster hosts are, for example.

Thanks,

4

Can pfsense curl or nc 10.0.50.100? by “inside the network” do you mean on k8s hosts? or other systems also on the same network?

5

Adding the service domain to pfsense pointing to traefik, all the network sees and makes curl of nginx.

I also modified metallb from L2 to BGP.

Still if I try to curl from outside the network (internet), going through cloudflare.
Cloudflare says it can’t reach the server.

So I think it must be the ISP router or pfSense blocking outside requests, not forwarding them.

Thing is I can use a vpn from outside, so some traffic is passed, I need to pass http/https.

6

Ok, managed to properly port-forward http/https traffic to traefik!

Thank you for your time, you actually pointed me in the right directions.