Trying to understand how to use the traefik ingress controllers on multiple servers in my home lab

General Discussions
1

I am new to Kubernetes in general. I have installed K3s on two servers (I know it’s not optimal to have an even number of servers), and am trying to get a first basic nginx web service to work.

I have gone through configuring the traefik helm chart, and am now struggling because the one replica prevents the requests to arrive half of the time (because the second IP doesn’t reply).

Here are my traefik values:

additionalArguments:
  - --certificatesresolvers.generic.acme.email=admin@[redacted]
  - --certificatesresolvers.generic.acme.caServer=https://acme-v02.api.letsencrypt.org/directory
  - --certificatesresolvers.generic.acme.httpChallenge.entryPoint=web
  - --certificatesresolvers.generic.acme.storage=/ssl-certs/acme-generic.json

logs:
  general:
    level: ERROR

ports:
  web:
    redirectTo:
      port: websecure
  websecure:
    tls:
      enabled: true
      certResolver: generic

ingressRoute:
  dashboard:
    enabled: false

persistence:
  enabled: true
  name: ssl-certs
  size: 1Gi
  path: /ssl-certs

deployment:
  initContainers:
    # Related issue: https://github.com/containous/traefik/issues/6972
    - name: volume-permissions
      image: busybox:1.36.1
      command: ["sh", "-c", "chmod -Rv 600 /ssl-certs/* || true"]
      volumeMounts:
        - name: ssl-certs
          mountPath: /ssl-certs

ingressClass:
  enabled: true
  isDefaultClass: true

I have also made an nginx deployment, service and ingress. This is boilerplate and irrelevant here, just know that I have used a domain that points to a reverse proxy balancing the requests between the two servers.

Here is what kubectl get all outputs:

NAME                           READY   STATUS    RESTARTS   AGE
pod/nginx-f7599d4c-4pvp8       1/1     Running   0          33m
pod/nginx-f7599d4c-lp68p       1/1     Running   0          33m
pod/traefik-7b87f44b56-p9vmg   1/1     Running   0          24m

NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP                   PORT(S)                      AGE
service/kubernetes   ClusterIP      10.43.0.1       <none>                        443/TCP                      172m
service/nginx        ClusterIP      10.43.149.101   <none>                        80/TCP                       33m
service/traefik      LoadBalancer   10.43.75.107    192.168.1.150,192.168.1.151   80:31880/TCP,443:31609/TCP   38m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx     2/2     2            2           33m
deployment.apps/traefik   1/1     1            1           38m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-f7599d4c       2         2         2       33m
replicaset.apps/traefik-5bcb455b96   0         0         0       38m
replicaset.apps/traefik-7b87f44b56   1         1         1       24m
replicaset.apps/traefik-db875d49b    0         0         0       26

(note the two external IP’s for the traefik service.)

From the load balancer server, sending a request to the first server (192.168.1.150) returns moved permanently as can be expected, but the other one just times out. I am thinking this is because there is only one traefik replica on one server, and the other one has nowhere to route the requests.

So here is my question: how can I tell traefik to deploy exactly one replica per host, so that requests can be handled as intended? Or should I make the traefik service use a single external IP?