Kubernetes behind pfSense

General Discussions
1

Cluster information:

Kubernetes version: 1.15.3 kubernetes
Cloud being used: ( bare-metal)
Installation method: KubeSpray
Host OS: Proxmox hypervisor and CoreOS virtual machine
CNI and version: Calico
CRI and version: Docker

Hello,
I am trying to be able to access some of my service in the outside world, but I am stuck on this part
My Kubernetes cluster was created following those steps (I am using a Bare metal server)

  • I install an hypervisor (Proxmox)
  • Create 4 virtual machine for Kubernetes (2 master, 2 nodes on CoreOS), and 1 pfSense
  • I provisioned everything using KubeSpray
  • Install metalLB as load balancer
    apiVersion: v1
    kind: ConfigMap
    metadata:
      namespace: metallb-system
      name: config
    data:
      config: |
        address-pools:
        - name: default
          protocol: layer2
          addresses:
          - 192.168.9.240/28
  • I installed nginx-ingress

So far so good, I have an external ip address but since the Kubernetes cluster is behind a pfSense, I have an internal ip address which is 192.168.9.240.
I would like to be able to enter a subdomain name and access some of my services. For example I type myservice.domainName.com and be redirect to a specific service like an url for example 192.168.9.240/myService.
I was thinking that maybe I should use a reverse proxy on the host ? But I am not sure. Any recommandations ? Even a tutorial could help me

Thank you in advance

1 Like
2

i’m stuck at the same spot. I can ping the ingress from the external world but unable to access http port. I thought HAproxy is necessary but after reading pfsense’ docs i changed my mind since i had one master.
I guess the problem lies with NAT forwarding, when i find a solution I’ll notify you. Best of luck!

3

I actually solved my problem since a long time ago and completely forgot to reply here.
I was pretty new to pfSense and networking by then :sweat_smile: and I changed a lot of things from then. BUT like you said it was just a NAT problem actually…

At first I did something quite complicated by installaing haproxy in pfSense and it was working. Later as I get a better understanding of networking, I just removed haproxy and added two simple nat rules from wan to the ip address of metalLB. On the port 80 et 443.

Hope it will help

EDIT1: Of course it will work if you pfSense has your public ip address otherwise I am not so sure how to do it, I would say probably with more NAT rules

2 Likes
4

Yesterday i installed HAproxy in pfsense and nothing worked haha. Guess i’m on your steps, i never installed metalLB ( quite new to the technology ). I’m using OVH and exposing pfsense with a Fail Over IP since i’m trying to accomplish a CARP cluster in a later stage. I’m pretty stuck at DNS resolution, idk how to make pfsense route the ingresses i create back to the kuberenetes API, that’s what led me to install HAproxy.
I’m also running everything in VMs on proxmox as yourself. I’ll try your setup. Thank you!

5

Ok please note my networking knowledge is far from great, I did not what was CARP before today :slight_smile:

First of all I am not using Proxmox anymore (I am using libvrit with Terraform) but when I was still using it I managed to install k8s on it and route things properly. MetalLB is quite simple to use, if I am not wrong you will need to have an external ip to access your stuff outside of your k8s cluster,

So I will explain clearly what I did, I install metalLB and nginx-ingress, please keep in my mind my pfSense has my pubic IP address.

For MetalLB I cnfigure it in Layer 2 like so (see here https://metallb.universe.tf/configuration/#layer-2-configuration)

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.9.240-192.168.9.250

After this I got an ip address in the range 192.168.9.240-192.168.9.250 so let’s say 192.168.9.241. Notice that this range of ip address is a network which pfSense has access to.

After I created ingress rules, check that my ingress get the external ip address provided by MetalLB, checked I can access the service inside the k8s cluster using the metalLB ip address.

Then I created a NAT rule in pfsense from my WAN to the ip address of metalLB
so the routing will be something like that
WAN => pfSense => MetalLB IP adress => Nginx ingress (domain for example)

I dunno if it’s enough clear

6

Just the part about ingress rules. For everything else i have the same setup.

7

You need an ingress rule otherwise, your routing will stop at MetalLB. MetalLB just create an external ip address, I don’t think it does something else

After I believe it’s not an obligation to create ingress rule to test, but just for the sake of example

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx-internal" => I have two ingress if you only have one you do not need to precise this
  labels:
    app.kubernetes.io/name: deluge
    app.kubernetes.io/managed-by: kubectl
    app.kubernetes.io/component: server
  name: deluge
  namespace: seedbox
spec:
  rules:
  - host: deluge..domainName.com
    http:
      paths:
      - backend:
          serviceName: deluge
          servicePort: 8112
status:
  loadBalancer: {}

So if I complete the routing stuff it would be as follow

WAN => pfSense => MetalLB IP adress => Nginx ingress => get the domain name => go he correct service

Maybe try to check the logs of MetalLB + nginx ingress.

Are you sure you did the NAT properly ?
Maybe also try without fail over at the beginning

8

Thank you for the quick replies. I now have a more general understanding of what is going on. There are broken pieces in my mind about certain parts of this. Guess i shouldn’t jump between things and better understand each component as you did.
Since you had this going on for you for a while, one last question. Do you like it? what are the cons of having such a setup?
As for me the reason i wanted to use pfsense is that in production we only have HAproxy in front of the cluster.

I just wanted to have PFsense in front of everything for security reasons.
since you told me you switched to something else, i wanted to know your opinion on this.

9

So I did this in a homelab to learn about k8s + pfSense.

I did not change too many thing just:

  • using libvrit with terraform
  • and moving my cluster from a kimsufi server to a NUC
  • use k3s instead of k8s

I am still using pfSense and I would quite recommend it as a firewall even for personal use, it’s quite good and there are a lot of resources. I find it hard mainly because my networking knowledge is still kind of poor
Now about k8s, I would say it’s another story, it’s quite fun for personal use and I learned a lot and keep learning. Now for professional use , I think calculating the cost it’s quite complex. However I can maybe give a few advises:

  • Be sure to already have a bit of devOps culutre before moving to k8s (usage of container, automatic CI/CD, ect…)
  • If you only have a few project with a few container and do not deploy often, I would see k8s could be overkill. A Docker compose would be preferable
  • if you have a lot of containers and need to push in production a lot, it’s a perfect tool
  • it’s gonna break at the beginning since it’s the learning curve is really steep
  • Don’t forget to set resourceQuotas and limits (great resource https://learnk8s.io/production-best-practices)
  • As this video is funny and may look stupid, it will tell you a few things to not do https://www.youtube.com/watch?v=9wvEwPLcLcA.

Feel free to pm if you any other question, good luck to you, have fun :smiley:

EDIT: Where I work a french company we have a pfSense in front of almost everything, it will clearly improve your security but I find it hard if no knowledge in networking otherwise it’s quite a breeze to use if you already got good networking knowledge

1 Like
10

Haha i enjoyed the video, guess that’s why you used terraform xD
I experimented with k3s myself and wrote a few articles on kubernetes https://kyouuma.blogspot.com/
If you’re interested in a carp setup i have a tutorial for that as well.
As you said, the biggest challenge lies in networking. I’ve been doing devops for a year now and used kubernetes/compose in production but never had the chance to build anything from the ground up to the application layer.
That’s why i committed to doing it with pfsense.
I also work for a french company, they all like PFsense a lot :stuck_out_tongue:

1 Like