# Kubernetes behind pfSense **URL:** \ **Category:** General Discussions\ **Created:** [September 16, 2019, 8:20am UTC](https://discuss.kubernetes.io/t/kubernetes-behind-pfsense/7941 "2019-09-16T08:20:28Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![sylflo](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/sylflo/32/3693_2.png) [@sylflo](https://discuss.kubernetes.io/u/sylflo)\ **Post date:** [September 16, 2019, 8:20am UTC](https://discuss.kubernetes.io/t/kubernetes-behind-pfsense/7941/1 "2019-09-16T08:20:28Z")
### Cluster information: Kubernetes version: 1.15.3 kubernetes Cloud being used: ( bare-metal) Installation method: KubeSpray Host OS: Proxmox hypervisor and CoreOS virtual machine CNI and version: Calico CRI and version: Docker Hello, I am trying to be able to access some of my service in the outside world, but I am stuck on this part My Kubernetes cluster was created following those steps (I am using a Bare metal server) - I install an hypervisor (Proxmox) - Create 4 virtual machine for Kubernetes (2 master, 2 nodes on CoreOS), and 1 pfSense - I provisioned everything using KubeSpray - Install metalLB as load balancer ```auto apiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system name: config data: config: | address-pools: - name: default protocol: layer2 addresses: - 192.168.9.240/28 ``` - I installed nginx-ingress So far so good, I have an external ip address but since the Kubernetes cluster is behind a pfSense, I have an internal ip address which is 192.168.9.240. I would like to be able to enter a subdomain name and access some of my services. For example I type [myservice.domainName.com](http://myservice.domainName.com) and be redirect to a specific service like an url for example 192.168.9.240/myService. I was thinking that maybe I should use a reverse proxy on the host ? But I am not sure. Any recommandations ? Even a tutorial could help me Thank you in advance ---
**Author:** ![Kyouuma](https://avatars.discourse-cdn.com/v4/letter/k/4da419/32.png) [@Kyouuma](https://discuss.kubernetes.io/u/Kyouuma)\ **Post date:** [May 6, 2020, 8:30pm UTC](https://discuss.kubernetes.io/t/kubernetes-behind-pfsense/7941/2 "2020-05-06T20:30:58Z")
i’m stuck at the same spot. I can ping the ingress from the external world but unable to access http port. I thought HAproxy is necessary but after reading pfsense’ docs i changed my mind since i had one master. I guess the problem lies with NAT forwarding, when i find a solution I’ll notify you. Best of luck! ---
**Author:** ![sylflo](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/sylflo/32/3693_2.png) [@sylflo](https://discuss.kubernetes.io/u/sylflo)\ **Post date:** [May 7, 2020, 8:20am UTC](https://discuss.kubernetes.io/t/kubernetes-behind-pfsense/7941/3 "2020-05-07T08:20:16Z")
I actually solved my problem since a long time ago and completely forgot to reply here. I was pretty new to pfSense and networking by then 😅 and I changed a lot of things from then. BUT like you said it was just a NAT problem actually… At first I did something quite complicated by installaing haproxy in pfSense and it was working. Later as I get a better understanding of networking, I just removed haproxy and added two simple nat rules from wan to the ip address of metalLB. On the port 80 et 443. Hope it will help EDIT1: Of course it will work if you pfSense has your public ip address otherwise I am not so sure how to do it, I would say probably with more NAT rules ---
**Author:** ![Kyouuma](https://avatars.discourse-cdn.com/v4/letter/k/4da419/32.png) [@Kyouuma](https://discuss.kubernetes.io/u/Kyouuma)\ **Post date:** [May 7, 2020, 12:26pm UTC](https://discuss.kubernetes.io/t/kubernetes-behind-pfsense/7941/4 "2020-05-07T12:26:57Z")
Yesterday i installed HAproxy in pfsense and nothing worked haha. Guess i’m on your steps, i never installed metalLB ( quite new to the technology ). I’m using OVH and exposing pfsense with a Fail Over IP since i’m trying to accomplish a CARP cluster in a later stage. I’m pretty stuck at DNS resolution, idk how to make pfsense route the ingresses i create back to the kuberenetes API, that’s what led me to install HAproxy. I’m also running everything in VMs on proxmox as yourself. I’ll try your setup. Thank you! ---
**Author:** ![sylflo](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/sylflo/32/3693_2.png) [@sylflo](https://discuss.kubernetes.io/u/sylflo)\ **Post date:** [May 7, 2020, 1:10pm UTC](https://discuss.kubernetes.io/t/kubernetes-behind-pfsense/7941/5 "2020-05-07T13:10:40Z")
Ok please note my networking knowledge is far from great, I did not what was CARP before today 🙂 First of all I am not using Proxmox anymore (I am using libvrit with Terraform) but when I was still using it I managed to install k8s on it and route things properly. MetalLB is quite simple to use, if I am not wrong you will need to have an external ip to access your stuff outside of your k8s cluster, So I will explain clearly what I did, I install metalLB and nginx-ingress, please keep in my mind my pfSense has my pubic IP address. For MetalLB I cnfigure it in Layer 2 like so (see here [https://metallb.universe.tf/configuration/#layer-2-configuration](https://metallb.universe.tf/configuration/#layer-2-configuration)) ```auto apiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system name: config data: config: | address-pools: - name: default protocol: layer2 addresses: - 192.168.9.240-192.168.9.250 ``` After this I got an ip address in the range 192.168.9.240-192.168.9.250 so let’s say 192.168.9.241. Notice that this range of ip address is a network which pfSense has access to. After I created ingress rules, check that my ingress get the external ip address provided by MetalLB, checked I can access the service inside the k8s cluster using the metalLB ip address. Then I created a NAT rule in pfsense from my WAN to the ip address of metalLB so the routing will be something like that WAN =\> pfSense =\> MetalLB IP adress =\> Nginx ingress (domain for example) I dunno if it’s enough clear ---
**Author:** ![Kyouuma](https://avatars.discourse-cdn.com/v4/letter/k/4da419/32.png) [@Kyouuma](https://discuss.kubernetes.io/u/Kyouuma)\ **Post date:** [May 7, 2020, 1:17pm UTC](https://discuss.kubernetes.io/t/kubernetes-behind-pfsense/7941/6 "2020-05-07T13:17:07Z")
Just the part about ingress rules. For everything else i have the same setup. ---
**Author:** ![sylflo](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/sylflo/32/3693_2.png) [@sylflo](https://discuss.kubernetes.io/u/sylflo)\ **Post date:** [May 7, 2020, 1:22pm UTC](https://discuss.kubernetes.io/t/kubernetes-behind-pfsense/7941/7 "2020-05-07T13:22:47Z")
You need an ingress rule otherwise, your routing will stop at MetalLB. MetalLB just create an external ip address, I don’t think it does something else After I believe it’s not an obligation to create ingress rule to test, but just for the sake of example ```auto apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: "nginx-internal" => I have two ingress if you only have one you do not need to precise this labels: app.kubernetes.io/name: deluge app.kubernetes.io/managed-by: kubectl app.kubernetes.io/component: server name: deluge namespace: seedbox spec: rules: - host: deluge..domainName.com http: paths: - backend: serviceName: deluge servicePort: 8112 status: loadBalancer: {} ``` So if I complete the routing stuff it would be as follow WAN =\> pfSense =\> MetalLB IP adress =\> Nginx ingress =\> get the domain name =\> go he correct service Maybe try to check the logs of MetalLB + nginx ingress. Are you sure you did the NAT properly ? Maybe also try without fail over at the beginning ---
**Author:** ![Kyouuma](https://avatars.discourse-cdn.com/v4/letter/k/4da419/32.png) [@Kyouuma](https://discuss.kubernetes.io/u/Kyouuma)\ **Post date:** [May 7, 2020, 1:33pm UTC](https://discuss.kubernetes.io/t/kubernetes-behind-pfsense/7941/8 "2020-05-07T13:33:04Z")
Thank you for the quick replies. I now have a more general understanding of what is going on. There are broken pieces in my mind about certain parts of this. Guess i shouldn’t jump between things and better understand each component as you did. Since you had this going on for you for a while, one last question. Do you like it? what are the cons of having such a setup? As for me the reason i wanted to use pfsense is that in production we only have HAproxy in front of the cluster. I just wanted to have PFsense in front of everything for security reasons. since you told me you switched to something else, i wanted to know your opinion on this. ---
**Author:** ![sylflo](https://sea2.discourse-cdn.com/flex016/user_avatar/discuss.kubernetes.io/sylflo/32/3693_2.png) [@sylflo](https://discuss.kubernetes.io/u/sylflo)\ **Post date:** [May 7, 2020, 1:47pm UTC](https://discuss.kubernetes.io/t/kubernetes-behind-pfsense/7941/9 "2020-05-07T13:47:34Z")
So I did this in a homelab to learn about k8s + pfSense. I did not change too many thing just: - using libvrit with terraform - and moving my cluster from a kimsufi server to a NUC - use k3s instead of k8s I am still using pfSense and I would quite recommend it as a firewall even for personal use, it’s quite good and there are a lot of resources. I find it hard mainly because my networking knowledge is still kind of poor Now about k8s, I would say it’s another story, it’s quite fun for personal use and I learned a lot and keep learning. Now for professional use , I think calculating the cost it’s quite complex. However I can maybe give a few advises: - Be sure to already have a bit of devOps culutre before moving to k8s (usage of container, automatic CI/CD, ect…) - If you only have a few project with a few container and do not deploy often, I would see k8s could be overkill. A Docker compose would be preferable - if you have a lot of containers and need to push in production a lot, it’s a perfect tool - it’s gonna break at the beginning since it’s the learning curve is really steep - Don’t forget to set resourceQuotas and limits (great resource [https://learnk8s.io/production-best-practices](https://learnk8s.io/production-best-practices)) - As this video is funny and may look stupid, it will tell you a few things to not do [https://www.youtube.com/watch?v=9wvEwPLcLcA](https://www.youtube.com/watch?v=9wvEwPLcLcA). Feel free to pm if you any other question, good luck to you, have fun 😃 EDIT: Where I work a french company we have a pfSense in front of almost everything, it will clearly improve your security but I find it hard if no knowledge in networking otherwise it’s quite a breeze to use if you already got good networking knowledge ---
**Author:** ![Kyouuma](https://avatars.discourse-cdn.com/v4/letter/k/4da419/32.png) [@Kyouuma](https://discuss.kubernetes.io/u/Kyouuma)\ **Post date:** [May 7, 2020, 2:14pm UTC](https://discuss.kubernetes.io/t/kubernetes-behind-pfsense/7941/10 "2020-05-07T14:14:26Z")
Haha i enjoyed the video, guess that’s why you used terraform xD I experimented with k3s myself and wrote a few articles on kubernetes [https://kyouuma.blogspot.com/](https://kyouuma.blogspot.com/) If you’re interested in a carp setup i have a tutorial for that as well. As you said, the biggest challenge lies in networking. I’ve been doing devops for a year now and used kubernetes/compose in production but never had the chance to build anything from the ground up to the application layer. That’s why i committed to doing it with pfsense. I also work for a french company, they all like PFsense a lot 😛