Helm install Nginx Ingress with a defined scope

Hi all,
I read an article said that : by default, each Ingress controller will listen to all the ingress events from all the namespaces and add corresponding directives and rules into Nginx configuration file.
So I want to use
–set controller.scope.enabled=true
–set controller.scope.namespace=namespace1,namespace2
To limit ingress controller only listen to ingress events from specific namespaces, is it possible ? How can I set multi values for “controller.scope.namespace” ?

Seems like something that you can test out pretty quickly in a local Kubernetes cluster (like in Docker for Desktop, Minikube, or kind) and blow it away when you’re done. Being able to build a proof-of-concept (POCs) is a valuable skill. :slight_smile:

So another question you could pose is how do I run multiple nginx ingress controllers?. That documentation poses some useful information on how the Nginx Ingress Controller works.

As for helm chart specifics, the helm landscape is pretty wartorn right now. Which chart are you trying to use?

The official documentation seems disorderly but it indicates at the top it can be segmented by namespace with --watch-namespace, before promptly not talking about helm installation until the very bottom of the page.

When I look at bitnami’s chart I don’t see those options and it’s not immediately clear to me how to configure this. The chart is just over-engineered making it somehow pretty useless imo.

Hi,
I’m using this chart : ingress-nginx/ingress-nginx
The document said : To change this behavior use the flag --watch-namespace to limit the scope to a particular namespace, so --set controller.scope.namespace=ns1 will work

but I want to limit the scope to several namespaces (with only 1 controller), I tried to test but it didn’t work
–set controller.scope.namespace=ns1,ns2 → will get error Error: failed parsing --set data: key “ns2” has no value
2 --set controller.scope.namespace= rows , the last one will override all.
I tried --set controller.scope.namespace={ns1,ns2} and result looks quite good

      containers:
      - args:
        - /nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/nginx-ingress-ingress-nginx-controller
        - --election-id=ingress-controller-leader
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/nginx-ingress-ingress-nginx-controller
        - --watch-namespace=[ns1 ns2]

I’m not sure if it works, you’re right, I should test more.

Does it work if you set it to only 1 namespace?

When I searched for nginx "--controller.scope.namespace" I did find this documented in a README.md in the deprecated helm charts github repo:

namespace to watch for ingress

Something that strikes me as interesting is that it says namespace and not namespaces.

My assumption is that nginx is using watch and I don’t think you can watch against multiple namespaces in a single API call (if you can, please teach me the way, because I vaguely recall fighting some code over this).

For the sake of getting you further along here. I would say your end goal is to have nginx work on limited namespaces and having an nginx per namespace running will do that, wont it?

It works if I set it to only 1 namespace

      containers:
      - args:
        - /nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/nginx-ingress-ingress-nginx-controller
        - --election-id=ingress-controller-leader
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/nginx-ingress-ingress-nginx-controller
        - --watch-namespace=ns1

The idea came when I read this article yesterday.
So I think maybe I should consider it in future, when I have 10 namespaces and I want
nginx-ingress-controller1 listen to all the ingress events from ns1 → 5
nginx-ingress-controller2 listen to all the ingress events from ns6 → 10

When searching for "helm" "controller.scope.namespace" to look specifically for examples of people using --controller.scope.namespace, I only see singular namespace usage of it.

I’ve noticed a few things with helm charts and I have my assumptions that could really diverge into a tangent of how to make better helm charts for users.

Something I might recommend is adopting this way of using and sharing charts. It’s a lot more concise when working with other people and understanding what repo you’re using.

helm install my-release nginx-ingress -n some-namespace --create-namespace --repo https://helm.nginx.com/stable

I don’t even bother adding repos anymore because that would be just one more thing a peer might have to install to do a task when I’m helping them out.

Maybe you’re right, I can only tell ingress controller to watch singular namespace, not multiple namespaces. So it watches All or only one namespace and I have to create 10 nginx ingress controllers to watch 10 namespaces independently.
I create “ingress” namespace then install with helm

helm install nginx-ingress ingress-nginx/ingress-nginx \
    --namespace ingress \
    --set controller.replicaCount=2 \
    --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
    --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux

Then I upgrade it

helm upgrade nginx-ingress ingress-nginx/ingress-nginx --namespace ingress --reuse-values --set controller.scope.enabled=true --set controller.scope.namespace=ns1

Create ingressrule1 in ns1 , ingressrule2 in ns2 , they are created without complain but I can only view ingressrule1 in Nginx Configuration , there is not ingressrule2, because it watches ns1 only

kubectl exec -it -n ingress nginx-ingress-ingress-nginx-controller-fbf5bfdcc-ngrmt -- cat /etc/nginx/nginx.conf

Then I try to upgrade again

helm upgrade nginx-ingress ingress-nginx/ingress-nginx --namespace ingress --reuse-values --set controller.scope.enabled=true --set controller.scope.namespace={ns1,ns2}

New controller pod cannot be created

kubectl get pod -n ingress
NAME                                                      READY   STATUS             RESTARTS   AGE
nginx-ingress-ingress-nginx-controller-5869488759-rbs5s   0/1     CrashLoopBackOff   7          14m
nginx-ingress-ingress-nginx-controller-fbf5bfdcc-ngrmt    1/1     Running            0          132m
nginx-ingress-ingress-nginx-controller-fbf5bfdcc-q4m59    1/1     Running            0          132m
kubectl logs nginx-ingress-ingress-nginx-controller-5869488759-rbs5s -n ingress
F0713 06:59:18.619837       7 main.go:100] No namespace with name [ns1 ns2] found: namespaces "[ns1 ns2]" not found

Or maybe I’m just not doing it right because the complain looks interesting :slight_smile: , somehow it understands “namespaces” ?

You could also go as far as to look through the charts and source code to figure it out.

First figure out how controller.scope.namespace is being used. Looking at controller-daemonset.yaml and controller-deployment.yaml it simply adds the --watch-namespace flag:

--watch-namespace={{ default "$(POD_NAMESPACE)" .Values.controller.scope.namespace }}

Next up is to seek that out in the nginx source code. The flag is defined here

		watchNamespace = flags.String("watch-namespace", apiv1.NamespaceAll,
			`Namespace the controller watches for updates to Kubernetes objects.
This includes Ingresses, Services and all configuration resources. All
namespaces are watched if this parameter is left empty.`)

Now to find watchNamespace variable and it’s just fed into config.Namespace after which, config just gets returned at the end of this big parseFlags function.

That’s fine, parseFlags is only referenced in 4 files and 2 of them are _test.go, so we can skip those and we just came from flags.go, so now we check out main.go.

In main.go I immediately searched for .Namespace since that’s the exported variable and I found that this is how it’s being used with the Go Kubernetes Client.

We’ve reached line main.go:100, where your error is actually happening at. Admittedly I didn’t read that snippet until I was already here staring at the protagonist.

The next thing to understand is how the Kubernetes Client API works.

I’m about to just gloss over some searching I did here. What I have to do to get through the client docs is just manual pattern scanning until I see what seems to fit logically. The Kubernetes Client documentation is not the easiest reference material to digest.

My conclusion is that .Namespaces() results in a NamespaceInterface, and Get() only takes a string argument called name. If it took multiple namespaces it would probably be an array of strings.

Get(ctx context.Context, name string, opts metav1.GetOptions) (*v1.Namespace, error)

I don’t really understand much except this part

Get() only takes a string argument called name . If it took multiple namespaces it would probably be an array of strings.
But thank you very much, I really appreciate you taking the time to find the answer .

For the novice traveler do note there are two nginx ingress implementations:
https://kubernetes.github.io/ingress-nginx
https://helm.nginx.com/stable

The first is the most popular one.

It bit me in the tail hard when figuring out why the ingress controller was behaving other than the documentation was telling me :sweat_smile: