Multiple kong ingress controller or just one to different environments

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: 1.12.6
Cloud being used: Azure Kubernetes Service
Installation method:
Host OS: Terraform
CNI and version:
CRI and version:


I want to set up in one kubernetes cluster, multiples environments of some web application of this way:

  • dev.my-domain.org to development environment
  • sandbox.my-domain.org to the sandbox environment
  • production.my-domain.org to the production environment

Currently, I have the dev.mydomain.org of this way:

  • a kong installation (include kong-ingress-controller)
    • Kong Postgres database
  • My application service installed via helm chart pulled from Azure Container Registry.

I am thinking in an implement with that same approach to sandbox and production environments each of them in their respective namespace, and I have the doubt with relation to kong-ingress-controller:

  • Should Is there one kong-ingress-controller by each environment. This means:


Or

  • Should Is there just one kong-ingress-controller which works and manage all the different ingress resources existing in the different namespaces, but each domain point to the same kong-ingress-controller…? This means:

Is this possible?

I am not sure about how kong does it’s magic, but for example nginx-ingress stays in it’s own namespace and monitors each of the other namespaces. and when you configure an ingress in a namespaces it just applies the configuration (creates an ingress for that namespace) using whatever rules you set up for it. including domains.

Is the philosophy of an ingress controller is manage many domains doesn’t it?

Currently, in my dev.my-domain.org environment my-ingress-application resource is pointing to my kong-ingress-controller, via kong-proxy public IP address:

âź© kubectl get ingress my-ingress-application
NAME                           HOSTS                                ADDRESS          PORTS     AGE
my-ingress-application   dev.mydomain.org   same-IP-Address   80, 443   103m
[I] 


âź© kubectl get svc/kong-proxy -n kong
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                      AGE
kong-proxy   LoadBalancer   10.0.174.245   same-IP-Address   80:31166/TCP,443:31525/TCP   3h42m
[I] 
âź© 

I have one doubt in order to have only one database to all these environments.
Should each of them to be configured in only one ingress resource?
This means of this way:

apiVersion: configuration.konghq.com/v1
kind: KongIngress
metadata:
   name: kong-ingress-config
   namespace: default
proxy:
   protocols:
      - http
      - https
   path: /
route:
   methods:
      - POST
      - GET
   strip_path: false
   preserve_host: true
 ---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress-application
  #namespace: default
  annotations:
    kubernetes.io/ingress.class: "kong"
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod # letsencrypt-staging

    # We are indicating the KongIngress resource
    configuration.konghq.com: kong-ingress-config

    certmanager.k8s.io/acme-challenge-type: http01
    kubernetes.io/tls-acme: "true"
    certmanager.k8s.io/acme-http01-edit-in-place: "true"
spec:
  rules:
 # DEV ENVIRONMENT
  - host: dev.mydomain.org
    http:
      paths:
        - path: "/"
          backend:
            serviceName: myapp
            servicePort: 80
  # SANDBOX ENVIRONMENT
  - host: sandbox.mydomain.org
    http:
      paths:
        - path: "/"
          backend:
            serviceName: myapp
            servicePort: 80  
  tls:
  - hosts:
    - dev.mydomain.org
    - sandbox.mydomain.org
    - production.mydomain.org 
    secretName: letsencrypt-prod #letsencrypt-staging

So, is possible that my my-ingress-application resource could to have multiple hosts pointing the same kong-ingress-controller (kong-proxy) and using the same secretName ?
Maybe something like this: ?

âź© kubectl get ingress my-ingress-application
    NAME                           HOSTS              ADDRESS             PORTS    AGE
    my-ingress-application   dev.mydomain.org         same-IP-Address   80, 443    103m
                             sandbox.mydomain.org     same-IP-Address   80, 443    103m
                             production.mydomain.org  same-IP-Address   80, 443    103m
[I] 

Could to be this a correct approach?

You can share it between environments.

In my personal opinion, but I want to remark that this is personal, I want to be able to change and test configurations without affecting production in any way. If you are sharing you cluster with some test/something environments and the production environment, I’d use separate Ingress controller for them.

That way you can make sure when you test some new Ingress config or something, it won’t affect your production environment. And you can even reserve more cpu/mem for production pods and scale them differently.

So, I’d share all except production, or something like that :slight_smile:

2 Likes

@rata Great recommendations, thank you!
Have the ability to change and test configurations without affecting production is something that I would too definitively.

I am going to choose some approach in order to have one kong for production and other for the remaining environments, it is a great idea.

Just one little doubt:

In my development deployment, I have installed kong-ingress-controller inside kong namespace and my app-service and ingress resource for that app-service inside the default namespace and the cert-manager inside the cert-manager namespace.

Having this way kong-ingress-controller (inside kong namespace) is in the ability to be reached by my ingress resource (in the default namespace) and work with cert-manager (inside cert-manager namespace) with some cluster issuers (which are not namespaced) and some cert-manager secrets (inside cert-manager namespace) and take the certificates issued (inside default namespace)

This means, that without matter if share or not the kong ingress controller between environments, kong is in the ability to monitor each of the other namespaces and when I configure an ingress using kong (in any namespaces) it just applies the configuration and use the kong controller

I am right?

Never used Kong, but usually you specify a “class” annotation (was that the name?) and only that controller takes action on that resource.

Does that answer or I misunderstood the question? :slight_smile:

Yes I supposse that it is the philosophy of an ingress-controller, takes action on any ingress resource without matter the namespace which they belong really?

Currently kong do that.
The annotation class is the same, quite simple
kubernetes.io/ingress.class: "kong"

Currently this is my ingress resource in dev environment and this should be taked by kong wothout matter that it and the ingress resource are located in different namespaces …

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: appdev-ingress
  namespace: development
  annotations:
    kubernetes.io/ingress.class: "kong"
    # configuration.konghq.com: kong-ingress-config
    certmanager.k8s.io/acme-challenge-type: http01
    kubernetes.io/tls-acme: "true"
    certmanager.k8s.io/acme-http01-edit-in-place: "true"
    certmanager.k8s.io/cluster-issuer: letsencrypt-staging

spec:
  rules:
  - host: appdev.example.com
    http:
      paths:
        - path: "/"
          backend:
            serviceName: app-dev  # located in development namespace
            servicePort: 80
  tls:
  - hosts:
    -  appdev.example.com
    secretName: letsencrypt-staging

Hi, I again, I got the following deployment:

  • namespace kong
    kong-ingress-controller (kong and kong-proxy)

  • namespace development
    app-service
    ingress-resource using that app-service and pointing to kong-ingress-controller located in namespace kong above

  • namespace kong-production
    kong-ingress-controller (kong and kong-proxy)

  • namespace production
    app-service
    ingress-resource using that app-service and pointing to kong-ingress-controller located in namespace kong-production above

In the first instance both ingress-resource (development and production namespace) it works
If you go to:

We can see that both ingress resources are working and seem like if they were integrated with their respective kong-ingress-controller in their own namespaces (kong to dev service and kong-production to prod service)

But when I update some ingress-resource in any of the namespaces mentioned, the kong-ingress-controller which is pointing that ingress-resource take over of all ingress operations inside my cluster and redirect all requests to their respective database.

As a result of this behavior, currently I was working with production ingress controller and resource, and this take over monitoring my cluster and has caught the requests made to zcrm365dev.possibilit.nl domain too, and it suppose that these requests are managed by the development ingress controller and resource.
Finally, in my kong ingress controller production database, I have routes of the appservice development and the appservice production.

  • This is my kong (development) database

The first time when the ingress resource in development environment was created, the kong-ingress-controller in the development environment (namespace kong) take over this route and was saved it.
Let’s have a look at the date in orange on April 3, 2019 and the route with cert-manager in red color.

  • This is my kong-production database

The first time when the ingress resource in production environment was created, the kong-ingress-controller in the production environment (namespace kong-production) take over this route and was saved it.
Let us have a look at the date in the first record on the blue rectangle, on April 4, 2019 at 10:27 hours, in which cert-manager configure it in the path attribute

But, then, I proceed to make some requests to zcrmdev domain and the production database take this request and store it, such as we can see in the third record in orange on on April 4, 2019 at 10:34 hours

I am not sure if maybe this could to have some influence or impact in the way of how these kong-ingress-controller in the different namespaces works because each one of them has registered the correct path according to request domain that correspond to each other, but, I think that the consequent route registers does not should be stored in the opposite database, I think that this kong-ingress-controller will receive some requests from any domain direction (zcm365dev or zcrm365prod) and kong-proxy will route the requests to their respective ingress resource and from there to their respective service pods in the different namespace and from there to their respective database.

But by other side, when I want to validate order certificates of each one of this domain, cert-manager pod does not know to what kong-ingress-controller use to perform the validation

Does sounds logic these affirmations? doesn’t?

Is this normal?

Does my ingress-controller have competition by taking over of the ingress operations inside my cluster?

Yeah, that is totally expected. The ingress controller is sometimes deployed to it’s own namespace. So, in that cases is needs to do it. And, in a way, it makes sense at least not to be forced to deploy one Ingress resource per namespace, IMHO.

bgarcial

    April 4

rata:
Never used Kong, but usually you specify a “class” annotation (was that the name?) and only that controller takes action on that resource.

Hi, I again, I got the following deployment:

  • namespace kong

    kong-ingress-controller (kong and kong-proxy)

  • namespace development

    app-service

    ingress-resource using that app-service and pointing to kong-ingress-controller located in namespace kong above

  • namespace kong-production

    kong-ingress-controller (kong and kong-proxy)

  • namespace production

    app-service

    ingress-resource using that app-service and pointing to kong-ingress-controller located in namespace kong-production above

In the first instance both ingress-resource (development and production namespace) it works

If you go to:

But when I update some ingress-resource in any of the namespaces mentioned, the kong-ingress-controller which is pointing that ingress-resource take over of all ingress operations inside my cluster and redirect all requests to their respective database.

I really don’t know about Kong, but usually when you deploy an Ingress controller, if you want to have more than one, you need to specify to which “class” each will take care of.

So, if you configure one Ingress controller to listen to class “kong-prod” and the others to “kong”, then you specify the class in the Ingress resource and that should do the trick and avoid what happened to you.

Contour, another Ingress resource, has a parameter to the binary IIRC (maybe I don’t, but it had some way to specify which class to react to).

Does my ingress-controller have competition by taking over of the ingress operations inside my cluster?

They shouldn’t if you configure them to listen to different class, IIUC.

1 Like

How to can I configure my ingress resource to listen to different class? I am using this class kubernetes.io/ingress.class: "kong" which point to kong-ingress-controller, this is a standard annotation, I think so, but how to can I differentiate one kong instance of other?

You are right,

I can configure each of my Kong Ingress controllers to listen for Ingress resources of a specific class using the --ingress-class CLI flag in the yaml manifest

So, in case for my dev environment, I could add --ingress-class dev-env to the Kong Ingress Controller’s CLI flag.

By example

The kong-ingress-controller yaml manifest should be:

- name: ingress-controller
  args:
  - /kong-ingress-controller
  # the kong URL points to the kong admin api server
  - --kong-url=https://localhost:8444
  - --admin-tls-skip-verify
  # the default service is the kong proxy service
  - --default-backend-service=kong/kong-proxy
  # Service from were we extract the IP address/es to use in Ingress status
  - --publish-service=kong/kong-proxy
  
  # INGRESS CLASS
  - --ingress-class=development

And then in my ingress resource if I want to point to that kong-ingress-controller
I should add the kubernetes.io/ingress.class: "development"

Any name cane be defined for you the ingress-class and the name you use as this CLI flag, you should use the same for the kubernetes.io/ingress.class annotation.

Glad it worked! :slight_smile: