Ingress - nginx - rewrite target

General Discussions
,
1

I am using Ingress as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend-backend-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.className: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
    - http:
        paths:
          # Route for /api -> flask-api-service with path rewrite
          - path: /api(/|$)(.*)
            pathType: Prefix
            backend:
              service:
                name: flask-api-service
                port:
                  number: 5000
          # Route for / -> frontend-service
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend-service
                port:
                  number: 80

I did not set the host variable as I don’t have a domain and IP address is not allowed.

Currently with this setup everything is mapped onto frontend-service, I was expecting e.g. /api/docs/ to go to flask-api-service as /docs

What am I doing wrong?

2

try using
pathType: ImplementationSpecific

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend-backend-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
    - host: ""
      http:
        paths:
          # Route for /api -> flask-api-service with path rewrite
          - path: /api(/|$)(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: flask-api-service
                port:
                  number: 5000
          # Route for / -> frontend-service
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: frontend-service
                port:
                  number: 80
3

Hey @jayeshmahajan thanks a lot for your response. I tried this just now. visiting /api/docs or any other endpoint on /api redirects me to the frontend-service no rewrite or redirect to flask-api-service is done.

4

Hi,
Try changing ingress logs to see which ingress rule processes requests

5

Hey @fox-md

Having added

    nginx.ingress.kubernetes.io/enable-access-log: "true"
    nginx.ingress.kubernetes.io/access-log-level: "info"

I get the following logs when executing kubectl logs -l app.kubernetes.io/name=ingress-nginx -n ingress-nginx

I1111 11:32:44.271621       7 nginx.go:327] Starting validation webhook on :8443 with keys /usr/local/certificates/cert /usr/local/certificates/key
E1111 11:32:44.280033       7 leaderelection.go:324] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:ingress-nginx:ingress-nginx" cannot create resource "configmaps" in API group "" in the namespace "ingress-nginx"
I1111 11:32:44.289300       7 status.go:137] removing address from ingress status ([172.16.210.10])
I1111 11:32:44.289399       7 nginx.go:388] Stopping admission controller
I1111 11:32:44.289471       7 nginx.go:396] Stopping NGINX process
E1111 11:32:44.289601       7 nginx.go:329] http: Server closed
2024/11/11 11:32:44 [notice] 60#60: signal process started
I1111 11:32:48.297003       7 nginx.go:409] NGINX process has stopped
I1111 11:32:48.297032       7 main.go:187] Handled quit, awaiting Pod deletion
E1111 11:32:55.004859       7 leaderelection.go:324] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:ingress-nginx:ingress-nginx" cannot create resource "configmaps" in API group "" in the namespace "ingress-nginx"
6

I am not sure this is sufficient.
There should be a configmap that is being used by ingress controller pods.
Try adding below log-format-upstream

apiVersion: v1
kind: ConfigMap
...
data:
  ...
  log-format-upstream: $remote_addr $host $remote_user [$time_local] $request $status
    $body_bytes_sent $http_referer $http_user_agent $request_length $request_time
    [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length
    $upstream_response_time $upstream_status $req_id $namespace $ingress_name
7

This did solve the permission issue indeed.

However this led to a different issue

W1111 13:02:09.712692       7 main.go:125] No IngressClass resource with name nginx found. Only annotation will be used.
I1111 13:02:09.724488       7 ssl.go:528] loading tls certificate from certificate path /usr/local/certificates/cert and key path /usr/local/certificates/key
I1111 13:02:09.752407       7 nginx.go:263] Starting NGINX Ingress controller
I1111 13:02:09.756734       7 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"cd084303-e148-4c01-b7df-f7600688ed14", APIVersion:"v1", ResourceVersion:"9313339", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
E1111 13:02:10.855077       7 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E1111 13:02:12.085901       7 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E1111 13:02:14.743175       7 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E1111 13:02:20.317423       7 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E1111 13:02:30.348598       7 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource
E1111 13:02:49.903192       7 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/reflector.go:125: Failed to list *v1beta1.Ingress: the server could not find the requested resource

Doing a search I found that the Failed to list *v1beta1.Ingress: the server could not find the requested resource is a versioning error. According to some sources, to solve this issue I would need to install the nginx-controller with helm. However this is not an option for me.

8

It seems that controller is built using older k8s libraries (v 1.18) and still tries to look for the v1beta1.Ingress.
Try checking whether your cluster has networking.k8s.io/v1beta1 group using the kubectl api-versions | grep networking command.
If that is not the case, controller should be updated to a newer version.

9

Thanks a lot for helping me out!

I only have networking.k8s.io/v1

It seems the problem was that kubernetes v1.27 didn’t support the newest version of the controller. I went back to v1.11.3 of the controller.

I no longer get any errors in the logs, but it also doesn’t give any info when trying to reach api/docs (or any other endpoint for that matter).

When accessing the /api endpoint, it doesn’t go to the frontend-service anymore tho. However it also doesn’t go to the flask-api-service tho, no clue where it is going. I checked my api deployment, no sign of it being accessed.

10

According to the matrix, GitHub - kubernetes/ingress-nginx: Ingress NGINX Controller for Kubernetes, 1.11.3 is the latest stable version. Indeed, this is the latest version that supports k8s version 1.27.

Have you enabled ingress logging? Last 2 items in the log entry should be namespace and ingress name. This way you know for sure who is serving request.

11

As you have stated I added

    nginx.ingress.kubernetes.io/enable-access-log: "true"
    nginx.ingress.kubernetes.io/access-log-level: "info"

Additionally I added:

nginx.ingress.kubernetes.io/enable-rewrite-log: "true"

However nothing is appearing in the logs when accessing any endpoint on my server.

All I get is

I1111 15:30:22.016360       7 controller.go:193] "Configuration changes detected, backend reload required"
I1111 15:30:22.053513       7 controller.go:213] "Backend successfully reloaded"
I1111 15:30:22.053726       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-8579885d9c-42rzn", UID:"87188540-010f-45af-ac1c-978742d42f65", APIVersion:"v1", ResourceVersion:"9314982", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
12

Try adding below:

Once pod picks up changes, tail controller logs and access desired resource.

13

This was already the case, this is the configmap ingress-nginx-controller

apiVersion: v1
data:
  log-format-upstream: $remote_addr $host $remote_user [$time_local] $request $status
    $body_bytes_sent $http_referer $http_user_agent $request_length $request_time
    [$proxy_upstream_name] [$proxy_alternative_upstream_name] $upstream_addr $upstream_response_length
    $upstream_response_time $upstream_status $req_id $namespace $ingress_name
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"allow-snippet-annotations":"false"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/name":"ingress-nginx","app.kubernete  creationTimestamp: "2024-11-11T13:30:24Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.11.3
  name: ingress-nginx-controller
  namespace: ingress-nginx
  resourceVersion: "9315101"

I restarted the controller deployment, but there are still no logs appearing.

My ingress currently:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: frontend-backend-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/enable-access-log: "true"
    nginx.ingress.kubernetes.io/access-log-level: "info"
    nginx.ingress.kubernetes.io/enable-rewrite-log: "true"
spec:
  rules:
    - host: ""
      http:
        paths:
          # Route for /api -> flask-api-service with path rewrite
          - path: /api(/|$)(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: flask-api-service
                port:
                  number: 5000
          # Route for / -> frontend-service
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: frontend-service
                port:
                  number: 80
14

Can you try accessing /api/docs resource and show controller logs?
It should return something similar to

10.32.0.1 rewrite.example.com - [11/Nov/2024:15:58:18 +0000] GET /dummy HTTP/1.1 200 612 - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 465 0.047 [default-web-80] [] 10.32.0.15:80 612 0.046 200 d24de037ff1cfca1dcadf286debc93d4 default rewrite

Try removing host: "" from ingress, just like here Ingress | Kubernetes.

15

This is all the logs I got from the controller since last restart

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.11.3
  Build:         0106de65cfccb74405a6dfa7d9daffc6f0a6ef1a
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.5

-------------------------------------------------------------------------------

W1111 15:44:13.961886       7 client_config.go:659] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1111 15:44:13.962109       7 main.go:205] "Creating API client" host="https://10.43.0.1:443"
I1111 15:44:13.967328       7 main.go:248] "Running in Kubernetes cluster" major="1" minor="27" git="v1.27.7+k3s2" state="clean" commit="575bce7689f4be112bd0099362fb8d5f2e39398e" platform="linux/amd64"
I1111 15:44:14.194570       7 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I1111 15:44:14.215088       7 ssl.go:535] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I1111 15:44:14.229129       7 nginx.go:271] "Starting NGINX Ingress controller"
I1111 15:44:14.237175       7 event.go:377] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"d69c093c-bdc8-4193-ac29-5d6c515b871c", APIVersion:"v1", ResourceVersion:"9315101", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I1111 15:44:15.333232       7 store.go:440] "Found valid IngressClass" ingress="default/frontend-backend-ingress" ingressclass="nginx"
I1111 15:44:15.333526       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"frontend-backend-ingress", UID:"b6e5b97e-df4b-49bd-a4ed-4115b798650f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"9318213", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1111 15:44:15.431158       7 nginx.go:317] "Starting NGINX process"
I1111 15:44:15.431235       7 leaderelection.go:254] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
I1111 15:44:15.431848       7 nginx.go:337] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I1111 15:44:15.432356       7 controller.go:193] "Configuration changes detected, backend reload required"
I1111 15:44:15.435820       7 status.go:85] "New leader elected" identity="ingress-nginx-controller-8579885d9c-42rzn"
I1111 15:44:15.468707       7 controller.go:213] "Backend successfully reloaded"
I1111 15:44:15.468787       7 controller.go:224] "Initial sync, sleeping for 1 second"
I1111 15:44:15.468874       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-ddcfbb74c-qfqmp", UID:"9c23e3d1-4771-4ff4-8f86-4f5b1a64f0ab", APIVersion:"v1", ResourceVersion:"9318595", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I1111 15:45:05.168320       7 leaderelection.go:268] successfully acquired lease ingress-nginx/ingress-nginx-leader
I1111 15:45:05.168415       7 status.go:85] "New leader elected" identity="ingress-nginx-controller-ddcfbb74c-qfqmp"
I1111 16:07:12.843700       7 controller.go:193] "Configuration changes detected, backend reload required"
I1111 16:07:12.876457       7 controller.go:213] "Backend successfully reloaded"
I1111 16:07:12.876680       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-ddcfbb74c-qfqmp", UID:"9c23e3d1-4771-4ff4-8f86-4f5b1a64f0ab", APIVersion:"v1", ResourceVersion:"9318595", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I1111 16:07:22.462188       7 main.go:107] "successfully validated configuration, accepting" ingress="default/frontend-backend-ingress"
I1111 16:07:22.471845       7 store.go:440] "Found valid IngressClass" ingress="default/frontend-backend-ingress" ingressclass="nginx"
I1111 16:07:22.472061       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"frontend-backend-ingress", UID:"cebfbd9d-ac82-4176-aeb1-9a90be85678f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"9319227", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1111 16:07:22.472474       7 controller.go:193] "Configuration changes detected, backend reload required"
I1111 16:07:22.505618       7 controller.go:213] "Backend successfully reloaded"
I1111 16:07:22.505947       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-ddcfbb74c-qfqmp", UID:"9c23e3d1-4771-4ff4-8f86-4f5b1a64f0ab", APIVersion:"v1", ResourceVersion:"9318595", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration

No matter what endpoint I access, no logs are added.

16

I see.

Try removing host: "" from ingress, just like here Ingress | Kubernetes.

17

This has been done, the logs above include that, also tried accessing endpoints after adjusting.

18

I have a stupid question. Are there other nginx ingress controllers deployed in other namespaces?
Do you see the frontend page when you call api/docs?

19

Just checked with

kubectl get deployments --all-namespaces

There is only 1 nginx controller and a traefik controller, however I double checked with kubectl get ingress and I am using nginx as class for my ingress.

When visiting / I get the homepage of my frontend service (as expected), however visiting /api or /api/docs gives a 404 and no acccess is logged in my backend deployment.

20

I see. Another shot in the dark… Are you sure you are trying to access the nginx ingress and not traefik? I am a bit confused by the fact that there are no access logs in nginx.

1 Like