NGinx Ingress Canary Annotation not working

General Discussions
1

Cluster information:

Kubernetes version: 1.21.2
Cloud being used: AKS
Installation method: Manifest https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.48.1/deploy/static/provider/cloud/deploy.yaml
Host OS: linux

This a real shot in the dark, but after 3 days of trying and failing to get canary based ingress traffic working, following all the instructions and guides, I’m at the point now where I don’t know where the problem is.

I’m running K8s (version 1.21.2) on AKS and NGINX Ingress installed from https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.48.1/deploy/static/provider/cloud/deploy.yaml (i’ve updated names and ingress class to ‘internal’ as it’s intended as an internal ingress, I have another one for external ingress)

I have an API service (talk-service) and it’s deployed to a blue and green deployment and have a nginx ingress rule for each so can address either in isolation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blue-talkservice-dev
  namespace: website-dev
  annotations:
    kubernetes.io/ingress.class: nginx-internal
spec:
  tls:
  - hosts:
    - blue-dev.talkservice.co.uk
    secretName: talkservice-dev-cert
  rules:
    - host: blue-dev.talkservice.co.uk
      http:
        paths:
          - backend:
              service:
                name: talkservice-dev-blue
                port:
                  number: 443
            pathType: ImplementationSpecific
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: green-talkservice-dev
  namespace: website-dev
  annotations:
    kubernetes.io/ingress.class: nginx-internal
spec:
  tls:
  - hosts:
    - green-dev.talkservice.co.uk
    secretName: talkservice-dev-cert
  rules:
    - host: green-dev.talkservice.co.uk
      http:
        paths:
          - backend:
              service:
                name: talkservice-dev-green
                port:
                  number: 443
            pathType: ImplementationSpecific

Hitting either of these green-dev.talkservice.co.uk or blue-dev.talkservice.co.uk works and I get a response from the service.

Both deployments are running the same code but to ensure zero down time when deploying I want to have dev.talkservice.co.uk talk to either blue or green, but when deploying I can swap traffic 100% to blue and deploy new code to green, then swap 100% traffic to green and deploy to blue. Once complete, swap traffic back to 50/50 blue and green.

I thought I could use the canary annotation for nginx ingress like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: talkservice-dev-green
  namespace: website-dev
  annotations:
    kubernetes.io/ingress.class: nginx-internal
  labels:
    app: talkservice-dev
spec:
  tls:
  - hosts:
    - dev.talkservice.co.uk
    secretName: talkservice-dev-cert
  rules:
    - host: dev.talkservice.co.uk
      http:
        paths:
          - backend:
              service:
                name: talkservice-dev-green
                port:
                  number: 443
            path: /
            pathType: ImplementationSpecific
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: talkservice-dev-blue
  namespace: website-dev
  annotations:
    kubernetes.io/ingress.class: nginx-internal
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "50"
  labels:
    app: talkservice-dev
spec:
  tls:
  - hosts:
    - dev.talkservice.co.uk
    secretName: talkservice-dev-cert
  rules:
    - host: dev.talkservice.co.uk
      http:
        paths:
          - backend:
              service:
                name: talkservice-dev-blue
                port:
                  number: 443
            path: /
            pathType: ImplementationSpecific
---
apiVersion: v1
data:
  allow-backend-server-header: "true"
  use-forwarded-headers: "true"
kind: ConfigMap
metadata:
  name: talkservice-dev-green
  namespace: website-dev
---
apiVersion: v1
data:
  allow-backend-server-header: "true"
  use-forwarded-headers: "true"
kind: ConfigMap
metadata:
  name: talkservice-dev-blue
  namespace: website-dev

However after applying it and provisioning through with no errors, the canary traffic rule doesn’t seem to work.

I’ve checked the logs on the ingress-controller and I just get traffic sent to green, even if I set the canary weight to 100% to the opposite service.

The nginx config just has dev.talkservice.co.uk talking to the green service with no rules etc for blue, I’ve read that the canary stuff is not put in the nginx.conf file but is done via lua, but all the documentation and guides I’ve read make no mention of further actions I need to take.

Am I missing something obvious that I can’t see from being so far down the rabbit hole?

2

I have the same issue. I have copied the link of your question to kubernetes user groups.