Addon: cert-manager

From 1.25

This addon installs Cert Manager. Cert-Manager is the de-facto standard solution for certificate management in Kubernetes clusters. It supports x.509 certificate management for Kubernetes and OpenShift clusters, retrieving certificates from private (internal) or public issuers, and ensures they are properly rotated and kept up to date.

Install this addon with:

microk8s enable cert-manager

Automatically generating Let’s Encrypt certificates for Ingress

One of the common use-cases of Cert-Manager is to configure Kubernetes Ingress resources with automatic TLS certificates from Let’s Encrypt.

Requirements

  1. A MicroK8s cluster with a public IP address. This is required to complete the HTTP challenges of Let’s Encrypt.

  2. A hostname that resolves to your public IP address, e.g. my-service.example.com.

  3. A properly configured ingress class for your MicroK8s cluster. The simplest way to do this is to use the ingress addon:

microk8s enable ingress dns

Create a ClusterIssuer

A ClusterIssuer resource is used to configure an account with Let’s Encrypt. All you need is an email address (make sure to use a valid email address).

Create a ClusterIssuer called lets-encrypt with the command below. Make sure to replace microk8s@example.com below with your email. Note that Let’s Encrypt will refuse to register accounts that use the example.com domain.

microk8s kubectl apply -f - <<EOF
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: lets-encrypt
spec:
 acme:
   email: microk8s@example.com
   server: https://acme-v02.api.letsencrypt.org/directory
   privateKeySecretRef:
     # Secret resource that will be used to store the account's private key.
     name: lets-encrypt-priviate-key
   # Add a single challenge solver, HTTP01 using nginx
   solvers:
   - http01:
       ingress:
         class: public
EOF

Verify that the ClusterIssuer was created successfully with microk8s kubectl get clusterissuer -o wide, which should produce output similar to:

NAME           READY   STATUS                                                 AGE
lets-encrypt   True    The ACME account was registered with the ACME server   2m19s

Deploy a service

For this example, we will deploy a simple microbot deployment:

microk8s kubectl create deploy --image cdkbot/microbot:1 --replicas 3 microbot
microk8s kubectl expose deploy microbot --port 80 --type ClusterIP

Ensure the service is up and running with microk8s kubectl get pod,svc:

NAME                          READY   STATUS    RESTARTS   AGE
pod/microbot-b6996696-sbp76   1/1     Running   0          11s
pod/microbot-b6996696-xmplm   1/1     Running   0          11s
pod/microbot-b6996696-8b82c   1/1     Running   0          11s

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.152.183.1     <none>        443/TCP   16m
service/microbot     ClusterIP   10.152.183.134   <none>        80/TCP    3s

Configure ingress

Next, create a Kubernetes ingress resource that forwards requests made to https://my-service.example.com to our microbot service.

Note that the cert-manager.io/cluster-issuer: lets-encrypt annotation tells Cert-Manager to automatically retrieve TLS certificates for our domain. The following example needs to reference the correct hostnames for your deployment - please substitute appropriately before running this command:

microk8s kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: microbot-ingress
 annotations:
   cert-manager.io/cluster-issuer: lets-encrypt
spec:
 tls:
 - hosts:
   - my-service.example.com
   secretName: microbot-ingress-tls
 rules:
 - host: my-service.example.com
   http:
     paths:
     - backend:
         service:
           name: microbot
           port:
             number: 80
       path: /
       pathType: Exact
EOF

After a while, Cert-Manager will automatically request a certificate from Let’s Encrypt, populate the microk8s-ingress-tls with it and configure the ingress. Finally, you should be able to access your service at its fully qualified domain.

Hi, @evilnick
Is there any way to use under microk8s 1.22 ?
I look forward to your reply.
Thanks.

@amo.mycena thanks for the message. Unfortunately no, this relies on features which were added in 1.25

Thanks for your reply.

Hi this worked great. I presume I don’t need to keep the microbot deployment running?

Will cert manager get a new TLS cert for any ingress configured with that annotation, or do I have to keep this ingress and just reconfigure the backend service for my ‘real’ website?

@bramo That’s great! No, the microbot service is just a simple test to verify that it is working, you can kill it.

Been trying to get this to work for days without success.

Today i reset my microk8s-cluster with snap remove microk8s --purge, then start with a fresh

sudo snap install microk8s --classic --channel=1.26/stable

and go on with the following commands:

sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube
su - $USER
microk8s status --wait-ready
sudo snap alias microk8s.kubectl kubectl
sudo snap alias microk8s.helm3 helm
helm repo add stable https://charts.helm.sh/stable
helm repo update
microk8s add-node 
    #then add the 2 other nodes with
microk8s join ....
microk8s enable dns:192.168.1.1,5.1.66.255          # CoreDNS
microk8s enable host-access                         # Allow Pods connecting to Host services smoothly
microk8s enable ingress                             # Ingress controller for external access
microk8s enable dashboard                           # The Kubernetes dashboard
microk8s enable metallb:'192.168.1.10-192.168.1.10,192.168.1.240-192.168.1.253'
microk8s enable cert-manager
kubectl apply -f clusterissuer.yml

Now after a few unsuccesful trials with my own clusterissuer and kuard as app i tried your microbot example from the top with the same result:

default     1s          Normal    ScalingReplicaSet   deployment/microbot                                  Scaled up replica set microbot-b778687f9 to 3
default     1s          Normal    SuccessfulCreate    replicaset/microbot-b778687f9                        Created pod: microbot-b778687f9-5jnvm
default     1s          Normal    SuccessfulCreate    replicaset/microbot-b778687f9                        Created pod: microbot-b778687f9-g68kh
default     1s          Normal    SuccessfulCreate    replicaset/microbot-b778687f9                        Created pod: microbot-b778687f9-dslhz
default     0s          Normal    Scheduled           pod/microbot-b778687f9-g68kh                         Successfully assigned default/microbot-b778687f9-g68kh to cluster-wf-technik01
default     0s          Normal    Scheduled           pod/microbot-b778687f9-5jnvm                         Successfully assigned default/microbot-b778687f9-5jnvm to cluster-wf-essen01
default     0s          Normal    Scheduled           pod/microbot-b778687f9-dslhz                         Successfully assigned default/microbot-b778687f9-dslhz to cluster-bs-dg01
default     0s          Normal    Pulling             pod/microbot-b778687f9-g68kh                         Pulling image "cdkbot/microbot:1"
default     0s          Normal    Pulling             pod/microbot-b778687f9-5jnvm                         Pulling image "cdkbot/microbot:1"
default     0s          Normal    Pulling             pod/microbot-b778687f9-dslhz                         Pulling image "cdkbot/microbot:1"
default     0s          Normal    Pulled              pod/microbot-b778687f9-g68kh                         Successfully pulled image "cdkbot/microbot:1" in 6.928614238s (6.928619642s including waiting)
default     0s          Normal    Created             pod/microbot-b778687f9-g68kh                         Created container microbot
default     1s          Normal    Started             pod/microbot-b778687f9-g68kh                         Started container microbot
default     0s          Normal    Pulled              pod/microbot-b778687f9-dslhz                         Successfully pulled image "cdkbot/microbot:1" in 45.021152957s (45.021157699s including waiting)
default     0s          Normal    Created             pod/microbot-b778687f9-dslhz                         Created container microbot
default     0s          Normal    Started             pod/microbot-b778687f9-dslhz                         Started container microbot
default     0s          Normal    Pulled              pod/microbot-b778687f9-5jnvm                         Successfully pulled image "cdkbot/microbot:1" in 47.301921331s (47.301925255s including waiting)
default     0s          Normal    Created             pod/microbot-b778687f9-5jnvm                         Created container microbot
default     0s          Normal    Started             pod/microbot-b778687f9-5jnvm                         Started container microbot
ingress     0s          Normal    RELOAD              pod/nginx-ingress-microk8s-controller-l46t4          NGINX reload triggered due to a change in configuration
ingress     0s          Normal    RELOAD              pod/nginx-ingress-microk8s-controller-pxtlb          NGINX reload triggered due to a change in configuration
ingress     1s          Normal    RELOAD              pod/nginx-ingress-microk8s-controller-dnqkv          NGINX reload triggered due to a change in configuration
default     0s          Normal    Sync                ingress/microbot-ingress                             Scheduled for sync
default     0s          Normal    Sync                ingress/microbot-ingress                             Scheduled for sync
ingress     0s          Normal    RELOAD              pod/nginx-ingress-microk8s-controller-pxtlb          NGINX reload triggered due to a change in configuration
ingress     0s          Normal    RELOAD              pod/nginx-ingress-microk8s-controller-l46t4          NGINX reload triggered due to a change in configuration
default     0s          Normal    CreateCertificate   ingress/microbot-ingress                             Successfully created Certificate "microbot-ingress-tls"
default     0s          Normal    Sync                ingress/microbot-ingress                             Scheduled for sync
ingress     0s          Normal    RELOAD              pod/nginx-ingress-microk8s-controller-dnqkv          NGINX reload triggered due to a change in configuration
default     0s          Normal    Issuing             certificate/microbot-ingress-tls                     Issuing certificate as Secret does not exist
default     0s          Normal    Generated           certificate/microbot-ingress-tls                     Stored new private key in temporary Secret resource "microbot-ingress-tls-p9j5v"
default     0s          Normal    Requested           certificate/microbot-ingress-tls                     Created new CertificateRequest resource "microbot-ingress-tls-tsmj9"
default     0s          Normal    cert-manager.io     certificaterequest/microbot-ingress-tls-tsmj9        Certificate request has been approved by cert-manager.io
default     1s          Normal    OrderCreated        certificaterequest/microbot-ingress-tls-tsmj9        Created Order resource default/microbot-ingress-tls-tsmj9-3484024173
default     1s          Normal    OrderPending        certificaterequest/microbot-ingress-tls-tsmj9        Waiting on certificate issuance from order default/microbot-ingress-tls-tsmj9-3484024173: ""
default     0s          Normal    Sync                ingress/microbot-ingress                             Scheduled for sync
default     0s          Normal    Sync                ingress/microbot-ingress                             Scheduled for sync
default     0s          Normal    Sync                ingress/microbot-ingress                             Scheduled for sync
default     1s          Normal    Created             order/microbot-ingress-tls-tsmj9-3484024173          Created Challenge resource "microbot-ingress-tls-tsmj9-3484024173-1852125348" for domain "dahoam.13mail.de"
default     0s          Normal    Started             challenge/microbot-ingress-tls-tsmj9-3484024173-1852125348   Challenge scheduled for processing
default     0s          Normal    Sync                ingress/cm-acme-http-solver-4c9x2                            Scheduled for sync
default     0s          Normal    Presented           challenge/microbot-ingress-tls-tsmj9-3484024173-1852125348   Presented challenge using HTTP-01 challenge mechanism
default     0s          Normal    Sync                ingress/cm-acme-http-solver-4c9x2                            Scheduled for sync
default     0s          Normal    Sync                ingress/cm-acme-http-solver-5tk2v                            Scheduled for sync
default     0s          Normal    Sync                ingress/cm-acme-http-solver-5tk2v                            Scheduled for sync
default     0s          Normal    Scheduled           pod/cm-acme-http-solver-9df5d                                Successfully assigned default/cm-acme-http-solver-9df5d to cluster-bs-dg01
default     0s          Normal    Sync                ingress/cm-acme-http-solver-4c9x2                            Scheduled for sync
default     0s          Normal    Sync                ingress/cm-acme-http-solver-5tk2v                            Scheduled for sync
default     0s          Normal    Sync                ingress/cm-acme-http-solver-hg7j9                            Scheduled for sync
default     0s          Normal    Sync                ingress/cm-acme-http-solver-hg7j9                            Scheduled for sync
default     0s          Normal    Sync                ingress/cm-acme-http-solver-hg7j9                            Scheduled for sync
default     0s          Normal    Pulling             pod/cm-acme-http-solver-9df5d                                Pulling image "quay.io/jetstack/cert-manager-acmesolver:v1.8.0"
ingress     1s          Normal    RELOAD              pod/nginx-ingress-microk8s-controller-pxtlb                  NGINX reload triggered due to a change in configuration
ingress     1s          Normal    RELOAD              pod/nginx-ingress-microk8s-controller-l46t4                  NGINX reload triggered due to a change in configuration
ingress     1s          Normal    RELOAD              pod/nginx-ingress-microk8s-controller-dnqkv                  NGINX reload triggered due to a change in configuration
default     0s          Normal    Pulled              pod/cm-acme-http-solver-9df5d                                Successfully pulled image "quay.io/jetstack/cert-manager-acmesolver:v1.8.0" in 18.433090934s (18.433100365s including waiting)
default     0s          Normal    Created             pod/cm-acme-http-solver-9df5d                                Created container acmesolver
default     0s          Normal    Started             pod/cm-acme-http-solver-9df5d                                Started container acmesolver
default     0s          Normal    Sync                ingress/cm-acme-http-solver-hg7j9                            Scheduled for sync
default     0s          Normal    Sync                ingress/cm-acme-http-solver-hg7j9                            Scheduled for sync
default     0s          Normal    Sync                ingress/cm-acme-http-solver-hg7j9                            Scheduled for sync
default     0s          Normal    Killing             pod/cm-acme-http-solver-9df5d                                Stopping container acmesolver
ingress     0s          Normal    RELOAD              pod/nginx-ingress-microk8s-controller-l46t4                  NGINX reload triggered due to a change in configuration
ingress     0s          Normal    RELOAD              pod/nginx-ingress-microk8s-controller-pxtlb                  NGINX reload triggered due to a change in configuration
default     0s          Warning   OrderFailed         certificaterequest/microbot-ingress-tls-tsmj9                Failed to wait for order resource "microbot-ingress-tls-tsmj9-3484024173" to become ready: order is in "invalid" state:
default     1s          Warning   Failed              certificate/microbot-ingress-tls                             The certificate request has failed to complete and will be retried: Failed to wait for order resource "microbot-ingress-tls-tsmj9-3484024173" to become ready: order is in "invalid" state:
ingress     1s          Normal    RELOAD              pod/nginx-ingress-microk8s-controller-dnqkv                  NGINX reload triggered due to a change in configuration
ingress     1s          Normal    RELOAD              pod/nginx-ingress-microk8s-controller-pxtlb                  NGINX reload triggered due to a change in configuration
ingress     1s          Normal    RELOAD              pod/nginx-ingress-microk8s-controller-l46t4                  NGINX reload triggered due to a change in configuration
kubectl describe certificaterequest microbot-ingress-tls-tsmj9
Name:         microbot-ingress-tls-tsmj9
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: microbot-ingress-tls
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: microbot-ingress-tls-p9j5v
API Version:  cert-manager.io/v1
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2022-12-21T18:52:59Z
  Generate Name:       microbot-ingress-tls-
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
          .:
          k:{"type":"Approved"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
    Manager:      cert-manager-certificaterequests-approver
    Operation:    Update
    Subresource:  status
    Time:         2022-12-21T18:52:59Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:cert-manager.io/certificate-name:
          f:cert-manager.io/certificate-revision:
          f:cert-manager.io/private-key-secret-name:
        f:generateName:
        f:ownerReferences:
          .:
          k:{"uid":"64aa8ec9-ce23-4a26-9637-d400783d1c3b"}:
      f:spec:
        .:
        f:issuerRef:
          .:
          f:group:
          f:kind:
          f:name:
        f:request:
        f:usages:
    Manager:      cert-manager-certificates-request-manager
    Operation:    Update
    Time:         2022-12-21T18:52:59Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          k:{"type":"Ready"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
        f:failureTime:
    Manager:      cert-manager-certificaterequests-issuer-acme
    Operation:    Update
    Subresource:  status
    Time:         2022-12-21T18:54:09Z
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  microbot-ingress-tls
    UID:                   64aa8ec9-ce23-4a26-9637-d400783d1c3b
  Resource Version:        27090
  UID:                     783f9bab-b1bf-4a59-8ce4-fcd5c4ccb88d
Spec:
  Extra:
    authentication.kubernetes.io/pod-name:
      cert-manager-69c6cb69f9-mwldw
    authentication.kubernetes.io/pod-uid:
      945ccb41-a5ae-4aaf-9535-f16dcb751809
  Groups:
    system:serviceaccounts
    system:serviceaccounts:cert-manager
    system:authenticated
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   lets-encrypt
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2dEQ0NBV2dDQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUtwaAp1eHpnaUk2blNUQUZFWEVYeVpxYit6eGpIY1c1ZWU0WDdKaHlXNThwMk1qZEZCQmVNNmg1WmQ2YlFTTVdCVFZKClowVUM3ZGk1Mms0QS9HMEl1OGpKbDR6cVRRS0pveVVuc2hwL3dhWkNtd1lrMUxmZ1czMmdvTmcySE5OR0wyRm8Kc1JDMHNadHFObFZEdXdtUUI2Y2dpRWhRR0ZwekZ0VlNYVjFPaEJ4R1JpdXo4TWtaRnVDeDRPZk9WVmJ1bmYyQQpBUWkwWHBGRXdWODdGMnk3aThLMldkVmc2cXY3dHhVaWduTVVSWDRmZVQ2bzF5R0hwdWZlRUJEL05oNmNXVk9SCndYb1ptS0plR09acU1jd2dLdE5tSlpwdzVGS2lMODlFMEtnY1lDUmZCTW9nalBXWkplSFVXOHgvZWplSEJDSG4KRldYUXhTaGZXdEVHZHlyVVdDTUNBd0VBQWFBN01Ea0dDU3FHU0liM0RRRUpEakVzTUNvd0d3WURWUjBSQkJRdwpFb0lRWkdGb2IyRnRMakV6YldGcGJDNWtaVEFMQmdOVkhROEVCQU1DQmFBd0RRWUpLb1pJaHZjTkFRRUxCUUFECmdnRUJBS1lGR05YVEwyM1RaMmliNCs3NDZIV3lVV1h4bW1SK01CQTNYaVJkOXlWNVY4NXp5b253bHdNdDExT0wKTTZoakR4VEVVZnpMcnJOaFgxcHpCaERIdDlSdDU2eWUxanppb3ZKcTVzTDQ0NktpUmpkNmI3dUNpT3Y0T0EwcAp0ZTVZUVIrSDd2cGFqMnBvY0tuNEtvUzNHTVZ5bjV0MnU4bkpyZDhHbFI1ZTBhSi9UQVhqcDhJaGVTbjFra3FNCmhieFdZbTJqM2JtTUZkMmpVblhpOC80ZTA5NVZEblpaVXV0azYxSzByZWVIUUVZaGZ5TkVna25Vb3BqS3JjaHAKVVdKbzU0NHlXYWsrcVFpZGNhazZDRlNPMEZOcGlpbWFWeGd1ZFFGU3EyZW9oUklnR0k5SndZVm5FRXFpYlU0cgpnSFBndFBXc29MSTJ0cGlqNzNWejZUbHVvdms9Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
  UID:      1e592ac4-61ca-435f-a6bf-b46b5389156a
  Usages:
    digital signature
    key encipherment
  Username:  system:serviceaccount:cert-manager:cert-manager
Status:
  Conditions:
    Last Transition Time:  2022-12-21T18:52:59Z
    Message:               Certificate request has been approved by cert-manager.io
    Reason:                cert-manager.io
    Status:                True
    Type:                  Approved
    Last Transition Time:  2022-12-21T18:52:59Z
    Message:               Failed to wait for order resource "microbot-ingress-tls-tsmj9-3484024173" to become ready: order is in "invalid" state: 
    Reason:                Failed
    Status:                False
    Type:                  Ready
  Failure Time:            2022-12-21T18:54:09Z
Events:
  Type     Reason           Age    From                                          Message
  ----     ------           ----   ----                                          -------
  Normal   cert-manager.io  4m16s  cert-manager-certificaterequests-approver     Certificate request has been approved by cert-manager.io
  Normal   OrderCreated     4m16s  cert-manager-certificaterequests-issuer-acme  Created Order resource default/microbot-ingress-tls-tsmj9-3484024173
  Normal   OrderPending     4m16s  cert-manager-certificaterequests-issuer-acme  Waiting on certificate issuance from order default/microbot-ingress-tls-tsmj9-3484024173: ""
  Warning  OrderFailed      3m6s   cert-manager-certificaterequests-issuer-acme  Failed to wait for order resource "microbot-ingress-tls-tsmj9-3484024173" to become ready: order is in "invalid" state:
kubectl describe certificate microbot-ingress-tls
Name:         microbot-ingress-tls
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2022-12-21T18:52:58Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          k:{"type":"Ready"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:observedGeneration:
            f:reason:
            f:status:
            f:type:
    Manager:      cert-manager-certificates-readiness
    Operation:    Update
    Subresource:  status
    Time:         2022-12-21T18:52:58Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
          .:
          k:{"type":"Issuing"}:
            .:
            f:observedGeneration:
            f:type:
    Manager:      cert-manager-certificates-trigger
    Operation:    Update
    Subresource:  status
    Time:         2022-12-21T18:52:58Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:ownerReferences:
          .:
          k:{"uid":"18e12242-3f02-4277-9814-3450face7ba8"}:
      f:spec:
        .:
        f:dnsNames:
        f:issuerRef:
          .:
          f:group:
          f:kind:
          f:name:
        f:secretName:
        f:usages:
    Manager:      cert-manager-ingress-shim
    Operation:    Update
    Time:         2022-12-21T18:52:58Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          k:{"type":"Issuing"}:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
        f:failedIssuanceAttempts:
        f:lastFailureTime:
    Manager:      cert-manager-certificates-issuing
    Operation:    Update
    Subresource:  status
    Time:         2022-12-21T18:54:09Z
  Owner References:
    API Version:           networking.k8s.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  microbot-ingress
    UID:                   18e12242-3f02-4277-9814-3450face7ba8
  Resource Version:        27100
  UID:                     64aa8ec9-ce23-4a26-9637-d400783d1c3b
Spec:
  Dns Names:
    something.mydomain.com
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       ClusterIssuer
    Name:       lets-encrypt
  Secret Name:  microbot-ingress-tls
  Usages:
    digital signature
    key encipherment
Status:
  Conditions:
    Last Transition Time:    2022-12-21T18:54:09Z
    Message:                 The certificate request has failed to complete and will be retried: Failed to wait for order resource "microbot-ingress-tls-tsmj9-3484024173" to become ready: order is in "invalid" state: 
    Observed Generation:     1
    Reason:                  Failed
    Status:                  False
    Type:                    Issuing
    Last Transition Time:    2022-12-21T18:52:58Z
    Message:                 Issuing certificate as Secret does not exist
    Observed Generation:     1
    Reason:                  DoesNotExist
    Status:                  False
    Type:                    Ready
  Failed Issuance Attempts:  1
  Last Failure Time:         2022-12-21T18:54:09Z
Events:
  Type     Reason     Age    From                                       Message
  ----     ------     ----   ----                                       -------
  Normal   Issuing    4m53s  cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal   Generated  4m52s  cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "microbot-ingress-tls-p9j5v"
  Normal   Requested  4m52s  cert-manager-certificates-request-manager  Created new CertificateRequest resource "microbot-ingress-tls-tsmj9"
  Warning  Failed     3m42s  cert-manager-certificates-issuing          The certificate request has failed to complete and will be retried: Failed to wait for order resource "microbot-ingress-tls-tsmj9-3484024173" to become ready: order is in "invalid" state:

I am at my wit’s end and am grateful for any help

During the

getting the next error:
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded

@Stanislav_Trifan sorry it isn’t working for you. That error indicates the pod isn’t being created in the expected timeframe, which could be for a number of reasons. Could you paste here the exact YAML of the resource you tried to create?

@evilnick this is the command I run (on microk8s 1.27.2 rev 5372):


stanislav@dev:~/workspace/homelab/kubecluster$ kubectl apply -f - <<EOF
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: lets-encrypt
spec:
 acme:
   email: _edited_@gmail.com
   server: https://acme-v02.api.letsencrypt.org/directory
   privateKeySecretRef:
     # Secret resource that will be used to store the account's private key.
     name: lets-encrypt-priviate-key
   # Add a single challenge solver, HTTP01 using nginx
   solvers:
   - http01:
       ingress:
         class: public
EOF

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded

@Stanislav_Trifan since I cannot reproduce the issue, could you please file an issue on Issues · canonical/microk8s · GitHub and attach a microk8s inspect tarball?

1 Like

Intead of microk8s-ingress-tls it should be microbot-ingress-tls

1 Like

There’s a typo in the original instructions—lets-encrypt-priviate-key should be lets-encrypt-private-key.

I know it does not impact anything.

Solver’s gets wrong class following instructions, need to manually set.

Here are my logs and solution from ChatGPT.

followed instructions on MicroK8s - Addon: cert-manager
but ended up with solvers with class .

I enabled all the required packages, and even rebooted the server.

my ClusterIssuer is as follows:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: lets-encrypt # Changed to match MicroK8s instructions
spec:
acme:
email: john_grabner@hotmail.com # Your email address
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: lets-encrypt-private-key # Corrected and matched name
solvers:
- http01:
ingress:
class: public

*** later you will see solver do not get class public but class . Not sure if this is an issue, but looks odd.

it is ready as can be seen here:

icrok8s kubectl get clusterissuer -o wide
NAME           READY   STATUS                                                 AGE
lets-encrypt   True    The ACME account was registered with the ACME server   23h

my service is running as can be seen here:

microk8s kubectl get pod,svc
NAME                                      READY   STATUS    RESTARTS        AGE
pod/cm-acme-http-solver-2dfxw             1/1     Running   2 (7m21s ago)   20h
pod/cm-acme-http-solver-9p244             1/1     Running   2 (7m21s ago)   20h
pod/express-deployment-7bc8645c58-kn6f4   1/1     Running   2 (7m21s ago)   21h
pod/mysql-deployment-59b74b59f9-4t8x8     2/2     Running   4 (7m21s ago)   41h

NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
service/cm-acme-http-solver-cf2sq   NodePort    10.152.183.29    <none>        8089:31437/TCP       20h
service/cm-acme-http-solver-dffrw   NodePort    10.152.183.222   <none>        8089:30921/TCP       20h
service/express-service             ClusterIP   10.152.183.208   <none>        3000/TCP,55001/TCP   21h
service/kubernetes                  ClusterIP   10.152.183.1     <none>        443/TCP              43h
service/mysql-service               NodePort    10.152.183.189   <none>        3306:30306/TCP       41h

my ingress is as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-before-ttl
  annotations:
    cert-manager.io/cluster-issuer: "lets-encrypt"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/proxy-body-size: 16m
spec:
  tls:
    - hosts:
        - ancient-script.org
        - www.ancient-script.org
      secretName: ancient-script-org-crt-secret
  rules:
    - host: ancient-script.org
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: express-service
                port:
                  number: 3000
    - host: www.ancient-script.org
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: express-service
                port:
                  number: 3000

k describe ingress

Name:             cm-acme-http-solver-rzl2p
Labels:           acme.cert-manager.io/http-domain=1683425454
                  acme.cert-manager.io/http-token=829886008
                  acme.cert-manager.io/http01-solver=true
Namespace:        default
Address:          127.0.0.1
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  www.ancient-script.org  
                          /.well-known/acme-challenge/6c1PhtRjg3Dz-MLKeDXe1mES-0YsZjtzN5NON17zwks   cm-acme-http-solver-cf2sq:8089 (10.1.96.84:8089)
Annotations:              kubernetes.io/ingress.class: public
                          nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
Events:                   <none>


Name:             cm-acme-http-solver-w4bdc
Labels:           acme.cert-manager.io/http-domain=1134561051
                  acme.cert-manager.io/http-token=1348800303
                  acme.cert-manager.io/http01-solver=true
Namespace:        default
Address:          127.0.0.1
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host                Path  Backends
  ----                ----  --------
  ancient-script.org  
                      /.well-known/acme-challenge/WkTcEHzZlXlUo3mtd4E2KKLzhSzynnz5bh5e7N7Yw_Y   cm-acme-http-solver-dffrw:8089 (10.1.96.93:8089)
Annotations:          kubernetes.io/ingress.class: public
                      nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
Events:               <none>


Name:             ingress-before-ttl
Labels:           <none>
Namespace:        default
Address:          127.0.0.1
Ingress Class:    public
Default backend:  <default>
TLS:
  ancient-script-org-crt-secret terminates ancient-script.org,www.ancient-script.org
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  ancient-script.org      
                          /   express-service:3000 (10.1.96.96:3000)
  www.ancient-script.org  
                          /   express-service:3000 (10.1.96.96:3000)
Annotations:              cert-manager.io/cluster-issuer: lets-encrypt
                          nginx.ingress.kubernetes.io/affinity: cookie
                          nginx.ingress.kubernetes.io/proxy-body-size: 16m
Events:                   <none>

k describe certificate

Name:         ancient-script-org-crt-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2024-10-08T16:43:08Z
  Generation:          1
  Owner References:
    API Version:           networking.k8s.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  ingress-before-ttl
    UID:                   f065e265-d6ab-46a7-a5bd-507d0226329d
  Resource Version:        237082
  UID:                     69adfc17-9572-4f90-a8b2-db0deaf6606b
Spec:
  Dns Names:
    ancient-script.org
    www.ancient-script.org
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       ClusterIssuer
    Name:       lets-encrypt
  Secret Name:  ancient-script-org-crt-secret
  Usages:
    digital signature
    key encipherment
Status:
  Conditions:
    Last Transition Time:        2024-10-08T16:43:08Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2024-10-08T16:43:08Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  ancient-script-org-crt-secret-4k8wr
Events:                          <none>

k -n cert-manager logs cert-manager-cainjector-dc95f9d66-m449x

I1009 13:28:16.672686       1 start.go:126] "starting" version="v1.8.0" revision="e466a521bc5455def8c224599c6edcd37e86410c"
I1009 13:28:26.691560       1 leaderelection.go:248] attempting to acquire leader lease kube-system/cert-manager-cainjector-leader-election...
I1009 13:29:47.067146       1 leaderelection.go:258] successfully acquired lease kube-system/cert-manager-cainjector-leader-election
I1009 13:29:47.067324       1 recorder.go:103] cert-manager/events "msg"="Normal"  "message"="cert-manager-cainjector-dc95f9d66-m449x_de1669f2-a803-4069-ac81-8f2c061ecf44 became leader" "object"={"kind":"Lease","namespace":"kube-system","name":"cert-manager-cainjector-leader-election","uid":"34db52f2-e5dc-4643-b33c-911f30212a0c","apiVersion":"coordination.k8s.io/v1","resourceVersion":"241297"} "reason"="LeaderElection"
I1009 13:29:47.168218       1 controller.go:178] cert-manager/certificate/mutatingwebhookconfiguration/controller/controller-for-certificate-mutatingwebhookconfiguration "msg"="Starting EventSource"  "source"="&{{%!s(*v1.MutatingWebhookConfiguration=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} []}) %!s(*cache.informerCache=&{0xc00050e360}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168226       1 controller.go:178] cert-manager/secret/customresourcedefinition/controller/controller-for-secret-customresourcedefinition "msg"="Starting EventSource"  "source"="&{{%!s(*v1.CustomResourceDefinition=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} { {  []   []}  [] <nil> false} {[] {  []   []} []}}) %!s(*cache.informerCache=&{0xc000720900}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168251       1 controller.go:178] cert-manager/certificate/mutatingwebhookconfiguration/controller/controller-for-certificate-mutatingwebhookconfiguration "msg"="Starting EventSource"  "source"="&{{%!s(*v1.Certificate=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} {<nil>  <nil> <nil> [] [] [] []  <nil> <nil> {  } false [] <nil> <nil> <nil> []} {[] <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}) %!s(*cache.informerCache=&{0xc00050e360}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168258       1 controller.go:178] cert-manager/secret/customresourcedefinition/controller/controller-for-secret-customresourcedefinition "msg"="Starting EventSource"  "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0xc000720900}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168271       1 controller.go:186] cert-manager/secret/customresourcedefinition/controller/controller-for-secret-customresourcedefinition "msg"="Starting Controller"  
I1009 13:29:47.168273       1 controller.go:178] cert-manager/certificate/mutatingwebhookconfiguration/controller/controller-for-certificate-mutatingwebhookconfiguration "msg"="Starting EventSource"  "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0xc00050e360}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168291       1 controller.go:186] cert-manager/certificate/mutatingwebhookconfiguration/controller/controller-for-certificate-mutatingwebhookconfiguration "msg"="Starting Controller"  
I1009 13:29:47.168321       1 controller.go:178] cert-manager/certificate/apiservice/controller/controller-for-certificate-apiservice "msg"="Starting EventSource"  "source"="&{{%!s(*v1.APIService=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} {<nil>   false [] 0 0} {[]}}) %!s(*cache.informerCache=&{0xc00050e360}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168352       1 controller.go:178] cert-manager/certificate/apiservice/controller/controller-for-certificate-apiservice "msg"="Starting EventSource"  "source"="&{{%!s(*v1.Certificate=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} {<nil>  <nil> <nil> [] [] [] []  <nil> <nil> {  } false [] <nil> <nil> <nil> []} {[] <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}) %!s(*cache.informerCache=&{0xc00050e360}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168370       1 controller.go:178] cert-manager/certificate/apiservice/controller/controller-for-certificate-apiservice "msg"="Starting EventSource"  "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0xc00050e360}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168377       1 controller.go:186] cert-manager/certificate/apiservice/controller/controller-for-certificate-apiservice "msg"="Starting Controller"  
I1009 13:29:47.168418       1 controller.go:178] cert-manager/secret/validatingwebhookconfiguration/controller/controller-for-secret-validatingwebhookconfiguration "msg"="Starting EventSource"  "source"="&{{%!s(*v1.ValidatingWebhookConfiguration=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} []}) %!s(*cache.informerCache=&{0xc000720900}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168446       1 controller.go:178] cert-manager/secret/validatingwebhookconfiguration/controller/controller-for-secret-validatingwebhookconfiguration "msg"="Starting EventSource"  "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0xc000720900}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168459       1 controller.go:186] cert-manager/secret/validatingwebhookconfiguration/controller/controller-for-secret-validatingwebhookconfiguration "msg"="Starting Controller"  
I1009 13:29:47.168580       1 controller.go:178] cert-manager/secret/mutatingwebhookconfiguration/controller/controller-for-secret-mutatingwebhookconfiguration "msg"="Starting EventSource"  "source"="&{{%!s(*v1.MutatingWebhookConfiguration=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} []}) %!s(*cache.informerCache=&{0xc000720900}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168611       1 controller.go:178] cert-manager/secret/apiservice/controller/controller-for-secret-apiservice "msg"="Starting EventSource"  "source"="&{{%!s(*v1.APIService=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} {<nil>   false [] 0 0} {[]}}) %!s(*cache.informerCache=&{0xc000720900}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168642       1 controller.go:178] cert-manager/certificate/validatingwebhookconfiguration/controller/controller-for-certificate-validatingwebhookconfiguration "msg"="Starting EventSource"  "source"="&{{%!s(*v1.ValidatingWebhookConfiguration=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} []}) %!s(*cache.informerCache=&{0xc00050e360}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168646       1 controller.go:178] cert-manager/secret/apiservice/controller/controller-for-secret-apiservice "msg"="Starting EventSource"  "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0xc000720900}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168661       1 controller.go:178] cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-customresourcedefinition "msg"="Starting EventSource"  "source"="&{{%!s(*v1.CustomResourceDefinition=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} { {  []   []}  [] <nil> false} {[] {  []   []} []}}) %!s(*cache.informerCache=&{0xc00050e360}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168669       1 controller.go:178] cert-manager/certificate/validatingwebhookconfiguration/controller/controller-for-certificate-validatingwebhookconfiguration "msg"="Starting EventSource"  "source"="&{{%!s(*v1.Certificate=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} {<nil>  <nil> <nil> [] [] [] []  <nil> <nil> {  } false [] <nil> <nil> <nil> []} {[] <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}) %!s(*cache.informerCache=&{0xc00050e360}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168667       1 controller.go:186] cert-manager/secret/apiservice/controller/controller-for-secret-apiservice "msg"="Starting Controller"  
I1009 13:29:47.168690       1 controller.go:178] cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-customresourcedefinition "msg"="Starting EventSource"  "source"="&{{%!s(*v1.Certificate=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} {<nil>  <nil> <nil> [] [] [] []  <nil> <nil> {  } false [] <nil> <nil> <nil> []} {[] <nil> <nil> <nil> <nil> <nil> <nil> <nil>}}) %!s(*cache.informerCache=&{0xc00050e360}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168643       1 controller.go:178] cert-manager/secret/mutatingwebhookconfiguration/controller/controller-for-secret-mutatingwebhookconfiguration "msg"="Starting EventSource"  "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0xc000720900}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168707       1 controller.go:178] cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-customresourcedefinition "msg"="Starting EventSource"  "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0xc00050e360}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168682       1 controller.go:178] cert-manager/certificate/validatingwebhookconfiguration/controller/controller-for-certificate-validatingwebhookconfiguration "msg"="Starting EventSource"  "source"="&{{%!s(*v1.Secret=&{{ } {      0 {{0 0 <nil>}} <nil> <nil> map[] map[] [] []  []} <nil> map[] map[] }) %!s(*cache.informerCache=&{0xc00050e360}) %!s(chan error=<nil>) %!s(func()=<nil>)}}"
I1009 13:29:47.168722       1 controller.go:186] cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-customresourcedefinition "msg"="Starting Controller"  
I1009 13:29:47.168742       1 controller.go:186] cert-manager/secret/mutatingwebhookconfiguration/controller/controller-for-secret-mutatingwebhookconfiguration "msg"="Starting Controller"  
I1009 13:29:47.168765       1 controller.go:186] cert-manager/certificate/validatingwebhookconfiguration/controller/controller-for-certificate-validatingwebhookconfiguration "msg"="Starting Controller"  
I1009 13:29:47.269434       1 controller.go:220] cert-manager/certificate/mutatingwebhookconfiguration/controller/controller-for-certificate-mutatingwebhookconfiguration "msg"="Starting workers"  "worker count"=1
I1009 13:29:47.269514       1 controller.go:220] cert-manager/secret/mutatingwebhookconfiguration/controller/controller-for-secret-mutatingwebhookconfiguration "msg"="Starting workers"  "worker count"=1
I1009 13:29:47.269541       1 controller.go:220] cert-manager/certificate/validatingwebhookconfiguration/controller/controller-for-certificate-validatingwebhookconfiguration "msg"="Starting workers"  "worker count"=1
I1009 13:29:47.269552       1 controller.go:220] cert-manager/secret/validatingwebhookconfiguration/controller/controller-for-secret-validatingwebhookconfiguration "msg"="Starting workers"  "worker count"=1
I1009 13:29:47.269552       1 controller.go:220] cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-customresourcedefinition "msg"="Starting workers"  "worker count"=1
I1009 13:29:47.269578       1 controller.go:220] cert-manager/certificate/apiservice/controller/controller-for-certificate-apiservice "msg"="Starting workers"  "worker count"=1
I1009 13:29:47.269585       1 controller.go:220] cert-manager/secret/customresourcedefinition/controller/controller-for-secret-customresourcedefinition "msg"="Starting workers"  "worker count"=1
I1009 13:29:47.269609       1 controller.go:220] cert-manager/secret/apiservice/controller/controller-for-secret-apiservice "msg"="Starting workers"  "worker count"=1
I1009 13:29:47.274745       1 controller.go:178] cert-manager/secret/mutatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1" 
I1009 13:29:47.274946       1 controller.go:178] cert-manager/secret/validatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1" 
I1009 13:29:47.277662       1 controller.go:178] cert-manager/secret/mutatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1" 
I1009 13:29:47.320780       1 controller.go:178] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="issuers.cert-manager.io" "resource_namespace"="" "resource_version"="v1" 
I1009 13:29:47.329779       1 controller.go:178] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="orders.acme.cert-manager.io" "resource_namespace"="" "resource_version"="v1" 

k -n cert-manager logs cert-manager-d5fcf78bc-xbbjr
I1009 13:28:16.753875       1 start.go:75] cert-manager "msg"="starting controller"  "git-commit"="e466a521bc5455def8c224599c6edcd37e86410c" "version"="v1.8.0"
I1009 13:28:16.753958       1 controller.go:242] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["10.152.183.10:53"] 
W1009 13:28:16.754403       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1009 13:28:16.759485       1 controller.go:70] cert-manager/controller "msg"="enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers ingress-shim issuers orders]"  
I1009 13:28:16.759893       1 controller.go:134] cert-manager/controller "msg"="starting leader election"  
I1009 13:28:16.760044       1 controller.go:91] cert-manager/controller "msg"="starting metrics server"  "address"={"IP":"::","Port":9402,"Zone":""}
I1009 13:28:16.761026       1 leaderelection.go:248] attempting to acquire leader lease kube-system/cert-manager-controller...
I1009 13:28:16.779671       1 leaderelection.go:258] successfully acquired lease kube-system/cert-manager-controller
I1009 13:28:16.780769       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-acme" 
I1009 13:28:16.785798       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca" 
I1009 13:28:16.786334       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-issuing" 
I1009 13:28:16.786707       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-ca" 
I1009 13:28:16.786728       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-selfsigned" 
I1009 13:28:16.786752       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-vault" 
I1009 13:28:16.786795       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-revision-manager" 
I1009 13:28:16.787342       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="orders" 
I1009 13:28:16.787584       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers" 
I1009 13:28:16.788110       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="issuers" 
I1009 13:28:16.788357       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="gateway-shim" 
I1009 13:28:16.788433       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-approver" 
I1009 13:28:16.792251       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim" 
I1009 13:28:16.792694       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-selfsigned" 
I1009 13:28:16.793091       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault" 
I1009 13:28:16.793312       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-venafi" 
I1009 13:28:16.793386       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-request-manager" 
I1009 13:28:16.898058       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="challenges" 
I1009 13:28:16.898561       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-key-manager" 
I1009 13:28:16.898986       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-metrics" 
I1009 13:28:16.899422       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-readiness" 
I1009 13:28:16.899684       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-acme" 
I1009 13:28:16.899781       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-trigger" 
I1009 13:28:16.900181       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-venafi" 
I1009 13:28:16.918352       1 util.go:84] cert-manager/controller/certificaterequests-issuer-acme/handleOwnedResource "msg"="owning resource not found in cache" "related_resource_kind"="CertificateRequest" "related_resource_name"="ancient-script-org-crt-secret-cczw9" "related_resource_namespace"="default" "resource_kind"="Order" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274" "resource_namespace"="default" "resource_version"="v1" 
E1009 13:28:16.987548       1 controller.go:166] cert-manager/orders "msg"="re-queuing item due to error processing" "error"="ACME client for issuer not initialised/available" "key"="default/ancient-script-org-crt-secret-cczw9-2481331274" 
I1009 13:28:16.988641       1 setup.go:202] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="lets-encrypt-private-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="lets-encrypt" "resource_namespace"="" "resource_version"="v1" 
I1009 13:28:16.999452       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="ancient-script.org" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-9p244" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:16.999454       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="www.ancient-script.org" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-2dfxw" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:16.999524       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="ancient-script.org" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-dffrw" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:16.999546       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="www.ancient-script.org" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-cf2sq" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:16.999572       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="ancient-script.org" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-w4bdc" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:16.999607       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="www.ancient-script.org" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-rzl2p" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
E1009 13:28:27.000640       1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://ancient-script.org/.well-known/acme-challenge/WkTcEHzZlXlUo3mtd4E2KKLzhSzynnz5bh5e7N7Yw_Y': Get \"http://ancient-script.org/.well-known/acme-challenge/WkTcEHzZlXlUo3mtd4E2KKLzhSzynnz5bh5e7N7Yw_Y\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" "dnsName"="ancient-script.org" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
E1009 13:28:27.000662       1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://www.ancient-script.org/.well-known/acme-challenge/6c1PhtRjg3Dz-MLKeDXe1mES-0YsZjtzN5NON17zwks': Get \"http://www.ancient-script.org/.well-known/acme-challenge/6c1PhtRjg3Dz-MLKeDXe1mES-0YsZjtzN5NON17zwks\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" "dnsName"="www.ancient-script.org" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
E1009 13:28:31.990103       1 controller.go:166] cert-manager/orders "msg"="re-queuing item due to error processing" "error"="context deadline exceeded" "key"="default/ancient-script-org-crt-secret-cczw9-2481331274" 
E1009 13:28:37.006482       1 controller.go:166] cert-manager/challenges "msg"="re-queuing item due to error processing" "error"="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s\": context deadline exceeded" "key"="default/ancient-script-org-crt-secret-cczw9-2481331274-3325911860" 
E1009 13:28:37.006486       1 controller.go:166] cert-manager/challenges "msg"="re-queuing item due to error processing" "error"="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s\": context deadline exceeded" "key"="default/ancient-script-org-crt-secret-cczw9-2481331274-1378496233" 
I1009 13:28:37.006632       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="ancient-script.org" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-9p244" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:37.006702       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="ancient-script.org" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-dffrw" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:37.006755       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="www.ancient-script.org" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-2dfxw" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:37.006764       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="ancient-script.org" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-w4bdc" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:37.006820       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="www.ancient-script.org" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-cf2sq" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:37.006881       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="www.ancient-script.org" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-rzl2p" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
E1009 13:28:47.007028       1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://www.ancient-script.org/.well-known/acme-challenge/6c1PhtRjg3Dz-MLKeDXe1mES-0YsZjtzN5NON17zwks': Get \"http://www.ancient-script.org/.well-known/acme-challenge/6c1PhtRjg3Dz-MLKeDXe1mES-0YsZjtzN5NON17zwks\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" "dnsName"="www.ancient-script.org" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
E1009 13:28:47.007059       1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://ancient-script.org/.well-known/acme-challenge/WkTcEHzZlXlUo3mtd4E2KKLzhSzynnz5bh5e7N7Yw_Y': Get \"http://ancient-script.org/.well-known/acme-challenge/WkTcEHzZlXlUo3mtd4E2KKLzhSzynnz5bh5e7N7Yw_Y\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" "dnsName"="ancient-script.org" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
E1009 13:28:51.991353       1 controller.go:166] cert-manager/orders "msg"="re-queuing item due to error processing" "error"="context deadline exceeded" "key"="default/ancient-script-org-crt-secret-cczw9-2481331274" 
E1009 13:28:57.011831       1 controller.go:166] cert-manager/challenges "msg"="re-queuing item due to error processing" "error"="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s\": context deadline exceeded" "key"="default/ancient-script-org-crt-secret-cczw9-2481331274-1378496233" 
E1009 13:28:57.011936       1 controller.go:166] cert-manager/challenges "msg"="re-queuing item due to error processing" "error"="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s\": context deadline exceeded" "key"="default/ancient-script-org-crt-secret-cczw9-2481331274-3325911860" 
I1009 13:28:57.012031       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="www.ancient-script.org" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-2dfxw" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:57.012099       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="www.ancient-script.org" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-cf2sq" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:57.012119       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="ancient-script.org" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-9p244" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:57.012160       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="www.ancient-script.org" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-rzl2p" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:57.012187       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="ancient-script.org" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-dffrw" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:28:57.012242       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="ancient-script.org" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-w4bdc" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
E1009 13:29:07.012540       1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://ancient-script.org/.well-known/acme-challenge/WkTcEHzZlXlUo3mtd4E2KKLzhSzynnz5bh5e7N7Yw_Y': Get \"http://ancient-script.org/.well-known/acme-challenge/WkTcEHzZlXlUo3mtd4E2KKLzhSzynnz5bh5e7N7Yw_Y\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" "dnsName"="ancient-script.org" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
E1009 13:29:07.012571       1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://www.ancient-script.org/.well-known/acme-challenge/6c1PhtRjg3Dz-MLKeDXe1mES-0YsZjtzN5NON17zwks': Get \"http://www.ancient-script.org/.well-known/acme-challenge/6c1PhtRjg3Dz-MLKeDXe1mES-0YsZjtzN5NON17zwks\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" "dnsName"="www.ancient-script.org" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
E1009 13:29:17.018327       1 controller.go:166] cert-manager/challenges "msg"="re-queuing item due to error processing" "error"="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s\": context deadline exceeded" "key"="default/ancient-script-org-crt-secret-cczw9-2481331274-1378496233" 
E1009 13:29:17.018330       1 controller.go:166] cert-manager/challenges "msg"="re-queuing item due to error processing" "error"="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s\": context deadline exceeded" "key"="default/ancient-script-org-crt-secret-cczw9-2481331274-3325911860" 
I1009 13:29:17.018531       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="ancient-script.org" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-9p244" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:29:17.018570       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="www.ancient-script.org" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-2dfxw" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:29:17.018616       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="ancient-script.org" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-dffrw" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:29:17.018665       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="www.ancient-script.org" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-cf2sq" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:29:17.018705       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="ancient-script.org" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-w4bdc" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:29:17.018760       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="www.ancient-script.org" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-rzl2p" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
E1009 13:29:17.130108       1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="www.ancient-script.org" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
E1009 13:29:17.222102       1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="ancient-script.org" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:29:27.130744       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="www.ancient-script.org" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-2dfxw" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:29:27.130812       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="www.ancient-script.org" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-cf2sq" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:29:27.130863       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="www.ancient-script.org" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-rzl2p" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
E1009 13:29:27.143235       1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="www.ancient-script.org" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:29:27.222713       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="ancient-script.org" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-9p244" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:29:27.222784       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="ancient-script.org" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-dffrw" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:29:27.222849       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="ancient-script.org" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-w4bdc" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
E1009 13:29:27.233812       1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="ancient-script.org" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-3325911860" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:29:37.143602       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="www.ancient-script.org" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-2dfxw" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="ancient-script-org-crt-secret-cczw9-2481331274-1378496233" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01" 
I1009 13:29:37.143678       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="www.ancient-script.org" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-cf2sq" "related_resource_namespace"="default" "related_resource_version"="v1"

k get ingress -w

NAME                        CLASS    HOSTS                                       ADDRESS     PORTS     AGE
cm-acme-http-solver-rzl2p   <none>   www.ancient-script.org                      127.0.0.1   80        21h
cm-acme-http-solver-w4bdc   <none>   ancient-script.org                          127.0.0.1   80        21h
ingress-before-ttl          public   ancient-script.org,www.ancient-script.org   127.0.0.1   80, 443   21h

**** notice the class for the solvers are while the ingress is public. Not sure if this is an issue, but looks strange.

**** nodejs server does get ancient-script.org/.well-known/acme-challenge and returns a 404 since nodejs does not implement this. Further implying that the solver are not properly hooked in.

I can access my website with HTTP://ancient-script.org from the cloud. i.e. DNS is fine and routing to server is fine.

chatgpt recommended:

Since the HTTP solver ingresses don’t have the public class, you can manually patch the ingresses for the solver to include it.

To patch the solver ingress, use the following command:

bash
Copy code

microk8s kubectl patch ingress cm-acme-http-solver-nf7ld -p '{"spec": {"ingressClassName": "public"}}'
microk8s kubectl patch ingress cm-acme-http-solver-s4fh6 -p '{"spec": {"ingressClassName": "public"}}'

This will update the solver ingresses to use the correct ingress class, ensuring that traffic for the ACME challenge paths is routed to the solver pods instead of your application.

k describe certificate
Name:         ancient-script-org-crt-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2024-10-09T14:07:17Z
  Generation:          1
  Owner References:
    API Version:           networking.k8s.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  ingress-before-ttl
    UID:                   1f65fec2-e3e2-4bfd-b5ae-f69f4235c3f2
  Resource Version:        248239
  UID:                     8de68bd1-fe9f-4824-a0b3-3a7455b56770
Spec:
  Dns Names:
    ancient-script.org
    www.ancient-script.org
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       ClusterIssuer
    Name:       lets-encrypt
  Secret Name:  ancient-script-org-crt-secret
  Usages:
    digital signature
    key encipherment
Status:
  Conditions:
    Last Transition Time:        2024-10-09T14:07:17Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
    Last Transition Time:        2024-10-09T14:07:17Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
  Next Private Key Secret Name:  ancient-script-org-crt-secret-m727d
Events:
  Type    Reason     Age    From                                       Message
  ----    ------     ----   ----                                       -------
  Normal  Issuing    2m53s  cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  2m53s  cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "ancient-script-org-crt-secret-m727d"
  Normal  Requested  2m53s  cert-manager-certificates-request-manager  Created new CertificateRequest resource "ancient-script-org-crt-secret-p6mm4"

still have issues, so not full solution.

Hi there, I ended up installing cert-manager myself as it’s pretty easy using argocd + their helm chart.

I did use the example ingress from here, and it contains pathType: Exact instead of pathType: Prefix. This tripped me up for ~15 minutes as “for some reason the image of the microbot example was not loading”. I’m running the cluster behind another ngnix reversed proxy, which means adding some header forwarding etc., so once I was able to acces the index page on the container I assumed some misconfiguration there was the issue for the image still returning a 404.

Turns out that the Ingress ‘does what it says’, and the example pathType: Exact will only match the exact url https://my-service.example.com without a subpath, which will serve up the index page as that is configured in the microbot image, so not https://my-service.example.com/microbot.png.

I suggest the docs shoud be changed to use pathType: Prefix.

{ Original post quoted the entire article, then inserted irrelevant links}

@saqrio Why are you trying to insert this into the instructions?

@evilnick
I followed the setps and i got below error

microk8s kubectl get certificates -A
NAMESPACE NAME READY SECRET AGE
cert-manager test-50gramx-com False tls-secret 84m
[ec2-user@ip-172-31-47-8 test]$ microk8s kubectl describe certificates test-50gramx-com -n cert-manager
Name: test-50gramx-com
Namespace: cert-manager
Labels:
Annotations:
API Version: cert-manager.io/v1
Kind: Certificate
Metadata:
Creation Timestamp: 2024-12-02T09:15:55Z
Generation: 1
Resource Version: 371697
UID: 4dda1a78-f6ed-4328-89f6-4af90eb321ee
Spec:
Common Name: test.50gramx.com
Dns Names:
test.50gramx.com
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-staging
Secret Name: tls-secret
Status:
Conditions:
Last Transition Time: 2024-12-02T09:15:55Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 1
Reason: DoesNotExist
Status: False
Type: Ready
Last Transition Time: 2024-12-02T09:15:55Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 1
Reason: DoesNotExist
Status: True
Type: Issuing
Next Private Key Secret Name: test-50gramx-com-4p4gw
Events: