"DeltaFIFO Pop Process" - Reason:slow event handlers blocking the queue

General Discussions
1

Hello,

CONTEXT:

I accidently deleted the namespace where Cert-manager runs. After redeploying (static manifest) Cert-manager’s webhook was failing to generate initial serving certificate but gladly I had a backup of the secret and I restored them(cert-manager-webhook-ca & cert-manager-webhook-tls).

Now webhook pod is working with no issues. However:

PROBLEM:

Cert-manager’s pod was working fine with no errors in the logs but after fixing the webhook, in the Cert-manager’s pod logs I noticed these messages (repeatedly with different IDs) here’s an example of one of the logs messages:

Trace[1788197141]: "DeltaFIFO Pop Process" ID:namespace/model-secrets,Depth:189,Reason:slow event handlers blocking the queue

I have also created an ingress in a different namespace and deleted it but i’d still see this in the certmanager’s logs:

ingress 'microservices/test-ingress' in work queue no longer exists

ATTEMPTS:

I increased the number of replicas in the Cert-manager Deployment from 1 to 2 replicas.

I increased the resources request and limits in the Cert-manager Deployment.

I created a ConfigMap and specified:

deltafifo-queue-history-size: "1056"

But none of the above attempts worked…

Environment:

Cert-manager : v1.10.0
Kubernetes: 1.21.14-gke.4300