Slow kubeapi response for pod creation possibly due to

Recently, new pod creation has significantly slowed, from 600ms → 8s.

Checking via kubectl -v6 i’ve isolated the problem to the kubeapi response to the create post, all other responses are very fast:

I0524 10:19:49.123517 3822186 round_trippers.go:454] POST https://etcd-01.mycluster:6443/api/v1/namespaces/default/pods?fieldManager=kubectl-client-side-apply 201 Created in **7795 milliseconds** pod/my_new_pod created 

On the kubeapi logs i see:

I0524 08:21:23.717701       1 trace.go:205] Trace[449263730]: "Call mutating webhook" configuration:stork-webhooks-cfg,,resource:/v1, Resource=pods,subresource:,operation:CREATE,UID:5ed0438a-551b-418d-8a93-5f9a0c91e6d3 (24-May-2023 08:21:16.080) (total time: 7636ms):
Trace[449263730]: ---"Request completed" 7636ms (08:21:23.717)
Trace[449263730]: [7.636966753s] [7.636966753s] END

I0524 08:21:23.722929       1 trace.go:205] Trace[1404157220]: "Create" url:/api/v1/namespaces/default/pods,user-agent:kubectl/v1.22.2 (linux/amd64) kubernetes/8b5a191,audit-id:fb3eea6b-629d-4529-aaa5-5029cbd76c52,client:,accept:application/json,protocol:HTTP/2.0 (24-May-2023 08:21:16.076) (total time: 7646ms):
Trace[1404157220]: ---"Object stored in database" 7645ms (08:21:23.722)
Trace[1404157220]: [7.64680376s] [7.64680376s] END

Recently, the response times went up from 600ms to ~8s. I suspect the mutating webhook ( to block the response for most of the time – looking in the logs, this webhook also responded in ~600ms previously, before times went up.

Any hints on how to investigate this further? I am unsure which component stork relates to, and how to find logs that could point me in the right direction.

Cluster information:

Kubernetes version: 1.22.4
Cloud being used: bare metal
Host OS: Centos 7
CNI and version:, portworx:23.3.1