Recently, new pod creation has significantly slowed, from 600ms → 8s.
Checking via kubectl -v6 i’ve isolated the problem to the kubeapi response to the create post, all other responses are very fast:
I0524 10:19:49.123517 3822186 round_trippers.go:454] POST https://etcd-01.mycluster:6443/api/v1/namespaces/default/pods?fieldManager=kubectl-client-side-apply 201 Created in **7795 milliseconds** pod/my_new_pod created
On the kubeapi logs i see:
I0524 08:21:23.717701 1 trace.go:205] Trace[449263730]: "Call mutating webhook" configuration:stork-webhooks-cfg,webhook:webhook.stork.libopenstorage.org,resource:/v1, Resource=pods,subresource:,operation:CREATE,UID:5ed0438a-551b-418d-8a93-5f9a0c91e6d3 (24-May-2023 08:21:16.080) (total time: 7636ms):
Trace[449263730]: ---"Request completed" 7636ms (08:21:23.717)
Trace[449263730]: [7.636966753s] [7.636966753s] END
I0524 08:21:23.722929 1 trace.go:205] Trace[1404157220]: "Create" url:/api/v1/namespaces/default/pods,user-agent:kubectl/v1.22.2 (linux/amd64) kubernetes/8b5a191,audit-id:fb3eea6b-629d-4529-aaa5-5029cbd76c52,client:172.22.63.25,accept:application/json,protocol:HTTP/2.0 (24-May-2023 08:21:16.076) (total time: 7646ms):
Trace[1404157220]: ---"Object stored in database" 7645ms (08:21:23.722)
Trace[1404157220]: [7.64680376s] [7.64680376s] END
Recently, the response times went up from 600ms to ~8s. I suspect the mutating webhook (webhook.stork.libopenstorage.org) to block the response for most of the time – looking in the logs, this webhook also responded in ~600ms previously, before times went up.
Any hints on how to investigate this further? I am unsure which component stork relates to, and how to find logs that could point me in the right direction.
Cluster information:
Kubernetes version: 1.22.4
Cloud being used: bare metal
Host OS: Centos 7
CNI and version: spectrumscale.csi.ibm.com:2.4.0, portworx:23.3.1