Hello Kubernetes Community,
I am currently facing one persistent problem with my website running inside a Kubernetes environment, and I am hoping someone can help me understand what might be causing it. The specific issue is that the API component of my website initially works normally after its pod starts, but after the pod has been running for several hours, API requests gradually become slower and eventually some requests stop receiving a response within the expected time. The website itself remains accessible, and the Kubernetes cluster continues reporting the affected pod as running, but requests routed to that application can take significantly longer than they did immediately after startup. Restarting the affected pod temporarily restores the API to normal response times, after which the same degradation eventually returns. I am trying to investigate this as one specific long-running API performance problem rather than several unrelated Kubernetes issues.
The API application handles normal website requests and returns JSON responses to the frontend. Immediately after a new pod is created, the same endpoints respond quickly and consistently under normal traffic. As the pod continues running, however, response times gradually increase even though there has not been a comparable increase in the number of requests being sent to the website. The problem is not that the application completely crashes; Kubernetes still shows the pod as running and the container does not necessarily report an obvious failure. Instead, the application appears to remain alive while becoming increasingly slow at processing incoming API requests. I have checked the basic application logs and can see that requests are still reaching the container, but the delay between receiving a request and producing its response becomes noticeably longer during the affected period.
I have been monitoring the pod’s resource usage while trying to reproduce the problem and have not yet found an obvious explanation from CPU usage alone. Memory consumption does appear to change over the lifetime of the pod, although I am still collecting enough information to determine whether that increase is significant or directly related to the slowdown. The application does not intentionally maintain an ever-growing in-memory dataset, so I am investigating whether some object, connection, cache, worker, or other runtime resource could be accumulating over time. I have also checked the container’s restart history and timestamps so that I can compare a freshly started pod with one that has been running long enough to exhibit the problem. The strongest clue so far is that restarting the same container image restores the API’s normal response time without requiring a new application deployment.
I have also checked the Kubernetes configuration to make sure that the affected workload has the expected resource requests and limits and that the service is routing requests to the intended pod. The application works normally when a fresh pod is running, so I do not currently suspect that the image itself is fundamentally broken. I am more interested in determining whether the long-running application process is gradually consuming a resource that is not being released properly. I have considered possibilities such as memory growth, open connections, file descriptors, connection pools, worker processes, or application-level caching, but I do not want to start changing Kubernetes limits randomly because that could hide the underlying problem. I would rather capture the state of the pod while the API is slow and compare it with the same measurements immediately after the pod starts.
The problem is particularly difficult to troubleshoot because restarting the pod removes the evidence that may explain why it became slow. I have therefore started collecting application logs, pod resource information, request timing data, and container statistics before performing a restart. I am also comparing the response time of the same API endpoint at different points in the pod’s lifecycle so I can determine whether latency increases gradually or changes suddenly after a particular event. I have not observed a complete node failure or cluster-wide outage when this happens, and other workloads can continue operating normally. This makes me think that the problem may be isolated to the application process running inside the affected container, although I would like to understand which Kubernetes-level metrics can confirm whether the pod is actually under resource pressure or whether the root cause is entirely inside the application runtime.
I would appreciate guidance from the Kubernetes community on how to systematically diagnose this specific situation where a website API pod remains in a healthy running state but becomes progressively slower after extended uptime, with a pod restart temporarily restoring normal performance. In particular, I would like recommendations for comparing pod resource usage, memory behaviour, container statistics, process counts, open file descriptors, network connections, probes, and other relevant metrics between a healthy newly started pod and a degraded long-running pod. I would also appreciate advice on which Kubernetes commands or monitoring tools would help determine whether the slowdown originates from Kubernetes resource management or from the application process itself. My goal is to identify the underlying cause of the gradual API degradation and fix it properly so the website can maintain consistent response times without relying on periodic pod restarts as a workaround. Sorry for long post!