How to prevent "Too many requests" Exception

General Discussions
1

There are several clusters accessed by a couple of applications, regularly, sometimes simultaneously.

The applications use the Kube API implementation for Java.

Too often, the Kube API server is prevented from doing its job, as it thinks too many requests have been received by the aforementioned applications.

What would be the best way to avoid this problem, without implementing some complicated request coordination for the applications using the Kubernetes API?

All I’ve found is information on how to limit requests further, etc.
However, I do not want to limit requests, I want the Kube API server to accept the ones received, no matter how much. Ideally it should just queue all requests, until they are all fulfilled in a safe non-overloading manner.

2

This seems relevant, but I’m not really sure. What version of k8s are you on and can you provide the actual error from the api-server?

3 
2021-04-08 11:27:20.742 ERROR 8 --- [   scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task

io.kubernetes.client.ApiException: Too Many Requests

	at io.kubernetes.client.ApiClient.handleResponse(ApiClient.java:882) ~[client-java-api-6.0.1.jar!/:na]

	at io.kubernetes.client.ApiClient.execute(ApiClient.java:798) ~[client-java-api-6.0.1.jar!/:na]

	at io.kubernetes.client.apis.AppsV1Api.listNamespacedDeploymentWithHttpInfo(AppsV1Api.java:3663) ~[client-java-api-6.0.1.jar!/:na]

	at io.kubernetes.client.apis.AppsV1Api.listNamespacedDeployment(AppsV1Api.java:3641) ~[client-java-api-6.0.1.jar!/:na]

	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]

	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) ~[na:na]

	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]

	at java.base/java.lang.reflect.Method.invoke(Method.java:564) ~[na:na]

	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.3.5.jar!/:5.3.5]

	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.3.5.jar!/:5.3.5]

	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]

	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na]

	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]

	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[na:na]

	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[na:na]

	at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]

This is the exception thrown, when using the Java implementation of the Kube API client. I currently cannot provide the ApiException’s responseBody, but I can log it next time this happens.

As far as I have seen, this error mostly happens, when more than one clients inside the applications I mentioned want to ask the Kube API for information from the cluster, at the same time. This overloads the Kube API server, apparently.

Kubernetes version: v1.19.4-rancher1-1

4

Maybe this. Check out the config files they mentioned for defaults perhaps.