Hello Team! I am relatively new to k8s and am hoping I can learn a lot from you all!
I need some advise on an issue I am facing with k8s 1.14 and running gitlab pipelines on it. Many jobs are throwing up exit code 137 errors and I found that it means that the container is being terminated abruptly.
Kubernetes version: 1.14
Cloud being used: AWS EKS
Installation method: EKS
Host OS: Amazon Linux
After digging in, I found the below logs:
kubelet: I0114 03:37:08.639450 4721 image_gc_manager.go:300] [imageGCManager]: Disk usage on image filesystem is at 95% which is over the high threshold (85%). Trying to free 3022784921 bytes down to the low threshold (80%).
kubelet: E0114 03:37:08.653132 4721 kubelet.go:1282] Image garbage collection failed once. Stats initialization may not have completed yet: failed to garbage collect required amount of images. Wanted to free 3022784921 bytes, but freed 0 bytes
kubelet: W0114 03:37:23.240990 4721 eviction_manager.go:397] eviction manager: timed out waiting for pods runner-u4zrz1by-project-12123209-concurrent-4zz892_gitlab-managed-apps(d9331870-367e-11ea-b638-0673fa95f662) to be cleaned up
kubelet: W0114 00:15:51.106881 4781 eviction_manager.go:333] eviction manager: attempting to reclaim ephemeral-storage
kubelet: I0114 00:15:51.106907 4781 container_gc.go:85] attempting to delete unused containers
kubelet: I0114 00:15:51.116286 4781 image_gc_manager.go:317] attempting to delete unused images
kubelet: I0114 00:15:51.130499 4781 eviction_manager.go:344] eviction manager: must evict pod(s) to reclaim ephemeral-storage
kubelet: I0114 00:15:51.130648 4781 eviction_manager.go:362] eviction manager: pods ranked for eviction:
And then the pods get terminated resulting in the exit code 137s.Can anyone help me understand the reason and a possible solution to overcome this?