An excellent alternative for training and evaluating your machine learning models in public and private clouds is to use Kubeflow — an open-source toolkit to distribute machine learning. It’s designed to leverage containers and Kubernetes. In this article, you’ll learn how you can train your deep learning model at scale with a managed on-premise Kubernetes cluster
https://learnk8s.io/blog/scaling-machine-learning-with-kubeflow-tensorflow