I recently worked on integrating Kubernetes in a testing pipeline which included spinning up a cluster for each run. Doing this one either has to spin up an external cluster in the cloud which is slow and costly or create a local cluster. We went with the local option but that brought other drawbacks, such as trouble debugging as the cluster isn’t persistent.
How do you handle Kubernetes in your testing pipeline?
Do you test a cluster creation or your application as a client of the cluster?
Minikube may be an option, take a look: https://github.com/kubernetes-client/python/blob/master/scripts/kube-init.sh - script to start minikube and configure kubectl to play with.
Have you looked at Kind? https://kind.sigs.k8s.io/
Here are some examples for using it in CI https://github.com/kind-ci/examples
For the project I was mentioned it was mostly about testing the cluster itself but I’m also interested in testing several services running in a cluster.
I actually use Minikube for my local environment and it’s quite good. The problem is that it runs locally which makes it quite hard to debug if some tests fail and then the CI runner is shut down. Preferably I would like to run the cluster externally and have it be persistent for a limited time to ease debugging.
Kind is actually what I ended up using for the project mentioned, really great piece of tooling!
As I mentioned in my response to Tomasz though the problem is that Kind (and Minikube) are local tools and non-persistent tools. Once the CI runner is done the cluster is gone which makes it hard to perform debugging or for example set up a staging environment for each PR.
Are you aware of any external service that would make it possible to quickly spin up a cluster and keep it alive for some time? Seems like it should exist or am I missing something?
My immediate thought it to use terraform in some capacity to achieve that and set up a timed tear down once the tests are done.