There seems to be an ever-growing amount of tools in this space. Theres 30 or so alone listed in the cncf landscape, and I know from experience there is no one-size-fits-all answer. With that, I figured it’d be worth starting a discussion on pros/cons or other pain points.
Example: There are still quite a few where if you want to build a container on top of k8s, it must be running with privileges or mount the docker.sock. Does any of them work well in a non-privileged mode? Theres been a few tools like buildah or orca-build, but is anyone running them in any sort of prod workflow?
4 Likes
how about openshift? it runs cri-o now days
Openshift is definitely a very nice batteries-included distribution. Honestly, it’s probably the easiest path for Enterprises trying to do an on-prem deployment.
CRI-O doesn’t solve the nested build issue, but I suspect that aspect of it will be solved over time as things continue to evolve.
For building images in kubernetes as part of a Continuous Delivery/Deployment pipeline, running Docker inside a Docker container vs. Docker outside of a Docker container, vs using some alternative unprivileged docker image builder (kaniko, buildah, img, umoci, orca-build) is a confusing array of choices. Using docker outside of a docker container by mounting /var/run/docker.sock is what we are currently using. The other tools are maturing quickly, and I’m sure we’ll be running an unprivileged tool inside a container in the next few months.
2 Likes
You shouldn’t have to worry about all different builder backends, but if you want to try some of them – you should look at Skaffold, it’s a good abstraction for that. I have a blog post coming up about how one can use Skaffold with our GitOps operator.
4 Likes
I agreed with @errordeveloper, the skaffold project is an ideal abstraction around most of the common approaches to building docker images on a k8s cluster - hopefully folks can add more Skaffold providers to cover all possible needs (e.g. for img/buildah or any cloud specific solutions etc). You can then use skaffold in whatever higher level CI/CD tools you pick.
I work on an open source automated CI/CD solution called Jenkins X which uses Skaffold for all docker image building & pre-commit development workflow. There’s a demo from KubeCon here if you’re interested in seeing it in action.
There’s still all kinds of different tools for the CI/CD side of things; but one thing @errordeveloper, WeaveWorks, CloudBees & the Jenkins X community all agree on: GitOps is the ideal approach to promoting releases, versions and changes through your environments - use git as the source of truth; then its versioned, audited and easy to revert changes that break things 
6 Likes
Hi @jstrachan, As far as I know, Helm is part of the workflow in Jenkins-X to build a chart and deploy a chart to the cluster. As Ksonnet is getting stable. Will it be part of the jenkinX workflow in GitOps?
Re. privileges to build: If you haven’t seen it, you may want to check out:
It is fairly new, though.
1 Like
we should be able to support it yeah - along with kustomize etc