So, after managing one too many Kubernetes clusters, it’s time I wrangle the developer workflow for some of the latest clusters. In the past I’ve used a handful of tools to wire up a pipeline but haven’t really had time to look at the scene for CI/CD kubernetes for over a year and know a lot has happened.
So, Kubernetes users, what are you all using for CI/CD pipelines today?
We use gitlab and it works fine until you get many runners and they could work on kube
Gitlab definitely has a great offering and really powerful configuration. Their CD takes into account the isolated effects of testing on a disparate ephemeral cluster. Plus, they’ve done a ton with their UI to make it really easy to visually see whats going on with each build in addition to offering the build-in container image registry. However, they do have some stability issues from time to time and pricing for runners can get interesting depending on how many repos you have etc…
I’m starting to look into Google Cloud Build with GCR which basically gives you an even easier way to approach CI/CD.
I have used Jenkins, Concourse, AWS Code Pipeline pretty extensively.
Jenkins can be wrestled into doing continuous delivery/deployment with declarative pipelines. It works ok, but feels kind of gross to debug and is very heavy for what it does.
Concourse works well for complicated pipelines. Seems a little heavy.
AWS Code Pipeline is a bit bare bones, but is extensible with Lambdas and Step functions for more complicated workflows.
We are in the process of evaluating Team City, ArgoCD, and a few others.
I am keeping more notes here
As of now, I am evaluating Jenkins-x. It looks promising and keeps improving. It is utilising Kubernetes so It works where k8s fits.
We use gitlab. We made our own customer runner called gkube-executor that can do anythiing to a kubernetes cluster we want during the execution of a CI job including the use of helm.
I use Github + S2I + Jenkins, with Openshift. I work for Red Hat, so that’s a good reason for me to use those tools. However, S2I really is super-easy to get started with, so I haven’t see a reason to switch.
After playing around with a few different options we have settled for Weave Flux for now, with a mix of Helm charts or direct manifest Deployments (depending on the respective dev team’s experience).
Builds are currently done via Docker Cloud’s Autobuild feature and are then picked up by Flux (albeit Flux can use any registry, it simply uses the defined image of the Pod Spec in the Deployment and its specified imagePullSecret to check for updates).
What we are still figuring out here is proper notification of successful/failed deployments (we’re not using Weave Cloud, but self hosted Flux).
Hi, thanks for mentioning flux above. I wrote this post to explain a bit more of the “gitops” thinking behind flux and similar tools. https://www.weave.works/blog/what-is-gitops-really
It is designed to provide CD that is managed by Kubernetes itself, from within the cluster. The idea is that CD occurs when Kubernetes is aware that the cluster needs to be updated, eg because a new image or config is in the repo.
When you work in this way it means you can work with any combination of CI, registry and config repo. Coordination is managed by workflow or CI tools.
There is a good article on the difference between using CIOps tools vs GitOps CD tools for pipelines.
Btw, Weave Cloud builds in the GitOps CD with full monitoring and Promotion (soon to be GA). Good idea to do deployments with understanding (monitoring) what’s happening in the cluster.
Disclaimer: I work for Weaveworks
Been involved with several CD flows for Kube with circleci.com, especially with their workflows in CircleCI 2.0.
Additional benefits are that each app’s workflow config is hosted in the app’s repo so not spread around the place.
I am using GitLab, its very good tool for CI/CD
Guess the tool doesn’t really matter if you know how to talk to the API’s and integrate well with it. Very good experience with Jenkins and the kubernetes runner plugins, we also did integrate deployments to k8s with gitlab ci super well.
After TeamCity, Bamboo and Jenkins, I am now a Gitlab User for the past 2.5 years and I don’t wanna go back. Pipelines run on K8s and it’s a bliss
I have used GitLab Ci, Jenkins, travis CI, CircleCI and etc.
But GitLab Ci is more powerful.
Interesting topic, add my personal experiences.
Bamboo - Heavy pipeline tool, old school style, bad experience of its new pipeline as code (bamboo yaml/java spec). Proper pipeline as code for it is long way to go. And seems Atlassian teams ignore customers feedback after IPO, improvement goes too slow.
Jenkins 2 - open source tool, most popular CICD. The new Jenkinsfile supports pipeline as code, not flat as Yaml, but good enough. There are some plugins for kubernetes deployment already.
Gitlab - Many people here talk about it already.
Bitbucket pipeline - Combine with bitbucket cloud, not cheap.
GoCD - Open source, people has no big interesting with it now in last 2 years. I don’t why, (not good marketing?), I still think it is a great tool.
Concourse - nice open source, suitable for small companies or small projects, but when deal with enterprise CI/CD pipeline, I can’t make the proper fine grained user management, can’t pass security review (My experience was stopped at last year, maybe it has new feature to overcome it, need feedback)
CircleCI 2 with workflow - used to be good for CI, with new feature workflow, support CD now. But still has issues, such as, I can’t hide secure value in environment variables from build logs.
Travis CI - Free for Github users. Simple to use. I have no experience as pay users.
Drone - Open source, written by Golang, I have experience with its version to 0.8.x. Can be installed with in-house solution, Good for CI, but when deal with CD, I still feel some difficulties. Not sure if any improvement for the CD now.
Buildkite - only open source for its agents, not server/UI, so you can build and manage your own agents, but you need export build logs to public cloud-based buildkite server/UI.
Teamcity - Seems Windows projects like it, I am Linux users. No comments on it.
AWS CodePipeline - manage service. Tightly with its own CodeCommit service, can integrate with other sources, such as Github.
We are using a combination of:
Whenever we commit back to BitBucket, Jenkins kicks off a build. Our build process uses the version number and buildId to dynamically update the Helm chart values. Then our images and Charts are pushed to Artifactory.
We then trigger deploys from RunDeck to the specific namespaces we want by referencing the Helm chart name and version.
Some of these integrations were written in house. If you want more details, please let me know.