Is there a quick way to safely bounce all nodes in a cluster?

dbwest · February 5, 2020, 3:16am

I was considering writing an Ansible script to in serial drain, reboot, and uncordon all nodes in a cluster.

Then I imagined this kind of script may already exist in some form.

Insight/tips?

I’ll consider it ‘safe’ if it doesn’t delete pvs. So the bar is low there

dbwest · February 5, 2020, 7:01pm

I made a script.

It is MIT Licensed. Use at your own peril.

stephendotcarter · February 5, 2020, 8:00pm

Kubo is a packaged version of Kubernetes which is deployed by BOSH. The drain script is what BOSH calls before stopping a worker VM.

This won’t be directly useable without some effort but might give some inspiration of what’s needed for a shutdown script.

github.com

cloudfoundry-incubator/kubo-release/blob/master/jobs/kubelet/templates/bin/drain.erb

#!/usr/bin/env bash
# vi: ft=sh

[ -z "$DEBUG" ] || set -x

set -eu

export PATH=/var/vcap/packages/kubernetes/bin:$PATH

LOG_DIR=/var/vcap/sys/log/kubelet
PIDFILE=/var/vcap/sys/run/kubernetes/kubelet.pid

trap ensure_safe_exit EXIT

main() {
  send_process_stdout_to_logfile
  send_process_stderr_to_logfile
  check_if_pidfile_exists
  check_if_pid_is_running

This file has been truncated. show original

rdxmb · February 5, 2020, 8:44pm

https://github.com/kubernetes-sigs/kubespray/tree/master/roles has an upgrade-role. I have never tested them.

Topic		Replies	Views
How to shut down and restart Kubernetes clusters? General Discussions	9	94389	June 26, 2023
Can we reboot the actual nodes using kubectl remotely? General Discussions	4	1599	March 5, 2019
How to drain a node and restart it? General Discussions	3	18891	June 13, 2021
Rollback when timeout General Discussions	4	801	June 1, 2021
Cluster reboot results in kube-controller-manager to crashloopbackoff apparently on master election General Discussions on-prem	7	5987	February 19, 2019

Is there a quick way to safely bounce all nodes in a cluster?

Related topics