Is there a quick way to safely bounce all nodes in a cluster?

dbwest · February 5, 2020, 3:16am

I was considering writing an Ansible script to in serial drain, reboot, and uncordon all nodes in a cluster.

Then I imagined this kind of script may already exist in some form.

Insight/tips?

I’ll consider it ‘safe’ if it doesn’t delete pvs. So the bar is low there

dbwest · February 5, 2020, 7:01pm

I made a script.

It is MIT Licensed. Use at your own peril.

stephendotcarter · February 5, 2020, 8:00pm

Kubo is a packaged version of Kubernetes which is deployed by BOSH. The drain script is what BOSH calls before stopping a worker VM.

This won’t be directly useable without some effort but might give some inspiration of what’s needed for a shutdown script.

github.com

cloudfoundry-incubator/kubo-release/blob/master/jobs/kubelet/templates/bin/drain.erb

#!/usr/bin/env bash
# vi: ft=sh

[ -z "$DEBUG" ] || set -x

set -eu

export PATH=/var/vcap/packages/kubernetes/bin:$PATH

LOG_DIR=/var/vcap/sys/log/kubelet
PIDFILE=/var/vcap/sys/run/kubernetes/kubelet.pid

trap ensure_safe_exit EXIT

main() {
  send_process_stdout_to_logfile
  send_process_stderr_to_logfile
  check_if_pidfile_exists
  check_if_pid_is_running

This file has been truncated. show original

rdxmb · February 5, 2020, 8:44pm

https://github.com/kubernetes-sigs/kubespray/tree/master/roles has an upgrade-role. I have never tested them.

Topic		Replies	Views
Rollback when timeout General Discussions	4	751	June 1, 2021
How To Test A Node For Failure [Force A Node To Fail] General Discussions	3	2008	January 10, 2021
Cluster reboot results in kube-controller-manager to crashloopbackoff apparently on master election General Discussions on-prem	7	5498	February 19, 2019
[Security Advisory] CVE-2023-5528: Insufficient input sanitization in in-tree storage plugin leads to privilege escalation on Windows nodes Announcements	0	981	November 14, 2023
Nodes crashed. node.kubernetes.io/unreachable:NoSchedule taint microk8s microk8s	3	2150	March 22, 2023

Is there a quick way to safely bounce all nodes in a cluster?

Related Topics