Summary
I am trying to stand up a simple kubernetes cluster with vagrant and ansible, but it is failing at the stage of setting up etcd.
Environment
Host: Fedora 34
Vagrant: 2.2.18
Vagrant backend: libvirt
Plugins: vagrant-libvirt
Logs
You can clearly see etcd fails to launch each time:
docker ps -a
8b570e095bfc 004811815584 "etcd --advertise-cl…" 4 minutes ago Exited (1) 4 minutes ago k8s_etcd_etcd-k8s-master_kube-system_07da62064f58755cfa30081d17a5d04c_15
Then getting logs of etcd container:
{"level":"info","ts":"2021-09-22T08:40:02.891Z","caller":"etcdmain/etcd.go:72","msg":"Running: ","args":["etcd","--advertise-client-urls=https://192.168.50.10:2379","--cert-file=/etc/kubernetes/pki/etcd/server.crt","--client-cert-auth=true","--data-dir=/var/lib/etcd","--initial-advertise-peer-urls=https://192.168.50.10:2380","--initial-cluster=k8s-master=https://192.168.50.10:2380","--key-file=/etc/kubernetes/pki/etcd/server.key","--listen-client-urls=https://127.0.0.1:2379,https://192.168.50.10:2379","--listen-metrics-urls=http://127.0.0.1:2381","--listen-peer-urls=https://192.168.50.10:2380","--name=k8s-master","--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt","--peer-client-cert-auth=true","--peer-key-file=/etc/kubernetes/pki/etcd/peer.key","--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt","--snapshot-count=10000","--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt"]}
{"level":"info","ts":"2021-09-22T08:40:02.891Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["https://192.168.50.10:2380"]}
{"level":"info","ts":"2021-09-22T08:40:02.891Z","caller":"embed/etcd.go:478","msg":"starting with peer TLS","tls-info":"cert = /etc/kubernetes/pki/etcd/peer.crt, key = /etc/kubernetes/pki/etcd/peer.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
{"level":"info","ts":"2021-09-22T08:40:02.891Z","caller":"embed/etcd.go:367","msg":"closing etcd server","name":"k8s-master","data-dir":"/var/lib/etcd","advertise-peer-urls":["https://192.168.50.10:2380"],"advertise-client-urls":["https://192.168.50.10:2379"]}
{"level":"info","ts":"2021-09-22T08:40:02.891Z","caller":"embed/etcd.go:369","msg":"closed etcd server","name":"k8s-master","data-dir":"/var/lib/etcd","advertise-peer-urls":["https://192.168.50.10:2380"],"advertise-client-urls":["https://192.168.50.10:2379"]}
{"level":"warn","ts":"2021-09-22T08:40:02.891Z","caller":"etcdmain/etcd.go:145","msg":"failed to start etcd","error":"listen tcp 192.168.50.10:2380: bind: cannot assign requested address"}
{"level":"fatal","ts":"2021-09-22T08:40:02.891Z","caller":"etcdmain/etcd.go:203","msg":"discovery failed","error":"listen tcp 192.168.50.10:2380: bind: cannot assign requested address","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdmain/etcd.go:203\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdmain/main.go:40\nmain.main\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/main.go:32\nruntime.main\n\t/home/remote/sbatsche/.gvm/gos/go1.16.3/src/runtime/proc.go:225"}
And this bit stands out:
{"level":"fatal","ts":"2021-09-22T08:40:02.891Z","caller":"etcdmain/etcd.go:203","msg":"discovery failed","error":"listen tcp 192.168.50.10:2380: bind: cannot assign requested address","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdmain/etcd.go:203\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdmain/main.go:40\nmain.main\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/main.go:32\nruntime.main\n\t/home/remote/sbatsche/.gvm/gos/go1.16.3/src/runtime/proc.go:225"}
Additional
Nothing is listening on that port:
netstat -tunlp | grep 2380
The device exists and is up:
ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:11:f9:ae brd ff:ff:ff:ff:ff:ff
inet 192.168.121.35/24 brd 192.168.121.255 scope global dynamic eth0
valid_lft 2767sec preferred_lft 2767sec
inet6 fe80::5054:ff:fe11:f9ae/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:b2:f8:37 brd ff:ff:ff:ff:ff:ff
inet 191.168.50.10/24 brd 191.168.50.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::5054:ff:feb2:f837/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:b1:f9:58:96 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 52:54:00:11:f9:ae brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 52:54:00:b2:f8:37 brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether 02:42:b1:f9:58:96 brd ff:ff:ff:ff:ff:ff
Vagrant File for Master
IMAGE_NAME = "brokencode/debian-10.10"
config.vm.define "k8s-master" do |master|
master.vm.box = IMAGE_NAME
master.vm.network "private_network", ip: "191.168.50.10"
#master.vm.provision "shell",
master.vm.hostname = "k8s-master"
master.vm.provision "ansible" do |ansible|
ansible.playbook = "ansible/k8s_master_setup.yaml"
ansible.extra_vars = {
node_ip: "192.168.50.10"
}
master.vm.provider "libvirt" do |v|
v.uri = "qemu+unix:///system"
v.host = "k8s-master"
v.memory =2048
v.cpus = 2
end
Ansible Playbook for Master
---
- hosts: all
become: true
tasks:
- name: Install base packages
apt:
name: "{{ packages }}"
state: present
update_cache: yes
vars:
packages:
- apt-transport-https
- ca-certificates
- curl
- git
- gnupg-agent
- software-properties-common
- name: Add Docker repository key
apt_key:
url: "https://download.docker.com/linux/debian/gpg"
state: present
- name: Add Docker repository
apt_repository:
repo: "deb https://download.docker.com/linux/debian buster stable"
state: present
- name: Install docker and its dependecies
apt:
name: "{{ packages }}"
state: present
update_cache: yes
vars:
packages:
- docker-ce
- docker-ce-cli
- containerd.io
notify:
- docker start
- name: Copy docker daemon config
synchronize:
src: ../files/docker/daemon.json
dest: "/etc/docker/"
- name: restart docker
systemd:
name: docker
state: restarted
daemon_reload: yes
- name: Add vagrant user to docker group
user:
name: vagrant
group: docker
- name: Remove swapfile from /etc/fstab
mount:
name: "{{ item }}"
fstype: swap
state: absent
with_items:
- swap
- none
- name: Disable swap
command: swapoff -a
when: ansible_swaptotal_mb > 0
- name: Add an apt signing key for Kubernetes
apt_key:
url: "https://packages.cloud.google.com/apt/doc/apt-key.gpg"
state: present
- name: Add repository for Kubernetes
apt_repository:
repo: "deb https://apt.kubernetes.io/ kubernetes-xenial main"
state: present
filename: kubernetes.list
- name: Install Kubernetes binaries
apt:
name: "{{ packages }}"
state: present
update_cache: yes
vars:
packages:
- kubelet
- kubeadm
- kubectl
- name: Configure node-ip {{ node_ip }} for kubelet
lineinfile:
path: '/etc/systemd/system/kubelet.service.d/10-kubeadm.conf'
line: 'Environment="KUBELET_EXTRA_ARGS=--node-ip={{ node_ip }}"'
regexp: 'KUBELET_EXTRA_ARGS='
insertafter: '\[Service\]'
state: present
- name: Restart Kubelet
systemd:
name: kubelet
state: restarted
daemon_reload: yes
- name: Initialize the Kubernetes cluster using kubeadm
command: kubeadm init --apiserver-advertise-address="192.168.50.10" --apiserver-cert-extra-sans="192.168.50.10" --node-name k8s-master --pod-network-cidr=192.168.0.0/16
- name: Setup kubeconfig for vagrant user
command: "{{ item }}"
with_items:
- mkdir -p /home/vagrant/.kube
- cp -i /etc/kubernetes/admin.conf /home/vagrant/.kube/config
- chown vagrant:vagrant /home/vagrant/.kube/config
- name: Install calico pod network
become: false
command: kubectl create -f https://docs.projectcalico.org/v3.4/getting-started/kubernetes/installation/hosted/calico.yaml
- name: Generate join command
command: kubeadm token create --print-join-command
register: join_command
- name: Copy join command to local file
local_action: copy content="{{ join_command.stdout_lines[0] }}" dest="./join-command"
handlers:
- name: docker start
systemd:
name: docker
state: started
Etcd manifest
As generated by kubeadm:
kind: Pod
metadata:
annotations:
kubeadm.kubernetes.io/etcd.advertise-client-urls: https://192.168.50.10:2379
creationTimestamp: null
labels:
component: etcd
tier: control-plane
name: etcd
namespace: kube-system
spec:
containers:
- command:
- etcd
- --advertise-client-urls=https://192.168.50.10:2379
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
- --client-cert-auth=true
- --data-dir=/var/lib/etcd
- --initial-advertise-peer-urls=https://192.168.50.10:2380
- --initial-cluster=k8s-master=https://192.168.50.10:2380
- --key-file=/etc/kubernetes/pki/etcd/server.key
- --listen-client-urls=https://127.0.0.1:2379,https://192.168.50.10:2379
- --listen-metrics-urls=http://127.0.0.1:2381
- --listen-peer-urls=https://192.168.50.10:2380
- --name=k8s-master
- --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
- --peer-client-cert-auth=true
- --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
- --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
- --snapshot-count=10000
- --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
image: k8s.gcr.io/etcd:3.5.0-0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 127.0.0.1
path: /health
port: 2381
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
name: etcd
resources:
requests:
cpu: 100m
memory: 100Mi
startupProbe:
failureThreshold: 24
httpGet:
host: 127.0.0.1
path: /health
port: 2381
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 15
volumeMounts:
- mountPath: /var/lib/etcd
name: etcd-data
- mountPath: /etc/kubernetes/pki/etcd
name: etcd-certs
hostNetwork: true
priorityClassName: system-node-critical
securityContext:
seccompProfile:
type: RuntimeDefault
volumes:
- hostPath:
path: /etc/kubernetes/pki/etcd
type: DirectoryOrCreate
name: etcd-certs
- hostPath:
path: /var/lib/etcd
type: DirectoryOrCreate
name: etcd-data
status: {}