Control Planes in microk8s

General Discussions microk8s
1

Hey there,

I’ve setup a 3 node microk8s cluster. In front of that, I have a two node loadbalancer (haproxy, keepalived) with a virtual IP (VIP). The loadbalancer is configured to access the kube-api. It distributes requests to all of the 3 micok8s nodes.

My question now: Can I somehow control on which microk8s nodes the control plane is running, e.g. when I add 20 more nodes, I don’t want to have the control plane running on all the nodes. I would rather like to have the control planes always be running on the first 3 nodes, so that my loadbalancer configuration does not need to be touched/changed.

Thanks for any useful hint,
hoba

1 Like
2

Hi @hoba
Not automatically. I’ve written some steps on how to setup worker node only with microk8s here.
Its a series of manual steps though.

3

Hi @hoba

I’m trying to reach the same, with haproxy in front of the apis (3x nodes and 3x workers without the apis), would you care sharing the config steps?

I get the following errors:

microk8s kubectl get nodes
E0920 13:06:21.566740  119997 memcache.go:265] couldn't get current server API group list: Get "https://k8cp.urbaman.it:16443/api?timeout=32s": EOF

curl -v -XGET  -H "User-Agent: kubectl/v1.28.1 (linux/amd64) kubernetes/8dc49c4" -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" 'https://k8cp.urbaman.it:16443/api'
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 10.0.50.64:16443...
* Connected to k8cp.urbaman.it (10.0.50.64) port 16443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
* error:0A000126:SSL routines::unexpected eof while reading
* Closing connection 0
curl: (35) error:0A000126:SSL routines::unexpected eof while reading

I used a haproxy config perfectly working with my previous kubeadm cluster.

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

#---------------------------------------------------------------------
# apiserver frontend which proxys to the microk8s nodes
#---------------------------------------------------------------------
frontend microk8s
    bind *:16443
    mode tcp
    option tcplog
    default_backend apiserver

#---------------------------------------------------------------------
# round robin balancing for microk8s
#---------------------------------------------------------------------
backend microk8s
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    option ssl-hello-chk
    balance     roundrobin
        server k8cp1 k8cp1.urbaman.it:16443 check
        server k8cp2 k8cp2.urbaman.it:16443 check
        server k8cp3 k8cp3.urbaman.it:16443 check

Thank you.