Launch configurations reference

General Discussions microk8s
1

NOTE: Launch configurations are available starting from MicroK8s 1.27 and newer

Launch configurations schema

The schema is defined in schema.go. For a full example config file, see full.yaml

Launch configurations version 0.1.0

---
# 'version' is the semantic version of the configuration file format.
version: 0.1.0

# 'persistentClusterToken' is a persistent token that may be used to authenticate join requests made to the local node.
# Persistent token length should be 32 characters. Other nodes will be able to join using the following command:
#   $ microk8s join $THIS_NODE_IP:25000/a74cddf30d2408d49fcd748a26021c6a
persistentClusterToken: "a74cddf30d2408d49fcd748a26021c6a"

# 'join' is configuration to join the local node to an already existing MicroK8s cluster.
join:
  url: "10.10.10.10:25000/a74cddf30d2408d49fcd748a26021c6a"       # 'url' is the cluster join URL
  worker: true                                                    # set 'worker' to true to join as a worker node

# 'extraSANs' is a list of extra Subject Alternate Names to add to the local API server.
extraSANs:
  - 10.10.10.10
  - microk8s.example.com

# 'extraKubeAPIServerArgs' is extra arguments to add to the local node kube-apiserver.
# Set a value to null to remove it from the arguments.
extraKubeAPIServerArgs:
  --authorization-mode: RBAC,Node
  --event-ttl: null

# 'extraKubeletServerArgs' is extra arguments to add to the local node kubelet.
# Set a value to null to remove it from the arguments.
extraKubeletArgs:
  --cluster-dns: 10.152.183.10

# 'extraKubeProxyArgs' is extra arguments to add to the local node kube-proxy.
# Set a value to null to remove it from the arguments.
extraKubeProxyArgs:
  --cluster-cidr: 10.1.0.0/16

# 'extraKubeControllerManagerArgs' is extra arguments to add to the local node kube-controller-manager.
# Set a value to null to remove it from the arguments.
extraKubeControllerManagerArgs:
  --leader-elect-lease-duration: 30s
  --leader-elect-renew-deadline: 15s

# 'extraKubeSchedulerArgs' is extra arguments to add to the local node kube-scheduler.
# Set a value to null to remove it from the arguments.
extraKubeSchedulerArgs:
  --leader-elect-lease-duration: 30s
  --leader-elect-renew-deadline: 15s

# 'extraContainerdArgs' is extra arguments to add to the local node containerd service.
# Set a value to null to remove it from the arguments.
extraContainerdArgs:
  -l: debug

# 'extraContainerdEnv' is extra environment variables (e.g. proxy configuration) for the local node containerd service.
# Set a value to null to remove it from the environment.
extraContainerdEnv:
  http_proxy: http://squid.internal:3128
  https_proxy: http://squid.internal:3128

# 'extraDqliteArgs' is extra arguments for the local node k8s-dqlite service.
# Set a value to null to remove it from the arguments.
extraDqliteArgs:
  --disk-mode: "true"

# 'extraDqliteEnv' is extra environment variables for the local node k8s-dqlite service.
# Set a value to null to remove it from the environment.
extraDqliteEnv:
  LIBRAFT_TRACE: "1"
  LIBDQLITE_TRACE: "1"

# 'extraMicroK8sClusterAgentArgs' is extra arguments for the local node cluster-agent service.
extraMicroK8sClusterAgentArgs:
  --min-tls-version: tls13

# 'extraMicroK8sClusterAgentEnv' is extra environment variables for the local node cluster-agent service.
extraMicroK8sClusterAgentEnv:
  GOFIPS: "1"

# 'extraMicroK8sAPIServerProxyArgs' is extra arguments for the local node apiserver-proxy service (kube-apiserver proxy for worker nodes).
extraMicroK8sAPIServerProxyArgs:
  --refresh-interval: "0"

# 'extraMicroK8sAPIServerProxyEnv' is extra environment variables for the local node apiserver-proxy service (kube-apiserver proxy for worker nodes).
extraMicroK8sAPIServerProxyEnv:
  GOFIPS: "1"

# 'extraEtcdArgs' is extra arguments to add to the local node etcd service.
# Set a value to null to remove it from the arguments.
extraEtcdArgs:
  --log-level: debug
  --enable-pprof: "true"

# 'extraEtcdEnv' is extra environment variables (e.g. proxy configuration) for the local node etcd service.
# Set a value to null to remove it from the environment.
extraEtcdEnv:
  GOFIPS: "1"

# 'extraFlanneldArgs' is extra arguments to add to the local node flanneld service.
# Set a value to null to remove it from the arguments.
extraFlanneldArgs:
  --kube-subnet-mgr: "true"
  --kubeconfig-file: $SNAP_DATA/credentials/kubelet.config
  --etcd-endpoints: null
  --etcd-cafile: null
  --etcd-certfile: null
  --etcd-keyfile: null

# 'extraFlanneldEnv' is extra environment variables (e.g. proxy configuration) for the local node flanneld service.
# Set a value to null to remove it from the environment.
extraFlanneldEnv:
  GOFIPS: "1"

# 'extraConfigFiles' is extra service configuration files to create (e.g. for configuring kube-apiserver encryption at rest).
# These files will be written at $SNAP_DATA/args/<filename>.
extraConfigFiles:
  flannel-network-mgr-config: |
    {"Network": "10.1.0.0/16", "Backend": {"Type": "vxlan"}}

# 'addonRepositories' is a list of addon repositories to configure on the local node.
addonRepositories:
  - name: github-repo
    url: https://github.com/org/repo
    reference: master                       # 'reference' is branch or tag to check out instead of the default
  - name: local-repo
    url: /path/to/local/repo                # 'url' can be a local directory

# 'addons' is a list of addons to enable or disable.
addons:
  - name: dns                               # 'name' is the name of the addon to enable.
  - name: gpu
    args: [--version=v22.9.1]               # 'args' is extra arguments that will be passed to the enable script.
                                            # refer to the documentation of each addon for more information.
  - name: registry
    disable: true                           # 'disable' should be set when the addon must be disabled instead.

# 'containerdRegistryConfigs' is used to configure registry mirrors. the key is the repository name (e.g. "docker.io")
# and the value will be written to the respective hosts.toml file
containerdRegistryConfigs:
  # Use `http://my.proxy:5000` as a DockerHub registry mirror.
  docker.io: |
    [host."http://my.proxy:5000"]
    capabilities = ["pull", "resolve"]

Examples

The following examples present launch configurations for common tasks. The examples are meant to be composable, you can mix and match as many of these configs as you want to suit your needs:

Enable DNS

Enable DNS addon, use host resolv.conf for upstream nameservers or fallback to 1.1.1.1.

---
version: 0.1.0
addons:
  - name: dns

# These arguments will be set by the 'dns' addon. We set them manually to avoid
# unnecessary Kubernetes restarts while the cluster is bootstrapping.
extraKubeletArgs:
  --cluster-dns: 10.152.183.10
  --cluster-domain: cluster.local

Enable DNS, RBAC, Metrics-Server, Dashboard, Hostpath Storage, Ingress and Cert-Manager

---
version: 0.1.0
addons:
  - name: dns
  - name: rbac
  - name: metrics-server
  - name: dashboard
  - name: hostpath-storage
  - name: ingress
  - name: cert-manager

# These arguments will be set by the 'dns' and 'rbac' addons. We set them manually to avoid
# unnecessary Kubernetes restarts while the cluster is bootstrapping.
extraKubeletArgs:
  --cluster-dns: 10.152.183.10
  --cluster-domain: cluster.local
extraKubeAPIServerArgs:
  --authorization-mode: RBAC,Node

Enable DNS with specific nameserver

Enable DNS addon and use a specific upstream nameserver (by specifying it as argument to the DNS addon):

---
version: 0.1.0
addons:
  - name: dns
    args: [1.1.1.1, 8.8.8.8]

Configure private registry mirrors

Configure private registry mirrors for docker.io and registry.k8s.io. Requires that you have manually pushed required images to the registry mirrors, or that you have set them up as a pull-through cache:

---
version: 0.1.0
addons:
  - name: dns
containerdRegistryConfigs:
  docker.io: |
    [host."http://dockerhub.internal.mirror:15050"]
    capabilities = ["pull", "resolve"]
  registry.k8s.io: |
    [host."http://registryk8sio.internal.mirror:15050"]
    capabilities = ["pull", "resolve"]

Configure proxy for access to image registries

Containerd will use this proxy when pulling images from the upstream repositories:

---
version: 0.1.0
addons:
  - name: dns
extraContainerdEnv:
  http_proxy: http://squid.internal:3128
  https_proxy: http://squid.internal:3128
  no_proxy: 10.0.0.0/8,127.0.0.1,192.168.0.0/16,172.16.0.0/12

Configure Subject Alternate Names for kube-apiserver certificates

Ensure the kube-apiserver can be accessed using the hostname k8s.cluster1.infra1.pod.

---
version: 0.1.0
extraSANs:
  - k8s.cluster1.infra1.prod

Configure Kubernetes services

Decrease lease times for kube-scheduler and kube-controller-manager from their default values. Also, tell kube-apiserver to prefer InternalIP when talking to the kubelets:

---
version: 0.1.0
extraKubeAPIServerArgs:
  --kubelet-preferred-address-types: InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP
extraKubeControllerManagerArgs:
  --leader-elect-lease-duration: 30s
  --leader-elect-renew-deadline: 15s
extraKubeSchedulerArgs:
  --leader-elect-lease-duration: 30s
  --leader-elect-renew-deadline: 15s

Configure host interfaces

Configure host interfaces used by Kubernetes services. In this example, the host uses 10.0.1.10 for control plane traffic, and 10.0.2.10 for NodePort services.

---
version: 0.1.0
extraKubeAPIServerArgs:
  --advertise-address: 10.0.1.10
extraKubeletArgs:
  --node-ip: 10.0.1.10
extraKubeProxyArgs:
  --nodeport-addresses: 10.0.2.10

Custom addon repositories

Configure custom addon repositories and enable custom addons:

---
version: 0.1.0
addonRepositories:
  - name: custom
    url: https://github.com/neoaggelos/microk8s-addons-extras
addons:
  - name: custom/clusterapi

Configure secrets encryption at rest

Create an extra config file with the encryption at rest config and pass it to kube-apiserver:

---
version: 0.1.0
extraKubeAPIServerArgs:
  --encryption-provider-config: "$SNAP_DATA/args/encryption-config.yaml"
extraConfigFiles:
  encryption-config.yaml: |
    apiVersion: apiserver.config.k8s.io/v1
    kind: EncryptionConfiguration
    resources:
      - resources:
          - secrets
          - configmaps
        providers:
          - aescbc:
              keys:
                - name: key1
                  secret: GneEFCl9TpamK4CcxRJcHFgUuFUnfTTnSW/LzwWRY5g=    # use 'openssl rand -base64 32'
          - identity: {}

For more details on the required configurations, see Encrypting Confidential Data at Rest | Kubernetes

Configure auditing policies

Create an extra config file with the auditing policy configuration and pass it to kube-apiserver:

---
version: 0.1.0
extraKubeAPIServerArgs:
  --audit-policy-file: "$SNAP_DATA/args/audit-policy.yaml"
  --audit-log-path: "-"
extraConfigFiles:
  audit-policy.yaml: |
    apiVersion: audit.k8s.io/v1
    kind: Policy
    omitStages:
      - "RequestReceived"
    rules:
      - level: RequestResponse
        resources:
        - group: ""
          resources: ["configmaps"]
      - level: Metadata
        resources:
        - group: ""
          resources: ["pods/log", "pods/status"]

For more details on the required configurations, see Auditing | Kubernetes

Configure clustering

Form a multi-node cluster that automatically provisions itself after installing. This requires separate configs for the init node and the rest of the cluster nodes:

On the cluster init node, configure a persistent token:

# configuration for init node of cluster
---
version: 0.1.0
persistentClusterToken: 3f6380c7c2950e74128ff037878b4925

On the rest of the nodes, specify the join string using the persistent token. This is identical to running microk8s join $join_url:

# configuration for other control plane nodes
---
version: 0.1.0
join:
  url: $FIRST_NODE_IP:25000/3f6380c7c2950e74128ff037878b4925

# configuration for other worker nodes
---
version: 0.1.0
join:
  url: $FIRST_NODE_IP:25000/3f6380c7c2950e74128ff037878b4925
  worker: true

3-node HA cluster with KubeOVN

On the cluster init node, enable kube-ovn and configure a cluster token:

---
version: 0.1.0
persistentClusterToken: 3f6380c7c2950e74128ff037878b4925
addons:
  - name: dns
  - name: kube-ovn
    args: [--force]

On the other two nodes, use a multi-part configuration to enable kube-ovn and then join the cluster:

---
version: 0.1.0
addons:
  - name: kube-ovn
    args: [--force]
---
version: 0.1.0
join:
  url: $FIRST_NODE_IP:25000/3f6380c7c2950e74128ff037878b4925

Specify a different service CIDR (before MicroK8s 1.28)

By default, MicroK8s uses the CIDR 10.152.183.0/24 for ClusterIP services. You may change it to 10.200.0.0/16 using the following config:

---
version: 0.1.0
extraKubeAPIServerArgs:
  --cluster-service-ip-range: 10.200.0.0/16
extraKubeletArgs:
  --cluster-dns: 10.200.0.10
  --cluster-domain: cluster.local
extraSANs:
  - 10.200.0.1
addons:
  - name: dns
    args: [/etc/resolv.conf, 10.200.0.10]

Kubernetes will use the first IP address of the specified range as the kubernetes service address, so we include it in the certificate Subject Alternate Names. Further, we set the ClusterIP address for the DNS service.

Specify a custom pod and service CIDR (requires MicroK8s 1.28 or newer)

Use 10.2.0.0/16 for pods and 10.94.0.0/24 for services. Make sure to add the 10.94.0.1 as an extra SAN as this address will be used by the kube-apiserver. The DNS addon will automatically use a valid service IP for the coredns service.

---
version: 0.2.0
extraCNIEnv:
  IPv4_CLUSTER_CIDR: "10.2.0.0/16"
  IPv4_SERVICE_CIDR: "10.94.0.0/24"
extraSANs:
  - 10.94.0.1
addons:
  - name: dns

Specify kubelet arguments

Configure kubelet for CPU isolation.

---
version: 0.1.0
extraKubeletArgs:
  --feature-gates: "CPUManager=true,TopologyManager=true"
  --system-reserved: "cpu=500m"
  --kube-reserved: "cpu=500m"
  --cpu-manager-policy: "static"
  --reserved-cpus: "1,2"