Kubernetes and Security Benchmarks
The CIS benchmark suites are widely accepted in security teams as authoritative, and issues have been created for a number of versions of Kubernetes. Each benchmark consists of a number controls with regards to a secure Kubernetes configuration, which are updated for each release of Kubernetes. The controls include sample code to check for the control, often based around running grep on a configuration file. These checks are then encoded within open source and vendor projects such as kube-bench, dev-sec.io, Twistlock Server Defender, etc…
However, the CIS benchmarks can be brittle, and have not kept up to date with changes in the project, particularly around ComponentConfig and kubeadm.
Issues with CIS Kubernetes Benchmark 1.4
Supported by kubeadm, but not as defaults
- 1.1.8, 1.2.1, 1.3.2 - Ensure
- 1.1.11, 1.1.24, 1.1.33, 1.1.36 - Enable admission controllers
- 1.1.15-1.1.18 - Enable auditing
- 1.1.30 - Enable strong ciphers
- 1.1.34-1.1.35 - Configure encryption provider
- 1.1.37 - Configure
All of these report errors in the CIS benchmark, but can be configured via kubeadm, mainly via the
extraArgs configuration of each component.
Control 1.1.1 - Disable anonymous authentication
--anonymous-auth=false to be set on controller-manager and scheduler. This does not currently work as kubeadm lays static manifests with HTTP readiness and liveness checks. With anonymous authentication disabled, these readiness and liveness checks do not pass subject access review and fail.
Create health check endpoint in API server
Add a new port for a
/healthz endpoint that is unauthenticated that can be consumed in the pod definition.
- Related issues: https://github.com/kubernetes/kubernetes/issues/43784
With anonymous-auth=true, only
/healthz and … accepts requests without requiring authorization. With recent versions of Kubernetes (specify), this option should be safe.
Control 1.4.12 - Ownership of etcd directories
Control 1.4.12 specifies that
/var/lib/etcd should be owned by etcd:etcd, however kubeadm sets up static pod manifests and runs etcd via kubelet with /var/lib/etcd mounted. The process runs as root within the namespace.
Remove control from benchmark
Easiest approach is to change the benchmark to not perform this check as the volume is confined within the container namespace. Kubeadm cannot guarantee the username or group of the etcd process even if it were to use an image that does not run as root.
Change check to !root
Assuming that upstream etcd container images are changed to not run as root, then it may be possible to do a filesystem group check on /var/lib/etcd to check that it’s not equal to 0, with the assumption etcd is running with fsgroup set in the security context.
Control 1.7.4 0 - Use of host network namespace
By design, kubeadm lays down control plane components on each host to live on the host network namespace.
Tooling should not fail control plane components.
Etcd, kube-apiserver, kube-controller-manager and kube-scheduler should be excluded from failures, or at scanners should accept an allow-list for kubeadm based systems to supply defaults.
Control 2.1.10 - Specifying TLS cert and private key file paths.
Kubeadm uses TLS bootstrapping, and these settings are now mutually exclusive with manually setting --tls-cert-file & --tls-private-key-file.
Remove control and replace with TLS bootstrapping checks
Use of kubelet TLS bootstrapping should be considered the default for some time.
Control 2.1.13 - Kubelet Server Certificate Rotation
The CIS benchmark requires enabling and configuring RotateKubeletServerCertificate. However, servlet certificate rotation is not fully implemented in Kubernetes. Kubelets are able to submit certificate signing requests but at present, there is no automatic signing mechanism. This is because the CSR mechanism does not contain enough information about host identity to reliably issue certificates.
Many operators work around this by using a “rubber stamp” mechanism that automatically allows all CSR requests, introducing more security vulnerabilities.
Host identity should be considered as a higher order concern, and could be solved within projects such as Cluster API.
We do not believe this control should continue to exist until the host identity problem is solved. Right now, kubeadm will produce self-signed certificates during kubelet bootstrap. This is currently the best option without external cluster orchestration such as Cluster API.
Changing scanning methodologies
All of the security scanners based on the CIS benchmarks currently operate by grepping configuration files on disk. This is brittle and cannot account for the various ways in which kubernetes components can be configured - currently by
Issues with CIS Docker benchmark
Many organisations will try to apply both the CIS benchmarks for Kubernetes as well as Docker. However, the CIS Docker benchmark is geared around deployments of Docker Swarm, and can mislead Kubernetes users into believing their install is not secure.
2.8 Enable user namespace support
User namespace support is not yet available in Kubernetes, and other mechanisms such as PodSecurityPolicies can achieve many of the same benefits.
5.18 Overwrite default ulimit
Ulimit support still is not aailable in Kubernetes, so this control cannot be apllied.
5.25, 2.18 Ensure containers are restricted from acquirigin additional privileges
Kubeadm does not set the allowPrivilegeEscalation setting, but some CNIs do, so most clusters will “fail” this control.
5.28 Ensure PIDs cgroup limit is used
Kubernetes does not support setting the cgroup limit
Other issues with cluster bootstrapping
PodSecurityPolicies for Kube-proxy and CNI
Kube-proxy and all CNIs require privileges to modify host networking, but most of the tools will