Kubernetes and Security Benchmarks

Kubernetes and Security Benchmarks

Context

The CIS benchmark suites are widely accepted in security teams as authoritative, and issues have been created for a number of versions of Kubernetes. Each benchmark consists of a number controls with regards to a secure Kubernetes configuration, which are updated for each release of Kubernetes. The controls include sample code to check for the control, often based around running grep on a configuration file. These checks are then encoded within open source and vendor projects such as kube-bench, dev-sec.io, Twistlock Server Defender, etc…

However, the CIS benchmarks can be brittle, and have not kept up to date with changes in the project, particularly around ComponentConfig and kubeadm.

Issues with CIS Kubernetes Benchmark 1.4

Supported by kubeadm, but not as defaults

  • 1.1.8, 1.2.1, 1.3.2 - Ensure --profiling is disabled
  • 1.1.11, 1.1.24, 1.1.33, 1.1.36 - Enable admission controllers
  • 1.1.15-1.1.18 - Enable auditing
  • 1.1.30 - Enable strong ciphers
  • 1.1.34-1.1.35 - Configure encryption provider
  • 1.1.37 - Configure AdvancedAuditing

All of these report errors in the CIS benchmark, but can be configured via kubeadm, mainly via the extraArgs configuration of each component.

Hard problems

Control 1.1.1 - Disable anonymous authentication

Requires --anonymous-auth=false to be set on controller-manager and scheduler. This does not currently work as kubeadm lays static manifests with HTTP readiness and liveness checks. With anonymous authentication disabled, these readiness and liveness checks do not pass subject access review and fail.

Solutions
Create health check endpoint in API server

Add a new port for a /healthz endpoint that is unauthenticated that can be consumed in the pod definition.

Change benchmark

With anonymous-auth=true, only /healthz and … accepts requests without requiring authorization. With recent versions of Kubernetes (specify), this option should be safe.

Control 1.4.12 - Ownership of etcd directories

Control 1.4.12 specifies that /var/lib/etcd should be owned by etcd:etcd, however kubeadm sets up static pod manifests and runs etcd via kubelet with /var/lib/etcd mounted. The process runs as root within the namespace.

Solutions

Remove control from benchmark

Easiest approach is to change the benchmark to not perform this check as the volume is confined within the container namespace. Kubeadm cannot guarantee the username or group of the etcd process even if it were to use an image that does not run as root.

Change check to !root

Assuming that upstream etcd container images are changed to not run as root, then it may be possible to do a filesystem group check on /var/lib/etcd to check that it’s not equal to 0, with the assumption etcd is running with fsgroup set in the security context.

Control 1.7.4 0 - Use of host network namespace

By design, kubeadm lays down control plane components on each host to live on the host network namespace.

Solutions

Tooling should not fail control plane components.
Etcd, kube-apiserver, kube-controller-manager and kube-scheduler should be excluded from failures, or at scanners should accept an allow-list for kubeadm based systems to supply defaults.

Control 2.1.10 - Specifying TLS cert and private key file paths.

Kubeadm uses TLS bootstrapping, and these settings are now mutually exclusive with manually setting --tls-cert-file & --tls-private-key-file.

Solutions
Remove control and replace with TLS bootstrapping checks

Use of kubelet TLS bootstrapping should be considered the default for some time.

Control 2.1.13 - Kubelet Server Certificate Rotation

The CIS benchmark requires enabling and configuring RotateKubeletServerCertificate. However, servlet certificate rotation is not fully implemented in Kubernetes. Kubelets are able to submit certificate signing requests but at present, there is no automatic signing mechanism. This is because the CSR mechanism does not contain enough information about host identity to reliably issue certificates.

Many operators work around this by using a “rubber stamp” mechanism that automatically allows all CSR requests, introducing more security vulnerabilities.

Host identity should be considered as a higher order concern, and could be solved within projects such as Cluster API.

Solutions
Remove control

We do not believe this control should continue to exist until the host identity problem is solved. Right now, kubeadm will produce self-signed certificates during kubelet bootstrap. This is currently the best option without external cluster orchestration such as Cluster API.

Changing scanning methodologies

All of the security scanners based on the CIS benchmarks currently operate by grepping configuration files on disk. This is brittle and cannot account for the various ways in which kubernetes components can be configured - currently by

Issues with CIS Docker benchmark

Many organisations will try to apply both the CIS benchmarks for Kubernetes as well as Docker. However, the CIS Docker benchmark is geared around deployments of Docker Swarm, and can mislead Kubernetes users into believing their install is not secure.

2.8 Enable user namespace support

User namespace support is not yet available in Kubernetes, and other mechanisms such as PodSecurityPolicies can achieve many of the same benefits.

5.18 Overwrite default ulimit

Ulimit support still is not aailable in Kubernetes, so this control cannot be apllied.
5.25, 2.18 Ensure containers are restricted from acquirigin additional privileges
Kubeadm does not set the allowPrivilegeEscalation setting, but some CNIs do, so most clusters will “fail” this control.

5.28 Ensure PIDs cgroup limit is used

Kubernetes does not support setting the cgroup limit

Other issues with cluster bootstrapping

PodSecurityPolicies for Kube-proxy and CNI

Kube-proxy and all CNIs require privileges to modify host networking, but most of the tools will

1 Like

Thanks for this awesome write up on the various Kubernetes CIS benchmark items that might not have obvious solutions to resolving on a kubeadm-based installation! I’m wondering what are your (and anyone in the community’s) thoughts on the overall effectiveness of the CIS benchmarks to measure the security of Kubernetes clusters. You make a few opinions in the beginning but I was wondering if we could dive a little further into a meaningful conversation about it?

As far as not being to stay up to date, what do you think about a collaborative effort between Kubernetes and CIS to provide the latest and greatest in best security practices for this wonderful container orchestration platform?

Thanks!

1 Like

The CIS Benchmark is a community effort - please sign up at workbench.cisecurity.org if you’d like to get involved. More help keeping it up-to-date as Kubernetes evolves would be very welcome!

2 Likes

Hi @beekay24. Have taken up @lizrice 's suggestion and started filing some tickets.

We are also scoping out what improvements we can do to be more secure by default. I’ll try and link these here as we go along.

2 Likes