Hello Kubernetes Community,
I am currently working on a project where I need to analyse the internal traffic within a single-node Kubernetes cluster, specifically at the packet level. My goal is to monitor the traffic between the Kubernetes API server and the kubelet, as well as the kubelet’s communication with the pods. I’m particularly interested in testing whether different container runtimes (runc, gVisor, and Kata Containers) disclose varying amounts of information depending on their isolation level.
The main challenge I’m facing is that Kubernetes communication is encrypted with TLS 1.3, which uses Perfect Forward Secrecy (PFS). This means that even though I have access to the Kubernetes keys stored in /etc/kubernetes/pki/, they are not sufficient to decrypt the traffic since PFS session keys are generated on a per-session basis. While SSL key logs could be a solution in other environments, Kubernetes components are written in Go, which does not natively support this.
Here’s what I’ve tried so far:
1. Log SSL Keys: Since Go lacks native SSL key logging support, this approach was unsuccessful.
2. MITM (Man in the middle) Proxy: I attempted to intercept traffic via a MITM proxy to decrypt the data, but the traffic remained encrypted. Decrypting kubernetes master api calls - Stack Overflow
3. Disable TLS: I tried disabling TLS for communication between the API server and kubelet, but after modifying the relevant configuration files, the Kubernetes system became non-functional.
4. Sidecar Container with tcpdump: I ran tcpdump from a sidecar container to capture traffic, but the results were encrypted, similar to when using Wireshark. Using sidecars to analyze and debug network traffic in OpenShift and Kubernetes pods | Red Hat Developer
5. Tools: I have also used Calico Enterprise and Kubeshark, which provide more user-friendly visualizations, but they do not offer decryption features.
Given these challenges, I’m seeking advice on how to proceed:
• Is there a way to decrypt the TLS 1.3 traffic or capture the session keys in a Kubernetes environment?
• Are there any known workarounds or tools that could help me analyze internal Kubernetes traffic at the packet level in the context of different container runtimes?
Any guidance or suggestions would be greatly appreciated!
Thank you!
Kubernetes version:
- Client Version: v1.31.1
- Kustomize Version: v5.4.2
- Server Version: v1.31.0
Cloud being used: bare-metal
Installation method: K8s installation guide
Host OS: Ubuntu 22.04.5 LTS
CNI and version: Calico v3.26.1
CRI and version: Containerd v1.7.22