HNS Issues on Kubernetes Cluster with Windows worker nodes

General Discussions Windows
1

Hello,

Need some pointers here on how to address these issues:

Recurring HNS endpoint “endpoint was not found” + HNS namespace leak (FailedKillPod) on Windows Server 2025 nodes — RKE2 v1.34 / containerd 2.1.5 / Calico v3.30.4 VXLAN

Environment

Orchestrator: RKE2 v1.34.2+rke2r1 (Kubernetes 1.34.2)

Control plane: Ubuntu 24.04.3 LTS, kernel 6.8.0-106

Windows workers: Windows Server 2025 Datacenter, builds 10.0.26100.32370, .32522, .32860

Hardware: Dell PowerEdge R760 / R960, amd64

Container runtime: containerd 2.1.5-k3s1 (both Linux and Windows)

CNI: Calico v3.30.4 (open source), Tigera operator v1.38.7 (Rancher-mirrored)

Encapsulation: VXLAN (vxlanMode: Always, ipipMode: Never), pod CIDR 10.42.0.0/16, natOutgoing enabled

Windows networking: HNS-backed (containerd + Calico HNS), VXLAN overlay

Workload: Windows container build/test agents (docker-in-docker style), moderate pod churn

Symptom 1 — Container fails to start: HNS endpoint missing

Windows pods intermittently get stuck in CrashLoopBackOff / RunContainerError with StartError (exit 128). Container status message:

failed to create containerd task: failed to create shim task:
hcs::CreateComputeSystem : The endpoint was not found.

Pods restart indefinitely (we’ve seen 40+ restarts over 27h on one). It clusters on specific nodes while others are fine. The node itself stays Ready with no resource pressure.

Symptom 2 — Pod stuck Terminating: HNS namespace leak / FailedKillPod

Separately (and we suspect related), pods get stuck in Terminating indefinitely, with kubelet unable to tear down the sandbox:

FailedKillPod … error killing pod … 0x803b0015

The HCN/HNS namespace appears to leak — the endpoint/namespace layer isn’t cleaned up on pod teardown, and subsequent pods on that node then can’t get an endpoint (feeding Symptom 1). A stuck-Terminating pod can also wedge anything using concurrencyPolicy: Forbid (e.g. a cleanup CronJob blocked for days).

Restarting the HNS service does not reliably clear it; only a full node reboot does

What we’ve tried

  • A watchdog DaemonSet that scans for stale HNS endpoints and StartError pods and attempts cleanup — it reduces but does not eliminate the issue (it reports “network healthy / NO_ACTION” even while endpoint was not found crashloops are active, so detection is imperfect).
  • Periodic node reboots as the only reliable remediation.

Questions

  1. Are stale HNS endpoint / HNS namespace leaks under containerd 2.1.5 + Calico VXLAN on Windows Server 2025 (26100) a known issue, and is there a fix or newer build that addresses it?
  2. Is there a supported way to reclaim leaked HNS namespaces/endpoints without a full reboot (e.g. a safe hnsdiag/Remove-HnsEndpoint procedure, or a CNI/containerd setting)?
  3. Is The endpoint was not found on hcs::CreateComputeSystem a downstream effect of the 0x803b0015 FailedKillPod leak, or a separate Calico HNS endpoint-GC bug?
  4. Any recommended Calico (v3.30.x) or kube-proxy (HNS) settings to harden endpoint cleanup on high-churn Windows nodes?

Appreciate any help or guidance on these issues.

Thanks,

PMS