Kubernetes version: v1.16.4
Cloud being used: vSphere, on prem
Installation method: kubeadm
Host OS: Ubuntu 18.04
CNI and version: Canal 3.8
CRI and version: Docker 18.09
I encountered this problem, when trying secure kubernetes dashboard with Network Policies. At the moment of writing, the dashboard consists of two pods: the dashboard itself and the metrics scraper. The dashboard periodically calls the metrics scraper, and thus needs access to it.
It appears, that the way the dashboard accesses the metrics scraper is via the API server proxy. It also appears, the pod and namespace selectors in a NetworkPolicy object do nothing to allow that type of access. I was able to use
0.0.0.0/0 in order to give access, but this defeats the purpose of having a NetworkPolicy. I would like to be a lot more concrete with what I’m allowing access from.
One peculiar thing, is that when access is working, the incoming IP address in the logs of metric scrape is displayed as
10.244.0.0 which is the entire Canal subnet that I’m using for kubernetes networking.
10.244.0.0/16 in the NetworkPolicy object also works, allowing access, but it is not much better. Specifying
10.244.0.0/32 (the IP address in the logs) does not allow access.
Note: I was just looking for link to Canal page on Calico web site but it disappeared. Similarly, any mention of Canal also disappeared from the kubernetes website. I’m guessing I should read it as that it’s deprecated. Anyway, at this stage I do not know if this is networking provider related or now, my current guess is that it’s not.
Below is how to reproduce this issue from scratch, that is without reliance on the dashboard. Any ideas how to resolve it are greatly appreciated.
# Create namespace where all our test objects will reside. At the end we delete the whole namespace with everything in it kubectl create namespace testing # This is the service account that we will give access to api server proxy to kubectl create serviceaccount -n testing curl # This role describes access to api server proxy for our nginx service, port 80 kubectl create role -n testing curl --verb=get --resource=services/proxy --resource-name=nginx:80 # Bind the account and the role above together kubectl create rolebinding -n testing curl --role=curl --serviceaccount=testing:curl # Create a nginx server kubectl run nginx -n testing --image=nginx --labels="app=nginx" --generator=run-pod/v1 # Create a curl container to test connectivity from kubectl run curl -n testing --serviceaccount=curl --image=curlimages/curl --labels="app=curl" --generator=run-pod/v1 sleep 999999 # Create service that exposes nginx (mainly for discovery purposes) kubectl create service -n testing clusterip nginx --tcp=80:80 # Try accessing nginx from the curl container - it works kubectl exec -n testing -it curl -- curl http://nginx # And also via api server proxy - it works kubectl exec -n testing -it curl -- sh -c 'curl -k "https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT_HTTPS/api/v1/namespaces/testing/services/nginx:80/proxy/" --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"' # Look and nginx logs. Note, how the source IP for the api server proxy call. In my case it's x.y.z.0 kubectl logs -n testing nginx # Let's create and apply NetworkPolicy objects for the pods we just created cat <<EOF > network-policy.yaml --- kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: nginx namespace: testing spec: podSelector: matchLabels: app: nginx policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: app: curl ports: - protocol: TCP port: 80 EOF kubectl apply -f network-policy.yaml # At this stage straight nginx call is still working: kubectl exec -n testing -it curl -- curl http://nginx # But api server proxy call no longer does kubectl exec -n testing -it curl -- sh -c 'curl -k "https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT_HTTPS/api/v1/namespaces/testing/services/nginx:80/proxy/" --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"' # clean up kubectl delete namespace testing