Why ImagePolicyWebhook cannot run as a pod itself?

General Discussions
1

Cluster information:

Kubernetes version: 1.26
Cloud being used: Azure
Installation method: kubeadm
Host OS: Linux
CNI and version: calico v3.24.1 + flannel 0.15.1
CRI and version: containerd://1.6.12

I followed the instructions on GitHub - flavio/kube-image-bouncer: Simple endpoint for the ImagePolicyWebhook and the GenericAdmissionWebhook Kubernetes admission controllers, but instead of installing it on the host or running in a docker I have arranged it as a ClusterIP service running on the same cluster. Temporarily, of course. Consider this an exercise.

Anyway, the service is working fine:

mark@cks-master:~ $ k exec test -- curl -s https://kube-image-bouncer.kube-image-bouncer.svc/image_policy --cacert /webhook.pem && echo
{"message":"Method Not Allowed"}
mark@cks-master:~ $

So, I have created all the necessary configuration, made the changes to the kube-apiserver static pod and:

mark@cks-master:~ $ k run test2 --image nginx
Error from server (Forbidden): pods "test2" is forbidden: Post "https://kube-image-bouncer.kube-image-bouncer.svc/image_policy?timeout=30s": dial tcp: lookup kube-image-bouncer.kube-image-bouncer.svc on 168.63.129.16:53: no such host
mark@cks-master:~ $

This looks like kube-image-bouncer.kube-image-bouncer.svc fails to resolve. But I know it resolves just fine when running in a pod - see the previous curl example.

I thought that the admission controllers are run by the kube-controller-manager, i.e. on a pod and thus should have no problem resolving kube-image-bouncer.kube-image-bouncer.svc.

Obviously, I misunderstand something. Can someone explain what is going on?

P.S.
I also asked the question on SO, but got no answers - kubernetes - Why am I unable to run ImagePolicyWebhook service as a regular ClusterIP service on the same cluster? - Stack Overflow

2

Hi,
You need to apply correct DNS policy DNS for Services and Pods | Kubernetes for pods running with hostNetwork.

3

Hi,
Thank you for your answer.
How do you know that image bouncer pods are running with hostNetwork ? I looked at GitHub - flavio/kube-image-bouncer: Simple endpoint for the ImagePolicyWebhook and the GenericAdmissionWebhook Kubernetes admission controllers and I may be looking in wrong places, where can one see it?

4

Hi,
The problem is in the kube-apiserver pod. It is using hostNetwork and is not able to resolve hosts known to CoreDNS by default. In order to change this behavior, ClusterFirstWithHostNet DNS policy should be configured for the kube-apiserver pod.
kubectl -n kube-system get pods -l component=kube-apiserver -o yaml | grep -i hostnetwork output will assure that hostNetwork is used by a pod.

5

I see.
Thank you.