Clusterip, nodeport: what kind of ip? ipvs iptable? no arp, not visible in kernel?

Hi, this is a general question about services ip type: clusterip, nodeport etc…
I was wondering, how to know what kind of ip it is? how to see it in linux kernel or with linux commands?

I am using microk8s, i am using default clusterip on services and i wanted to identify them in linux kernel:

on linux i can see a calico net interface, used for pods:

4: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 66:e8:e7:fa:d1:f3 brd ff:ff:ff:ff:ff:ff
inet 10.1.128.192/32 scope global vxlan.calico
valid_lft forever preferred_lft forever
inet6 fe80::64e8:e7ff:fefa:d1f3/64 scope link

kube-system pod/metrics-server-6b6844c455-jkbgs 1/1 Running 0 2d15h 10.1.128.198 microk8s
kube-system pod/calico-kube-controllers-86c46c6b67-mg72x 1/1 Running 0 3d10h 10.1.128.193 microk8s

ok for that

and i can see on services clusterip/nodeport with ips not related to net interfaces. i suppose it il virtual ip but of what kind?

kube-system service/kube-dns ClusterIP 10.152.183.10 53/UDP,53/TCP,9153/TCP 3d10h k8s-app=kube-dns
kube-system service/metrics-server ClusterIP 10.152.183.199 443/TCP 2d15h k8s-app=metrics-server

iptables-save -t nat is empty and filter table show only access right obout a network which is unknown:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A FORWARD -s 10.1.0.0/16 -m comment --comment “generated for MicroK8s pods” -j ACCEPT
-A FORWARD -d 10.1.0.0/16 -m comment --comment “generated for MicroK8s pods” -j ACCEPT
COMMIT

Completed on Mon Nov 21 11:09:17 2022

Warning: iptables-legacy tables present, use iptables-legacy-save to see them

k8sadm@microk8s:~$ sudo iptables-save -t nat

Generated by iptables-save v1.8.7 on Mon Nov 21 11:09:26 2022

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT

ipvs show nothing:
k8sadm@microk8s:~$ sudo ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
→ RemoteAddress:Port Forward Weight ActiveConn InActConn

So how to identify these ips?

Regards,

Are you running iptables inside the pods or in the node one which the pods live?

Are you running kube-proxy or something else?

Hi.

on the node. It is a lxd container with all needed rights.

i don’t think microk8s use kube-proxy by default.

here are all objects:

NAMESPACE            NAME                                             READY   STATUS    RESTARTS   AGE     IP              NODE       NOMINATED NODE   READINESS GATES
default              pod/nginx-76d6c9b8c-vd7ld                        1/1     Running   0          5d11h   10.1.128.194    microk8s   <none>           <none>
kube-system          pod/hostpath-provisioner-5885cf6485-nhgfm        1/1     Running   0          5d11h   10.1.128.196    microk8s   <none>           <none>
kube-system          pod/calico-node-mgrjn                            1/1     Running   0          5d11h   172.16.99.124   microk8s   <none>           <none>
kube-system          pod/metrics-server-6b6844c455-jkbgs              1/1     Running   0          4d16h   10.1.128.198    microk8s   <none>           <none>
kube-system          pod/calico-kube-controllers-86c46c6b67-mg72x     1/1     Running   0          5d11h   10.1.128.193    microk8s   <none>           <none>
ingress              pod/nginx-ingress-microk8s-controller-c7mw2      1/1     Running   0          5d11h   10.1.128.197    microk8s   <none>           <none>
kube-system          pod/coredns-d489fb88-zlzdw                       1/1     Running   0          5d11h   10.1.128.195    microk8s   <none>           <none>
kube-system          pod/dashboard-metrics-scraper-64bcc67c9c-nbl9c   1/1     Running   0          4d16h   10.1.128.199    microk8s   <none>           <none>
kube-system          pod/kubernetes-dashboard-74b66d7f9c-6cszf        1/1     Running   0          4d16h   10.1.128.200    microk8s   <none>           <none>
container-registry   pod/registry-6674bf676f-67kgm                    1/1     Running   0          4d16h   10.1.128.202    microk8s   <none>           <none>
default              pod/grafounette-grafana-6dd4f5f7fb-r2p9w         1/1     Running   0          4d16h   10.1.128.205    microk8s   <none>           <none>
default              pod/postgres-postgresql-0                        1/1     Running   0          4d15h   10.1.128.207    microk8s   <none>           <none>

NAMESPACE            NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE     SELECTOR
default              service/kubernetes                  ClusterIP   10.152.183.1     <none>        443/TCP                  5d11h   <none>
kube-system          service/kube-dns                    ClusterIP   10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP   5d11h   k8s-app=kube-dns
kube-system          service/metrics-server              ClusterIP   10.152.183.199   <none>        443/TCP                  4d16h   k8s-app=metrics-server
kube-system          service/kubernetes-dashboard        ClusterIP   10.152.183.213   <none>        443/TCP                  4d16h   k8s-app=kubernetes-dashboard
kube-system          service/dashboard-metrics-scraper   ClusterIP   10.152.183.243   <none>        8000/TCP                 4d16h   k8s-app=dashboard-metrics-scraper
container-registry   service/registry                    NodePort    10.152.183.27    <none>        5000:32000/TCP           4d16h   app=registry
default              service/grafounette-grafana         ClusterIP   10.152.183.206   <none>        80/TCP                   4d16h   app.kubernetes.io/instance=grafounette,app.kubernetes.io/name=grafana
default              service/postgres-postgresql-hl      ClusterIP   None             <none>        5432/TCP                 4d15h   app.kubernetes.io/component=primary,app.kubernetes.io/instance=postgres,app.ku
bernetes.io/name=postgresql
default              service/postgres-postgresql         ClusterIP   10.152.183.235   <none>        5432/TCP                 4d15h   app.kubernetes.io/component=primary,app.kubernetes.io/instance=postgres,app.ku
bernetes.io/name=postgresql

NAMESPACE     NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE     CONTAINERS               IMAGES                             
          SELECTOR
kube-system   daemonset.apps/calico-node                         1         1         1       1            1           kubernetes.io/os=linux   5d11h   calico-node              docker.io/calico/node:v3.23.3      
          k8s-app=calico-node
ingress       daemonset.apps/nginx-ingress-microk8s-controller   1         1         1       1            1           <none>                   5d11h   nginx-ingress-microk8s   k8s.gcr.io/ingress-nginx/controller
:v1.2.0   name=nginx-ingress-microk8s

NAMESPACE            NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS                  IMAGES                                            SELECTOR
kube-system          deployment.apps/calico-kube-controllers     1/1     1            1           5d11h   calico-kube-controllers     docker.io/calico/kube-controllers:v3.23.3         k8s-app=calico-kube-control
lers
default              deployment.apps/nginx                       1/1     1            1           5d11h   nginx                       nginx                                             app=nginx
kube-system          deployment.apps/coredns                     1/1     1            1           5d11h   coredns                     coredns/coredns:1.9.3                             k8s-app=kube-dns
kube-system          deployment.apps/hostpath-provisioner        1/1     1            1           5d11h   hostpath-provisioner        cdkbot/hostpath-provisioner:1.4.1                 k8s-app=hostpath-provisione
r
kube-system          deployment.apps/metrics-server              1/1     1            1           4d16h   metrics-server              k8s.gcr.io/metrics-server/metrics-server:v0.5.2   k8s-app=metrics-server
kube-system          deployment.apps/dashboard-metrics-scraper   1/1     1            1           4d16h   dashboard-metrics-scraper   kubernetesui/metrics-scraper:v1.0.8               k8s-app=dashboard-metrics-s
craper
kube-system          deployment.apps/kubernetes-dashboard        1/1     1            1           4d16h   kubernetes-dashboard        kubernetesui/dashboard:v2.7.0                     k8s-app=kubernetes-dashboar
d
container-registry   deployment.apps/registry                    1/1     1            1           4d16h   registry                    registry:2.8.1                                    app=registry
default              deployment.apps/grafounette-grafana         1/1     1            1           4d16h   grafana                     grafana/grafana:9.2.5                             app.kubernetes.io/instance=
grafounette,app.kubernetes.io/name=grafana

NAMESPACE   NAME                                   READY   AGE     CONTAINERS   IMAGES
default     statefulset.apps/postgres-postgresql   1/1     4d15h   postgresql   docker.io/bitnami/postgresql:15.1.0-debian-11-r0

Regards.

Without something implementing Services (kube-proxy, cilium, something), then Service IPs probably don’t work at all.

yes it works.

for example:

k8sadm@microk8s:~$ curl -k https://10.152.183.199


{
  "paths": [
    "/apis",
    "/apis/metrics.k8s.io",
    "/apis/metrics.k8s.io/v1beta1",
    "/healthz",
    "/healthz/log",
    "/healthz/metadata-informer-sync",
    "/healthz/ping",
    "/healthz/poststarthook/generic-apiserver-start-informers",
    "/healthz/poststarthook/max-in-flight-filter",
    "/livez",
    "/livez/log",
    "/livez/metadata-informer-sync",
    "/livez/metric-collection-timely",
    "/livez/ping",
    "/livez/poststarthook/generic-apiserver-start-informers",
    "/livez/poststarthook/max-in-flight-filter",
    "/metrics",
    "/openapi/v2",
    "/readyz",
    "/readyz/informer-sync",
    "/readyz/log",
    "/readyz/metadata-informer-sync",
    "/readyz/metric-storage-ready",
    "/readyz/ping",
    "/readyz/poststarthook/generic-apiserver-start-informers",
    "/readyz/poststarthook/max-in-flight-filter",
    "/readyz/shutdown",
    "/version"
  ]
}k8sadm@microk8s:~$ 

Ok, so something is implementing it. I kube-proxy running maybe not as a pod?

no, there is no kube-proxy