Clusterip, nodeport: what kind of ip? ipvs iptable? no arp, not visible in kernel?

General Discussions
1

Hi, this is a general question about services ip type: clusterip, nodeport etc…
I was wondering, how to know what kind of ip it is? how to see it in linux kernel or with linux commands?

I am using microk8s, i am using default clusterip on services and i wanted to identify them in linux kernel:

on linux i can see a calico net interface, used for pods:

4: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 66:e8:e7:fa:d1:f3 brd ff:ff:ff:ff:ff:ff
inet 10.1.128.192/32 scope global vxlan.calico
valid_lft forever preferred_lft forever
inet6 fe80::64e8:e7ff:fefa:d1f3/64 scope link

kube-system pod/metrics-server-6b6844c455-jkbgs 1/1 Running 0 2d15h 10.1.128.198 microk8s
kube-system pod/calico-kube-controllers-86c46c6b67-mg72x 1/1 Running 0 3d10h 10.1.128.193 microk8s

ok for that

and i can see on services clusterip/nodeport with ips not related to net interfaces. i suppose it il virtual ip but of what kind?

kube-system service/kube-dns ClusterIP 10.152.183.10 53/UDP,53/TCP,9153/TCP 3d10h k8s-app=kube-dns
kube-system service/metrics-server ClusterIP 10.152.183.199 443/TCP 2d15h k8s-app=metrics-server

iptables-save -t nat is empty and filter table show only access right obout a network which is unknown:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A FORWARD -s 10.1.0.0/16 -m comment --comment “generated for MicroK8s pods” -j ACCEPT
-A FORWARD -d 10.1.0.0/16 -m comment --comment “generated for MicroK8s pods” -j ACCEPT
COMMIT

Completed on Mon Nov 21 11:09:17 2022

Warning: iptables-legacy tables present, use iptables-legacy-save to see them

k8sadm@microk8s:~$ sudo iptables-save -t nat

Generated by iptables-save v1.8.7 on Mon Nov 21 11:09:26 2022

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT

ipvs show nothing:
k8sadm@microk8s:~$ sudo ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
→ RemoteAddress:Port Forward Weight ActiveConn InActConn

So how to identify these ips?

Regards,

2

Are you running iptables inside the pods or in the node one which the pods live?

Are you running kube-proxy or something else?

3

Hi.

on the node. It is a lxd container with all needed rights.

i don’t think microk8s use kube-proxy by default.

here are all objects:

NAMESPACE            NAME                                             READY   STATUS    RESTARTS   AGE     IP              NODE       NOMINATED NODE   READINESS GATES
default              pod/nginx-76d6c9b8c-vd7ld                        1/1     Running   0          5d11h   10.1.128.194    microk8s   <none>           <none>
kube-system          pod/hostpath-provisioner-5885cf6485-nhgfm        1/1     Running   0          5d11h   10.1.128.196    microk8s   <none>           <none>
kube-system          pod/calico-node-mgrjn                            1/1     Running   0          5d11h   172.16.99.124   microk8s   <none>           <none>
kube-system          pod/metrics-server-6b6844c455-jkbgs              1/1     Running   0          4d16h   10.1.128.198    microk8s   <none>           <none>
kube-system          pod/calico-kube-controllers-86c46c6b67-mg72x     1/1     Running   0          5d11h   10.1.128.193    microk8s   <none>           <none>
ingress              pod/nginx-ingress-microk8s-controller-c7mw2      1/1     Running   0          5d11h   10.1.128.197    microk8s   <none>           <none>
kube-system          pod/coredns-d489fb88-zlzdw                       1/1     Running   0          5d11h   10.1.128.195    microk8s   <none>           <none>
kube-system          pod/dashboard-metrics-scraper-64bcc67c9c-nbl9c   1/1     Running   0          4d16h   10.1.128.199    microk8s   <none>           <none>
kube-system          pod/kubernetes-dashboard-74b66d7f9c-6cszf        1/1     Running   0          4d16h   10.1.128.200    microk8s   <none>           <none>
container-registry   pod/registry-6674bf676f-67kgm                    1/1     Running   0          4d16h   10.1.128.202    microk8s   <none>           <none>
default              pod/grafounette-grafana-6dd4f5f7fb-r2p9w         1/1     Running   0          4d16h   10.1.128.205    microk8s   <none>           <none>
default              pod/postgres-postgresql-0                        1/1     Running   0          4d15h   10.1.128.207    microk8s   <none>           <none>

NAMESPACE            NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE     SELECTOR
default              service/kubernetes                  ClusterIP   10.152.183.1     <none>        443/TCP                  5d11h   <none>
kube-system          service/kube-dns                    ClusterIP   10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP   5d11h   k8s-app=kube-dns
kube-system          service/metrics-server              ClusterIP   10.152.183.199   <none>        443/TCP                  4d16h   k8s-app=metrics-server
kube-system          service/kubernetes-dashboard        ClusterIP   10.152.183.213   <none>        443/TCP                  4d16h   k8s-app=kubernetes-dashboard
kube-system          service/dashboard-metrics-scraper   ClusterIP   10.152.183.243   <none>        8000/TCP                 4d16h   k8s-app=dashboard-metrics-scraper
container-registry   service/registry                    NodePort    10.152.183.27    <none>        5000:32000/TCP           4d16h   app=registry
default              service/grafounette-grafana         ClusterIP   10.152.183.206   <none>        80/TCP                   4d16h   app.kubernetes.io/instance=grafounette,app.kubernetes.io/name=grafana
default              service/postgres-postgresql-hl      ClusterIP   None             <none>        5432/TCP                 4d15h   app.kubernetes.io/component=primary,app.kubernetes.io/instance=postgres,app.ku
bernetes.io/name=postgresql
default              service/postgres-postgresql         ClusterIP   10.152.183.235   <none>        5432/TCP                 4d15h   app.kubernetes.io/component=primary,app.kubernetes.io/instance=postgres,app.ku
bernetes.io/name=postgresql

NAMESPACE     NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE     CONTAINERS               IMAGES                             
          SELECTOR
kube-system   daemonset.apps/calico-node                         1         1         1       1            1           kubernetes.io/os=linux   5d11h   calico-node              docker.io/calico/node:v3.23.3      
          k8s-app=calico-node
ingress       daemonset.apps/nginx-ingress-microk8s-controller   1         1         1       1            1           <none>                   5d11h   nginx-ingress-microk8s   k8s.gcr.io/ingress-nginx/controller
:v1.2.0   name=nginx-ingress-microk8s

NAMESPACE            NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS                  IMAGES                                            SELECTOR
kube-system          deployment.apps/calico-kube-controllers     1/1     1            1           5d11h   calico-kube-controllers     docker.io/calico/kube-controllers:v3.23.3         k8s-app=calico-kube-control
lers
default              deployment.apps/nginx                       1/1     1            1           5d11h   nginx                       nginx                                             app=nginx
kube-system          deployment.apps/coredns                     1/1     1            1           5d11h   coredns                     coredns/coredns:1.9.3                             k8s-app=kube-dns
kube-system          deployment.apps/hostpath-provisioner        1/1     1            1           5d11h   hostpath-provisioner        cdkbot/hostpath-provisioner:1.4.1                 k8s-app=hostpath-provisione
r
kube-system          deployment.apps/metrics-server              1/1     1            1           4d16h   metrics-server              k8s.gcr.io/metrics-server/metrics-server:v0.5.2   k8s-app=metrics-server
kube-system          deployment.apps/dashboard-metrics-scraper   1/1     1            1           4d16h   dashboard-metrics-scraper   kubernetesui/metrics-scraper:v1.0.8               k8s-app=dashboard-metrics-s
craper
kube-system          deployment.apps/kubernetes-dashboard        1/1     1            1           4d16h   kubernetes-dashboard        kubernetesui/dashboard:v2.7.0                     k8s-app=kubernetes-dashboar
d
container-registry   deployment.apps/registry                    1/1     1            1           4d16h   registry                    registry:2.8.1                                    app=registry
default              deployment.apps/grafounette-grafana         1/1     1            1           4d16h   grafana                     grafana/grafana:9.2.5                             app.kubernetes.io/instance=
grafounette,app.kubernetes.io/name=grafana

NAMESPACE   NAME                                   READY   AGE     CONTAINERS   IMAGES
default     statefulset.apps/postgres-postgresql   1/1     4d15h   postgresql   docker.io/bitnami/postgresql:15.1.0-debian-11-r0

Regards.

4

Without something implementing Services (kube-proxy, cilium, something), then Service IPs probably don’t work at all.

5

yes it works.

for example:

k8sadm@microk8s:~$ curl -k https://10.152.183.199


{
  "paths": [
    "/apis",
    "/apis/metrics.k8s.io",
    "/apis/metrics.k8s.io/v1beta1",
    "/healthz",
    "/healthz/log",
    "/healthz/metadata-informer-sync",
    "/healthz/ping",
    "/healthz/poststarthook/generic-apiserver-start-informers",
    "/healthz/poststarthook/max-in-flight-filter",
    "/livez",
    "/livez/log",
    "/livez/metadata-informer-sync",
    "/livez/metric-collection-timely",
    "/livez/ping",
    "/livez/poststarthook/generic-apiserver-start-informers",
    "/livez/poststarthook/max-in-flight-filter",
    "/metrics",
    "/openapi/v2",
    "/readyz",
    "/readyz/informer-sync",
    "/readyz/log",
    "/readyz/metadata-informer-sync",
    "/readyz/metric-storage-ready",
    "/readyz/ping",
    "/readyz/poststarthook/generic-apiserver-start-informers",
    "/readyz/poststarthook/max-in-flight-filter",
    "/readyz/shutdown",
    "/version"
  ]
}k8sadm@microk8s:~$
6

Ok, so something is implementing it. I kube-proxy running maybe not as a pod?

7

no, there is no kube-proxy