Service Topology and EndpointSlices complete usage

zorruch · April 22, 2020, 1:56pm

Cluster information:

Kubernetes version: 1.18.2
Cloud being used: bare-metal
Installation method: kubeadm
Host OS: ubuntu 18.04
CNI and version: calico
CRI and version:

hi.

I want to start cluster with Service Topology.

I read:
Service Topology - Kubernetes EndpointSlices - Kubernetes
and
Enabling EndpointSlices - Kubernetes

I enabled startup options for the service kube-proxy:
–feature-gates=EndpointSliceProxying=true

and created service:
dns.yaml:

 apiVersion: v1
 kind: Service
 metadata:
   name: kube-dns2
 spec:
   ports:
   - name: dns
     port: 53
     protocol: UDP
     targetPort: 53
   selector:
     k8s-app: kube-dns
   topologyKeys:
   - "kubernetes.io/hostname"

 # kubectl  get svc kube-dns2  -n  kube-system
NAME        TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
kube-dns2   ClusterIP   10.106.3.98   <none>        53/UDP    14h

Automatically created endpointslice:

# kubectl  describe  endpointslice kube-dns2-b8rp4  -n  kube-system   
Name:         kube-dns2-b8rp4
Namespace:    kube-system
Labels:       endpointslice.kubernetes.io/managed-by=endpointslice-controller.k8s.io
              kubernetes.io/service-name=kube-dns2
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2020-04-21T23:34:13Z
AddressType:  IPv4
Ports:
  Name  Port  Protocol
  ----  ----  --------
  dns   53    UDP
Endpoints:
  - Addresses:  10.244.148.214
    Conditions:
      Ready:    true
    Hostname:   <unset>
    TargetRef:  Pod/coredns-6cfdfcf764-d4rjd
    Topology:   kubernetes.io/hostname=kuber-01
  - Addresses:  10.244.40.27
    Conditions:
      Ready:    true
    Hostname:   <unset>
    TargetRef:  Pod/coredns-6cfdfcf764-m5gf8
    Topology:   kubernetes.io/hostname=kuber-04
  - Addresses:  10.244.235.166
    Conditions:
      Ready:    true
    Hostname:   <unset>
    TargetRef:  Pod/coredns-6cfdfcf764-vwfgh
    Topology:   kubernetes.io/hostname=kuber-06
  - Addresses:  10.244.235.182
    Conditions:
      Ready:    true
    Hostname:   <unset>
    TargetRef:  Pod/coredns-6cfdfcf764-qrs5n
    Topology:   kubernetes.io/hostname=kuber-06

I use ipvs for kube-proxy
On the node in the ipvsadm -ln command listing

UDP  10.106.3.98:53 rr
  -> 10.244.40.27:53              Masq    1      0          0
  -> 10.244.40.54:53              Masq    1      0          0
  -> 10.244.97.70:53              Masq    1      0          0
  -> 10.244.97.75:53              Masq    1      0          0
  -> 10.244.97.76:53              Masq    1      0          0
  -> 10.244.97.78:53              Masq    1      0          0
  -> 10.244.106.1:53              Masq    1      0          0
  -> 10.244.106.2:53              Masq    1      0          0
  -> 10.244.106.24:53             Masq    1      0          0
  -> 10.244.106.50:53             Masq    1      0          0

How can I use the created service? Ipvs rules are created for proxying requests to all my pods, I want the request to be sent only to the local dns

I read that kube-proxy should switch from using endpoint to endpointslice when using the option --feature-gates=EndpointSliceProxying=true. but it does not work

I will be happy for any help or manual for using Service Topology or EndpointSlices .

thockin · April 22, 2020, 3:21pm

Caution that the topology API is being reconsidered right now, so it may ultimately be different. It’s alpha for a reason.

zorruch · April 22, 2020, 3:31pm

I understand that this is an alpha version
I will be glad if you point out a mistake or help me understand the concept of how this should work

thockin · April 22, 2020, 4:06pm

Did you enable the ServiceTopology gate?

zorruch · April 22, 2020, 4:12pm

no

in which component should i do this?
kube-proxy, kube-api, kube-sheduler?

thanks

thockin · April 22, 2020, 5:54pm

In general, just enable the same set of gates in all components

zorruch · April 23, 2020, 1:58pm

I enabled gates:
kube-apiserver: --feature-gates=ServiceTopology=true,EndpointSlice=true
kube-controller-manager: --feature-gates=ServiceTopology=true,EndpointSlice=true
kube-scheduler: --feature-gates=ServiceTopology=true,EndpointSlice=true
on all master server

kube-proxy: --feature-gates=EndpointSliceProxying=true,ServiceTopology=true,EndpointSlice=true on all server

But I still see ipvs rule:

UDP  10.106.3.98:53 rr
  -> 10.244.40.27:53              Masq    1      0          0
  -> 10.244.40.54:53              Masq    1      0          0
  -> 10.244.63.131:53             Masq    1      0          0
  -> 10.244.63.148:53             Masq    1      0          0
  -> 10.244.97.70:53              Masq    1      0          0
  -> 10.244.97.75:53              Masq    1      0          0
  -> 10.244.97.76:53              Masq    1      0          0
  -> 10.244.97.78:53              Masq    1      0          0
  -> 10.244.106.1:53              Masq    1      0          0

thanks

zorruch · May 18, 2020, 3:40pm

hmm … my proxy log:

docker logs c72778eaa26a
I0518 15:34:39.987860 1 node.go:136] Successfully retrieved node IP: хх.хх.хх.хх
I0518 15:34:39.987896 1 server_others.go:259] Using ipvs Proxier.
I0518 15:34:39.988118 1 server.go:583] Version: v1.18.2
I0518 15:34:39.988455 1 conntrack.go:100] Set sysctl ‘net/netfilter/nf_conntrack_max’ to 262144
I0518 15:34:39.988477 1 conntrack.go:52] Setting nf_conntrack_max to 262144
I0>518 15:34:39.988697 1 config.go:133] Starting endpoints config controller
I0518 15:34:39.988713 1 shared_informer.go:223] Waiting for caches to sync for endpoints config
I0518 15:34:39.988731 1 config.go:315] Starting service config controller
I0518 15:34:39.988735 1 shared_informer.go:223] Waiting for caches to sync for service config
I0518 15:34:40.288830 1 shared_informer.go:230] Caches are synced for service config
I0518 15:34:40.288831 1 shared_informer.go:230] Caches are synced for endpoints config

how can I understand that the kube-proxy really reads the configuration Endpoint Slices?
In the log I can not read it

robscott · May 18, 2020, 4:48pm

Hey @zorruch, that’s a great question! I’m not sure what log level you’re running kube-proxy with, but it might be worth temporarily increasing it to debug this. Can you try adding --v=4 as a kube-proxy flag here? What version of kube-proxy are you running, is it also 1.18? With more verbose logging, you should see one of the following in your log output to indicate whether Endpoints or EndpointSlices are being used:

Calling handler.OnEndpointsSynced()
Calling handler.OnEndpointSlicesSynced()

zorruch · May 22, 2020, 3:52pm

thanks for your reply
I use an image k8s.gcr.io/kube-proxy:v1.18.2
and run with parameters:

 containers:
 - command:
   - /usr/local/bin/kube-proxy
   - --config=/var/lib/kube-proxy/config.conf
   - --hostname-override=$(NODE_NAME)
   - --feature-gates=EndpointSliceProxying=true,ServiceTopology=true,EndpointSlice=true
  - --v=4
   env:
   - name: NODE_NAME
     valueFrom:
       fieldRef:
         apiVersion: v1
         fieldPath: spec.nodeName

In the log I see:

$ docker logs ec146b67c267 2>&1 | grep OnEndpointsSynced
I0522 15:42:39.826037 1 config.go:140] Calling handler.OnEndpointsSynced()
$ docker logs ec146b67c267 2>&1 | grep OnEndpointSlicesSynced
$

kube-proxy does not switch to use EndpointSlices
Am I running it with invalid parameters?

zorruch · June 26, 2020, 3:05pm

I write option feature gates option in a config kube-proxy:

featureGates:
   EndpointSlice: true
   EndpointSliceProxying: true
   ServiceTopology: true

everything is working

thanks all for reply

Topic		Replies	Views
Kubernetes offline installation, kube-dns has no endpoints. Need help to resolve the issue General Discussions development , network	3	2473	September 6, 2022
CoreDNS not working General Discussions	13	12952	April 3, 2023
Network Hairpin issue General Discussions	0	746	January 19, 2021
K8s on premise - coredns and clusterissuer problems General Discussions	0	738	March 16, 2023
Kubernetes Pod DNS Trick General Discussions	4	637	October 23, 2020

Service Topology and EndpointSlices complete usage

Cluster information:

Related Topics