I have 2 small developmental clusters. Currently each of these clusters shows the same performance degradation of a service exposed via a NodePort.
The problem, in a nutshell, is that if we send/receive a stream of data to the service worker node and port then we receive the throughput that we expect: bi-directional 10,000 hundred byte messages / second.
However if we send the stream of data to a cluster node other than the worker node, and we rely on the kubernetes service mapping to then relay that stream of data to the worker node, the performance we see is significantly less (by at least a factor of 10).
Here are the nodes in one of the clusters
[dsargrad@malta ~]$ k get nodes
NAME STATUS ROLES AGE VERSION
chile.corp.sensis.com Ready <none> 52d v1.17.0
malta.corp.sensis.com Ready master 52d v1.17.0
The service pod of interest (solace0-pubsubplus-0) is running on chile.
The NodePort service maps the internal port (55555) to the Node Port 30968.
[dsargrad@malta ~]$ k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
artemis NodePort 10.96.188.86 8161:31112/TCP,61616:32192/TCP 2d19h
cxf-proxy NodePort 10.96.134.209 8080:31234/TCP 28d
kubernetes ClusterIP 10.96.0.1 443/TCP 52d
nginx NodePort 10.96.188.234 80:32323/TCP 28d
solace0-pubsubplus NodePort 10.96.61.231 2222:32416/TCP,8080:31668/TCP,1943:30968/TCP,55555:31253/TCP,55003:30876/TCP,55443:31689/TCP,8008:32723/TCP,1443:30348/TCP,5672:32594/TCP,1883:30199/TCP,9000:30469/TCP 41h
In test configuration 1, we successfully send/receive 10,000msg/sec directly to chile.
In test configuration 2, though we try to send/receive exactly the same data stream (10,000 msg/sec)… the performance we see is less than a tenth of that
We are on a 1gigabit network. This is a closed network and a closed test fixture. There is little else happening on the three machines of interest.
Its not clear to me how to isolate this. Is this a problem with flannel? Is it a problem at the OS level (seems unlikely)?
Test Configuration
Cluster information:
Kubernetes version:
1.17
Cloud being used: (put bare-metal if not on a public cloud)
bare-metal
Installation method:
kubeadm
Host OS:
Centos 7
CNI and version:
flannel
CRI and version:
docker
You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.