Kubernetes NodePort Service Performance Degradation

I have 2 small developmental clusters. Currently each of these clusters shows the same performance degradation of a service exposed via a NodePort.

The problem, in a nutshell, is that if we send/receive a stream of data to the service worker node and port then we receive the throughput that we expect: bi-directional 10,000 hundred byte messages / second.

However if we send the stream of data to a cluster node other than the worker node, and we rely on the kubernetes service mapping to then relay that stream of data to the worker node, the performance we see is significantly less (by at least a factor of 10).

Here are the nodes in one of the clusters

[dsargrad@malta ~]$ k get nodes
NAME                    STATUS   ROLES    AGE   VERSION
chile.corp.sensis.com   Ready    <none>   52d   v1.17.0
malta.corp.sensis.com   Ready    master   52d   v1.17.0

The service pod of interest (solace0-pubsubplus-0) is running on chile.

The NodePort service maps the internal port (55555) to the Node Port 30968.
[dsargrad@malta ~]$ k get svc
artemis NodePort 8161:31112/TCP,61616:32192/TCP 2d19h
cxf-proxy NodePort 8080:31234/TCP 28d
kubernetes ClusterIP 443/TCP 52d
nginx NodePort 80:32323/TCP 28d
solace0-pubsubplus NodePort 2222:32416/TCP,8080:31668/TCP,1943:30968/TCP,55555:31253/TCP,55003:30876/TCP,55443:31689/TCP,8008:32723/TCP,1443:30348/TCP,5672:32594/TCP,1883:30199/TCP,9000:30469/TCP 41h

In test configuration 1, we successfully send/receive 10,000msg/sec directly to chile.

In test configuration 2, though we try to send/receive exactly the same data stream (10,000 msg/sec)… the performance we see is less than a tenth of that

We are on a 1gigabit network. This is a closed network and a closed test fixture. There is little else happening on the three machines of interest.

Its not clear to me how to isolate this. Is this a problem with flannel? Is it a problem at the OS level (seems unlikely)?

Test Configuration

Cluster information:

Kubernetes version:
Cloud being used: (put bare-metal if not on a public cloud)
Installation method:
Host OS:
Centos 7
CNI and version:
CRI and version:

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.