Multicast fails within same pod, succeeds in different pods

General Discussions
1

Cluster information:

Kubernetes version: v1.17.1
Cloud being used: bare-metal
Installation method: kubeadm
Host OS: Ubuntu 18.04 LTS Server
CNI and version: weave/flannel/kube-router/calico (latest releases of each)
CRI and version: docker/containerd (19.03.5/1.2.10)

Problem:

I am attempting to bring up a ROS 2 installation on Kubernetes, ideally using multiple containers in a single pod. Under the hood, ROS 2 relies upon DDS for communication, which is based upon UDP multicast.

When I bring up a simple pod deployment with two containers in a producer-consumer configuration, the consumer rarely (if ever) receives a message from the producer. When I bring up two pods, each with a single container the same producer-consumer configuration, the consumer always receives the messages.

Surprises

Every once in a while, the consumer will start up and receive messages as expected.

Furthermore, if one logs into the consumer with kubectl exec -it ros2-1 -c consumer /bin/bash then runs /ros_entrypoint.sh ros2 run demo_nodes_cpp listener, messages are sometimes received from the producer in the single pod scenario.

Expected Behavior

Successful messages appear in the logs of the consumer container as:

[INFO] [1579805884.017171859] [listener]: I heard: [Hello World: 1]
[INFO] [1579805885.017168023] [listener]: I heard: [Hello World: 2]
[INFO] [1579805886.017025092] [listener]: I heard: [Hello World: 3]

Actual Behavior

No such log messages are observed from the consumer.

Steps to Reproduce:

Failure within same pod

  1. Bring up a kubernetes cluster

  2. Apply the following pod definition: ros2-1.yaml

     apiVersion: v1
 kind: Pod
 metadata:
   name: ros2-1
 spec:
   containers:
   - name: producer
     image: osrf/ros2:nightly
     args: ["ros2", "run", "demo_nodes_cpp", "talker"]
   - name: consumer
     image: osrf/ros2:nightly
     args: ["ros2", "run", "demo_nodes_cpp", "listener"]
   restartPolicy: Never

  3. Watch for messages from the consumer with kubectl logs --follow ros2-1 consumer.

Success in different pods

  1. Bring up a kubernetes cluster

  2. Apply the following pod definition: ros2-2.yaml

     apiVersion: v1
 kind: Pod
 metadata:
   name: ros2-2-producer
 spec:
   containers:
   - name: producer
     image: osrf/ros2:nightly
     args: ["ros2", "run", "demo_nodes_cpp", "talker"]
   restartPolicy: Never
 ---
 apiVersion: v1
 kind: Pod
 metadata:
   name: ros2-2-consumer
 spec:
   containers:
   - name: consumer
     image: osrf/ros2:nightly
     args: ["ros2", "run", "demo_nodes_cpp", "listener"]
   restartPolicy: Never

  3. Watch for messages from the consumer with kubectl logs --follow ros2-2-consumer.

Questions:

  1. What is causing a single pod deployment to fail, but multi pod deployment to succeed?
  2. I am unfamiliar with debugging networking issues within the Kubernetes environment, while fairly experienced on bare-metal. How should I go about investigating this issue under flannel, weave, or kube-router?