sessionAffinity based on clientIP is not maintained for a new UDP request next to a TCP request

lets say, we have two independent Kubernetes clusters Cluster 1 & Cluster 2 , Each of them has two replicas of same application Pod. Like Cluster 1 : Pod A & Pod B and Cluster 2 : Pod C & Pod D . Application code in Pod A(client) wants to connect to any Pod running in cluster 2 via NodePort/Loadbalancer service over TCP protocol where the response would carry a UDP Port stating the client(Pod A) to initiate a new UDP connection on supplied UDP port for further communication.

To summarize the need:

1.Pod A → Pod C over TCP, and response carries a UDP port

2.Pod A → Pod C over UDP port captured in response

But here, while making UDP call, the request is going to other Pod i.e., Pod D. SessionAffinity set to clientIP is not helping. I want both TCP and UDP calls to fall on same pod.
Any idea ?