NodePort, persistent connections, separate instances

General Discussions
1

I am running a multi-node cluster of UDP game servers that need to be accessible to the player on a single domain name and port, i.e. example.com:19132. The game runs in a way that requires a series of UDP packets to be sent consistently to the same server after the initial connection.

The problem is, I want to have multiple and separate instances of the game. If I use a NodePort service, I’m afraid individual packets will get routed to different instances mid-session and ruin the connection.

Is there a way to do that without limiting the amount of scaling I can do? Am I limited to only running one instance per IP or is there a better way of balancing load consistently?

2
3

The problem is, I’m making a server for an existing game, so I have no control over the game itself, and the game only allows adding servers on a single IP and port as an entry point.

Looks like Agones does a fantastic job of managing game servers with an allocator but the problem I’m facing is more about load balancing due to the port restriction. I unfortunately cannot use the allocator as there has to be a single entry point for the player.

4

Just to be clear - what does that external IP resolve to? A load balancer or proxy or something?

Setting the service session affinity SHOULD do what you want as long as the first-contact LB respects it.

5

Well the architecture is Client <-> L7 proxy <-> Game server. I already have some load balancing through the L7 proxy to the downstream game servers, and I need to expose multiple proxy instances to also balance out the players between the proxies.

What exactly do you mean by service session affinity? Is that an agones feature or the Kubernetes ipvs source hashing load balancing method?

6

What do you do for L4 load balancing to those L7’s ?

Kube service API has a field to enable explicit session affinity. The first level (L4) load balancer should always send traffic from a given client to the same back end. Unless that back end is not available. Then that back end (the L7) can do whatever it needs to with the traffic.

That assumes your L4 is smart enough to not spray packets. If not, I amnot sure I see how to do it.

7

What do you do for L4 load balancing to those L7’s ?

That was exactly what I was trying to ask, haha, what should I be doing?

Kube service API has a field to enable explicit session affinity. The first level (L4) load balancer should always send traffic from a given client to the same back end.

Does that mean I can just setup a NodePort service of my L7s using the session affinity setting, and that should be enough to have the same client reach the same backend?

That assumes your L4 is smart enough to not spray packets.

Is there a way to get a smarter L4 than just however k8s handles it?

8

Well, that depends on where you are running and what infrastructure you have
available. This doesn’t seem to be a Kubernetes-specific issue - how would you
normally expose a VIP to these clients which is consistently routed to N
backends?

If you are on a cloud, they all have L4 LBs that do this.

If you are not, you need something to do that. You can set up F5 or IPVS or
something - that’s outside the domain of k8s.

Think about how the traffic and routing decisions are made. I don’t know
enough about your situation to fill in the blanks, but:

Client sends UDP traffic to an IP. How did they find that IP? How do they
know which port?

That IP is routed to a pod (your L7). If there are N > 1 such pods, who
decides which one gets used by this client, and how do they steer it there?

Once it is at your L7, you can make app-aware decisions and keep state, such
that responses and subsequent packets are handled.

Subsequent packets from the client are sent to that same IP. How do you ensure
they go the same L7 (which has state!) as before? Or how do you do
state-sharing between L7s?

There are too many decisions here to answer in the abstract. As an example,
here’s how it works (if I get your arch correctly) on Google Cloud:

You deploy your L7s as pods.

You create a Service with type=LoadBalancer (and session affinity on) for those
L7s, that gives you a public IP.

You put that IP into a DNS record somewhere.

Your clients send UDP to that IP.

Our L4 infrastructure (which only understands VMs) hashes on client IP and
sends the UDP traffic to a specific VM.

That VM hashes on client IP and service IP and sends the UDP traffic to a
specific L7 pod (and remembers that decision).

That’s your pod, you do whatever you need, and if there is a response it comes
from that pod.

The VM reverses the connection state and replies directly to your client (which
is blissfully unaware of any of this :slight_smile:

Subsequent packets are sent to the same VM, which has connection state and
forwards to the same pod, which may have game state.

That all assumes you have enough info IN YOUR OWN PACKET to do the L7 routing.
If you need unique ports for each game, its a different story.

k8s doesn’t “handle it”. We have an API for “I need an external LB, someone
please make that happen”, and we have a handful of cloud-provider modules which
implement that request. If you are not in a cloud, you need to decide who is
answering that request and how.

When I mentioned the affinity API, what I meant is that whomever implements the
external LB has to support it, or it won’t work.