How to manually configure Topology-aware traffic routing

General Discussions
1

Yes I know about Topology Aware Hints, but to put it simply, this feature just doesn’t work.
So, how can I (if at all) configure a service with topology aware routing?

My use case is rather simple, I have 3 redis replicas, each pod in a different node, each node in a different AZ.
I want any requests to the redis service to be routed in the same AZ as the pods who made the request, very simple.
I don’t care about edge cases like if 1 AZ doesn’t have redis running, I’ll take care of that.

So, is this even possible to configure manual?

2

This is an on-going discussion in sig-network. Hints assumes a larger scale. To run across a small set of endpoints requires different tuning.

3

This is a cool feature. Guessing it depends on the CNI to support it?

You could just build your workload to have the zone in it’s pod labels, use service selectors for each zone, and make deployments zonal. Getting a free meal out of k8s doesn’t seem like it’ll be easy persay.

4

Here’s a visual of what I would do. Can’t say I love it, but it’s valid. (Edit, I realize I use two different regions instead of AZs here. You’ll have to forgive me, I’m not fixing the typo)

    ┌───────────────────────┐   ┌───────────────────────┐
    │ Name: redis-us-west-1 │   │ Name: redis-us-west-2 │
    ├───────────────────────┤   ├───────────────────────┤
    │ Kind: StatefulSet     │   │ Kind: StatefulSet     │
    │ Replicas: 1           │   │ Replicas: 1           │
    │ Pod Labels:           │   │ Pod Labels:           │
    │   App: redis          │   │   App: redis          │
    │   Zone: us-west-1     │   │   Zone: us-west-2     │
    └──────────▲────────────┘   └──────────▲────────────┘
               │                           │
               │                           │
               │                           │
               │                           │
               │                           │
    ┌──────────┴────────────┐   ┌──────────┴────────────┐
    │ Name: redis-us-west-1 │   │ Name: redis-us-west-2 │
    ├───────────────────────┤   ├───────────────────────┤
    │ Kind: Service         │   │ Kind: Service         │
    │ Selectors:            │   │ Selectors:            │
    │   App: redis          │   │   App: redis          │
    │   Zone: us-west-1     │   │   Zone: us-west-2     │
    └──────────▲────────────┘   └──────────▲────────────┘
               │                           │
               │                           │
               │                           │
   ┌───────────┴────────────┐   ┌──────────┴─────────────┐
   │ Name: app-us-west-1    │   │ Name: app-us-west-2    │
   ├────────────────────────┤   ├────────────────────────┤
   │ Kind: Deployment       │   │ Kind: Deployment       │
   │ Pod Env:               │   │ Pod Env:               │
   │   Redis Host: >        │   │   Redis Host: >        │
   │     redis-us-west-1    │   │     redis-us-west-2    │
┌──┤►Replicas: HPA Managed  │ ┌─┤►Replicas: HPA Managed  │
│  └────────────────────────┘ │ └────────────────────────┘
│                             │
│  ┌────────────────────────┐ │ ┌────────────────────────┐
└──┤Name: app-us-west-1     │ └─┤Name: app-us-west-2     │
   ├────────────────────────┤   ├────────────────────────┤
   │Kind: HPA               │   │Kind: HPA               │
   │This should scale up    │   │This should scale up    │
   │replicas in the zone    │   │replicas in the zone    │
   │desired, when needed    │   │desired, when needed    │
   │and scale to 0 when     │   │and scale to 0 when     │
   │not needed              │   │not needed              │
   └────────────────────────┘   └────────────────────────┘
5

Wrote a controller to plug node labels onto pods. It might be useful in making some sorta generic workaround too. Idk, feel free to let me know it’s terrible, lol.

6

This is more of a scheduling solution, not network routing.
And having to configure each of my deployments with different config per AZ isn’t maintanable.