I have a very simple scenario. I’ve got 2 different auto-scaled node pools: A and B and 2 different types of jobs: jA (can only use nodes A) and jB (can only use nodes B). Every job creates a single pod and uses a single node, there can’t be more than 1 pod running on a single node at the same time.
I want 2 strict FIFO queues to nodes of type A and nodes of type B. I thought that this was a default scheduling strategy, but recently I discovered it is not. There were 4 unscheduled pods and the last one created was scheduled before the first one.
I thought about using Kueue, but they require specifying a nominal quota. Is the solution to create 2 cluster queues qA and qB and specify the nominal quota of pods to max_number_of_nodes in the node pool? Are there simpler solutions?
Kubernetes version: 1.27.3-gke.1700
Cloud being used: GKE
CNI and version:
CRI and version:
You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.