Hi,
I am trying to submit a k8 job to a cluster nodepool with autoscaling enabled.
Cloud being used: GWS
Script to generate the nodepool:
gcloud beta container node-pools create dws-32vcpu1-nodepool
–cluster=lir-ml-cluster
–zone=us-central1
–node-locations=us-central1-c
–num-nodes=0
–machine-type=e2-highcpu-32
–enable-autoscaling
–total-min-nodes=0
–total-max-nodes=1
–reservation-affinity=none
–enable-queued-provisioning
–scopes https://www.googleapis.com/auth/devstorage.read_write
–disk-size 100
Job.yaml:
apiVersion: batch/v1
kind: Job
metadata:
name: k8-test
labels:
kueue.x-k8s.io/queue-name: dws-local-queue
annotations:
provreq.kueue.x-k8s.io/maxRunDurationSeconds: “2000”
spec:
backoffLimit: 0
template:
metadata:
labels:
job-name: k8-test # Required label for the Pod template
kueue.x-k8s.io/queue-name: dws-local-queue
spec:
nodeSelector:
cloud.google.com/gke-nodepool: dws-32vcpu1-nodepool
containers:
resources:
requests:
cpu: “32”
memory: “32Gi”
restartPolicy: Never
The resultant pod produces the following errors:
Pod didn’t trigger scale-up: 1 max node group size reached
0/3 nodes are available: 3 node(s) didn’t match Pod’s node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
The three nodes I believe are from the deployment nodepool, where i do not want to run this job. As no node has yet been activated from the nodepool, the job doesnt run, but it also fails activate the new nodes from the specified nodepool as I expect it should due to the autoscheduler.
I’m lost for ideas on what to look for to resolve this issue, so any assistance is much appeciated.