Need help with Taints and Tolerances + Node Affinity

General Discussions
1

Cluster information:

Kubernetes version:
v1.20.5
Cloud being used: bare-metal
Installation method: cluster managed by Helm + Rancher
Host OS: Centos 7
CNI and version: unknown
CRI and version: unknown

I have a cluster of 7 nodes with 1 etcd, 1 ctrl plane, and 5 workers. 4 of these worker nodes are dedicated for a document ingesting tool, and the last one is a general use node.

The taints and labels are laid out as such

kwork01 has no taint or label (general use node)

(dedicated node) ingest01 has the label of “role=ingest” and the taint of “ingestProtect=ingest:NoSchedule”
(dedicated node) ingest02 has the label of “role=ingest” and the taint of “ingestProtect=ingest:NoSchedule”
(dedicated node) ingest03 has the label of “role=ingest” and the taint of “ingestProtect=ingest:NoSchedule”
(dedicated node) ingest04 has the label of “role=ingest” and the taint of “ingestProtect=ingest:NoSchedule”

For what I am trying to do, I am pretty sure I need to use a combination of Node Affinity and Taint/Tolerations, so I hammered this into my helmchart:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: role
                operator: In
                values:
                  - ingest

  tolerations:
    - key: "ingestProtect"
      operator: "Exist"

With this combination, it just dumps all of my replicas onto kwork01.

I’ve tried the following as well and I get the same result unless I taint kwork01, then I just get failures on deployment of the tool altogether, citing that the pod can’t tolerate the node taints, cluster wide

  tolerations:
  - key: "ingestProtect"
    operator: "Equal"
    value: "ingest"
    effect: "NoSchedule"

I tried setting the Effect in both the taint and in the yaml to “PreferNoSchedule” - which “works” and I can deploy to those nodes, but I must taint kwork01 or it gets targeted.

My desired end results is to have the ingesting tool only ever get deployed on it’s dedicated nodes without other pods ending up deployed on those same nodes. Additionally I don’t want to taint the kwork01 general use node in order to keep it accessible for other usage without having to add a bunch of info to all of the helm charts that already exist.

Any input would be great. What am I doing wrong in my YAML or on my node’s taint config?

2

I’m not sure to that using noeeAffinity is necessary.

You can use Taint as you’re doing so general pods (without any taint) will run on generic node, and you can use Taint + nodeSelector for the specific pods that should run on the specific nodes.

For nodeSelector, you will need to add a label to your 4 nodes, and use this label in the nodeSelector of the Pods you want to run on these specific nodes

3

I wouldn’t use taints to deploy apps on specific nodes but rathar labels with nodeaffinity and node antiafrinity.