volcano.sh/volcano@v1.9.0/docs/design/node-group.md (about)

     1  ## Introduction
     2  
     3  nodegroup is the nodes with the same specified label, on the basis of nodegroup, different queues and nodegroups have some relationship about affinity and anti-affinity.
     4  
     5  case1: different departments (like nlp and recommend) use different nodes (nodegroup1 and nodegroup2), nlp queue uses nodegroup1 first, and nodegroup2 can also be used when resources are insufficient, but recommend queue can only use nodegroup2.
     6  
     7  ![](images/node-group-case-1.png)
     8  
     9  case2: recommend queue can use private cloud nodes or public cloud nodes, but tts queue can only use private cloud nodes because its task takes a long time, which is not easy to the release of nodes in public cloud (for saving money)
    10  
    11  ![](images/node-group-case-2.png)
    12  
    13  ## Solution
    14  
    15  1. First, we need mark out some nodes(add `volcano.sh/nodegroup-name` labels) which are the same group.
    16  2. Second, we need to express the relationship between queue and nodegroup, such as affinity and anti-affinity, so I add the `queue.spec.affinity` field in Queue.
    17     ```yaml
    18     apiVersion: scheduling.volcano.sh/v1beta1
    19     kind: Queue
    20     metadata:
    21       name: default
    22       spec:
    23         reclaimable: true
    24         weight: 1
    25         affinity:            # added field
    26           nodeGroupAffinity:
    27             requiredDuringSchedulingIgnoredDuringExecution:
    28             - groupname1
    29             - gropuname2
    30             preferredDuringSchedulingIgnoredDuringExecution:
    31             - groupname1
    32           nodeGroupAntiAffinity:
    33             requiredDuringSchedulingIgnoredDuringExecution:
    34             - groupname3
    35             - gropuname4
    36             preferredDuringSchedulingIgnoredDuringExecution:
    37             - groupname3
    38     ```
    39  
    40  affinity configure:
    41  1. affinity.nodeGroupAffinity.requiredDuringSchedulingIgnoredDuringExecution, hard constraints, such as `nlp = nodegroup1,nodegroup2`, it means that task in queue=nlp can ony run on the nodes in nodegroup1 or nodegroup2.
    42  2. affinity.nodeGroupAffinity.preferredDuringSchedulingIgnoredDuringExecution, soft constraints, such as `nlp = nodegroup1`, it means that task in queue=nlp runs on nodegroup1 first, but if the resources of nodegroup1 is insufficient, it can also run on other nodegroups. Combine rule1 and rule2, task in queue=nlp runs on nodegroup1 first,  but if the resources of nodegroup1 is insufficient, it can also run on nodegroup2.
    43  3. affinity.nodeGroupAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution, hard constraints, such as `nlp = nodegroup1`, it means that task in queue=nlp can run on any nodegroups but nodegroup1.
    44  4. affinity.nodeGroupAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution, soft constraints, such as `nlp = nodegroup1`, it means that task in queue=nlp runs on any other nodegroups, but if the resources of other nodegroup is insufficient, it can also run on nodegroup1.
    45  
    46  we also need to enable nodegroup plugin.
    47  
    48  ```yaml
    49  actions: "reclaim, allocate, backfill, preempt"
    50  tiers:
    51  - plugins:
    52    - name: priority
    53    - name: gang
    54    - name: conformance
    55  - plugins:
    56    - name: drf
    57    - name: predicates
    58    - name: proportion
    59    - name: nodegroup
    60      enablePredicate: true
    61      enableNodeOrder: true
    62  ```
    63  
    64  risk: The resources of the queue can not be too different from the resources of the nodegroup(such as queue.capability.memory = 1024G,but all the memory of binding nodegroup is 512G), otherwise it may cause that task can be scheduled to run from the queue's point of view, but cannot find a suitable node.
    65  
    66  ## Implement
    67  
    68  Hard constraints are implemented by using PredicateFn, and soft constraints are implemented by using NodeOrderFn.
    69  
    70  predict flow chart:
    71  
    72  ![](images/nodegroup_plugin-predict-1.png)
    73  
    74  score flow chart:
    75  
    76  ![](images/nodegroup_plugin-score-1.png)