volcano.sh/volcano@v1.9.0/docs/design/tdm-plugin.md (about)

     1  # Time Division Multiplexing (TDM) plugin
     2  
     3  ## Introduction
     4  
     5  In a co-located environment, some nodes are in both Kubernetes cluster and Yarn cluster. For these nodes, Kubernetes and Yarn cluster can use these resource by time-sharing multiplexing.
     6  For example, in 8:00~24:00 the on-line service in kubernetes can use these nodes. And in 0:00~8:00, kubernetes need evict the workload on these nodes, left the resource for Yarn cluster.
     7  
     8  ## Solution
     9  
    10  First, we need mark out these nodes(by add `volcano.sh/revocable-zone` for nodes) which are in both Kubernetes cluster and Yarn cluster. We call these node are `revocable node`. For the kubernetes 
    11  workload, we add `volcano.sh/preemptable: "true"` annotation for pod/podgroup to distinguish which pod can be dispatched to the `revocable node`. We call these pod are `preemptable task`.
    12  TDM plugin enhance the volcano time-sharing multiplexing resource ability. It will try to dispatch `preemptable task` to `revocable node` in node revocable time and evict the `preemptable task` from 
    13  `revocable node` out of revocable time.
    14  
    15  ![tdmsolution](./images/tdmsolution.png)
    16  
    17  1. Add `volcano.sh/preemptable` annotaion for Pod/PodGroup. For volcano job when add this annotaion in job level, Pod/PodGroup will inherit this annotation. The pod with `volcano.sh/preemptable: "true"` annotation can be dispatched to `revocable node`.
    18  Otherwise, the pod can not be dispatched to `revocable node`.
    19  2. Add `tdm` plugin and config for volcano scheduler. `tdm.revocable-zone` is a const prefix(), `rz1` is the revocable zone name and the value is a time frame. 
    20  
    21  ```
    22    tiers:
    23      - plugins:
    24        - name: tdm
    25          arguments:
    26            tdm.revocable-zone.rz1: 1:00-4:00
    27            tdm.evict.period: 1m
    28  
    29  ```
    30  3. Add `volcano.sh/revocable-zone: rz1` label for nodes. Nodes which have this label is a `revocable node`. The value indicates in which time period, the node can be used by kubernetes (run pod).
    31  4. Add `VictimTasksFn` for scheduler framework, it returns all the tasks which will be evicted in the scheduler period.
    32  5. TDM plugin:
    33  - PredicateFn: For the `non-preemptable task`, they can not use `revocable nodes`; For the `preemptable task`, they can use `revocable node` in revocable time period, otherwise they also can not use the `revocable node`.
    34  - NodeOrderFn: For the `preemptable task`, they preferred to use `revocable nodes` in revocable time period.
    35  - VictimTasksFn: In 1VictimTasksFn1, the plugin can decide which tasks/pods should be evicted, e.g. based on Job/PodGroup info.
    36  - PreemptableFn: When there is `non-preemptable task` pending, select appropriate `preemptable task` on `non-revocable nodes` to evict.
    37  
    38  ## Feature interaction
    39  
    40  - Preempt action: This TDM plugin need work with preempt action. If does not config preempt action, TDM plugin will miss PreemptableFn.