volcano.sh/volcano@v1.9.0/docs/design/dedicated-volume.md (about)

     1  ---
     2  title: Dedicated Volumes
     3  authors:
     4    - "@hzxuzhonghu"
     5  reviewers:
     6    - "@k82cn"
     7    - "@zrss"
     8  approvers:
     9    - "@k82cn"
    10  creation-date: 2019-11-21
    11  last-updated: 2019-11-21
    12  status: review
    13  ---
    14  
    15  # Dedicated Volumes
    16  
    17  ## Summary
    18  
    19  The purpose of this is to allow mount dedicated volumes per pod of a Volcano Job.
    20  
    21  ## Motivation
    22  
    23  Volume mount was supported from begin. But there are few limitations:
    24  
    25  1. We can specify volumes of Job scope, all the volumes are shared by all pods within a Job.
    26  
    27  2. We specify volumes by setting `TaskSpec.PodTemplateSpec.Volumes`, but similarly they are shared by pods within a task.
    28  
    29  But in real world, scenarios like DL/BigData, etc requires high performance. Shared storage has some performance issue,
    30  like io limit, read/write conflicts.
    31  
    32  Also some cloud vendors do not support volumes mounting to multiple nodes, it is to prevent data inconsistent.
    33  
    34  Given the above issues, we should support pods within volcano mounting volumes exclusively.
    35  
    36  ## Goals
    37  
    38  Job TaskSpec will contain an additional structure called `VolumeClaimTemplates` to control exclusive volumes.
    39  
    40  ## Proposal
    41  
    42  ### Implementation Details
    43  
    44  #### API Changes
    45  
    46  Following changes will be made to the TaskSpec and VolumeSpec for volcano Job.
    47  
    48  ```go
    49  // TaskSpec specifies the task specification of Job
    50  type TaskSpec struct {
    51      ...
    52  
    53  	// The volumes mount on pods of the Task
    54      // Depends on the `VolumeSpec.GenerateName`, they can be dedicated or shared.
    55      // If `GenerateName` is specified and `VolumeClaimName` is not, the Job controller will generate a dedicated PVC for each pod.
    56  	Volumes []VolumeSpec `json:"volumes,omitempty" protobuf:"bytes,3,opt,name=volumes"`
    57      ...
    58  }
    59  ```
    60  
    61  ```go
    62  // VolumeSpec defines the specification of Volume, e.g. PVC
    63  type VolumeSpec struct {
    64  	// Path within the container at which the volume should be mounted.  Must
    65  	// not contain ':'.
    66  	MountPath string `json:"mountPath" protobuf:"bytes,1,opt,name=mountPath"`
    67  
    68  	// defined the PVC name
    69  	VolumeClaimName string `json:"volumeClaimName,omitempty" protobuf:"bytes,2,opt,name=volumeClaimName"`
    70  
    71      // If `VolumeClaimName` is empty, then the job controller will generate a name with `{task_index}` suffixed for each task instance.
    72      // Note: it can be set for task scoped only.
    73  	GenerateName string `json:"generateName,omitempty" protobuf:"bytes,4,opt,name=generateName"`
    74  
    75  	// VolumeClaim defines the PVC used by the VolumeMount.
    76  	VolumeClaim *v1.PersistentVolumeClaimSpec `json:"volumeClaim,omitempty" protobuf:"bytes,3,opt,name=volumeClaim"`
    77  }
    78  ```
    79  
    80  - By default, this is empty. The task instance will use volumes defined in `JobSpec.Volumes` and `TaskSpec.Template`.
    81  
    82  - If `Volumes` are specified, these pvcs are referenced by all the pods of the task.
    83    If the VolumeSpec specifies the `GenerateName` while the `VolumeClaimName` left empty,  the pvc name is generated with task index suffixed by job controller.
    84    Otherwise, the explicitly declared pvc will be shared by all pods of a task.
    85  
    86  - If the pvcs does not exist, job controller will create them.
    87  
    88  
    89  #### Implementation
    90  
    91  Consider the following demo job:
    92  
    93  ```yaml
    94  apiVersion: batch.volcano.sh/v1alpha1
    95  kind: Job
    96  metadata:
    97    name: demo
    98  spec:
    99    tasks:
   100    - name: task
   101      replicas: 2
   102      template:
   103        spec:
   104          containers:
   105          - name: ps
   106            image: ps-img
   107      volumes:
   108      - mountPath: /data
   109        generateName: train-pvc-
   110        volumeClaim:
   111          accessModes:
   112            - ReadWriteOnce
   113          resources:
   114            requests:
   115              storage: 8Gi
   116  ```
   117  Job controller creates persistent volume claim from `TaskSpec.Volumes`, the total number of pvcs built here is equal to `TaskSpec.Replicas` * `len(TaskSpec.Volumes)`.
   118  
   119  As is known the two pods created by job controller is: `demo-task-0` and `demo-task-1`. The name is combined with job name, task name and pod ordinal.
   120  For the above example, it will create 2 persistent volume claims, one for each pod.
   121  We should make use of the pod ordinal here to generate the pvc name as well, with the format: `<TaskSpec.Volumes[i].GenerateName><task_index>`.
   122  For the above example, the pvc created for pod `demo-task-0` is `train-pvc-0`, and the pvc for `demo-task-1` is `train-pvc-1`.
   123