k8s.io/test-infra@v0.0.0-20240520184403-27c6b4c223d8/config/jobs/kubernetes/sig-cloud-provider/gcp/gpu/README.md (about)

     1  GPU Job Scheduling
     2  ======
     3  
     4  A finite number of GPU projects is available (at time of writing, seventeen). If all the jobs
     5  are left to run on an interval without further intervention, we will exhaust the available projects
     6  and jobs will fail trying to get one.
     7  
     8  This file sets out an explicit schedule to minimise the number of concurrent jobs we've used.
     9  It's probably going to go stale, but is accurate at time of writing.
    10  
    11  Currently we only need to use six projects, leaving eleven for PR jobs and other exceptional events. 
    12  
    13  Artisanal
    14  ------
    15  
    16  ### 2 hours
    17  d) `ci-kubernetes-e2e-gce-device-plugin-gpu` (22m) (01:30)  
    18  e) `ci-kubernetes-e2e-gce-device-plugin-gpu-beta` (22m) (00:00)
    19  
    20  ### 3 hours
    21  g) `ci-cri-containerd-e2e-gce-device-plugin-gpu` (22m) (01:30)
    22  
    23  ### 4 hours
    24  h) `ci-kubernetes-e2e-gci-gke-autoscaling-gpu-k80` (1h33m) (00:00)  
    25  i) `ci-kubernetes-e2e-gci-gke-autoscaling-gpu-p100` (1h33m) (02:00)
    26  
    27  ### 6 hours
    28  k) `ci-kubernetes-e2e-gce-device-plugin-gpu-stable1` (23m) (03:00)  
    29  
    30  ### 12 hours
    31  s) `ci-kubernetes-e2e-gce-device-plugin-gpu-stable2` (25m) (08:00)  
    32  t) `ci-kubernetes-e2e-gce-gpu-stable2-stable1-cluster-upgrade` (40m) (04:00)  
    33  u) `ci-kubernetes-e2e-gce-gpu-stable2-stable1-master-upgrade` (30m) (05:00)  
    34  v) `ci-kubernetes-e2e-gce-gpu-stable1-beta-cluster-upgrade` (40m) (01:00)  
    35  w) `ci-kubernetes-e2e-gce-gpu-stable1-beta-master-upgrade` (30m) (07:00)  
    36  x) `ci-kubernetes-e2e-gce-gpu-stable1-master-cluster-upgrade` (40m) (03:00)  
    37  y) `ci-kubernetes-e2e-gce-gpu-stable1-master-master-upgrade` (30m - 50m on failure) (09:00)  
    38  z) `ci-kubernetes-e2e-gce-gpu-beta-stable1-cluster-downgrade` (40m) (10:00)  
    39  @) `ci-kubernetes-e2e-gce-gpu-master-stable1-cluster-downgrade` (40m) (11:00)
    40  
    41  Visualisation
    42  -----
    43  
    44  This sequence repeats in the afternoon.
    45  
    46  ```
    47  |00:00|00:30|01:00|01:30|02:00|02:30|03:00|03:30|04:00|04:30|05:00|05:30|06:00|06:30|07:00|07:30|08:00|08:30|09:00|09:30|10:00|10:30|11:00|11:30|
    48  |-----|-----|-----|--d--|--a--|--b--|--c--|--d--|--a--|--b--|--c--|--d--|-----|-----|-----|--d--|--a--|--b--|--c--|--d--|--a--|--b--|--c--|--d--|
    49  |--e--|-----|-----|--g--|-----|-----|--k--|-----|--e--|--g--|-----|-----|--e--|-----|-----|--g--|--e--|-----|--k--|-----|--e--|--g--|-----|-----|
    50  |-----------h-----------|-----------i-----------|-----------h-----------|-----------i-----------|-----------h-----------|-----------i-----------|
    51  |-----|-----|-----v-----|-----|-----|-----x-----|-----t-----|-----u-----|-----|-----|-----w-----|--s--|-----|-----y-----|-----z-----|-----@-----|
    52  ```