sigs.k8s.io/kueue@v0.6.2/keps/1145-additional-labels/README.md (about)

     1  # KEP-1145: Additional labels, annotations, tolerations and selector
     2  
     3  <!-- toc -->
     4  - [Summary](#summary)
     5  - [Motivation](#motivation)
     6    - [Goals](#goals)
     7    - [Non-Goals](#non-goals)
     8  - [Proposal](#proposal)
     9    - [User Stories (Optional)](#user-stories-optional)
    10      - [Story 1](#story-1)
    11      - [Story 2](#story-2)
    12    - [Risks and Mitigations](#risks-and-mitigations)
    13  - [Design Details](#design-details)
    14    - [Test Plan](#test-plan)
    15        - [Prerequisite testing updates](#prerequisite-testing-updates)
    16      - [Unit Tests](#unit-tests)
    17      - [Integration tests](#integration-tests)
    18    - [Graduation Criteria](#graduation-criteria)
    19  - [Implementation History](#implementation-history)
    20  - [Drawbacks](#drawbacks)
    21  - [Alternatives](#alternatives)
    22  <!-- /toc -->
    23  
    24  ## Summary
    25  
    26  Allow (AdmissionCheck)[https://github.com/kubernetes-sigs/kueue/tree/main/keps/993-two-phase-admission] 
    27  to set some additional location/admission information
    28  on workload's pods, so that they land on the machines of the `AdmissionCheck`
    29  choosing.
    30  
    31  ## Motivation
    32  
    33  `AdmissionChecks` is a "plugin" mechanism for Kueue. Some of the plugins while 
    34  admitting may either narrow down on which machines the workload should start (by adding
    35  some extra node selector/tolerations) or add some tracking information
    36  that may be needed by other system components outside of Kueue.
    37  
    38  ### Goals
    39  
    40  * Establish a mechanism through which `AdmissionChecks` can add annotations/labels/node selector/tolerations
    41    to workload pods.
    42  
    43  ### Non-Goals
    44  
    45  * Allow `AdmissionChecks` to change pod definition in other way, for instance change command or image name.
    46  * Hard code any specific labels/annotations that are passed down.
    47  
    48  ## Proposal
    49  
    50  * Add a struct in `Workload's` `AdmissionCheckStatus` that allows 
    51    to set annotations, labels, selectors and tolerations on PodSets.
    52  * Expend PodSetInfo in Integration Framework to pass this information.
    53  * Modify existing integrations to support setting more data on the pods.
    54  
    55  ### User Stories (Optional)
    56  
    57  #### Story 1
    58  
    59  My external controller managing an `AdmissionCheck` preallocates specific
    60  nodes (tainted and labeled) for a Workload to run, before it is 
    61  admitted (and creates any worker pods). Workload pods need to be modified
    62  so that they end up on correct nodes.
    63  
    64  #### Story 2
    65  
    66  My `AdmissionCheck` does budget checking and would like to mark pods that are possibly crossing budget with a 
    67  specific label for easier tracking and alerting.
    68  
    69  ### Risks and Mitigations
    70  
    71  Some integrations may not support setting all of the desired information
    72  into the pods due to their CRD limitations.
    73  
    74  ## Design Details
    75  
    76  Modify WorkloadStatus API object:
    77  
    78  ```
    79  // WorkloadStatus defines the observed state of Workload
    80  type WorkloadStatus struct {
    81      [...]
    82      AdmissionChecks []metav1.AdmissionCheckState `json:"admissionChecks,omitempty" patchStrategy:"merge" patchMergeKey:"type"`
    83  }
    84  
    85  type AdmissionCheckState struct {
    86      // name identifies the admission check.
    87      // +required
    88      // +kubebuilder:validation:Required
    89      // +kubebuilder:validation:MaxLength=316
    90      Name string `json:"name"`
    91  
    92      [...]
    93  
    94      // +optional
    95      // +listType=atomic
    96      PodSetUpdates []PodSetUpdate `json:"podSetUpdates,omitempty"`
    97  }
    98  
    99  // PodSetUpdate contains a list of pod set modifications suggested by AdmissionChecks.
   100  // The modifications should be additive only - modifications of already existing keys
   101  // are not allowed and will result in failure during workload admission.
   102  type PodSetUpdate struct {
   103      // Name of the PodSet to modify. Should match to one of the Workload's PodSets.
   104      Name string `json:"name"`
   105  
   106      // +optional
   107      Labels map[string]string `json:"labels,omitempty"
   108  
   109      // +optional
   110      Annotations map[string]string `json:"annotations,omitempty" 
   111  
   112      // +optional
   113      NodeSelector map[string]string `json:"nodeSelector,omitempty"
   114  
   115      // +optional
   116      Tolerations []corev1.Toleration `json:"tolerations,omitempty"
   117  }
   118  ```
   119  
   120  Each of the `AdmissionChecks` controllers should populate the needed fields before 
   121  (or at the same time) flipping the corresponding `AdmissionCheck` to `True`. The values
   122  from `PodSetUpdates` are combined with `ResourceFlavors` into integration framework
   123  `PodSetInfo` (that will get the missing labels, tolerations and annotations fields) 
   124  and then passed to the framework controller for application on the CRD.
   125  
   126  ### Test Plan
   127  
   128  [x] I/we understand the owners of the involved components may require updates to
   129  existing tests to make this code solid enough prior to committing the changes necessary
   130  to implement this enhancement.
   131  
   132  ##### Prerequisite testing updates
   133  
   134  None.
   135  
   136  #### Unit Tests
   137  
   138  Existing unit test should be updated to tests whether the new data are correctly
   139  passed and applied on CRD. Changes may be needed in all framework integrations.
   140  
   141  #### Integration tests
   142  
   143  Existing integration test should be updated to tests whether the new data are correctly
   144  passed and applied on CRD. Changes may be needed in all framework integrations.
   145  
   146  ### Graduation Criteria
   147  
   148  We will graduate this feature to stable together with the whole AdmissionCheck/two
   149  phase admission process.
   150  
   151  ## Implementation History
   152  
   153  2023-09-22 KEP 
   154  
   155  ## Drawbacks
   156  
   157  * `Workload` status becomes even more loaded. 
   158  * Additional requirements for framework integrations.
   159  
   160  ## Alternatives
   161  
   162  * `AdmissionCheck` could try to modify Workload's Spec, but then the spec would be reconciled
   163    with the Job and the changes would be discarded and Workload recreated with the matching Spec.
   164    
   165  * `AdmissionCheck` could try to catch Job's pods after they are created and modify them on the fly
   166    via WebHook. This would however put an additional requirements on the `AdmissionCheck` to understand
   167    how framework integrations work and what Pods are created by them.