sigs.k8s.io/kueue@v0.6.2/site/content/en/docs/tasks/run_plain_pods.md (about)

     1  ---
     2  title: "Run Plain Pods"
     3  date: 2023-09-27
     4  weight: 6
     5  description: >
     6    Run a single Pod, or a group of Pods as a Kueue-managed job.
     7  ---
     8  
     9  This page shows how to leverage Kueue's scheduling and resource management
    10  capabilities when running plain Pods. Kueue supports management of both
    11  [individual Pods](#running-a-single-pod-admitted-by-kueue), or
    12  [Pod groups](#running-a-group-of-pods-to-be-admitted-together).
    13  
    14  This guide is for [batch users](/docs/tasks#batch-user) that have a basic understanding of Kueue. For more information, see [Kueue's overview](/docs/overview).
    15  
    16  ## Before you begin
    17  
    18  1. By default, the integration for `v1/pod` is not enabled.
    19     Learn how to [install Kueue with a custom manager configuration](/docs/installation/#install-a-custom-configured-released-version)
    20     and enable the `pod` integration.
    21  
    22     A configuration for Kueue with enabled pod integration would look like follows:
    23     ```yaml
    24     apiVersion: config.kueue.x-k8s.io/v1beta1
    25     kind: Configuration
    26     integrations:
    27       frameworks:
    28        - "pod"
    29       podOptions:
    30         # You can change namespaceSelector to define in which 
    31         # namespaces kueue will manage the pods.
    32         namespaceSelector:
    33           matchExpressions:
    34           - key: kubernetes.io/metadata.name
    35             operator: NotIn
    36             values: [ kube-system, kueue-system ]
    37         # Kueue uses podSelector to manage pods with particular 
    38         # labels. The default podSelector will match all the pods. 
    39         podSelector:
    40           matchExpressions:
    41           - key: kueue-job
    42             operator: In
    43             values: [ "true", "True", "yes" ]
    44     ```
    45  
    46  2. Kueue will run webhooks for all created pods if the pod integration is enabled. The webhook namespaceSelector could be 
    47     used to filter the pods to reconcile. The default webhook namespaceSelector is:
    48     ```yaml
    49     matchExpressions:
    50     - key: kubernetes.io/metadata.name
    51       operator: NotIn
    52       values: [ kube-system, kueue-system ]
    53     ```
    54     
    55     When you [install Kueue via Helm](/docs/installation/#install-via-helm), the webhook namespace selector 
    56     will match the `integrations.podOptions.namespaceSelector` in the `values.yaml`.
    57  
    58     Make sure that namespaceSelector never matches the kueue namespace, otherwise the 
    59     Kueue deployment won't be able to create Pods.
    60  
    61  3. Pods that belong to other API resources managed by Kueue are excluded from being queued by `pod` integration. 
    62     For example, pods managed by `batch/v1.Job` won't be managed by `pod` integration.
    63  
    64  4. Check [Administer cluster quotas](/docs/tasks/administer_cluster_quotas) for details on the initial Kueue setup.
    65  
    66  ## Running a single Pod admitted by Kueue
    67  
    68  When running Pods on Kueue, take into consideration the following aspects:
    69  
    70  ### a. Queue selection
    71  
    72  The target [local queue](/docs/concepts/local_queue) should be specified in the `metadata.labels` section of the Pod configuration.
    73  
    74  ```yaml
    75  metadata:
    76    labels:
    77      kueue.x-k8s.io/queue-name: user-queue
    78  ```
    79  
    80  ### b. Configure the resource needs
    81  
    82  The resource needs of the workload can be configured in the `spec.containers`.
    83  
    84  ```yaml
    85      - resources:
    86          requests:
    87            cpu: 3
    88  ```
    89  
    90  ### c. The "managed" label
    91  
    92  Kueue will inject the `kueue.x-k8s.io/managed=true` label to indicate which pods are managed by it.
    93  
    94  ### d. Limitations
    95  
    96  - A Kueue managed Pod cannot be created in `kube-system` or `kueue-system` namespaces.
    97  - In case of [preemption](/docs/concepts/cluster_queue/#preemption), the Pod will
    98    be terminated and deleted.
    99  
   100  ## Example Pod
   101  
   102  Here is a sample Pod that just sleeps for a few seconds:
   103  
   104  {{< include "examples/pods-kueue/kueue-pod.yaml" "yaml" >}}
   105  
   106  You can create the Pod using the following command:
   107  ```sh
   108  # Create the pod
   109  kubectl apply -f kueue-pod.yaml
   110  ```
   111  
   112  ## Running a group of Pods to be admitted together
   113  
   114  In order to run a set of Pods as a single unit, called Pod group, add the
   115  "pod-group-name" label, and the "pod-group-total-count" annotation to all
   116  members of the group, consistently:
   117  
   118  ```yaml
   119  metadata:
   120    labels:
   121      kueue.x-k8s.io/pod-group-name: "group-name"
   122    annotations:
   123      kueue.x-k8s.io/pod-group-total-count: "2"
   124  ```
   125  
   126  ### Feature limitations
   127  
   128  Kueue provides only the minimal required functionality of running Pod groups,
   129  just for the need of environments where the Pods are managed by external
   130  controllers directly, without a Job-level CRD.
   131  
   132  As a consequence of this design decision, Kueue does not re-implement core
   133  functionalities that are available in the Kubernetes Job API, such as advanced retry
   134  policies. In particular, Kueue does not re-create failed Pods.
   135  
   136  This design choice impacts the scenario of
   137  [preemption](/docs/concepts/cluster_queue/#preemption).
   138  When a Kueue needs to preempt a workload that represents a Pod group, kueue sends
   139  delete requests for all of the Pods in the group. It is the responsibility of the
   140  user or controller that created the original Pods to create replacement Pods.
   141  
   142  **NOTE:** We recommend using the kubernetes Job API or similar CRDs such as
   143  JobSet, MPIJob, RayJob (see more [here](/docs/tasks/#batch-user)).
   144  
   145  ### Termination
   146  
   147  Kueue considers a Pod group as successful, and marks the associated Workload as
   148  finished, when the number of succeeded Pods equals the Pod group size.
   149  
   150  If a Pod group is not successful, there are two ways you may want to use to
   151  terminate execution of a Pod group to free the reserved resources:
   152  1. Issue a Delete request for the Workload object. Kueue will terminate all
   153     remaining Pods.
   154  2. Set the `kueue.x-k8s.io/retriable-in-group: false` annotation on at least
   155     one Pod in the group (can be a replacement Pod). Kueue will mark the workload
   156     as finished once all Pods are terminated.
   157  
   158  ### Example Pod group
   159  
   160  Here is a sample Pod group that just sleeps for a few seconds:
   161  
   162  {{< include "examples/pods-kueue/kueue-pod-group.yaml" "yaml" >}}
   163  
   164  You can create the Pod group using the following command:
   165  ```sh
   166  kubectl apply -f kueue-pod-group.yaml
   167  ```
   168  
   169  The name of the associated Workload created by Kueue equals the name of the Pod
   170  group. In this example it is `sample-group`, you can inspect the workload using:
   171  ```sh
   172  kubectl describe workload/sample-group
   173  ```