sigs.k8s.io/kueue@v0.6.2/site/content/en/docs/tasks/administer_cluster_quotas.md (about)

     1  ---
     2  title: "Administer Cluster Quotas"
     3  date: 2022-03-14
     4  weight: 3
     5  description: >
     6    Manage your cluster resource quotas and to establish fair sharing rules among the tenants.
     7  ---
     8  
     9  This page shows you how to manage your cluster resource quotas and to establish
    10  fair sharing rules among the tenants.
    11  
    12  The intended audience for this page are [batch administrators](/docs/tasks#batch-administrator).
    13  
    14  ## Before you begin
    15  
    16  Make sure the following conditions are met:
    17  
    18  - A Kubernetes cluster is running.
    19  - The kubectl command-line tool has communication with your cluster.
    20  - [Kueue is installed](/docs/installation).
    21  
    22  ## Single ClusterQueue and single ResourceFlavor setup
    23  
    24  In the following steps, you will create a queuing system with a single
    25  ClusterQueue and a single [ResourceFlavor](/docs/concepts/cluster_queue#resourceflavor-object)
    26  to govern the quota of your cluster.
    27  
    28  You can perform all these steps at once by
    29  applying [examples/admin/single-clusterqueue-setup.yaml](/examples/admin/single-clusterqueue-setup.yaml):
    30  
    31  ```shell
    32  kubectl apply -f examples/admin/single-clusterqueue-setup.yaml
    33  ```
    34  
    35  ### 1. Create a [ClusterQueue](/docs/concepts/cluster_queue)
    36  
    37  Create a single ClusterQueue to represent the resource quotas for your entire
    38  cluster.
    39  
    40  Write the manifest for the ClusterQueue. It should look similar to the following:
    41  
    42  ```yaml
    43  # cluster-queue.yaml
    44  apiVersion: kueue.x-k8s.io/v1beta1
    45  kind: ClusterQueue
    46  metadata:
    47    name: "cluster-queue"
    48  spec:
    49    namespaceSelector: {} # match all.
    50    resourceGroups:
    51    - coveredResources: ["cpu", "memory"]
    52      flavors:
    53      - name: "default-flavor"
    54        resources:
    55        - name: "cpu"
    56          nominalQuota: 9
    57        - name: "memory"
    58          nominalQuota: 36Gi
    59  ```
    60  
    61  To create the ClusterQueue, run the following command:
    62  
    63  ```shell
    64  kubectl apply -f cluster-queue.yaml
    65  ```
    66  
    67  This ClusterQueue governs the usage of [resource types](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-types)
    68  `cpu` and `memory`. Each resource type has a single [resource flavor](/docs/concepts/cluster_queue#resourceflavor-object),
    69  named `default` with a nominal quota.
    70  
    71  The empty `namespaceSelector` allows any namespace to use these resources.
    72  
    73  ### 2. Create a [ResourceFlavor](/docs/concepts/cluster_queue#resourceflavor-object)
    74  
    75  The ClusterQueue is not ready to be used yet, as the `default` flavor is not
    76  defined.
    77  
    78  Typically, a resource flavor has node labels and/or taints to scope which nodes
    79  can provide it. However, since we are using a single flavor to represent all the
    80  resources available in the cluster, you can create an empty ResourceFlavor.
    81  
    82  Write the manifest for the ResourceFlavor. It should look similar to the following:
    83  
    84  ```yaml
    85  # default-flavor.yaml
    86  apiVersion: kueue.x-k8s.io/v1beta1
    87  kind: ResourceFlavor
    88  metadata:
    89    name: "default-flavor"
    90  ```
    91  
    92  To create the ResourceFlavor, run the following command:
    93  
    94  ```shell
    95  kubectl apply -f default-flavor.yaml
    96  ```
    97  
    98  The `.metadata.name` matches the `.spec.resourceGroups[0].flavors[0].name`
    99  field in the ClusterQueue.
   100  
   101  ### 3. Create [LocalQueues](/docs/concepts/local_queue)
   102  
   103  Users cannot directly send [workloads](/docs/concepts/workload) to
   104  ClusterQueues. Instead, users need to send their workloads to a Queue in their
   105  namespace.
   106  Thus, for the queuing system to be complete, you need to create a Queue in
   107  each namespace that needs access to the ClusterQueue.
   108  
   109  Write the manifest for the LocalQueue. It should look similar to the following:
   110  
   111  ```yaml
   112  # default-user-queue.yaml
   113  apiVersion: kueue.x-k8s.io/v1beta1
   114  kind: LocalQueue
   115  metadata:
   116    namespace: "default"
   117    name: "user-queue"
   118  spec:
   119    clusterQueue: "cluster-queue"
   120  ```
   121  
   122  To create the LocalQueue, run the following command:
   123  
   124  ```shell
   125  kubectl apply -f default-user-queue.yaml
   126  ```
   127  
   128  ## Multiple ResourceFlavors setup
   129  
   130  You can define quotas for different [resource flavors](/docs/concepts/cluster_queue#resourceflavor-object).
   131  
   132  For the rest of this section, assume that your cluster has nodes with two CPU
   133  architectures, namely `x86` and `arm`, specified in the node label `cpu-arch`.
   134  
   135  
   136  ### 1. Create ResourceFlavors
   137  
   138  Write the manifests for the ResourceFlavors. They should look similar to the
   139  following:
   140  
   141  ```yaml
   142  # flavor-x86.yaml
   143  apiVersion: kueue.x-k8s.io/v1beta1
   144  kind: ResourceFlavor
   145  metadata:
   146    name: "x86"
   147  spec:
   148    nodeLabels:
   149      cpu-arch: x86
   150  ```
   151  
   152  ```yaml
   153  # flavor-arm.yaml
   154  apiVersion: kueue.x-k8s.io/v1beta1
   155  kind: ResourceFlavor
   156  metadata:
   157    name: "arm"
   158  spec:
   159    nodeLabels:
   160      cpu-arch: arm
   161  ```
   162  
   163  To create the ResourceFlavors, run the following command:
   164  
   165  ```shell
   166  kubectl apply -f flavor-x86.yaml -f flavor-arm.yaml
   167  ```
   168  
   169  The labels set in the ResourceFlavors should match the labels in your nodes.
   170  If you are using [cluster autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler)
   171  (or equivalent controllers), make sure it is configured to add those labels when
   172  adding new nodes.
   173  
   174  ### 2. Create a ClusterQueue referencing the flavors
   175  
   176  Write the manifest for the ClusterQueue that references the flavors. It should
   177  look similar to the following:
   178  
   179  ```yaml
   180  # cluster-queue.yaml
   181  apiVersion: kueue.x-k8s.io/v1beta1
   182  kind: ClusterQueue
   183  metadata:
   184    name: "cluster-queue"
   185  spec:
   186    namespaceSelector: {} # match all.
   187    resourceGroups:
   188    - coveredResources: ["cpu"]
   189      flavors:
   190      - name: "x86"
   191        resources:
   192        - name: "cpu"
   193          nominalQuota: 9
   194      - name: "arm"
   195        resources:
   196        - name: "cpu"
   197          nominalQuota: 12
   198    - coveredResources: ["memory"]
   199      flavors:
   200      - name: "default-flavor"
   201        resources:
   202        - name: "memory"
   203          nominalQuota: 84Gi
   204  ```
   205  
   206  The flavor names in the fields `.spec.resourceGroups[*].flavors[*].name`
   207  should match the names of the ResourceFlavors created earlier.
   208  
   209  Note that `memory` is referencing the `default-flavor` flavor created in the [single flavor setup.](#single-clusterqueue-and-single-resourceflavor-setup)
   210  This means that you don't want to distinguish if the memory is given from `x86`
   211  or `arm` nodes.
   212  
   213  To create the ClusterQueue, run the following command:
   214  
   215  ```shell
   216  kubectl apply -f cluster-queue.yaml
   217  ```
   218  
   219  ## Multiple ClusterQueues and borrowing cohorts
   220  
   221  Two or more ClusterQueues can borrow unused quota from other ClusterQueues in
   222  the same [cohort](/docs/concepts/cluster_queue#cohort).
   223  
   224  Using the following example, you can establish a cohort `team-ab` that includes
   225  ClusterQueues `team-a-cq` and `team-b-cq`.
   226  
   227  ```yaml
   228  # team-a-cq.yaml
   229  apiVersion: kueue.x-k8s.io/v1beta1
   230  kind: ClusterQueue
   231  metadata:
   232    name: "team-a-cq"
   233  spec:
   234    namespaceSelector: {} # match all.
   235    cohort: "team-ab"
   236    resourceGroups:
   237    - coveredResources: ["cpu", "memory"]
   238      flavors:
   239      - name: "default-flavor"
   240        resources:
   241        - name: "cpu"
   242          nominalQuota: 9
   243          borrowingLimit: 6
   244        - name: "memory"
   245          nominalQuota: 36Gi
   246          borrowingLimit: 24Gi
   247  ```
   248  
   249  ```yaml
   250  apiVersion: kueue.x-k8s.io/v1beta1
   251  kind: ClusterQueue
   252  metadata:
   253    name: "team-b-cq"
   254  spec:
   255    namespaceSelector: {}
   256    cohort: "team-ab"
   257    resourceGroups:
   258    - coveredResources: ["cpu", "memory"]
   259      flavors:
   260      - name: "default-flavor"
   261        resources:
   262        - name: "cpu"
   263          nominalQuota: 12
   264        - name: "memory"
   265          nominalQuota: 48Gi
   266  ```
   267  
   268  Note that the ClusterQueue `team-a-cq` also defines [borrowingLimit](/docs/concepts/cluster_queue#borrowingLimit).
   269  This restricts the ability of the ClusterQueue to borrow the unused quota from
   270  the cohort up to the configured `borrowingLimit`, even if the quota is completely unused.
   271  
   272  To create these ClusterQueues, save the preceding manifests and run the
   273  following command:
   274  
   275  ```shell
   276  kubectl apply -f team-a-cq.yaml -f team-b-cq.yaml
   277  ```
   278  
   279  ## Multiple ClusterQueue with dedicated and fallback flavors
   280  
   281  A ClusterQueue can borrow resources from the [cohort](/docs/concepts/cluster_queue#cohort)
   282  even if the ClusterQueue has zero nominalQuota for a flavor. This allows you to
   283  give dedicated quota for a flavor and fallback to quota for a different flavor,
   284  shared with other tenants.
   285  
   286  Such setup can be accomplished with a ClusterQueue for each tenant and an extra
   287  ClusterQueue for the shared resources. For example, the manifests for two
   288  tenants look like the following:
   289  
   290  ```yaml
   291  # team-a-cq.yaml
   292  apiVersion: kueue.x-k8s.io/v1beta1
   293  kind: ClusterQueue
   294  metadata:
   295    name: "team-a-cq"
   296  spec:
   297    namespaceSelector: {} # match all.
   298    cohort: "team-ab"
   299    resourceGroups:
   300    - coveredResources: ["cpu"]
   301      flavors:
   302      - name: "arm"
   303        resources:
   304        - name: "cpu"
   305          nominalQuota: 9
   306          borrowingLimit: 0
   307      - name: "x86"
   308        resources:
   309        - name: "cpu"
   310          nominalQuota: 0
   311    - coveredResources: ["memory"]
   312      flavors:
   313      - name: "default-flavor"
   314        resources:
   315        - name: "memory"
   316          nominalQuota: 36Gi
   317  ```
   318  
   319  ```yaml
   320  # team-b-cq.yaml
   321  apiVersion: kueue.x-k8s.io/v1beta1
   322  kind: ClusterQueue
   323  metadata:
   324    name: "team-b-cq"
   325  spec:
   326    namespaceSelector: {} # match all.
   327    cohort: "team-ab"
   328    resourceGroups:
   329    - coveredResources: ["cpu"]
   330      flavors:
   331      - name: "arm"
   332        resources:
   333        - name: "cpu"
   334          nominalQuota: 12
   335          borrowingLimit: 0
   336      - name: "x86"
   337        resources:
   338        - name: "cpu"
   339          nominalQuota: 0
   340    - coveredResources: ["memory"]
   341      flavors:
   342      - name: "default-flavor"
   343        resources:
   344        - name: "memory"
   345          nominalQuota: 48Gi
   346  ```
   347  
   348  ```yaml
   349  # shared-cq.yaml
   350  apiVersion: kueue.x-k8s.io/v1beta1
   351  kind: ClusterQueue
   352  metadata:
   353    name: "shared-cq"
   354  spec:
   355    namespaceSelector: {} # match all.
   356    cohort: "team-ab"
   357    resourceGroups:
   358    - coveredResources: ["cpu"]
   359      flavors:
   360      - name: "x86"
   361        resources:
   362        - name: "cpu"
   363          nominalQuota: 6
   364    - coveredResources: ["memory"]
   365      flavors:
   366      - name: "default-flavor"
   367        resources:
   368        - name: "memory"
   369          nominalQuota: 24Gi
   370  ```
   371  
   372  Note the following setup:
   373  
   374  - `team-a-cq` and `team-b-cq` define a `borrowingLimit: 0` 
   375    for the `arm` flavor. Therefore, they can't borrow this flavor from each
   376    other.
   377  - `team-a-cq` and `team-b-cq` define `nominalQuota: 0` for the `x86` flavor.
   378    Therefore, they don't have any dedicated quota for the flavor and they can
   379    only borrow it from `shared-cq`.
   380  
   381  To create these ClusterQueues, save the preceding manifests and run the
   382  following command:
   383  
   384  ```shell
   385  kubectl apply -f team-a-cq.yaml -f team-b-cq.yaml -f shared-cq.yaml
   386  ```