sigs.k8s.io/kueue@v0.6.2/config/components/crd/bases/kueue.x-k8s.io_clusterqueues.yaml (about) 1 --- 2 apiVersion: apiextensions.k8s.io/v1 3 kind: CustomResourceDefinition 4 metadata: 5 annotations: 6 controller-gen.kubebuilder.io/version: v0.14.0 7 name: clusterqueues.kueue.x-k8s.io 8 spec: 9 group: kueue.x-k8s.io 10 names: 11 kind: ClusterQueue 12 listKind: ClusterQueueList 13 plural: clusterqueues 14 singular: clusterqueue 15 scope: Cluster 16 versions: 17 - additionalPrinterColumns: 18 - description: Cohort that this ClusterQueue belongs to 19 jsonPath: .spec.cohort 20 name: Cohort 21 type: string 22 - description: The queueing strategy used to prioritize workloads 23 jsonPath: .spec.queueingStrategy 24 name: Strategy 25 priority: 1 26 type: string 27 - description: Number of pending workloads 28 jsonPath: .status.pendingWorkloads 29 name: Pending Workloads 30 type: integer 31 - description: Number of admitted workloads that haven't finished yet 32 jsonPath: .status.admittedWorkloads 33 name: Admitted Workloads 34 priority: 1 35 type: integer 36 name: v1beta1 37 schema: 38 openAPIV3Schema: 39 description: ClusterQueue is the Schema for the clusterQueue API. 40 properties: 41 apiVersion: 42 description: |- 43 APIVersion defines the versioned schema of this representation of an object. 44 Servers should convert recognized schemas to the latest internal value, and 45 may reject unrecognized values. 46 More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources 47 type: string 48 kind: 49 description: |- 50 Kind is a string value representing the REST resource this object represents. 51 Servers may infer this from the endpoint the client submits requests to. 52 Cannot be updated. 53 In CamelCase. 54 More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds 55 type: string 56 metadata: 57 type: object 58 spec: 59 description: ClusterQueueSpec defines the desired state of ClusterQueue 60 properties: 61 admissionChecks: 62 description: admissionChecks lists the AdmissionChecks required by 63 this ClusterQueue 64 items: 65 type: string 66 type: array 67 cohort: 68 description: |- 69 cohort that this ClusterQueue belongs to. CQs that belong to the 70 same cohort can borrow unused resources from each other. 71 72 73 A CQ can be a member of a single borrowing cohort. A workload submitted 74 to a queue referencing this CQ can borrow quota from any CQ in the cohort. 75 Only quota for the [resource, flavor] pairs listed in the CQ can be 76 borrowed. 77 If empty, this ClusterQueue cannot borrow from any other ClusterQueue and 78 vice versa. 79 80 81 A cohort is a name that links CQs together, but it doesn't reference any 82 object. 83 84 85 Validation of a cohort name is equivalent to that of object names: 86 subdomain in DNS (RFC 1123). 87 type: string 88 flavorFungibility: 89 description: |- 90 flavorFungibility defines whether a workload should try the next flavor 91 before borrowing or preempting in the flavor being evaluated. 92 properties: 93 whenCanBorrow: 94 default: Borrow 95 description: |- 96 whenCanBorrow determines whether a workload should try the next flavor 97 before borrowing in current flavor. The possible values are: 98 99 100 - `Borrow` (default): allocate in current flavor if borrowing 101 is possible. 102 - `TryNextFlavor`: try next flavor even if the current 103 flavor has enough resources to borrow. 104 enum: 105 - Borrow 106 - TryNextFlavor 107 type: string 108 whenCanPreempt: 109 default: TryNextFlavor 110 description: |- 111 whenCanPreempt determines whether a workload should try the next flavor 112 before borrowing in current flavor. The possible values are: 113 114 115 - `Preempt`: allocate in current flavor if it's possible to preempt some workloads. 116 - `TryNextFlavor` (default): try next flavor even if there are enough 117 candidates for preemption in the current flavor. 118 enum: 119 - Preempt 120 - TryNextFlavor 121 type: string 122 type: object 123 namespaceSelector: 124 description: |- 125 namespaceSelector defines which namespaces are allowed to submit workloads to 126 this clusterQueue. Beyond this basic support for policy, a policy agent like 127 Gatekeeper should be used to enforce more advanced policies. 128 Defaults to null which is a nothing selector (no namespaces eligible). 129 If set to an empty selector `{}`, then all namespaces are eligible. 130 properties: 131 matchExpressions: 132 description: matchExpressions is a list of label selector requirements. 133 The requirements are ANDed. 134 items: 135 description: |- 136 A label selector requirement is a selector that contains values, a key, and an operator that 137 relates the key and values. 138 properties: 139 key: 140 description: key is the label key that the selector applies 141 to. 142 type: string 143 operator: 144 description: |- 145 operator represents a key's relationship to a set of values. 146 Valid operators are In, NotIn, Exists and DoesNotExist. 147 type: string 148 values: 149 description: |- 150 values is an array of string values. If the operator is In or NotIn, 151 the values array must be non-empty. If the operator is Exists or DoesNotExist, 152 the values array must be empty. This array is replaced during a strategic 153 merge patch. 154 items: 155 type: string 156 type: array 157 required: 158 - key 159 - operator 160 type: object 161 type: array 162 matchLabels: 163 additionalProperties: 164 type: string 165 description: |- 166 matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels 167 map is equivalent to an element of matchExpressions, whose key field is "key", the 168 operator is "In", and the values array contains only "value". The requirements are ANDed. 169 type: object 170 type: object 171 x-kubernetes-map-type: atomic 172 preemption: 173 description: |- 174 preemption describes policies to preempt Workloads from this ClusterQueue 175 or the ClusterQueue's cohort. 176 177 178 Preemption can happen in two scenarios: 179 180 181 - When a Workload fits within the nominal quota of the ClusterQueue, but 182 the quota is currently borrowed by other ClusterQueues in the cohort. 183 Preempting Workloads in other ClusterQueues allows this ClusterQueue to 184 reclaim its nominal quota. 185 - When a Workload doesn't fit within the nominal quota of the ClusterQueue 186 and there are admitted Workloads in the ClusterQueue with lower priority. 187 188 189 The preemption algorithm tries to find a minimal set of Workloads to 190 preempt to accomomdate the pending Workload, preempting Workloads with 191 lower priority first. 192 properties: 193 borrowWithinCohort: 194 description: |- 195 borrowWithinCohort provides configuration to allow preemption within 196 cohort while borrowing. 197 properties: 198 maxPriorityThreshold: 199 description: |- 200 maxPriorityThreshold allows to restrict the set of workloads which 201 might be preempted by a borrowing workload, to only workloads with 202 priority less than or equal to the specified threshold priority. 203 When the threshold is not specified, then any workload satisfying the 204 policy can be preempted by the borrowing workload. 205 format: int32 206 type: integer 207 policy: 208 default: Never 209 description: |- 210 policy determines the policy for preemption to reclaim quota within cohort while borrowing. 211 Possible values are: 212 - `Never` (default): do not allow for preemption, in other 213 ClusterQueues within the cohort, for a borrowing workload. 214 - `LowerPriority`: allow preemption, in other ClusterQueues 215 within the cohort, for a borrowing workload, but only if 216 the preempted workloads are of lower priority. 217 enum: 218 - Never 219 - LowerPriority 220 type: string 221 type: object 222 reclaimWithinCohort: 223 default: Never 224 description: |- 225 reclaimWithinCohort determines whether a pending Workload can preempt 226 Workloads from other ClusterQueues in the cohort that are using more than 227 their nominal quota. The possible values are: 228 229 230 - `Never` (default): do not preempt Workloads in the cohort. 231 - `LowerPriority`: if the pending Workload fits within the nominal 232 quota of its ClusterQueue, only preempt Workloads in the cohort that have 233 lower priority than the pending Workload. 234 - `Any`: if the pending Workload fits within the nominal quota of its 235 ClusterQueue, preempt any Workload in the cohort, irrespective of 236 priority. 237 enum: 238 - Never 239 - LowerPriority 240 - Any 241 type: string 242 withinClusterQueue: 243 default: Never 244 description: |- 245 withinClusterQueue determines whether a pending Workload that doesn't fit 246 within the nominal quota for its ClusterQueue, can preempt active Workloads in 247 the ClusterQueue. The possible values are: 248 249 250 - `Never` (default): do not preempt Workloads in the ClusterQueue. 251 - `LowerPriority`: only preempt Workloads in the ClusterQueue that have 252 lower priority than the pending Workload. 253 - `LowerOrNewerEqualPriority`: only preempt Workloads in the ClusterQueue that 254 either have a lower priority than the pending workload or equal priority 255 and are newer than the pending workload. 256 enum: 257 - Never 258 - LowerPriority 259 - LowerOrNewerEqualPriority 260 type: string 261 type: object 262 queueingStrategy: 263 default: BestEffortFIFO 264 description: |- 265 QueueingStrategy indicates the queueing strategy of the workloads 266 across the queues in this ClusterQueue. This field is immutable. 267 Current Supported Strategies: 268 269 270 - StrictFIFO: workloads are ordered strictly by creation time. 271 Older workloads that can't be admitted will block admitting newer 272 workloads even if they fit available quota. 273 - BestEffortFIFO: workloads are ordered by creation time, 274 however older workloads that can't be admitted will not block 275 admitting newer workloads that fit existing quota. 276 enum: 277 - StrictFIFO 278 - BestEffortFIFO 279 type: string 280 resourceGroups: 281 description: |- 282 resourceGroups describes groups of resources. 283 Each resource group defines the list of resources and a list of flavors 284 that provide quotas for these resources. 285 Each resource and each flavor can only form part of one resource group. 286 resourceGroups can be up to 16. 287 items: 288 properties: 289 coveredResources: 290 description: |- 291 coveredResources is the list of resources covered by the flavors in this 292 group. 293 Examples: cpu, memory, vendor.com/gpu. 294 The list cannot be empty and it can contain up to 16 resources. 295 items: 296 description: ResourceName is the name identifying various 297 resources in a ResourceList. 298 type: string 299 maxItems: 16 300 minItems: 1 301 type: array 302 flavors: 303 description: |- 304 flavors is the list of flavors that provide the resources of this group. 305 Typically, different flavors represent different hardware models 306 (e.g., gpu models, cpu architectures) or pricing models (on-demand vs spot 307 cpus). 308 Each flavor MUST list all the resources listed for this group in the same 309 order as the .resources field. 310 The list cannot be empty and it can contain up to 16 flavors. 311 items: 312 properties: 313 name: 314 description: |- 315 name of this flavor. The name should match the .metadata.name of a 316 ResourceFlavor. If a matching ResourceFlavor does not exist, the 317 ClusterQueue will have an Active condition set to False. 318 type: string 319 resources: 320 description: |- 321 resources is the list of quotas for this flavor per resource. 322 There could be up to 16 resources. 323 items: 324 properties: 325 borrowingLimit: 326 anyOf: 327 - type: integer 328 - type: string 329 description: |- 330 borrowingLimit is the maximum amount of quota for the [flavor, resource] 331 combination that this ClusterQueue is allowed to borrow from the unused 332 quota of other ClusterQueues in the same cohort. 333 In total, at a given time, Workloads in a ClusterQueue can consume a 334 quantity of quota equal to nominalQuota+borrowingLimit, assuming the other 335 ClusterQueues in the cohort have enough unused quota. 336 If null, it means that there is no borrowing limit. 337 If not null, it must be non-negative. 338 borrowingLimit must be null if spec.cohort is empty. 339 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ 340 x-kubernetes-int-or-string: true 341 lendingLimit: 342 anyOf: 343 - type: integer 344 - type: string 345 description: |- 346 lendingLimit is the maximum amount of unused quota for the [flavor, resource] 347 combination that this ClusterQueue can lend to other ClusterQueues in the same cohort. 348 In total, at a given time, ClusterQueue reserves for its exclusive use 349 a quantity of quota equals to nominalQuota - lendingLimit. 350 If null, it means that there is no lending limit, meaning that 351 all the nominalQuota can be borrowed by other clusterQueues in the cohort. 352 If not null, it must be non-negative. 353 lendingLimit must be null if spec.cohort is empty. 354 This field is in alpha stage. To be able to use this field, 355 enable the feature gate LendingLimit, which is disabled by default. 356 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ 357 x-kubernetes-int-or-string: true 358 name: 359 description: name of this resource. 360 type: string 361 nominalQuota: 362 anyOf: 363 - type: integer 364 - type: string 365 description: |- 366 nominalQuota is the quantity of this resource that is available for 367 Workloads admitted by this ClusterQueue at a point in time. 368 The nominalQuota must be non-negative. 369 nominalQuota should represent the resources in the cluster available for 370 running jobs (after discounting resources consumed by system components 371 and pods not managed by kueue). In an autoscaled cluster, nominalQuota 372 should account for resources that can be provided by a component such as 373 Kubernetes cluster-autoscaler. 374 375 376 If the ClusterQueue belongs to a cohort, the sum of the quotas for each 377 (flavor, resource) combination defines the maximum quantity that can be 378 allocated by a ClusterQueue in the cohort. 379 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ 380 x-kubernetes-int-or-string: true 381 required: 382 - name 383 - nominalQuota 384 type: object 385 maxItems: 16 386 minItems: 1 387 type: array 388 x-kubernetes-list-map-keys: 389 - name 390 x-kubernetes-list-type: map 391 required: 392 - name 393 - resources 394 type: object 395 maxItems: 16 396 minItems: 1 397 type: array 398 x-kubernetes-list-map-keys: 399 - name 400 x-kubernetes-list-type: map 401 required: 402 - coveredResources 403 - flavors 404 type: object 405 maxItems: 16 406 type: array 407 x-kubernetes-list-type: atomic 408 stopPolicy: 409 default: None 410 description: |- 411 stopPolicy - if set to a value different from None, the ClusterQueue is considered Inactive, no new reservation being 412 made. 413 414 415 Depending on its value, its associated workloads will: 416 417 418 - None - Workloads are admitted 419 - HoldAndDrain - Admitted workloads are evicted and Reserving workloads will cancel the reservation. 420 - Hold - Admitted workloads will run to completion and Reserving workloads will cancel the reservation. 421 enum: 422 - None 423 - Hold 424 - HoldAndDrain 425 type: string 426 type: object 427 status: 428 description: ClusterQueueStatus defines the observed state of ClusterQueue 429 properties: 430 admittedWorkloads: 431 description: |- 432 admittedWorkloads is the number of workloads currently admitted to this 433 clusterQueue and haven't finished yet. 434 format: int32 435 type: integer 436 conditions: 437 description: |- 438 conditions hold the latest available observations of the ClusterQueue 439 current state. 440 items: 441 description: "Condition contains details for one aspect of the current 442 state of this API Resource.\n---\nThis struct is intended for 443 direct use as an array at the field path .status.conditions. For 444 example,\n\n\n\ttype FooStatus struct{\n\t // Represents the 445 observations of a foo's current state.\n\t // Known .status.conditions.type 446 are: \"Available\", \"Progressing\", and \"Degraded\"\n\t // 447 +patchMergeKey=type\n\t // +patchStrategy=merge\n\t // +listType=map\n\t 448 \ // +listMapKey=type\n\t Conditions []metav1.Condition `json:\"conditions,omitempty\" 449 patchStrategy:\"merge\" patchMergeKey:\"type\" protobuf:\"bytes,1,rep,name=conditions\"`\n\n\n\t 450 \ // other fields\n\t}" 451 properties: 452 lastTransitionTime: 453 description: |- 454 lastTransitionTime is the last time the condition transitioned from one status to another. 455 This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. 456 format: date-time 457 type: string 458 message: 459 description: |- 460 message is a human readable message indicating details about the transition. 461 This may be an empty string. 462 maxLength: 32768 463 type: string 464 observedGeneration: 465 description: |- 466 observedGeneration represents the .metadata.generation that the condition was set based upon. 467 For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date 468 with respect to the current state of the instance. 469 format: int64 470 minimum: 0 471 type: integer 472 reason: 473 description: |- 474 reason contains a programmatic identifier indicating the reason for the condition's last transition. 475 Producers of specific condition types may define expected values and meanings for this field, 476 and whether the values are considered a guaranteed API. 477 The value should be a CamelCase string. 478 This field may not be empty. 479 maxLength: 1024 480 minLength: 1 481 pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ 482 type: string 483 status: 484 description: status of the condition, one of True, False, Unknown. 485 enum: 486 - "True" 487 - "False" 488 - Unknown 489 type: string 490 type: 491 description: |- 492 type of condition in CamelCase or in foo.example.com/CamelCase. 493 --- 494 Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be 495 useful (see .node.status.conditions), the ability to deconflict is important. 496 The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt) 497 maxLength: 316 498 pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ 499 type: string 500 required: 501 - lastTransitionTime 502 - message 503 - reason 504 - status 505 - type 506 type: object 507 type: array 508 x-kubernetes-list-map-keys: 509 - type 510 x-kubernetes-list-type: map 511 flavorsReservation: 512 description: |- 513 flavorsReservation are the reserved quotas, by flavor, currently in use by the 514 workloads assigned to this ClusterQueue. 515 items: 516 properties: 517 name: 518 description: name of the flavor. 519 type: string 520 resources: 521 description: resources lists the quota usage for the resources 522 in this flavor. 523 items: 524 properties: 525 borrowed: 526 anyOf: 527 - type: integer 528 - type: string 529 description: |- 530 Borrowed is quantity of quota that is borrowed from the cohort. In other 531 words, it's the used quota that is over the nominalQuota. 532 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ 533 x-kubernetes-int-or-string: true 534 name: 535 description: name of the resource 536 type: string 537 total: 538 anyOf: 539 - type: integer 540 - type: string 541 description: |- 542 total is the total quantity of used quota, including the amount borrowed 543 from the cohort. 544 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ 545 x-kubernetes-int-or-string: true 546 required: 547 - name 548 type: object 549 maxItems: 16 550 type: array 551 x-kubernetes-list-map-keys: 552 - name 553 x-kubernetes-list-type: map 554 required: 555 - name 556 - resources 557 type: object 558 maxItems: 16 559 type: array 560 x-kubernetes-list-map-keys: 561 - name 562 x-kubernetes-list-type: map 563 flavorsUsage: 564 description: |- 565 flavorsUsage are the used quotas, by flavor, currently in use by the 566 workloads admitted in this ClusterQueue. 567 items: 568 properties: 569 name: 570 description: name of the flavor. 571 type: string 572 resources: 573 description: resources lists the quota usage for the resources 574 in this flavor. 575 items: 576 properties: 577 borrowed: 578 anyOf: 579 - type: integer 580 - type: string 581 description: |- 582 Borrowed is quantity of quota that is borrowed from the cohort. In other 583 words, it's the used quota that is over the nominalQuota. 584 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ 585 x-kubernetes-int-or-string: true 586 name: 587 description: name of the resource 588 type: string 589 total: 590 anyOf: 591 - type: integer 592 - type: string 593 description: |- 594 total is the total quantity of used quota, including the amount borrowed 595 from the cohort. 596 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ 597 x-kubernetes-int-or-string: true 598 required: 599 - name 600 type: object 601 maxItems: 16 602 type: array 603 x-kubernetes-list-map-keys: 604 - name 605 x-kubernetes-list-type: map 606 required: 607 - name 608 - resources 609 type: object 610 maxItems: 16 611 type: array 612 x-kubernetes-list-map-keys: 613 - name 614 x-kubernetes-list-type: map 615 pendingWorkloads: 616 description: |- 617 pendingWorkloads is the number of workloads currently waiting to be 618 admitted to this clusterQueue. 619 format: int32 620 type: integer 621 pendingWorkloadsStatus: 622 description: |- 623 PendingWorkloadsStatus contains the information exposed about the current 624 status of the pending workloads in the cluster queue. 625 properties: 626 clusterQueuePendingWorkload: 627 description: Head contains the list of top pending workloads. 628 items: 629 description: |- 630 ClusterQueuePendingWorkload contains the information identifying a pending workload 631 in the cluster queue. 632 properties: 633 name: 634 description: Name indicates the name of the pending workload. 635 type: string 636 namespace: 637 description: Namespace indicates the name of the pending 638 workload. 639 type: string 640 required: 641 - name 642 - namespace 643 type: object 644 type: array 645 x-kubernetes-list-type: atomic 646 lastChangeTime: 647 description: LastChangeTime indicates the time of the last change 648 of the structure. 649 format: date-time 650 type: string 651 required: 652 - lastChangeTime 653 type: object 654 reservingWorkloads: 655 description: |- 656 reservingWorkloads is the number of workloads currently reserving quota in this 657 clusterQueue. 658 format: int32 659 type: integer 660 type: object 661 type: object 662 served: true 663 storage: true 664 subresources: 665 status: {}