sigs.k8s.io/kueue@v0.6.2/charts/kueue/templates/crd/kueue.x-k8s.io_clusterqueues.yaml (about) 1 --- 2 apiVersion: apiextensions.k8s.io/v1 3 kind: CustomResourceDefinition 4 metadata: 5 annotations: 6 {{- if .Values.enableCertManager }} 7 cert-manager.io/inject-ca-from: {{ .Release.Namespace }}/{{ include "kueue.fullname" . }}-serving-cert 8 {{- end }} 9 controller-gen.kubebuilder.io/version: v0.14.0 10 name: clusterqueues.kueue.x-k8s.io 11 spec: 12 conversion: 13 strategy: Webhook 14 webhook: 15 clientConfig: 16 service: 17 name: {{ include "kueue.fullname" . }}-webhook-service 18 namespace: '{{ .Release.Namespace }}' 19 path: /convert 20 conversionReviewVersions: 21 - v1 22 group: kueue.x-k8s.io 23 names: 24 kind: ClusterQueue 25 listKind: ClusterQueueList 26 plural: clusterqueues 27 singular: clusterqueue 28 scope: Cluster 29 versions: 30 - additionalPrinterColumns: 31 - description: Cohort that this ClusterQueue belongs to 32 jsonPath: .spec.cohort 33 name: Cohort 34 type: string 35 - description: The queueing strategy used to prioritize workloads 36 jsonPath: .spec.queueingStrategy 37 name: Strategy 38 priority: 1 39 type: string 40 - description: Number of pending workloads 41 jsonPath: .status.pendingWorkloads 42 name: Pending Workloads 43 type: integer 44 - description: Number of admitted workloads that haven't finished yet 45 jsonPath: .status.admittedWorkloads 46 name: Admitted Workloads 47 priority: 1 48 type: integer 49 name: v1beta1 50 schema: 51 openAPIV3Schema: 52 description: ClusterQueue is the Schema for the clusterQueue API. 53 properties: 54 apiVersion: 55 description: |- 56 APIVersion defines the versioned schema of this representation of an object. 57 Servers should convert recognized schemas to the latest internal value, and 58 may reject unrecognized values. 59 More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources 60 type: string 61 kind: 62 description: |- 63 Kind is a string value representing the REST resource this object represents. 64 Servers may infer this from the endpoint the client submits requests to. 65 Cannot be updated. 66 In CamelCase. 67 More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds 68 type: string 69 metadata: 70 type: object 71 spec: 72 description: ClusterQueueSpec defines the desired state of ClusterQueue 73 properties: 74 admissionChecks: 75 description: admissionChecks lists the AdmissionChecks required by 76 this ClusterQueue 77 items: 78 type: string 79 type: array 80 cohort: 81 description: |- 82 cohort that this ClusterQueue belongs to. CQs that belong to the 83 same cohort can borrow unused resources from each other. 84 85 86 A CQ can be a member of a single borrowing cohort. A workload submitted 87 to a queue referencing this CQ can borrow quota from any CQ in the cohort. 88 Only quota for the [resource, flavor] pairs listed in the CQ can be 89 borrowed. 90 If empty, this ClusterQueue cannot borrow from any other ClusterQueue and 91 vice versa. 92 93 94 A cohort is a name that links CQs together, but it doesn't reference any 95 object. 96 97 98 Validation of a cohort name is equivalent to that of object names: 99 subdomain in DNS (RFC 1123). 100 type: string 101 flavorFungibility: 102 description: |- 103 flavorFungibility defines whether a workload should try the next flavor 104 before borrowing or preempting in the flavor being evaluated. 105 properties: 106 whenCanBorrow: 107 default: Borrow 108 description: |- 109 whenCanBorrow determines whether a workload should try the next flavor 110 before borrowing in current flavor. The possible values are: 111 112 113 - `Borrow` (default): allocate in current flavor if borrowing 114 is possible. 115 - `TryNextFlavor`: try next flavor even if the current 116 flavor has enough resources to borrow. 117 enum: 118 - Borrow 119 - TryNextFlavor 120 type: string 121 whenCanPreempt: 122 default: TryNextFlavor 123 description: |- 124 whenCanPreempt determines whether a workload should try the next flavor 125 before borrowing in current flavor. The possible values are: 126 127 128 - `Preempt`: allocate in current flavor if it's possible to preempt some workloads. 129 - `TryNextFlavor` (default): try next flavor even if there are enough 130 candidates for preemption in the current flavor. 131 enum: 132 - Preempt 133 - TryNextFlavor 134 type: string 135 type: object 136 namespaceSelector: 137 description: |- 138 namespaceSelector defines which namespaces are allowed to submit workloads to 139 this clusterQueue. Beyond this basic support for policy, a policy agent like 140 Gatekeeper should be used to enforce more advanced policies. 141 Defaults to null which is a nothing selector (no namespaces eligible). 142 If set to an empty selector `{}`, then all namespaces are eligible. 143 properties: 144 matchExpressions: 145 description: matchExpressions is a list of label selector requirements. 146 The requirements are ANDed. 147 items: 148 description: |- 149 A label selector requirement is a selector that contains values, a key, and an operator that 150 relates the key and values. 151 properties: 152 key: 153 description: key is the label key that the selector applies 154 to. 155 type: string 156 operator: 157 description: |- 158 operator represents a key's relationship to a set of values. 159 Valid operators are In, NotIn, Exists and DoesNotExist. 160 type: string 161 values: 162 description: |- 163 values is an array of string values. If the operator is In or NotIn, 164 the values array must be non-empty. If the operator is Exists or DoesNotExist, 165 the values array must be empty. This array is replaced during a strategic 166 merge patch. 167 items: 168 type: string 169 type: array 170 required: 171 - key 172 - operator 173 type: object 174 type: array 175 matchLabels: 176 additionalProperties: 177 type: string 178 description: |- 179 matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels 180 map is equivalent to an element of matchExpressions, whose key field is "key", the 181 operator is "In", and the values array contains only "value". The requirements are ANDed. 182 type: object 183 type: object 184 x-kubernetes-map-type: atomic 185 preemption: 186 description: |- 187 preemption describes policies to preempt Workloads from this ClusterQueue 188 or the ClusterQueue's cohort. 189 190 191 Preemption can happen in two scenarios: 192 193 194 - When a Workload fits within the nominal quota of the ClusterQueue, but 195 the quota is currently borrowed by other ClusterQueues in the cohort. 196 Preempting Workloads in other ClusterQueues allows this ClusterQueue to 197 reclaim its nominal quota. 198 - When a Workload doesn't fit within the nominal quota of the ClusterQueue 199 and there are admitted Workloads in the ClusterQueue with lower priority. 200 201 202 The preemption algorithm tries to find a minimal set of Workloads to 203 preempt to accomomdate the pending Workload, preempting Workloads with 204 lower priority first. 205 properties: 206 borrowWithinCohort: 207 description: |- 208 borrowWithinCohort provides configuration to allow preemption within 209 cohort while borrowing. 210 properties: 211 maxPriorityThreshold: 212 description: |- 213 maxPriorityThreshold allows to restrict the set of workloads which 214 might be preempted by a borrowing workload, to only workloads with 215 priority less than or equal to the specified threshold priority. 216 When the threshold is not specified, then any workload satisfying the 217 policy can be preempted by the borrowing workload. 218 format: int32 219 type: integer 220 policy: 221 default: Never 222 description: |- 223 policy determines the policy for preemption to reclaim quota within cohort while borrowing. 224 Possible values are: 225 - `Never` (default): do not allow for preemption, in other 226 ClusterQueues within the cohort, for a borrowing workload. 227 - `LowerPriority`: allow preemption, in other ClusterQueues 228 within the cohort, for a borrowing workload, but only if 229 the preempted workloads are of lower priority. 230 enum: 231 - Never 232 - LowerPriority 233 type: string 234 type: object 235 reclaimWithinCohort: 236 default: Never 237 description: |- 238 reclaimWithinCohort determines whether a pending Workload can preempt 239 Workloads from other ClusterQueues in the cohort that are using more than 240 their nominal quota. The possible values are: 241 242 243 - `Never` (default): do not preempt Workloads in the cohort. 244 - `LowerPriority`: if the pending Workload fits within the nominal 245 quota of its ClusterQueue, only preempt Workloads in the cohort that have 246 lower priority than the pending Workload. 247 - `Any`: if the pending Workload fits within the nominal quota of its 248 ClusterQueue, preempt any Workload in the cohort, irrespective of 249 priority. 250 enum: 251 - Never 252 - LowerPriority 253 - Any 254 type: string 255 withinClusterQueue: 256 default: Never 257 description: |- 258 withinClusterQueue determines whether a pending Workload that doesn't fit 259 within the nominal quota for its ClusterQueue, can preempt active Workloads in 260 the ClusterQueue. The possible values are: 261 262 263 - `Never` (default): do not preempt Workloads in the ClusterQueue. 264 - `LowerPriority`: only preempt Workloads in the ClusterQueue that have 265 lower priority than the pending Workload. 266 - `LowerOrNewerEqualPriority`: only preempt Workloads in the ClusterQueue that 267 either have a lower priority than the pending workload or equal priority 268 and are newer than the pending workload. 269 enum: 270 - Never 271 - LowerPriority 272 - LowerOrNewerEqualPriority 273 type: string 274 type: object 275 queueingStrategy: 276 default: BestEffortFIFO 277 description: |- 278 QueueingStrategy indicates the queueing strategy of the workloads 279 across the queues in this ClusterQueue. This field is immutable. 280 Current Supported Strategies: 281 282 283 - StrictFIFO: workloads are ordered strictly by creation time. 284 Older workloads that can't be admitted will block admitting newer 285 workloads even if they fit available quota. 286 - BestEffortFIFO: workloads are ordered by creation time, 287 however older workloads that can't be admitted will not block 288 admitting newer workloads that fit existing quota. 289 enum: 290 - StrictFIFO 291 - BestEffortFIFO 292 type: string 293 resourceGroups: 294 description: |- 295 resourceGroups describes groups of resources. 296 Each resource group defines the list of resources and a list of flavors 297 that provide quotas for these resources. 298 Each resource and each flavor can only form part of one resource group. 299 resourceGroups can be up to 16. 300 items: 301 properties: 302 coveredResources: 303 description: |- 304 coveredResources is the list of resources covered by the flavors in this 305 group. 306 Examples: cpu, memory, vendor.com/gpu. 307 The list cannot be empty and it can contain up to 16 resources. 308 items: 309 description: ResourceName is the name identifying various 310 resources in a ResourceList. 311 type: string 312 maxItems: 16 313 minItems: 1 314 type: array 315 flavors: 316 description: |- 317 flavors is the list of flavors that provide the resources of this group. 318 Typically, different flavors represent different hardware models 319 (e.g., gpu models, cpu architectures) or pricing models (on-demand vs spot 320 cpus). 321 Each flavor MUST list all the resources listed for this group in the same 322 order as the .resources field. 323 The list cannot be empty and it can contain up to 16 flavors. 324 items: 325 properties: 326 name: 327 description: |- 328 name of this flavor. The name should match the .metadata.name of a 329 ResourceFlavor. If a matching ResourceFlavor does not exist, the 330 ClusterQueue will have an Active condition set to False. 331 type: string 332 resources: 333 description: |- 334 resources is the list of quotas for this flavor per resource. 335 There could be up to 16 resources. 336 items: 337 properties: 338 borrowingLimit: 339 anyOf: 340 - type: integer 341 - type: string 342 description: |- 343 borrowingLimit is the maximum amount of quota for the [flavor, resource] 344 combination that this ClusterQueue is allowed to borrow from the unused 345 quota of other ClusterQueues in the same cohort. 346 In total, at a given time, Workloads in a ClusterQueue can consume a 347 quantity of quota equal to nominalQuota+borrowingLimit, assuming the other 348 ClusterQueues in the cohort have enough unused quota. 349 If null, it means that there is no borrowing limit. 350 If not null, it must be non-negative. 351 borrowingLimit must be null if spec.cohort is empty. 352 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ 353 x-kubernetes-int-or-string: true 354 lendingLimit: 355 anyOf: 356 - type: integer 357 - type: string 358 description: |- 359 lendingLimit is the maximum amount of unused quota for the [flavor, resource] 360 combination that this ClusterQueue can lend to other ClusterQueues in the same cohort. 361 In total, at a given time, ClusterQueue reserves for its exclusive use 362 a quantity of quota equals to nominalQuota - lendingLimit. 363 If null, it means that there is no lending limit, meaning that 364 all the nominalQuota can be borrowed by other clusterQueues in the cohort. 365 If not null, it must be non-negative. 366 lendingLimit must be null if spec.cohort is empty. 367 This field is in alpha stage. To be able to use this field, 368 enable the feature gate LendingLimit, which is disabled by default. 369 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ 370 x-kubernetes-int-or-string: true 371 name: 372 description: name of this resource. 373 type: string 374 nominalQuota: 375 anyOf: 376 - type: integer 377 - type: string 378 description: |- 379 nominalQuota is the quantity of this resource that is available for 380 Workloads admitted by this ClusterQueue at a point in time. 381 The nominalQuota must be non-negative. 382 nominalQuota should represent the resources in the cluster available for 383 running jobs (after discounting resources consumed by system components 384 and pods not managed by kueue). In an autoscaled cluster, nominalQuota 385 should account for resources that can be provided by a component such as 386 Kubernetes cluster-autoscaler. 387 388 389 If the ClusterQueue belongs to a cohort, the sum of the quotas for each 390 (flavor, resource) combination defines the maximum quantity that can be 391 allocated by a ClusterQueue in the cohort. 392 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ 393 x-kubernetes-int-or-string: true 394 required: 395 - name 396 - nominalQuota 397 type: object 398 maxItems: 16 399 minItems: 1 400 type: array 401 x-kubernetes-list-map-keys: 402 - name 403 x-kubernetes-list-type: map 404 required: 405 - name 406 - resources 407 type: object 408 maxItems: 16 409 minItems: 1 410 type: array 411 x-kubernetes-list-map-keys: 412 - name 413 x-kubernetes-list-type: map 414 required: 415 - coveredResources 416 - flavors 417 type: object 418 maxItems: 16 419 type: array 420 x-kubernetes-list-type: atomic 421 stopPolicy: 422 default: None 423 description: |- 424 stopPolicy - if set to a value different from None, the ClusterQueue is considered Inactive, no new reservation being 425 made. 426 427 428 Depending on its value, its associated workloads will: 429 430 431 - None - Workloads are admitted 432 - HoldAndDrain - Admitted workloads are evicted and Reserving workloads will cancel the reservation. 433 - Hold - Admitted workloads will run to completion and Reserving workloads will cancel the reservation. 434 enum: 435 - None 436 - Hold 437 - HoldAndDrain 438 type: string 439 type: object 440 status: 441 description: ClusterQueueStatus defines the observed state of ClusterQueue 442 properties: 443 admittedWorkloads: 444 description: |- 445 admittedWorkloads is the number of workloads currently admitted to this 446 clusterQueue and haven't finished yet. 447 format: int32 448 type: integer 449 conditions: 450 description: |- 451 conditions hold the latest available observations of the ClusterQueue 452 current state. 453 items: 454 description: "Condition contains details for one aspect of the current 455 state of this API Resource.\n---\nThis struct is intended for 456 direct use as an array at the field path .status.conditions. For 457 example,\n\n\n\ttype FooStatus struct{\n\t // Represents the 458 observations of a foo's current state.\n\t // Known .status.conditions.type 459 are: \"Available\", \"Progressing\", and \"Degraded\"\n\t // 460 +patchMergeKey=type\n\t // +patchStrategy=merge\n\t // +listType=map\n\t 461 \ // +listMapKey=type\n\t Conditions []metav1.Condition `json:\"conditions,omitempty\" 462 patchStrategy:\"merge\" patchMergeKey:\"type\" protobuf:\"bytes,1,rep,name=conditions\"`\n\n\n\t 463 \ // other fields\n\t}" 464 properties: 465 lastTransitionTime: 466 description: |- 467 lastTransitionTime is the last time the condition transitioned from one status to another. 468 This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. 469 format: date-time 470 type: string 471 message: 472 description: |- 473 message is a human readable message indicating details about the transition. 474 This may be an empty string. 475 maxLength: 32768 476 type: string 477 observedGeneration: 478 description: |- 479 observedGeneration represents the .metadata.generation that the condition was set based upon. 480 For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date 481 with respect to the current state of the instance. 482 format: int64 483 minimum: 0 484 type: integer 485 reason: 486 description: |- 487 reason contains a programmatic identifier indicating the reason for the condition's last transition. 488 Producers of specific condition types may define expected values and meanings for this field, 489 and whether the values are considered a guaranteed API. 490 The value should be a CamelCase string. 491 This field may not be empty. 492 maxLength: 1024 493 minLength: 1 494 pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ 495 type: string 496 status: 497 description: status of the condition, one of True, False, Unknown. 498 enum: 499 - "True" 500 - "False" 501 - Unknown 502 type: string 503 type: 504 description: |- 505 type of condition in CamelCase or in foo.example.com/CamelCase. 506 --- 507 Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be 508 useful (see .node.status.conditions), the ability to deconflict is important. 509 The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt) 510 maxLength: 316 511 pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ 512 type: string 513 required: 514 - lastTransitionTime 515 - message 516 - reason 517 - status 518 - type 519 type: object 520 type: array 521 x-kubernetes-list-map-keys: 522 - type 523 x-kubernetes-list-type: map 524 flavorsReservation: 525 description: |- 526 flavorsReservation are the reserved quotas, by flavor, currently in use by the 527 workloads assigned to this ClusterQueue. 528 items: 529 properties: 530 name: 531 description: name of the flavor. 532 type: string 533 resources: 534 description: resources lists the quota usage for the resources 535 in this flavor. 536 items: 537 properties: 538 borrowed: 539 anyOf: 540 - type: integer 541 - type: string 542 description: |- 543 Borrowed is quantity of quota that is borrowed from the cohort. In other 544 words, it's the used quota that is over the nominalQuota. 545 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ 546 x-kubernetes-int-or-string: true 547 name: 548 description: name of the resource 549 type: string 550 total: 551 anyOf: 552 - type: integer 553 - type: string 554 description: |- 555 total is the total quantity of used quota, including the amount borrowed 556 from the cohort. 557 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ 558 x-kubernetes-int-or-string: true 559 required: 560 - name 561 type: object 562 maxItems: 16 563 type: array 564 x-kubernetes-list-map-keys: 565 - name 566 x-kubernetes-list-type: map 567 required: 568 - name 569 - resources 570 type: object 571 maxItems: 16 572 type: array 573 x-kubernetes-list-map-keys: 574 - name 575 x-kubernetes-list-type: map 576 flavorsUsage: 577 description: |- 578 flavorsUsage are the used quotas, by flavor, currently in use by the 579 workloads admitted in this ClusterQueue. 580 items: 581 properties: 582 name: 583 description: name of the flavor. 584 type: string 585 resources: 586 description: resources lists the quota usage for the resources 587 in this flavor. 588 items: 589 properties: 590 borrowed: 591 anyOf: 592 - type: integer 593 - type: string 594 description: |- 595 Borrowed is quantity of quota that is borrowed from the cohort. In other 596 words, it's the used quota that is over the nominalQuota. 597 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ 598 x-kubernetes-int-or-string: true 599 name: 600 description: name of the resource 601 type: string 602 total: 603 anyOf: 604 - type: integer 605 - type: string 606 description: |- 607 total is the total quantity of used quota, including the amount borrowed 608 from the cohort. 609 pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ 610 x-kubernetes-int-or-string: true 611 required: 612 - name 613 type: object 614 maxItems: 16 615 type: array 616 x-kubernetes-list-map-keys: 617 - name 618 x-kubernetes-list-type: map 619 required: 620 - name 621 - resources 622 type: object 623 maxItems: 16 624 type: array 625 x-kubernetes-list-map-keys: 626 - name 627 x-kubernetes-list-type: map 628 pendingWorkloads: 629 description: |- 630 pendingWorkloads is the number of workloads currently waiting to be 631 admitted to this clusterQueue. 632 format: int32 633 type: integer 634 pendingWorkloadsStatus: 635 description: |- 636 PendingWorkloadsStatus contains the information exposed about the current 637 status of the pending workloads in the cluster queue. 638 properties: 639 clusterQueuePendingWorkload: 640 description: Head contains the list of top pending workloads. 641 items: 642 description: |- 643 ClusterQueuePendingWorkload contains the information identifying a pending workload 644 in the cluster queue. 645 properties: 646 name: 647 description: Name indicates the name of the pending workload. 648 type: string 649 namespace: 650 description: Namespace indicates the name of the pending 651 workload. 652 type: string 653 required: 654 - name 655 - namespace 656 type: object 657 type: array 658 x-kubernetes-list-type: atomic 659 lastChangeTime: 660 description: LastChangeTime indicates the time of the last change 661 of the structure. 662 format: date-time 663 type: string 664 required: 665 - lastChangeTime 666 type: object 667 reservingWorkloads: 668 description: |- 669 reservingWorkloads is the number of workloads currently reserving quota in this 670 clusterQueue. 671 format: int32 672 type: integer 673 type: object 674 type: object 675 served: true 676 storage: true 677 subresources: 678 status: {}