github.com/1aal/kubeblocks@v0.0.0-20231107070852-e1c03e598921/docs/user_docs/kubeblocks-for-kafka/cluster-management/scale.md (about)

     1  ---
     2  title: Scale for a Kafka cluster
     3  description: How to scale a Kafka cluster, horizontal scaling, vertical scaling
     4  keywords: [mysql, horizontal scaling, vertical scaling]
     5  sidebar_position: 3
     6  sidebar_label: Scale
     7  ---
     8  
     9  # Scale a Kafka cluster
    10  
    11  You can scale a Kafka cluster in two ways, vertical scaling and horizontal scaling.
    12  
    13  ## Vertical scaling
    14  
    15  You can vertically scale a cluster by changing resource requirements and limits (CPU and storage). For example, if you need to change the resource class from 1C2G to 2C4G, vertical scaling is what you need.
    16  
    17  :::note
    18  
    19  During the vertical scaling process, all pods restart in the order of learner -> follower -> leader and the leader pod may change after the restarting.
    20  
    21  :::
    22  
    23  ### Before you start
    24  
    25  Check whether the cluster status is `Running`. Otherwise, the following operations may fail.
    26  
    27  ```bash
    28  kbcli cluster list
    29  >
    30  NAME    NAMESPACE   CLUSTER-DEFINITION   VERSION       TERMINATION-POLICY   STATUS    CREATED-TIME                 
    31  ivy85   default     kafka                kafka-3.3.2   Delete               Running   Jul 19,2023 18:01 UTC+0800   
    32  ```
    33  
    34  ### Steps
    35  
    36  1. Change configuration. There are 3 ways to apply vertical scaling.
    37  
    38     **Option 1.** (**Recommended**) Use kbcli
    39  
    40     Configure the parameters `--components`, `--memory`, and `--cpu` and run the command.
    41  
    42     ```bash
    43      kbcli cluster vscale ivy85 --components="broker" --memory="4Gi" --cpu="2" 
    44     ```
    45  
    46     - `--components` value can be `broker` or `controller`.
    47       - broker: all nodes in the combined mode, or all the broker node in the separated node.
    48       - controller: all the corresponding nodes in the separated mode.
    49     - `--memory` describes the requested and limited size of the component memory.
    50     - `--cpu` describes the requested and limited size of the component CPU.
    51  
    52     **Option 2.** Create an OpsRequest
    53    
    54     Apply an OpsRequest to the specified cluster. Configure the parameters according to your needs.
    55  
    56     ```bash
    57     kubectl apply -f - <<EOF
    58     apiVersion: apps.kubeblocks.io/v1alpha1
    59     kind: OpsRequest
    60     metadata:
    61       name: ops-vertical-scaling
    62     spec:
    63       clusterRef: ivy85
    64       type: VerticalScaling 
    65       verticalScaling:
    66       - componentName: broker
    67         requests:
    68           memory: "2Gi"
    69           cpu: "1000m"
    70         limits:
    71           memory: "4Gi"
    72           cpu: "2000m"
    73     EOF
    74     ```
    75    
    76     **Option 3.** Change the YAML file of the cluster
    77  
    78     Change the configuration of `spec.componentSpecs.resources` in the YAML file. `spec.componentSpecs.resources` controls the requirement and limit of resources and changing them triggers a vertical scaling.
    79  
    80     ***Example***
    81  
    82     ```yaml
    83     apiVersion: apps.kubeblocks.io/v1alpha1
    84     kind: Cluster
    85     metadata:
    86       name: ivy85
    87       namespace: default
    88     spec:
    89       clusterDefinitionRef: kafka
    90       clusterVersionRef: kafka-3.3.2
    91       componentSpecs:
    92       - name: broker
    93         componentDefRef: broker
    94         replicas: 1
    95         resources: # Change the values of resources.
    96           requests:
    97             memory: "2Gi"
    98             cpu: "1000m"
    99           limits:
   100             memory: "4Gi"
   101             cpu: "2000m"
   102         volumeClaimTemplates:
   103         - name: data
   104           spec:
   105             accessModes:
   106               - ReadWriteOnce
   107             resources:
   108               requests:
   109                 storage: 1Gi
   110       terminationPolicy: Halt
   111     ```
   112    
   113  2. Check the cluste status to validate the vertical scaling.
   114  
   115      ```bash
   116      kbcli cluster list mysql-cluster
   117      >
   118      NAME                 NAMESPACE        CLUSTER-DEFINITION        VERSION                TERMINATION-POLICY        STATUS                 CREATED-TIME
   119      ivy85                 default          kafka                kafka-3.3.2            Delete                    VerticalScaling        Jan 29,2023 14:29 UTC+0800
   120      ```
   121  
   122     - STATUS=VerticalScaling: it means the vertical scaling is in progress.
   123     - STATUS=Running: it means the vertical scaling operation has been applied.
   124     - STATUS=Abnormal: it means the vertical scaling is abnormal. The reason may be that the number of the normal instances is less than that of the total instance or the leader instance is running properly while others are abnormal.
   125       > To solve the problem, you can manually check whether this error is caused by insufficient resources. Then if AutoScaling is supported by the Kubernetes cluster, the system recovers when there are enough resources. Otherwise, you can create enough resources and troubleshoot with `kubectl describe` command.
   126  
   127  :::note
   128  
   129  Vertical scaling does not synchronize parameters related to CPU and memory and it is required to manually call the opsRequest of configuration to change parameters accordingly. Refer to [Configuration](./../configuration/configuration.md) for instructions.
   130  
   131  :::
   132  
   133  3. Check whether the corresponding resources change.
   134  
   135      ```bash
   136      kbcli cluster describe ivy85
   137      ```
   138  
   139  ## Horizontal scaling
   140  
   141  Horizontal scaling changes the amount of pods. For example, you can apply horizontal scaling to scale pods up from three to five. The scaling process includes the backup and restoration of data.
   142  
   143  ### Before you start
   144  
   145  - Check whether the cluster STATUS is `Running`. Otherwise, the following operations may fail.
   146  - You are not recommended to perform horizontal scaling on controller node, including the controller node both in combined mode and separated node.
   147  - When scaling in horizontally, you must know the topic partition storage, if the topic has only one replication, data loss may caused when you scale in broker.
   148  
   149   ```bash
   150  kbcli cluster list
   151  >
   152  NAME    NAMESPACE   CLUSTER-DEFINITION   VERSION       TERMINATION-POLICY   STATUS    CREATED-TIME                 
   153  ivy85   default     kafka                kafka-3.3.2   Delete               Running   Jul 19,2023 18:01 UTC+0800   
   154  ```
   155  
   156  ### Steps
   157  
   158  1. Change configuration. There are 3 ways to apply horizontal scaling.
   159  
   160     **Option 1.** (**Recommended**) Use kbcli
   161  
   162     Configure the parameters `--components` and `--replicas`, and run the command.
   163  
   164     ```bash
   165     kbcli cluster hscale mysql-cluster \
   166     --components="broker" --replicas=3
   167     ```
   168  
   169   
   170     - `--components` value can be `broker` or `controller`.
   171       - broker: all nodes in the combined mode, or all the broker node in the separated node.
   172       - controller: all the corresponding nodes in the separated mode.
   173     - `--memory` describes the requested and limited size of the component memory.
   174     - `--cpu` describes the requested and limited size of the component CPU.
   175  
   176     **Option 2.** Create an OpsRequest
   177  
   178     Apply an OpsRequest to a specified cluster. Configure the parameters according to your needs.
   179  
   180     ```bash
   181     kubectl apply -f - <<EOF
   182     apiVersion: apps.kubeblocks.io/v1alpha1
   183     kind: OpsRequest
   184     metadata:
   185       name: ops-horizontal-scaling
   186     spec:
   187       clusterRef: ivy85
   188       type: HorizontalScaling
   189       horizontalScaling:
   190       - componentName: broker
   191         replicas: 3
   192     EOF
   193     ```
   194  
   195     **Option 3.** Change the YAML file of the cluster
   196  
   197     Change the configuration of `spec.componentSpecs.replicas` in the YAML file. `spec.componentSpecs.replicas` stands for the pod amount and changing this value triggers a horizontal scaling of a cluster.
   198  
   199     ***Example***
   200  
   201     ```yaml
   202     apiVersion: apps.kubeblocks.io/v1alpha1
   203     kind: Cluster
   204     metadata:
   205      apiVersion: apps.kubeblocks.io/v1alpha1
   206     kind: Cluster
   207     metadata:
   208       name: ivy85
   209       namespace: default
   210     spec:
   211       clusterDefinitionRef: kafka
   212       clusterVersionRef: kafka-3.3.2 
   213       componentSpecs:
   214       - name: broker
   215         componentDefRef: broker
   216         replicas: 1 # Change the pod amount.
   217         volumeClaimTemplates:
   218         - name: data
   219           spec:
   220             accessModes:
   221               - ReadWriteOnce
   222             resources:
   223               requests:
   224                 storage: 1Gi
   225      terminationPolicy: Halt
   226     ```
   227  
   228  2. Validate the horizontal scaling operation.
   229  
   230     Check the cluster STATUS to identify the horizontal scaling status.
   231  
   232     ```bash
   233     kbcli cluster list ivy85
   234     ```
   235  
   236     - STATUS=HorizontalScaling: it means horizontal scaling is in progress.
   237     - STATUS=Running: it means horizontal scaling has been applied.
   238  
   239  3. Check whether the corresponding resources change.
   240  
   241      ```bash
   242      kbcli cluster describe ivy85
   243      ```
   244  
   245  ### Handle the snapshot exception
   246  
   247  If `STATUS=ConditionsError` occurs during the horizontal scaling process, you can find the cause from `cluster.status.condition.message` for troubleshooting.
   248  In the example below, a snapshot exception occurs.
   249  
   250  ```bash
   251  Status:
   252    conditions: 
   253    - lastTransitionTime: "2023-02-08T04:20:26Z"
   254      message: VolumeSnapshot/ivy85-kafka-scaling-dbqgp: Failed to set default snapshot
   255        class with error cannot find default snapshot class
   256      reason: ApplyResourcesFailed
   257      status: "False"
   258      type: ApplyResources
   259  ```
   260  
   261  ***Reason***
   262  
   263  This exception occurs because the `VolumeSnapshotClass` is not configured. This exception can be fixed after configuring `VolumeSnapshotClass`, but the horizontal scaling cannot continue to run. It is because the wrong backup (volumesnapshot is generated by backup) and volumesnapshot generated before still exist. Delete these two wrong resources and then KubeBlocks re-generates new resources.
   264  
   265  ***Steps:***
   266  
   267  1. Configure the VolumeSnapshotClass by running the command below.
   268  
   269     ```bash
   270     kubectl create -f - <<EOF
   271     apiVersion: snapshot.storage.k8s.io/v1
   272     kind: VolumeSnapshotClass
   273     metadata:
   274       name: csi-aws-vsc
   275       annotations:
   276         snapshot.storage.kubernetes.io/is-default-class: "true"
   277     driver: ebs.csi.aws.com
   278     deletionPolicy: Delete
   279     EOF
   280     ```
   281  
   282  2. Delete the wrong backup (volumesnapshot is generated by backup) and volumesnapshot resources.
   283  
   284     ```bash
   285     kubectl delete backup -l app.kubernetes.io/instance=ivy85
   286     
   287     kubectl delete volumesnapshot -l app.kubernetes.io/instance=ivy85
   288  
   289     ```
   290  
   291  ***Result***
   292  
   293  The horizontal scaling continues after backup and volumesnapshot are deleted and the cluster restores to running status.