github.com/1aal/kubeblocks@v0.0.0-20231107070852-e1c03e598921/docs/user_docs/kubeblocks-for-mysql/cluster-management/scale-for-apecloud-mysql.md (about)

     1  ---
     2  title: Scale for a MySQL cluster
     3  description: How to scale a MySQL cluster, horizontal scaling, vertical scaling
     4  keywords: [mysql, horizontal scaling, vertical scaling]
     5  sidebar_position: 2
     6  sidebar_label: Scale
     7  ---
     8  
     9  # Scale for an ApeCloud MySQL cluster
    10  
    11  You can scale a MySQL cluster in two ways, vertical scaling and horizontal scaling.
    12  
    13  ## Vertical scaling
    14  
    15  You can vertically scale a cluster by changing resource requirements and limits (CPU and storage). For example, if you need to change the resource class from 1C2G to 2C4G, vertical scaling is what you need.
    16  
    17  :::note
    18  
    19  During the vertical scaling process, all pods restart in the order of learner -> follower -> leader and the leader pod may change after the restarting.
    20  
    21  :::
    22  
    23  ### Before you start
    24  
    25  Check whether the cluster status is `Running`. Otherwise, the following operations may fail.
    26  
    27  ```bash
    28  kbcli cluster list mysql-cluster
    29  >
    30  NAME                 NAMESPACE        CLUSTER-DEFINITION        VERSION                TERMINATION-POLICY        STATUS         CREATED-TIME
    31  mysql-cluster        default          apecloud-mysql            ac-mysql-8.0.30        Delete                    Running        Jan 29,2023 14:29 UTC+0800
    32  ```
    33  
    34  ### Steps
    35  
    36  1. Change configuration. There are 3 ways to apply vertical scaling.
    37  
    38     **Option 1.** (**Recommended**) Use kbcli
    39  
    40     Configure the parameters `--components`, `--memory`, and `--cpu` and run the command.
    41  
    42     ```bash
    43     kbcli cluster vscale mysql-cluster \
    44     --components="mysql" \
    45     --memory="4Gi" --cpu="2" \
    46     ```
    47  
    48     - `--components` describes the component name ready for vertical scaling.
    49     - `--memory` describes the requested and limited size of the component memory.
    50     - `--cpu` describes the requested and limited size of the component CPU.
    51  
    52     **Option 2.** Create an OpsRequest
    53    
    54     Apply an OpsRequest to the specified cluster. Configure the parameters according to your needs.
    55  
    56     ```bash
    57     kubectl apply -f - <<EOF
    58     apiVersion: apps.kubeblocks.io/v1alpha1
    59     kind: OpsRequest
    60     metadata:
    61       name: ops-vertical-scaling
    62     spec:
    63       clusterRef: mysql-cluster
    64       type: VerticalScaling 
    65       verticalScaling:
    66       - componentName: mysql
    67         requests:
    68           memory: "2Gi"
    69           cpu: "1000m"
    70         limits:
    71           memory: "4Gi"
    72           cpu: "2000m"
    73     EOF
    74     ```
    75    
    76     **Option 3.** Change the YAML file of the cluster
    77  
    78     Change the configuration of `spec.componentSpecs.resources` in the YAML file. `spec.componentSpecs.resources` controls the requirement and limit of resources and changing them triggers a vertical scaling.
    79  
    80     ***Example***
    81  
    82     ```yaml
    83     apiVersion: apps.kubeblocks.io/v1alpha1
    84     kind: Cluster
    85     metadata:
    86       name: mysql-cluster
    87       namespace: default
    88     spec:
    89       clusterDefinitionRef: apecloud-mysql
    90       clusterVersionRef: ac-mysql-8.0.30
    91       componentSpecs:
    92       - name: mysql
    93         componentDefRef: mysql
    94         replicas: 1
    95         resources: # Change the values of resources.
    96           requests:
    97             memory: "2Gi"
    98             cpu: "1000m"
    99           limits:
   100             memory: "4Gi"
   101             cpu: "2000m"
   102         volumeClaimTemplates:
   103         - name: data
   104           spec:
   105             accessModes:
   106               - ReadWriteOnce
   107             resources:
   108               requests:
   109                 storage: 1Gi
   110       terminationPolicy: Halt
   111     ```
   112  
   113  2. Check the cluste status to validate the vertical scaling.
   114  
   115      ```bash
   116      kbcli cluster list mysql-cluster
   117      >
   118      NAME                 NAMESPACE        CLUSTER-DEFINITION        VERSION                TERMINATION-POLICY        STATUS                 CREATED-TIME
   119      mysql-cluster        default          apecloud-mysql            ac-mysql-8.0.30        Delete                    VerticalScaling        Jan 29,2023 14:29 UTC+0800
   120      ```
   121  
   122     - STATUS=VerticalScaling: it means the vertical scaling is in progress.
   123     - STATUS=Running: it means the vertical scaling operation has been applied.
   124     - STATUS=Abnormal: it means the vertical scaling is abnormal. The reason may be that the number of the normal instances is less than that of the total instance or the leader instance is running properly while others are abnormal.
   125       > To solve the problem, you can manually check whether this error is caused by insufficient resources. Then if AutoScaling is supported by the Kubernetes cluster, the system recovers when there are enough resources. Otherwise, you can create enough resources and troubleshoot with `kubectl describe` command.
   126  
   127  :::note
   128  
   129  Vertical scaling does not synchronize parameters related to CPU and memory and it is required to manually call the opsRequest of configuration to change parameters accordingly. Refer to [Configuration](./../configuration/configuration.md) for instructions.
   130  
   131  :::
   132  
   133  3. Check whether the corresponding resources change.
   134  
   135      ```bash
   136      kbcli cluster describe mysql-cluster
   137      ```
   138  
   139  ## Horizontal scaling
   140  
   141  Horizontal scaling changes the amount of pods. For example, you can apply horizontal scaling to scale pods up from three to five. The scaling process includes the backup and restore of data.
   142  
   143  ### Before you start
   144  
   145  Check whether the cluster STATUS is `Running`. Otherwise, the following operations may fail.
   146  
   147  ```bash
   148  kbcli cluster list mysql-cluster
   149  >
   150  NAME                 NAMESPACE        CLUSTER-DEFINITION        VERSION                TERMINATION-POLICY        STATUS         CREATED-TIME
   151  mysql-cluster        default          apecloud-mysql            ac-mysql-8.0.30        Delete                    Running        Jan 29,2023 14:29 UTC+0800
   152  ```
   153  
   154  ### Steps
   155  
   156  1. Change configuration. There are 3 ways to apply horizontal scaling.
   157  
   158     **Option 1.** (**Recommended**) Use kbcli
   159  
   160     Configure the parameters `--components` and `--replicas`, and run the command.
   161  
   162     ```bash
   163     kbcli cluster hscale mysql-cluster \
   164     --components="mysql" --replicas=3
   165     ```
   166  
   167     - `--components` describes the component name ready for horizontal scaling.
   168     - `--replicas` describes the replicas with the specified components.
   169  
   170     **Option 2.** Create an OpsRequest
   171  
   172     Apply an OpsRequest to a specified cluster. Configure the parameters according to your needs.
   173  
   174     ```bash
   175     kubectl apply -f - <<EOF
   176     apiVersion: apps.kubeblocks.io/v1alpha1
   177     kind: OpsRequest
   178     metadata:
   179       name: ops-horizontal-scaling
   180     spec:
   181       clusterRef: mysql-cluster
   182       type: HorizontalScaling
   183       horizontalScaling:
   184       - componentName: mysql
   185         replicas: 3
   186     EOF
   187     ```
   188  
   189     **Option 3.** Change the YAML file of the cluster
   190  
   191     Change the configuration of `spec.componentSpecs.replicas` in the YAML file. `spec.componentSpecs.replicas` stands for the pod amount and changing this value triggers a horizontal scaling of a cluster.
   192  
   193     ***Example***
   194  
   195     ```yaml
   196     apiVersion: apps.kubeblocks.io/v1alpha1
   197     kind: Cluster
   198     metadata:
   199      apiVersion: apps.kubeblocks.io/v1alpha1
   200     kind: Cluster
   201     metadata:
   202       name: mysql-cluster
   203       namespace: default
   204     spec:
   205       clusterDefinitionRef: apecloud-mysql
   206       clusterVersionRef: ac-mysql-8.0.30
   207       componentSpecs:
   208       - name: mysql
   209         componentDefRef: mysql
   210         replicas: 1 # Change the pod amount.
   211         volumeClaimTemplates:
   212         - name: data
   213           spec:
   214             accessModes:
   215               - ReadWriteOnce
   216             resources:
   217               requests:
   218                 storage: 1Gi
   219      terminationPolicy: Halt
   220     ```
   221  
   222  2. Validate the horizontal scaling operation.
   223  
   224     Check the cluster STATUS to identify the horizontal scaling status.
   225  
   226     ```bash
   227     kbcli cluster list mysql-cluster
   228     ```
   229  
   230     - STATUS=HorizontalScaling: it means horizontal scaling is in progress.
   231     - STATUS=Running: it means horizontal scaling has been applied.
   232  
   233  3. Check whether the corresponding resources change.
   234  
   235      ```bash
   236      kbcli cluster describe mysql-cluster
   237      ```
   238  
   239  ### Handle the snapshot exception
   240  
   241  If `STATUS=ConditionsError` occurs during the horizontal scaling process, you can find the cause from `cluster.status.condition.message` for troubleshooting.
   242  In the example below, a snapshot exception occurs.
   243  
   244  ```bash
   245  Status:
   246    conditions: 
   247    - lastTransitionTime: "2023-02-08T04:20:26Z"
   248      message: VolumeSnapshot/mysql-cluster-mysql-scaling-dbqgp: Failed to set default snapshot
   249        class with error cannot find default snapshot class
   250      reason: ApplyResourcesFailed
   251      status: "False"
   252      type: ApplyResources
   253  ```
   254  
   255  ***Reason***
   256  
   257  This exception occurs because the `VolumeSnapshotClass` is not configured. This exception can be fixed after configuring `VolumeSnapshotClass`, but the horizontal scaling cannot continue to run. It is because the wrong backup (volumesnapshot is generated by backup) and volumesnapshot generated before still exist. Delete these two wrong resources and then KubeBlocks re-generates new resources.
   258  
   259  ***Steps:***
   260  
   261  1. Configure the VolumeSnapshotClass by running the command below.
   262  
   263     ```bash
   264     kubectl create -f - <<EOF
   265     apiVersion: snapshot.storage.k8s.io/v1
   266     kind: VolumeSnapshotClass
   267     metadata:
   268       name: csi-aws-vsc
   269       annotations:
   270         snapshot.storage.kubernetes.io/is-default-class: "true"
   271     driver: ebs.csi.aws.com
   272     deletionPolicy: Delete
   273     EOF
   274     ```
   275  
   276  2. Delete the wrong backup (volumesnapshot is generated by backup) and volumesnapshot resources.
   277  
   278     ```bash
   279     kubectl delete backup -l app.kubernetes.io/instance=mysql-cluster
   280     
   281     kubectl delete volumesnapshot -l app.kubernetes.io/instance=mysql-cluster
   282     ```
   283  
   284  ***Result***
   285  
   286  The horizontal scaling continues after backup and volumesnapshot are deleted and the cluster restores to running status.