github.com/1aal/kubeblocks@v0.0.0-20231107070852-e1c03e598921/docs/user_docs/kubeblocks-for-mongodb/migration/migration-kbcli.md (about)

     1  ---
     2  title: Migrate data in MongoDB to KubeBlocks
     3  description: How to migrate data in MongDB v5.* to KubeBlocks by kbcli migration
     4  keywords: [mongodb, migration, kbcli migration, migrate data in MongoDB to KubeBlocks]
     5  sidebar_position: 1
     6  sidebar_label: Migration by kbcli
     7  ---
     8  
     9  # Migrate data in MongoDB to KubeBlocks
    10  
    11  ## Before you start
    12  
    13  ### Enable kbcli migration
    14  
    15  1. Install KubeBlocks: You can install KubeBlocks by [kbcli](./../../installation/install-with-kbcli/install-kubeblocks-with-kbcli.md) or by [Helm](./../../installation/install-with-helm/install-kubeblocks-with-helm.md).
    16  2. [Enable the migration add-on](./../../overview/supported-addons.md).
    17  
    18     ```bash
    19     kbcli addon list
    20  
    21     kbcli addon enable migration
    22     ```
    23  
    24  ### Configure the source
    25  
    26  Modify the configuration of the source to support CDC.
    27  
    28  * Replication cluster instance: no scaling is required. You need to provide the address of the primary for migration.
    29  * Standalone instance: it is required to scale a Standalone to a Replica Set with one node to use CDC. Refer to [Convert a Standalone to a Replica Set](https://www.mongodb.com/docs/manual/tutorial/convert-standalone-to-replica-set/) for details.
    30  
    31  ### Check the account permission
    32  
    33  Make sure both the source and sink account meet the following permissions.
    34  
    35  * The source account
    36    The read permission of the repository to be migrated, admin, and local.
    37  * The sink account
    38    The read and write permission of the repository to be migrated and the read permission of admin and local.
    39  
    40  ### Prepare data sampling
    41  
    42  It is recommended to prepare data sampling for verification after the migration to ensure correctness.
    43  
    44  ## Migrate data
    45  
    46  ### Steps
    47  
    48  1. Create a migration task.
    49  
    50     ```bash
    51     kbcli migration create mytask --template apecloud-mongo2mongo \
    52     --source user:123456@127.0.0.1:5432/db_test \
    53     --sink user:123456@127.0.0.2:5432/db_test \
    54     --migration-object '"public.table_test_1","public.table_test_2"'
    55     ```
    56  
    57     :paperclip: Table 1. Options explanation
    58  
    59     | Option     | Descriprion |
    60     | :--------- | :---------- |
    61     | mystask    | The name of the migration task. You can customize it. |
    62     | `--template` | It specifies the migration template. `--template apecloud-mongo2mongo` means that this migration task uses the template of migrating from MongoDB to MongoDB created by KubeBlocks. Run `kbcli migration templates` to view all available templates and the supported database information.   |
    63     | `--source`  | It specifies the source. `user:123456@127.0.0.1:5432/db_test` in the above example follows the format `${user_name}:${password}@${database connection url}/${database}`. For this guide, the connection URL uses the public network address. |
    64     | `--sink`     | It specifies the destination. `user:123456@127.0.0.2:5432/db_test` in the above example follows the format `${user_name}:${password}@${database connection url}/${database}`. For this guide, the connection URL uses the service address inside the Kubernetes cluster. |
    65     | `--migration-object`  | It specifies the migration object. The above example describes data in "public.table_test_1" and "public.table_test_2", including structure data, stock data, and incremental data generated during running migration task, will be migrated to the sink.    |
    66  
    67  2. (Optional) Specify migration steps by the flag `--steps`.
    68  
    69     The default steps follow the order precheck -> structure initialization -> data initialization -> incremental migration. You can use `--steps` to specify migration steps. For example, perform tasks in the order of precheck -> structure initialization -> data initialization -> incremental migration.
    70  
    71     ```bash
    72     kbcli migration create mytask --template apecloud-mysql2mysql \
    73     --source user:123456@127.0.0.1:5432/db_test \
    74     --sink user:123456@127.0.0.2:5432/db_test \
    75     --migration-object '"public.table_test_1","public.table_test_2"'
    76     --steps precheck=true,init-struct=false,init-data=true,cdc=true
    77     ```
    78  
    79  3. View the task status.
    80  
    81     ```bash
    82     # View the migration task list
    83     kbcli migration list
    84  
    85     # View the details of a specified task
    86     kbcli migration describe ${migration-task-name}
    87     ```
    88  
    89     Pay attention to Initialization, CDC, and CDC Metrics.
    90  
    91     * Initialization
    92       * Precheck: If the status shows `Failed`, it means the initialization precheck does not pass. Troubleshoot the initialization by [the following examples in troubleshooting](#troubleshooting).
    93       * Init-struct: Structure initialization. Idempotent processing logic is adopted. A failure occurs only when a severe problem occurs, such as failing to connect a database.
    94       * Init-data: Data initialization. If there is a large amount of stock data, it takes a long time to perform this step and you should pay attention to Status.
    95     * CDC: Incremental migration. Based on the timestamp recorded by the system before the init-data step, the system starts data migration following eventual consistency and performs capturing the source library WAL (Write Ahead Log) changes -> writing to the sink. Under normal circumstances, the CDC phase continues if the migration link is not actively terminated.
    96     * CDC Metrics: Incremental migration indicators. Currently, the indicators mainly provide the WAL LSN (Log Sequencer Number) of the source library and the corresponding timestamp (note that the timestamp shows the local time zone of the Pod Container runtime) when the CDC process has completed "capturing -> writing" process.
    97  
    98       :::note
    99  
   100       The CDC Metrics are updated every 10 minutes by the system, i.e. if there exists continuous data writing into the source, metrics.timestamp here delays 10 minutes compared with the current time.
   101  
   102       :::
   103  
   104  4. Validate the migration with the prepared data sampling.
   105  
   106  ### Troubleshooting
   107  
   108  If any step above fails, run the command below to troubleshoot the failure.
   109  
   110  ```bash
   111  # --step: Specify the step. Allowed values: precheck,init-struct,init-data,cdc
   112  kbcli migration logs ${migration-task-name} --step ${step-name}
   113  ```
   114  
   115  ## Switch applications
   116  
   117  ### Before you start
   118  
   119  * Make sure the KubeBlocks migration task runs normally.
   120  * To differentiate the dialogue information and to improve data security, it is recommended to create and authorize another account dedicated to data migration.
   121  * For safety concerns, it is necessary to stop the business write and switch the application during off-peak hours.
   122  * Before switching the application, it is recommended to prepare data sampling for verification after switching to ensure correctness.
   123  
   124  ### Steps
   125  
   126  1. Check the migration task status and ensure the task is performed normally.
   127     1. Describe the migration task details and all steps in Initialization are `Complete` and CDC is `Running`.
   128  
   129        ```bash
   130        kbcli migration describe ${migration-task-name}
   131        ```
   132  
   133     2. Under the prerequisite that there exists continuous write into the source, observe whether the timestamp is still in progress and whether there is almost no delay. For example,
   134  
   135        ```bash
   136        kbcli migration logs ${migration-task-name} --step cdc | grep current_position
   137        ```
   138  
   139        The results update every 10 seconds.
   140  
   141        ![Timestamp](../../../img/pgsql-migration-timestamp.png)
   142  2. Pause the business and stop new business data from being written into the source.
   143  3. View the migration status again and ensure the migration task runs normally, lasting at least one minute.
   144  
   145     Refer to the operations in step 1 and observe whether the link is normal and the timestamp meets the expectation.
   146  4. Use the sink to restore the business.
   147  5. Validate the switch with the prepared data sampling.
   148  
   149  ## Clean up the environment
   150  
   151  After the migration task is completed, you can terminate the migration task and function.
   152  
   153  ### Terminate the migration task
   154  
   155  Deleting the migration task does not affect the data in the source and sink.
   156  
   157  ```bash
   158  kbcli migration terminate ${migration-task-name}
   159  ```
   160  
   161  ### Terminate kbcli migration
   162  
   163  1. Check whether there are running migration tasks.
   164  
   165     ```bash
   166     kbcli migration list
   167     ```
   168  
   169  2. Disable the migration add-on.
   170  
   171     ```bash
   172     kbcli addon disable migration
   173     ```
   174  
   175  3. Delete the Kubernetes CRD (Custom Resource Definition) manually.
   176  
   177     ```bash
   178     kubectl delete crd migrationtasks.datamigration.apecloud.io migrationtemplates.datamigration.apecloud.io serialjobs.common.apecloud.io
   179     ```