github.com/pingcap/tiflow@v0.0.0-20240520035814-5bf52d54e205/dm/tests/README.md (about)

     1  
     2  ## Preparations
     3  
     4  1. The following executables must be copied or generated or linked into these locations.
     5  
     6      * `bin/tidb-server` can be downloaded from [tidb-master-linux-amd64](https://download.pingcap.org/tidb-master-linux-amd64.tar.gz) or installed by [tiup](https://github.com/pingcap/tiup), you can use the command `find ~/.tiup -name tidb-server` to locate `tidb-server` binary file and copy it
     7      * `bin/sync_diff_inspector` # can be downloaded from [tidb-enterprise-tools-latest-linux-amd64](http://download.pingcap.org/tidb-enterprise-tools-latest-linux-amd64.tar.gz) or build from [source code](https://github.com/pingcap/tidb-tools)
     8      * `bin/minio` can be build from (https://github.com/minio/minio)
     9      * `bin/dm-master.test` # generated by `make dm_integration_test_build`
    10      * `bin/dm-worker.test` # generated by `make dm_integration_test_build`
    11      * `bin/dm-master.test.current` # generated from `bin/dm-master.test` by `make compatibility_test`
    12      * `bin/dm-worker.test.current` # generated from `bin/dm-worker.test` by `make compatibility_test`
    13      * `bin/dm-master.test.previous` # generated from `bin/dm-master.test` by `make compatibility_test`
    14      * `bin/dm-worker.test.previous` # generated from `bin/dm-worker.test` by `make compatibility_test`
    15      * [gh-ost](https://github.com/github/gh-ost) # must be added to path, or you can `export GHOST_BINARY=/path/to/gh-ost-binary`
    16      * [pt-online-schema-change](https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html) # must be added to path, or you can `export PTOSC_BINARY=/path/to/pt-osc-binary`
    17      * [GNU sed](https://www.gnu.org/software/sed/) # must be added to path, BSD sed user should be attention
    18      * [GNU awk](https://www.gnu.org/software/gawk/) # must be added to path
    19  
    20  2. The following programs must be installed:
    21  
    22      * `mysql` (the CLI client, currently [not supported for mysql client 8.0](https://github.com/pingcap/tidb/issues/14021))
    23      * `python2.7` or `python3.x`
    24  
    25  3. The user executing the tests must have permission to create the folder `/tmp/dm_test`. All test artifacts will be written into this folder.
    26  
    27  ## Running
    28  
    29  ### Unit Test
    30  
    31  1. Run `make unit_test` to run unit test
    32  
    33  ### Integration Test
    34  
    35  1. Run `make dm_integration_test_build` to generate DM related binary for integration test
    36  
    37  2. Setup two MySQL servers (the first one: 5.6 ~ 5.7; the second one: 8.0.21, suggest you are same as [CI](https://github.com/PingCAP-QE/ci/blob/main/jenkins/pipelines/ci/dm/dm_ghpr_new_test.groovy#L164-L172)) with [binlog enabled first](https://dev.mysql.com/doc/refman/5.7/en/replication-howto-masterbaseconfig.html) and [set `GTID_MODE=ON`](https://dev.mysql.com/doc/refman/5.7/en/replication-mode-change-online-enable-gtids.html), You need set the mysql port and root password according to the following table.
    38  
    39      | MySQL | Host | Port| PASSWORD |
    40      | :------------ | :---------- | :------ | :---- |
    41      | MySQL1 | `127.0.0.1` | `3306` | `123456` |
    42      | MySQL2 | `127.0.0.1` | `3307` | `123456` |
    43  
    44      > For the second MySQL server (with 8.0 version), you should setup it with `--default-authentication-plugin=mysql_native_password`.
    45  
    46  3. Run `make integration_test` to execute the integration tests. This command will
    47  
    48      1. Check that all required executables exist.
    49      2. Execute `tests/run.sh`
    50  
    51      > If want to run one integration test case only, just pass the CASE parameter, such as `make integration_test CASE=sharding`.
    52  
    53      > There exists some environment variables that you can set by yourself, including `RESET_MASTER`,`ONLINE_DDL_ENABLE`. If `RESET_MASTER` is not set or set to true, `RESET MASTER` will be executed at upstream MySQL before each case.
    54  
    55      > The online DDL test using pt-osc doesn't work if the upstream MySQL has different connect port and bind port (often caused by port forwarding via NAT). In this case, you must specify the real IP and port of MySQL. Otherwise you can skip online DDL test by `export ONLINE_DDL_ENABLE=false`.
    56  
    57  4. After executing the tests, run `make coverage` to get a coverage report at `/tmp/dm_test/all_cov.html`.
    58  
    59  ### Compatibility Test
    60  
    61  1. Run Compatibility test
    62  
    63      Run `make compatibility_test` will run compatibility test.
    64  
    65  2. Setup two MySQL servers with [binlog enabled first](https://dev.mysql.com/doc/refman/5.7/en/replication-howto-masterbaseconfig.html) and [set `GTID_MODE=ON`](https://dev.mysql.com/doc/refman/5.7/en/replication-mode-change-online-enable-gtids.html), export proper environment variable. Variables and their default values are showed in following table.
    66  
    67      | MySQL | Host | Port| PASSWORD |
    68      | :------------ | :---------- | :------ | :---- |
    69      | MySQL1 | `MYSQL_HOST1` / `127.0.0.1` | `MYSQL_PORT1` / `3306` | `MYSQL_PASSWORD1` / `123456` |
    70      | MySQL2 | `MYSQL_HOST2` / `127.0.0.1` | `MYSQL_PORT2` / `3307` | `MYSQL_PASSWORD2` / `123456` |
    71  
    72  3. Run other test with different version's dm-master and dm-worker
    73  
    74     Run `make compatibility_test CASE={test_case}`, for example, `make compatibility_test CASE=http_apis` will run test `http_apis` with previous dm-master and current dm-worker, and also current dm-master and previous dm-worker. The `test_case` can be `all`, and will run all the test cases.
    75  
    76  ## Writing new tests
    77  
    78  1. New integration tests can be written as shell scripts in `tests/TEST_NAME/run.sh`. The script should exit with a nonzero error code on failure.
    79  2. Add TEST_NAME to existing group in [run_group.sh](./run_group.sh), or add a new group for it.
    80  3. If you add a new group, the name of the new group must be added to CI.
    81     * [dm-integration-test](https://github.com/PingCAP-QE/ci/blob/main/pipelines/pingcap/tiflow/latest/pull_dm_integration_test.groovy)
    82  
    83  > Note the integration test runs in parallel, and new added test case is not listed in CI, so remember to add the `TEST_NAME` of new added test case to file `others_integration_2.txt` in a newline.
    84  
    85  Several convenient commands are provided:
    86  
    87  * `run_dm_master <WORKDIR> <PORT> <CONFIG>` — Starts `dm-master` using config provided, on given port, running in workdir.
    88  * `run_dm_worker <WORKDIR> <PORT> <CONFIG>` — Starts `dm-worker` using config provided, on given port, running in workdir.
    89  * `run_dm_ctl <WORKDIR> <MASTER_ADDR> <DMCTL_COMMAND>` - Runs `dmctl` with given command in non-interactive mode.
    90  * `run_sql <SQL> <PORT> <PASSWORD>` — Executes an SQL query in database based on port provided
    91  * `run_sql_file <path_to_SQL_file> <HOST> <PORT> <PASSWORD>` — Executes all SQLs in given file to the database on port provided
    92  * `run_sql_file_online_ddl <path_to_SQL_file> <HOST> <PORT> <PASSWORD> <DB> <ONLINE DDL TOOL>` — Executes all SQLs in given file, will auto switch DDL to online DDL command.
    93  * `check_contains <TEXT>` — Checks if the previous `run_sql`/`run_sql_file` result contains the given text (in `-E` format)
    94  * `check_sync_diff <WORKDIR> <CONFIG>` - Runs `sync_diff_inspector` to check diff between databases, using config file provided
    95  * `check_rpc_alive <CHECK_TOOL> <ADDR>` - Wrapper to check a rpc address is available, at most 10 times.
    96  * `check_port_alive <PORT>` - Wrapper to check a port is alive, at most 20 times.
    97  * `check_port <HOST> <PORT>` - Checks a host:port is alive.
    98  * `wait_process_exit <process_name>` - Wait for one or more processes to exit by given process name.
    99  * `check_metric <PORT> <METRIC_NAME> <RETRY_COUNT> <LOWER BOUND> <UPPER BOUND>` - check metric value from prometheus.
   100  * `truncate_trace_events <PORT>` - truncate trace server events records.
   101