vitess.io/vitess@v0.16.2/examples/compose/README.md (about) 1 # Local Vitess cluster using docker-compose 2 3 This directory has a docker-compose sample application. 4 To understand it better, you can run it. 5 6 First you will need to [install docker-compose](https://docs.docker.com/compose/install/). 7 8 ### Before you begin 9 You will need to create a docker-compose.yml file. There are 2 ways to do this. 10 1. Run `go run vtcompose/vtcompose.go --args`. [Instructions can be found here](#programatically-create-vitess-configuration-for-docker "Generate docker-compose") 11 2. Use the `docker-compose.beginners.yml` to generate your `docker-compose.yml` file. Run: 12 ``` 13 vitess/examples/compose$ cp docker-compose.beginners.yml docker-compose.yml 14 ``` 15 Create your .env file 16 ``` 17 vitess/examples/compose$ cp template.env .env 18 ``` 19 20 You can then proceed to the instructions under [`Start the Cluster`](#start-the-cluster "Start the cluster") section. 21 22 ### Programatically create Vitess configuration for Docker 23 To create a configuration to your specifications, run vtcompose. Creates corresponding docker-compose file, vschema files per keyspace, and loads schemas. 24 ``` 25 vitess/examples/compose$ go run vtcompose/vtcompose.go --args(optional) 26 ``` 27 28 Use `-h` or `--help` to get list of flags with descriptions. 29 30 Flags available: 31 * **baseDockerComposeFile** - Specifies starting docker-compose yaml file. 32 * **baseVschemaFile** - Specifies starting vschema json file. 33 * **topologyFlags** - Specifies Vitess topology flags config 34 * **webPort** - Specifies web port to be used. 35 * **gRpcPort** - Specifies gRPC port to be used. 36 * **mySqlPort** - Specifies mySql port to be used. 37 * **cell** - `Specifies Vitess cell name to be used. 38 * **keyspaceData** - List of `keyspace_name:num_of_shards:num_of_replica_tablets:schema_file_names:<optional>lookup_keyspace_name` separated by ';'. 39 * This is where you specify most of the data for the program to build your vSchema and docker-compose files. 40 * Examples you can run; 41 * Default 42 ``` 43 go run vtcompose/vtcompose.go 44 ``` 45 * Use `0` for `num_of_shards` to specify an unsharded keyspace 46 ``` 47 go run vtcompose/vtcompose.go -keyspaceData="test_keyspace:0:2:create_messages.sql" 48 ``` 49 * Multiple keyspaces with sharded test_keyspace 50 ``` 51 go run vtcompose/vtcompose.go -keyspaceData="test_keyspace:2:1:create_messages.sql,create_tokens.sql:lookup_keyspace;lookup_keyspace:1:1:create_tokens_token_lookup.sql,create_messages_message_lookup.sql" 52 ``` 53 * **externalDbData** - Specifies which databases/keyspaces are external and provides data along with it to connect to the external db. 54 List of `<external_db_name>,<DB_HOST>,<DB_PORT>,<DB_USER>,<DB_PASS>,<DB_CHARSET>` separated by ';'. 55 When using this, make sure to have the external_db_name/keyspace in the `keyspaceData` flag with no schema_file_names specified. 56 ``` 57 go run vtcompose/vtcompose.go -keyspaceData="commerce:0:2::" -externalDbData="commerce:external_db_host:3306:external_db_user:external_db_password:" 58 ``` 59 60 61 ### Start the cluster 62 To start Consul(which saves the topology config), vtctld, vtgate, vtorc and a few vttablets with MySQL running on them. 63 ``` 64 vitess/examples/compose$ docker-compose up -d 65 ``` 66 67 ### Check cluster status 68 Check the status of the cluster. 69 ``` 70 vitess/examples/compose$ docker-compose ps 71 Name Command State Ports 72 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 73 compose_consul1_1 docker-entrypoint.sh agent ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp, 74 0.0.0.0:8400->8400/tcp,:::8400->8400/tcp, 0.0.0.0:8500->8500/tcp,:::8500->8500/tcp, 75 0.0.0.0:8600->8600/tcp,:::8600->8600/tcp, 8600/udp 76 compose_consul2_1 docker-entrypoint.sh agent ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp, 8400/tcp, 8500/tcp, 8600/tcp, 77 8600/udp 78 compose_consul3_1 docker-entrypoint.sh agent ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp, 8400/tcp, 8500/tcp, 8600/tcp, 79 8600/udp 80 compose_external_db_host_1 docker-entrypoint.sh --ser ... Up (healthy) 0.0.0.0:59816->3306/tcp, 33060/tcp 81 compose_schemaload_1 sh -c /script/schemaload.sh Exit 0 82 compose_set_keyspace_durability_policy_1 sh -c /script/set_keyspace ... Exit 0 83 compose_vreplication_1 sh -c [ $EXTERNAL_DB -eq 1 ... Exit 0 84 compose_vtctld_1 sh -c /vt/bin/vtctld -top ... Up 0.0.0.0:59823->15999/tcp, 0.0.0.0:15000->8080/tcp,:::15000->8080/tcp 85 compose_vtgate_1 sh -c /vt/bin/vtgate -topo ... Up 0.0.0.0:15306->15306/tcp,:::15306->15306/tcp, 0.0.0.0:59830->15999/tcp, 86 0.0.0.0:15099->8080/tcp,:::15099->8080/tcp 87 compose_vtorc_1 sh -c /script/vtorc-up.sh Up (healthy) 0.0.0.0:13000->3000/tcp,:::13000->3000/tcp 88 compose_vttablet100_1 sh -c [ $EXTERNAL_DB -eq 1 ... Exit 0 89 compose_vttablet101_1 sh -c /script/vttablet-up. ... Up (healthy) 0.0.0.0:59824->15999/tcp, 0.0.0.0:59825->3306/tcp, 90 0.0.0.0:15101->8080/tcp,:::15101->8080/tcp 91 compose_vttablet102_1 sh -c /script/vttablet-up. ... Up (healthy) 0.0.0.0:59826->15999/tcp, 0.0.0.0:59827->3306/tcp, 92 0.0.0.0:15102->8080/tcp,:::15102->8080/tcp 93 compose_vttablet103_1 sh -c /script/vttablet-up. ... Up (healthy) 0.0.0.0:65284->15999/tcp, 0.0.0.0:65285->3306/tcp, 94 0.0.0.0:15103->8080/tcp,:::15103->8080/tcp 95 ``` 96 97 ### Check the status of the containers 98 You can check the logs of the containers (vtgate, vttablet101, vttablet102, vttablet103) at any time. 99 For example to check vtgate logs, run the following; 100 ``` 101 vitess/examples/compose$ docker-compose logs -f vtgate 102 ``` 103 104 ### Load the schema 105 This step is carried out by the **schemaload** container 106 107 We need to create a few tables into our new cluster. To do that, we can run the `ApplySchema` command. 108 ``` 109 vitess/examples/compose$ ./lvtctl.sh ApplySchema -- --sql "$(cat tables/create_messages.sql)" test_keyspace 110 ``` 111 112 ### Create Vschema 113 This step is carried out by the **schemaload** container 114 115 Create Vschema 116 ``` 117 vitess/examples/compose$ ./lvtctl.sh ApplyVschema -- --vschema '{"sharded": false }' test_keyspace 118 ``` 119 120 ### Connect to vgate and run queries 121 vtgate responds to the MySQL protocol, so we can connect to it using the default MySQL client command line. 122 ``` 123 vitess/examples/compose$ mysql --port=15306 --host=127.0.0.1 124 ``` 125 **Note that you may need to replace `127.0.0.1` with `docker ip` or `docker-machine ip`** 126 127 You can also use the `./lmysql.sh` helper script. 128 ``` 129 vitess/examples/compose$ ./lmysql.sh --port=15306 --host=<DOCKER_HOST_IP> 130 ``` 131 132 where `<DOCKER_HOST_IP>` is `docker-machine ip` or external docker host ip addr 133 134 ### Play around with vtctl commands 135 136 ``` 137 vitess/examples/compose$ ./lvtctl.sh Help 138 ``` 139 140 ## Exploring 141 142 - vtctld web ui: 143 http://localhost:15000 144 145 - vttablets web ui: 146 http://localhost:15101/debug/status 147 http://localhost:15102/debug/status 148 http://localhost:15103/debug/status 149 150 - vtgate web ui: 151 http://localhost:15099/debug/status 152 153 - vtorc ui: 154 http://localhost:13000 155 156 - Stream querylog 157 `curl -S localhost:15099/debug/querylog` 158 159 **Note that you may need to replace `localhost` with `docker ip` or `docker-machine ip`** 160 161 ## Troubleshooting 162 If the cluster gets in a bad state, you most likely will have to stop and kill the containers. Note: you will lose all the data. 163 ``` 164 vitess/examples/compose$ docker-compose kill 165 vitess/examples/compose$ docker-compose rm 166 ``` 167 168 ## Advanced Usage 169 170 ### External mysql instance 171 The compose example has the capability to run against an external mysql instance. Kindly take care to secure your connection to the database. 172 To start vitess against unsharded external mysql, change the following variables in your .env file to match your external database; 173 ``` 174 KEYSPACE=external_db_name 175 DB=external_db_name 176 EXTERNAL_DB=1 177 DB_HOST=external_db_host 178 DB_PORT=external_db_port 179 DB_USER=external_db_user 180 DB_PASS=external_db_password 181 DB_CHARSET=CHARACTER SET utf8 COLLATE utf8_general_ci 182 ``` 183 184 **Important** Ensure you have log bin enabled on your external database. 185 Refer to the .cnf files [here](./external_db/mysql) 186 187 ### Start the cluster 188 To simulate running vitess against an external cluster, simply change 189 `EXTERNAL_DB=1` when you copy your `template.env` then run the command below 190 ``` 191 vitess/examples/compose$ docker-compose up -d 192 ``` 193 194 ### Check cluster status 195 Once vitess starts, check cluster status 196 ``` 197 vitess/examples/compose$ docker-compose ps 198 199 Name Command State Ports 200 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 201 compose_consul1_1 docker-entrypoint.sh agent ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp, 0.0.0.0:8400->8400/tcp, 0.0.0.0:8500->8500/tcp, 0.0.0.0:8600->8600/tcp, 8600/udp 202 compose_consul2_1 docker-entrypoint.sh agent ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp, 8400/tcp, 8500/tcp, 8600/tcp, 8600/udp 203 compose_consul3_1 docker-entrypoint.sh agent ... Up 8300/tcp, 8301/tcp, 8301/udp, 8302/tcp, 8302/udp, 8400/tcp, 8500/tcp, 8600/tcp, 8600/udp 204 compose_external_db_host_1 docker-entrypoint.sh mysqld Up (healthy) 0.0.0.0:32846->3306/tcp, 33060/tcp 205 compose_schemaload_1 sh -c /script/schemaload.sh Exit 0 206 compose_vreplication_1 sh -c [ $EXTERNAL_DB -eq 1 ... Exit 0 207 compose_vtctld_1 sh -c /vt/bin/vtctld -top ... Up 0.0.0.0:32847->15999/tcp, 0.0.0.0:15000->8080/tcp 208 compose_vtgate_1 sh -c /vt/bin/vtgate -topo ... Up 0.0.0.0:15306->15306/tcp, 0.0.0.0:32856->15999/tcp, 0.0.0.0:15099->8080/tcp 209 compose_vtorc_1 sh -c /script/vtorc-up.sh Up (healthy) 0.0.0.0:13000->3000/tcp 210 compose_vttablet100_1 sh -c [ $EXTERNAL_DB -eq 1 ... Up (healthy) 0.0.0.0:32848->15999/tcp, 0.0.0.0:32849->3306/tcp, 0.0.0.0:15100->8080/tcp 211 compose_vttablet101_1 sh -c /script/vttablet-up. ... Up (healthy) 0.0.0.0:32850->15999/tcp, 0.0.0.0:32851->3306/tcp, 0.0.0.0:15101->8080/tcp 212 compose_vttablet102_1 sh -c /script/vttablet-up. ... Up (healthy) 0.0.0.0:32853->15999/tcp, 0.0.0.0:32855->3306/tcp, 0.0.0.0:15102->8080/tcp 213 compose_vttablet103_1 sh -c /script/vttablet-up. ... Up (healthy) 0.0.0.0:32852->15999/tcp, 0.0.0.0:32854->3306/tcp, 0.0.0.0:15103->8080/tcp 214 215 ``` 216 217 ### Check replication 218 The vreplication container included performs the following actions; 219 1. Identifies the correct source and destination tablets for you 220 2. Copies schema from external database to Vitess managed tablet 221 3. Starts VReplication from external keyspace to managed keyspace 222 4. Prints out helpful debug information for you. 223 ``` 224 vitess/examples/compose$ docker-compose logs -f vreplication 225 vreplication_1 | + /vt/bin/vtctlclient --server vtctld:15999 VReplicationExec local-0000000101 'insert into _vt.vreplication (db_name, source, pos, max_tps, max_replication_lag, tablet_types, time_updated, transaction_timestamp, state) values('\''commerce'\'', '\''keyspace:\"ext_commerce\" shard:\"0\" filter:<rules:<match:\"/.*\" > > on_ddl:EXEC_IGNORE '\'', '\'''\'', 9999, 9999, '\''primary'\'', 0, 0, '\''Running'\'')' 226 vreplication_1 | + /vt/bin/vtctlclient --server vtctld:15999 VReplicationExec local-0000000101 'select * from _vt.vreplication' 227 vreplication_1 | +----+----------+--------------------------------+-----+----------+---------+---------------------+------+--------------+--------------+-----------------------+---------+---------+----------+ 228 vreplication_1 | | id | workflow | source | pos | stop_pos | max_tps | max_replication_lag | cell | tablet_types | time_updated | transaction_timestamp | state | message | db_name | 229 vreplication_1 | +----+----------+--------------------------------+-----+----------+---------+---------------------+------+--------------+--------------+-----------------------+---------+---------+----------+ 230 vreplication_1 | | 1 | | keyspace:"ext_commerce" | | | 9999 | 9999 | | primary | 0 | 0 | Running | | commerce | 231 vreplication_1 | | | | shard:"0" | | | | | | | | | | | | 232 vreplication_1 | | | | filter:<rules:<match:"/.*" > > | | | | | | | | | | | | 233 vreplication_1 | | | | on_ddl:EXEC_IGNORE | | | | | | | | | | | | 234 vreplication_1 | +----+----------+--------------------------------+-----+----------+---------+---------------------+------+--------------+--------------+-----------------------+---------+---------+----------+ 235 compose_vreplication_1 exited with code 0 236 ``` 237 238 ### Connect to vgate and run queries 239 vtgate responds to the MySQL protocol, so we can connect to it using the default MySQL client command line. 240 Verify that data was copied and is replicating successfully. 241 ```sh 242 vitess/examples/compose$ ./lmysql --port=15306 --host=<host of machine containers are running in e.g. 127.0.0.1, docker-machine ip e.t.c> 243 244 mysql> show databases; 245 mysql> show databases; 246 +--------------+ 247 | Databases | 248 +--------------+ 249 | commerce | 250 | ext_commerce | 251 +--------------+ 252 2 rows in set (0.01 sec) 253 mysql> use commerce@replica; 254 Reading table information for completion of table and column names 255 You can turn off this feature to get a quicker startup with -A 256 257 Database changed 258 mysql> show tables; 259 +--------------------+ 260 | Tables_in_commerce | 261 +--------------------+ 262 | users | 263 +--------------------+ 264 1 row in set (0.00 sec) 265 266 mysql> select count(*) from users; 267 +----------+ 268 | count(*) | 269 +----------+ 270 | 1000 | 271 +----------+ 272 1 row in set (0.00 sec) 273 274 ``` 275 276 ## Helper Scripts 277 The following helper scripts are included to help you perform various actions easily 278 * vitess/examples/compose/lvtctl.sh 279 * vitess/examples/compose/lmysql.sh 280 281 You may run them as below 282 ``` 283 vitess/examples/compose$ ./lvtctl.sh <args> 284 ``` 285 286 To run against a specific compose service/container, use the environment variable **$CS** 287 288 ``` 289 vitess/examples/compose$ (export CS=vttablet101; ./lvtctl.sh <args> ) 290 ``` 291 292 ## Custom Image Tags 293 You may specify a custom `vitess:lite` image tag by setting the evnironment variable `VITESS_TAG`. 294 This is optional and defaults to the `latest` tag. Example; 295 * Set `VITESS_TAG=8.0.0` in your `.env` before running `docker-compose up -d` 296 * Run `VITESS_TAG=8.0.0; docker-compose up -d` 297 298 299 ## Reference 300 Checkout this excellent post about [The Life of a Vitess Cluster](https://vitess.io/blog/2020-04-27-life-of-a-cluster/) 301 302