github.com/pingcap/tiflow@v0.0.0-20240520035814-5bf52d54e205/README_Engine.md (about) 1 # Tiflow engine 2 3 ## Introduction 4 5 This repo implements a new engine of distributed task scheduler. 6 7 ### Master 8 9 Master is set to process the requests from outside and to schedule and dispatch tasks. 10 11 ### Executor 12 13 Executor is the worker process to run the tasks. 14 15 ## Build 16 17 Simply run `make engine` to compile. 18 19 ## Run Test 20 21 Use `make engine_unit_test` to run unit test and integrated test. 22 23 ## Deploy Demonstration 24 25 ### Single Master and Single Executor on Two Nodes 26 27 #### Start Master on Single Node 28 29 ```[shell] 30 ./bin/tiflow master --config=./deployments/engine/docker-compose/config/master.toml --addr 0.0.0.0:10240 --advertise-addr ${ip0}:10240 31 ``` 32 33 Replace **ip0** with your advertising ip. 34 35 #### Start Executor 36 37 ```[shell] 38 ./bin/tiflow executor --config=./deployments/engine/docker-compose/config/executor.toml --join ${ip0}:10240 --addr 0.0.0.0:10241 --advertise-addr ${ip1}:10241 39 ``` 40 41 Replace **ip1** with your advertising executor ip. 42 43 ### Three Master and One Executor on Three Nodes 44 45 Scaling out executor is quite simple. Right now we only support static config of masters, without scaling in and out dynamically. 46 47 In this case, we assume three node ips are ip0, ip1 and ip2. Replace them with real ip or hostname when operating. 48 49 #### Start Master on Node (ip0) 50 51 ```[shell] 52 ./bin/tiflow master --config=./deployments/engine/docker-compose/config/master.toml --addr 0.0.0.0:10240 --advertise-addr http://${ip0}:10240 53 ``` 54 55 Deploying masters for ip1 and ip2 are similar. 56 57 #### Start Executor on Node (ip0) 58 59 ```[shell] 60 ./bin/tiflow executor --config=./deployments/engine/docker-compose/config/executor.toml --join ${ip0}:10240 --addr 0.0.0.0:10241 --advertise-addr ${ip1}:10241 61 ``` 62 63 ## Run engine in docker 64 65 This section shows how to start the cluster and run some tests by docker and docker-compose. 66 67 ### Preparations 68 69 The following programs must be installed: 70 71 * docker 72 * docker-compose 73 74 Besides, make sure you have run the docker daemon. We recommend that you provide docker with at least 6+ cores and 8G+ memory. Of course, the more resources, the better. 75 76 Then, change directory to working directory: 77 78 ```bash 79 cd ./deployments/engine/docker-compose 80 ``` 81 82 ### Build 83 84 To build this repo, you can run `../run-engine.sh build` in working directory (or simply run `make engine_image` in root directory of tiflow project). 85 86 If you have build the binary on your local Linux, you can try `../run-engine.sh build-local`. 87 88 ### Deploy 89 90 There are several configure files to use. The file name suggests the number of running server-master and executor nodes. 91 For example, `1m1e.yaml` means this file contains one server-master and one executor. 92 Use `../run-engine.sh deploy ./deployments/engine/docker-compose/1m1e.yaml` to deploy cluster. 93 94 ### Destroy 95 96 Use `../run-engine.sh stop ./deployments/engine/docker-compose/1m1e.yaml` to destroy the cluster. 97 98 ### Cleanup 99 100 sudo rm -rf /tmp/df/master 101 102 ## Manager engine cluster by [helm](https://github.com/helm/helm) in local K8s 103 ### Install tools 104 * [helm](https://helm.sh/docs/intro/install/) 105 * [kind](https://kind.sigs.k8s.io/) 106 * [kubectl](https://kubernetes.io/docs/tasks/tools/) 107 108 ### Create a k8s cluster 109 ``` 110 $ kind create cluster 111 Creating cluster "kind" ... 112 â Ensuring node image (kindest/node:v1.24.0) đŧ 113 â Preparing nodes đĻ 114 â Writing configuration đ 115 â Starting control-plane đšī¸ 116 â Installing CNI đ 117 â Installing StorageClass đž 118 Set kubectl context to "kind-kind" 119 You can now use your cluster with: 120 121 kubectl cluster-info --context kind-kind 122 Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community đ 123 124 $ kubectl cluster-info --context kind-kind 125 ``` 126 127 ### Load engine image to k8s cluster 128 ``` 129 $ make engine_image 130 $ kind load docker-image dataflow:test 131 ``` 132 133 ### Deploy engine cluster via helm 134 ``` 135 $ cd deployments/engine/helm 136 $ helm install test ./tiflow 137 NAME: test 138 LAST DEPLOYED: Fri Jul 22 18:52:02 2022 139 NAMESPACE: default 140 STATUS: deployed 141 REVISION: 1 142 TEST SUITE: None 143 $ helm list 144 NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION 145 t1 default 1 2022-07-22 18:51:49.82093217 +0800 CST deployed tiflow-0.1.0 dev 146 test default 1 2022-07-22 18:52:02.96320918 +0800 CST deployed tiflow-0.1.0 dev 147 148 $ kubectl get pods 149 NAME READY STATUS RESTARTS AGE 150 test-chaos-test-case-rvhx2 1/1 Running 0 6m58s 151 test-executor-0 1/1 Running 0 6m58s 152 test-executor-1 1/1 Running 0 6m58s 153 test-executor-2 1/1 Running 0 6m58s 154 test-executor-3 1/1 Running 0 6m58s 155 test-metastore-business-etcd-0 1/1 Running 0 6m58s 156 test-metastore-framework-mysql-0 1/1 Running 0 6m58s 157 test-server-master-0 1/1 Running 1 (4m49s ago) 6m58s 158 test-server-master-1 1/1 Running 1 (4m49s ago) 6m58s 159 test-server-master-2 1/1 Running 0 6m58s 160 ``` 161 162 ## Contribute