github.com/pingcap/tiflow@v0.0.0-20240520035814-5bf52d54e205/README_Engine.md (about)

     1  # Tiflow engine
     2  
     3  ## Introduction
     4  
     5  This repo implements a new engine of distributed task scheduler.
     6  
     7  ### Master
     8  
     9  Master is set to process the requests from outside and to schedule and dispatch tasks.
    10  
    11  ### Executor
    12  
    13  Executor is the worker process to run the tasks.
    14  
    15  ## Build
    16  
    17  Simply run `make engine` to compile.
    18  
    19  ## Run Test
    20  
    21  Use `make engine_unit_test` to run unit test and integrated test.
    22  
    23  ## Deploy Demonstration
    24  
    25  ### Single Master and Single Executor on Two Nodes
    26  
    27  #### Start Master on Single Node
    28  
    29  ```[shell]
    30  ./bin/tiflow master --config=./deployments/engine/docker-compose/config/master.toml --addr 0.0.0.0:10240 --advertise-addr ${ip0}:10240 
    31  ```
    32  
    33  Replace **ip0** with your advertising ip.
    34  
    35  #### Start Executor
    36  
    37  ```[shell]
    38  ./bin/tiflow executor --config=./deployments/engine/docker-compose/config/executor.toml --join ${ip0}:10240 --addr 0.0.0.0:10241 --advertise-addr ${ip1}:10241
    39  ```
    40  
    41  Replace **ip1** with your advertising executor ip.
    42  
    43  ### Three Master and One Executor on Three Nodes
    44  
    45  Scaling out executor is quite simple. Right now we only support static config of masters, without scaling in and out dynamically.
    46  
    47  In this case, we assume three node ips are ip0, ip1 and ip2. Replace them with real ip or hostname when operating.
    48  
    49  #### Start Master on Node (ip0)
    50  
    51  ```[shell]
    52  ./bin/tiflow master --config=./deployments/engine/docker-compose/config/master.toml --addr 0.0.0.0:10240 --advertise-addr http://${ip0}:10240
    53  ```
    54  
    55  Deploying masters for ip1 and ip2 are similar.
    56  
    57  #### Start Executor on Node (ip0)
    58  
    59  ```[shell]
    60  ./bin/tiflow executor --config=./deployments/engine/docker-compose/config/executor.toml --join ${ip0}:10240 --addr 0.0.0.0:10241 --advertise-addr ${ip1}:10241
    61  ```
    62  
    63  ## Run engine in docker
    64  
    65  This section shows how to start the cluster and run some tests by docker and docker-compose.
    66  
    67  ### Preparations
    68  
    69  The following programs must be installed:
    70  
    71  * docker
    72  * docker-compose
    73  
    74  Besides, make sure you have run the docker daemon. We recommend that you provide docker with at least 6+ cores and 8G+ memory. Of course, the more resources, the better.
    75  
    76  Then, change directory to working directory:
    77  
    78  ```bash
    79  cd ./deployments/engine/docker-compose
    80  ```
    81  
    82  ### Build
    83  
    84  To build this repo, you can run `../run-engine.sh build` in working directory (or simply run `make engine_image` in root directory of tiflow project).
    85  
    86  If you have build the binary on your local Linux, you can try `../run-engine.sh build-local`.
    87  
    88  ### Deploy
    89  
    90  There are several configure files to use. The file name suggests the number of running server-master and executor nodes.   
    91  For example, `1m1e.yaml` means this file contains one server-master and one executor.   
    92  Use `../run-engine.sh deploy ./deployments/engine/docker-compose/1m1e.yaml` to deploy cluster.
    93  
    94  ### Destroy
    95  
    96  Use `../run-engine.sh stop ./deployments/engine/docker-compose/1m1e.yaml` to destroy the cluster.
    97  
    98  ### Cleanup
    99  
   100  sudo rm -rf /tmp/df/master
   101  
   102  ## Manager engine cluster by [helm](https://github.com/helm/helm) in local K8s
   103  ### Install tools
   104  * [helm](https://helm.sh/docs/intro/install/)
   105  * [kind](https://kind.sigs.k8s.io/)
   106  * [kubectl](https://kubernetes.io/docs/tasks/tools/)
   107  
   108  ### Create a k8s cluster
   109  ```
   110  $ kind create cluster
   111  Creating cluster "kind" ...
   112   ✓ Ensuring node image (kindest/node:v1.24.0) đŸ–ŧ 
   113   ✓ Preparing nodes đŸ“Ļ  
   114   ✓ Writing configuration 📜 
   115   ✓ Starting control-plane 🕹ī¸ 
   116   ✓ Installing CNI 🔌 
   117   ✓ Installing StorageClass 💾 
   118  Set kubectl context to "kind-kind"
   119  You can now use your cluster with:
   120  
   121  kubectl cluster-info --context kind-kind
   122  Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
   123  
   124  $ kubectl cluster-info --context kind-kind
   125  ```
   126  
   127  ### Load engine image to k8s cluster
   128  ```
   129  $ make engine_image
   130  $ kind load docker-image dataflow:test
   131  ```
   132  
   133  ### Deploy engine cluster via helm
   134  ```
   135  $ cd deployments/engine/helm
   136  $ helm install test ./tiflow
   137  NAME: test
   138  LAST DEPLOYED: Fri Jul 22 18:52:02 2022
   139  NAMESPACE: default
   140  STATUS: deployed
   141  REVISION: 1
   142  TEST SUITE: None
   143  $ helm list
   144  NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
   145  t1      default         1               2022-07-22 18:51:49.82093217 +0800 CST  deployed        tiflow-0.1.0    dev        
   146  test    default         1               2022-07-22 18:52:02.96320918 +0800 CST  deployed        tiflow-0.1.0    dev   
   147  
   148  $ kubectl get pods 
   149  NAME                               READY   STATUS    RESTARTS        AGE
   150  test-chaos-test-case-rvhx2         1/1     Running   0               6m58s
   151  test-executor-0                    1/1     Running   0               6m58s
   152  test-executor-1                    1/1     Running   0               6m58s
   153  test-executor-2                    1/1     Running   0               6m58s
   154  test-executor-3                    1/1     Running   0               6m58s
   155  test-metastore-business-etcd-0     1/1     Running   0               6m58s
   156  test-metastore-framework-mysql-0   1/1     Running   0               6m58s
   157  test-server-master-0               1/1     Running   1 (4m49s ago)   6m58s
   158  test-server-master-1               1/1     Running   1 (4m49s ago)   6m58s
   159  test-server-master-2               1/1     Running   0               6m58s
   160  ```
   161  
   162  ## Contribute