go.etcd.io/etcd@v3.3.27+incompatible/functional/README.md (about) 1 ## etcd Functional Testing 2 3 [`functional`](https://godoc.org/github.com/coreos/etcd/functional) verifies the correct behavior of etcd under various system and network malfunctions. It sets up an etcd cluster under high pressure loads and continuously injects failures into the cluster. Then it expects the etcd cluster to recover within a few seconds. This has been extremely helpful to find critical bugs. 4 5 See [`rpcpb.Case`](https://godoc.org/github.com/coreos/etcd/functional/rpcpb#Case) for all failure cases. 6 7 See [functional.yaml](https://github.com/coreos/etcd/blob/master/functional.yaml) for an example configuration. 8 9 ### Run locally 10 11 ```bash 12 PASSES=functional ./test 13 ``` 14 15 ### Run with Docker 16 17 ```bash 18 pushd .. 19 make build-docker-functional 20 popd 21 ``` 22 23 And run [example scripts](./scripts). 24 25 ```bash 26 # run 3 agents for 3-node local etcd cluster 27 ./scripts/docker-local-agent.sh 1 28 ./scripts/docker-local-agent.sh 2 29 ./scripts/docker-local-agent.sh 3 30 31 # to run only 1 tester round 32 ./scripts/docker-local-tester.sh 33 ``` 34 35 ## etcd Proxy 36 37 Proxy layer that simulates various network conditions. 38 39 Test locally 40 41 ```bash 42 $ ./build 43 $ ./bin/etcd 44 45 $ make build-functional 46 47 $ ./bin/etcd-proxy --help 48 $ ./bin/etcd-proxy --from localhost:23790 --to localhost:2379 --http-port 2378 --verbose 49 50 $ ETCDCTL_API=3 ./bin/etcdctl --endpoints localhost:2379 put foo bar 51 $ ETCDCTL_API=3 ./bin/etcdctl --endpoints localhost:23790 put foo bar 52 ``` 53 54 Proxy overhead per request is under 500μs 55 56 ```bash 57 $ go build -v -o ./bin/benchmark ./tools/benchmark 58 59 $ ./bin/benchmark \ 60 --endpoints localhost:2379 \ 61 --conns 5 \ 62 --clients 15 \ 63 put \ 64 --key-size 48 \ 65 --val-size 50000 \ 66 --total 10000 67 68 <<COMMENT 69 Summary: 70 Total: 8.4611 secs. 71 Slowest: 0.1324 secs. 72 Fastest: 0.0011 secs. 73 Average: 0.0121 secs. 74 Stddev: 0.0125 secs. 75 Requests/sec: 1181.8758 76 77 Response time histogram: 78 0.0011 [1] | 79 0.0142 [7899] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ 80 0.0273 [1339] |∎∎∎∎∎∎ 81 0.0405 [543] |∎∎ 82 0.0536 [67] | 83 0.0667 [49] | 84 0.0798 [9] | 85 0.0930 [15] | 86 0.1061 [42] | 87 0.1192 [21] | 88 0.1324 [15] | 89 90 Latency distribution: 91 10% in 0.0049 secs. 92 25% in 0.0064 secs. 93 50% in 0.0085 secs. 94 75% in 0.0126 secs. 95 90% in 0.0243 secs. 96 95% in 0.0307 secs. 97 99% in 0.0686 secs. 98 99.9% in 0.1294 secs. 99 COMMENT 100 101 $ ./bin/benchmark \ 102 --endpoints localhost:23790 \ 103 --conns 5 \ 104 --clients 15 \ 105 put \ 106 --key-size 48 \ 107 --val-size 50000 \ 108 --total 10000 109 110 <<COMMENT 111 Summary: 112 Total: 9.1128 secs. 113 Slowest: 0.1363 secs. 114 Fastest: 0.0015 secs. 115 Average: 0.0131 secs. 116 Stddev: 0.0113 secs. 117 Requests/sec: 1097.3613 118 119 Response time histogram: 120 0.0015 [1] | 121 0.0150 [7407] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ 122 0.0285 [2017] |∎∎∎∎∎∎∎∎∎∎ 123 0.0419 [440] |∎∎ 124 0.0554 [30] | 125 0.0689 [13] | 126 0.0824 [12] | 127 0.0959 [48] | 128 0.1093 [2] | 129 0.1228 [16] | 130 0.1363 [14] | 131 132 Latency distribution: 133 10% in 0.0054 secs. 134 25% in 0.0071 secs. 135 50% in 0.0100 secs. 136 75% in 0.0153 secs. 137 90% in 0.0241 secs. 138 95% in 0.0297 secs. 139 99% in 0.0584 secs. 140 99.9% in 0.1312 secs. 141 COMMENT 142 ``` 143 144 Delay client transmit 145 146 ```bash 147 $ curl -L http://localhost:2378/delay-tx -X PUT \ 148 -d "latency=5s&random-variable=100ms" 149 # added send latency 5s±100ms (current latency 4.92143955s) 150 151 $ curl -L http://localhost:2378/delay-tx 152 # current send latency 4.92143955s 153 154 $ ETCDCTL_API=3 ./bin/etcdctl \ 155 --endpoints localhost:23790 \ 156 --command-timeout=3s \ 157 put foo bar 158 # Error: context deadline exceeded 159 160 $ curl -L http://localhost:2378/delay-tx -X DELETE 161 # removed latency 4.92143955s 162 163 $ curl -L http://localhost:2378/delay-tx 164 # current send latency 0s 165 166 $ ETCDCTL_API=3 ./bin/etcdctl \ 167 --endpoints localhost:23790 \ 168 --command-timeout=3s \ 169 put foo bar 170 # OK 171 ``` 172 173 Pause client transmit 174 175 ```bash 176 $ curl -L http://localhost:2378/pause-tx -X PUT 177 # paused forwarding [tcp://localhost:23790 -> tcp://localhost:2379] 178 179 $ ETCDCTL_API=3 ./bin/etcdctl \ 180 --endpoints localhost:23790 \ 181 put foo bar 182 # Error: context deadline exceeded 183 184 $ curl -L http://localhost:2378/pause-tx -X DELETE 185 # unpaused forwarding [tcp://localhost:23790 -> tcp://localhost:2379] 186 ``` 187 188 Drop client packets 189 190 ```bash 191 $ curl -L http://localhost:2378/blackhole-tx -X PUT 192 # blackholed; dropping packets [tcp://localhost:23790 -> tcp://localhost:2379] 193 194 $ ETCDCTL_API=3 ./bin/etcdctl --endpoints localhost:23790 put foo bar 195 # Error: context deadline exceeded 196 197 $ curl -L http://localhost:2378/blackhole-tx -X DELETE 198 # unblackholed; restart forwarding [tcp://localhost:23790 -> tcp://localhost:2379] 199 ``` 200 201 Trigger leader election 202 203 ```bash 204 $ ./build 205 $ make build-functional 206 207 $ rm -rf /tmp/etcd-proxy-data.s* 208 $ goreman -f ./functional/Procfile-proxy start 209 210 $ ETCDCTL_API=3 ./bin/etcdctl \ 211 --endpoints localhost:13790,localhost:23790,localhost:33790 \ 212 member list 213 214 # isolate s1 when s1 is the current leader 215 $ curl -L http://localhost:1381/blackhole-tx -X PUT 216 $ curl -L http://localhost:1381/blackhole-rx -X PUT 217 # s1 becomes follower after election timeout 218 ```