github.com/number571/tendermint@v0.34.11-gost/test/e2e/README.md

github.com/number571/tendermint@v0.34.11-gost/test/e2e/README.md (about)

1 # End-to-End Tests
2
3 Spins up and tests Tendermint networks in Docker Compose based on a testnet manifest. To run the CI testnet:
4
5 ```sh
6 make
7 ./build/runner -f networks/ci.toml
8 ```
9
10 This creates and runs a testnet named `ci` under `networks/ci/`.
11
12 ## Conceptual Overview
13
14 End-to-end testnets are used to test Tendermint functionality as a user would use it, by spinning up a set of nodes with various configurations and making sure the nodes and network behave correctly. The background for the E2E test suite is outlined in [RFC-001](https://github.com/number571/tendermint/blob/master/docs/rfc/rfc-001-end-to-end-testing.md).
15
16 The end-to-end tests can be thought of in this manner:
17
18 1. Does a certain (valid!) testnet configuration result in a block-producing network where all nodes eventually reach the latest height?
19
20 2. If so, does each node in that network satisfy all invariants specified by the Go E2E tests?
21
22 The above should hold for any arbitrary, valid network configuration, and that configuration space should be searched and tested by randomly generating testnets.
23
24 A testnet configuration is specified as a TOML testnet manifest (see below). The testnet runner uses the manifest to configure a set of Docker containers and start them in some order. The manifests can be written manually (to test specific configurations) or generated randomly by the testnet generator (to test a wide range of configuration permutations).
25
26 When running a testnet, the runner will first start the Docker nodes in some sequence, submit random transactions, and wait for the nodes to come online and the first blocks to be produced. This may involve e.g. waiting for nodes to fast sync and/or state sync. If specified, it will then run any misbehaviors (e.g. double-signing) and perturbations (e.g. killing or disconnecting nodes). It then waits for the testnet to stabilize, with all nodes online and having reached the latest height.
27
28 Once the testnet stabilizes, a set of Go end-to-end tests are run against the live testnet to verify network invariants (for example that blocks are identical across nodes). These use the RPC client to interact with the network, and should consider the entire network as a black box (i.e. it should not test any network or node internals, only externally visible behavior via RPC). The tests may use the `testNode()` helper to run parallel tests against each individual testnet node, and/or inspect the full blockchain history via `fetchBlockChain()`.
29
30 The tests must take into account the network and/or node configuration, and tolerate that the network is still live and producing blocks. For example, validator tests should only run against nodes that are actually validators, and take into account the node's block retention and/or state sync configuration to not query blocks that don't exist.
31
32 ## Testnet Manifests
33
34 Testnets are specified as TOML manifests. For an example see [`networks/ci.toml`](networks/ci.toml), and for documentation see [`pkg/manifest.go`](pkg/manifest.go).
35
36 ## Random Testnet Generation
37
38 Random (but deterministic) combinations of testnets can be generated with `generator`:
39
40 ```sh
41 ./build/generator -d networks/generated/
42
43 # Split networks into 8 groups (by filename)
44 ./build/generator -g 8 -d networks/generated/
45 ```
46
47 Multiple testnets can be run with the `run-multiple.sh` script:
48
49 ```sh
50 ./run-multiple.sh networks/generated/gen-group3-*.toml
51 ```
52
53 ## Test Stages
54
55 The test runner has the following stages, which can also be executed explicitly by running `./build/runner -f <manifest> <stage>`:
56
57 * `setup`: generates configuration files.
58
59 * `start`: starts Docker containers.
60
61 * `load`: generates a transaction load against the testnet nodes.
62
63 * `perturb`: runs any requested perturbations (e.g. node restarts or network disconnects).
64
65 * `wait`: waits for a few blocks to be produced, and for all nodes to catch up to it.
66
67 * `test`: runs test cases in `tests/` against all nodes in a running testnet.
68
69 * `stop`: stops Docker containers.
70
71 * `cleanup`: removes configuration files and Docker containers/networks.
72
73 Auxiliary commands:
74
75 * `logs`: outputs all node logs.
76
77 * `tail`: tails (follows) node logs until canceled.
78
79 ## Tests
80
81 Test cases are written as normal Go tests in `tests/`. They use a `testNode()` helper which executes each test as a parallel subtest for each node in the network.
82
83 ### Running Manual Tests
84
85 To run tests manually, set the `E2E_MANIFEST` environment variable to the path of the testnet manifest (e.g. `networks/ci.toml`) and run them as normal, e.g.:
86
87 ```sh
88 ./build/runner -f networks/ci.toml start
89 E2E_MANIFEST=networks/ci.toml go test -v ./tests/...
90 ```
91
92 Optionally, `E2E_NODE` specifies the name of a single testnet node to test.
93
94 These environment variables can also be specified in `tests/e2e_test.go` to run tests from an editor or IDE:
95
96 ```go
97 func init() {
98 // This can be used to manually specify a testnet manifest and/or node to
99 // run tests against. The testnet must have been started by the runner first.
100 os.Setenv("E2E_MANIFEST", "networks/ci.toml")
101 os.Setenv("E2E_NODE", "validator01")
102 }
103 ```
104
105 ### Debugging Failures
106
107 If a command or test fails, the runner simply exits with an error message and
108 non-zero status code. The testnet is left running with data in the testnet
109 directory, and can be inspected with e.g. `docker ps`, `docker logs`, or
110 `./build/runner -f <manifest> logs` or `tail`. To shut down and remove the
111 testnet, run `./build/runner -f <manifest> cleanup`.
112
113 If the standard `log_level` is not detailed enough (e.g. you want "debug" level
114 logging for certain modules), you can change it in the manifest file.
115
116 Each node exposes a [pprof](https://golang.org/pkg/runtime/pprof/) server. To
117 find out the local port, run `docker port <NODENAME> 6060 | awk -F: '{print
118 $2}'`. Then you may perform any queries supported by the pprof tool. Julia
119 Evans has a [great
120 post](https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/) on this
121 subject.
122
123 ```bash
124 export PORT=$(docker port full01 6060 | awk -F: '{print $2}')
125
126 go tool pprof http://localhost:$PORT/debug/pprof/goroutine
127 go tool pprof http://localhost:$PORT/debug/pprof/heap
128 go tool pprof http://localhost:$PORT/debug/pprof/threadcreate
129 go tool pprof http://localhost:$PORT/debug/pprof/block
130 go tool pprof http://localhost:$PORT/debug/pprof/mutex
131 ```
132
133 ## Enabling IPv6
134
135 Docker does not enable IPv6 by default. To do so, enter the following in
136 `daemon.json` (or in the Docker for Mac UI under Preferences → Docker Engine):
137
138 ```json
139 {
140 "ipv6": true,
141 "fixed-cidr-v6": "2001:db8:1::/64"
142 }
143 ```
144
145 ## Benchmarking testnets
146
147 It is also possible to run a simple benchmark on a testnet. This is done through the `benchmark` command. This manages the entire process: setting up the environment, starting the test net, waiting for a considerable amount of blocks to be used (currently 100), and then returning the following metrics from the sample of the blockchain:
148
149 - Average time to produce a block
150 - Standard deviation of producing a block
151 - Minimum and maximum time to produce a block