github.com/badrootd/nibiru-cometbft@v0.37.5-0.20240307173500-2a75559eee9b/test/e2e/README.md (about)

     1  # End-to-End Tests
     2  
     3  Spins up and tests CometBFT networks in Docker Compose based on a testnet manifest. To run the CI testnet:
     4  
     5  ```sh
     6  make
     7  ./build/runner -f networks/ci.toml
     8  ```
     9  
    10  This creates and runs a testnet named `ci` under `networks/ci/` (determined by the manifest filename).
    11  
    12  ## Testnet Manifests
    13  
    14  Testnets are specified as TOML manifests. For an example see [`networks/ci.toml`](networks/ci.toml), and for documentation see [`pkg/manifest.go`](pkg/manifest.go).
    15  
    16  ## Random Testnet Generation
    17  
    18  Random (but deterministic) combinations of testnets can be generated with `generator`:
    19  
    20  ```sh
    21  ./build/generator -d networks/generated/
    22  
    23  # Split networks into 8 groups (by filename)
    24  ./build/generator -g 8 -d networks/generated/
    25  ```
    26  
    27  Multiple testnets can be run with the `run-multiple.sh` script:
    28  
    29  ```sh
    30  ./run-multiple.sh networks/generated/gen-group3-*.toml
    31  ```
    32  
    33  Testnets running different versions of CometBFT can be generated by the
    34  generator. For example:
    35  
    36  ```sh
    37  # Generate testnets randomly choosing between v0.34.21 (making up 1/3rd of the
    38  # network) and v0.34.22 (making up 2/3rds of the network).
    39  ./build/generator -m "v0.34.21:1,v0.34.22:2" -d networks/generated/
    40  
    41  # "local" refers to the current local code. The E2E node built from the local
    42  # code will be run on 2/3rds of the network, whereas the v0.34.23 E2E node will
    43  # be run on the remaining 1/3rd.
    44  ./build/generator -m "v0.34.23:1,local:2" -d networks/generated/
    45  
    46  # Using "latest" will cause the generator to auto-detect the latest
    47  # non-pre-release version tag in the current Git repository that is closest to
    48  # the CometBFT version in the current local code (as specified in
    49  # ../../version/version.go).
    50  #
    51  # In the example below, if the local version.TMCoreSemVer value is "v0.34.24",
    52  # for example, and the latest official release is v0.34.23, then 1/3rd of the
    53  # network will run v0.34.23 and the remaining 2/3rds will run the E2E node built
    54  # from the local code.
    55  ./build/generator -m "latest:1,local:2" -d networks/generated/
    56  ```
    57  
    58  **NB**: The corresponding Docker images for the relevant versions of the E2E
    59  node (the `cometbft/e2e-node` image) must be available on the local machine,
    60  or via [Docker Hub](https://hub.docker.com/r/cometbft/e2e-node).
    61  
    62  Multiversion testnets can also perform uncoordinated upgrades. Nodes containing a
    63  perturbation of type `upgrade` will upgrade to the target version specified in
    64  testnet's attribute `upgrade_version` of the testnet manifest.
    65  The generator generates this type of perturbation both on full nodes and on light nodes.
    66  Perturbations of type `upgrade` are a noop if the node's version matches the
    67  one in `upgrade_version`.
    68  
    69  ## Test Stages
    70  
    71  The test runner has the following stages, which can also be executed explicitly by running `./build/runner -f <manifest> <stage>`:
    72  
    73  * `setup`: generates configuration files.
    74  
    75  * `start`: starts Docker containers.
    76  
    77  * `load`: generates a transaction load against the testnet nodes.
    78  
    79  * `perturb`: runs any requested perturbations (e.g. node restarts or network disconnects).
    80  
    81  * `wait`: waits for a few blocks to be produced, and for all nodes to catch up to it.
    82  
    83  * `test`: runs test cases in `tests/` against all nodes in a running testnet.
    84  
    85  * `stop`: stops Docker containers.
    86  
    87  * `cleanup`: removes configuration files and Docker containers/networks.
    88  
    89  Auxiliary commands:
    90  
    91  * `logs`: outputs all node logs.
    92  
    93  * `tail`: tails (follows) node logs until canceled.
    94  
    95  ## Tests
    96  
    97  Test cases are written as normal Go tests in `tests/`. They use a `testNode()` helper which executes each test as a parallel subtest for each node in the network.
    98  
    99  ### Running Manual Tests
   100  
   101  To run tests manually, set the `E2E_MANIFEST` environment variable to the path of the testnet manifest (e.g. `networks/ci.toml`) and run them as normal, e.g.:
   102  
   103  ```sh
   104  ./build/runner -f networks/ci.toml start
   105  E2E_MANIFEST=networks/ci.toml go test -v ./tests/...
   106  ```
   107  
   108  Optionally, `E2E_NODE` specifies the name of a single testnet node to test.
   109  
   110  These environment variables can also be specified in `tests/e2e_test.go` to run tests from an editor or IDE:
   111  
   112  ```go
   113  func init() {
   114  	// This can be used to manually specify a testnet manifest and/or node to
   115  	// run tests against. The testnet must have been started by the runner first.
   116  	os.Setenv("E2E_MANIFEST", "networks/ci.toml")
   117  	os.Setenv("E2E_NODE", "validator01")
   118  }
   119  ```
   120  
   121  ### Debugging Failures
   122  
   123  If a command or test fails, the runner simply exits with an error message and
   124  non-zero status code. The testnet is left running with data in the testnet
   125  directory, and can be inspected with e.g. `docker ps`, `docker logs`, or
   126  `./build/runner -f <manifest> logs` or `tail`. To shut down and remove the
   127  testnet, run `./build/runner -f <manifest> cleanup`.
   128  
   129  If the standard `log_level` is not detailed enough (e.g. you want "debug" level
   130  logging for certain modules), you can change it in the manifest file.
   131  
   132  Each node exposes a [pprof](https://golang.org/pkg/runtime/pprof/) server. To
   133  find out the local port, run `docker port <NODENAME> 6060 | awk -F: '{print
   134  $2}'`. Then you may perform any queries supported by the pprof tool. Julia
   135  Evans has a [great
   136  post](https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/) on this
   137  subject.
   138  
   139  ```bash
   140  export PORT=$(docker port full01 6060 | awk -F: '{print $2}')
   141  
   142  go tool pprof http://localhost:$PORT/debug/pprof/goroutine
   143  go tool pprof http://localhost:$PORT/debug/pprof/heap
   144  go tool pprof http://localhost:$PORT/debug/pprof/threadcreate
   145  go tool pprof http://localhost:$PORT/debug/pprof/block
   146  go tool pprof http://localhost:$PORT/debug/pprof/mutex
   147  ```
   148  
   149  ## Enabling IPv6
   150  
   151  Docker does not enable IPv6 by default. To do so, enter the following in
   152  `daemon.json` (or in the Docker for Mac UI under Preferences → Docker Engine):
   153  
   154  ```json
   155  {
   156    "ipv6": true,
   157    "fixed-cidr-v6": "2001:db8:1::/64"
   158  }
   159  ```
   160  
   161  ## Benchmarking Testnets
   162  
   163  It is also possible to run a simple benchmark on a testnet. This is done through the `benchmark` command. This manages the entire process: setting up the environment, starting the test net, waiting for a considerable amount of blocks to be used (currently 100), and then returning the following metrics from the sample of the blockchain:
   164  
   165  - Average time to produce a block
   166  - Standard deviation of producing a block
   167  - Minimum and maximum time to produce a block
   168  
   169  ## Running Individual Nodes
   170  
   171  The E2E test harness is designed to run several nodes of varying configurations within docker. It is also possible to run a single node in the case of running larger, geographically-dispersed testnets. To run a single node you can either run:
   172  
   173  **Built-in**
   174  
   175  ```bash
   176  make node
   177  cometbft init validator
   178  CMTHOME=$HOME/.cometbft ./build/node ./node/built-in.toml
   179  ```
   180  
   181  To make things simpler the e2e application can also be run in the `cometbft` binary
   182  by running
   183  
   184  ```bash
   185  cometbft start --proxy-app e2e
   186  ```
   187  
   188  However this won't offer the same level of configurability of the application.
   189  
   190  **Socket**
   191  
   192  ```bash
   193  make node
   194  cometbft init validator
   195  cometbft start
   196  ./build/node ./node.socket.toml
   197  ```
   198  
   199  Check `node/config.go` to see how the settings of the test application can be tweaked.