github.com/noirx94/tendermintmp@v0.0.1/docs/rfc/rfc-001-end-to-end-testing.md (about)

     1  # RFC 001: End-to-End Testing
     2  
     3  ## Changelog
     4  
     5  - 2020-09-07: Initial draft (@erikgrinaker)
     6  
     7  - 2020-09-08: Minor improvements (@erikgrinaker)
     8  
     9  ## Authors
    10  
    11  - Erik Grinaker (@erikgrinaker)
    12  
    13  ## Context
    14  
    15  The current set of end-to-end tests under `test/` are very limited, mostly focusing on P2P testing in a standard configuration. They do not test various configurations (e.g. fast sync reactor versions, state sync, block pruning, genesis vs InitChain setup), nor do they test various network topologies (e.g. sentry node architecture). This leads to poor test coverage, which has caused several serious bugs to go unnoticed.
    16  
    17  We need an end-to-end test suite that can run a large number of combinations of configuration options, genesis settings, network topologies, ABCI interactions, and failure scenarios and check that the network is still functional. This RFC outlines the basic requirements and design considerations, but does not propose a specific implementation - a later ADR will be submitted for this.
    18  
    19  This RFC will not cover comprehensive chaos testing, only a few simple scenarios (e.g. abrupt process termination and network partitioning). Chaos testing of the core consensus algorithm should be implemented e.g. via Jepsen tests or a similar framework, or alternatively be added to these end-to-end tests at a later time. Similarly, malicious or adversarial behavior is out of scope for the first implementation, but may be added later.
    20  
    21  ## Proposal
    22  
    23  ### Functional Coverage
    24  
    25  The following lists the functionality we would like to test:
    26  
    27  #### Environments
    28  
    29  - **Topology:** single node, 4 nodes (seeds and persistent), sentry architecture, NAT (UPnP)
    30  - **Networking:** IPv4, IPv6
    31  - **ABCI connection:** UNIX socket, TCP, gRPC
    32  - **PrivVal:** file, UNIX socket, TCP
    33  
    34  #### Node/App Configurations
    35  
    36  - **Database:** goleveldb, cleveldb, boltdb, rocksdb, badgerdb
    37  - **Fast sync:** disabled, v0, v1, v2
    38  - **State sync:** disabled, enabled
    39  - **Block pruning:** none, keep 20, keep 1, keep random
    40  - **Role:** validator, full node
    41  - **App persistence:** enabled, disabled
    42  
    43  #### Geneses
    44  
    45  - **Validators:** none (InitChain), given
    46  - **Initial height:** 1, 1000
    47  - **App state:** none, given
    48  
    49  #### Behaviors
    50  
    51  - **Recovery:** stop/start, power cycling, validator outage, network partition, total network loss
    52  - **Validators:** add, remove, change power
    53  
    54  ### Functional Combinations
    55  
    56  Running separate tests for all combinations of the above functionality is not feasible, as there are millions of them. However, the functionality can be grouped into three broad classes:
    57  
    58  - **Global:** affects the entire network, needing a separate testnet for each combination (e.g. topology, network protocol, genesis settings)
    59  
    60  - **Local:** affects a single node, and can be varied per node in a testnet (e.g. ABCI/privval connections, database backend, block pruning)
    61  
    62  - **Temporal:** can be run after each other in the same testnet (e.g. recovery and validator changes)
    63  
    64  Thus, we can run separate testnets for all combinations of global options (on the order of 100). In each testnet, we run nodes with randomly generated node configurations optimized for broad coverage (i.e. if one node is using GoLevelDB, then no other node should use it if possible). And in each testnet, we sequentially and randomly pick nodes to stop/start, power cycle, add/remove, disconnect, and so on.
    65  
    66  All of the settings should be specified in a testnet configuration (or alternatively the seed that generated it) such that it can be retrieved from CI and debugged locally.
    67  
    68  A custom ABCI application will have to be built that can exhibit the necessary behavior (e.g. make validator changes, prune blocks, enable/disable persistence, and so on).
    69  
    70  ### Test Stages
    71  
    72  Given a test configuration, the test runner has the following stages:
    73  
    74  - **Setup:** configures the Docker containers and networks, but does not start them.
    75  
    76  - **Initialization:** starts the Docker containers, performs fast sync/state sync.
    77  
    78  - **Perturbation:** adds/removes validators, restarts nodes, perturbs networking, etc - liveness and readiness checked between each operation.
    79  
    80  - **Testing:** runs RPC tests independently against all network nodes, making sure data matches expectations and invariants hold.
    81  
    82  ### Tests
    83  
    84  The general approach will be to put the network through a sequence of operations (see stages above), check basic liveness and readiness after each operation, and then once the network stabilizes run an RPC test suite against each node in the network.
    85  
    86  The test suite will do black-box testing against a single node's RPC service. We will be testing the behavior of the network as a whole, e.g. that a fast synced node correctly catches up to the chain head and serves basic block data via RPC. Thus the tests will not send e.g. P2P messages or examine the node database, as these are considered internal implementation details - if the network behaves correctly, presumably the internal components function correctly. Comprehensive component testing (e.g. each and every RPC method parameter) should be done via unit/integration tests.
    87  
    88  The tests must take into account the node configuration (e.g. some nodes may be pruned, others may not be validators), and should somehow be provided access to expected data (i.e. complete block headers for the entire chain).
    89  
    90  The test suite should use the Tendermint RPC client and the Tendermint light client, to exercise the client code as well.
    91  
    92  ### Implementation Considerations
    93  
    94  The testnets should run in Docker Compose, both locally and in CI. This makes it easier to reproduce test failures locally. Supporting multiple test-runners (e.g. on VMs or Kubernetes) is out of scope. The same image should be used for all tests, with configuration passed via a mounted volume.
    95  
    96  There does not appear to be any off-the-shelf solutions that would do this for us, so we will have to roll our own on top of Docker Compose. This gives us more flexibility, but is estimated to be a few weeks of work.
    97  
    98  Testnets should be configured via a YAML file. These are used as inputs for the test runner, which e.g. generates Docker Compose configurations from them. An additional layer on top should generate these testnet configurations from a YAML file that specifies all the option combinations to test.
    99  
   100  Comprehensive testnets should run against master nightly. However, a small subset of representative testnets should run for each pull request, e.g. a four-node IPv4 network with state sync and fast sync.
   101  
   102  Tests should be written using the standard Go test framework (and e.g. Testify), with a helper function to fetch info from the test configuration. The test runner will run the tests separately for each network node, and the test must vary its expectations based on the node's configuration.
   103  
   104  It should be possible to launch a specific testnet and run individual test cases from the IDE or local terminal against a it.
   105  
   106  If possible, the existing `testnet` command should be extended to set up the network topologies needed by the end-to-end tests.
   107  
   108  ## Status
   109  
   110  Accepted
   111  
   112  ## Consequences
   113  
   114  ### Positive
   115  
   116  - Comprehensive end-to-end test coverage of basic Tendermint functionality, exercising common code paths in the same way that users would
   117  
   118  - Test environments can easily be reproduced locally and debugged via standard tooling
   119  
   120  ### Negative
   121  
   122  - Limited coverage of consensus correctness testing (e.g. Jepsen)
   123  
   124  - No coverage of malicious or adversarial behavior
   125  
   126  - Have to roll our own test framework, which takes engineering resources
   127  
   128  - Possibly slower CI times, depending on which tests are run
   129  
   130  - Operational costs and overhead, e.g. infrastructure costs and system maintenance
   131  
   132  ### Neutral
   133  
   134  - No support for alternative infrastructure platforms, e.g. Kubernetes or VMs
   135  
   136  ## References
   137  
   138  - [#5291: new end-to-end test suite](https://github.com/tendermint/tendermint/issues/5291)