github.com/Team-Kujira/tendermint@v0.34.24-indexer/spec/abci/apps.md (about)

     1  ---
     2  order: 2
     3  title: Applications
     4  ---
     5  
     6  # Applications
     7  
     8  Please ensure you've first read the spec for [ABCI Methods and Types](abci.md)
     9  
    10  Here we cover the following components of ABCI applications:
    11  
    12  - [Connection State](#connection-state) - the interplay between ABCI connections and application state
    13    and the differences between `CheckTx` and `DeliverTx`.
    14  - [Transaction Results](#transaction-results) - rules around transaction
    15    results and validity
    16  - [Validator Set Updates](#validator-updates) - how validator sets are
    17    changed during `InitChain` and `EndBlock`
    18  - [Query](#query) - standards for using the `Query` method and proofs about the
    19    application state
    20  - [Crash Recovery](#crash-recovery) - handshake protocol to synchronize
    21    Tendermint and the application on startup.
    22  - [State Sync](#state-sync) - rapid bootstrapping of new nodes by restoring state machine snapshots
    23  
    24  ## Connection State
    25  
    26  Since Tendermint maintains four concurrent ABCI connections, it is typical
    27  for an application to maintain a distinct state for each, and for the states to
    28  be synchronized during `Commit`.
    29  
    30  ### Concurrency
    31  
    32  In principle, each of the four ABCI connections operate concurrently with one
    33  another. This means applications need to ensure access to state is
    34  thread safe. In practice, both the
    35  [default in-process ABCI client](https://github.com/tendermint/tendermint/blob/v0.34.4/abci/client/local_client.go#L18)
    36  and the
    37  [default Go ABCI
    38  server](https://github.com/tendermint/tendermint/blob/v0.34.4/abci/server/socket_server.go#L32)
    39  use global locks across all connections, so they are not
    40  concurrent at all. This means if your app is written in Go, and compiled in-process with Tendermint
    41  using the default `NewLocalClient`, or run out-of-process using the default `SocketServer`,
    42  ABCI messages from all connections will be linearizable (received one at a
    43  time).
    44  
    45  The existence of this global mutex means Go application developers can get
    46  thread safety for application state by routing *all* reads and writes through the ABCI
    47  system. Thus it may be *unsafe* to expose application state directly to an RPC
    48  interface, and unless explicit measures are taken, all queries should be routed through the ABCI Query method.
    49  
    50  ### BeginBlock
    51  
    52  The BeginBlock request can be used to run some code at the beginning of
    53  every block. It also allows Tendermint to send the current block hash
    54  and header to the application, before it sends any of the transactions.
    55  
    56  The app should remember the latest height and header (ie. from which it
    57  has run a successful Commit) so that it can tell Tendermint where to
    58  pick up from when it restarts. See information on the Handshake, below.
    59  
    60  ### Commit
    61  
    62  Application state should only be persisted to disk during `Commit`.
    63  
    64  Before `Commit` is called, Tendermint locks and flushes the mempool so that no new messages will
    65  be received on the mempool connection. This provides an opportunity to safely update all four connection
    66  states to the latest committed state at once.
    67  
    68  When `Commit` completes, it unlocks the mempool.
    69  
    70  WARNING: if the ABCI app logic processing the `Commit` message sends a
    71  `/broadcast_tx_sync` or `/broadcast_tx_commit` and waits for the response
    72  before proceeding, it will deadlock. Executing those `broadcast_tx` calls
    73  involves acquiring a lock that is held during the `Commit` call, so it's not
    74  possible. If you make the call to the `broadcast_tx` endpoints concurrently,
    75  that's no problem, it just can't be part of the sequential logic of the
    76  `Commit` function.
    77  
    78  ### Consensus Connection
    79  
    80  The Consensus Connection should maintain a `DeliverTxState` - the working state
    81  for block execution. It should be updated by the calls to `BeginBlock`, `DeliverTx`,
    82  and `EndBlock` during block execution and committed to disk as the "latest
    83  committed state" during `Commit`.
    84  
    85  Updates made to the `DeliverTxState` by each method call must be readable by each subsequent method -
    86  ie. the updates are linearizable.
    87  
    88  ### Mempool Connection
    89  
    90  The mempool Connection should maintain a `CheckTxState`
    91  to sequentially process pending transactions in the mempool that have
    92  not yet been committed. It should be initialized to the latest committed state
    93  at the end of every `Commit`.
    94  
    95  Before calling `Commit`, Tendermint will lock and flush the mempool connection,
    96  ensuring that all existing CheckTx are responded to and no new ones can begin.
    97  The `CheckTxState` may be updated concurrently with the `DeliverTxState`, as
    98  messages may be sent concurrently on the Consensus and Mempool connections.
    99  
   100  After `Commit`, while still holding the mempool lock, CheckTx is run again on all transactions that remain in the
   101  node's local mempool after filtering those included in the block.
   102  An additional `Type` parameter is made available to the CheckTx function that
   103  indicates whether an incoming transaction is new (`CheckTxType_New`), or a
   104  recheck (`CheckTxType_Recheck`).
   105  
   106  Finally, after re-checking transactions in the mempool, Tendermint will unlock
   107  the mempool connection. New transactions are once again able to be processed through CheckTx.
   108  
   109  Note that CheckTx is just a weak filter to keep invalid transactions out of the block chain.
   110  CheckTx doesn't have to check everything that affects transaction validity; the
   111  expensive things can be skipped.  It's weak because a Byzantine node doesn't
   112  care about CheckTx; it can propose a block full of invalid transactions if it wants.
   113  
   114  #### Replay Protection
   115  
   116  To prevent old transactions from being replayed, CheckTx must implement
   117  replay protection.
   118  
   119  It is possible for old transactions to be sent to the application. So
   120  it is important CheckTx implements some logic to handle them.
   121  
   122  ### Query Connection
   123  
   124  The Info Connection should maintain a `QueryState` for answering queries from the user,
   125  and for initialization when Tendermint first starts up (both described further
   126  below).
   127  It should always contain the latest committed state associated with the
   128  latest committed block.
   129  
   130  `QueryState` should be set to the latest `DeliverTxState` at the end of every `Commit`,
   131  after the full block has been processed and the state committed to disk.
   132  Otherwise it should never be modified.
   133  
   134  Tendermint Core currently uses the Query connection to filter peers upon
   135  connecting, according to IP address or node ID. For instance,
   136  returning non-OK ABCI response to either of the following queries will
   137  cause Tendermint to not connect to the corresponding peer:
   138  
   139  - `p2p/filter/addr/<ip addr>`, where `<ip addr>` is an IP address.
   140  - `p2p/filter/id/<id>`, where `<is>` is the hex-encoded node ID (the hash of
   141    the node's p2p pubkey).
   142  
   143  Note: these query formats are subject to change!
   144  
   145  ### Snapshot Connection
   146  
   147  The Snapshot Connection is optional, and is only used to serve state sync snapshots for other nodes
   148  and/or restore state sync snapshots to a local node being bootstrapped.
   149  
   150  For more information, see [the state sync section of this document](#state-sync).
   151  
   152  ## Transaction Results
   153  
   154  The `Info` and `Log` fields are non-deterministic values for debugging/convenience purposes
   155  that are otherwise ignored.
   156  
   157  The `Data` field must be strictly deterministic, but can be arbitrary data.
   158  
   159  ### Gas
   160  
   161  Ethereum introduced the notion of `gas` as an abstract representation of the
   162  cost of resources used by nodes when processing transactions. Every operation in the
   163  Ethereum Virtual Machine uses some amount of gas, and gas can be accepted at a market-variable price.
   164  Users propose a maximum amount of gas for their transaction; if the tx uses less, they get
   165  the difference credited back. Tendermint adopts a similar abstraction,
   166  though uses it only optionally and weakly, allowing applications to define
   167  their own sense of the cost of execution.
   168  
   169  In Tendermint, the [ConsensusParams.Block.MaxGas](../proto/types/params.proto) limits the amount of `gas` that can be used in a block.
   170  The default value is `-1`, meaning no limit, or that the concept of gas is
   171  meaningless.
   172  
   173  Responses contain a `GasWanted` and `GasUsed` field. The former is the maximum
   174  amount of gas the sender of a tx is willing to use, and the latter is how much it actually
   175  used. Applications should enforce that `GasUsed <= GasWanted` - ie. tx execution
   176  should halt before it can use more resources than it requested.
   177  
   178  When `MaxGas > -1`, Tendermint enforces the following rules:
   179  
   180  - `GasWanted <= MaxGas` for all txs in the mempool
   181  - `(sum of GasWanted in a block) <= MaxGas` when proposing a block
   182  
   183  If `MaxGas == -1`, no rules about gas are enforced.
   184  
   185  Note that Tendermint does not currently enforce anything about Gas in the consensus, only the mempool.
   186  This means it does not guarantee that committed blocks satisfy these rules!
   187  It is the application's responsibility to return non-zero response codes when gas limits are exceeded.
   188  
   189  The `GasUsed` field is ignored completely by Tendermint. That said, applications should enforce:
   190  
   191  - `GasUsed <= GasWanted` for any given transaction
   192  - `(sum of GasUsed in a block) <= MaxGas` for every block
   193  
   194  In the future, we intend to add a `Priority` field to the responses that can be
   195  used to explicitly prioritize txs in the mempool for inclusion in a block
   196  proposal. See [#1861](https://github.com/tendermint/tendermint/issues/1861).
   197  
   198  ### CheckTx
   199  
   200  If `Code != 0`, it will be rejected from the mempool and hence
   201  not broadcasted to other peers and not included in a proposal block.
   202  
   203  `Data` contains the result of the CheckTx transaction execution, if any. It is
   204  semantically meaningless to Tendermint.
   205  
   206  `Events` include any events for the execution, though since the transaction has not
   207  been committed yet, they are effectively ignored by Tendermint.
   208  
   209  ### DeliverTx
   210  
   211  DeliverTx is the workhorse of the blockchain. Tendermint sends the
   212  DeliverTx requests asynchronously but in order, and relies on the
   213  underlying socket protocol (ie. TCP) to ensure they are received by the
   214  app in order. They have already been ordered in the global consensus by
   215  the Tendermint protocol.
   216  
   217  If DeliverTx returns `Code != 0`, the transaction will be considered invalid,
   218  though it is still included in the block.
   219  
   220  DeliverTx also returns a [Code, Data, and Log](../../proto/abci/types.proto#L189-L191).
   221  
   222  `Data` contains the result of the CheckTx transaction execution, if any. It is
   223  semantically meaningless to Tendermint.
   224  
   225  Both the `Code` and `Data` are included in a structure that is hashed into the
   226  `LastResultsHash` of the next block header.
   227  
   228  `Events` include any events for the execution, which Tendermint will use to index
   229  the transaction by. This allows transactions to be queried according to what
   230  events took place during their execution.
   231  
   232  ## Updating the Validator Set
   233  
   234  The application may set the validator set during InitChain, and may update it during
   235  EndBlock.
   236  
   237  Note that the maximum total power of the validator set is bounded by
   238  `MaxTotalVotingPower = MaxInt64 / 8`. Applications are responsible for ensuring
   239  they do not make changes to the validator set that cause it to exceed this
   240  limit.
   241  
   242  Additionally, applications must ensure that a single set of updates does not contain any duplicates -
   243  a given public key can only appear once within a given update. If an update includes
   244  duplicates, the block execution will fail irrecoverably.
   245  
   246  ### InitChain
   247  
   248  The `InitChain` method can return a list of validators.
   249  If the list is empty, Tendermint will use the validators loaded in the genesis
   250  file.
   251  If the list returned by `InitChain` is not empty, Tendermint will use its contents as the validator set.
   252  This way the application can set the initial validator set for the
   253  blockchain.
   254  
   255  ### EndBlock
   256  
   257  Updates to the Tendermint validator set can be made by returning
   258  `ValidatorUpdate` objects in the `ResponseEndBlock`:
   259  
   260  ```protobuf
   261  message ValidatorUpdate {
   262    tendermint.crypto.keys.PublicKey pub_key
   263    int64 power
   264  }
   265  
   266  message PublicKey {
   267    oneof {
   268      ed25519 bytes = 1;
   269    }
   270  ```
   271  
   272  The `pub_key` currently supports only one type:
   273  
   274  - `type = "ed25519"`
   275  
   276  The `power` is the new voting power for the validator, with the
   277  following rules:
   278  
   279  - power must be non-negative
   280  - if power is 0, the validator must already exist, and will be removed from the
   281    validator set
   282  - if power is non-0:
   283      - if the validator does not already exist, it will be added to the validator
   284      set with the given power
   285      - if the validator does already exist, its power will be adjusted to the given power
   286  - the total power of the new validator set must not exceed MaxTotalVotingPower
   287  
   288  Note the updates returned in block `H` will only take effect at block `H+2`.
   289  
   290  ## Consensus Parameters
   291  
   292  ConsensusParams enforce certain limits in the blockchain, like the maximum size
   293  of blocks, amount of gas used in a block, and the maximum acceptable age of
   294  evidence. They can be set in InitChain and updated in EndBlock.
   295  
   296  ### BlockParams.MaxBytes
   297  
   298  The maximum size of a complete Protobuf encoded block.
   299  This is enforced by Tendermint consensus.
   300  
   301  This implies a maximum transaction size that is this MaxBytes, less the expected size of
   302  the header, the validator set, and any included evidence in the block.
   303  
   304  Must have `0 < MaxBytes < 100 MB`.
   305  
   306  ### BlockParams.MaxGas
   307  
   308  The maximum of the sum of `GasWanted` that will be allowed in a proposed block.
   309  This is *not* enforced by Tendermint consensus.
   310  It is left to the app to enforce (ie. if txs are included past the
   311  limit, they should return non-zero codes). It is used by Tendermint to limit the
   312  txs included in a proposed block.
   313  
   314  Must have `MaxGas >= -1`.
   315  If `MaxGas == -1`, no limit is enforced.
   316  
   317  ### EvidenceParams.MaxAgeDuration
   318  
   319  This is the maximum age of evidence in time units.
   320  This is enforced by Tendermint consensus.
   321  
   322  If a block includes evidence older than this (AND the evidence was created more
   323  than `MaxAgeNumBlocks` ago), the block will be rejected (validators won't vote
   324  for it).
   325  
   326  Must have `MaxAgeDuration > 0`.
   327  
   328  ### EvidenceParams.MaxAgeNumBlocks
   329  
   330  This is the maximum age of evidence in blocks.
   331  This is enforced by Tendermint consensus.
   332  
   333  If a block includes evidence older than this (AND the evidence was created more
   334  than `MaxAgeDuration` ago), the block will be rejected (validators won't vote
   335  for it).
   336  
   337  Must have `MaxAgeNumBlocks > 0`.
   338  
   339  ### EvidenceParams.MaxNum
   340  
   341  This is the maximum number of evidence that can be committed to a single block.
   342  
   343  The product of this and the `MaxEvidenceBytes` must not exceed the size of
   344  a block minus it's overhead ( ~ `MaxBytes`).
   345  
   346  Must have `MaxNum > 0`.
   347  
   348  ### Updates
   349  
   350  The application may set the ConsensusParams during InitChain, and update them during
   351  EndBlock. If the ConsensusParams is empty, it will be ignored. Each field
   352  that is not empty will be applied in full. For instance, if updating the
   353  Block.MaxBytes, applications must also set the other Block fields (like
   354  Block.MaxGas), even if they are unchanged, as they will otherwise cause the
   355  value to be updated to 0.
   356  
   357  #### InitChain
   358  
   359  ResponseInitChain includes a ConsensusParams.
   360  If ConsensusParams is nil, Tendermint will use the params loaded in the genesis
   361  file. If ConsensusParams is not nil, Tendermint will use it.
   362  This way the application can determine the initial consensus params for the
   363  blockchain.
   364  
   365  #### EndBlock
   366  
   367  ResponseEndBlock includes a ConsensusParams.
   368  If ConsensusParams nil, Tendermint will do nothing.
   369  If ConsensusParam is not nil, Tendermint will use it.
   370  This way the application can update the consensus params over time.
   371  
   372  Note the updates returned in block `H` will take effect right away for block
   373  `H+1`.
   374  
   375  ## Query
   376  
   377  Query is a generic method with lots of flexibility to enable diverse sets
   378  of queries on application state. Tendermint makes use of Query to filter new peers
   379  based on ID and IP, and exposes Query to the user over RPC.
   380  
   381  Note that calls to Query are not replicated across nodes, but rather query the
   382  local node's state - hence they may return stale reads. For reads that require
   383  consensus, use a transaction.
   384  
   385  The most important use of Query is to return Merkle proofs of the application state at some height
   386  that can be used for efficient application-specific light-clients.
   387  
   388  Note Tendermint has technically no requirements from the Query
   389  message for normal operation - that is, the ABCI app developer need not implement
   390  Query functionality if they do not wish too.
   391  
   392  ### Query Proofs
   393  
   394  The Tendermint block header includes a number of hashes, each providing an
   395  anchor for some type of proof about the blockchain. The `ValidatorsHash` enables
   396  quick verification of the validator set, the `DataHash` gives quick
   397  verification of the transactions included in the block, etc.
   398  
   399  The `AppHash` is unique in that it is application specific, and allows for
   400  application-specific Merkle proofs about the state of the application.
   401  While some applications keep all relevant state in the transactions themselves
   402  (like Bitcoin and its UTXOs), others maintain a separated state that is
   403  computed deterministically *from* transactions, but is not contained directly in
   404  the transactions themselves (like Ethereum contracts and accounts).
   405  For such applications, the `AppHash` provides a much more efficient way to verify light-client proofs.
   406  
   407  ABCI applications can take advantage of more efficient light-client proofs for
   408  their state as follows:
   409  
   410  - return the Merkle root of the deterministic application state in
   411  `ResponseCommit.Data`. This Merkle root will be included as the `AppHash` in the next block.
   412  - return efficient Merkle proofs about that application state in `ResponseQuery.Proof`
   413    that can be verified using the `AppHash` of the corresponding block.
   414  
   415  For instance, this allows an application's light-client to verify proofs of
   416  absence in the application state, something which is much less efficient to do using the block hash.
   417  
   418  Some applications (eg. Ethereum, Cosmos-SDK) have multiple "levels" of Merkle trees,
   419  where the leaves of one tree are the root hashes of others. To support this, and
   420  the general variability in Merkle proofs, the `ResponseQuery.Proof` has some minimal structure:
   421  
   422  ```protobuf
   423  message ProofOps {
   424    repeated ProofOp ops
   425  }
   426  
   427  message ProofOp {
   428    string type = 1;
   429    bytes key = 2;
   430    bytes data = 3;
   431  }
   432  ```
   433  
   434  Each `ProofOp` contains a proof for a single key in a single Merkle tree, of the specified `type`.
   435  This allows ABCI to support many different kinds of Merkle trees, encoding
   436  formats, and proofs (eg. of presence and absence) just by varying the `type`.
   437  The `data` contains the actual encoded proof, encoded according to the `type`.
   438  When verifying the full proof, the root hash for one ProofOp is the value being
   439  verified for the next ProofOp in the list. The root hash of the final ProofOp in
   440  the list should match the `AppHash` being verified against.
   441  
   442  ### Peer Filtering
   443  
   444  When Tendermint connects to a peer, it sends two queries to the ABCI application
   445  using the following paths, with no additional data:
   446  
   447  - `/p2p/filter/addr/<IP:PORT>`, where `<IP:PORT>` denote the IP address and
   448    the port of the connection
   449  - `p2p/filter/id/<ID>`, where `<ID>` is the peer node ID (ie. the
   450    pubkey.Address() for the peer's PubKey)
   451  
   452  If either of these queries return a non-zero ABCI code, Tendermint will refuse
   453  to connect to the peer.
   454  
   455  ### Paths
   456  
   457  Queries are directed at paths, and may optionally include additional data.
   458  
   459  The expectation is for there to be some number of high level paths
   460  differentiating concerns, like `/p2p`, `/store`, and `/app`. Currently,
   461  Tendermint only uses `/p2p`, for filtering peers. For more advanced use, see the
   462  implementation of
   463  [Query in the Cosmos-SDK](https://github.com/cosmos/cosmos-sdk/blob/v0.23.1/baseapp/baseapp.go#L333).
   464  
   465  ## Crash Recovery
   466  
   467  On startup, Tendermint calls the `Info` method on the Info Connection to get the latest
   468  committed state of the app. The app MUST return information consistent with the
   469  last block it succesfully completed Commit for.
   470  
   471  If the app succesfully committed block H, then `last_block_height = H` and `last_block_app_hash = <hash returned by Commit for block H>`. If the app
   472  failed during the Commit of block H, then `last_block_height = H-1` and
   473  `last_block_app_hash = <hash returned by Commit for block H-1, which is the hash in the header of block H>`.
   474  
   475  We now distinguish three heights, and describe how Tendermint syncs itself with
   476  the app.
   477  
   478  ```md
   479  storeBlockHeight = height of the last block Tendermint saw a commit for
   480  stateBlockHeight = height of the last block for which Tendermint completed all
   481      block processing and saved all ABCI results to disk
   482  appBlockHeight = height of the last block for which ABCI app succesfully
   483      completed Commit
   484  
   485  ```
   486  
   487  Note we always have `storeBlockHeight >= stateBlockHeight` and `storeBlockHeight >= appBlockHeight`
   488  Note also Tendermint never calls Commit on an ABCI app twice for the same height.
   489  
   490  The procedure is as follows.
   491  
   492  First, some simple start conditions:
   493  
   494  If `appBlockHeight == 0`, then call InitChain.
   495  
   496  If `storeBlockHeight == 0`, we're done.
   497  
   498  Now, some sanity checks:
   499  
   500  If `storeBlockHeight < appBlockHeight`, error
   501  If `storeBlockHeight < stateBlockHeight`, panic
   502  If `storeBlockHeight > stateBlockHeight+1`, panic
   503  
   504  Now, the meat:
   505  
   506  If `storeBlockHeight == stateBlockHeight && appBlockHeight < storeBlockHeight`,
   507  replay all blocks in full from `appBlockHeight` to `storeBlockHeight`.
   508  This happens if we completed processing the block, but the app forgot its height.
   509  
   510  If `storeBlockHeight == stateBlockHeight && appBlockHeight == storeBlockHeight`, we're done.
   511  This happens if we crashed at an opportune spot.
   512  
   513  If `storeBlockHeight == stateBlockHeight+1`
   514  This happens if we started processing the block but didn't finish.
   515  
   516  If `appBlockHeight < stateBlockHeight`
   517      replay all blocks in full from `appBlockHeight` to `storeBlockHeight-1`,
   518      and replay the block at `storeBlockHeight` using the WAL.
   519  This happens if the app forgot the last block it committed.
   520  
   521  If `appBlockHeight == stateBlockHeight`,
   522      replay the last block (storeBlockHeight) in full.
   523  This happens if we crashed before the app finished Commit
   524  
   525  If `appBlockHeight == storeBlockHeight`
   526      update the state using the saved ABCI responses but dont run the block against the real app.
   527  This happens if we crashed after the app finished Commit but before Tendermint saved the state.
   528  
   529  ## State Sync
   530  
   531  A new node joining the network can simply join consensus at the genesis height and replay all
   532  historical blocks until it is caught up. However, for large chains this can take a significant
   533  amount of time, often on the order of days or weeks.
   534  
   535  State sync is an alternative mechanism for bootstrapping a new node, where it fetches a snapshot
   536  of the state machine at a given height and restores it. Depending on the application, this can
   537  be several orders of magnitude faster than replaying blocks.
   538  
   539  Note that state sync does not currently backfill historical blocks, so the node will have a
   540  truncated block history - users are advised to consider the broader network implications of this in
   541  terms of block availability and auditability. This functionality may be added in the future.
   542  
   543  For details on the specific ABCI calls and types, see the [methods and types section](abci.md).
   544  
   545  ### Taking Snapshots
   546  
   547  Applications that want to support state syncing must take state snapshots at regular intervals. How
   548  this is accomplished is entirely up to the application. A snapshot consists of some metadata and
   549  a set of binary chunks in an arbitrary format:
   550  
   551  - `Height (uint64)`: The height at which the snapshot is taken. It must be taken after the given
   552    height has been committed, and must not contain data from any later heights.
   553  
   554  - `Format (uint32)`: An arbitrary snapshot format identifier. This can be used to version snapshot
   555    formats, e.g. to switch from Protobuf to MessagePack for serialization. The application can use
   556    this when restoring to choose whether to accept or reject a snapshot.
   557  
   558  - `Chunks (uint32)`: The number of chunks in the snapshot. Each chunk contains arbitrary binary
   559    data, and should be less than 16 MB; 10 MB is a good starting point.
   560  
   561  - `Hash ([]byte)`: An arbitrary hash of the snapshot. This is used to check whether a snapshot is
   562    the same across nodes when downloading chunks.
   563  
   564  - `Metadata ([]byte)`: Arbitrary snapshot metadata, e.g. chunk hashes for verification or any other
   565    necessary info.
   566  
   567  For a snapshot to be considered the same across nodes, all of these fields must be identical. When
   568  sent across the network, snapshot metadata messages are limited to 4 MB.
   569  
   570  When a new node is running state sync and discovering snapshots, Tendermint will query an existing
   571  application via the ABCI `ListSnapshots` method to discover available snapshots, and load binary
   572  snapshot chunks via `LoadSnapshotChunk`. The application is free to choose how to implement this
   573  and which formats to use, but must provide the following guarantees:
   574  
   575  - **Consistent:** A snapshot must be taken at a single isolated height, unaffected by
   576    concurrent writes. This can be accomplished by using a data store that supports ACID
   577    transactions with snapshot isolation.
   578  
   579  - **Asynchronous:** Taking a snapshot can be time-consuming, so it must not halt chain progress,
   580    for example by running in a separate thread.
   581  
   582  - **Deterministic:** A snapshot taken at the same height in the same format must be identical
   583    (at the byte level) across nodes, including all metadata. This ensures good availability of
   584    chunks, and that they fit together across nodes.
   585  
   586  A very basic approach might be to use a datastore with MVCC transactions (such as RocksDB),
   587  start a transaction immediately after block commit, and spawn a new thread which is passed the
   588  transaction handle. This thread can then export all data items, serialize them using e.g.
   589  Protobuf, hash the byte stream, split it into chunks, and store the chunks in the file system
   590  along with some metadata - all while the blockchain is applying new blocks in parallel.
   591  
   592  A more advanced approach might include incremental verification of individual chunks against the
   593  chain app hash, parallel or batched exports, compression, and so on.
   594  
   595  Old snapshots should be removed after some time - generally only the last two snapshots are needed
   596  (to prevent the last one from being removed while a node is restoring it).
   597  
   598  ### Bootstrapping a Node
   599  
   600  An empty node can be state synced by setting the configuration option `statesync.enabled =
   601  true`. The node also needs the chain genesis file for basic chain info, and configuration for
   602  light client verification of the restored snapshot: a set of Tendermint RPC servers, and a
   603  trusted header hash and corresponding height from a trusted source, via the `statesync`
   604  configuration section.
   605  
   606  Once started, the node will connect to the P2P network and begin discovering snapshots. These
   607  will be offered to the local application via the `OfferSnapshot` ABCI method. Once a snapshot
   608  is accepted Tendermint will fetch and apply the snapshot chunks. After all chunks have been
   609  successfully applied, Tendermint verifies the app's `AppHash` against the chain using the light
   610  client, then switches the node to normal consensus operation.
   611  
   612  #### Snapshot Discovery
   613  
   614  When the empty node join the P2P network, it asks all peers to report snapshots via the
   615  `ListSnapshots` ABCI call (limited to 10 per node). After some time, the node picks the most
   616  suitable snapshot (generally prioritized by height, format, and number of peers), and offers it
   617  to the application via `OfferSnapshot`. The application can choose a number of responses,
   618  including accepting or rejecting it, rejecting the offered format, rejecting the peer who sent
   619  it, and so on. Tendermint will keep discovering and offering snapshots until one is accepted or
   620  the application aborts.
   621  
   622  #### Snapshot Restoration
   623  
   624  Once a snapshot has been accepted via `OfferSnapshot`, Tendermint begins downloading chunks from
   625  any peers that have the same snapshot (i.e. that have identical metadata fields). Chunks are
   626  spooled in a temporary directory, and then given to the application in sequential order via
   627  `ApplySnapshotChunk` until all chunks have been accepted.
   628  
   629  The method for restoring snapshot chunks is entirely up to the application.
   630  
   631  During restoration, the application can respond to `ApplySnapshotChunk` with instructions for how
   632  to continue. This will typically be to accept the chunk and await the next one, but it can also
   633  ask for chunks to be refetched (either the current one or any number of previous ones), P2P peers
   634  to be banned, snapshots to be rejected or retried, and a number of other responses - see the ABCI
   635  reference for details.
   636  
   637  If Tendermint fails to fetch a chunk after some time, it will reject the snapshot and try a
   638  different one via `OfferSnapshot` - the application can choose whether it wants to support
   639  restarting restoration, or simply abort with an error.
   640  
   641  #### Snapshot Verification
   642  
   643  Once all chunks have been accepted, Tendermint issues an `Info` ABCI call to retrieve the
   644  `LastBlockAppHash`. This is compared with the trusted app hash from the chain, retrieved and
   645  verified using the light client. Tendermint also checks that `LastBlockHeight` corresponds to the
   646  height of the snapshot.
   647  
   648  This verification ensures that an application is valid before joining the network. However, the
   649  snapshot restoration may take a long time to complete, so applications may want to employ additional
   650  verification during the restore to detect failures early. This might e.g. include incremental
   651  verification of each chunk against the app hash (using bundled Merkle proofs), checksums to
   652  protect against data corruption by the disk or network, and so on. However, it is important to
   653  note that the only trusted information available is the app hash, and all other snapshot metadata
   654  can be spoofed by adversaries.
   655  
   656  Apps may also want to consider state sync denial-of-service vectors, where adversaries provide
   657  invalid or harmful snapshots to prevent nodes from joining the network. The application can
   658  counteract this by asking Tendermint to ban peers. As a last resort, node operators can use
   659  P2P configuration options to whitelist a set of trusted peers that can provide valid snapshots.
   660  
   661  #### Transition to Consensus
   662  
   663  Once the snapshots have all been restored, Tendermint gathers additional information necessary for
   664  bootstrapping the node (e.g. chain ID, consensus parameters, validator sets, and block headers)
   665  from the genesis file and light client RPC servers. It also fetches and records the `AppVersion`
   666  from the ABCI application.
   667  
   668  Once the state machine has been restored and Tendermint has gathered this additional
   669  information, it transitions to block sync (if enabled) to fetch any remaining blocks up the chain
   670  head, and then transitions to regular consensus operation. At this point the node operates like
   671  any other node, apart from having a truncated block history at the height of the restored snapshot.