github.com/aakash4dev/cometbft@v0.38.2/spec/abci/abci++_app_requirements.md (about)

     1  ---
     2  order: 3
     3  title: Requirements for the Application
     4  ---
     5  
     6  # Requirements for the Application
     7  
     8  - [Formal Requirements](#formal-requirements)
     9    - [Consensus Connection Requirements](#consensus-connection-requirements)
    10    - [Mempool Connection Requirements](#mempool-connection-requirements)
    11  - [Managing the Application state and related topics](#managing-the-application-state-and-related-topics)
    12    - [Connection State](#connection-state)
    13      - [Concurrency](#concurrency)
    14      - [Finalize Block](#finalizeblock)
    15      - [Commit](#commit)
    16      - [Candidate States](#candidate-states)
    17    - [States and ABCI++ Connections](#states-and-abci%2B%2B-connections) 
    18      - [Consensus Connection](#consensus-connection)
    19      - [Mempool Connection](#mempool-connection)
    20      - [Info/Query Connection](#infoquery-connection)
    21      - [Snapshot Connection](#snapshot-connection)
    22    - [Transaction Results](#transaction-results)
    23    - [Updating the Validator Set](#updating-the-validator-set)
    24    - [Consensus Parameters](#consensus-parameters)
    25      - [List of Parameters](#list-of-parameters)
    26      - [Updating Consensus Parameters](#updating-consensus-parameters)
    27    - [Query](#query)
    28      - [Query Proofs](#query-proofs)
    29      - [Peer Filtering](#peer-filtering)
    30      - [Paths](#paths)
    31    - [Crash Recovery](#crash-recovery)
    32    - [State Sync](#state-sync)
    33  - [Application configuration required to switch to ABCI2.0](#application-configuration-required-to-switch-to-abci-20)
    34  
    35  
    36  ## Formal Requirements
    37  
    38  ### Consensus Connection Requirements
    39  
    40  This section specifies what CometBFT expects from the Application. It is structured as a set
    41  of formal requirements that can be used for testing and verification of the Application's logic.
    42  
    43  Let *p* and *q* be two correct processes.
    44  Let *r<sub>p</sub>* (resp. *r<sub>q</sub>*) be a round of height *h* where *p* (resp. *q*) is the
    45  proposer.
    46  Let *s<sub>p,h-1</sub>* be *p*'s Application's state committed for height *h-1*.
    47  Let *v<sub>p</sub>* (resp. *v<sub>q</sub>*) be the block that *p*'s (resp. *q*'s) CometBFT passes
    48  on to the Application
    49  via `RequestPrepareProposal` as proposer of round *r<sub>p</sub>* (resp *r<sub>q</sub>*), height *h*,
    50  also known as the raw proposal.
    51  Let *u<sub>p</sub>* (resp. *u<sub>q</sub>*) the possibly modified block *p*'s (resp. *q*'s) Application
    52  returns via `ResponsePrepareProposal` to CometBFT, also known as the prepared proposal.
    53  
    54  Process *p*'s prepared proposal can differ in two different rounds where *p* is the proposer.
    55  
    56  * Requirement 1 [`PrepareProposal`, timeliness]: If *p*'s Application fully executes prepared blocks in
    57    `PrepareProposal` and the network is in a synchronous period while processes *p* and *q* are in *r<sub>p</sub>*,
    58    then the value of *TimeoutPropose* at *q* must be such that *q*'s propose timer does not time out
    59    (which would result in *q* prevoting `nil` in *r<sub>p</sub>*).
    60  
    61  Full execution of blocks at `PrepareProposal` time stands on CometBFT's critical path. Thus,
    62  Requirement 1 ensures the Application or operator will set a value for `TimeoutPropose` such that the time it takes
    63  to fully execute blocks in `PrepareProposal` does not interfere with CometBFT's propose timer.
    64  Note that the violation of Requirement 1 may lead to further rounds, but will not
    65  compromise liveness because even though `TimeoutPropose` is used as the initial
    66  value for proposal timeouts, CometBFT will be dynamically adjust these timeouts
    67  such that they will eventually be enough for completing `PrepareProposal`.
    68  
    69  * Requirement 2 [`PrepareProposal`, tx-size]: When *p*'s Application calls `ResponsePrepareProposal`, the
    70    total size in bytes of the transactions returned does not exceed `RequestPrepareProposal.max_tx_bytes`.
    71  
    72  Busy blockchains might seek to gain full visibility into transactions in CometBFT's mempool,
    73  rather than having visibility only on *a* subset of those transactions that fit in a block.
    74  The application can do so by setting `ConsensusParams.Block.MaxBytes` to -1.
    75  This instructs CometBFT (a) to enforce the maximum possible value for `MaxBytes` (100 MB) at CometBFT level,
    76  and (b) to provide *all* transactions in the mempool when calling `RequestPrepareProposal`.
    77  Under these settings, the aggregated size of all transactions may exceed `RequestPrepareProposal.max_tx_bytes`.
    78  Hence, Requirement 2 ensures that the size in bytes of the transaction list returned by the application will never
    79  cause the resulting block to go beyond its byte size limit.
    80  
    81  * Requirement 3 [`PrepareProposal`, `ProcessProposal`, coherence]: For any two correct processes *p* and *q*,
    82    if *q*'s CometBFT calls `RequestProcessProposal` on *u<sub>p</sub>*,
    83    *q*'s Application returns Accept in `ResponseProcessProposal`.
    84  
    85  Requirement 3 makes sure that blocks proposed by correct processes *always* pass the correct receiving process's
    86  `ProcessProposal` check.
    87  On the other hand, if there is a deterministic bug in `PrepareProposal` or `ProcessProposal` (or in both),
    88  strictly speaking, this makes all processes that hit the bug byzantine. This is a problem in practice,
    89  as very often validators are running the Application from the same codebase, so potentially *all* would
    90  likely hit the bug at the same time. This would result in most (or all) processes prevoting `nil`, with the
    91  serious consequences on CometBFT's liveness that this entails. Due to its criticality, Requirement 3 is a
    92  target for extensive testing and automated verification.
    93  
    94  * Requirement 4 [`ProcessProposal`, determinism-1]: `ProcessProposal` is a (deterministic) function of the current
    95    state and the block that is about to be applied. In other words, for any correct process *p*, and any arbitrary block *u*,
    96    if *p*'s CometBFT calls `RequestProcessProposal` on *u* at height *h*,
    97    then *p*'s Application's acceptance or rejection **exclusively** depends on *u* and *s<sub>p,h-1</sub>*.
    98  
    99  * Requirement 5 [`ProcessProposal`, determinism-2]: For any two correct processes *p* and *q*, and any arbitrary
   100    block *u*,
   101    if *p*'s (resp. *q*'s) CometBFT calls `RequestProcessProposal` on *u* at height *h*,
   102    then *p*'s Application accepts *u* if and only if *q*'s Application accepts *u*.
   103    Note that this requirement follows from Requirement 4 and the Agreement property of consensus.
   104  
   105  Requirements 4 and 5 ensure that all correct processes will react in the same way to a proposed block, even
   106  if the proposer is Byzantine. However, `ProcessProposal` may contain a bug that renders the
   107  acceptance or rejection of the block non-deterministic, and therefore prevents processes hitting
   108  the bug from fulfilling Requirements 4 or 5 (effectively making those processes Byzantine).
   109  In such a scenario, CometBFT's liveness cannot be guaranteed.
   110  Again, this is a problem in practice if most validators are running the same software, as they are likely
   111  to hit the bug at the same point. There is currently no clear solution to help with this situation, so
   112  the Application designers/implementors must proceed very carefully with the logic/implementation
   113  of `ProcessProposal`. As a general rule `ProcessProposal` SHOULD always accept the block.
   114  
   115  According to the Tendermint consensus algorithm, currently adopted in CometBFT,
   116  a correct process can broadcast at most one precommit
   117  message in round *r*, height *h*.
   118  Since, as stated in the [Methods](./abci++_methods.md#extendvote) section, `ResponseExtendVote`
   119  is only called when the consensus algorithm
   120  is about to broadcast a non-`nil` precommit message, a correct process can only produce one vote extension
   121  in round *r*, height *h*.
   122  Let *e<sup>r</sup><sub>p</sub>* be the vote extension that the Application of a correct process *p* returns via
   123  `ResponseExtendVote` in round *r*, height *h*.
   124  Let *w<sup>r</sup><sub>p</sub>* be the proposed block that *p*'s CometBFT passes to the Application via `RequestExtendVote`
   125  in round *r*, height *h*.
   126  
   127  * Requirement 6 [`ExtendVote`, `VerifyVoteExtension`, coherence]: For any two different correct
   128    processes *p* and *q*, if *q* receives *e<sup>r</sup><sub>p</sub>* from *p* in height *h*, *q*'s
   129    Application returns Accept in `ResponseVerifyVoteExtension`.
   130  
   131  Requirement 6 constrains the creation and handling of vote extensions in a similar way as Requirement 3
   132  constrains the creation and handling of proposed blocks.
   133  Requirement 6 ensures that extensions created by correct processes *always* pass the `VerifyVoteExtension`
   134  checks performed by correct processes receiving those extensions.
   135  However, if there is a (deterministic) bug in `ExtendVote` or `VerifyVoteExtension` (or in both),
   136  we will face the same liveness issues as described for Requirement 5, as Precommit messages with invalid vote
   137  extensions will be discarded.
   138  
   139  * Requirement 7 [`VerifyVoteExtension`, determinism-1]: `VerifyVoteExtension` is a (deterministic) function of
   140    the current state, the vote extension received, and the prepared proposal that the extension refers to.
   141    In other words, for any correct process *p*, and any arbitrary vote extension *e*, and any arbitrary
   142    block *w*, if *p*'s (resp. *q*'s) CometBFT calls `RequestVerifyVoteExtension` on *e* and *w* at height *h*,
   143    then *p*'s Application's acceptance or rejection **exclusively** depends on *e*, *w* and *s<sub>p,h-1</sub>*.
   144  
   145  * Requirement 8 [`VerifyVoteExtension`, determinism-2]: For any two correct processes *p* and *q*,
   146    and any arbitrary vote extension *e*, and any arbitrary block *w*,
   147    if *p*'s (resp. *q*'s) CometBFT calls `RequestVerifyVoteExtension` on *e* and *w* at height *h*,
   148    then *p*'s Application accepts *e* if and only if *q*'s Application accepts *e*.
   149    Note that this requirement follows from Requirement 7 and the Agreement property of consensus.
   150  
   151  Requirements 7 and 8 ensure that the validation of vote extensions will be deterministic at all
   152  correct processes.
   153  Requirements 7 and 8 protect against arbitrary vote extension data from Byzantine processes,
   154  in a similar way as Requirements 4 and 5 protect against arbitrary proposed blocks.
   155  Requirements 7 and 8 can be violated by a bug inducing non-determinism in
   156  `VerifyVoteExtension`. In this case liveness can be compromised.
   157  Extra care should be put in the implementation of `ExtendVote` and `VerifyVoteExtension`.
   158  As a general rule, `VerifyVoteExtension` SHOULD always accept the vote extension.
   159  
   160  * Requirement 9 [*all*, no-side-effects]: *p*'s calls to `RequestPrepareProposal`,
   161    `RequestProcessProposal`, `RequestExtendVote`, and `RequestVerifyVoteExtension` at height *h* do
   162    not modify *s<sub>p,h-1</sub>*.
   163  
   164  
   165  * Requirement 10 [`ExtendVote`, `FinalizeBlock`, non-dependency]: for any correct process *p*,
   166  and any vote extension *e* that *p* received at height *h*, the computation of
   167  *s<sub>p,h</sub>* does not depend on *e*.
   168  
   169  The call to correct process *p*'s `RequestFinalizeBlock` at height *h*, with block *v<sub>p,h</sub>*
   170  passed as parameter, creates state *s<sub>p,h</sub>*.
   171  Additionally, *p*'s `FinalizeBlock` creates a set of transaction results *T<sub>p,h</sub>*.
   172  
   173  * Requirement 11 [`FinalizeBlock`, determinism-1]: For any correct process *p*,
   174    *s<sub>p,h</sub>* exclusively depends on *s<sub>p,h-1</sub>* and *v<sub>p,h</sub>*.
   175  
   176  * Requirement 12 [`FinalizeBlock`, determinism-2]: For any correct process *p*,
   177    the contents of *T<sub>p,h</sub>* exclusively depend on *s<sub>p,h-1</sub>* and *v<sub>p,h</sub>*.
   178  
   179  Note that Requirements 11 and 12, combined with the Agreement property of consensus ensure
   180  state machine replication, i.e., the Application state evolves consistently at all correct processes.
   181  
   182  Also, notice that neither `PrepareProposal` nor `ExtendVote` have determinism-related
   183  requirements associated.
   184  Indeed, `PrepareProposal` is not required to be deterministic:
   185  
   186  * *u<sub>p</sub>* may depend on *v<sub>p</sub>* and *s<sub>p,h-1</sub>*, but may also depend on other values or operations.
   187  * *v<sub>p</sub> = v<sub>q</sub> &#8655; u<sub>p</sub> = u<sub>q</sub>*.
   188  
   189  Likewise, `ExtendVote` can also be non-deterministic:
   190  
   191  * *e<sup>r</sup><sub>p</sub>* may depend on *w<sup>r</sup><sub>p</sub>* and *s<sub>p,h-1</sub>*,
   192    but may also depend on other values or operations.
   193  * *w<sup>r</sup><sub>p</sub> = w<sup>r</sup><sub>q</sub> &#8655;
   194    e<sup>r</sup><sub>p</sub> = e<sup>r</sup><sub>q</sub>*
   195  
   196  ### Mempool Connection Requirements
   197  
   198  Let *CheckTxCodes<sub>tx,p,h</sub>* denote the set of result codes returned by *p*'s Application,
   199  via `ResponseCheckTx`,
   200  to successive calls to `RequestCheckTx` occurring while the Application is at height *h*
   201  and having transaction *tx* as parameter.
   202  *CheckTxCodes<sub>tx,p,h</sub>* is a set since *p*'s Application may
   203  return different result codes during height *h*.
   204  If *CheckTxCodes<sub>tx,p,h</sub>* is a singleton set, i.e. the Application always returned
   205  the same result code in `ResponseCheckTx` while at height *h*,
   206  we define *CheckTxCode<sub>tx,p,h</sub>* as the singleton value of *CheckTxCodes<sub>tx,p,h</sub>*.
   207  If *CheckTxCodes<sub>tx,p,h</sub>* is not a singleton set, *CheckTxCode<sub>tx,p,h</sub>* is undefined.
   208  Let predicate *OK(CheckTxCode<sub>tx,p,h</sub>)* denote whether *CheckTxCode<sub>tx,p,h</sub>* is `SUCCESS`.
   209  
   210  * Requirement 13 [`CheckTx`, eventual non-oscillation]: For any transaction *tx*,
   211    there exists a boolean value *b*,
   212    and a height *h<sub>stable</sub>* such that,
   213    for any correct process *p*,
   214    *CheckTxCode<sub>tx,p,h</sub>* is defined, and
   215    *OK(CheckTxCode<sub>tx,p,h</sub>) = b*
   216    for any height *h &#8805; h<sub>stable</sub>*.
   217  
   218  Requirement 13 ensures that
   219  a transaction will eventually stop oscillating between `CheckTx` success and failure
   220  if it stays in *p's* mempool for long enough.
   221  This condition on the Application's behavior allows the mempool to ensure that
   222  a transaction will leave the mempool of all full nodes,
   223  either because it is expunged everywhere due to failing `CheckTx` calls,
   224  or because it stays valid long enough to be gossipped, proposed and decided.
   225  Although Requirement 13 defines a global *h<sub>stable</sub>*, application developers
   226  can consider such stabilization height as local to process *p* (*h<sub>p,stable</sub>*),
   227  without loss for generality.
   228  In contrast, the value of *b* MUST be the same across all processes.
   229  
   230  ## Managing the Application state and related topics
   231  
   232  ### Connection State
   233  
   234  CometBFT maintains four concurrent ABCI++ connections, namely
   235  [Consensus Connection](#consensus-connection),
   236  [Mempool Connection](#mempool-connection),
   237  [Info/Query Connection](#infoquery-connection), and
   238  [Snapshot Connection](#snapshot-connection).
   239  It is common for an application to maintain a distinct copy of
   240  the state for each connection, which are synchronized upon `Commit` calls.
   241  
   242  #### Concurrency
   243  
   244  In principle, each of the four ABCI++ connections operates concurrently with one
   245  another. This means applications need to ensure access to state is
   246  thread safe. Both the
   247  [default in-process ABCI client](https://github.com/aakash4dev/cometbft/blob/main/abci/client/local_client.go#L13)
   248  and the
   249  [default Go ABCI server](https://github.com/aakash4dev/cometbft/blob/main/abci/server/socket_server.go#L20)
   250  use a global lock to guard the handling of events across all connections, so they are not
   251  concurrent at all. This means whether your app is compiled in-process with
   252  CometBFT using the `NewLocalClient`, or run out-of-process using the `SocketServer`,
   253  ABCI messages from all connections are received in sequence, one at a
   254  time.
   255  
   256  The existence of this global mutex means Go application developers can get thread safety for application state by routing all reads and writes through the ABCI system. Thus it may be unsafe to expose application state directly to an RPC interface, and unless explicit measures are taken, all queries should be routed through the ABCI Query method.
   257  
   258  <!--
   259  This is no longer the case starting from v0.36.0: the global locks have been removed and it is
   260  up to the Application to synchronize access to its state when handling
   261  ABCI++ methods on all connections.
   262   -->
   263  
   264  <!--
   265   TODO CHeck with Sergio whether this is still the case
   266   -->
   267  <!--
   268  Nevertheless, as all ABCI calls are now synchronous, ABCI messages using the same connection are
   269  still received in sequence.
   270   -->
   271  
   272  #### FinalizeBlock
   273  
   274  When the consensus algorithm decides on a block, CometBFT uses `FinalizeBlock` to send the
   275  decided block's data to the Application, which uses it to transition its state, but MUST NOT persist it;
   276  persisting MUST be done during `Commit`.
   277  
   278  The Application must remember the latest height from which it
   279  has run a successful `Commit` so that it can tell CometBFT where to
   280  pick up from when it recovers from a crash. See information on the Handshake
   281  [here](#crash-recovery).
   282  
   283  #### Commit
   284  
   285  The Application should persist its state during `Commit`, before returning from it.
   286  
   287  Before invoking `Commit`, CometBFT locks the mempool and flushes the mempool connection. This ensures that
   288  no new messages
   289  will be received on the mempool connection during this processing step, providing an opportunity to safely
   290  update all four
   291  connection states to the latest committed state at the same time.
   292  
   293  When `Commit` returns, CometBFT unlocks the mempool.
   294  
   295  WARNING: if the ABCI app logic processing the `Commit` message sends a
   296  `/broadcast_tx_sync` or `/broadcast_tx` and waits for the response
   297  before proceeding, it will deadlock. Executing `broadcast_tx` calls
   298  involves acquiring the mempool lock that CometBFT holds during the `Commit` call.
   299  Synchronous mempool-related calls must be avoided as part of the sequential logic of the
   300  `Commit` function.
   301  
   302  #### Candidate States
   303  
   304  CometBFT calls `PrepareProposal` when it is about to send a proposed block to the network.
   305  Likewise, CometBFT calls `ProcessProposal` upon reception of a proposed block from the
   306  network. The proposed block's data
   307  that is disclosed to the Application by these two methods is the following:
   308  
   309  * the transaction list
   310  * the `LastCommit` referring to the previous block
   311  * the block header's hash (except in `PrepareProposal`, where it is not known yet)
   312  * list of validators that misbehaved
   313  * the block's timestamp
   314  * `NextValidatorsHash`
   315  * Proposer address
   316  
   317  The Application may decide to *immediately* execute the given block (i.e., upon `PrepareProposal`
   318  or `ProcessProposal`). There are two main reasons why the Application may want to do this:
   319  
   320  * *Avoiding invalid transactions in blocks*.
   321    In order to be sure that the block does not contain *any* invalid transaction, there may be
   322    no way other than fully executing the transactions in the block as though it was the *decided*
   323    block.
   324  * *Quick `FinalizeBlock` execution*.
   325    Upon reception of the decided block via `FinalizeBlock`, if that same block was executed
   326    upon `PrepareProposal` or `ProcessProposal` and the resulting state was kept in memory, the
   327    Application can simply apply that state (faster) to the main state, rather than reexecuting
   328    the decided block (slower).
   329  
   330  `PrepareProposal`/`ProcessProposal` can be called many times for a given height. Moreover,
   331  it is not possible to accurately predict which of the blocks proposed in a height will be decided,
   332  being delivered to the Application in that height's `FinalizeBlock`.
   333  Therefore, the state resulting from executing a proposed block, denoted a *candidate state*, should
   334  be kept in memory as a possible final state for that height. When `FinalizeBlock` is called, the Application should
   335  check if the decided block corresponds to one of its candidate states; if so, it will apply it as
   336  its *ExecuteTxState* (see [Consensus Connection](#consensus-connection) below),
   337  which will be persisted during the upcoming `Commit` call.
   338  
   339  Under adverse conditions (e.g., network instability), the consensus algorithm might take many rounds.
   340  In this case, potentially many proposed blocks will be disclosed to the Application for a given height.
   341  By the nature of Tendermint consensus algorithm, currently adopted in CometBFT, the number of proposed blocks received by the Application
   342  for a particular height cannot be bound, so Application developers must act with care and use mechanisms
   343  to bound memory usage. As a general rule, the Application should be ready to discard candidate states
   344  before `FinalizeBlock`, even if one of them might end up corresponding to the
   345  decided block and thus have to be reexecuted upon `FinalizeBlock`.
   346  
   347  ### States and ABCI++ Connections
   348  
   349  #### Consensus Connection
   350  
   351  The Consensus Connection should maintain an *ExecuteTxState* &mdash; the working state
   352  for block execution. It should be updated by the call to `FinalizeBlock`
   353  during block execution and committed to disk as the "latest
   354  committed state" during `Commit`. Execution of a proposed block (via `PrepareProposal`/`ProcessProposal`)
   355  **must not** update the *ExecuteTxState*, but rather be kept as a separate candidate state until `FinalizeBlock`
   356  confirms which of the candidate states (if any) can be used to update *ExecuteTxState*.
   357  
   358  #### Mempool Connection
   359  
   360  The mempool Connection maintains *CheckTxState*. CometBFT sequentially processes an incoming
   361  transaction (via RPC from client or P2P from the gossip layer) against *CheckTxState*.
   362  If the processing does not return any error, the transaction is accepted into the mempool
   363  and CometBFT starts gossipping it.
   364  *CheckTxState* should be reset to the latest committed state
   365  at the end of every `Commit`.
   366  
   367  During the execution of a consensus instance, the *CheckTxState* may be updated concurrently with the
   368  *ExecuteTxState*, as messages may be sent concurrently on the Consensus and Mempool connections.
   369  At the end of the consensus instance, as described above, CometBFT locks the mempool and flushes
   370  the mempool connection before calling `Commit`. This ensures that all pending `CheckTx` calls are
   371  responded to and no new ones can begin.
   372  
   373  After the `Commit` call returns, while still holding the mempool lock, `CheckTx` is run again on all
   374  transactions that remain in the node's local mempool after filtering those included in the block.
   375  Parameter `Type` in `RequestCheckTx`
   376  indicates whether an incoming transaction is new (`CheckTxType_New`), or a
   377  recheck (`CheckTxType_Recheck`).
   378  
   379  Finally, after re-checking transactions in the mempool, CometBFT will unlock
   380  the mempool connection. New transactions are once again able to be processed through `CheckTx`.
   381  
   382  Note that `CheckTx` is just a weak filter to keep invalid transactions out of the mempool and,
   383  ultimately, ouf of the blockchain.
   384  Since the transaction cannot be guaranteed to be checked against the exact same state as it
   385  will be executed as part of a (potential) decided block, `CheckTx` shouldn't check *everything*
   386  that affects the transaction's validity, in particular those checks whose validity may depend on
   387  transaction ordering. `CheckTx` is weak because a Byzantine node need not care about `CheckTx`;
   388  it can propose a block full of invalid transactions if it wants. The mechanism ABCI++ has
   389  in place for dealing with such behavior is `ProcessProposal`.
   390  
   391  ##### Replay Protection
   392  
   393  It is possible for old transactions to be sent again to the Application. This is typically
   394  undesirable for all transactions, except for a generally small subset of them which are idempotent.
   395  
   396  The mempool has a mechanism to prevent duplicated transactions from being processed.
   397  This mechanism is nevertheless best-effort (currently based on the indexer)
   398  and does not provide any guarantee of non duplication.
   399  It is thus up to the Application to implement an application-specific
   400  replay protection mechanism with strong guarantees as part of the logic in `CheckTx`.
   401  
   402  #### Info/Query Connection
   403  
   404  The Info (or Query) Connection should maintain a `QueryState`. This connection has two
   405  purposes: 1) having the application answer the queries CometBFT receives from users
   406  (see section [Query](#query)),
   407  and 2) synchronizing CometBFT and the Application at start up time (see
   408  [Crash Recovery](#crash-recovery))
   409  or after state sync (see [State Sync](#state-sync)).
   410  
   411  `QueryState` is a read-only copy of *ExecuteTxState* as it was after the last
   412  `Commit`, i.e.
   413  after the full block has been processed and the state committed to disk.
   414  
   415  #### Snapshot Connection
   416  
   417  The Snapshot Connection is used to serve state sync snapshots for other nodes
   418  and/or restore state sync snapshots to a local node being bootstrapped.
   419  Snapshot management is optional: an Application may choose not to implement it.
   420  
   421  For more information, see Section [State Sync](#state-sync).
   422  
   423  ### Transaction Results
   424  
   425  The Application is expected to return a list of
   426  [`ExecTxResult`](./abci%2B%2B_methods.md#exectxresult) in
   427  [`ResponseFinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). The list of transaction
   428  results MUST respect the same order as the list of transactions delivered via
   429  [`RequestFinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock).
   430  This section discusses the fields inside this structure, along with the fields in
   431  [`ResponseCheckTx`](./abci%2B%2B_methods.md#checktx),
   432  whose semantics are similar.
   433  
   434  The `Info` and `Log` fields are
   435  non-deterministic values for debugging/convenience purposes. CometBFT logs them but they
   436  are otherwise ignored.
   437  
   438  #### Gas
   439  
   440  Ethereum introduced the notion of *gas* as an abstract representation of the
   441  cost of the resources consumed by nodes when processing a transaction. Every operation in the
   442  Ethereum Virtual Machine uses some amount of gas.
   443  Gas has a market-variable price based on which miners can accept or reject to execute a
   444  particular operation.
   445  
   446  Users propose a maximum amount of gas for their transaction; if the transaction uses less, they get
   447  the difference credited back. CometBFT adopts a similar abstraction,
   448  though uses it only optionally and weakly, allowing applications to define
   449  their own sense of the cost of execution.
   450  
   451  In CometBFT, the [ConsensusParams.Block.MaxGas](#consensus-parameters) limits the amount of
   452  total gas that can be used by all transactions in a block.
   453  The default value is `-1`, which means the block gas limit is not enforced, or that the concept of
   454  gas is meaningless.
   455  
   456  Responses contain a `GasWanted` and `GasUsed` field. The former is the maximum
   457  amount of gas the sender of a transaction is willing to use, and the latter is how much it actually
   458  used. Applications should enforce that `GasUsed <= GasWanted` &mdash; i.e. transaction execution
   459  or validation should fail before it can use more resources than it requested.
   460  
   461  When `MaxGas > -1`, CometBFT enforces the following rules:
   462  
   463  * `GasWanted <= MaxGas` for every transaction in the mempool
   464  * `(sum of GasWanted in a block) <= MaxGas` when proposing a block
   465  
   466  If `MaxGas == -1`, no rules about gas are enforced.
   467  
   468  In v0.34.x and earlier versions, CometBFT does not enforce anything about Gas in consensus,
   469  only in the mempool.
   470  This means it does not guarantee that committed blocks satisfy these rules.
   471  It is the application's responsibility to return non-zero response codes when gas limits are exceeded
   472  when executing the transactions of a block.
   473  Since the introduction of `PrepareProposal` and `ProcessProposal` in v.0.37.x, it is now possible
   474  for the Application to enforce that all blocks proposed (and voted for) in consensus &mdash; and thus all
   475  blocks decided &mdash; respect the `MaxGas` limits described above.
   476  
   477  Since the Application should enforce that `GasUsed <= GasWanted` when executing a transaction, and
   478  it can use `PrepareProposal` and `ProcessProposal` to enforce that `(sum of GasWanted in a block) <= MaxGas`
   479  in all proposed or prevoted blocks,
   480  we have:
   481  
   482  * `(sum of GasUsed in a block) <= MaxGas` for every block
   483  
   484  The `GasUsed` field is ignored by CometBFT.
   485  
   486  #### Specifics of `ResponseCheckTx`
   487  
   488  If `Code != 0`, it will be rejected from the mempool and hence
   489  not broadcasted to other peers and not included in a proposal block.
   490  
   491  `Data` contains the result of the `CheckTx` transaction execution, if any. It does not need to be
   492  deterministic since, given a transaction, nodes' Applications
   493  might have a different *CheckTxState* values when they receive it and check their validity
   494  via `CheckTx`.
   495  CometBFT ignores this value in `ResponseCheckTx`.
   496  
   497  From v0.34.x on, there is a `Priority` field in `ResponseCheckTx` that can be
   498  used to explicitly prioritize transactions in the mempool for inclusion in a block
   499  proposal.
   500  
   501  #### Specifics of `ExecTxResult`
   502  
   503  `FinalizeBlock` is the workhorse of the blockchain. CometBFT delivers the decided block,
   504  including the list of all its transactions synchronously to the Application.
   505  The block delivered (and thus the transaction order) is the same at all correct nodes as guaranteed
   506  by the Agreement property of consensus.
   507  
   508  The `Data` field in `ExecTxResult` contains an array of bytes with the transaction result.
   509  It must be deterministic (i.e., the same value must be returned at all nodes), but it can contain arbitrary
   510  data. Likewise, the value of `Code` must be deterministic.
   511  If `Code != 0`, the transaction will be marked invalid,
   512  though it is still included in the block. Invalid transactions are not indexed, as they are
   513  considered analogous to those that failed `CheckTx`.
   514  
   515  Both the `Code` and `Data` are included in a structure that is hashed into the
   516  `LastResultsHash` of the block header in the next height.
   517  
   518  `Events` include any events for the execution, which CometBFT will use to index
   519  the transaction by. This allows transactions to be queried according to what
   520  events took place during their execution.
   521  
   522  ### Updating the Validator Set
   523  
   524  The application may set the validator set during
   525  [`InitChain`](./abci%2B%2B_methods.md#initchain), and may update it during
   526  [`FinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). In both cases, a structure of type
   527  [`ValidatorUpdate`](./abci%2B%2B_methods.md#validatorupdate) is returned.
   528  
   529  The `InitChain` method, used to initialize the Application, can return a list of validators.
   530  If the list is empty, CometBFT will use the validators loaded from the genesis
   531  file.
   532  If the list returned by `InitChain` is not empty, CometBFT will use its contents as the validator set.
   533  This way the application can set the initial validator set for the
   534  blockchain.
   535  
   536  Applications must ensure that a single set of validator updates does not contain duplicates, i.e.
   537  a given public key can only appear once within a given update. If an update includes
   538  duplicates, the block execution will fail irrecoverably.
   539  
   540  Structure `ValidatorUpdate` contains a public key, which is used to identify the validator:
   541  The public key currently supports three types:
   542  
   543  * `ed25519`
   544  * `secp256k1`
   545  * `sr25519`
   546  
   547  Structure `ValidatorUpdate` also contains an `ìnt64` field denoting the validator's new power.
   548  Applications must ensure that
   549  `ValidatorUpdate` structures abide by the following rules:
   550  
   551  * power must be non-negative
   552  * if power is set to 0, the validator must be in the validator set; it will be removed from the set
   553  * if power is greater than 0:
   554      * if the validator is not in the validator set, it will be added to the
   555        set with the given power
   556      * if the validator is in the validator set, its power will be adjusted to the given power
   557  * the total power of the new validator set must not exceed `MaxTotalVotingPower`, where
   558    `MaxTotalVotingPower = MaxInt64 / 8`
   559  
   560  Note the updates returned after processing the block at height `H` will only take effect
   561  at block `H+2` (see Section [Methods](./abci%2B%2B_methods.md)).
   562  
   563  ### Consensus Parameters
   564  
   565  `ConsensusParams` are global parameters that apply to all validators in a blockchain.
   566  They enforce certain limits in the blockchain, like the maximum size
   567  of blocks, amount of gas used in a block, and the maximum acceptable age of
   568  evidence. They can be set in
   569  [`InitChain`](./abci%2B%2B_methods.md#initchain), and updated in
   570  [`FinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock).
   571  These parameters are deterministically set and/or updated by the Application, so
   572  all full nodes have the same value at a given height.
   573  
   574  #### List of Parameters
   575  
   576  These are the current consensus parameters (as of v0.37.x):
   577  
   578  1. [BlockParams.MaxBytes](#blockparamsmaxbytes)
   579  2. [BlockParams.MaxGas](#blockparamsmaxgas)
   580  3. [EvidenceParams.MaxAgeDuration](#evidenceparamsmaxageduration)
   581  4. [EvidenceParams.MaxAgeNumBlocks](#evidenceparamsmaxagenumblocks)
   582  5. [EvidenceParams.MaxBytes](#evidenceparamsmaxbytes)
   583  6. [ValidatorParams.PubKeyTypes](#validatorparamspubkeytypes)
   584  7. [VersionParams.App](#versionparamsapp)
   585  <!--
   586   6. [SynchronyParams.MessageDelay](#synchronyparamsmessagedelay)
   587  7. [SynchronyParams.Precision](#synchronyparamsprecision)
   588  8. [TimeoutParams.Propose](#timeoutparamspropose)
   589  9. [TimeoutParams.ProposeDelta](#timeoutparamsproposedelta)
   590  10. [TimeoutParams.Vote](#timeoutparamsvote)
   591  11. [TimeoutParams.VoteDelta](#timeoutparamsvotedelta)
   592  12. [TimeoutParams.Commit](#timeoutparamscommit)
   593  13. [TimeoutParams.BypassCommitTimeout](#timeoutparamsbypasscommittimeout) 
   594  -->
   595  
   596  ##### BlockParams.MaxBytes
   597  
   598  The maximum size of a complete Protobuf encoded block.
   599  This is enforced by the consensus algorithm.
   600  
   601  This implies a maximum transaction size that is this `MaxBytes`, less the expected size of
   602  the header, the validator set, and any included evidence in the block.
   603  
   604  If the Application wants full control over the size of blocks,
   605  it can do so by enforcing a byte limit set up at the Application level.
   606  This Application-internal limit is used by `PrepareProposal` to bound the total size
   607  of transactions it returns, and by `ProcessProposal` to reject any received block
   608  whose total transaction size is bigger than the enforced limit.
   609  In such case, the Application MAY set `MaxBytes` to -1.
   610  
   611  If the Application sets value -1, consensus will:
   612  
   613  - consider that the actual value to enforce is 100 MB
   614  - will provide *all* transactions in the mempool in calls to `PrepareProposal`
   615  
   616  Must have `MaxBytes == -1` OR `0 < MaxBytes <= 100 MB`.
   617  
   618  ##### BlockParams.MaxGas
   619  
   620  The maximum of the sum of `GasWanted` that will be allowed in a proposed block.
   621  This is *not* enforced by the consensus algorithm.
   622  It is left to the Application to enforce (ie. if transactions are included past the
   623  limit, they should return non-zero codes). It is used by CometBFT to limit the
   624  transactions included in a proposed block.
   625  
   626  Must have `MaxGas >= -1`.
   627  If `MaxGas == -1`, no limit is enforced.
   628  
   629  ##### EvidenceParams.MaxAgeDuration
   630  
   631  This is the maximum age of evidence in time units.
   632  This is enforced by the consensus algorithm.
   633  
   634  If a block includes evidence older than this (AND the evidence was created more
   635  than `MaxAgeNumBlocks` ago), the block will be rejected (validators won't vote
   636  for it).
   637  
   638  Must have `MaxAgeDuration > 0`.
   639  
   640  ##### EvidenceParams.MaxAgeNumBlocks
   641  
   642  This is the maximum age of evidence in blocks.
   643  This is enforced by the consensus algorithm.
   644  
   645  If a block includes evidence older than this (AND the evidence was created more
   646  than `MaxAgeDuration` ago), the block will be rejected (validators won't vote
   647  for it).
   648  
   649  Must have `MaxAgeNumBlocks > 0`.
   650  
   651  ##### EvidenceParams.MaxBytes
   652  
   653  This is the maximum size of total evidence in bytes that can be committed to a
   654  single block. It should fall comfortably under the max block bytes.
   655  
   656  Its value must not exceed the size of
   657  a block minus its overhead ( ~ `BlockParams.MaxBytes`).
   658  
   659  Must have `MaxBytes > 0`.
   660  
   661  ##### ValidatorParams.PubKeyTypes
   662  
   663  The parameter restricts the type of keys validators can use. The parameter uses ABCI pubkey naming, not Amino names.
   664  
   665  ##### VersionParams.App
   666  
   667  This is the version of the ABCI application.
   668  <!--
   669  ##### SynchronyParams.MessageDelay
   670  
   671  This sets a bound on how long a proposal message may take to reach all
   672  validators on a network and still be considered valid.
   673  
   674  This parameter is part of the
   675  [proposer-based timestamps](../consensus/proposer-based-timestamp)
   676  (PBTS) algorithm.
   677  
   678  
   679  ##### SynchronyParams.Precision
   680  
   681  This sets a bound on how skewed a proposer's clock may be from any validator
   682  on the network while still producing valid proposals.
   683  
   684  This parameter is part of the
   685  [proposer-based timestamps](../consensus/proposer-based-timestamp)
   686  (PBTS) algorithm.
   687  
   688  
   689  ##### TimeoutParams.Propose
   690  
   691  Timeout in ms of the propose step of the consensus algorithm.
   692  This value is the initial timeout at every height (round 0).
   693  
   694  The value in subsequent rounds is modified by parameter `ProposeDelta`.
   695  When a new height is started, the `Propose` timeout value is reset to this
   696  parameter.
   697  
   698  If a node waiting for a proposal message does not receive one matching its
   699  current height and round before this timeout, the node will issue a
   700  `nil` prevote for the round and advance to the next step.
   701  
   702  ##### TimeoutParams.ProposeDelta
   703  
   704  Increment in ms to be added to the `Propose` timeout every time the
   705  consensus algorithm advances one round in a given height.
   706  
   707  When a new height is started, the `Propose` timeout value is reset.
   708  
   709  ##### TimeoutParams.Vote
   710  
   711  Timeout in ms of the prevote and precommit steps of the consensus
   712  algorithm.
   713  This value is the initial timeout at every height (round 0).
   714  
   715  The value in subsequent rounds is modified by parameter `VoteDelta`.
   716  When a new height is started, the `Vote` timeout value is reset to this
   717  parameter.
   718  
   719  The `Vote` timeout does not begin until a quorum of votes has been received.
   720  Once a quorum of votes has been seen and this timeout elapses, Tendermint will
   721  proceed to the next step of the consensus algorithm. If Tendermint receives
   722  all of the remaining votes before the end of the timeout, it will proceed
   723  to the next step immediately.
   724  
   725  ##### TimeoutParams.VoteDelta
   726  
   727  Increment in ms to be added to the `Vote` timeout every time the
   728  consensus algorithm advances one round in a given height.
   729  
   730  When a new height is started, the `Vote` timeout value is reset.
   731  
   732  ##### TimeoutParams.Commit
   733  
   734  This configures how long the consensus algorithm will wait after receiving a quorum of
   735  precommits before beginning consensus for the next height. This can be
   736  used to allow slow precommits to arrive for inclusion in the next height
   737  before progressing.
   738  
   739  ##### TimeoutParams.BypassCommitTimeout
   740  
   741  This configures the node to proceed immediately to the next height once the
   742  node has received all precommits for a block, forgoing the remaining commit timeout.
   743  Setting this parameter to `false` (the default) causes Tendermint to wait
   744  for the full commit timeout configured in `TimeoutParams.Commit`.
   745  -->
   746  
   747  ##### ABCIParams.VoteExtensionsEnableHeight
   748  
   749  This parameter is either 0 or a positive height at which vote extensions
   750  become mandatory. If the value is zero (which is the default), vote
   751  extensions are not required. Otherwise, at all heights greater than the
   752  configured height `H` vote extensions must be present (even if empty).
   753  When the configured height `H` is reached, `PrepareProposal` will not
   754  include vote extensions yet, but `ExtendVote` and `VerifyVoteExtension` will
   755  be called. Then, when reaching height `H+1`, `PrepareProposal` will
   756  include the vote extensions from height `H`. For all heights after `H`
   757  
   758  * vote extensions cannot be disabled,
   759  * they are mandatory: all precommit messages sent MUST have an extension
   760    attached. Nevertheless, the application MAY provide 0-length
   761    extensions.
   762  
   763  Must always be set to a future height. Once set to a value different from
   764  0, its value must not be changed.
   765  
   766  #### Updating Consensus Parameters
   767  
   768  The application may set the `ConsensusParams` during
   769  [`InitChain`](./abci%2B%2B_methods.md#initchain),
   770  and update them during
   771  [`FinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock).
   772  If the `ConsensusParams` is empty, it will be ignored. Each field
   773  that is not empty will be applied in full. For instance, if updating the
   774  `Block.MaxBytes`, applications must also set the other `Block` fields (like
   775  `Block.MaxGas`), even if they are unchanged, as they will otherwise cause the
   776  value to be updated to the default.
   777  
   778  ##### `InitChain`
   779  
   780  `ResponseInitChain` includes a `ConsensusParams` parameter.
   781  If `ConsensusParams` is `nil`, CometBFT will use the params loaded in the genesis
   782  file. If `ConsensusParams` is not `nil`, CometBFT will use it.
   783  This way the application can determine the initial consensus parameters for the
   784  blockchain.
   785  
   786  ##### `FinalizeBlock`, `PrepareProposal`/`ProcessProposal`
   787  
   788  `ResponseFinalizeBlock` accepts a `ConsensusParams` parameter.
   789  If `ConsensusParams` is `nil`, CometBFT will do nothing.
   790  If `ConsensusParams` is not `nil`, CometBFT will use it.
   791  This way the application can update the consensus parameters over time.
   792  
   793  The updates returned in block `H` will take effect right away for block
   794  `H+1`.
   795  
   796  ### `Query`
   797  
   798  `Query` is a generic method with lots of flexibility to enable diverse sets
   799  of queries on application state. CometBFT makes use of `Query` to filter new peers
   800  based on ID and IP, and exposes `Query` to the user over RPC.
   801  
   802  Note that calls to `Query` are not replicated across nodes, but rather query the
   803  local node's state - hence they may return stale reads. For reads that require
   804  consensus, use a transaction.
   805  
   806  The most important use of `Query` is to return Merkle proofs of the application state at some height
   807  that can be used for efficient application-specific light-clients.
   808  
   809  Note CometBFT has technically no requirements from the `Query`
   810  message for normal operation - that is, the ABCI app developer need not implement
   811  Query functionality if they do not wish to.
   812  
   813  #### Query Proofs
   814  
   815  The CometBFT block header includes a number of hashes, each providing an
   816  anchor for some type of proof about the blockchain. The `ValidatorsHash` enables
   817  quick verification of the validator set, the `DataHash` gives quick
   818  verification of the transactions included in the block.
   819  
   820  The `AppHash` is unique in that it is application specific, and allows for
   821  application-specific Merkle proofs about the state of the application.
   822  While some applications keep all relevant state in the transactions themselves
   823  (like Bitcoin and its UTXOs), others maintain a separated state that is
   824  computed deterministically *from* transactions, but is not contained directly in
   825  the transactions themselves (like Ethereum contracts and accounts).
   826  For such applications, the `AppHash` provides a much more efficient way to verify light-client proofs.
   827  
   828  ABCI applications can take advantage of more efficient light-client proofs for
   829  their state as follows:
   830  
   831  * return the Merkle root of the deterministic application state in
   832    `ResponseFinalizeBlock.Data`. This Merkle root will be included as the `AppHash` in the next block.
   833  * return efficient Merkle proofs about that application state in `ResponseQuery.Proof`
   834    that can be verified using the `AppHash` of the corresponding block.
   835  
   836  For instance, this allows an application's light-client to verify proofs of
   837  absence in the application state, something which is much less efficient to do using the block hash.
   838  
   839  Some applications (eg. Ethereum, Cosmos-SDK) have multiple "levels" of Merkle trees,
   840  where the leaves of one tree are the root hashes of others. To support this, and
   841  the general variability in Merkle proofs, the `ResponseQuery.Proof` has some minimal structure:
   842  
   843  ```protobuf
   844  message ProofOps {
   845    repeated ProofOp ops = 1
   846  }
   847  
   848  message ProofOp {
   849    string type = 1;
   850    bytes key   = 2;
   851    bytes data  = 3;
   852  }
   853  ```
   854  
   855  Each `ProofOp` contains a proof for a single key in a single Merkle tree, of the specified `type`.
   856  This allows ABCI to support many different kinds of Merkle trees, encoding
   857  formats, and proofs (eg. of presence and absence) just by varying the `type`.
   858  The `data` contains the actual encoded proof, encoded according to the `type`.
   859  When verifying the full proof, the root hash for one ProofOp is the value being
   860  verified for the next ProofOp in the list. The root hash of the final ProofOp in
   861  the list should match the `AppHash` being verified against.
   862  
   863  #### Peer Filtering
   864  
   865  When CometBFT connects to a peer, it sends two queries to the ABCI application
   866  using the following paths, with no additional data:
   867  
   868  * `/p2p/filter/addr/<IP:PORT>`, where `<IP:PORT>` denote the IP address and
   869    the port of the connection
   870  * `p2p/filter/id/<ID>`, where `<ID>` is the peer node ID (ie. the
   871    pubkey.Address() for the peer's PubKey)
   872  
   873  If either of these queries return a non-zero ABCI code, CometBFT will refuse
   874  to connect to the peer.
   875  
   876  #### Paths
   877  
   878  Queries are directed at paths, and may optionally include additional data.
   879  
   880  The expectation is for there to be some number of high level paths
   881  differentiating concerns, like `/p2p`, `/store`, and `/app`. Currently,
   882  CometBFT only uses `/p2p`, for filtering peers. For more advanced use, see the
   883  implementation of
   884  [Query in the Cosmos-SDK](https://github.com/cosmos/cosmos-sdk/blob/e2037f7696fed4fdd4bc076f9e7053fe8178a881/baseapp/abci.go#L557-L565).
   885  
   886  ### Crash Recovery
   887  
   888  CometBFT and the application are expected to crash together and there should not 
   889  exist a scenario where the application has persisted state of a height greater than the
   890  latest height persisted by CometBFT.
   891  
   892  In practice, persisting the state of a height consists of three steps, the last of which 
   893  is the call to the application's `Commit` method, the only place where the application is expected to
   894  persist/commit its state.
   895  On startup (upon recovery), CometBFT calls the `Info` method on the Info Connection to get the latest
   896  committed state of the app. The app MUST return information consistent with the
   897  last block for which it successfully completed `Commit`. 
   898  
   899  The three steps performed before the state of a height is considered persisted are: 
   900  - The block is stored by CometBFT in the blockstore
   901  - CometBFT has stored the state returned by the application through `FinalizeBlockResponse`
   902  - The application has committed its state within `Commit`. 
   903    
   904  The following diagram depicts the order in which these events happen, and the corresponding
   905  ABCI functions that are called and executed by CometBFT and the application:
   906  
   907  
   908  ``` 
   909  APP:                                              Execute block                         Persist application state
   910                                                   /     return ResultFinalizeBlock            /
   911                                                  /                                           /  
   912  Event: ------------- block_stored ------------ / ------------ state_stored --------------- / ----- app_persisted_state
   913                            |                   /                   |                       /        |
   914  CometBFT: Decide --- Persist block -- Call FinalizeBlock - Persist results ---------- Call Commit -- 
   915              on        in the                                (txResults, validator
   916             Block      block store                              updates...)
   917  
   918  ```
   919  
   920  As these three steps are not atomic, we observe different cases based on which steps have been executed
   921  before the crash occurred
   922  (we assume that at least `block_stored` has been executed, otherwise, there is no state persisted, 
   923  and the operations for this height are repeated entirely):
   924  
   925  - `block_stored`: we replay `FinalizeBlock` and the steps afterwards.
   926  - `block_stored` and `state_stored`: As the app did not persist its state within `Commit`, we need to re-execute
   927    `FinalizeBlock` to retrieve the results and compare them to the state stored by CometBFT within `state_stored`. 
   928    The expected case is that the states will match, otherwise CometBFT panics.
   929  - `block_stored`, `state_stored`, `app_persisted_state`: we move on to the next height. 
   930  
   931  Based on the sequence of these events, CometBFT will panic if any of the steps in the sequence happen out of order,
   932  that is if: 
   933  - The application has persisted a block at a height higher than the blocked saved during `state_stored`.
   934  - The `block_stored` step persisted a block at a height smaller than the `state_stored`
   935  - And the difference between the heights of the blocks persisted by `state_stored` and `block_stored` is more 
   936  than 1 (this corresponds to a scenario where we stored two blocks in the block store but never persisted the state of the first
   937  block, which should never happen).
   938  
   939  A special case is when a crash happens before the first block is committed - that is, after calling 
   940  `InitChain`. In that case, the application's state should still be at height 0 and thus `InitChain`
   941  will be called again. 
   942  
   943  
   944  ### State Sync
   945  
   946  A new node joining the network can simply join consensus at the genesis height and replay all
   947  historical blocks until it is caught up. However, for large chains this can take a significant
   948  amount of time, often on the order of days or weeks.
   949  
   950  State sync is an alternative mechanism for bootstrapping a new node, where it fetches a snapshot
   951  of the state machine at a given height and restores it. Depending on the application, this can
   952  be several orders of magnitude faster than replaying blocks.
   953  
   954  Note that state sync does not currently backfill historical blocks, so the node will have a
   955  truncated block history - users are advised to consider the broader network implications of this in
   956  terms of block availability and auditability. This functionality may be added in the future.
   957  
   958  For details on the specific ABCI calls and types, see the
   959  [methods](abci%2B%2B_methods.md) section.
   960  
   961  #### Taking Snapshots
   962  
   963  Applications that want to support state syncing must take state snapshots at regular intervals. How
   964  this is accomplished is entirely up to the application. A snapshot consists of some metadata and
   965  a set of binary chunks in an arbitrary format:
   966  
   967  * `Height (uint64)`: The height at which the snapshot is taken. It must be taken after the given
   968    height has been committed, and must not contain data from any later heights.
   969  
   970  * `Format (uint32)`: An arbitrary snapshot format identifier. This can be used to version snapshot
   971    formats, e.g. to switch from Protobuf to MessagePack for serialization. The application can use
   972    this when restoring to choose whether to accept or reject a snapshot.
   973  
   974  * `Chunks (uint32)`: The number of chunks in the snapshot. Each chunk contains arbitrary binary
   975    data, and should be less than 16 MB; 10 MB is a good starting point.
   976  
   977  * `Hash ([]byte)`: An arbitrary hash of the snapshot. This is used to check whether a snapshot is
   978    the same across nodes when downloading chunks.
   979  
   980  * `Metadata ([]byte)`: Arbitrary snapshot metadata, e.g. chunk hashes for verification or any other
   981    necessary info.
   982  
   983  For a snapshot to be considered the same across nodes, all of these fields must be identical. When
   984  sent across the network, snapshot metadata messages are limited to 4 MB.
   985  
   986  When a new node is running state sync and discovering snapshots, CometBFT will query an existing
   987  application via the ABCI `ListSnapshots` method to discover available snapshots, and load binary
   988  snapshot chunks via `LoadSnapshotChunk`. The application is free to choose how to implement this
   989  and which formats to use, but must provide the following guarantees:
   990  
   991  * **Consistent:** A snapshot must be taken at a single isolated height, unaffected by
   992    concurrent writes. This can be accomplished by using a data store that supports ACID
   993    transactions with snapshot isolation.
   994  
   995  * **Asynchronous:** Taking a snapshot can be time-consuming, so it must not halt chain progress,
   996    for example by running in a separate thread.
   997  
   998  * **Deterministic:** A snapshot taken at the same height in the same format must be identical
   999    (at the byte level) across nodes, including all metadata. This ensures good availability of
  1000    chunks, and that they fit together across nodes.
  1001  
  1002  A very basic approach might be to use a datastore with MVCC transactions (such as RocksDB),
  1003  start a transaction immediately after block commit, and spawn a new thread which is passed the
  1004  transaction handle. This thread can then export all data items, serialize them using e.g.
  1005  Protobuf, hash the byte stream, split it into chunks, and store the chunks in the file system
  1006  along with some metadata - all while the blockchain is applying new blocks in parallel.
  1007  
  1008  A more advanced approach might include incremental verification of individual chunks against the
  1009  chain app hash, parallel or batched exports, compression, and so on.
  1010  
  1011  Old snapshots should be removed after some time - generally only the last two snapshots are needed
  1012  (to prevent the last one from being removed while a node is restoring it).
  1013  
  1014  #### Bootstrapping a Node
  1015  
  1016  An empty node can be state synced by setting the configuration option `statesync.enabled =
  1017  true`. The node also needs the chain genesis file for basic chain info, and configuration for
  1018  light client verification of the restored snapshot: a set of CometBFT RPC servers, and a
  1019  trusted header hash and corresponding height from a trusted source, via the `statesync`
  1020  configuration section.
  1021  
  1022  Once started, the node will connect to the P2P network and begin discovering snapshots. These
  1023  will be offered to the local application via the `OfferSnapshot` ABCI method. Once a snapshot
  1024  is accepted CometBFT will fetch and apply the snapshot chunks. After all chunks have been
  1025  successfully applied, CometBFT verifies the app's `AppHash` against the chain using the light
  1026  client, then switches the node to normal consensus operation.
  1027  
  1028  ##### Snapshot Discovery
  1029  
  1030  When the empty node joins the P2P network, it asks all peers to report snapshots via the
  1031  `ListSnapshots` ABCI call (limited to 10 per node). After some time, the node picks the most
  1032  suitable snapshot (generally prioritized by height, format, and number of peers), and offers it
  1033  to the application via `OfferSnapshot`. The application can choose a number of responses,
  1034  including accepting or rejecting it, rejecting the offered format, rejecting the peer who sent
  1035  it, and so on. CometBFT will keep discovering and offering snapshots until one is accepted or
  1036  the application aborts.
  1037  
  1038  ##### Snapshot Restoration
  1039  
  1040  Once a snapshot has been accepted via `OfferSnapshot`, CometBFT begins downloading chunks from
  1041  any peers that have the same snapshot (i.e. that have identical metadata fields). Chunks are
  1042  spooled in a temporary directory, and then given to the application in sequential order via
  1043  `ApplySnapshotChunk` until all chunks have been accepted.
  1044  
  1045  The method for restoring snapshot chunks is entirely up to the application.
  1046  
  1047  During restoration, the application can respond to `ApplySnapshotChunk` with instructions for how
  1048  to continue. This will typically be to accept the chunk and await the next one, but it can also
  1049  ask for chunks to be refetched (either the current one or any number of previous ones), P2P peers
  1050  to be banned, snapshots to be rejected or retried, and a number of other responses - see the ABCI
  1051  reference for details.
  1052  
  1053  If CometBFT fails to fetch a chunk after some time, it will reject the snapshot and try a
  1054  different one via `OfferSnapshot` - the application can choose whether it wants to support
  1055  restarting restoration, or simply abort with an error.
  1056  
  1057  ##### Snapshot Verification
  1058  
  1059  Once all chunks have been accepted, CometBFT issues an `Info` ABCI call to retrieve the
  1060  `LastBlockAppHash`. This is compared with the trusted app hash from the chain, retrieved and
  1061  verified using the light client. CometBFT also checks that `LastBlockHeight` corresponds to the
  1062  height of the snapshot.
  1063  
  1064  This verification ensures that an application is valid before joining the network. However, the
  1065  snapshot restoration may take a long time to complete, so applications may want to employ additional
  1066  verification during the restore to detect failures early. This might e.g. include incremental
  1067  verification of each chunk against the app hash (using bundled Merkle proofs), checksums to
  1068  protect against data corruption by the disk or network, and so on. However, it is important to
  1069  note that the only trusted information available is the app hash, and all other snapshot metadata
  1070  can be spoofed by adversaries.
  1071  
  1072  Apps may also want to consider state sync denial-of-service vectors, where adversaries provide
  1073  invalid or harmful snapshots to prevent nodes from joining the network. The application can
  1074  counteract this by asking CometBFT to ban peers. As a last resort, node operators can use
  1075  P2P configuration options to whitelist a set of trusted peers that can provide valid snapshots.
  1076  
  1077  ##### Transition to Consensus
  1078  
  1079  Once the snapshots have all been restored, CometBFT gathers additional information necessary for
  1080  bootstrapping the node (e.g. chain ID, consensus parameters, validator sets, and block headers)
  1081  from the genesis file and light client RPC servers. It also calls `Info` to verify the following:
  1082  
  1083  * that the app hash from the snapshot it has delivered to the Application matches the apphash
  1084    stored in the next height's block
  1085  * that the version that the Application returns in `ResponseInfo` matches the version in the
  1086    current height's block header
  1087  
  1088  Once the state machine has been restored and CometBFT has gathered this additional
  1089  information, it transitions to consensus. As of ABCI 2.0, CometBFT ensures the neccessary conditions
  1090  to switch are met [RFC-100](./../../docs/rfc/rfc-100-abci-vote-extension-propag.md#base-implementation-persist-and-propagate-extended-commit-history).
  1091  From the application's point of view, these operations are transparent, unless the application has just upgraded to ABCI 2.0. 
  1092  In that case, the application needs to be properly configured and aware of certain constraints in terms of when
  1093  to provide vote extensions. More details can be found in the section below. 
  1094  
  1095  Once a node switches to consensus, it operates like any other node, apart from having a truncated block history at the height of the restored snapshot.
  1096  
  1097  ## Application configuration required to switch to ABCI 2.0
  1098  
  1099  Introducing vote extensions requires changes to the configuration of the application.
  1100  
  1101  First of all, switching to a version of CometBFT with vote extensions, requires a coordinated upgrade. 
  1102  For a detailed description on the upgrade path, please refer to the corresponding 
  1103  [section](./../../docs/rfc/rfc-100-abci-vote-extension-propag.md#upgrade-path) in RFC-100.
  1104  
  1105  There is a newly introduced [**consensus parameter**](./abci%2B%2B_app_requirements.md#abciparamsvoteextensionsenableheight): `VoteExtensionsEnableHeight`. 
  1106  This parameter represents the height at which vote extensions are 
  1107  required for consensus to proceed, with 0 being the default value (no vote extensions).
  1108  A chain can enable vote extensions either:
  1109  * at genesis by setting `VoteExtensionsEnableHeight` to be equal, e.g., to the `InitialHeight`
  1110  * or via the application logic by changing the `ConsensusParam` to configure the
  1111  `VoteExtensionsEnableHeight`.
  1112  
  1113  Once the (coordinated) upgrade to ABCI 2.0 has taken place, at height  *h<sub>u</sub>*,
  1114  the value of `VoteExtensionsEnableHeight` MAY be set to some height, *h<sub>e</sub>*,
  1115  which MUST be higher than the current height of the chain. Thus the earliest value for 
  1116   *h<sub>e</sub>* is  *h<sub>u</sub>* + 1.
  1117  
  1118  Once a node reaches the configured height,
  1119  for all heights *h ≥ h<sub>e</sub>*, the consensus algorithm will
  1120  reject as invalid any precommit messages that do not have signed vote extension data.
  1121  If the application requires it, a 0-length vote extension is allowed, but it MUST be signed
  1122  and present in the precommit message.
  1123  Likewise, for all heights *h < h<sub>e</sub>*, any precommit messages that *do* have vote extensions
  1124  will also be rejected as malformed.
  1125  Height *h<sub>e</sub>* is somewhat special, as calls to `PrepareProposal` MUST NOT
  1126  have vote extension data, but all precommit votes in that height MUST carry a vote extension,
  1127  even if the extension is `nil`. 
  1128  Height *h<sub>e</sub> + 1* is the first height for which `PrepareProposal` MUST have vote
  1129  extension data and all precommit votes in that height MUST have a vote extension.
  1130  
  1131  Corollary, [CometBFT will decide](./abci%2B%2B_comet_expected_behavior.md#handling-upgrades-to-abci-20) which data to store, and require for successful operations, based on the current height
  1132  of the chain.