github.com/KYVENetwork/cometbft/v38@v38.0.3/spec/abci/abci++_app_requirements.md (about) 1 --- 2 order: 3 3 title: Requirements for the Application 4 --- 5 6 # Requirements for the Application 7 8 - [Requirements for the Application](#requirements-for-the-application) 9 - [Formal Requirements](#formal-requirements) 10 - [Consensus Connection Requirements](#consensus-connection-requirements) 11 - [Mempool Connection Requirements](#mempool-connection-requirements) 12 - [Managing the Application state and related topics](#managing-the-application-state-and-related-topics) 13 - [Connection State](#connection-state) 14 - [Concurrency](#concurrency) 15 - [FinalizeBlock](#finalizeblock) 16 - [Commit](#commit) 17 - [Candidate States](#candidate-states) 18 - [States and ABCI++ Connections](#states-and-abci-connections) 19 - [Consensus Connection](#consensus-connection) 20 - [Mempool Connection](#mempool-connection) 21 - [Replay Protection](#replay-protection) 22 - [Info/Query Connection](#infoquery-connection) 23 - [Snapshot Connection](#snapshot-connection) 24 - [Transaction Results](#transaction-results) 25 - [Gas](#gas) 26 - [Specifics of `ResponseCheckTx`](#specifics-of-responsechecktx) 27 - [Specifics of `ExecTxResult`](#specifics-of-exectxresult) 28 - [Updating the Validator Set](#updating-the-validator-set) 29 - [Consensus Parameters](#consensus-parameters) 30 - [List of Parameters](#list-of-parameters) 31 - [ABCIParams.VoteExtensionsEnableHeight](#abciparamsvoteextensionsenableheight) 32 - [BlockParams.MaxBytes](#blockparamsmaxbytes) 33 - [BlockParams.MaxGas](#blockparamsmaxgas) 34 - [EvidenceParams.MaxAgeDuration](#evidenceparamsmaxageduration) 35 - [EvidenceParams.MaxAgeNumBlocks](#evidenceparamsmaxagenumblocks) 36 - [EvidenceParams.MaxBytes](#evidenceparamsmaxbytes) 37 - [ValidatorParams.PubKeyTypes](#validatorparamspubkeytypes) 38 - [VersionParams.App](#versionparamsapp) 39 - [Updating Consensus Parameters](#updating-consensus-parameters) 40 - [`InitChain`](#initchain) 41 - [`FinalizeBlock`, `PrepareProposal`/`ProcessProposal`](#finalizeblock-prepareproposalprocessproposal) 42 - [`Query`](#query) 43 - [Query Proofs](#query-proofs) 44 - [Peer Filtering](#peer-filtering) 45 - [Paths](#paths) 46 - [Crash Recovery](#crash-recovery) 47 - [State Sync](#state-sync) 48 - [Taking Snapshots](#taking-snapshots) 49 - [Bootstrapping a Node](#bootstrapping-a-node) 50 - [Snapshot Discovery](#snapshot-discovery) 51 - [Snapshot Restoration](#snapshot-restoration) 52 - [Snapshot Verification](#snapshot-verification) 53 - [Transition to Consensus](#transition-to-consensus) 54 - [Application configuration required to switch to ABCI 2.0](#application-configuration-required-to-switch-to-abci-20) 55 56 57 ## Formal Requirements 58 59 ### Consensus Connection Requirements 60 61 This section specifies what CometBFT expects from the Application. It is structured as a set 62 of formal requirements that can be used for testing and verification of the Application's logic. 63 64 Let *p* and *q* be two correct processes. 65 Let *r<sub>p</sub>* (resp. *r<sub>q</sub>*) be a round of height *h* where *p* (resp. *q*) is the 66 proposer. 67 Let *s<sub>p,h-1</sub>* be *p*'s Application's state committed for height *h-1*. 68 Let *v<sub>p</sub>* (resp. *v<sub>q</sub>*) be the block that *p*'s (resp. *q*'s) CometBFT passes 69 on to the Application 70 via `RequestPrepareProposal` as proposer of round *r<sub>p</sub>* (resp *r<sub>q</sub>*), height *h*, 71 also known as the raw proposal. 72 Let *u<sub>p</sub>* (resp. *u<sub>q</sub>*) the possibly modified block *p*'s (resp. *q*'s) Application 73 returns via `ResponsePrepareProposal` to CometBFT, also known as the prepared proposal. 74 75 Process *p*'s prepared proposal can differ in two different rounds where *p* is the proposer. 76 77 - Requirement 1 [`PrepareProposal`, timeliness]: If *p*'s Application fully executes prepared blocks in 78 `PrepareProposal` and the network is in a synchronous period while processes *p* and *q* are in *r<sub>p</sub>*, 79 then the value of *TimeoutPropose* at *q* must be such that *q*'s propose timer does not time out 80 (which would result in *q* prevoting `nil` in *r<sub>p</sub>*). 81 82 Full execution of blocks at `PrepareProposal` time stands on CometBFT's critical path. Thus, 83 Requirement 1 ensures the Application or operator will set a value for `TimeoutPropose` such that the time it takes 84 to fully execute blocks in `PrepareProposal` does not interfere with CometBFT's propose timer. 85 Note that the violation of Requirement 1 may lead to further rounds, but will not 86 compromise liveness because even though `TimeoutPropose` is used as the initial 87 value for proposal timeouts, CometBFT will be dynamically adjust these timeouts 88 such that they will eventually be enough for completing `PrepareProposal`. 89 90 - Requirement 2 [`PrepareProposal`, tx-size]: When *p*'s Application calls `ResponsePrepareProposal`, the 91 total size in bytes of the transactions returned does not exceed `RequestPrepareProposal.max_tx_bytes`. 92 93 Busy blockchains might seek to gain full visibility into transactions in CometBFT's mempool, 94 rather than having visibility only on *a* subset of those transactions that fit in a block. 95 The application can do so by setting `ConsensusParams.Block.MaxBytes` to -1. 96 This instructs CometBFT (a) to enforce the maximum possible value for `MaxBytes` (100 MB) at CometBFT level, 97 and (b) to provide *all* transactions in the mempool when calling `RequestPrepareProposal`. 98 Under these settings, the aggregated size of all transactions may exceed `RequestPrepareProposal.max_tx_bytes`. 99 Hence, Requirement 2 ensures that the size in bytes of the transaction list returned by the application will never 100 cause the resulting block to go beyond its byte size limit. 101 102 - Requirement 3 [`PrepareProposal`, `ProcessProposal`, coherence]: For any two correct processes *p* and *q*, 103 if *q*'s CometBFT calls `RequestProcessProposal` on *u<sub>p</sub>*, 104 *q*'s Application returns Accept in `ResponseProcessProposal`. 105 106 Requirement 3 makes sure that blocks proposed by correct processes *always* pass the correct receiving process's 107 `ProcessProposal` check. 108 On the other hand, if there is a deterministic bug in `PrepareProposal` or `ProcessProposal` (or in both), 109 strictly speaking, this makes all processes that hit the bug byzantine. This is a problem in practice, 110 as very often validators are running the Application from the same codebase, so potentially *all* would 111 likely hit the bug at the same time. This would result in most (or all) processes prevoting `nil`, with the 112 serious consequences on CometBFT's liveness that this entails. Due to its criticality, Requirement 3 is a 113 target for extensive testing and automated verification. 114 115 - Requirement 4 [`ProcessProposal`, determinism-1]: `ProcessProposal` is a (deterministic) function of the current 116 state and the block that is about to be applied. In other words, for any correct process *p*, and any arbitrary block *u*, 117 if *p*'s CometBFT calls `RequestProcessProposal` on *u* at height *h*, 118 then *p*'s Application's acceptance or rejection **exclusively** depends on *u* and *s<sub>p,h-1</sub>*. 119 120 - Requirement 5 [`ProcessProposal`, determinism-2]: For any two correct processes *p* and *q*, and any arbitrary 121 block *u*, 122 if *p*'s (resp. *q*'s) CometBFT calls `RequestProcessProposal` on *u* at height *h*, 123 then *p*'s Application accepts *u* if and only if *q*'s Application accepts *u*. 124 Note that this requirement follows from Requirement 4 and the Agreement property of consensus. 125 126 Requirements 4 and 5 ensure that all correct processes will react in the same way to a proposed block, even 127 if the proposer is Byzantine. However, `ProcessProposal` may contain a bug that renders the 128 acceptance or rejection of the block non-deterministic, and therefore prevents processes hitting 129 the bug from fulfilling Requirements 4 or 5 (effectively making those processes Byzantine). 130 In such a scenario, CometBFT's liveness cannot be guaranteed. 131 Again, this is a problem in practice if most validators are running the same software, as they are likely 132 to hit the bug at the same point. There is currently no clear solution to help with this situation, so 133 the Application designers/implementors must proceed very carefully with the logic/implementation 134 of `ProcessProposal`. As a general rule `ProcessProposal` SHOULD always accept the block. 135 136 According to the Tendermint consensus algorithm, currently adopted in CometBFT, 137 a correct process can broadcast at most one precommit 138 message in round *r*, height *h*. 139 Since, as stated in the [Methods](./abci++_methods.md#extendvote) section, `ResponseExtendVote` 140 is only called when the consensus algorithm 141 is about to broadcast a non-`nil` precommit message, a correct process can only produce one vote extension 142 in round *r*, height *h*. 143 Let *e<sup>r</sup><sub>p</sub>* be the vote extension that the Application of a correct process *p* returns via 144 `ResponseExtendVote` in round *r*, height *h*. 145 Let *w<sup>r</sup><sub>p</sub>* be the proposed block that *p*'s CometBFT passes to the Application via `RequestExtendVote` 146 in round *r*, height *h*. 147 148 - Requirement 6 [`ExtendVote`, `VerifyVoteExtension`, coherence]: For any two different correct 149 processes *p* and *q*, if *q* receives *e<sup>r</sup><sub>p</sub>* from *p* in height *h*, *q*'s 150 Application returns Accept in `ResponseVerifyVoteExtension`. 151 152 Requirement 6 constrains the creation and handling of vote extensions in a similar way as Requirement 3 153 constrains the creation and handling of proposed blocks. 154 Requirement 6 ensures that extensions created by correct processes *always* pass the `VerifyVoteExtension` 155 checks performed by correct processes receiving those extensions. 156 However, if there is a (deterministic) bug in `ExtendVote` or `VerifyVoteExtension` (or in both), 157 we will face the same liveness issues as described for Requirement 5, as Precommit messages with invalid vote 158 extensions will be discarded. 159 160 - Requirement 7 [`VerifyVoteExtension`, determinism-1]: `VerifyVoteExtension` is a (deterministic) function of 161 the current state, the vote extension received, and the prepared proposal that the extension refers to. 162 In other words, for any correct process *p*, and any arbitrary vote extension *e*, and any arbitrary 163 block *w*, if *p*'s (resp. *q*'s) CometBFT calls `RequestVerifyVoteExtension` on *e* and *w* at height *h*, 164 then *p*'s Application's acceptance or rejection **exclusively** depends on *e*, *w* and *s<sub>p,h-1</sub>*. 165 166 - Requirement 8 [`VerifyVoteExtension`, determinism-2]: For any two correct processes *p* and *q*, 167 and any arbitrary vote extension *e*, and any arbitrary block *w*, 168 if *p*'s (resp. *q*'s) CometBFT calls `RequestVerifyVoteExtension` on *e* and *w* at height *h*, 169 then *p*'s Application accepts *e* if and only if *q*'s Application accepts *e*. 170 Note that this requirement follows from Requirement 7 and the Agreement property of consensus. 171 172 Requirements 7 and 8 ensure that the validation of vote extensions will be deterministic at all 173 correct processes. 174 Requirements 7 and 8 protect against arbitrary vote extension data from Byzantine processes, 175 in a similar way as Requirements 4 and 5 protect against arbitrary proposed blocks. 176 Requirements 7 and 8 can be violated by a bug inducing non-determinism in 177 `VerifyVoteExtension`. In this case liveness can be compromised. 178 Extra care should be put in the implementation of `ExtendVote` and `VerifyVoteExtension`. 179 As a general rule, `VerifyVoteExtension` SHOULD always accept the vote extension. 180 181 - Requirement 9 [*all*, no-side-effects]: *p*'s calls to `RequestPrepareProposal`, 182 `RequestProcessProposal`, `RequestExtendVote`, and `RequestVerifyVoteExtension` at height *h* do 183 not modify *s<sub>p,h-1</sub>*. 184 185 186 - Requirement 10 [`ExtendVote`, `FinalizeBlock`, non-dependency]: for any correct process *p*, 187 and any vote extension *e* that *p* received at height *h*, the computation of 188 *s<sub>p,h</sub>* does not depend on *e*. 189 190 The call to correct process *p*'s `RequestFinalizeBlock` at height *h*, with block *v<sub>p,h</sub>* 191 passed as parameter, creates state *s<sub>p,h</sub>*. 192 Additionally, *p*'s `FinalizeBlock` creates a set of transaction results *T<sub>p,h</sub>*. 193 194 - Requirement 11 [`FinalizeBlock`, determinism-1]: For any correct process *p*, 195 *s<sub>p,h</sub>* exclusively depends on *s<sub>p,h-1</sub>* and *v<sub>p,h</sub>*. 196 197 - Requirement 12 [`FinalizeBlock`, determinism-2]: For any correct process *p*, 198 the contents of *T<sub>p,h</sub>* exclusively depend on *s<sub>p,h-1</sub>* and *v<sub>p,h</sub>*. 199 200 Note that Requirements 11 and 12, combined with the Agreement property of consensus ensure 201 state machine replication, i.e., the Application state evolves consistently at all correct processes. 202 203 Also, notice that neither `PrepareProposal` nor `ExtendVote` have determinism-related 204 requirements associated. 205 Indeed, `PrepareProposal` is not required to be deterministic: 206 207 - *u<sub>p</sub>* may depend on *v<sub>p</sub>* and *s<sub>p,h-1</sub>*, but may also depend on other values or operations. 208 - *v<sub>p</sub> = v<sub>q</sub> ⇏ u<sub>p</sub> = u<sub>q</sub>*. 209 210 Likewise, `ExtendVote` can also be non-deterministic: 211 212 - *e<sup>r</sup><sub>p</sub>* may depend on *w<sup>r</sup><sub>p</sub>* and *s<sub>p,h-1</sub>*, 213 but may also depend on other values or operations. 214 - *w<sup>r</sup><sub>p</sub> = w<sup>r</sup><sub>q</sub> ⇏ 215 e<sup>r</sup><sub>p</sub> = e<sup>r</sup><sub>q</sub>* 216 217 ### Mempool Connection Requirements 218 219 Let *CheckTxCodes<sub>tx,p,h</sub>* denote the set of result codes returned by *p*'s Application, 220 via `ResponseCheckTx`, 221 to successive calls to `RequestCheckTx` occurring while the Application is at height *h* 222 and having transaction *tx* as parameter. 223 *CheckTxCodes<sub>tx,p,h</sub>* is a set since *p*'s Application may 224 return different result codes during height *h*. 225 If *CheckTxCodes<sub>tx,p,h</sub>* is a singleton set, i.e. the Application always returned 226 the same result code in `ResponseCheckTx` while at height *h*, 227 we define *CheckTxCode<sub>tx,p,h</sub>* as the singleton value of *CheckTxCodes<sub>tx,p,h</sub>*. 228 If *CheckTxCodes<sub>tx,p,h</sub>* is not a singleton set, *CheckTxCode<sub>tx,p,h</sub>* is undefined. 229 Let predicate *OK(CheckTxCode<sub>tx,p,h</sub>)* denote whether *CheckTxCode<sub>tx,p,h</sub>* is `SUCCESS`. 230 231 - Requirement 13 [`CheckTx`, eventual non-oscillation]: For any transaction *tx*, 232 there exists a boolean value *b*, 233 and a height *h<sub>stable</sub>* such that, 234 for any correct process *p*, 235 *CheckTxCode<sub>tx,p,h</sub>* is defined, and 236 *OK(CheckTxCode<sub>tx,p,h</sub>) = b* 237 for any height *h ≥ h<sub>stable</sub>*. 238 239 Requirement 13 ensures that 240 a transaction will eventually stop oscillating between `CheckTx` success and failure 241 if it stays in *p's* mempool for long enough. 242 This condition on the Application's behavior allows the mempool to ensure that 243 a transaction will leave the mempool of all full nodes, 244 either because it is expunged everywhere due to failing `CheckTx` calls, 245 or because it stays valid long enough to be gossipped, proposed and decided. 246 Although Requirement 13 defines a global *h<sub>stable</sub>*, application developers 247 can consider such stabilization height as local to process *p* (*h<sub>p,stable</sub>*), 248 without loss for generality. 249 In contrast, the value of *b* MUST be the same across all processes. 250 251 ## Managing the Application state and related topics 252 253 ### Connection State 254 255 CometBFT maintains four concurrent ABCI++ connections, namely 256 [Consensus Connection](#consensus-connection), 257 [Mempool Connection](#mempool-connection), 258 [Info/Query Connection](#infoquery-connection), and 259 [Snapshot Connection](#snapshot-connection). 260 It is common for an application to maintain a distinct copy of 261 the state for each connection, which are synchronized upon `Commit` calls. 262 263 #### Concurrency 264 265 In principle, each of the four ABCI++ connections operates concurrently with one 266 another. This means applications need to ensure access to state is 267 thread safe. Both the 268 [default in-process ABCI client](https://github.com/KYVENetwork/cometbft/v38/blob/v0.38.x/abci/client/local_client.go#L13) 269 and the 270 [default Go ABCI server](https://github.com/KYVENetwork/cometbft/v38/blob/v0.38.x/abci/server/socket_server.go#L20) 271 use a global lock to guard the handling of events across all connections, so they are not 272 concurrent at all. This means whether your app is compiled in-process with 273 CometBFT using the `NewLocalClient`, or run out-of-process using the `SocketServer`, 274 ABCI messages from all connections are received in sequence, one at a 275 time. 276 277 The existence of this global mutex means Go application developers can get thread safety for application state by routing all reads and writes through the ABCI system. Thus it may be unsafe to expose application state directly to an RPC interface, and unless explicit measures are taken, all queries should be routed through the ABCI Query method. 278 279 #### FinalizeBlock 280 281 When the consensus algorithm decides on a block, CometBFT uses `FinalizeBlock` to send the 282 decided block's data to the Application, which uses it to transition its state, but MUST NOT persist it; 283 persisting MUST be done during `Commit`. 284 285 The Application must remember the latest height from which it 286 has run a successful `Commit` so that it can tell CometBFT where to 287 pick up from when it recovers from a crash. See information on the Handshake 288 [here](#crash-recovery). 289 290 #### Commit 291 292 The Application should persist its state during `Commit`, before returning from it. 293 294 Before invoking `Commit`, CometBFT locks the mempool and flushes the mempool connection. This ensures that 295 no new messages 296 will be received on the mempool connection during this processing step, providing an opportunity to safely 297 update all four 298 connection states to the latest committed state at the same time. 299 300 When `Commit` returns, CometBFT unlocks the mempool. 301 302 WARNING: if the ABCI app logic processing the `Commit` message sends a 303 `/broadcast_tx_sync` or `/broadcast_tx` and waits for the response 304 before proceeding, it will deadlock. Executing `broadcast_tx` calls 305 involves acquiring the mempool lock that CometBFT holds during the `Commit` call. 306 Synchronous mempool-related calls must be avoided as part of the sequential logic of the 307 `Commit` function. 308 309 #### Candidate States 310 311 CometBFT calls `PrepareProposal` when it is about to send a proposed block to the network. 312 Likewise, CometBFT calls `ProcessProposal` upon reception of a proposed block from the 313 network. The proposed block's data 314 that is disclosed to the Application by these two methods is the following: 315 316 - the transaction list 317 - the `LastCommit` referring to the previous block 318 - the block header's hash (except in `PrepareProposal`, where it is not known yet) 319 - list of validators that misbehaved 320 - the block's timestamp 321 - `NextValidatorsHash` 322 - Proposer address 323 324 The Application may decide to *immediately* execute the given block (i.e., upon `PrepareProposal` 325 or `ProcessProposal`). There are two main reasons why the Application may want to do this: 326 327 - *Avoiding invalid transactions in blocks*. 328 In order to be sure that the block does not contain *any* invalid transaction, there may be 329 no way other than fully executing the transactions in the block as though it was the *decided* 330 block. 331 - *Quick `FinalizeBlock` execution*. 332 Upon reception of the decided block via `FinalizeBlock`, if that same block was executed 333 upon `PrepareProposal` or `ProcessProposal` and the resulting state was kept in memory, the 334 Application can simply apply that state (faster) to the main state, rather than reexecuting 335 the decided block (slower). 336 337 `PrepareProposal`/`ProcessProposal` can be called many times for a given height. Moreover, 338 it is not possible to accurately predict which of the blocks proposed in a height will be decided, 339 being delivered to the Application in that height's `FinalizeBlock`. 340 Therefore, the state resulting from executing a proposed block, denoted a *candidate state*, should 341 be kept in memory as a possible final state for that height. When `FinalizeBlock` is called, the Application should 342 check if the decided block corresponds to one of its candidate states; if so, it will apply it as 343 its *ExecuteTxState* (see [Consensus Connection](#consensus-connection) below), 344 which will be persisted during the upcoming `Commit` call. 345 346 Under adverse conditions (e.g., network instability), the consensus algorithm might take many rounds. 347 In this case, potentially many proposed blocks will be disclosed to the Application for a given height. 348 By the nature of Tendermint consensus algorithm, currently adopted in CometBFT, the number of proposed blocks received by the Application 349 for a particular height cannot be bound, so Application developers must act with care and use mechanisms 350 to bound memory usage. As a general rule, the Application should be ready to discard candidate states 351 before `FinalizeBlock`, even if one of them might end up corresponding to the 352 decided block and thus have to be reexecuted upon `FinalizeBlock`. 353 354 ### [States and ABCI++ Connections](#states-and-abci-connections) 355 356 #### Consensus Connection 357 358 The Consensus Connection should maintain an *ExecuteTxState* — the working state 359 for block execution. It should be updated by the call to `FinalizeBlock` 360 during block execution and committed to disk as the "latest 361 committed state" during `Commit`. Execution of a proposed block (via `PrepareProposal`/`ProcessProposal`) 362 **must not** update the *ExecuteTxState*, but rather be kept as a separate candidate state until `FinalizeBlock` 363 confirms which of the candidate states (if any) can be used to update *ExecuteTxState*. 364 365 #### Mempool Connection 366 367 The mempool Connection maintains *CheckTxState*. CometBFT sequentially processes an incoming 368 transaction (via RPC from client or P2P from the gossip layer) against *CheckTxState*. 369 If the processing does not return any error, the transaction is accepted into the mempool 370 and CometBFT starts gossipping it. 371 *CheckTxState* should be reset to the latest committed state 372 at the end of every `Commit`. 373 374 During the execution of a consensus instance, the *CheckTxState* may be updated concurrently with the 375 *ExecuteTxState*, as messages may be sent concurrently on the Consensus and Mempool connections. 376 At the end of the consensus instance, as described above, CometBFT locks the mempool and flushes 377 the mempool connection before calling `Commit`. This ensures that all pending `CheckTx` calls are 378 responded to and no new ones can begin. 379 380 After the `Commit` call returns, while still holding the mempool lock, `CheckTx` is run again on all 381 transactions that remain in the node's local mempool after filtering those included in the block. 382 Parameter `Type` in `RequestCheckTx` 383 indicates whether an incoming transaction is new (`CheckTxType_New`), or a 384 recheck (`CheckTxType_Recheck`). 385 386 Finally, after re-checking transactions in the mempool, CometBFT will unlock 387 the mempool connection. New transactions are once again able to be processed through `CheckTx`. 388 389 Note that `CheckTx` is just a weak filter to keep invalid transactions out of the mempool and, 390 ultimately, ouf of the blockchain. 391 Since the transaction cannot be guaranteed to be checked against the exact same state as it 392 will be executed as part of a (potential) decided block, `CheckTx` shouldn't check *everything* 393 that affects the transaction's validity, in particular those checks whose validity may depend on 394 transaction ordering. `CheckTx` is weak because a Byzantine node need not care about `CheckTx`; 395 it can propose a block full of invalid transactions if it wants. The mechanism ABCI++ has 396 in place for dealing with such behavior is `ProcessProposal`. 397 398 ##### Replay Protection 399 400 It is possible for old transactions to be sent again to the Application. This is typically 401 undesirable for all transactions, except for a generally small subset of them which are idempotent. 402 403 The mempool has a mechanism to prevent duplicated transactions from being processed. 404 This mechanism is nevertheless best-effort (currently based on the indexer) 405 and does not provide any guarantee of non duplication. 406 It is thus up to the Application to implement an application-specific 407 replay protection mechanism with strong guarantees as part of the logic in `CheckTx`. 408 409 #### Info/Query Connection 410 411 The Info (or Query) Connection should maintain a `QueryState`. This connection has two 412 purposes: 1) having the application answer the queries CometBFT receives from users 413 (see section [Query](#query)), 414 and 2) synchronizing CometBFT and the Application at start up time (see 415 [Crash Recovery](#crash-recovery)) 416 or after state sync (see [State Sync](#state-sync)). 417 418 `QueryState` is a read-only copy of *ExecuteTxState* as it was after the last 419 `Commit`, i.e. 420 after the full block has been processed and the state committed to disk. 421 422 #### Snapshot Connection 423 424 The Snapshot Connection is used to serve state sync snapshots for other nodes 425 and/or restore state sync snapshots to a local node being bootstrapped. 426 Snapshot management is optional: an Application may choose not to implement it. 427 428 For more information, see Section [State Sync](#state-sync). 429 430 ### Transaction Results 431 432 The Application is expected to return a list of 433 [`ExecTxResult`](./abci%2B%2B_methods.md#exectxresult) in 434 [`ResponseFinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). The list of transaction 435 results MUST respect the same order as the list of transactions delivered via 436 [`RequestFinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). 437 This section discusses the fields inside this structure, along with the fields in 438 [`ResponseCheckTx`](./abci%2B%2B_methods.md#checktx), 439 whose semantics are similar. 440 441 The `Info` and `Log` fields are 442 non-deterministic values for debugging/convenience purposes. CometBFT logs them but they 443 are otherwise ignored. 444 445 #### Gas 446 447 Ethereum introduced the notion of *gas* as an abstract representation of the 448 cost of the resources consumed by nodes when processing a transaction. Every operation in the 449 Ethereum Virtual Machine uses some amount of gas. 450 Gas has a market-variable price based on which miners can accept or reject to execute a 451 particular operation. 452 453 Users propose a maximum amount of gas for their transaction; if the transaction uses less, they get 454 the difference credited back. CometBFT adopts a similar abstraction, 455 though uses it only optionally and weakly, allowing applications to define 456 their own sense of the cost of execution. 457 458 In CometBFT, the [ConsensusParams.Block.MaxGas](#consensus-parameters) limits the amount of 459 total gas that can be used by all transactions in a block. 460 The default value is `-1`, which means the block gas limit is not enforced, or that the concept of 461 gas is meaningless. 462 463 Responses contain a `GasWanted` and `GasUsed` field. The former is the maximum 464 amount of gas the sender of a transaction is willing to use, and the latter is how much it actually 465 used. Applications should enforce that `GasUsed <= GasWanted` — i.e. transaction execution 466 or validation should fail before it can use more resources than it requested. 467 468 When `MaxGas > -1`, CometBFT enforces the following rules: 469 470 - `GasWanted <= MaxGas` for every transaction in the mempool 471 - `(sum of GasWanted in a block) <= MaxGas` when proposing a block 472 473 If `MaxGas == -1`, no rules about gas are enforced. 474 475 In v0.34.x and earlier versions, CometBFT does not enforce anything about Gas in consensus, 476 only in the mempool. 477 This means it does not guarantee that committed blocks satisfy these rules. 478 It is the application's responsibility to return non-zero response codes when gas limits are exceeded 479 when executing the transactions of a block. 480 Since the introduction of `PrepareProposal` and `ProcessProposal` in v.0.37.x, it is now possible 481 for the Application to enforce that all blocks proposed (and voted for) in consensus — and thus all 482 blocks decided — respect the `MaxGas` limits described above. 483 484 Since the Application should enforce that `GasUsed <= GasWanted` when executing a transaction, and 485 it can use `PrepareProposal` and `ProcessProposal` to enforce that `(sum of GasWanted in a block) <= MaxGas` 486 in all proposed or prevoted blocks, 487 we have: 488 489 - `(sum of GasUsed in a block) <= MaxGas` for every block 490 491 The `GasUsed` field is ignored by CometBFT. 492 493 #### Specifics of `ResponseCheckTx` 494 495 If `Code != 0`, it will be rejected from the mempool and hence 496 not broadcasted to other peers and not included in a proposal block. 497 498 `Data` contains the result of the `CheckTx` transaction execution, if any. It does not need to be 499 deterministic since, given a transaction, nodes' Applications 500 might have a different *CheckTxState* values when they receive it and check their validity 501 via `CheckTx`. 502 CometBFT ignores this value in `ResponseCheckTx`. 503 504 From v0.34.x on, there is a `Priority` field in `ResponseCheckTx` that can be 505 used to explicitly prioritize transactions in the mempool for inclusion in a block 506 proposal. 507 508 #### Specifics of `ExecTxResult` 509 510 `FinalizeBlock` is the workhorse of the blockchain. CometBFT delivers the decided block, 511 including the list of all its transactions synchronously to the Application. 512 The block delivered (and thus the transaction order) is the same at all correct nodes as guaranteed 513 by the Agreement property of consensus. 514 515 The `Data` field in `ExecTxResult` contains an array of bytes with the transaction result. 516 It must be deterministic (i.e., the same value must be returned at all nodes), but it can contain arbitrary 517 data. Likewise, the value of `Code` must be deterministic. 518 If `Code != 0`, the transaction will be marked invalid, 519 though it is still included in the block. Invalid transactions are not indexed, as they are 520 considered analogous to those that failed `CheckTx`. 521 522 Both the `Code` and `Data` are included in a structure that is hashed into the 523 `LastResultsHash` of the block header in the next height. 524 525 `Events` include any events for the execution, which CometBFT will use to index 526 the transaction by. This allows transactions to be queried according to what 527 events took place during their execution. 528 529 ### Updating the Validator Set 530 531 The application may set the validator set during 532 [`InitChain`](./abci%2B%2B_methods.md#initchain), and may update it during 533 [`FinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). In both cases, a structure of type 534 [`ValidatorUpdate`](./abci%2B%2B_methods.md#validatorupdate) is returned. 535 536 The `InitChain` method, used to initialize the Application, can return a list of validators. 537 If the list is empty, CometBFT will use the validators loaded from the genesis 538 file. 539 If the list returned by `InitChain` is not empty, CometBFT will use its contents as the validator set. 540 This way the application can set the initial validator set for the 541 blockchain. 542 543 Applications must ensure that a single set of validator updates does not contain duplicates, i.e. 544 a given public key can only appear once within a given update. If an update includes 545 duplicates, the block execution will fail irrecoverably. 546 547 Structure `ValidatorUpdate` contains a public key, which is used to identify the validator: 548 The public key currently supports three types: 549 550 - `ed25519` 551 - `secp256k1` 552 - `sr25519` 553 554 Structure `ValidatorUpdate` also contains an `ìnt64` field denoting the validator's new power. 555 Applications must ensure that 556 `ValidatorUpdate` structures abide by the following rules: 557 558 - power must be non-negative 559 - if power is set to 0, the validator must be in the validator set; it will be removed from the set 560 - if power is greater than 0: 561 - if the validator is not in the validator set, it will be added to the 562 set with the given power 563 - if the validator is in the validator set, its power will be adjusted to the given power 564 - the total power of the new validator set must not exceed `MaxTotalVotingPower`, where 565 `MaxTotalVotingPower = MaxInt64 / 8` 566 567 Note the updates returned after processing the block at height `H` will only take effect 568 at block `H+2` (see Section [Methods](./abci%2B%2B_methods.md)). 569 570 ### Consensus Parameters 571 572 `ConsensusParams` are global parameters that apply to all validators in a blockchain. 573 They enforce certain limits in the blockchain, like the maximum size 574 of blocks, amount of gas used in a block, and the maximum acceptable age of 575 evidence. They can be set in 576 [`InitChain`](./abci%2B%2B_methods.md#initchain), and updated in 577 [`FinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). 578 These parameters are deterministically set and/or updated by the Application, so 579 all full nodes have the same value at a given height. 580 581 #### List of Parameters 582 583 These are the current consensus parameters (as of v0.38.x): 584 585 1. [ABCIParams.VoteExtensionsEnableHeight](#abciparamsvoteextensionsenableheight) 586 2. [BlockParams.MaxBytes](#blockparamsmaxbytes) 587 3. [BlockParams.MaxGas](#blockparamsmaxgas) 588 4. [EvidenceParams.MaxAgeDuration](#evidenceparamsmaxageduration) 589 5. [EvidenceParams.MaxAgeNumBlocks](#evidenceparamsmaxagenumblocks) 590 6. [EvidenceParams.MaxBytes](#evidenceparamsmaxbytes) 591 7. [ValidatorParams.PubKeyTypes](#validatorparamspubkeytypes) 592 8. [VersionParams.App](#versionparamsapp) 593 594 ##### ABCIParams.VoteExtensionsEnableHeight 595 596 This parameter is either 0 or a positive height at which vote extensions 597 become mandatory. If the value is zero (which is the default), vote 598 extensions are not expected. Otherwise, at all heights greater than the 599 configured height `H` vote extensions must be present (even if empty). 600 When the configured height `H` is reached, `PrepareProposal` will not 601 include vote extensions yet, but `ExtendVote` and `VerifyVoteExtension` will 602 be called. Then, when reaching height `H+1`, `PrepareProposal` will 603 include the vote extensions from height `H`. For all heights after `H` 604 605 - vote extensions cannot be disabled, 606 - they are mandatory: all precommit messages sent MUST have an extension 607 attached. Nevertheless, the application MAY provide 0-length 608 extensions. 609 610 Must always be set to a future height, 0, or the same height that was previously set. 611 Once the chain's height reaches the value set, it cannot be changed to a different value. 612 613 ##### BlockParams.MaxBytes 614 615 The maximum size of a complete Protobuf encoded block. 616 This is enforced by the consensus algorithm. 617 618 This implies a maximum transaction size that is `MaxBytes`, less the expected size of 619 the header, the validator set, and any included evidence in the block. 620 621 The Application should be aware that honest validators *may* produce and 622 broadcast blocks with up to the configured `MaxBytes` size. 623 As a result, the consensus 624 [timeout parameters](../../docs/core/configuration.md#consensus-timeouts-explained) 625 adopted by nodes should be configured so as to account for the worst-case 626 latency for the delivery of a full block with `MaxBytes` size to all validators. 627 628 If the Application wants full control over the size of blocks, 629 it can do so by enforcing a byte limit set up at the Application level. 630 This Application-internal limit is used by `PrepareProposal` to bound the total size 631 of transactions it returns, and by `ProcessProposal` to reject any received block 632 whose total transaction size is bigger than the enforced limit. 633 In such case, the Application MAY set `MaxBytes` to -1. 634 635 If the Application sets value -1, consensus will: 636 637 - consider that the actual value to enforce is 100 MB 638 - will provide *all* transactions in the mempool in calls to `PrepareProposal` 639 640 Must have `MaxBytes == -1` OR `0 < MaxBytes <= 100 MB`. 641 642 > Bear in mind that the default value for the `BlockParams.MaxBytes` consensus 643 > parameter accepts as valid blocks with size up to 21 MB. 644 > If the Application's use case does not need blocks of that size, 645 > or if the impact (specially on bandwidth consumption and block latency) 646 > of propagating blocks of that size was not evaluated, 647 > it is strongly recommended to wind down this default value. 648 649 ##### BlockParams.MaxGas 650 651 The maximum of the sum of `GasWanted` that will be allowed in a proposed block. 652 This is *not* enforced by the consensus algorithm. 653 It is left to the Application to enforce (ie. if transactions are included past the 654 limit, they should return non-zero codes). It is used by CometBFT to limit the 655 transactions included in a proposed block. 656 657 Must have `MaxGas >= -1`. 658 If `MaxGas == -1`, no limit is enforced. 659 660 ##### EvidenceParams.MaxAgeDuration 661 662 This is the maximum age of evidence in time units. 663 This is enforced by the consensus algorithm. 664 665 If a block includes evidence older than this (AND the evidence was created more 666 than `MaxAgeNumBlocks` ago), the block will be rejected (validators won't vote 667 for it). 668 669 Must have `MaxAgeDuration > 0`. 670 671 ##### EvidenceParams.MaxAgeNumBlocks 672 673 This is the maximum age of evidence in blocks. 674 This is enforced by the consensus algorithm. 675 676 If a block includes evidence older than this (AND the evidence was created more 677 than `MaxAgeDuration` ago), the block will be rejected (validators won't vote 678 for it). 679 680 Must have `MaxAgeNumBlocks > 0`. 681 682 ##### EvidenceParams.MaxBytes 683 684 This is the maximum size of total evidence in bytes that can be committed to a 685 single block. It should fall comfortably under the max block bytes. 686 687 Its value must not exceed the size of 688 a block minus its overhead ( ~ `BlockParams.MaxBytes`). 689 690 Must have `MaxBytes > 0`. 691 692 ##### ValidatorParams.PubKeyTypes 693 694 The parameter restricts the type of keys validators can use. The parameter uses ABCI pubkey naming, not Amino names. 695 696 ##### VersionParams.App 697 698 This is the version of the ABCI application. 699 700 #### Updating Consensus Parameters 701 702 The application may set the `ConsensusParams` during 703 [`InitChain`](./abci%2B%2B_methods.md#initchain), 704 and update them during 705 [`FinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). 706 If the `ConsensusParams` is empty, it will be ignored. Each field 707 that is not empty will be applied in full. For instance, if updating the 708 `Block.MaxBytes`, applications must also set the other `Block` fields (like 709 `Block.MaxGas`), even if they are unchanged, as they will otherwise cause the 710 value to be updated to the default. 711 712 ##### `InitChain` 713 714 `ResponseInitChain` includes a `ConsensusParams` parameter. 715 If `ConsensusParams` is `nil`, CometBFT will use the params loaded in the genesis 716 file. If `ConsensusParams` is not `nil`, CometBFT will use it. 717 This way the application can determine the initial consensus parameters for the 718 blockchain. 719 720 ##### `FinalizeBlock`, `PrepareProposal`/`ProcessProposal` 721 722 `ResponseFinalizeBlock` accepts a `ConsensusParams` parameter. 723 If `ConsensusParams` is `nil`, CometBFT will do nothing. 724 If `ConsensusParams` is not `nil`, CometBFT will use it. 725 This way the application can update the consensus parameters over time. 726 727 The updates returned in block `H` will take effect right away for block 728 `H+1`. 729 730 ### `Query` 731 732 `Query` is a generic method with lots of flexibility to enable diverse sets 733 of queries on application state. CometBFT makes use of `Query` to filter new peers 734 based on ID and IP, and exposes `Query` to the user over RPC. 735 736 Note that calls to `Query` are not replicated across nodes, but rather query the 737 local node's state - hence they may return stale reads. For reads that require 738 consensus, use a transaction. 739 740 The most important use of `Query` is to return Merkle proofs of the application state at some height 741 that can be used for efficient application-specific light-clients. 742 743 Note CometBFT has technically no requirements from the `Query` 744 message for normal operation - that is, the ABCI app developer need not implement 745 Query functionality if they do not wish to. 746 747 #### Query Proofs 748 749 The CometBFT block header includes a number of hashes, each providing an 750 anchor for some type of proof about the blockchain. The `ValidatorsHash` enables 751 quick verification of the validator set, the `DataHash` gives quick 752 verification of the transactions included in the block. 753 754 The `AppHash` is unique in that it is application specific, and allows for 755 application-specific Merkle proofs about the state of the application. 756 While some applications keep all relevant state in the transactions themselves 757 (like Bitcoin and its UTXOs), others maintain a separated state that is 758 computed deterministically *from* transactions, but is not contained directly in 759 the transactions themselves (like Ethereum contracts and accounts). 760 For such applications, the `AppHash` provides a much more efficient way to verify light-client proofs. 761 762 ABCI applications can take advantage of more efficient light-client proofs for 763 their state as follows: 764 765 - return the Merkle root of the deterministic application state in 766 `ResponseFinalizeBlock.Data`. This Merkle root will be included as the `AppHash` in the next block. 767 - return efficient Merkle proofs about that application state in `ResponseQuery.Proof` 768 that can be verified using the `AppHash` of the corresponding block. 769 770 For instance, this allows an application's light-client to verify proofs of 771 absence in the application state, something which is much less efficient to do using the block hash. 772 773 Some applications (eg. Ethereum, Cosmos-SDK) have multiple "levels" of Merkle trees, 774 where the leaves of one tree are the root hashes of others. To support this, and 775 the general variability in Merkle proofs, the `ResponseQuery.Proof` has some minimal structure: 776 777 ```protobuf 778 message ProofOps { 779 repeated ProofOp ops = 1 780 } 781 782 message ProofOp { 783 string type = 1; 784 bytes key = 2; 785 bytes data = 3; 786 } 787 ``` 788 789 Each `ProofOp` contains a proof for a single key in a single Merkle tree, of the specified `type`. 790 This allows ABCI to support many different kinds of Merkle trees, encoding 791 formats, and proofs (eg. of presence and absence) just by varying the `type`. 792 The `data` contains the actual encoded proof, encoded according to the `type`. 793 When verifying the full proof, the root hash for one ProofOp is the value being 794 verified for the next ProofOp in the list. The root hash of the final ProofOp in 795 the list should match the `AppHash` being verified against. 796 797 #### Peer Filtering 798 799 When CometBFT connects to a peer, it sends two queries to the ABCI application 800 using the following paths, with no additional data: 801 802 - `/p2p/filter/addr/<IP:PORT>`, where `<IP:PORT>` denote the IP address and 803 the port of the connection 804 - `p2p/filter/id/<ID>`, where `<ID>` is the peer node ID (ie. the 805 pubkey.Address() for the peer's PubKey) 806 807 If either of these queries return a non-zero ABCI code, CometBFT will refuse 808 to connect to the peer. 809 810 #### Paths 811 812 Queries are directed at paths, and may optionally include additional data. 813 814 The expectation is for there to be some number of high level paths 815 differentiating concerns, like `/p2p`, `/store`, and `/app`. Currently, 816 CometBFT only uses `/p2p`, for filtering peers. For more advanced use, see the 817 implementation of 818 [Query in the Cosmos-SDK](https://github.com/cosmos/cosmos-sdk/blob/e2037f7696fed4fdd4bc076f9e7053fe8178a881/baseapp/abci.go#L557-L565). 819 820 ### Crash Recovery 821 822 CometBFT and the application are expected to crash together and there should not 823 exist a scenario where the application has persisted state of a height greater than the 824 latest height persisted by CometBFT. 825 826 In practice, persisting the state of a height consists of three steps, the last of which 827 is the call to the application's `Commit` method, the only place where the application is expected to 828 persist/commit its state. 829 On startup (upon recovery), CometBFT calls the `Info` method on the Info Connection to get the latest 830 committed state of the app. The app MUST return information consistent with the 831 last block for which it successfully completed `Commit`. 832 833 The three steps performed before the state of a height is considered persisted are: 834 835 - The block is stored by CometBFT in the blockstore 836 - CometBFT has stored the state returned by the application through `FinalizeBlockResponse` 837 - The application has committed its state within `Commit`. 838 839 The following diagram depicts the order in which these events happen, and the corresponding 840 ABCI functions that are called and executed by CometBFT and the application: 841 842 843 ``` 844 APP: Execute block Persist application state 845 / return ResultFinalizeBlock / 846 / / 847 Event: ------------- block_stored ------------ / ------------ state_stored --------------- / ----- app_persisted_state 848 | / | / | 849 CometBFT: Decide --- Persist block -- Call FinalizeBlock - Persist results ---------- Call Commit -- 850 on in the (txResults, validator 851 Block block store updates...) 852 853 ``` 854 855 As these three steps are not atomic, we observe different cases based on which steps have been executed 856 before the crash occurred 857 (we assume that at least `block_stored` has been executed, otherwise, there is no state persisted, 858 and the operations for this height are repeated entirely): 859 860 - `block_stored`: we replay `FinalizeBlock` and the steps afterwards. 861 - `block_stored` and `state_stored`: As the app did not persist its state within `Commit`, we need to re-execute 862 `FinalizeBlock` to retrieve the results and compare them to the state stored by CometBFT within `state_stored`. 863 The expected case is that the states will match, otherwise CometBFT panics. 864 - `block_stored`, `state_stored`, `app_persisted_state`: we move on to the next height. 865 866 Based on the sequence of these events, CometBFT will panic if any of the steps in the sequence happen out of order, 867 that is if: 868 869 - The application has persisted a block at a height higher than the blocked saved during `state_stored`. 870 - The `block_stored` step persisted a block at a height smaller than the `state_stored` 871 - And the difference between the heights of the blocks persisted by `state_stored` and `block_stored` is more 872 than 1 (this corresponds to a scenario where we stored two blocks in the block store but never persisted the state of the first 873 block, which should never happen). 874 875 A special case is when a crash happens before the first block is committed - that is, after calling 876 `InitChain`. In that case, the application's state should still be at height 0 and thus `InitChain` 877 will be called again. 878 879 880 ### State Sync 881 882 A new node joining the network can simply join consensus at the genesis height and replay all 883 historical blocks until it is caught up. However, for large chains this can take a significant 884 amount of time, often on the order of days or weeks. 885 886 State sync is an alternative mechanism for bootstrapping a new node, where it fetches a snapshot 887 of the state machine at a given height and restores it. Depending on the application, this can 888 be several orders of magnitude faster than replaying blocks. 889 890 Note that state sync does not currently backfill historical blocks, so the node will have a 891 truncated block history - users are advised to consider the broader network implications of this in 892 terms of block availability and auditability. This functionality may be added in the future. 893 894 For details on the specific ABCI calls and types, see the 895 [methods](abci%2B%2B_methods.md) section. 896 897 #### Taking Snapshots 898 899 Applications that want to support state syncing must take state snapshots at regular intervals. How 900 this is accomplished is entirely up to the application. A snapshot consists of some metadata and 901 a set of binary chunks in an arbitrary format: 902 903 - `Height (uint64)`: The height at which the snapshot is taken. It must be taken after the given 904 height has been committed, and must not contain data from any later heights. 905 906 - `Format (uint32)`: An arbitrary snapshot format identifier. This can be used to version snapshot 907 formats, e.g. to switch from Protobuf to MessagePack for serialization. The application can use 908 this when restoring to choose whether to accept or reject a snapshot. 909 910 - `Chunks (uint32)`: The number of chunks in the snapshot. Each chunk contains arbitrary binary 911 data, and should be less than 16 MB; 10 MB is a good starting point. 912 913 - `Hash ([]byte)`: An arbitrary hash of the snapshot. This is used to check whether a snapshot is 914 the same across nodes when downloading chunks. 915 916 - `Metadata ([]byte)`: Arbitrary snapshot metadata, e.g. chunk hashes for verification or any other 917 necessary info. 918 919 For a snapshot to be considered the same across nodes, all of these fields must be identical. When 920 sent across the network, snapshot metadata messages are limited to 4 MB. 921 922 When a new node is running state sync and discovering snapshots, CometBFT will query an existing 923 application via the ABCI `ListSnapshots` method to discover available snapshots, and load binary 924 snapshot chunks via `LoadSnapshotChunk`. The application is free to choose how to implement this 925 and which formats to use, but must provide the following guarantees: 926 927 - **Consistent:** A snapshot must be taken at a single isolated height, unaffected by 928 concurrent writes. This can be accomplished by using a data store that supports ACID 929 transactions with snapshot isolation. 930 931 - **Asynchronous:** Taking a snapshot can be time-consuming, so it must not halt chain progress, 932 for example by running in a separate thread. 933 934 - **Deterministic:** A snapshot taken at the same height in the same format must be identical 935 (at the byte level) across nodes, including all metadata. This ensures good availability of 936 chunks, and that they fit together across nodes. 937 938 A very basic approach might be to use a datastore with MVCC transactions (such as RocksDB), 939 start a transaction immediately after block commit, and spawn a new thread which is passed the 940 transaction handle. This thread can then export all data items, serialize them using e.g. 941 Protobuf, hash the byte stream, split it into chunks, and store the chunks in the file system 942 along with some metadata - all while the blockchain is applying new blocks in parallel. 943 944 A more advanced approach might include incremental verification of individual chunks against the 945 chain app hash, parallel or batched exports, compression, and so on. 946 947 Old snapshots should be removed after some time - generally only the last two snapshots are needed 948 (to prevent the last one from being removed while a node is restoring it). 949 950 #### Bootstrapping a Node 951 952 An empty node can be state synced by setting the configuration option `statesync.enabled = 953 true`. The node also needs the chain genesis file for basic chain info, and configuration for 954 light client verification of the restored snapshot: a set of CometBFT RPC servers, and a 955 trusted header hash and corresponding height from a trusted source, via the `statesync` 956 configuration section. 957 958 Once started, the node will connect to the P2P network and begin discovering snapshots. These 959 will be offered to the local application via the `OfferSnapshot` ABCI method. Once a snapshot 960 is accepted CometBFT will fetch and apply the snapshot chunks. After all chunks have been 961 successfully applied, CometBFT verifies the app's `AppHash` against the chain using the light 962 client, then switches the node to normal consensus operation. 963 964 ##### Snapshot Discovery 965 966 When the empty node joins the P2P network, it asks all peers to report snapshots via the 967 `ListSnapshots` ABCI call (limited to 10 per node). After some time, the node picks the most 968 suitable snapshot (generally prioritized by height, format, and number of peers), and offers it 969 to the application via `OfferSnapshot`. The application can choose a number of responses, 970 including accepting or rejecting it, rejecting the offered format, rejecting the peer who sent 971 it, and so on. CometBFT will keep discovering and offering snapshots until one is accepted or 972 the application aborts. 973 974 ##### Snapshot Restoration 975 976 Once a snapshot has been accepted via `OfferSnapshot`, CometBFT begins downloading chunks from 977 any peers that have the same snapshot (i.e. that have identical metadata fields). Chunks are 978 spooled in a temporary directory, and then given to the application in sequential order via 979 `ApplySnapshotChunk` until all chunks have been accepted. 980 981 The method for restoring snapshot chunks is entirely up to the application. 982 983 During restoration, the application can respond to `ApplySnapshotChunk` with instructions for how 984 to continue. This will typically be to accept the chunk and await the next one, but it can also 985 ask for chunks to be refetched (either the current one or any number of previous ones), P2P peers 986 to be banned, snapshots to be rejected or retried, and a number of other responses - see the ABCI 987 reference for details. 988 989 If CometBFT fails to fetch a chunk after some time, it will reject the snapshot and try a 990 different one via `OfferSnapshot` - the application can choose whether it wants to support 991 restarting restoration, or simply abort with an error. 992 993 ##### Snapshot Verification 994 995 Once all chunks have been accepted, CometBFT issues an `Info` ABCI call to retrieve the 996 `LastBlockAppHash`. This is compared with the trusted app hash from the chain, retrieved and 997 verified using the light client. CometBFT also checks that `LastBlockHeight` corresponds to the 998 height of the snapshot. 999 1000 This verification ensures that an application is valid before joining the network. However, the 1001 snapshot restoration may take a long time to complete, so applications may want to employ additional 1002 verification during the restore to detect failures early. This might e.g. include incremental 1003 verification of each chunk against the app hash (using bundled Merkle proofs), checksums to 1004 protect against data corruption by the disk or network, and so on. However, it is important to 1005 note that the only trusted information available is the app hash, and all other snapshot metadata 1006 can be spoofed by adversaries. 1007 1008 Apps may also want to consider state sync denial-of-service vectors, where adversaries provide 1009 invalid or harmful snapshots to prevent nodes from joining the network. The application can 1010 counteract this by asking CometBFT to ban peers. As a last resort, node operators can use 1011 P2P configuration options to whitelist a set of trusted peers that can provide valid snapshots. 1012 1013 ##### Transition to Consensus 1014 1015 Once the snapshots have all been restored, CometBFT gathers additional information necessary for 1016 bootstrapping the node (e.g. chain ID, consensus parameters, validator sets, and block headers) 1017 from the genesis file and light client RPC servers. It also calls `Info` to verify the following: 1018 1019 - that the app hash from the snapshot it has delivered to the Application matches the apphash 1020 stored in the next height's block 1021 1022 - that the version that the Application returns in `ResponseInfo` matches the version in the 1023 current height's block header 1024 1025 Once the state machine has been restored and CometBFT has gathered this additional 1026 information, it transitions to consensus. As of ABCI 2.0, CometBFT ensures the necessary conditions 1027 to switch are met [RFC-100](https://github.com/KYVENetwork/cometbft/v38/blob/v0.38.x/docs/rfc/rfc-100-abci-vote-extension-propag.md#base-implementation-persist-and-propagate-extended-commit-history). 1028 From the application's point of view, these operations are transparent, unless the application has just upgraded to ABCI 2.0. 1029 In that case, the application needs to be properly configured and aware of certain constraints in terms of when 1030 to provide vote extensions. More details can be found in the section below. 1031 1032 Once a node switches to consensus, it operates like any other node, apart from having a truncated block history at the height of the restored snapshot. 1033 1034 ## Application configuration required to switch to ABCI 2.0 1035 1036 Introducing vote extensions requires changes to the configuration of the application. 1037 1038 First of all, switching to a version of CometBFT with vote extensions, requires a coordinated upgrade. 1039 For a detailed description on the upgrade path, please refer to the corresponding 1040 [section](https://github.com/KYVENetwork/cometbft/v38/blob/v0.38.x/docs/rfc/rfc-100-abci-vote-extension-propag.md#upgrade-path) in RFC-100. 1041 1042 There is a newly introduced [**consensus parameter**](./abci%2B%2B_app_requirements.md#abciparamsvoteextensionsenableheight): `VoteExtensionsEnableHeight`. 1043 This parameter represents the height at which vote extensions are 1044 required for consensus to proceed, with 0 being the default value (no vote extensions). 1045 A chain can enable vote extensions either: 1046 1047 - at genesis by setting `VoteExtensionsEnableHeight` to be equal, e.g., to the `InitialHeight` 1048 - or via the application logic by changing the `ConsensusParam` to configure the 1049 `VoteExtensionsEnableHeight`. 1050 1051 Once the (coordinated) upgrade to ABCI 2.0 has taken place, at height *h<sub>u</sub>*, 1052 the value of `VoteExtensionsEnableHeight` MAY be set to some height, *h<sub>e</sub>*, 1053 which MUST be higher than the current height of the chain. Thus the earliest value for 1054 *h<sub>e</sub>* is *h<sub>u</sub>* + 1. 1055 1056 Once a node reaches the configured height, 1057 for all heights *h ≥ h<sub>e</sub>*, the consensus algorithm will 1058 reject as invalid any precommit messages that do not have signed vote extension data. 1059 If the application requires it, a 0-length vote extension is allowed, but it MUST be signed 1060 and present in the precommit message. 1061 Likewise, for all heights *h < h<sub>e</sub>*, any precommit messages that *do* have vote extensions 1062 will also be rejected as malformed. 1063 Height *h<sub>e</sub>* is somewhat special, as calls to `PrepareProposal` MUST NOT 1064 have vote extension data, but all precommit votes in that height MUST carry a vote extension, 1065 even if the extension is `nil`. 1066 Height *h<sub>e</sub> + 1* is the first height for which `PrepareProposal` MUST have vote 1067 extension data and all precommit votes in that height MUST have a vote extension. 1068 1069 Corollary, [CometBFT will decide](./abci%2B%2B_comet_expected_behavior.md#handling-upgrades-to-abci-20) which data to store, and require for successful operations, based on the current height 1070 of the chain.