github.com/aakash4dev/cometbft@v0.38.2/spec/abci/abci++_app_requirements.md (about) 1 --- 2 order: 3 3 title: Requirements for the Application 4 --- 5 6 # Requirements for the Application 7 8 - [Formal Requirements](#formal-requirements) 9 - [Consensus Connection Requirements](#consensus-connection-requirements) 10 - [Mempool Connection Requirements](#mempool-connection-requirements) 11 - [Managing the Application state and related topics](#managing-the-application-state-and-related-topics) 12 - [Connection State](#connection-state) 13 - [Concurrency](#concurrency) 14 - [Finalize Block](#finalizeblock) 15 - [Commit](#commit) 16 - [Candidate States](#candidate-states) 17 - [States and ABCI++ Connections](#states-and-abci%2B%2B-connections) 18 - [Consensus Connection](#consensus-connection) 19 - [Mempool Connection](#mempool-connection) 20 - [Info/Query Connection](#infoquery-connection) 21 - [Snapshot Connection](#snapshot-connection) 22 - [Transaction Results](#transaction-results) 23 - [Updating the Validator Set](#updating-the-validator-set) 24 - [Consensus Parameters](#consensus-parameters) 25 - [List of Parameters](#list-of-parameters) 26 - [Updating Consensus Parameters](#updating-consensus-parameters) 27 - [Query](#query) 28 - [Query Proofs](#query-proofs) 29 - [Peer Filtering](#peer-filtering) 30 - [Paths](#paths) 31 - [Crash Recovery](#crash-recovery) 32 - [State Sync](#state-sync) 33 - [Application configuration required to switch to ABCI2.0](#application-configuration-required-to-switch-to-abci-20) 34 35 36 ## Formal Requirements 37 38 ### Consensus Connection Requirements 39 40 This section specifies what CometBFT expects from the Application. It is structured as a set 41 of formal requirements that can be used for testing and verification of the Application's logic. 42 43 Let *p* and *q* be two correct processes. 44 Let *r<sub>p</sub>* (resp. *r<sub>q</sub>*) be a round of height *h* where *p* (resp. *q*) is the 45 proposer. 46 Let *s<sub>p,h-1</sub>* be *p*'s Application's state committed for height *h-1*. 47 Let *v<sub>p</sub>* (resp. *v<sub>q</sub>*) be the block that *p*'s (resp. *q*'s) CometBFT passes 48 on to the Application 49 via `RequestPrepareProposal` as proposer of round *r<sub>p</sub>* (resp *r<sub>q</sub>*), height *h*, 50 also known as the raw proposal. 51 Let *u<sub>p</sub>* (resp. *u<sub>q</sub>*) the possibly modified block *p*'s (resp. *q*'s) Application 52 returns via `ResponsePrepareProposal` to CometBFT, also known as the prepared proposal. 53 54 Process *p*'s prepared proposal can differ in two different rounds where *p* is the proposer. 55 56 * Requirement 1 [`PrepareProposal`, timeliness]: If *p*'s Application fully executes prepared blocks in 57 `PrepareProposal` and the network is in a synchronous period while processes *p* and *q* are in *r<sub>p</sub>*, 58 then the value of *TimeoutPropose* at *q* must be such that *q*'s propose timer does not time out 59 (which would result in *q* prevoting `nil` in *r<sub>p</sub>*). 60 61 Full execution of blocks at `PrepareProposal` time stands on CometBFT's critical path. Thus, 62 Requirement 1 ensures the Application or operator will set a value for `TimeoutPropose` such that the time it takes 63 to fully execute blocks in `PrepareProposal` does not interfere with CometBFT's propose timer. 64 Note that the violation of Requirement 1 may lead to further rounds, but will not 65 compromise liveness because even though `TimeoutPropose` is used as the initial 66 value for proposal timeouts, CometBFT will be dynamically adjust these timeouts 67 such that they will eventually be enough for completing `PrepareProposal`. 68 69 * Requirement 2 [`PrepareProposal`, tx-size]: When *p*'s Application calls `ResponsePrepareProposal`, the 70 total size in bytes of the transactions returned does not exceed `RequestPrepareProposal.max_tx_bytes`. 71 72 Busy blockchains might seek to gain full visibility into transactions in CometBFT's mempool, 73 rather than having visibility only on *a* subset of those transactions that fit in a block. 74 The application can do so by setting `ConsensusParams.Block.MaxBytes` to -1. 75 This instructs CometBFT (a) to enforce the maximum possible value for `MaxBytes` (100 MB) at CometBFT level, 76 and (b) to provide *all* transactions in the mempool when calling `RequestPrepareProposal`. 77 Under these settings, the aggregated size of all transactions may exceed `RequestPrepareProposal.max_tx_bytes`. 78 Hence, Requirement 2 ensures that the size in bytes of the transaction list returned by the application will never 79 cause the resulting block to go beyond its byte size limit. 80 81 * Requirement 3 [`PrepareProposal`, `ProcessProposal`, coherence]: For any two correct processes *p* and *q*, 82 if *q*'s CometBFT calls `RequestProcessProposal` on *u<sub>p</sub>*, 83 *q*'s Application returns Accept in `ResponseProcessProposal`. 84 85 Requirement 3 makes sure that blocks proposed by correct processes *always* pass the correct receiving process's 86 `ProcessProposal` check. 87 On the other hand, if there is a deterministic bug in `PrepareProposal` or `ProcessProposal` (or in both), 88 strictly speaking, this makes all processes that hit the bug byzantine. This is a problem in practice, 89 as very often validators are running the Application from the same codebase, so potentially *all* would 90 likely hit the bug at the same time. This would result in most (or all) processes prevoting `nil`, with the 91 serious consequences on CometBFT's liveness that this entails. Due to its criticality, Requirement 3 is a 92 target for extensive testing and automated verification. 93 94 * Requirement 4 [`ProcessProposal`, determinism-1]: `ProcessProposal` is a (deterministic) function of the current 95 state and the block that is about to be applied. In other words, for any correct process *p*, and any arbitrary block *u*, 96 if *p*'s CometBFT calls `RequestProcessProposal` on *u* at height *h*, 97 then *p*'s Application's acceptance or rejection **exclusively** depends on *u* and *s<sub>p,h-1</sub>*. 98 99 * Requirement 5 [`ProcessProposal`, determinism-2]: For any two correct processes *p* and *q*, and any arbitrary 100 block *u*, 101 if *p*'s (resp. *q*'s) CometBFT calls `RequestProcessProposal` on *u* at height *h*, 102 then *p*'s Application accepts *u* if and only if *q*'s Application accepts *u*. 103 Note that this requirement follows from Requirement 4 and the Agreement property of consensus. 104 105 Requirements 4 and 5 ensure that all correct processes will react in the same way to a proposed block, even 106 if the proposer is Byzantine. However, `ProcessProposal` may contain a bug that renders the 107 acceptance or rejection of the block non-deterministic, and therefore prevents processes hitting 108 the bug from fulfilling Requirements 4 or 5 (effectively making those processes Byzantine). 109 In such a scenario, CometBFT's liveness cannot be guaranteed. 110 Again, this is a problem in practice if most validators are running the same software, as they are likely 111 to hit the bug at the same point. There is currently no clear solution to help with this situation, so 112 the Application designers/implementors must proceed very carefully with the logic/implementation 113 of `ProcessProposal`. As a general rule `ProcessProposal` SHOULD always accept the block. 114 115 According to the Tendermint consensus algorithm, currently adopted in CometBFT, 116 a correct process can broadcast at most one precommit 117 message in round *r*, height *h*. 118 Since, as stated in the [Methods](./abci++_methods.md#extendvote) section, `ResponseExtendVote` 119 is only called when the consensus algorithm 120 is about to broadcast a non-`nil` precommit message, a correct process can only produce one vote extension 121 in round *r*, height *h*. 122 Let *e<sup>r</sup><sub>p</sub>* be the vote extension that the Application of a correct process *p* returns via 123 `ResponseExtendVote` in round *r*, height *h*. 124 Let *w<sup>r</sup><sub>p</sub>* be the proposed block that *p*'s CometBFT passes to the Application via `RequestExtendVote` 125 in round *r*, height *h*. 126 127 * Requirement 6 [`ExtendVote`, `VerifyVoteExtension`, coherence]: For any two different correct 128 processes *p* and *q*, if *q* receives *e<sup>r</sup><sub>p</sub>* from *p* in height *h*, *q*'s 129 Application returns Accept in `ResponseVerifyVoteExtension`. 130 131 Requirement 6 constrains the creation and handling of vote extensions in a similar way as Requirement 3 132 constrains the creation and handling of proposed blocks. 133 Requirement 6 ensures that extensions created by correct processes *always* pass the `VerifyVoteExtension` 134 checks performed by correct processes receiving those extensions. 135 However, if there is a (deterministic) bug in `ExtendVote` or `VerifyVoteExtension` (or in both), 136 we will face the same liveness issues as described for Requirement 5, as Precommit messages with invalid vote 137 extensions will be discarded. 138 139 * Requirement 7 [`VerifyVoteExtension`, determinism-1]: `VerifyVoteExtension` is a (deterministic) function of 140 the current state, the vote extension received, and the prepared proposal that the extension refers to. 141 In other words, for any correct process *p*, and any arbitrary vote extension *e*, and any arbitrary 142 block *w*, if *p*'s (resp. *q*'s) CometBFT calls `RequestVerifyVoteExtension` on *e* and *w* at height *h*, 143 then *p*'s Application's acceptance or rejection **exclusively** depends on *e*, *w* and *s<sub>p,h-1</sub>*. 144 145 * Requirement 8 [`VerifyVoteExtension`, determinism-2]: For any two correct processes *p* and *q*, 146 and any arbitrary vote extension *e*, and any arbitrary block *w*, 147 if *p*'s (resp. *q*'s) CometBFT calls `RequestVerifyVoteExtension` on *e* and *w* at height *h*, 148 then *p*'s Application accepts *e* if and only if *q*'s Application accepts *e*. 149 Note that this requirement follows from Requirement 7 and the Agreement property of consensus. 150 151 Requirements 7 and 8 ensure that the validation of vote extensions will be deterministic at all 152 correct processes. 153 Requirements 7 and 8 protect against arbitrary vote extension data from Byzantine processes, 154 in a similar way as Requirements 4 and 5 protect against arbitrary proposed blocks. 155 Requirements 7 and 8 can be violated by a bug inducing non-determinism in 156 `VerifyVoteExtension`. In this case liveness can be compromised. 157 Extra care should be put in the implementation of `ExtendVote` and `VerifyVoteExtension`. 158 As a general rule, `VerifyVoteExtension` SHOULD always accept the vote extension. 159 160 * Requirement 9 [*all*, no-side-effects]: *p*'s calls to `RequestPrepareProposal`, 161 `RequestProcessProposal`, `RequestExtendVote`, and `RequestVerifyVoteExtension` at height *h* do 162 not modify *s<sub>p,h-1</sub>*. 163 164 165 * Requirement 10 [`ExtendVote`, `FinalizeBlock`, non-dependency]: for any correct process *p*, 166 and any vote extension *e* that *p* received at height *h*, the computation of 167 *s<sub>p,h</sub>* does not depend on *e*. 168 169 The call to correct process *p*'s `RequestFinalizeBlock` at height *h*, with block *v<sub>p,h</sub>* 170 passed as parameter, creates state *s<sub>p,h</sub>*. 171 Additionally, *p*'s `FinalizeBlock` creates a set of transaction results *T<sub>p,h</sub>*. 172 173 * Requirement 11 [`FinalizeBlock`, determinism-1]: For any correct process *p*, 174 *s<sub>p,h</sub>* exclusively depends on *s<sub>p,h-1</sub>* and *v<sub>p,h</sub>*. 175 176 * Requirement 12 [`FinalizeBlock`, determinism-2]: For any correct process *p*, 177 the contents of *T<sub>p,h</sub>* exclusively depend on *s<sub>p,h-1</sub>* and *v<sub>p,h</sub>*. 178 179 Note that Requirements 11 and 12, combined with the Agreement property of consensus ensure 180 state machine replication, i.e., the Application state evolves consistently at all correct processes. 181 182 Also, notice that neither `PrepareProposal` nor `ExtendVote` have determinism-related 183 requirements associated. 184 Indeed, `PrepareProposal` is not required to be deterministic: 185 186 * *u<sub>p</sub>* may depend on *v<sub>p</sub>* and *s<sub>p,h-1</sub>*, but may also depend on other values or operations. 187 * *v<sub>p</sub> = v<sub>q</sub> ⇏ u<sub>p</sub> = u<sub>q</sub>*. 188 189 Likewise, `ExtendVote` can also be non-deterministic: 190 191 * *e<sup>r</sup><sub>p</sub>* may depend on *w<sup>r</sup><sub>p</sub>* and *s<sub>p,h-1</sub>*, 192 but may also depend on other values or operations. 193 * *w<sup>r</sup><sub>p</sub> = w<sup>r</sup><sub>q</sub> ⇏ 194 e<sup>r</sup><sub>p</sub> = e<sup>r</sup><sub>q</sub>* 195 196 ### Mempool Connection Requirements 197 198 Let *CheckTxCodes<sub>tx,p,h</sub>* denote the set of result codes returned by *p*'s Application, 199 via `ResponseCheckTx`, 200 to successive calls to `RequestCheckTx` occurring while the Application is at height *h* 201 and having transaction *tx* as parameter. 202 *CheckTxCodes<sub>tx,p,h</sub>* is a set since *p*'s Application may 203 return different result codes during height *h*. 204 If *CheckTxCodes<sub>tx,p,h</sub>* is a singleton set, i.e. the Application always returned 205 the same result code in `ResponseCheckTx` while at height *h*, 206 we define *CheckTxCode<sub>tx,p,h</sub>* as the singleton value of *CheckTxCodes<sub>tx,p,h</sub>*. 207 If *CheckTxCodes<sub>tx,p,h</sub>* is not a singleton set, *CheckTxCode<sub>tx,p,h</sub>* is undefined. 208 Let predicate *OK(CheckTxCode<sub>tx,p,h</sub>)* denote whether *CheckTxCode<sub>tx,p,h</sub>* is `SUCCESS`. 209 210 * Requirement 13 [`CheckTx`, eventual non-oscillation]: For any transaction *tx*, 211 there exists a boolean value *b*, 212 and a height *h<sub>stable</sub>* such that, 213 for any correct process *p*, 214 *CheckTxCode<sub>tx,p,h</sub>* is defined, and 215 *OK(CheckTxCode<sub>tx,p,h</sub>) = b* 216 for any height *h ≥ h<sub>stable</sub>*. 217 218 Requirement 13 ensures that 219 a transaction will eventually stop oscillating between `CheckTx` success and failure 220 if it stays in *p's* mempool for long enough. 221 This condition on the Application's behavior allows the mempool to ensure that 222 a transaction will leave the mempool of all full nodes, 223 either because it is expunged everywhere due to failing `CheckTx` calls, 224 or because it stays valid long enough to be gossipped, proposed and decided. 225 Although Requirement 13 defines a global *h<sub>stable</sub>*, application developers 226 can consider such stabilization height as local to process *p* (*h<sub>p,stable</sub>*), 227 without loss for generality. 228 In contrast, the value of *b* MUST be the same across all processes. 229 230 ## Managing the Application state and related topics 231 232 ### Connection State 233 234 CometBFT maintains four concurrent ABCI++ connections, namely 235 [Consensus Connection](#consensus-connection), 236 [Mempool Connection](#mempool-connection), 237 [Info/Query Connection](#infoquery-connection), and 238 [Snapshot Connection](#snapshot-connection). 239 It is common for an application to maintain a distinct copy of 240 the state for each connection, which are synchronized upon `Commit` calls. 241 242 #### Concurrency 243 244 In principle, each of the four ABCI++ connections operates concurrently with one 245 another. This means applications need to ensure access to state is 246 thread safe. Both the 247 [default in-process ABCI client](https://github.com/aakash4dev/cometbft/blob/main/abci/client/local_client.go#L13) 248 and the 249 [default Go ABCI server](https://github.com/aakash4dev/cometbft/blob/main/abci/server/socket_server.go#L20) 250 use a global lock to guard the handling of events across all connections, so they are not 251 concurrent at all. This means whether your app is compiled in-process with 252 CometBFT using the `NewLocalClient`, or run out-of-process using the `SocketServer`, 253 ABCI messages from all connections are received in sequence, one at a 254 time. 255 256 The existence of this global mutex means Go application developers can get thread safety for application state by routing all reads and writes through the ABCI system. Thus it may be unsafe to expose application state directly to an RPC interface, and unless explicit measures are taken, all queries should be routed through the ABCI Query method. 257 258 <!-- 259 This is no longer the case starting from v0.36.0: the global locks have been removed and it is 260 up to the Application to synchronize access to its state when handling 261 ABCI++ methods on all connections. 262 --> 263 264 <!-- 265 TODO CHeck with Sergio whether this is still the case 266 --> 267 <!-- 268 Nevertheless, as all ABCI calls are now synchronous, ABCI messages using the same connection are 269 still received in sequence. 270 --> 271 272 #### FinalizeBlock 273 274 When the consensus algorithm decides on a block, CometBFT uses `FinalizeBlock` to send the 275 decided block's data to the Application, which uses it to transition its state, but MUST NOT persist it; 276 persisting MUST be done during `Commit`. 277 278 The Application must remember the latest height from which it 279 has run a successful `Commit` so that it can tell CometBFT where to 280 pick up from when it recovers from a crash. See information on the Handshake 281 [here](#crash-recovery). 282 283 #### Commit 284 285 The Application should persist its state during `Commit`, before returning from it. 286 287 Before invoking `Commit`, CometBFT locks the mempool and flushes the mempool connection. This ensures that 288 no new messages 289 will be received on the mempool connection during this processing step, providing an opportunity to safely 290 update all four 291 connection states to the latest committed state at the same time. 292 293 When `Commit` returns, CometBFT unlocks the mempool. 294 295 WARNING: if the ABCI app logic processing the `Commit` message sends a 296 `/broadcast_tx_sync` or `/broadcast_tx` and waits for the response 297 before proceeding, it will deadlock. Executing `broadcast_tx` calls 298 involves acquiring the mempool lock that CometBFT holds during the `Commit` call. 299 Synchronous mempool-related calls must be avoided as part of the sequential logic of the 300 `Commit` function. 301 302 #### Candidate States 303 304 CometBFT calls `PrepareProposal` when it is about to send a proposed block to the network. 305 Likewise, CometBFT calls `ProcessProposal` upon reception of a proposed block from the 306 network. The proposed block's data 307 that is disclosed to the Application by these two methods is the following: 308 309 * the transaction list 310 * the `LastCommit` referring to the previous block 311 * the block header's hash (except in `PrepareProposal`, where it is not known yet) 312 * list of validators that misbehaved 313 * the block's timestamp 314 * `NextValidatorsHash` 315 * Proposer address 316 317 The Application may decide to *immediately* execute the given block (i.e., upon `PrepareProposal` 318 or `ProcessProposal`). There are two main reasons why the Application may want to do this: 319 320 * *Avoiding invalid transactions in blocks*. 321 In order to be sure that the block does not contain *any* invalid transaction, there may be 322 no way other than fully executing the transactions in the block as though it was the *decided* 323 block. 324 * *Quick `FinalizeBlock` execution*. 325 Upon reception of the decided block via `FinalizeBlock`, if that same block was executed 326 upon `PrepareProposal` or `ProcessProposal` and the resulting state was kept in memory, the 327 Application can simply apply that state (faster) to the main state, rather than reexecuting 328 the decided block (slower). 329 330 `PrepareProposal`/`ProcessProposal` can be called many times for a given height. Moreover, 331 it is not possible to accurately predict which of the blocks proposed in a height will be decided, 332 being delivered to the Application in that height's `FinalizeBlock`. 333 Therefore, the state resulting from executing a proposed block, denoted a *candidate state*, should 334 be kept in memory as a possible final state for that height. When `FinalizeBlock` is called, the Application should 335 check if the decided block corresponds to one of its candidate states; if so, it will apply it as 336 its *ExecuteTxState* (see [Consensus Connection](#consensus-connection) below), 337 which will be persisted during the upcoming `Commit` call. 338 339 Under adverse conditions (e.g., network instability), the consensus algorithm might take many rounds. 340 In this case, potentially many proposed blocks will be disclosed to the Application for a given height. 341 By the nature of Tendermint consensus algorithm, currently adopted in CometBFT, the number of proposed blocks received by the Application 342 for a particular height cannot be bound, so Application developers must act with care and use mechanisms 343 to bound memory usage. As a general rule, the Application should be ready to discard candidate states 344 before `FinalizeBlock`, even if one of them might end up corresponding to the 345 decided block and thus have to be reexecuted upon `FinalizeBlock`. 346 347 ### States and ABCI++ Connections 348 349 #### Consensus Connection 350 351 The Consensus Connection should maintain an *ExecuteTxState* — the working state 352 for block execution. It should be updated by the call to `FinalizeBlock` 353 during block execution and committed to disk as the "latest 354 committed state" during `Commit`. Execution of a proposed block (via `PrepareProposal`/`ProcessProposal`) 355 **must not** update the *ExecuteTxState*, but rather be kept as a separate candidate state until `FinalizeBlock` 356 confirms which of the candidate states (if any) can be used to update *ExecuteTxState*. 357 358 #### Mempool Connection 359 360 The mempool Connection maintains *CheckTxState*. CometBFT sequentially processes an incoming 361 transaction (via RPC from client or P2P from the gossip layer) against *CheckTxState*. 362 If the processing does not return any error, the transaction is accepted into the mempool 363 and CometBFT starts gossipping it. 364 *CheckTxState* should be reset to the latest committed state 365 at the end of every `Commit`. 366 367 During the execution of a consensus instance, the *CheckTxState* may be updated concurrently with the 368 *ExecuteTxState*, as messages may be sent concurrently on the Consensus and Mempool connections. 369 At the end of the consensus instance, as described above, CometBFT locks the mempool and flushes 370 the mempool connection before calling `Commit`. This ensures that all pending `CheckTx` calls are 371 responded to and no new ones can begin. 372 373 After the `Commit` call returns, while still holding the mempool lock, `CheckTx` is run again on all 374 transactions that remain in the node's local mempool after filtering those included in the block. 375 Parameter `Type` in `RequestCheckTx` 376 indicates whether an incoming transaction is new (`CheckTxType_New`), or a 377 recheck (`CheckTxType_Recheck`). 378 379 Finally, after re-checking transactions in the mempool, CometBFT will unlock 380 the mempool connection. New transactions are once again able to be processed through `CheckTx`. 381 382 Note that `CheckTx` is just a weak filter to keep invalid transactions out of the mempool and, 383 ultimately, ouf of the blockchain. 384 Since the transaction cannot be guaranteed to be checked against the exact same state as it 385 will be executed as part of a (potential) decided block, `CheckTx` shouldn't check *everything* 386 that affects the transaction's validity, in particular those checks whose validity may depend on 387 transaction ordering. `CheckTx` is weak because a Byzantine node need not care about `CheckTx`; 388 it can propose a block full of invalid transactions if it wants. The mechanism ABCI++ has 389 in place for dealing with such behavior is `ProcessProposal`. 390 391 ##### Replay Protection 392 393 It is possible for old transactions to be sent again to the Application. This is typically 394 undesirable for all transactions, except for a generally small subset of them which are idempotent. 395 396 The mempool has a mechanism to prevent duplicated transactions from being processed. 397 This mechanism is nevertheless best-effort (currently based on the indexer) 398 and does not provide any guarantee of non duplication. 399 It is thus up to the Application to implement an application-specific 400 replay protection mechanism with strong guarantees as part of the logic in `CheckTx`. 401 402 #### Info/Query Connection 403 404 The Info (or Query) Connection should maintain a `QueryState`. This connection has two 405 purposes: 1) having the application answer the queries CometBFT receives from users 406 (see section [Query](#query)), 407 and 2) synchronizing CometBFT and the Application at start up time (see 408 [Crash Recovery](#crash-recovery)) 409 or after state sync (see [State Sync](#state-sync)). 410 411 `QueryState` is a read-only copy of *ExecuteTxState* as it was after the last 412 `Commit`, i.e. 413 after the full block has been processed and the state committed to disk. 414 415 #### Snapshot Connection 416 417 The Snapshot Connection is used to serve state sync snapshots for other nodes 418 and/or restore state sync snapshots to a local node being bootstrapped. 419 Snapshot management is optional: an Application may choose not to implement it. 420 421 For more information, see Section [State Sync](#state-sync). 422 423 ### Transaction Results 424 425 The Application is expected to return a list of 426 [`ExecTxResult`](./abci%2B%2B_methods.md#exectxresult) in 427 [`ResponseFinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). The list of transaction 428 results MUST respect the same order as the list of transactions delivered via 429 [`RequestFinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). 430 This section discusses the fields inside this structure, along with the fields in 431 [`ResponseCheckTx`](./abci%2B%2B_methods.md#checktx), 432 whose semantics are similar. 433 434 The `Info` and `Log` fields are 435 non-deterministic values for debugging/convenience purposes. CometBFT logs them but they 436 are otherwise ignored. 437 438 #### Gas 439 440 Ethereum introduced the notion of *gas* as an abstract representation of the 441 cost of the resources consumed by nodes when processing a transaction. Every operation in the 442 Ethereum Virtual Machine uses some amount of gas. 443 Gas has a market-variable price based on which miners can accept or reject to execute a 444 particular operation. 445 446 Users propose a maximum amount of gas for their transaction; if the transaction uses less, they get 447 the difference credited back. CometBFT adopts a similar abstraction, 448 though uses it only optionally and weakly, allowing applications to define 449 their own sense of the cost of execution. 450 451 In CometBFT, the [ConsensusParams.Block.MaxGas](#consensus-parameters) limits the amount of 452 total gas that can be used by all transactions in a block. 453 The default value is `-1`, which means the block gas limit is not enforced, or that the concept of 454 gas is meaningless. 455 456 Responses contain a `GasWanted` and `GasUsed` field. The former is the maximum 457 amount of gas the sender of a transaction is willing to use, and the latter is how much it actually 458 used. Applications should enforce that `GasUsed <= GasWanted` — i.e. transaction execution 459 or validation should fail before it can use more resources than it requested. 460 461 When `MaxGas > -1`, CometBFT enforces the following rules: 462 463 * `GasWanted <= MaxGas` for every transaction in the mempool 464 * `(sum of GasWanted in a block) <= MaxGas` when proposing a block 465 466 If `MaxGas == -1`, no rules about gas are enforced. 467 468 In v0.34.x and earlier versions, CometBFT does not enforce anything about Gas in consensus, 469 only in the mempool. 470 This means it does not guarantee that committed blocks satisfy these rules. 471 It is the application's responsibility to return non-zero response codes when gas limits are exceeded 472 when executing the transactions of a block. 473 Since the introduction of `PrepareProposal` and `ProcessProposal` in v.0.37.x, it is now possible 474 for the Application to enforce that all blocks proposed (and voted for) in consensus — and thus all 475 blocks decided — respect the `MaxGas` limits described above. 476 477 Since the Application should enforce that `GasUsed <= GasWanted` when executing a transaction, and 478 it can use `PrepareProposal` and `ProcessProposal` to enforce that `(sum of GasWanted in a block) <= MaxGas` 479 in all proposed or prevoted blocks, 480 we have: 481 482 * `(sum of GasUsed in a block) <= MaxGas` for every block 483 484 The `GasUsed` field is ignored by CometBFT. 485 486 #### Specifics of `ResponseCheckTx` 487 488 If `Code != 0`, it will be rejected from the mempool and hence 489 not broadcasted to other peers and not included in a proposal block. 490 491 `Data` contains the result of the `CheckTx` transaction execution, if any. It does not need to be 492 deterministic since, given a transaction, nodes' Applications 493 might have a different *CheckTxState* values when they receive it and check their validity 494 via `CheckTx`. 495 CometBFT ignores this value in `ResponseCheckTx`. 496 497 From v0.34.x on, there is a `Priority` field in `ResponseCheckTx` that can be 498 used to explicitly prioritize transactions in the mempool for inclusion in a block 499 proposal. 500 501 #### Specifics of `ExecTxResult` 502 503 `FinalizeBlock` is the workhorse of the blockchain. CometBFT delivers the decided block, 504 including the list of all its transactions synchronously to the Application. 505 The block delivered (and thus the transaction order) is the same at all correct nodes as guaranteed 506 by the Agreement property of consensus. 507 508 The `Data` field in `ExecTxResult` contains an array of bytes with the transaction result. 509 It must be deterministic (i.e., the same value must be returned at all nodes), but it can contain arbitrary 510 data. Likewise, the value of `Code` must be deterministic. 511 If `Code != 0`, the transaction will be marked invalid, 512 though it is still included in the block. Invalid transactions are not indexed, as they are 513 considered analogous to those that failed `CheckTx`. 514 515 Both the `Code` and `Data` are included in a structure that is hashed into the 516 `LastResultsHash` of the block header in the next height. 517 518 `Events` include any events for the execution, which CometBFT will use to index 519 the transaction by. This allows transactions to be queried according to what 520 events took place during their execution. 521 522 ### Updating the Validator Set 523 524 The application may set the validator set during 525 [`InitChain`](./abci%2B%2B_methods.md#initchain), and may update it during 526 [`FinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). In both cases, a structure of type 527 [`ValidatorUpdate`](./abci%2B%2B_methods.md#validatorupdate) is returned. 528 529 The `InitChain` method, used to initialize the Application, can return a list of validators. 530 If the list is empty, CometBFT will use the validators loaded from the genesis 531 file. 532 If the list returned by `InitChain` is not empty, CometBFT will use its contents as the validator set. 533 This way the application can set the initial validator set for the 534 blockchain. 535 536 Applications must ensure that a single set of validator updates does not contain duplicates, i.e. 537 a given public key can only appear once within a given update. If an update includes 538 duplicates, the block execution will fail irrecoverably. 539 540 Structure `ValidatorUpdate` contains a public key, which is used to identify the validator: 541 The public key currently supports three types: 542 543 * `ed25519` 544 * `secp256k1` 545 * `sr25519` 546 547 Structure `ValidatorUpdate` also contains an `ìnt64` field denoting the validator's new power. 548 Applications must ensure that 549 `ValidatorUpdate` structures abide by the following rules: 550 551 * power must be non-negative 552 * if power is set to 0, the validator must be in the validator set; it will be removed from the set 553 * if power is greater than 0: 554 * if the validator is not in the validator set, it will be added to the 555 set with the given power 556 * if the validator is in the validator set, its power will be adjusted to the given power 557 * the total power of the new validator set must not exceed `MaxTotalVotingPower`, where 558 `MaxTotalVotingPower = MaxInt64 / 8` 559 560 Note the updates returned after processing the block at height `H` will only take effect 561 at block `H+2` (see Section [Methods](./abci%2B%2B_methods.md)). 562 563 ### Consensus Parameters 564 565 `ConsensusParams` are global parameters that apply to all validators in a blockchain. 566 They enforce certain limits in the blockchain, like the maximum size 567 of blocks, amount of gas used in a block, and the maximum acceptable age of 568 evidence. They can be set in 569 [`InitChain`](./abci%2B%2B_methods.md#initchain), and updated in 570 [`FinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). 571 These parameters are deterministically set and/or updated by the Application, so 572 all full nodes have the same value at a given height. 573 574 #### List of Parameters 575 576 These are the current consensus parameters (as of v0.37.x): 577 578 1. [BlockParams.MaxBytes](#blockparamsmaxbytes) 579 2. [BlockParams.MaxGas](#blockparamsmaxgas) 580 3. [EvidenceParams.MaxAgeDuration](#evidenceparamsmaxageduration) 581 4. [EvidenceParams.MaxAgeNumBlocks](#evidenceparamsmaxagenumblocks) 582 5. [EvidenceParams.MaxBytes](#evidenceparamsmaxbytes) 583 6. [ValidatorParams.PubKeyTypes](#validatorparamspubkeytypes) 584 7. [VersionParams.App](#versionparamsapp) 585 <!-- 586 6. [SynchronyParams.MessageDelay](#synchronyparamsmessagedelay) 587 7. [SynchronyParams.Precision](#synchronyparamsprecision) 588 8. [TimeoutParams.Propose](#timeoutparamspropose) 589 9. [TimeoutParams.ProposeDelta](#timeoutparamsproposedelta) 590 10. [TimeoutParams.Vote](#timeoutparamsvote) 591 11. [TimeoutParams.VoteDelta](#timeoutparamsvotedelta) 592 12. [TimeoutParams.Commit](#timeoutparamscommit) 593 13. [TimeoutParams.BypassCommitTimeout](#timeoutparamsbypasscommittimeout) 594 --> 595 596 ##### BlockParams.MaxBytes 597 598 The maximum size of a complete Protobuf encoded block. 599 This is enforced by the consensus algorithm. 600 601 This implies a maximum transaction size that is this `MaxBytes`, less the expected size of 602 the header, the validator set, and any included evidence in the block. 603 604 If the Application wants full control over the size of blocks, 605 it can do so by enforcing a byte limit set up at the Application level. 606 This Application-internal limit is used by `PrepareProposal` to bound the total size 607 of transactions it returns, and by `ProcessProposal` to reject any received block 608 whose total transaction size is bigger than the enforced limit. 609 In such case, the Application MAY set `MaxBytes` to -1. 610 611 If the Application sets value -1, consensus will: 612 613 - consider that the actual value to enforce is 100 MB 614 - will provide *all* transactions in the mempool in calls to `PrepareProposal` 615 616 Must have `MaxBytes == -1` OR `0 < MaxBytes <= 100 MB`. 617 618 ##### BlockParams.MaxGas 619 620 The maximum of the sum of `GasWanted` that will be allowed in a proposed block. 621 This is *not* enforced by the consensus algorithm. 622 It is left to the Application to enforce (ie. if transactions are included past the 623 limit, they should return non-zero codes). It is used by CometBFT to limit the 624 transactions included in a proposed block. 625 626 Must have `MaxGas >= -1`. 627 If `MaxGas == -1`, no limit is enforced. 628 629 ##### EvidenceParams.MaxAgeDuration 630 631 This is the maximum age of evidence in time units. 632 This is enforced by the consensus algorithm. 633 634 If a block includes evidence older than this (AND the evidence was created more 635 than `MaxAgeNumBlocks` ago), the block will be rejected (validators won't vote 636 for it). 637 638 Must have `MaxAgeDuration > 0`. 639 640 ##### EvidenceParams.MaxAgeNumBlocks 641 642 This is the maximum age of evidence in blocks. 643 This is enforced by the consensus algorithm. 644 645 If a block includes evidence older than this (AND the evidence was created more 646 than `MaxAgeDuration` ago), the block will be rejected (validators won't vote 647 for it). 648 649 Must have `MaxAgeNumBlocks > 0`. 650 651 ##### EvidenceParams.MaxBytes 652 653 This is the maximum size of total evidence in bytes that can be committed to a 654 single block. It should fall comfortably under the max block bytes. 655 656 Its value must not exceed the size of 657 a block minus its overhead ( ~ `BlockParams.MaxBytes`). 658 659 Must have `MaxBytes > 0`. 660 661 ##### ValidatorParams.PubKeyTypes 662 663 The parameter restricts the type of keys validators can use. The parameter uses ABCI pubkey naming, not Amino names. 664 665 ##### VersionParams.App 666 667 This is the version of the ABCI application. 668 <!-- 669 ##### SynchronyParams.MessageDelay 670 671 This sets a bound on how long a proposal message may take to reach all 672 validators on a network and still be considered valid. 673 674 This parameter is part of the 675 [proposer-based timestamps](../consensus/proposer-based-timestamp) 676 (PBTS) algorithm. 677 678 679 ##### SynchronyParams.Precision 680 681 This sets a bound on how skewed a proposer's clock may be from any validator 682 on the network while still producing valid proposals. 683 684 This parameter is part of the 685 [proposer-based timestamps](../consensus/proposer-based-timestamp) 686 (PBTS) algorithm. 687 688 689 ##### TimeoutParams.Propose 690 691 Timeout in ms of the propose step of the consensus algorithm. 692 This value is the initial timeout at every height (round 0). 693 694 The value in subsequent rounds is modified by parameter `ProposeDelta`. 695 When a new height is started, the `Propose` timeout value is reset to this 696 parameter. 697 698 If a node waiting for a proposal message does not receive one matching its 699 current height and round before this timeout, the node will issue a 700 `nil` prevote for the round and advance to the next step. 701 702 ##### TimeoutParams.ProposeDelta 703 704 Increment in ms to be added to the `Propose` timeout every time the 705 consensus algorithm advances one round in a given height. 706 707 When a new height is started, the `Propose` timeout value is reset. 708 709 ##### TimeoutParams.Vote 710 711 Timeout in ms of the prevote and precommit steps of the consensus 712 algorithm. 713 This value is the initial timeout at every height (round 0). 714 715 The value in subsequent rounds is modified by parameter `VoteDelta`. 716 When a new height is started, the `Vote` timeout value is reset to this 717 parameter. 718 719 The `Vote` timeout does not begin until a quorum of votes has been received. 720 Once a quorum of votes has been seen and this timeout elapses, Tendermint will 721 proceed to the next step of the consensus algorithm. If Tendermint receives 722 all of the remaining votes before the end of the timeout, it will proceed 723 to the next step immediately. 724 725 ##### TimeoutParams.VoteDelta 726 727 Increment in ms to be added to the `Vote` timeout every time the 728 consensus algorithm advances one round in a given height. 729 730 When a new height is started, the `Vote` timeout value is reset. 731 732 ##### TimeoutParams.Commit 733 734 This configures how long the consensus algorithm will wait after receiving a quorum of 735 precommits before beginning consensus for the next height. This can be 736 used to allow slow precommits to arrive for inclusion in the next height 737 before progressing. 738 739 ##### TimeoutParams.BypassCommitTimeout 740 741 This configures the node to proceed immediately to the next height once the 742 node has received all precommits for a block, forgoing the remaining commit timeout. 743 Setting this parameter to `false` (the default) causes Tendermint to wait 744 for the full commit timeout configured in `TimeoutParams.Commit`. 745 --> 746 747 ##### ABCIParams.VoteExtensionsEnableHeight 748 749 This parameter is either 0 or a positive height at which vote extensions 750 become mandatory. If the value is zero (which is the default), vote 751 extensions are not required. Otherwise, at all heights greater than the 752 configured height `H` vote extensions must be present (even if empty). 753 When the configured height `H` is reached, `PrepareProposal` will not 754 include vote extensions yet, but `ExtendVote` and `VerifyVoteExtension` will 755 be called. Then, when reaching height `H+1`, `PrepareProposal` will 756 include the vote extensions from height `H`. For all heights after `H` 757 758 * vote extensions cannot be disabled, 759 * they are mandatory: all precommit messages sent MUST have an extension 760 attached. Nevertheless, the application MAY provide 0-length 761 extensions. 762 763 Must always be set to a future height. Once set to a value different from 764 0, its value must not be changed. 765 766 #### Updating Consensus Parameters 767 768 The application may set the `ConsensusParams` during 769 [`InitChain`](./abci%2B%2B_methods.md#initchain), 770 and update them during 771 [`FinalizeBlock`](./abci%2B%2B_methods.md#finalizeblock). 772 If the `ConsensusParams` is empty, it will be ignored. Each field 773 that is not empty will be applied in full. For instance, if updating the 774 `Block.MaxBytes`, applications must also set the other `Block` fields (like 775 `Block.MaxGas`), even if they are unchanged, as they will otherwise cause the 776 value to be updated to the default. 777 778 ##### `InitChain` 779 780 `ResponseInitChain` includes a `ConsensusParams` parameter. 781 If `ConsensusParams` is `nil`, CometBFT will use the params loaded in the genesis 782 file. If `ConsensusParams` is not `nil`, CometBFT will use it. 783 This way the application can determine the initial consensus parameters for the 784 blockchain. 785 786 ##### `FinalizeBlock`, `PrepareProposal`/`ProcessProposal` 787 788 `ResponseFinalizeBlock` accepts a `ConsensusParams` parameter. 789 If `ConsensusParams` is `nil`, CometBFT will do nothing. 790 If `ConsensusParams` is not `nil`, CometBFT will use it. 791 This way the application can update the consensus parameters over time. 792 793 The updates returned in block `H` will take effect right away for block 794 `H+1`. 795 796 ### `Query` 797 798 `Query` is a generic method with lots of flexibility to enable diverse sets 799 of queries on application state. CometBFT makes use of `Query` to filter new peers 800 based on ID and IP, and exposes `Query` to the user over RPC. 801 802 Note that calls to `Query` are not replicated across nodes, but rather query the 803 local node's state - hence they may return stale reads. For reads that require 804 consensus, use a transaction. 805 806 The most important use of `Query` is to return Merkle proofs of the application state at some height 807 that can be used for efficient application-specific light-clients. 808 809 Note CometBFT has technically no requirements from the `Query` 810 message for normal operation - that is, the ABCI app developer need not implement 811 Query functionality if they do not wish to. 812 813 #### Query Proofs 814 815 The CometBFT block header includes a number of hashes, each providing an 816 anchor for some type of proof about the blockchain. The `ValidatorsHash` enables 817 quick verification of the validator set, the `DataHash` gives quick 818 verification of the transactions included in the block. 819 820 The `AppHash` is unique in that it is application specific, and allows for 821 application-specific Merkle proofs about the state of the application. 822 While some applications keep all relevant state in the transactions themselves 823 (like Bitcoin and its UTXOs), others maintain a separated state that is 824 computed deterministically *from* transactions, but is not contained directly in 825 the transactions themselves (like Ethereum contracts and accounts). 826 For such applications, the `AppHash` provides a much more efficient way to verify light-client proofs. 827 828 ABCI applications can take advantage of more efficient light-client proofs for 829 their state as follows: 830 831 * return the Merkle root of the deterministic application state in 832 `ResponseFinalizeBlock.Data`. This Merkle root will be included as the `AppHash` in the next block. 833 * return efficient Merkle proofs about that application state in `ResponseQuery.Proof` 834 that can be verified using the `AppHash` of the corresponding block. 835 836 For instance, this allows an application's light-client to verify proofs of 837 absence in the application state, something which is much less efficient to do using the block hash. 838 839 Some applications (eg. Ethereum, Cosmos-SDK) have multiple "levels" of Merkle trees, 840 where the leaves of one tree are the root hashes of others. To support this, and 841 the general variability in Merkle proofs, the `ResponseQuery.Proof` has some minimal structure: 842 843 ```protobuf 844 message ProofOps { 845 repeated ProofOp ops = 1 846 } 847 848 message ProofOp { 849 string type = 1; 850 bytes key = 2; 851 bytes data = 3; 852 } 853 ``` 854 855 Each `ProofOp` contains a proof for a single key in a single Merkle tree, of the specified `type`. 856 This allows ABCI to support many different kinds of Merkle trees, encoding 857 formats, and proofs (eg. of presence and absence) just by varying the `type`. 858 The `data` contains the actual encoded proof, encoded according to the `type`. 859 When verifying the full proof, the root hash for one ProofOp is the value being 860 verified for the next ProofOp in the list. The root hash of the final ProofOp in 861 the list should match the `AppHash` being verified against. 862 863 #### Peer Filtering 864 865 When CometBFT connects to a peer, it sends two queries to the ABCI application 866 using the following paths, with no additional data: 867 868 * `/p2p/filter/addr/<IP:PORT>`, where `<IP:PORT>` denote the IP address and 869 the port of the connection 870 * `p2p/filter/id/<ID>`, where `<ID>` is the peer node ID (ie. the 871 pubkey.Address() for the peer's PubKey) 872 873 If either of these queries return a non-zero ABCI code, CometBFT will refuse 874 to connect to the peer. 875 876 #### Paths 877 878 Queries are directed at paths, and may optionally include additional data. 879 880 The expectation is for there to be some number of high level paths 881 differentiating concerns, like `/p2p`, `/store`, and `/app`. Currently, 882 CometBFT only uses `/p2p`, for filtering peers. For more advanced use, see the 883 implementation of 884 [Query in the Cosmos-SDK](https://github.com/cosmos/cosmos-sdk/blob/e2037f7696fed4fdd4bc076f9e7053fe8178a881/baseapp/abci.go#L557-L565). 885 886 ### Crash Recovery 887 888 CometBFT and the application are expected to crash together and there should not 889 exist a scenario where the application has persisted state of a height greater than the 890 latest height persisted by CometBFT. 891 892 In practice, persisting the state of a height consists of three steps, the last of which 893 is the call to the application's `Commit` method, the only place where the application is expected to 894 persist/commit its state. 895 On startup (upon recovery), CometBFT calls the `Info` method on the Info Connection to get the latest 896 committed state of the app. The app MUST return information consistent with the 897 last block for which it successfully completed `Commit`. 898 899 The three steps performed before the state of a height is considered persisted are: 900 - The block is stored by CometBFT in the blockstore 901 - CometBFT has stored the state returned by the application through `FinalizeBlockResponse` 902 - The application has committed its state within `Commit`. 903 904 The following diagram depicts the order in which these events happen, and the corresponding 905 ABCI functions that are called and executed by CometBFT and the application: 906 907 908 ``` 909 APP: Execute block Persist application state 910 / return ResultFinalizeBlock / 911 / / 912 Event: ------------- block_stored ------------ / ------------ state_stored --------------- / ----- app_persisted_state 913 | / | / | 914 CometBFT: Decide --- Persist block -- Call FinalizeBlock - Persist results ---------- Call Commit -- 915 on in the (txResults, validator 916 Block block store updates...) 917 918 ``` 919 920 As these three steps are not atomic, we observe different cases based on which steps have been executed 921 before the crash occurred 922 (we assume that at least `block_stored` has been executed, otherwise, there is no state persisted, 923 and the operations for this height are repeated entirely): 924 925 - `block_stored`: we replay `FinalizeBlock` and the steps afterwards. 926 - `block_stored` and `state_stored`: As the app did not persist its state within `Commit`, we need to re-execute 927 `FinalizeBlock` to retrieve the results and compare them to the state stored by CometBFT within `state_stored`. 928 The expected case is that the states will match, otherwise CometBFT panics. 929 - `block_stored`, `state_stored`, `app_persisted_state`: we move on to the next height. 930 931 Based on the sequence of these events, CometBFT will panic if any of the steps in the sequence happen out of order, 932 that is if: 933 - The application has persisted a block at a height higher than the blocked saved during `state_stored`. 934 - The `block_stored` step persisted a block at a height smaller than the `state_stored` 935 - And the difference between the heights of the blocks persisted by `state_stored` and `block_stored` is more 936 than 1 (this corresponds to a scenario where we stored two blocks in the block store but never persisted the state of the first 937 block, which should never happen). 938 939 A special case is when a crash happens before the first block is committed - that is, after calling 940 `InitChain`. In that case, the application's state should still be at height 0 and thus `InitChain` 941 will be called again. 942 943 944 ### State Sync 945 946 A new node joining the network can simply join consensus at the genesis height and replay all 947 historical blocks until it is caught up. However, for large chains this can take a significant 948 amount of time, often on the order of days or weeks. 949 950 State sync is an alternative mechanism for bootstrapping a new node, where it fetches a snapshot 951 of the state machine at a given height and restores it. Depending on the application, this can 952 be several orders of magnitude faster than replaying blocks. 953 954 Note that state sync does not currently backfill historical blocks, so the node will have a 955 truncated block history - users are advised to consider the broader network implications of this in 956 terms of block availability and auditability. This functionality may be added in the future. 957 958 For details on the specific ABCI calls and types, see the 959 [methods](abci%2B%2B_methods.md) section. 960 961 #### Taking Snapshots 962 963 Applications that want to support state syncing must take state snapshots at regular intervals. How 964 this is accomplished is entirely up to the application. A snapshot consists of some metadata and 965 a set of binary chunks in an arbitrary format: 966 967 * `Height (uint64)`: The height at which the snapshot is taken. It must be taken after the given 968 height has been committed, and must not contain data from any later heights. 969 970 * `Format (uint32)`: An arbitrary snapshot format identifier. This can be used to version snapshot 971 formats, e.g. to switch from Protobuf to MessagePack for serialization. The application can use 972 this when restoring to choose whether to accept or reject a snapshot. 973 974 * `Chunks (uint32)`: The number of chunks in the snapshot. Each chunk contains arbitrary binary 975 data, and should be less than 16 MB; 10 MB is a good starting point. 976 977 * `Hash ([]byte)`: An arbitrary hash of the snapshot. This is used to check whether a snapshot is 978 the same across nodes when downloading chunks. 979 980 * `Metadata ([]byte)`: Arbitrary snapshot metadata, e.g. chunk hashes for verification or any other 981 necessary info. 982 983 For a snapshot to be considered the same across nodes, all of these fields must be identical. When 984 sent across the network, snapshot metadata messages are limited to 4 MB. 985 986 When a new node is running state sync and discovering snapshots, CometBFT will query an existing 987 application via the ABCI `ListSnapshots` method to discover available snapshots, and load binary 988 snapshot chunks via `LoadSnapshotChunk`. The application is free to choose how to implement this 989 and which formats to use, but must provide the following guarantees: 990 991 * **Consistent:** A snapshot must be taken at a single isolated height, unaffected by 992 concurrent writes. This can be accomplished by using a data store that supports ACID 993 transactions with snapshot isolation. 994 995 * **Asynchronous:** Taking a snapshot can be time-consuming, so it must not halt chain progress, 996 for example by running in a separate thread. 997 998 * **Deterministic:** A snapshot taken at the same height in the same format must be identical 999 (at the byte level) across nodes, including all metadata. This ensures good availability of 1000 chunks, and that they fit together across nodes. 1001 1002 A very basic approach might be to use a datastore with MVCC transactions (such as RocksDB), 1003 start a transaction immediately after block commit, and spawn a new thread which is passed the 1004 transaction handle. This thread can then export all data items, serialize them using e.g. 1005 Protobuf, hash the byte stream, split it into chunks, and store the chunks in the file system 1006 along with some metadata - all while the blockchain is applying new blocks in parallel. 1007 1008 A more advanced approach might include incremental verification of individual chunks against the 1009 chain app hash, parallel or batched exports, compression, and so on. 1010 1011 Old snapshots should be removed after some time - generally only the last two snapshots are needed 1012 (to prevent the last one from being removed while a node is restoring it). 1013 1014 #### Bootstrapping a Node 1015 1016 An empty node can be state synced by setting the configuration option `statesync.enabled = 1017 true`. The node also needs the chain genesis file for basic chain info, and configuration for 1018 light client verification of the restored snapshot: a set of CometBFT RPC servers, and a 1019 trusted header hash and corresponding height from a trusted source, via the `statesync` 1020 configuration section. 1021 1022 Once started, the node will connect to the P2P network and begin discovering snapshots. These 1023 will be offered to the local application via the `OfferSnapshot` ABCI method. Once a snapshot 1024 is accepted CometBFT will fetch and apply the snapshot chunks. After all chunks have been 1025 successfully applied, CometBFT verifies the app's `AppHash` against the chain using the light 1026 client, then switches the node to normal consensus operation. 1027 1028 ##### Snapshot Discovery 1029 1030 When the empty node joins the P2P network, it asks all peers to report snapshots via the 1031 `ListSnapshots` ABCI call (limited to 10 per node). After some time, the node picks the most 1032 suitable snapshot (generally prioritized by height, format, and number of peers), and offers it 1033 to the application via `OfferSnapshot`. The application can choose a number of responses, 1034 including accepting or rejecting it, rejecting the offered format, rejecting the peer who sent 1035 it, and so on. CometBFT will keep discovering and offering snapshots until one is accepted or 1036 the application aborts. 1037 1038 ##### Snapshot Restoration 1039 1040 Once a snapshot has been accepted via `OfferSnapshot`, CometBFT begins downloading chunks from 1041 any peers that have the same snapshot (i.e. that have identical metadata fields). Chunks are 1042 spooled in a temporary directory, and then given to the application in sequential order via 1043 `ApplySnapshotChunk` until all chunks have been accepted. 1044 1045 The method for restoring snapshot chunks is entirely up to the application. 1046 1047 During restoration, the application can respond to `ApplySnapshotChunk` with instructions for how 1048 to continue. This will typically be to accept the chunk and await the next one, but it can also 1049 ask for chunks to be refetched (either the current one or any number of previous ones), P2P peers 1050 to be banned, snapshots to be rejected or retried, and a number of other responses - see the ABCI 1051 reference for details. 1052 1053 If CometBFT fails to fetch a chunk after some time, it will reject the snapshot and try a 1054 different one via `OfferSnapshot` - the application can choose whether it wants to support 1055 restarting restoration, or simply abort with an error. 1056 1057 ##### Snapshot Verification 1058 1059 Once all chunks have been accepted, CometBFT issues an `Info` ABCI call to retrieve the 1060 `LastBlockAppHash`. This is compared with the trusted app hash from the chain, retrieved and 1061 verified using the light client. CometBFT also checks that `LastBlockHeight` corresponds to the 1062 height of the snapshot. 1063 1064 This verification ensures that an application is valid before joining the network. However, the 1065 snapshot restoration may take a long time to complete, so applications may want to employ additional 1066 verification during the restore to detect failures early. This might e.g. include incremental 1067 verification of each chunk against the app hash (using bundled Merkle proofs), checksums to 1068 protect against data corruption by the disk or network, and so on. However, it is important to 1069 note that the only trusted information available is the app hash, and all other snapshot metadata 1070 can be spoofed by adversaries. 1071 1072 Apps may also want to consider state sync denial-of-service vectors, where adversaries provide 1073 invalid or harmful snapshots to prevent nodes from joining the network. The application can 1074 counteract this by asking CometBFT to ban peers. As a last resort, node operators can use 1075 P2P configuration options to whitelist a set of trusted peers that can provide valid snapshots. 1076 1077 ##### Transition to Consensus 1078 1079 Once the snapshots have all been restored, CometBFT gathers additional information necessary for 1080 bootstrapping the node (e.g. chain ID, consensus parameters, validator sets, and block headers) 1081 from the genesis file and light client RPC servers. It also calls `Info` to verify the following: 1082 1083 * that the app hash from the snapshot it has delivered to the Application matches the apphash 1084 stored in the next height's block 1085 * that the version that the Application returns in `ResponseInfo` matches the version in the 1086 current height's block header 1087 1088 Once the state machine has been restored and CometBFT has gathered this additional 1089 information, it transitions to consensus. As of ABCI 2.0, CometBFT ensures the neccessary conditions 1090 to switch are met [RFC-100](./../../docs/rfc/rfc-100-abci-vote-extension-propag.md#base-implementation-persist-and-propagate-extended-commit-history). 1091 From the application's point of view, these operations are transparent, unless the application has just upgraded to ABCI 2.0. 1092 In that case, the application needs to be properly configured and aware of certain constraints in terms of when 1093 to provide vote extensions. More details can be found in the section below. 1094 1095 Once a node switches to consensus, it operates like any other node, apart from having a truncated block history at the height of the restored snapshot. 1096 1097 ## Application configuration required to switch to ABCI 2.0 1098 1099 Introducing vote extensions requires changes to the configuration of the application. 1100 1101 First of all, switching to a version of CometBFT with vote extensions, requires a coordinated upgrade. 1102 For a detailed description on the upgrade path, please refer to the corresponding 1103 [section](./../../docs/rfc/rfc-100-abci-vote-extension-propag.md#upgrade-path) in RFC-100. 1104 1105 There is a newly introduced [**consensus parameter**](./abci%2B%2B_app_requirements.md#abciparamsvoteextensionsenableheight): `VoteExtensionsEnableHeight`. 1106 This parameter represents the height at which vote extensions are 1107 required for consensus to proceed, with 0 being the default value (no vote extensions). 1108 A chain can enable vote extensions either: 1109 * at genesis by setting `VoteExtensionsEnableHeight` to be equal, e.g., to the `InitialHeight` 1110 * or via the application logic by changing the `ConsensusParam` to configure the 1111 `VoteExtensionsEnableHeight`. 1112 1113 Once the (coordinated) upgrade to ABCI 2.0 has taken place, at height *h<sub>u</sub>*, 1114 the value of `VoteExtensionsEnableHeight` MAY be set to some height, *h<sub>e</sub>*, 1115 which MUST be higher than the current height of the chain. Thus the earliest value for 1116 *h<sub>e</sub>* is *h<sub>u</sub>* + 1. 1117 1118 Once a node reaches the configured height, 1119 for all heights *h ≥ h<sub>e</sub>*, the consensus algorithm will 1120 reject as invalid any precommit messages that do not have signed vote extension data. 1121 If the application requires it, a 0-length vote extension is allowed, but it MUST be signed 1122 and present in the precommit message. 1123 Likewise, for all heights *h < h<sub>e</sub>*, any precommit messages that *do* have vote extensions 1124 will also be rejected as malformed. 1125 Height *h<sub>e</sub>* is somewhat special, as calls to `PrepareProposal` MUST NOT 1126 have vote extension data, but all precommit votes in that height MUST carry a vote extension, 1127 even if the extension is `nil`. 1128 Height *h<sub>e</sub> + 1* is the first height for which `PrepareProposal` MUST have vote 1129 extension data and all precommit votes in that height MUST have a vote extension. 1130 1131 Corollary, [CometBFT will decide](./abci%2B%2B_comet_expected_behavior.md#handling-upgrades-to-abci-20) which data to store, and require for successful operations, based on the current height 1132 of the chain.