github.com/badrootd/nibiru-cometbft@v0.37.5-0.20240307173500-2a75559eee9b/spec/abci/abci++_app_requirements.md

github.com/badrootd/nibiru-cometbft@v0.37.5-0.20240307173500-2a75559eee9b/spec/abci/abci++_app_requirements.md (about)

1 ---
2 order: 3
3 title: Requirements for the Application
4 ---
5
6 # Requirements for the Application
7
8 ## Formal Requirements
9
10 ### Consensus Connection Requirements
11
12 This section specifies what CometBFT expects from the Application. It is structured as a set
13 of formal requirements that can be used for testing and verification of the Application's logic.
14
15 Let *p* and *q* be two correct processes.
16 Let *rp* (resp. *rq*) be a round of height *h* where *p* (resp. *q*) is the
17 proposer.
18 Let *sp,h-1* be *p*'s Application's state committed for height *h-1*.
19 Let *vp* (resp. *vq*) be the block that *p*'s (resp. *q*'s) CometBFT passes
20 on to the Application
21 via `RequestPrepareProposal` as proposer of round *rp* (resp *rq*), height *h*,
22 also known as the raw proposal.
23 Let *up* (resp. *uq*) the possibly modified block *p*'s (resp. *q*'s) Application
24 returns via `ResponsePrepareProposal` to CometBFT, also known as the prepared proposal.
25
26 Process *p*'s prepared proposal can differ in two different rounds where *p* is the proposer.
27
28 * Requirement 1 [`PrepareProposal`, timeliness]: If *p*'s Application fully executes prepared blocks in
29 `PrepareProposal` and the network is in a synchronous period while processes *p* and *q* are in *rp*,
30 then the value of *TimeoutPropose* at *q* must be such that *q*'s propose timer does not time out
31 (which would result in *q* prevoting `nil` in *rp*).
32
33 Full execution of blocks at `PrepareProposal` time stands on CometBFT's critical path. Thus,
34 Requirement 1 ensures the Application or operator will set a value for `TimeoutPropose` such that the time it takes
35 to fully execute blocks in `PrepareProposal` does not interfere with CometBFT's propose timer.
36 Note that violation of Requirement 1 may just lead to further rounds, but will not compromise liveness.
37
38 * Requirement 2 [`PrepareProposal`, tx-size]: When *p*'s Application calls `ResponsePrepareProposal`, the
39 total size in bytes of the transactions returned does not exceed `RequestPrepareProposal.max_tx_bytes`.
40
41 Busy blockchains might seek to maximize the amount of transactions included in each block. Under those conditions,
42 CometBFT might choose to increase the transactions passed to the Application via `RequestPrepareProposal.txs`
43 beyond the `RequestPrepareProposal.max_tx_bytes` limit. The idea is that, if the Application drops some of
44 those transactions, it can still return a transaction list whose byte size is as close to
45 `RequestPrepareProposal.max_tx_bytes` as possible. Thus, Requirement 2 ensures that the size in bytes of the
46 transaction list returned by the application will never cause the resulting block to go beyond its byte size
47 limit.
48
49 * Requirement 3 [`PrepareProposal`, `ProcessProposal`, coherence]: For any two correct processes *p* and *q*,
50 if *q*'s CometBFT calls `RequestProcessProposal` on *up*,
51 *q*'s Application returns Accept in `ResponseProcessProposal`.
52
53 Requirement 3 makes sure that blocks proposed by correct processes *always* pass the correct receiving process's
54 `ProcessProposal` check.
55 On the other hand, if there is a deterministic bug in `PrepareProposal` or `ProcessProposal` (or in both),
56 strictly speaking, this makes all processes that hit the bug byzantine. This is a problem in practice,
57 as very often validators are running the Application from the same codebase, so potentially *all* would
58 likely hit the bug at the same time. This would result in most (or all) processes prevoting `nil`, with the
59 serious consequences on CometBFT's liveness that this entails. Due to its criticality, Requirement 3 is a
60 target for extensive testing and automated verification.
61
62 * Requirement 4 [`ProcessProposal`, determinism-1]: `ProcessProposal` is a (deterministic) function of the current
63 state and the block that is about to be applied. In other words, for any correct process *p*, and any arbitrary block *u*,
64 if *p*'s CometBFT calls `RequestProcessProposal` on *u* at height *h*,
65 then *p*'s Application's acceptance or rejection **exclusively** depends on *u* and *sp,h-1*.
66
67 * Requirement 5 [`ProcessProposal`, determinism-2]: For any two correct processes *p* and *q*, and any arbitrary
68 block *u*,
69 if *p*'s (resp. *q*'s) CometBFT calls `RequestProcessProposal` on *u* at height *h*,
70 then *p*'s Application accepts *u* if and only if *q*'s Application accepts *u*.
71 Note that this requirement follows from Requirement 4 and the Agreement property of consensus.
72
73 Requirements 4 and 5 ensure that all correct processes will react in the same way to a proposed block, even
74 if the proposer is Byzantine. However, `ProcessProposal` may contain a bug that renders the
75 acceptance or rejection of the block non-deterministic, and therefore prevents processes hitting
76 the bug from fulfilling Requirements 4 or 5 (effectively making those processes Byzantine).
77 In such a scenario, CometBFT's liveness cannot be guaranteed.
78 Again, this is a problem in practice if most validators are running the same software, as they are likely
79 to hit the bug at the same point. There is currently no clear solution to help with this situation, so
80 the Application designers/implementors must proceed very carefully with the logic/implementation
81 of `ProcessProposal`. As a general rule `ProcessProposal` SHOULD always accept the block.
82
83 According to the Tendermint consensus algorithm, currently adopted in CometBFT,
84 a correct process can broadcast at most one precommit
85 message in round *r*, height *h*.
86
87 
97
98 
133 * Requirement 9 [*all*, no-side-effects]: *p*'s calls to `RequestPrepareProposal`,
134 `RequestProcessProposal`,
135 
139 at height *h* do
140 not modify *sp,h-1*.
141
142 
147 The call to correct process *p*'s `BeginBlock - DeliverTx - EndBlock` sequence at height *h*, with block *vp,h*
148 passed as parameter, creates state *sp,h*.
149 Additionally, *p*'s `DeliverTx` on transactions creates a set of transaction results *Tp,h*.
150
151 * Requirement 11 [`BeginBlock - DeliverTx - EndBlock`, determinism-1]: For any correct process *p*,
152 *sp,h* exclusively depends on *sp,h-1* and *vp,h*.
153
154 * Requirement 12 [`BeginBlock - DeliverTx - EndBlock`, determinism-2]: For any correct process *p*,
155 the contents of *Tp,h* exclusively depend on *sp,h-1* and *vp,h*.
156
157 Note that Requirements 11 and 12, combined with the Agreement property of consensus ensure
158 state machine replication, i.e., the Application state evolves consistently at all correct processes.
159
160 Also, notice that `PrepareProposal`  has determinism-related
161 requirements associated.
162 Indeed, `PrepareProposal` is not required to be deterministic:
163
164 * *up* may depend on *vp* and *sp,h-1*, but may also depend on other values or operations.
165 * *vp = vq &#8655; up = uq*.
166
167 
175
176 ### Mempool Connection Requirements
177
178 Let *CheckTxCodestx,p,h* denote the set of result codes returned by *p*'s Application,
179 via `ResponseCheckTx`,
180 to successive calls to `RequestCheckTx` occurring while the Application is at height *h*
181 and having transaction *tx* as parameter.
182 *CheckTxCodestx,p,h* is a set since *p*'s Application may
183 return different result codes during height *h*.
184 If *CheckTxCodestx,p,h* is a singleton set, i.e. the Application always returned
185 the same result code in `ResponseCheckTx` while at height *h*,
186 we define *CheckTxCodetx,p,h* as the singleton value of *CheckTxCodestx,p,h*.
187 If *CheckTxCodestx,p,h* is not a singleton set, *CheckTxCodetx,p,h* is undefined.
188 Let predicate *OK(CheckTxCodetx,p,h)* denote whether *CheckTxCodetx,p,h* is `SUCCESS`.
189
190 * Requirement 13 [`CheckTx`, eventual non-oscillation]: For any transaction *tx*,
191 there exists a boolean value *b*,
192 and a height *hstable* such that,
193 for any correct process *p*,
194 *CheckTxCodetx,p,h* is defined, and
195 *OK(CheckTxCodetx,p,h) = b*
196 for any height *h &#8805; hstable*.
197
198 Requirement 13 ensures that
199 a transaction will eventually stop oscillating between `CheckTx` success and failure
200 if it stays in *p's* mempool for long enough.
201 This condition on the Application's behavior allows the mempool to ensure that
202 a transaction will leave the mempool of all full nodes,
203 either because it is expunged everywhere due to failing `CheckTx` calls,
204 or because it stays valid long enough to be gossipped, proposed and decided.
205 Although Requirement 13 defines a global *hstable*, application developers
206 can consider such stabilization height as local to process *p* (*hp,stable*),
207 without loss for generality.
208 In contrast, the value of *b* MUST be the same across all processes.
209
210 ## Managing the Application state and related topics
211
212 ### Connection State
213
214 CometBFT maintains four concurrent ABCI++ connections, namely
215 [Consensus Connection](#consensus-connection),
216 [Mempool Connection](#mempool-connection),
217 [Info/Query Connection](#infoquery-connection), and
218 [Snapshot Connection](#snapshot-connection).
219 It is common for an application to maintain a distinct copy of
220 the state for each connection, which are synchronized upon `Commit` calls.
221
222 #### Concurrency
223
224 In principle, each of the four ABCI++ connections operates concurrently with one
225 another. This means applications need to ensure access to state is
226 thread safe. Both the
227 [default in-process ABCI client](https://github.com/cometbft/cometbft/blob/v0.37.x/abci/client/local_client.go#L13)
228 and the
229 [default Go ABCI server](https://github.com/cometbft/cometbft/blob/v0.37.x/abci/server/socket_server.go#L20)
230 use a global lock to guard the handling of events across all connections, so they are not
231 concurrent at all. This means whether your app is compiled in-process with
232 CometBFT using the `NewLocalClient`, or run out-of-process using the `SocketServer`,
233 ABCI messages from all connections are received in sequence, one at a
234 time.
235
236 The existence of this global mutex means Go application developers can get thread safety for application state by routing all reads and writes through the ABCI system. Thus it may be unsafe to expose application state directly to an RPC interface, and unless explicit measures are taken, all queries should be routed through the ABCI Query method.
237
238 
243
244 
247 
251
252 #### BeginBlock - DeliverTx - EndBlock
253
254 When the consensus algorithm decides on a block, CometBFT uses the sequence of calls to
255 `BeginBlock`, `DeliverTx` and `EndBlock` to send the
256 decided block's data to the Application, which uses it to transition its state.
257
258 The sequence of `DeliverTx` calls is asynchronous but all those calls are enclosed by calls to `BeginBlock` and `EndBlock` which are synchronous.
259
260 The Application must remember the latest height from which it
261 has run a successful `Commit` so that it can tell CometBFT where to
262 pick up from when it recovers from a crash. See information on the Handshake
263 [here](#crash-recovery).
264
265 #### Commit
266
267 The Application should persist its state during `Commit`, before returning from it.
268
269 Before invoking `Commit`, CometBFT locks the mempool and flushes the mempool connection. This ensures that
270 no new messages
271 will be received on the mempool connection during this processing step, providing an opportunity to safely
272 update all four
273 connection states to the latest committed state at the same time.
274
275 When `Commit` returns, CometBFT unlocks the mempool.
276
277 WARNING: if the ABCI app logic processing the `Commit` message sends a
278 `/broadcast_tx_sync` or `/broadcast_tx` and waits for the response
279 before proceeding, it will deadlock. Executing `broadcast_tx` calls
280 involves acquiring the mempool lock that CometBFT holds during the `Commit` call.
281 Synchronous mempool-related calls must be avoided as part of the sequential logic of the
282 `Commit` function.
283
284 #### Candidate States
285
286 CometBFT calls `PrepareProposal` when it is about to send a proposed block to the network.
287 Likewise, CometBFT calls `ProcessProposal` upon reception of a proposed block from the
288 network. The proposed block's data
289 that is disclosed to the Application by these two methods is the following:
290
291 * the transaction list
292 * the `LastCommit` referring to the previous block
293 * the block header's hash (except in `PrepareProposal`, where it is not known yet)
294 * list of validators that misbehaved
295 * the block's timestamp
296 * `NextValidatorsHash`
297 * Proposer address
298
299 The Application may decide to *immediately* execute the given block (i.e., upon `PrepareProposal`
300 or `ProcessProposal`). There are two main reasons why the Application may want to do this:
301
302 * *Avoiding invalid transactions in blocks*.
303 In order to be sure that the block does not contain *any* invalid transaction, there may be
304 no way other than fully executing the transactions in the block as though it was the *decided*
305 block.
306 * *Quick `BeginBlock-DeliverTx-EndBlock` execution*.
307 Upon reception of the decided block via `BeginBlock`, if that same block was executed
308 upon `PrepareProposal` or `ProcessProposal` and the resulting state was kept in memory, the
309 Application can simply apply that state (faster) to the main state, rather than reexecuting
310 the decided block (slower).
311
312 `PrepareProposal`/`ProcessProposal` can be called many times for a given height. Moreover,
313 it is not possible to accurately predict which of the blocks proposed in a height will be decided,
314 being delivered to the Application in that height's block execution calls.
315 Therefore, the state resulting from executing a proposed block, denoted a *candidate state*, should
316 be kept in memory as a possible final state for that height. When the block execution functions are called, the Application should
317 check if the decided block corresponds to one of its candidate states; if so, it will apply it as
318 its *ExecuteTxState* (see [Consensus Connection](#consensus-connection) below),
319 which will be persisted during the upcoming `Commit` call.
320
321 Under adverse conditions (e.g., network instability), the consensus algorithm might take many rounds.
322 In this case, potentially many proposed blocks will be disclosed to the Application for a given height.
323 By the nature of Tendermint consensus algorithm, currently adopted in CometBFT, the number of proposed blocks received by the Application
324 for a particular height cannot be bound, so Application developers must act with care and use mechanisms
325 to bound memory usage. As a general rule, the Application should be ready to discard candidate states
326 before block execution, even if one of them might end up corresponding to the
327 decided block and thus have to be reexecuted upon `BeginBlock-DeliverTx-EndBlock`.
328
329 ### States and ABCI++ Connections
330
331 #### Consensus Connection
332
333 The Consensus Connection should maintain an *ExecuteTxState* &mdash; the working state
334 for block execution. It should be updated by the calls to the block execution functions
335 during block execution and committed to disk as the "latest
336 committed state" during `Commit`. Execution of a proposed block (via `PrepareProposal`/`ProcessProposal`)
337 **must not** update the *ExecuteTxState*, but rather be kept as a separate candidate state until `BeginBlock-DeliverTx-EndBlock`
338 confirms which of the candidate states (if any) can be used to update *ExecuteTxState*.
339
340 #### Mempool Connection
341
342 The mempool Connection maintains *CheckTxState*. CometBFT sequentially processes an incoming
343 transaction (via RPC from client or P2P from the gossip layer) against *CheckTxState*.
344 If the processing does not return any error, the transaction is accepted into the mempool
345 and CometBFT starts gossipping it.
346 *CheckTxState* should be reset to the latest committed state
347 at the end of every `Commit`.
348
349 During the execution of a consensus instance, the *CheckTxState* may be updated concurrently with the
350 *ExecuteTxState*, as messages may be sent concurrently on the Consensus and Mempool connections.
351 At the end of the consensus instance, as described above, CometBFT locks the mempool and flushes
352 the mempool connection before calling `Commit`. This ensures that all pending `CheckTx` calls are
353 responded to and no new ones can begin.
354
355 After the `Commit` call returns, while still holding the mempool lock, `CheckTx` is run again on all
356 transactions that remain in the node's local mempool after filtering those included in the block.
357 Parameter `Type` in `RequestCheckTx`
358 indicates whether an incoming transaction is new (`CheckTxType_New`), or a
359 recheck (`CheckTxType_Recheck`).
360
361 Finally, after re-checking transactions in the mempool, CometBFT will unlock
362 the mempool connection. New transactions are once again able to be processed through `CheckTx`.
363
364 Note that `CheckTx` is just a weak filter to keep invalid transactions out of the mempool and,
365 utimately, ouf of the blockchain.
366 Since the transaction cannot be guaranteed to be checked against the exact same state as it
367 will be executed as part of a (potential) decided block, `CheckTx` shouldn't check *everything*
368 that affects the transaction's validity, in particular those checks whose validity may depend on
369 transaction ordering. `CheckTx` is weak because a Byzantine node need not care about `CheckTx`;
370 it can propose a block full of invalid transactions if it wants. The mechanism ABCI++ has
371 in place for dealing with such behavior is `ProcessProposal`.
372
373 ##### Replay Protection
374
375 It is possible for old transactions to be sent again to the Application. This is typically
376 undesirable for all transactions, except for a generally small subset of them which are idempotent.
377
378 The mempool has a mechanism to prevent duplicated transactions from being processed.
379 This mechanism is nevertheless best-effort (currently based on the indexer)
380 and does not provide any guarantee of non duplication.
381 It is thus up to the Application to implement an application-specific
382 replay protection mechanism with strong guarantees as part of the logic in `CheckTx`.
383
384 #### Info/Query Connection
385
386 The Info (or Query) Connection should maintain a `QueryState`. This connection has two
387 purposes: 1) having the application answer the queries CometBFT receives from users
388 (see section [Query](#query)),
389 and 2) synchronizing CometBFT and the Application at start up time (see
390 [Crash Recovery](#crash-recovery))
391 or after state sync (see [State Sync](#state-sync)).
392
393 `QueryState` is a read-only copy of *ExecuteTxState* as it was after the last
394 `Commit`, i.e.
395 after the full block has been processed and the state committed to disk.
396
397 #### Snapshot Connection
398
399 The Snapshot Connection is used to serve state sync snapshots for other nodes
400 and/or restore state sync snapshots to a local node being bootstrapped.
401 Snapshop management is optional: an Application may choose not to implement it.
402
403 For more information, see Section [State Sync](#state-sync).
404
405 ### Transaction Results
406
407 For each transaction within a block, the Application is expected to return a result within
408 [`ResponseDeliverTx`](./abci%2B%2B_methods.md#delivertx).
409 
413 This section discusses the fields inside `ResponseDeliverTx` along with the fields in
414 [`ResponseCheckTx`](./abci%2B%2B_methods.md#checktx),
415 whose semantics are similar.
416
417 The `Info` and `Log` fields are
418 non-deterministic values for debugging/convenience purposes. CometBFT logs them but they
419 are otherwise ignored.
420
421 #### Gas
422
423 Ethereum introduced the notion of *gas* as an abstract representation of the
424 cost of the resources consumed by nodes when processing a transaction. Every operation in the
425 Ethereum Virtual Machine uses some amount of gas.
426 Gas has a market-variable price based on which miners can accept or reject to execute a
427 particular operation.
428
429 Users propose a maximum amount of gas for their transaction; if the transaction uses less, they get
430 the difference credited back. CometBFT adopts a similar abstraction,
431 though uses it only optionally and weakly, allowing applications to define
432 their own sense of the cost of execution.
433
434 In CometBFT, the [ConsensusParams.Block.MaxGas](#consensus-parameters) limits the amount of
435 total gas that can be used by all transactions in a block.
436 The default value is `-1`, which means the block gas limit is not enforced, or that the concept of
437 gas is meaningless.
438
439 Responses contain a `GasWanted` and `GasUsed` field. The former is the maximum
440 amount of gas the sender of a transaction is willing to use, and the latter is how much it actually
441 used. Applications should enforce that `GasUsed <= GasWanted` &mdash; i.e. transaction execution
442 or validation should fail before it can use more resources than it requested.
443
444 When `MaxGas > -1`, CometBFT enforces the following rules:
445
446 * `GasWanted <= MaxGas` for every transaction in the mempool
447 * `(sum of GasWanted in a block) <= MaxGas` when proposing a block
448
449 If `MaxGas == -1`, no rules about gas are enforced.
450
451 In v0.34.x and earlier versions, CometBFT does not enforce anything about Gas in consensus,
452 only in the mempool.
453 This means it does not guarantee that committed blocks satisfy these rules.
454 It is the application's responsibility to return non-zero response codes when gas limits are exceeded
455 when executing the transactions of a block.
456 Since the introduction of `PrepareProposal` and `ProcessProposal` in v.0.37.x, it is now possible
457 for the Application to enforce that all blocks proposed (and voted for) in consensus &mdash; and thus all
458 blocks decided &mdash; respect the `MaxGas` limits described above.
459
460 Since the Application should enforce that `GasUsed <= GasWanted` when executing a transaction, and
461 it can use `PrepareProposal` and `ProcessProposal` to enforce that `(sum of GasWanted in a block) <= MaxGas`
462 in all proposed or prevoted blocks,
463 we have:
464
465 * `(sum of GasUsed in a block) <= MaxGas` for every block
466
467 The `GasUsed` field is ignored by CometBFT.
468
469 #### Specifics of `ResponseCheckTx`
470
471 If `Code != 0`, it will be rejected from the mempool and hence
472 not broadcasted to other peers and not included in a proposal block.
473
474 `Data` contains the result of the `CheckTx` transaction execution, if any. It does not need to be
475 deterministic since, given a transaction, nodes' Applications
476 might have a different *CheckTxState* values when they receive it and check their validity
477 via `CheckTx`.
478 CometBFT ignores this value in `ResponseCheckTx`.
479
480 From v0.34.x on, there is a `Priority` field in `ResponseCheckTx` that can be
481 used to explicitly prioritize transactions in the mempool for inclusion in a block
482 proposal.
483
484 #### Specifics of `ResponseDeliverTx`
485
486 The `BeginBlock-DeliverTx-EndBlock` sequence is the workhorse of the blockchain.
487 A sequence of `DeliverTx` calls delivers the decided block,
488 one transaction at a time, to the Application.
489 The block delivered (and thus the transaction order) is the same at all correct nodes as guaranteed
490 by the Agreement property of consensus.
491
492 The `Data` field contains an array of bytes with the transaction result.
493 It must be deterministic (i.e., the same value must be returned at all nodes), but it can contain arbitrary
494 data. Likewise, the value of `Code` must be deterministic.
495 If `Code != 0`, the transaction will be marked invalid,
496 though it is still included in the block. Invalid transactions are not indexed, as they are
497 considered analogous to those that failed `CheckTx`.
498
499 Both the `Code` and `Data` are included in a structure that is hashed into the
500 `LastResultsHash` of the block header in the next height.
501
502 `Events` include any events for the execution, which CometBFT will use to index
503 the transaction by. This allows transactions to be queried according to what
504 events took place during their execution.
505
506 ### Updating the Validator Set
507
508 The application may set the validator set during
509 [`InitChain`](./abci%2B%2B_methods.md#initchain), and may update it during
510 [`EndBlock`](./abci%2B%2B_methods.md#endblock). In both cases, a structure of type
511 [`ValidatorUpdate`](./abci%2B%2B_methods.md#validatorupdate) is returned.
512
513 The `InitChain` method, used to initialize the Application, can return a list of validators.
514 If the list is empty, CometBFT will use the validators loaded from the genesis
515 file.
516 If the list returned by `InitChain` is not empty, CometBFT will use its contents as the validator set.
517 This way the application can set the initial validator set for the
518 blockchain.
519
520 Applications must ensure that a single set of validator updates does not contain duplicates, i.e.
521 a given public key can only appear once within a given update. If an update includes
522 duplicates, the block execution will fail irrecoverably.
523
524 Structure `ValidatorUpdate` contains a public key, which is used to identify the validator:
525 The public key currently supports three types:
526
527 * `ed25519`
528 * `secp256k1`
529 * `sr25519`
530
531 Structure `ValidatorUpdate` also contains an `ìnt64` field denoting the validator's new power.
532 Applications must ensure that
533 `ValidatorUpdate` structures abide by the following rules:
534
535 * power must be non-negative
536 * if power is set to 0, the validator must be in the validator set; it will be removed from the set
537 * if power is greater than 0:
538 * if the validator is not in the validator set, it will be added to the
539 set with the given power
540 * if the validator is in the validator set, its power will be adjusted to the given power
541 * the total power of the new validator set must not exceed `MaxTotalVotingPower`, where
542 `MaxTotalVotingPower = MaxInt64 / 8`
543
544 Note the updates returned after processing the block at height `H` will only take effect
545 at block `H+2` (see Section [Methods](./abci%2B%2B_methods.md)).
546
547 ### Consensus Parameters
548
549 `ConsensusParams` are global parameters that apply to all validators in a blockchain.
550 They enforce certain limits in the blockchain, like the maximum size
551 of blocks, amount of gas used in a block, and the maximum acceptable age of
552 evidence. They can be set in
553 [`InitChain`](./abci%2B%2B_methods.md#initchain), and updated in
554 [`EndBlock`](./abci%2B%2B_methods.md#endblock).
555 These parameters are deterministically set and/or updated by the Application, so
556 all full nodes have the same value at a given height.
557
558 #### List of Parameters
559
560 These are the current consensus parameters (as of v0.37.x):
561
562 1. [BlockParams.MaxBytes](#blockparamsmaxbytes)
563 2. [BlockParams.MaxGas](#blockparamsmaxgas)
564 3. [EvidenceParams.MaxAgeDuration](#evidenceparamsmaxageduration)
565 4. [EvidenceParams.MaxAgeNumBlocks](#evidenceparamsmaxagenumblocks)
566 5. [EvidenceParams.MaxBytes](#evidenceparamsmaxbytes)
567 6. [ValidatorParams.PubKeyTypes](#validatorparamspubkeytypes)
568 7. [VersionParams.App](#versionparamsapp)
569 
579
580 ##### BlockParams.MaxBytes
581
582 The maximum size of a complete Protobuf encoded block.
583 This is enforced by the consensus algorithm.
584
585 This implies a maximum transaction size that is `MaxBytes`, less the expected size of
586 the header, the validator set, and any included evidence in the block.
587
588 The Application should be aware that honest validators _may_ produce and
589 broadcast blocks with up to the configured `MaxBytes` size.
590 As a result, the consensus
591 [timeout parameters](../../docs/core/configuration.md#consensus-timeouts-explained)
592 adopted by nodes should be configured so as to account for the worst-case
593 latency for the delivery of a full block with `MaxBytes` size to all validators.
594
595 Must have `0 < MaxBytes <= 100 MB`.
596
597 > Bear in mind that the default value for the `BlockParams.MaxBytes` consensus
598 > parameter accepts as valid blocks with size up to 21 MB.
599 > If the Application's use case does not need blocks of that size,
600 > or if the impact (specially on bandwidth consumption and block latency)
601 > of propagating blocks of that size was not evaluated,
602 > it is strongly recommended to wind down this default value.
603
604 ##### BlockParams.MaxGas
605
606 The maximum of the sum of `GasWanted` that will be allowed in a proposed block.
607 This is *not* enforced by the consensus algorithm.
608 It is left to the Application to enforce (ie. if transactions are included past the
609 limit, they should return non-zero codes). It is used by CometBFT to limit the
610 transactions included in a proposed block.
611
612 Must have `MaxGas >= -1`.
613 If `MaxGas == -1`, no limit is enforced.
614
615 ##### EvidenceParams.MaxAgeDuration
616
617 This is the maximum age of evidence in time units.
618 This is enforced by the consensus algorithm.
619
620 If a block includes evidence older than this (AND the evidence was created more
621 than `MaxAgeNumBlocks` ago), the block will be rejected (validators won't vote
622 for it).
623
624 Must have `MaxAgeDuration > 0`.
625
626 ##### EvidenceParams.MaxAgeNumBlocks
627
628 This is the maximum age of evidence in blocks.
629 This is enforced by the consensus algorithm.
630
631 If a block includes evidence older than this (AND the evidence was created more
632 than `MaxAgeDuration` ago), the block will be rejected (validators won't vote
633 for it).
634
635 Must have `MaxAgeNumBlocks > 0`.
636
637 ##### EvidenceParams.MaxBytes
638
639 This is the maximum size of total evidence in bytes that can be committed to a
640 single block. It should fall comfortably under the max block bytes.
641
642 Its value must not exceed the size of
643 a block minus its overhead ( ~ `BlockParams.MaxBytes`).
644
645 Must have `MaxBytes > 0`.
646
647 ##### ValidatorParams.PubKeyTypes
648
649 The parameter restricts the type of keys validators can use. The parameter uses ABCI pubkey naming, not Amino names.
650
651 ##### VersionParams.App
652
653 This is the version of the ABCI application.
654 
732 
752 #### Updating Consensus Parameters
753
754 The application may set the `ConsensusParams` during
755 [`InitChain`](./abci%2B%2B_methods.md#initchain),
756 and update them during
757 [`EndBlock`](./abci%2B%2B_methods.md#endblock).
758 If the `ConsensusParams` is empty, it will be ignored. Each field
759 that is not empty will be applied in full. For instance, if updating the
760 `Block.MaxBytes`, applications must also set the other `Block` fields (like
761 `Block.MaxGas`), even if they are unchanged, as they will otherwise cause the
762 value to be updated to the default.
763
764 ##### `InitChain`
765
766 `ResponseInitChain` includes a `ConsensusParams` parameter.
767 If `ConsensusParams` is `nil`, CometBFT will use the params loaded in the genesis
768 file. If `ConsensusParams` is not `nil`, CometBFT will use it.
769 This way the application can determine the initial consensus parameters for the
770 blockchain.
771
772 ##### `EndBlock`, `PrepareProposal`/`ProcessProposal`
773
774 `ResponseEndBlock` accepts a `ConsensusParams` parameter.
775 If `ConsensusParams` is `nil`, CometBFT will do nothing.
776 If `ConsensusParams` is not `nil`, CometBFT will use it.
777 This way the application can update the consensus parameters over time.
778
779 The updates returned in block `H` will take effect right away for block
780 `H+1`.
781
782 ### `Query`
783
784 `Query` is a generic method with lots of flexibility to enable diverse sets
785 of queries on application state. CometBFT makes use of `Query` to filter new peers
786 based on ID and IP, and exposes `Query` to the user over RPC.
787
788 Note that calls to `Query` are not replicated across nodes, but rather query the
789 local node's state - hence they may return stale reads. For reads that require
790 consensus, use a transaction.
791
792 The most important use of `Query` is to return Merkle proofs of the application state at some height
793 that can be used for efficient application-specific light-clients.
794
795 Note CometBFT has technically no requirements from the `Query`
796 message for normal operation - that is, the ABCI app developer need not implement
797 Query functionality if they do not wish to.
798
799 #### Query Proofs
800
801 The CometBFT block header includes a number of hashes, each providing an
802 anchor for some type of proof about the blockchain. The `ValidatorsHash` enables
803 quick verification of the validator set, the `DataHash` gives quick
804 verification of the transactions included in the block.
805
806 The `AppHash` is unique in that it is application specific, and allows for
807 application-specific Merkle proofs about the state of the application.
808 While some applications keep all relevant state in the transactions themselves
809 (like Bitcoin and its UTXOs), others maintain a separated state that is
810 computed deterministically *from* transactions, but is not contained directly in
811 the transactions themselves (like Ethereum contracts and accounts).
812 For such applications, the `AppHash` provides a much more efficient way to verify light-client proofs.
813
814 ABCI applications can take advantage of more efficient light-client proofs for
815 their state as follows:
816
817 * return the Merkle root of the deterministic application state in
818 `Commit.Data`. This Merkle root will be included as the `AppHash` in the next block.
819 * return efficient Merkle proofs about that application state in `ResponseQuery.Proof`
820 that can be verified using the `AppHash` of the corresponding block.
821
822 For instance, this allows an application's light-client to verify proofs of
823 absence in the application state, something which is much less efficient to do using the block hash.
824
825 Some applications (eg. Ethereum, Cosmos-SDK) have multiple "levels" of Merkle trees,
826 where the leaves of one tree are the root hashes of others. To support this, and
827 the general variability in Merkle proofs, the `ResponseQuery.Proof` has some minimal structure:
828
829 ```protobuf
830 message ProofOps {
831 repeated ProofOp ops = 1
832 }
833
834 message ProofOp {
835 string type = 1;
836 bytes key = 2;
837 bytes data = 3;
838 }
839 ```
840
841 Each `ProofOp` contains a proof for a single key in a single Merkle tree, of the specified `type`.
842 This allows ABCI to support many different kinds of Merkle trees, encoding
843 formats, and proofs (eg. of presence and absence) just by varying the `type`.
844 The `data` contains the actual encoded proof, encoded according to the `type`.
845 When verifying the full proof, the root hash for one ProofOp is the value being
846 verified for the next ProofOp in the list. The root hash of the final ProofOp in
847 the list should match the `AppHash` being verified against.
848
849 #### Peer Filtering
850
851 When CometBFT connects to a peer, it sends two queries to the ABCI application
852 using the following paths, with no additional data:
853
854 * `/p2p/filter/addr/<IP:PORT>`, where `<IP:PORT>` denote the IP address and
855 the port of the connection
856 * `p2p/filter/id/<ID>`, where `<ID>` is the peer node ID (ie. the
857 pubkey.Address() for the peer's PubKey)
858
859 If either of these queries return a non-zero ABCI code, CometBFT will refuse
860 to connect to the peer.
861
862 #### Paths
863
864 Queries are directed at paths, and may optionally include additional data.
865
866 The expectation is for there to be some number of high level paths
867 differentiating concerns, like `/p2p`, `/store`, and `/app`. Currently,
868 CometBFT only uses `/p2p`, for filtering peers. For more advanced use, see the
869 implementation of
870 [Query in the Cosmos-SDK](https://github.com/cosmos/cosmos-sdk/blob/v0.23.1/baseapp/baseapp.go#L333).
871
872 ### Crash Recovery
873
874 On startup, CometBFT calls the `Info` method on the Info Connection to get the latest
875 committed state of the app. The app MUST return information consistent with the
876 last block it succesfully completed Commit for.
877
878 If the app succesfully committed block H, then `last_block_height = H` and `last_block_app_hash = <hash returned by Commit for block H>`. If the app
879 failed during the Commit of block H, then `last_block_height = H-1` and
880 `last_block_app_hash = <hash returned by Commit for block H-1, which is the hash in the header of block H>`.
881
882 We now distinguish three heights, and describe how CometBFT syncs itself with
883 the app.
884
885 ```md
886 storeBlockHeight = height of the last block CometBFT saw a commit for
887 stateBlockHeight = height of the last block for which CometBFT completed all
888 block processing and saved all ABCI results to disk
889 appBlockHeight = height of the last block for which ABCI app succesfully
890 completed Commit
891
892 ```
893
894 Note we always have `storeBlockHeight >= stateBlockHeight` and `storeBlockHeight >= appBlockHeight`
895 Note also CometBFT never calls Commit on an ABCI app twice for the same height.
896
897 The procedure is as follows.
898
899 First, some simple start conditions:
900
901 If `appBlockHeight == 0`, then call InitChain.
902
903 If `storeBlockHeight == 0`, we're done.
904
905 Now, some sanity checks:
906
907 If `storeBlockHeight < appBlockHeight`, error
908 If `storeBlockHeight < stateBlockHeight`, panic
909 If `storeBlockHeight > stateBlockHeight+1`, panic
910
911 Now, the meat:
912
913 If `storeBlockHeight == stateBlockHeight && appBlockHeight < storeBlockHeight`,
914 replay all blocks in full from `appBlockHeight` to `storeBlockHeight`.
915 This happens if we completed processing the block, but the app forgot its height.
916
917 If `storeBlockHeight == stateBlockHeight && appBlockHeight == storeBlockHeight`, we're done.
918 This happens if we crashed at an opportune spot.
919
920 If `storeBlockHeight == stateBlockHeight+1`
921 This happens if we started processing the block but didn't finish.
922
923 If `appBlockHeight < stateBlockHeight`
924 replay all blocks in full from `appBlockHeight` to `storeBlockHeight-1`,
925 and replay the block at `storeBlockHeight` using the WAL.
926 This happens if the app forgot the last block it committed.
927
928 If `appBlockHeight == stateBlockHeight`,
929 replay the last block (storeBlockHeight) in full.
930 This happens if we crashed before the app finished Commit
931
932 If `appBlockHeight == storeBlockHeight`
933 update the state using the saved ABCI responses but dont run the block against the real app.
934 This happens if we crashed after the app finished Commit but before CometBFT saved the state.
935
936 ### State Sync
937
938 A new node joining the network can simply join consensus at the genesis height and replay all
939 historical blocks until it is caught up. However, for large chains this can take a significant
940 amount of time, often on the order of days or weeks.
941
942 State sync is an alternative mechanism for bootstrapping a new node, where it fetches a snapshot
943 of the state machine at a given height and restores it. Depending on the application, this can
944 be several orders of magnitude faster than replaying blocks.
945
946 Note that state sync does not currently backfill historical blocks, so the node will have a
947 truncated block history - users are advised to consider the broader network implications of this in
948 terms of block availability and auditability. This functionality may be added in the future.
949
950 For details on the specific ABCI calls and types, see the
951 [methods](abci%2B%2B_methods.md) section.
952
953 #### Taking Snapshots
954
955 Applications that want to support state syncing must take state snapshots at regular intervals. How
956 this is accomplished is entirely up to the application. A snapshot consists of some metadata and
957 a set of binary chunks in an arbitrary format:
958
959 * `Height (uint64)`: The height at which the snapshot is taken. It must be taken after the given
960 height has been committed, and must not contain data from any later heights.
961
962 * `Format (uint32)`: An arbitrary snapshot format identifier. This can be used to version snapshot
963 formats, e.g. to switch from Protobuf to MessagePack for serialization. The application can use
964 this when restoring to choose whether to accept or reject a snapshot.
965
966 * `Chunks (uint32)`: The number of chunks in the snapshot. Each chunk contains arbitrary binary
967 data, and should be less than 16 MB; 10 MB is a good starting point.
968
969 * `Hash ([]byte)`: An arbitrary hash of the snapshot. This is used to check whether a snapshot is
970 the same across nodes when downloading chunks.
971
972 * `Metadata ([]byte)`: Arbitrary snapshot metadata, e.g. chunk hashes for verification or any other
973 necessary info.
974
975 For a snapshot to be considered the same across nodes, all of these fields must be identical. When
976 sent across the network, snapshot metadata messages are limited to 4 MB.
977
978 When a new node is running state sync and discovering snapshots, CometBFT will query an existing
979 application via the ABCI `ListSnapshots` method to discover available snapshots, and load binary
980 snapshot chunks via `LoadSnapshotChunk`. The application is free to choose how to implement this
981 and which formats to use, but must provide the following guarantees:
982
983 * **Consistent:** A snapshot must be taken at a single isolated height, unaffected by
984 concurrent writes. This can be accomplished by using a data store that supports ACID
985 transactions with snapshot isolation.
986
987 * **Asynchronous:** Taking a snapshot can be time-consuming, so it must not halt chain progress,
988 for example by running in a separate thread.
989
990 * **Deterministic:** A snapshot taken at the same height in the same format must be identical
991 (at the byte level) across nodes, including all metadata. This ensures good availability of
992 chunks, and that they fit together across nodes.
993
994 A very basic approach might be to use a datastore with MVCC transactions (such as RocksDB),
995 start a transaction immediately after block commit, and spawn a new thread which is passed the
996 transaction handle. This thread can then export all data items, serialize them using e.g.
997 Protobuf, hash the byte stream, split it into chunks, and store the chunks in the file system
998 along with some metadata - all while the blockchain is applying new blocks in parallel.
999
1000 A more advanced approach might include incremental verification of individual chunks against the
1001 chain app hash, parallel or batched exports, compression, and so on.
1002
1003 Old snapshots should be removed after some time - generally only the last two snapshots are needed
1004 (to prevent the last one from being removed while a node is restoring it).
1005
1006 #### Bootstrapping a Node
1007
1008 An empty node can be state synced by setting the configuration option `statesync.enabled =
1009 true`. The node also needs the chain genesis file for basic chain info, and configuration for
1010 light client verification of the restored snapshot: a set of CometBFT RPC servers, and a
1011 trusted header hash and corresponding height from a trusted source, via the `statesync`
1012 configuration section.
1013
1014 Once started, the node will connect to the P2P network and begin discovering snapshots. These
1015 will be offered to the local application via the `OfferSnapshot` ABCI method. Once a snapshot
1016 is accepted CometBFT will fetch and apply the snapshot chunks. After all chunks have been
1017 successfully applied, CometBFT verifies the app's `AppHash` against the chain using the light
1018 client, then switches the node to normal consensus operation.
1019
1020 ##### Snapshot Discovery
1021
1022 When the empty node joins the P2P network, it asks all peers to report snapshots via the
1023 `ListSnapshots` ABCI call (limited to 10 per node). After some time, the node picks the most
1024 suitable snapshot (generally prioritized by height, format, and number of peers), and offers it
1025 to the application via `OfferSnapshot`. The application can choose a number of responses,
1026 including accepting or rejecting it, rejecting the offered format, rejecting the peer who sent
1027 it, and so on. CometBFT will keep discovering and offering snapshots until one is accepted or
1028 the application aborts.
1029
1030 ##### Snapshot Restoration
1031
1032 Once a snapshot has been accepted via `OfferSnapshot`, CometBFT begins downloading chunks from
1033 any peers that have the same snapshot (i.e. that have identical metadata fields). Chunks are
1034 spooled in a temporary directory, and then given to the application in sequential order via
1035 `ApplySnapshotChunk` until all chunks have been accepted.
1036
1037 The method for restoring snapshot chunks is entirely up to the application.
1038
1039 During restoration, the application can respond to `ApplySnapshotChunk` with instructions for how
1040 to continue. This will typically be to accept the chunk and await the next one, but it can also
1041 ask for chunks to be refetched (either the current one or any number of previous ones), P2P peers
1042 to be banned, snapshots to be rejected or retried, and a number of other responses - see the ABCI
1043 reference for details.
1044
1045 If CometBFT fails to fetch a chunk after some time, it will reject the snapshot and try a
1046 different one via `OfferSnapshot` - the application can choose whether it wants to support
1047 restarting restoration, or simply abort with an error.
1048
1049 ##### Snapshot Verification
1050
1051 Once all chunks have been accepted, CometBFT issues an `Info` ABCI call to retrieve the
1052 `LastBlockAppHash`. This is compared with the trusted app hash from the chain, retrieved and
1053 verified using the light client. CometBFT also checks that `LastBlockHeight` corresponds to the
1054 height of the snapshot.
1055
1056 This verification ensures that an application is valid before joining the network. However, the
1057 snapshot restoration may take a long time to complete, so applications may want to employ additional
1058 verification during the restore to detect failures early. This might e.g. include incremental
1059 verification of each chunk against the app hash (using bundled Merkle proofs), checksums to
1060 protect against data corruption by the disk or network, and so on. However, it is important to
1061 note that the only trusted information available is the app hash, and all other snapshot metadata
1062 can be spoofed by adversaries.
1063
1064 Apps may also want to consider state sync denial-of-service vectors, where adversaries provide
1065 invalid or harmful snapshots to prevent nodes from joining the network. The application can
1066 counteract this by asking CometBFT to ban peers. As a last resort, node operators can use
1067 P2P configuration options to whitelist a set of trusted peers that can provide valid snapshots.
1068
1069 ##### Transition to Consensus
1070
1071 Once the snapshots have all been restored, CometBFT gathers additional information necessary for
1072 bootstrapping the node (e.g. chain ID, consensus parameters, validator sets, and block headers)
1073 from the genesis file and light client RPC servers. It also calls `Info` to verify the following:
1074
1075 * that the app hash from the snapshot it has delivered to the Application matches the apphash
1076 stored in the next height's block
1077 * that the version that the Application returns in `ResponseInfo` matches the version in the
1078 current height's block header
1079
1080 Once the state machine has been restored and CometBFT has gathered this additional
1081 information, it transitions to block sync (if enabled) to fetch any remaining blocks up the chain
1082 head, and then transitions to regular consensus operation. At this point the node operates like
1083 any other node, apart from having a truncated block history at the height of the restored snapshot.