github.com/badrootd/celestia-core@v0.0.0-20240305091328-aa4207a4b25d/docs/celestia-architecture/adr-007-minimal-changes-to-tendermint.md (about)

     1  # ADR 007: From Ukraine, with Love
     2  
     3  ## Changelog
     4  
     5  - 2021-08-20: Initial Description
     6  - 2022-05-03: Update pointing to ADR 008
     7  
     8  ## Context
     9  
    10  Currently, our fork of tendermint includes changes to how to erasure block data, minor changes to the header to commit
    11  to that data, additions to serve data availability sampling, along with some miscellaneous modification to adhere to the
    12  spec. Instead of incorporating all of these changes into our fork of tendermint, we will only make the strictly
    13  necessary changes and the other services and their code to the new celestia-node repo. Notably, we will also refactor
    14  some of the remaining necessary changes to be more isolated from the rest of the tendermint codebase. Both of these
    15  strategies should significantly streamline pulling updates from upstream, and allow us to iterate faster since most
    16  changes will be isolated to celestia-node.
    17  
    18  Update: many of the changes described below have since been minimized or removed. Please see ADR 008 for a summarized list of changes. Notably, we removed intermediate state roots, adopted two new methods from ABCI++ instead of PreprocessTxs, and are still signing over the PartSetHeader. 
    19  
    20  ## Decision
    21  
    22  Treat tendermint more as a "black box".
    23  
    24  ## Detailed Design
    25  
    26  ### Overview
    27  
    28  We keep the bare-minimum changes to tendermint in our fork, celestia-core. Where necessary and possible we augment the
    29  tendermint node in a separate process, via celestia-node, which communicates with the tendermint node via RPC. All data
    30  availability sampling logic, including all Celestia-specific networking logic not already provided by tendermint, is
    31  moved into celestia node:
    32  
    33  ![core node relation](./img/core-node-relation.jpg)
    34  
    35  The detailed design of celestia-node will be defined in the repository itself.
    36  
    37  ### Necessary changes to tendermint
    38  
    39  #### Changing the repo import names to celestiaorg
    40  
    41  - Rebrand (https://github.com/celestiaorg/celestia-core/pull/476)
    42  
    43  #### Changes to the README.md other basic things
    44  
    45  - update github templates (https://github.com/celestiaorg/celestia-core/pull/405)
    46  - update README.md (https://github.com/celestiaorg/celestia-core/pull/10)
    47  
    48  #### Adding the extra types of block data
    49  
    50  - Update core data types (https://github.com/celestiaorg/celestia-core/pull/17)
    51      - Create the Message/Messages types
    52      - Proto and the tendermint version
    53      - Create the IntermediateStateRoots type
    54      - Proto and the tendermint version
    55  - Data availability for evidence (https://github.com/celestiaorg/celestia-core/pull/19)
    56      - Add both types to `types.Data`
    57      - Modify proto
    58      - Add `EvidenceData` to `types.Data`
    59  
    60  #### Add the HeaderHash to the Commit
    61  
    62  - Add header hash to commit(https://github.com/celestiaorg/celestia-core/pull/198)
    63  
    64  #### Adding the consts package in types
    65  
    66  #### Remove iavl as a dependency
    67  
    68  - remove iavl as a dependency (https://github.com/celestiaorg/celestia-core/pull/129)
    69  
    70  #### Using the `DataAvailabilityHeader` to calculate the DataHash
    71  
    72  The `DataAvailabilityHeader` struct will be used by celestia-core as well as by the celestia-node. It might make sense
    73  to (eventually) move the struct together with all the DA-related code into a separate repository and go-module.
    74  @Wondertan explored this as part of [#427](https://github.com/celestiaorg/celestia-core/pull/427#issue-674512464). This
    75  way all client implementations can depend on that module without running into circular dependencies. Hence, we only
    76  describe how to hash the block data here:
    77  
    78  - Update core types (https://github.com/celestiaorg/celestia-core/pull/17)
    79      - Replace the `Data.Hash()` with `DAH.Hash()`
    80      - Use DAH to fill DataHash when filling the header
    81      - Fill the DAH when making a block to generate the data hash
    82  
    83  #### Add availableDataOriginalSharesUsed to the header
    84  
    85  - Add availableDataOriginalSharesUsed to the header (https://github.com/celestiaorg/celestia-core/pull/262)
    86  
    87  #### Reap some number of transactions probably using the app or some other mech
    88  
    89  - Enforce a minimum square size (https://github.com/celestiaorg/celestia-core/pull/282)
    90  - Use squares with a width that is a power of two(https://github.com/celestiaorg/celestia-core/pull/331)
    91  - Adopt reamping from the mempool to max square size (https://github.com/celestiaorg/celestia-core/issues/77)
    92  - Proposal: Decide on a mech to pick square size and communicate that to the
    93    app (https://github.com/celestiaorg/celestia-core/issues/454)
    94  - Also see ABCI++ for a less hacky solution
    95  
    96  #### Filling the DAH using share merging and splitting
    97  
    98  - Compute Shares (not merged) (https://github.com/celestiaorg/celestia-core/pull/60)
    99      - part II (not merged) (https://github.com/celestiaorg/celestia-core/pull/63)
   100      - while this was not merged, we will need some function to compute the shares that make up the block data
   101  - Share Splitting (https://github.com/celestiaorg/celestia-core/pull/246)
   102      - Serialize each constituent of block data
   103      - Split into shares
   104          - Txs (contiguous)
   105          - Messages (not contiguous)
   106          - Evidence (contiguous)
   107          - IntermediateStateRoots (contiguous)
   108  - Combine shares into original square
   109  - ExtendBlockData
   110  - Generate nmt root of each row and col
   111  - Use those roots to generate the DataHash
   112  - Share Merging (https://github.com/celestiaorg/celestia-core/pull/261)
   113      - Sort by namespace
   114      - Parse each reserved type
   115      - Parse remaining messages
   116  
   117  #### Add the wrapper around nmt to erasure namespaces
   118  
   119  - Implement rsmt tree wrapper for nmt (https://github.com/celestiaorg/celestia-core/pull/238)
   120  
   121  #### Add PreprocessTxs to ABCI
   122  
   123  - Add PreprocessTxs method to ABCI (https://github.com/celestiaorg/celestia-core/pull/110)
   124  - Add method to ABCI interface
   125  - Create sync and async versions
   126  - Add sync version the the CreateProposalBlock method of BlockExecutor
   127  
   128  #### Fill the DAH while making the block
   129  
   130  - Basic DA functionality (https://github.com/celestiaorg/celestia-core/pull/83)
   131  
   132  #### Only produce blocks on some interval
   133  
   134  - Control block times (https://github.com/cometbft/cometbft/issues/5911)
   135  
   136  #### Stop signing over the PartSetHeader
   137  
   138  - Replace canonical blockID with just a hash in the CononicalVote
   139  - Replace the LastBlockID in the header with just a hash
   140  
   141  #### Optionally remove some unused code
   142  
   143  - Removing misc unused code (https://github.com/celestiaorg/celestia-core/pull/208)
   144  - Remove docs deployment (https://github.com/celestiaorg/celestia-core/pull/134)
   145  - Start deleting docs (https://github.com/celestiaorg/celestia-core/pull/209)
   146  - Remove tendermint-db in favor of badgerdb (https://github.com/celestiaorg/celestia-core/pull/241)
   147  - Delete blockchain 2 until further notice (https://github.com/celestiaorg/celestia-core/pull/309)
   148  - We don’t need to support using out of process apps
   149  
   150  #### Nice to Haves
   151  
   152  - More efficient hashing (https://github.com/celestiaorg/celestia-core/pull/351)
   153  
   154  We should also take this opportunity to refactor as many additions to tendermint into their own package as possible.
   155  This will hopefully make updating to future versions of tendermint easier. For example, when we fill the data
   156  availability header, instead of using a method on `Block`, it could be handled by a function that takes `types.Data` as
   157  input and returns the DAH, the number of shares used in the square, along with the obligatory error.
   158  
   159  ```go
   160  func FillDataAvailabilityHeader(data types.Data) (types.DataAvailabilityHeader, numOrigDataShares, error)
   161  ```
   162  
   163  We could perform a similar treatment to the `splitIntoShares` methods and their helper method `ComputeShares`. Instead
   164  of performing the share splitting logic in those methods, we could keep it in a different package and instead call the
   165  equivalent function to compute the shares.
   166  
   167  Beyond refactoring and some minor additions, we will also have to remove and revert quite a few changes to get to the
   168  minimum desired changes specified above.
   169  
   170  ### Changes that will need to be reverted
   171  
   172  #### IPLD Plugin
   173  
   174  - Introduction (https://github.com/celestiaorg/celestia-core/pull/144)
   175  - Initial integration (https://github.com/celestiaorg/celestia-core/pull/152)
   176  - Custom Multihash (https://github.com/celestiaorg/celestia-core/pull/155)
   177  - Putting data during proposal (https://github.com/celestiaorg/celestia-core/pull/178)
   178  - Module name (https://github.com/celestiaorg/celestia-core/pull/151)
   179  - Update rsmt2d (https://github.com/celestiaorg/celestia-core/pull/290)
   180  - Make plugin a package (https://github.com/celestiaorg/celestia-core/pull/294)
   181  
   182  #### Adding DAH to Stuff
   183  
   184  - Adding DAH to Proposal (https://github.com/celestiaorg/celestia-core/pull/248/files)
   185  - Blockmeta (https://github.com/celestiaorg/celestia-core/pull/372)
   186  
   187  #### Embedding DAS
   188  
   189  - GetLeafData (https://github.com/celestiaorg/celestia-core/pull/212)
   190  - RetrieveBlockData (https://github.com/celestiaorg/celestia-core/pull/232)
   191  - ValidateAvailability (https://github.com/celestiaorg/celestia-core/pull/270)
   192  - Prevent double writes to IPFS (https://github.com/celestiaorg/celestia-core/pull/271)
   193  - Stop Pinning (https://github.com/celestiaorg/celestia-core/pull/276)
   194  - Rework IPFS Node (https://github.com/celestiaorg/celestia-core/pull/334)
   195  - Refactor for putting the block (https://github.com/celestiaorg/celestia-core/pull/338)
   196  - Config for IPFS node (https://github.com/celestiaorg/celestia-core/pull/340)
   197  - IPLD Dag instead of CoreAPI (https://github.com/celestiaorg/celestia-core/pull/352)
   198  - Adding the DAG to the blockstore (https://github.com/celestiaorg/celestia-core/pull/356)
   199  - Saving and Loading using IPFS (https://github.com/celestiaorg/celestia-core/pull/374)
   200  - Manual Providing (https://github.com/celestiaorg/celestia-core/pull/375)
   201  - Refactor node provider (https://github.com/celestiaorg/celestia-core/pull/400)
   202  - DAS in light client workaround (https://github.com/celestiaorg/celestia-core/pull/413)
   203  
   204  #### BlockID and PartSetHeader
   205  
   206  - Decouple ParSetHeader from BlockID (https://github.com/celestiaorg/celestia-core/pull/441)
   207  - Stop Signing over the PartSetHeader (https://github.com/celestiaorg/celestia-core/pull/457)
   208  - We still don’t want to sign over the PartSetHeader, but we will not be able to use the same mechanism used in the
   209    linked PR, as that way requires decoupling of the PSH from the BlockID
   210  - Remove PSH from some consensus messages (https://github.com/celestiaorg/celestia-core/pull/479)
   211  
   212  Note: This ADR overrides ADR 005 Decouple BlockID and the PartSetHeader. The PartSetHeader and the BlockID will mostly
   213  remain the same. This will make pulling changes from upstream much easier
   214  
   215  ## Status
   216  
   217  Accepted
   218  
   219  ## Consequences
   220  
   221  ### Positive
   222  
   223  - Pulling changes from upstream is streamlined
   224  - Separation of functionality will help us iterate faster
   225  - Creates a great opportunity for reconsidering past design choices without fully starting from scratch
   226  - Prepare for future designs
   227  - Don’t have to have two p2p stacks in a single repo
   228  
   229  ### Negative
   230  
   231  - Perform some computation multiple times
   232  - Running multiple nodes instead of a single node is less convenient for node operators (but only in the case the full
   233    celestia-node wants to participate in the consensus protocol)
   234  
   235  ## References
   236  
   237  Tracking Issue #491