github.com/badrootd/celestia-core@v0.0.0-20240305091328-aa4207a4b25d/docs/celestia-architecture/adr-007-minimal-changes-to-tendermint.md (about) 1 # ADR 007: From Ukraine, with Love 2 3 ## Changelog 4 5 - 2021-08-20: Initial Description 6 - 2022-05-03: Update pointing to ADR 008 7 8 ## Context 9 10 Currently, our fork of tendermint includes changes to how to erasure block data, minor changes to the header to commit 11 to that data, additions to serve data availability sampling, along with some miscellaneous modification to adhere to the 12 spec. Instead of incorporating all of these changes into our fork of tendermint, we will only make the strictly 13 necessary changes and the other services and their code to the new celestia-node repo. Notably, we will also refactor 14 some of the remaining necessary changes to be more isolated from the rest of the tendermint codebase. Both of these 15 strategies should significantly streamline pulling updates from upstream, and allow us to iterate faster since most 16 changes will be isolated to celestia-node. 17 18 Update: many of the changes described below have since been minimized or removed. Please see ADR 008 for a summarized list of changes. Notably, we removed intermediate state roots, adopted two new methods from ABCI++ instead of PreprocessTxs, and are still signing over the PartSetHeader. 19 20 ## Decision 21 22 Treat tendermint more as a "black box". 23 24 ## Detailed Design 25 26 ### Overview 27 28 We keep the bare-minimum changes to tendermint in our fork, celestia-core. Where necessary and possible we augment the 29 tendermint node in a separate process, via celestia-node, which communicates with the tendermint node via RPC. All data 30 availability sampling logic, including all Celestia-specific networking logic not already provided by tendermint, is 31 moved into celestia node: 32 33  34 35 The detailed design of celestia-node will be defined in the repository itself. 36 37 ### Necessary changes to tendermint 38 39 #### Changing the repo import names to celestiaorg 40 41 - Rebrand (https://github.com/celestiaorg/celestia-core/pull/476) 42 43 #### Changes to the README.md other basic things 44 45 - update github templates (https://github.com/celestiaorg/celestia-core/pull/405) 46 - update README.md (https://github.com/celestiaorg/celestia-core/pull/10) 47 48 #### Adding the extra types of block data 49 50 - Update core data types (https://github.com/celestiaorg/celestia-core/pull/17) 51 - Create the Message/Messages types 52 - Proto and the tendermint version 53 - Create the IntermediateStateRoots type 54 - Proto and the tendermint version 55 - Data availability for evidence (https://github.com/celestiaorg/celestia-core/pull/19) 56 - Add both types to `types.Data` 57 - Modify proto 58 - Add `EvidenceData` to `types.Data` 59 60 #### Add the HeaderHash to the Commit 61 62 - Add header hash to commit(https://github.com/celestiaorg/celestia-core/pull/198) 63 64 #### Adding the consts package in types 65 66 #### Remove iavl as a dependency 67 68 - remove iavl as a dependency (https://github.com/celestiaorg/celestia-core/pull/129) 69 70 #### Using the `DataAvailabilityHeader` to calculate the DataHash 71 72 The `DataAvailabilityHeader` struct will be used by celestia-core as well as by the celestia-node. It might make sense 73 to (eventually) move the struct together with all the DA-related code into a separate repository and go-module. 74 @Wondertan explored this as part of [#427](https://github.com/celestiaorg/celestia-core/pull/427#issue-674512464). This 75 way all client implementations can depend on that module without running into circular dependencies. Hence, we only 76 describe how to hash the block data here: 77 78 - Update core types (https://github.com/celestiaorg/celestia-core/pull/17) 79 - Replace the `Data.Hash()` with `DAH.Hash()` 80 - Use DAH to fill DataHash when filling the header 81 - Fill the DAH when making a block to generate the data hash 82 83 #### Add availableDataOriginalSharesUsed to the header 84 85 - Add availableDataOriginalSharesUsed to the header (https://github.com/celestiaorg/celestia-core/pull/262) 86 87 #### Reap some number of transactions probably using the app or some other mech 88 89 - Enforce a minimum square size (https://github.com/celestiaorg/celestia-core/pull/282) 90 - Use squares with a width that is a power of two(https://github.com/celestiaorg/celestia-core/pull/331) 91 - Adopt reamping from the mempool to max square size (https://github.com/celestiaorg/celestia-core/issues/77) 92 - Proposal: Decide on a mech to pick square size and communicate that to the 93 app (https://github.com/celestiaorg/celestia-core/issues/454) 94 - Also see ABCI++ for a less hacky solution 95 96 #### Filling the DAH using share merging and splitting 97 98 - Compute Shares (not merged) (https://github.com/celestiaorg/celestia-core/pull/60) 99 - part II (not merged) (https://github.com/celestiaorg/celestia-core/pull/63) 100 - while this was not merged, we will need some function to compute the shares that make up the block data 101 - Share Splitting (https://github.com/celestiaorg/celestia-core/pull/246) 102 - Serialize each constituent of block data 103 - Split into shares 104 - Txs (contiguous) 105 - Messages (not contiguous) 106 - Evidence (contiguous) 107 - IntermediateStateRoots (contiguous) 108 - Combine shares into original square 109 - ExtendBlockData 110 - Generate nmt root of each row and col 111 - Use those roots to generate the DataHash 112 - Share Merging (https://github.com/celestiaorg/celestia-core/pull/261) 113 - Sort by namespace 114 - Parse each reserved type 115 - Parse remaining messages 116 117 #### Add the wrapper around nmt to erasure namespaces 118 119 - Implement rsmt tree wrapper for nmt (https://github.com/celestiaorg/celestia-core/pull/238) 120 121 #### Add PreprocessTxs to ABCI 122 123 - Add PreprocessTxs method to ABCI (https://github.com/celestiaorg/celestia-core/pull/110) 124 - Add method to ABCI interface 125 - Create sync and async versions 126 - Add sync version the the CreateProposalBlock method of BlockExecutor 127 128 #### Fill the DAH while making the block 129 130 - Basic DA functionality (https://github.com/celestiaorg/celestia-core/pull/83) 131 132 #### Only produce blocks on some interval 133 134 - Control block times (https://github.com/cometbft/cometbft/issues/5911) 135 136 #### Stop signing over the PartSetHeader 137 138 - Replace canonical blockID with just a hash in the CononicalVote 139 - Replace the LastBlockID in the header with just a hash 140 141 #### Optionally remove some unused code 142 143 - Removing misc unused code (https://github.com/celestiaorg/celestia-core/pull/208) 144 - Remove docs deployment (https://github.com/celestiaorg/celestia-core/pull/134) 145 - Start deleting docs (https://github.com/celestiaorg/celestia-core/pull/209) 146 - Remove tendermint-db in favor of badgerdb (https://github.com/celestiaorg/celestia-core/pull/241) 147 - Delete blockchain 2 until further notice (https://github.com/celestiaorg/celestia-core/pull/309) 148 - We don’t need to support using out of process apps 149 150 #### Nice to Haves 151 152 - More efficient hashing (https://github.com/celestiaorg/celestia-core/pull/351) 153 154 We should also take this opportunity to refactor as many additions to tendermint into their own package as possible. 155 This will hopefully make updating to future versions of tendermint easier. For example, when we fill the data 156 availability header, instead of using a method on `Block`, it could be handled by a function that takes `types.Data` as 157 input and returns the DAH, the number of shares used in the square, along with the obligatory error. 158 159 ```go 160 func FillDataAvailabilityHeader(data types.Data) (types.DataAvailabilityHeader, numOrigDataShares, error) 161 ``` 162 163 We could perform a similar treatment to the `splitIntoShares` methods and their helper method `ComputeShares`. Instead 164 of performing the share splitting logic in those methods, we could keep it in a different package and instead call the 165 equivalent function to compute the shares. 166 167 Beyond refactoring and some minor additions, we will also have to remove and revert quite a few changes to get to the 168 minimum desired changes specified above. 169 170 ### Changes that will need to be reverted 171 172 #### IPLD Plugin 173 174 - Introduction (https://github.com/celestiaorg/celestia-core/pull/144) 175 - Initial integration (https://github.com/celestiaorg/celestia-core/pull/152) 176 - Custom Multihash (https://github.com/celestiaorg/celestia-core/pull/155) 177 - Putting data during proposal (https://github.com/celestiaorg/celestia-core/pull/178) 178 - Module name (https://github.com/celestiaorg/celestia-core/pull/151) 179 - Update rsmt2d (https://github.com/celestiaorg/celestia-core/pull/290) 180 - Make plugin a package (https://github.com/celestiaorg/celestia-core/pull/294) 181 182 #### Adding DAH to Stuff 183 184 - Adding DAH to Proposal (https://github.com/celestiaorg/celestia-core/pull/248/files) 185 - Blockmeta (https://github.com/celestiaorg/celestia-core/pull/372) 186 187 #### Embedding DAS 188 189 - GetLeafData (https://github.com/celestiaorg/celestia-core/pull/212) 190 - RetrieveBlockData (https://github.com/celestiaorg/celestia-core/pull/232) 191 - ValidateAvailability (https://github.com/celestiaorg/celestia-core/pull/270) 192 - Prevent double writes to IPFS (https://github.com/celestiaorg/celestia-core/pull/271) 193 - Stop Pinning (https://github.com/celestiaorg/celestia-core/pull/276) 194 - Rework IPFS Node (https://github.com/celestiaorg/celestia-core/pull/334) 195 - Refactor for putting the block (https://github.com/celestiaorg/celestia-core/pull/338) 196 - Config for IPFS node (https://github.com/celestiaorg/celestia-core/pull/340) 197 - IPLD Dag instead of CoreAPI (https://github.com/celestiaorg/celestia-core/pull/352) 198 - Adding the DAG to the blockstore (https://github.com/celestiaorg/celestia-core/pull/356) 199 - Saving and Loading using IPFS (https://github.com/celestiaorg/celestia-core/pull/374) 200 - Manual Providing (https://github.com/celestiaorg/celestia-core/pull/375) 201 - Refactor node provider (https://github.com/celestiaorg/celestia-core/pull/400) 202 - DAS in light client workaround (https://github.com/celestiaorg/celestia-core/pull/413) 203 204 #### BlockID and PartSetHeader 205 206 - Decouple ParSetHeader from BlockID (https://github.com/celestiaorg/celestia-core/pull/441) 207 - Stop Signing over the PartSetHeader (https://github.com/celestiaorg/celestia-core/pull/457) 208 - We still don’t want to sign over the PartSetHeader, but we will not be able to use the same mechanism used in the 209 linked PR, as that way requires decoupling of the PSH from the BlockID 210 - Remove PSH from some consensus messages (https://github.com/celestiaorg/celestia-core/pull/479) 211 212 Note: This ADR overrides ADR 005 Decouple BlockID and the PartSetHeader. The PartSetHeader and the BlockID will mostly 213 remain the same. This will make pulling changes from upstream much easier 214 215 ## Status 216 217 Accepted 218 219 ## Consequences 220 221 ### Positive 222 223 - Pulling changes from upstream is streamlined 224 - Separation of functionality will help us iterate faster 225 - Creates a great opportunity for reconsidering past design choices without fully starting from scratch 226 - Prepare for future designs 227 - Don’t have to have two p2p stacks in a single repo 228 229 ### Negative 230 231 - Perform some computation multiple times 232 - Running multiple nodes instead of a single node is less convenient for node operators (but only in the case the full 233 celestia-node wants to participate in the consensus protocol) 234 235 ## References 236 237 Tracking Issue #491