github.com/ari-anchor/sei-tendermint@v0.0.0-20230519144642-dc826b7b56bb/docs/architecture/adr-069-flexible-node-initialization.md (about) 1 # ADR 069: Flexible Node Initialization 2 3 ## Changlog 4 5 - 2021-06-09: Initial Draft (@tychoish) 6 7 - 2021-07-21: Major Revision (@tychoish) 8 9 ## Status 10 11 Proposed. 12 13 ## Context 14 15 In an effort to support [Go-API-Stability](./adr-060-go-api-stability.md), 16 during the 0.35 development cycle, we have attempted to reduce the the API 17 surface area by moving most of the interface of the `node` package into 18 unexported functions, as well as moving the reactors to an `internal` 19 package. Having this coincide with the 0.35 release made a lot of sense 20 because these interfaces were _already_ changing as a result of the `p2p` 21 [refactor](./adr-061-p2p-refactor-scope.md), so it made sense to think a bit 22 more about how tendermint exposes this API. 23 24 While the interfaces of the P2P layer and most of the node package are already 25 internalized, this precludes some operational patterns that are important to 26 users who use tendermint as a library. Specifically, introspecting the 27 tendermint node service and replacing components is not supported in the latest 28 version of the code, and some of these use cases would require maintaining a 29 vendor copy of the code. Adding these features requires rather extensive 30 (internal/implementation) changes to the `node` and `rpc` packages, and this 31 ADR describes a model for changing the way that tendermint nodes initialize, in 32 service of providing this kind of functionality. 33 34 We consider node initialization, because the current implemention 35 provides strong connections between all components, as well as between 36 the components of the node and the RPC layer, and being able to think 37 about the interactions of these components will help enable these 38 features and help define the requirements of the node package. 39 40 ## Alternative Approaches 41 42 These alternatives are presented to frame the design space and to 43 contextualize the decision in terms of product requirements. These 44 ideas are not inherently bad, and may even be possible or desireable 45 in the (distant) future, and merely provide additional context for how 46 we, in the moment came to our decision(s). 47 48 ### Do Nothing 49 50 The current implementation is functional and sufficient for the vast 51 majority of use cases (e.g., all users of the Cosmos-SDK as well as 52 anyone who runs tendermint and the ABCI application in separate 53 processes). In the current implementation, and even previous versions, 54 modifying node initialization or injecting custom components required 55 copying most of the `node` package, which required such users 56 to maintain a vendored copy of tendermint. 57 58 While this is (likely) not tenable in the long term, as users do want 59 more modularity, and the current service implementation is brittle and 60 difficult to maintain, in the short term it may be possible to delay 61 implementation somewhat. Eventually, however, we will need to make the 62 `node` package easier to maintain and reason about. 63 64 ### Generic Service Pluggability 65 66 One possible system design would export interfaces (in the Golang 67 sense) for all components of the system, to permit runtime dependency 68 injection of all components in the system, so that users can compose 69 tendermint nodes of arbitrary user-supplied components. 70 71 Although this level of customization would provide benefits, it would be a huge 72 undertaking (particularly with regards to API design work) that we do not have 73 scope for at the moment. Eventually providing support for some kinds of 74 pluggability may be useful, so the current solution does not explicitly 75 foreclose the possibility of this alternative. 76 77 ### Abstract Dependency Based Startup and Shutdown 78 79 The main proposal in this document makes tendermint node initialization simpler 80 and more abstract, but the system lacks a number of 81 features which daemon/service initialization could provide, such as a 82 system allowing the authors of services to control initialization and shutdown order 83 of components using dependency relationships. 84 85 Such a system could work by allowing services to declare 86 initialization order dependencies to other reactors (by ID, perhaps) 87 so that the node could decide the initialization based on the 88 dependencies declared by services rather than requiring the node to 89 encode this logic directly. 90 91 This level of configuration is probably more complicated than is needed. Given 92 that the authors of components in the current implementation of tendermint 93 already *do* need to know about other components, a dependency-based system 94 would probably be overly-abstract at this stage. 95 96 ## Decisions 97 98 - To the greatest extent possible, factor the code base so that 99 packages are responsible for their own initialization, and minimize 100 the amount of code in the `node` package itself. 101 102 - As a design goal, reduce direct coupling and dependencies between 103 components in the implementation of `node`. 104 105 - Begin iterating on a more-flexible internal framework for 106 initializing tendermint nodes to make the initatilization process 107 less hard-coded by the implementation of the node objects. 108 109 - Reactors should not need to expose their interfaces *within* the 110 implementation of the node type 111 112 - This refactoring should be entirely opaque to users. 113 114 - These node initialization changes should not require a 115 reevaluation of the `service.Service` or a generic initialization 116 orchestration framework. 117 118 - Do not proactively provide a system for injecting 119 components/services within a tendtermint node, though make it 120 possible to retrofit this kind of plugability in the future if 121 needed. 122 123 - Prioritize implementation of p2p-based statesync reactor to obviate 124 need for users to inject a custom state-sync provider. 125 126 ## Detailed Design 127 128 The [current 129 nodeImpl](https://github.com/tendermint/tendermint/blob/master/node/node.go#L47) 130 includes direct references to the implementations of each of the 131 reactors, which should be replaced by references to `service.Service` 132 objects. This will require moving construction of the [rpc 133 service](https://github.com/tendermint/tendermint/blob/master/node/node.go#L771) 134 into the constructor of 135 [makeNode](https://github.com/tendermint/tendermint/blob/master/node/node.go#L126). One 136 possible implementation of this would be to eliminate the current 137 `ConfigureRPC` method on the node package and instead [configure it 138 here](https://github.com/tendermint/tendermint/pull/6798/files#diff-375d57e386f20eaa5f09f02bb9d28bfc48ac3dca18d0325f59492208219e5618R441). 139 140 To avoid adding complexity to the `node` package, we will add a 141 composite service implementation to the `service` package 142 that implements `service.Service` and is composed of a sequence of 143 underlying `service.Service` objects and handles their 144 startup/shutdown in the specified sequential order. 145 146 Consensus, blocksync (*née* fast sync), and statesync all depend on 147 each other, and have significant initialization dependencies that are 148 presently encoded in the `node` package. As part of this change, a 149 new package/component (likely named `blocks` located at 150 `internal/blocks`) will encapsulate the initialization of these block 151 management areas of the code. 152 153 ### Injectable Component Option 154 155 This section briefly describes a possible implementation for 156 user-supplied services running within a node. This should not be 157 implemented unless user-supplied components are a hard requirement for 158 a user. 159 160 In order to allow components to be replaced, a new public function 161 will be added to the public interface of `node` with a signature that 162 resembles the following: 163 164 ```go 165 func NewWithServices(conf *config.Config, 166 logger log.Logger, 167 cf proxy.ClientCreator, 168 gen *types.GenesisDoc, 169 srvs []service.Service, 170 ) (service.Service, error) { 171 ``` 172 173 The `service.Service` objects will be initialized in the order supplied, after 174 all pre-configured/default services have started (and shut down in reverse 175 order). The given services may implement additional interfaces, allowing them 176 to replace specific default services. `NewWithServices` will validate input 177 service lists with the following rules: 178 179 - None of the services may already be running. 180 - The caller may not supply more than one replacement reactor for a given 181 default service type. 182 183 If callers violate any of these rules, `NewWithServices` will return 184 an error. To retract support for this kind of operation in the future, 185 the function can be modified to *always* return an error. 186 187 ## Consequences 188 189 ### Positive 190 191 - The node package will become easier to maintain. 192 193 - It will become easier to add additional services within tendermint 194 nodes. 195 196 - It will become possible to replace default components in the node 197 package without vendoring the tendermint repo and modifying internal 198 code. 199 200 - The current end-to-end (e2e) test suite will be able to prevent any 201 regressions, and the new functionality can be thoroughly unit tested. 202 203 - The scope of this project is very narrow, which minimizes risk. 204 205 ### Negative 206 207 - This increases our reliance on the `service.Service` interface which 208 is probably not an interface that we want to fully commit to. 209 210 - This proposal implements a fairly minimal set of functionality and 211 leaves open the possibility for many additional features which are 212 not included in the scope of this proposal. 213 214 ### Neutral 215 216 N/A 217 218 ## Open Questions 219 220 - To what extent does this new initialization framework need to accommodate 221 the legacy p2p stack? Would it be possible to delay a great deal of this 222 work to the 0.36 cycle to avoid this complexity? 223 224 - Answer: _depends on timing_, and the requirement to ship pluggable reactors in 0.35. 225 226 - Where should additional public types be exported for the 0.35 227 release? 228 229 Related to the general project of API stabilization we want to deprecate 230 the `types` package, and move its contents into a new `pkg` hierarchy; 231 however, the design of the `pkg` interface is currently underspecified. 232 If `types` is going to remain for the 0.35 release, then we should consider 233 the impact of using multiple organizing modalities for this code within a 234 single release. 235 236 ## Future Work 237 238 - Improve or simplify the `service.Service` interface. There are some 239 pretty clear limitations with this interface as written (there's no 240 way to timeout slow startup or shut down, the cycle between the 241 `service.BaseService` and `service.Service` implementations is 242 troubling, the default panic in `OnReset` seems troubling.) 243 244 - As part of the refactor of `service.Service` have all services/nodes 245 respect the lifetime of a `context.Context` object, and avoid the 246 current practice of creating `context.Context` objects in p2p and 247 reactor code. This would be required for in-process multi-tenancy. 248 249 - Support explicit dependencies between components and allow for 250 parallel startup, so that different reactors can startup at the same 251 time, where possible. 252 253 ## References 254 255 - [the component 256 graph](https://peter.bourgon.org/go-for-industrial-programming/#the-component-graph) 257 as a framing for internal service construction. 258 259 ## Appendix 260 261 ### Dependencies 262 263 There's a relationship between the blockchain and consensus reactor 264 described by the following dependency graph makes replacing some of 265 these components more difficult relative to other reactors or 266 components. 267 268 ![consensus blockchain dependency graph](./img/consensus_blockchain.png)