github.com/fiagdao/tendermint@v0.32.11-0.20220824195748-2087fcc480c1/docs/guides/go-built-in.md (about) 1 <!--- 2 order: 2 3 ---> 4 5 # Creating a built-in application in Go 6 7 ## Guide assumptions 8 9 This guide is designed for beginners who want to get started with a Tendermint 10 Core application from scratch. It does not assume that you have any prior 11 experience with Tendermint Core. 12 13 Tendermint Core is Byzantine Fault Tolerant (BFT) middleware that takes a state 14 transition machine - written in any programming language - and securely 15 replicates it on many machines. 16 17 Although Tendermint Core is written in the Golang programming language, prior 18 knowledge of it is not required for this guide. You can learn it as we go due 19 to it's simplicity. However, you may want to go through [Learn X in Y minutes 20 Where X=Go](https://learnxinyminutes.com/docs/go/) first to familiarize 21 yourself with the syntax. 22 23 By following along with this guide, you'll create a Tendermint Core project 24 called kvstore, a (very) simple distributed BFT key-value store. 25 26 ## Built-in app vs external app 27 28 Running your application inside the same process as Tendermint Core will give 29 you the best possible performance. 30 31 For other languages, your application have to communicate with Tendermint Core 32 through a TCP, Unix domain socket or gRPC. 33 34 ## 1.1 Installing Go 35 36 Please refer to [the official guide for installing 37 Go](https://golang.org/doc/install). 38 39 Verify that you have the latest version of Go installed: 40 41 ```sh 42 $ go version 43 go version go1.13.1 darwin/amd64 44 ``` 45 46 Make sure you have `$GOPATH` environment variable set: 47 48 ```sh 49 $ echo $GOPATH 50 /Users/melekes/go 51 ``` 52 53 ## 1.2 Creating a new Go project 54 55 We'll start by creating a new Go project. 56 57 ```sh 58 $ mkdir kvstore 59 $ cd kvstore 60 ``` 61 62 Inside the example directory create a `main.go` file with the following content: 63 64 ```go 65 package main 66 67 import ( 68 "fmt" 69 ) 70 71 func main() { 72 fmt.Println("Hello, Tendermint Core") 73 } 74 ``` 75 76 When run, this should print "Hello, Tendermint Core" to the standard output. 77 78 ```sh 79 $ go run main.go 80 Hello, Tendermint Core 81 ``` 82 83 ## 1.3 Writing a Tendermint Core application 84 85 Tendermint Core communicates with the application through the Application 86 BlockChain Interface (ABCI). All message types are defined in the [protobuf 87 file](https://github.com/tendermint/tendermint/blob/master/abci/types/types.proto). 88 This allows Tendermint Core to run applications written in any programming 89 language. 90 91 Create a file called `app.go` with the following content: 92 93 ```go 94 package main 95 96 import ( 97 abcitypes "github.com/tendermint/tendermint/abci/types" 98 ) 99 100 type KVStoreApplication struct {} 101 102 var _ abcitypes.Application = (*KVStoreApplication)(nil) 103 104 func NewKVStoreApplication() *KVStoreApplication { 105 return &KVStoreApplication{} 106 } 107 108 func (KVStoreApplication) Info(req abcitypes.RequestInfo) abcitypes.ResponseInfo { 109 return abcitypes.ResponseInfo{} 110 } 111 112 func (KVStoreApplication) SetOption(req abcitypes.RequestSetOption) abcitypes.ResponseSetOption { 113 return abcitypes.ResponseSetOption{} 114 } 115 116 func (KVStoreApplication) DeliverTx(req abcitypes.RequestDeliverTx) abcitypes.ResponseDeliverTx { 117 return abcitypes.ResponseDeliverTx{Code: 0} 118 } 119 120 func (KVStoreApplication) CheckTx(req abcitypes.RequestCheckTx) abcitypes.ResponseCheckTx { 121 return abcitypes.ResponseCheckTx{Code: 0} 122 } 123 124 func (KVStoreApplication) Commit() abcitypes.ResponseCommit { 125 return abcitypes.ResponseCommit{} 126 } 127 128 func (KVStoreApplication) Query(req abcitypes.RequestQuery) abcitypes.ResponseQuery { 129 return abcitypes.ResponseQuery{Code: 0} 130 } 131 132 func (KVStoreApplication) InitChain(req abcitypes.RequestInitChain) abcitypes.ResponseInitChain { 133 return abcitypes.ResponseInitChain{} 134 } 135 136 func (KVStoreApplication) BeginBlock(req abcitypes.RequestBeginBlock) abcitypes.ResponseBeginBlock { 137 return abcitypes.ResponseBeginBlock{} 138 } 139 140 func (KVStoreApplication) EndBlock(req abcitypes.RequestEndBlock) abcitypes.ResponseEndBlock { 141 return abcitypes.ResponseEndBlock{} 142 } 143 ``` 144 145 Now I will go through each method explaining when it's called and adding 146 required business logic. 147 148 ### 1.3.1 CheckTx 149 150 When a new transaction is added to the Tendermint Core, it will ask the 151 application to check it (validate the format, signatures, etc.). 152 153 ```go 154 import "bytes" 155 156 func (app *KVStoreApplication) isValid(tx []byte) (code uint32) { 157 // check format 158 parts := bytes.Split(tx, []byte("=")) 159 if len(parts) != 2 { 160 return 1 161 } 162 163 key, value := parts[0], parts[1] 164 165 // check if the same key=value already exists 166 err := app.db.View(func(txn *badger.Txn) error { 167 item, err := txn.Get(key) 168 if err != nil && err != badger.ErrKeyNotFound { 169 return err 170 } 171 if err == nil { 172 return item.Value(func(val []byte) error { 173 if bytes.Equal(val, value) { 174 code = 2 175 } 176 return nil 177 }) 178 } 179 return nil 180 }) 181 if err != nil { 182 panic(err) 183 } 184 185 return code 186 } 187 188 func (app *KVStoreApplication) CheckTx(req abcitypes.RequestCheckTx) abcitypes.ResponseCheckTx { 189 code := app.isValid(req.Tx) 190 return abcitypes.ResponseCheckTx{Code: code, GasWanted: 1} 191 } 192 ``` 193 194 Don't worry if this does not compile yet. 195 196 If the transaction does not have a form of `{bytes}={bytes}`, we return `1` 197 code. When the same key=value already exist (same key and value), we return `2` 198 code. For others, we return a zero code indicating that they are valid. 199 200 Note that anything with non-zero code will be considered invalid (`-1`, `100`, 201 etc.) by Tendermint Core. 202 203 Valid transactions will eventually be committed given they are not too big and 204 have enough gas. To learn more about gas, check out ["the 205 specification"](https://docs.tendermint.com/master/spec/abci/apps.html#gas). 206 207 For the underlying key-value store we'll use 208 [badger](https://github.com/dgraph-io/badger), which is an embeddable, 209 persistent and fast key-value (KV) database. 210 211 ```go 212 import "github.com/dgraph-io/badger" 213 214 type KVStoreApplication struct { 215 db *badger.DB 216 currentBatch *badger.Txn 217 } 218 219 func NewKVStoreApplication(db *badger.DB) *KVStoreApplication { 220 return &KVStoreApplication{ 221 db: db, 222 } 223 } 224 ``` 225 226 ### 1.3.2 BeginBlock -> DeliverTx -> EndBlock -> Commit 227 228 When Tendermint Core has decided on the block, it's transfered to the 229 application in 3 parts: `BeginBlock`, one `DeliverTx` per transaction and 230 `EndBlock` in the end. DeliverTx are being transfered asynchronously, but the 231 responses are expected to come in order. 232 233 ``` 234 func (app *KVStoreApplication) BeginBlock(req abcitypes.RequestBeginBlock) abcitypes.ResponseBeginBlock { 235 app.currentBatch = app.db.NewTransaction(true) 236 return abcitypes.ResponseBeginBlock{} 237 } 238 239 ``` 240 241 Here we create a batch, which will store block's transactions. 242 243 ```go 244 func (app *KVStoreApplication) DeliverTx(req abcitypes.RequestDeliverTx) abcitypes.ResponseDeliverTx { 245 code := app.isValid(req.Tx) 246 if code != 0 { 247 return abcitypes.ResponseDeliverTx{Code: code} 248 } 249 250 parts := bytes.Split(req.Tx, []byte("=")) 251 key, value := parts[0], parts[1] 252 253 err := app.currentBatch.Set(key, value) 254 if err != nil { 255 panic(err) 256 } 257 258 return abcitypes.ResponseDeliverTx{Code: 0} 259 } 260 ``` 261 262 If the transaction is badly formatted or the same key=value already exist, we 263 again return the non-zero code. Otherwise, we add it to the current batch. 264 265 In the current design, a block can include incorrect transactions (those who 266 passed CheckTx, but failed DeliverTx or transactions included by the proposer 267 directly). This is done for performance reasons. 268 269 Note we can't commit transactions inside the `DeliverTx` because in such case 270 `Query`, which may be called in parallel, will return inconsistent data (i.e. 271 it will report that some value already exist even when the actual block was not 272 yet committed). 273 274 `Commit` instructs the application to persist the new state. 275 276 ```go 277 func (app *KVStoreApplication) Commit() abcitypes.ResponseCommit { 278 app.currentBatch.Commit() 279 return abcitypes.ResponseCommit{Data: []byte{}} 280 } 281 ``` 282 283 ### 1.3.3 Query 284 285 Now, when the client wants to know whenever a particular key/value exist, it 286 will call Tendermint Core RPC `/abci_query` endpoint, which in turn will call 287 the application's `Query` method. 288 289 Applications are free to provide their own APIs. But by using Tendermint Core 290 as a proxy, clients (including [light client 291 package](https://godoc.org/github.com/tendermint/tendermint/lite2)) can leverage 292 the unified API across different applications. Plus they won't have to call the 293 otherwise separate Tendermint Core API for additional proofs. 294 295 Note we don't include a proof here. 296 297 ```go 298 func (app *KVStoreApplication) Query(reqQuery abcitypes.RequestQuery) (resQuery abcitypes.ResponseQuery) { 299 resQuery.Key = reqQuery.Data 300 err := app.db.View(func(txn *badger.Txn) error { 301 item, err := txn.Get(reqQuery.Data) 302 if err != nil && err != badger.ErrKeyNotFound { 303 return err 304 } 305 if err == badger.ErrKeyNotFound { 306 resQuery.Log = "does not exist" 307 } else { 308 return item.Value(func(val []byte) error { 309 resQuery.Log = "exists" 310 resQuery.Value = val 311 return nil 312 }) 313 } 314 return nil 315 }) 316 if err != nil { 317 panic(err) 318 } 319 return 320 } 321 ``` 322 323 The complete specification can be found 324 [here](https://docs.tendermint.com/master/spec/abci/). 325 326 ## 1.4 Starting an application and a Tendermint Core instance in the same process 327 328 Put the following code into the "main.go" file: 329 330 ```go 331 package main 332 333 import ( 334 "flag" 335 "fmt" 336 "os" 337 "os/signal" 338 "path/filepath" 339 "syscall" 340 341 "github.com/dgraph-io/badger" 342 "github.com/pkg/errors" 343 "github.com/spf13/viper" 344 345 abci "github.com/tendermint/tendermint/abci/types" 346 cfg "github.com/tendermint/tendermint/config" 347 tmflags "github.com/tendermint/tendermint/libs/cli/flags" 348 "github.com/tendermint/tendermint/libs/log" 349 nm "github.com/tendermint/tendermint/node" 350 "github.com/tendermint/tendermint/p2p" 351 "github.com/tendermint/tendermint/privval" 352 "github.com/tendermint/tendermint/proxy" 353 ) 354 355 var configFile string 356 357 func init() { 358 flag.StringVar(&configFile, "config", "$HOME/.tendermint/config/config.toml", "Path to config.toml") 359 } 360 361 func main() { 362 db, err := badger.Open(badger.DefaultOptions("/tmp/badger")) 363 if err != nil { 364 fmt.Fprintf(os.Stderr, "failed to open badger db: %v", err) 365 os.Exit(1) 366 } 367 defer db.Close() 368 app := NewKVStoreApplication(db) 369 370 flag.Parse() 371 372 node, err := newTendermint(app, configFile) 373 if err != nil { 374 fmt.Fprintf(os.Stderr, "%v", err) 375 os.Exit(2) 376 } 377 378 node.Start() 379 defer func() { 380 node.Stop() 381 node.Wait() 382 }() 383 384 c := make(chan os.Signal, 1) 385 signal.Notify(c, os.Interrupt, syscall.SIGTERM) 386 <-c 387 os.Exit(0) 388 } 389 390 func newTendermint(app abci.Application, configFile string) (*nm.Node, error) { 391 // read config 392 config := cfg.DefaultConfig() 393 config.RootDir = filepath.Dir(filepath.Dir(configFile)) 394 viper.SetConfigFile(configFile) 395 if err := viper.ReadInConfig(); err != nil { 396 return nil, errors.Wrap(err, "viper failed to read config file") 397 } 398 if err := viper.Unmarshal(config); err != nil { 399 return nil, errors.Wrap(err, "viper failed to unmarshal config") 400 } 401 if err := config.ValidateBasic(); err != nil { 402 return nil, errors.Wrap(err, "config is invalid") 403 } 404 405 // create logger 406 logger := log.NewTMLogger(log.NewSyncWriter(os.Stdout)) 407 var err error 408 logger, err = tmflags.ParseLogLevel(config.LogLevel, logger, cfg.DefaultLogLevel()) 409 if err != nil { 410 return nil, errors.Wrap(err, "failed to parse log level") 411 } 412 413 // read private validator 414 pv := privval.LoadFilePV( 415 config.PrivValidatorKeyFile(), 416 config.PrivValidatorStateFile(), 417 ) 418 419 // read node key 420 nodeKey, err := p2p.LoadNodeKey(config.NodeKeyFile()) 421 if err != nil { 422 return nil, errors.Wrap(err, "failed to load node's key") 423 } 424 425 // create node 426 node, err := nm.NewNode( 427 config, 428 pv, 429 nodeKey, 430 proxy.NewLocalClientCreator(app), 431 nm.DefaultGenesisDocProviderFunc(config), 432 nm.DefaultDBProvider, 433 nm.DefaultMetricsProvider(config.Instrumentation), 434 logger) 435 if err != nil { 436 return nil, errors.Wrap(err, "failed to create new Tendermint node") 437 } 438 439 return node, nil 440 } 441 ``` 442 443 This is a huge blob of code, so let's break it down into pieces. 444 445 First, we initialize the Badger database and create an app instance: 446 447 ```go 448 db, err := badger.Open(badger.DefaultOptions("/tmp/badger")) 449 if err != nil { 450 fmt.Fprintf(os.Stderr, "failed to open badger db: %v", err) 451 os.Exit(1) 452 } 453 defer db.Close() 454 app := NewKVStoreApplication(db) 455 ``` 456 457 For **Windows** users, restarting this app will make badger throw an error as it requires value log to be truncated. For more information on this, visit [here](https://github.com/dgraph-io/badger/issues/744). 458 This can be avoided by setting the truncate option to true, like this: 459 460 ```go 461 db, err := badger.Open(badger.DefaultOptions("/tmp/badger").WithTruncate(true)) 462 ``` 463 464 Then we use it to create a Tendermint Core `Node` instance: 465 466 ```go 467 flag.Parse() 468 469 node, err := newTendermint(app, configFile) 470 if err != nil { 471 fmt.Fprintf(os.Stderr, "%v", err) 472 os.Exit(2) 473 } 474 475 ... 476 477 // create node 478 node, err := nm.NewNode( 479 config, 480 pv, 481 nodeKey, 482 proxy.NewLocalClientCreator(app), 483 nm.DefaultGenesisDocProviderFunc(config), 484 nm.DefaultDBProvider, 485 nm.DefaultMetricsProvider(config.Instrumentation), 486 logger) 487 if err != nil { 488 return nil, errors.Wrap(err, "failed to create new Tendermint node") 489 } 490 ``` 491 492 `NewNode` requires a few things including a configuration file, a private 493 validator, a node key and a few others in order to construct the full node. 494 495 Note we use `proxy.NewLocalClientCreator` here to create a local client instead 496 of one communicating through a socket or gRPC. 497 498 [viper](https://github.com/spf13/viper) is being used for reading the config, 499 which we will generate later using the `tendermint init` command. 500 501 ```go 502 config := cfg.DefaultConfig() 503 config.RootDir = filepath.Dir(filepath.Dir(configFile)) 504 viper.SetConfigFile(configFile) 505 if err := viper.ReadInConfig(); err != nil { 506 return nil, errors.Wrap(err, "viper failed to read config file") 507 } 508 if err := viper.Unmarshal(config); err != nil { 509 return nil, errors.Wrap(err, "viper failed to unmarshal config") 510 } 511 if err := config.ValidateBasic(); err != nil { 512 return nil, errors.Wrap(err, "config is invalid") 513 } 514 ``` 515 516 We use `FilePV`, which is a private validator (i.e. thing which signs consensus 517 messages). Normally, you would use `SignerRemote` to connect to an external 518 [HSM](https://kb.certus.one/hsm.html). 519 520 ```go 521 pv := privval.LoadFilePV( 522 config.PrivValidatorKeyFile(), 523 config.PrivValidatorStateFile(), 524 ) 525 526 ``` 527 528 `nodeKey` is needed to identify the node in a p2p network. 529 530 ```go 531 nodeKey, err := p2p.LoadNodeKey(config.NodeKeyFile()) 532 if err != nil { 533 return nil, errors.Wrap(err, "failed to load node's key") 534 } 535 ``` 536 537 As for the logger, we use the build-in library, which provides a nice 538 abstraction over [go-kit's 539 logger](https://github.com/go-kit/kit/tree/master/log). 540 541 ```go 542 logger := log.NewTMLogger(log.NewSyncWriter(os.Stdout)) 543 var err error 544 logger, err = tmflags.ParseLogLevel(config.LogLevel, logger, cfg.DefaultLogLevel()) 545 if err != nil { 546 return nil, errors.Wrap(err, "failed to parse log level") 547 } 548 ``` 549 550 Finally, we start the node and add some signal handling to gracefully stop it 551 upon receiving SIGTERM or Ctrl-C. 552 553 ```go 554 node.Start() 555 defer func() { 556 node.Stop() 557 node.Wait() 558 }() 559 560 c := make(chan os.Signal, 1) 561 signal.Notify(c, os.Interrupt, syscall.SIGTERM) 562 <-c 563 os.Exit(0) 564 ``` 565 566 ## 1.5 Getting Up and Running 567 568 We are going to use [Go modules](https://github.com/golang/go/wiki/Modules) for 569 dependency management. 570 571 ```sh 572 $ go mod init github.com/me/example 573 $ go build 574 ``` 575 576 This should build the binary. 577 578 To create a default configuration, nodeKey and private validator files, let's 579 execute `tendermint init`. But before we do that, we will need to install 580 Tendermint Core. Please refer to [the official 581 guide](https://docs.tendermint.com/master/introduction/install.html). If you're 582 installing from source, don't forget to checkout the latest release (`git checkout vX.Y.Z`). 583 584 ```sh 585 $ rm -rf /tmp/example 586 $ TMHOME="/tmp/example" tendermint init 587 588 I[2019-07-16|18:40:36.480] Generated private validator module=main keyFile=/tmp/example/config/priv_validator_key.json stateFile=/tmp/example2/data/priv_validator_state.json 589 I[2019-07-16|18:40:36.481] Generated node key module=main path=/tmp/example/config/node_key.json 590 I[2019-07-16|18:40:36.482] Generated genesis file module=main path=/tmp/example/config/genesis.json 591 ``` 592 593 We are ready to start our application: 594 595 ```sh 596 $ ./example -config "/tmp/example/config/config.toml" 597 598 badger 2019/07/16 18:42:25 INFO: All 0 tables opened in 0s 599 badger 2019/07/16 18:42:25 INFO: Replaying file id: 0 at offset: 0 600 badger 2019/07/16 18:42:25 INFO: Replay took: 695.227s 601 E[2019-07-16|18:42:25.818] Couldn't connect to any seeds module=p2p 602 I[2019-07-16|18:42:26.853] Executed block module=state height=1 validTxs=0 invalidTxs=0 603 I[2019-07-16|18:42:26.865] Committed state module=state height=1 txs=0 appHash= 604 ``` 605 606 Now open another tab in your terminal and try sending a transaction: 607 608 ```sh 609 $ curl -s 'localhost:26657/broadcast_tx_commit?tx="tendermint=rocks"' 610 { 611 "jsonrpc": "2.0", 612 "id": "", 613 "result": { 614 "check_tx": { 615 "gasWanted": "1" 616 }, 617 "deliver_tx": {}, 618 "hash": "1B3C5A1093DB952C331B1749A21DCCBB0F6C7F4E0055CD04D16346472FC60EC6", 619 "height": "128" 620 } 621 } 622 ``` 623 624 Response should contain the height where this transaction was committed. 625 626 Now let's check if the given key now exists and its value: 627 628 ``` 629 $ curl -s 'localhost:26657/abci_query?data="tendermint"' 630 { 631 "jsonrpc": "2.0", 632 "id": "", 633 "result": { 634 "response": { 635 "log": "exists", 636 "key": "dGVuZGVybWludA==", 637 "value": "cm9ja3M=" 638 } 639 } 640 } 641 ``` 642 643 "dGVuZGVybWludA==" and "cm9ja3M=" are the base64-encoding of the ASCII of 644 "tendermint" and "rocks" accordingly. 645 646 ## Outro 647 648 I hope everything went smoothly and your first, but hopefully not the last, 649 Tendermint Core application is up and running. If not, please [open an issue on 650 Github](https://github.com/tendermint/tendermint/issues/new/choose). To dig 651 deeper, read [the docs](https://docs.tendermint.com/master/).