github.com/noirx94/tendermintmp@v0.0.1/docs/tutorials/go-built-in.md (about) 1 <!--- 2 order: 2 3 ---> 4 5 # Creating a built-in application in Go 6 7 ## Guide assumptions 8 9 This guide is designed for beginners who want to get started with a Tendermint 10 Core application from scratch. It does not assume that you have any prior 11 experience with Tendermint Core. 12 13 Tendermint Core is Byzantine Fault Tolerant (BFT) middleware that takes a state 14 transition machine - written in any programming language - and securely 15 replicates it on many machines. 16 17 Although Tendermint Core is written in the Golang programming language, prior 18 knowledge of it is not required for this guide. You can learn it as we go due 19 to it's simplicity. However, you may want to go through [Learn X in Y minutes 20 Where X=Go](https://learnxinyminutes.com/docs/go/) first to familiarize 21 yourself with the syntax. 22 23 By following along with this guide, you'll create a Tendermint Core project 24 called kvstore, a (very) simple distributed BFT key-value store. 25 26 ## Built-in app vs external app 27 28 Running your application inside the same process as Tendermint Core will give 29 you the best possible performance. 30 31 For other languages, your application have to communicate with Tendermint Core 32 through a TCP, Unix domain socket or gRPC. 33 34 ## 1.1 Installing Go 35 36 Please refer to [the official guide for installing 37 Go](https://golang.org/doc/install). 38 39 Verify that you have the latest version of Go installed: 40 41 ```bash 42 $ go version 43 go version go1.13.1 darwin/amd64 44 ``` 45 46 Make sure you have `$GOPATH` environment variable set: 47 48 ```bash 49 $ echo $GOPATH 50 /Users/melekes/go 51 ``` 52 53 ## 1.2 Creating a new Go project 54 55 We'll start by creating a new Go project. 56 57 ```bash 58 mkdir kvstore 59 cd kvstore 60 ``` 61 62 Inside the example directory create a `main.go` file with the following content: 63 64 ```go 65 package main 66 67 import ( 68 "fmt" 69 ) 70 71 func main() { 72 fmt.Println("Hello, Tendermint Core") 73 } 74 ``` 75 76 When run, this should print "Hello, Tendermint Core" to the standard output. 77 78 ```bash 79 $ go run main.go 80 Hello, Tendermint Core 81 ``` 82 83 ## 1.3 Writing a Tendermint Core application 84 85 Tendermint Core communicates with the application through the Application 86 BlockChain Interface (ABCI). All message types are defined in the [protobuf 87 file](https://github.com/tendermint/tendermint/blob/master/proto/tendermint/abci/types.proto). 88 This allows Tendermint Core to run applications written in any programming 89 language. 90 91 Create a file called `app.go` with the following content: 92 93 ```go 94 package main 95 96 import ( 97 abcitypes "github.com/tendermint/tendermint/abci/types" 98 ) 99 100 type KVStoreApplication struct {} 101 102 var _ abcitypes.Application = (*KVStoreApplication)(nil) 103 104 func NewKVStoreApplication() *KVStoreApplication { 105 return &KVStoreApplication{} 106 } 107 108 func (KVStoreApplication) Info(req abcitypes.RequestInfo) abcitypes.ResponseInfo { 109 return abcitypes.ResponseInfo{} 110 } 111 112 func (KVStoreApplication) SetOption(req abcitypes.RequestSetOption) abcitypes.ResponseSetOption { 113 return abcitypes.ResponseSetOption{} 114 } 115 116 func (KVStoreApplication) DeliverTx(req abcitypes.RequestDeliverTx) abcitypes.ResponseDeliverTx { 117 return abcitypes.ResponseDeliverTx{Code: 0} 118 } 119 120 func (KVStoreApplication) CheckTx(req abcitypes.RequestCheckTx) abcitypes.ResponseCheckTx { 121 return abcitypes.ResponseCheckTx{Code: 0} 122 } 123 124 func (KVStoreApplication) Commit() abcitypes.ResponseCommit { 125 return abcitypes.ResponseCommit{} 126 } 127 128 func (KVStoreApplication) Query(req abcitypes.RequestQuery) abcitypes.ResponseQuery { 129 return abcitypes.ResponseQuery{Code: 0} 130 } 131 132 func (KVStoreApplication) InitChain(req abcitypes.RequestInitChain) abcitypes.ResponseInitChain { 133 return abcitypes.ResponseInitChain{} 134 } 135 136 func (KVStoreApplication) BeginBlock(req abcitypes.RequestBeginBlock) abcitypes.ResponseBeginBlock { 137 return abcitypes.ResponseBeginBlock{} 138 } 139 140 func (KVStoreApplication) EndBlock(req abcitypes.RequestEndBlock) abcitypes.ResponseEndBlock { 141 return abcitypes.ResponseEndBlock{} 142 } 143 144 func (KVStoreApplication) ListSnapshots(abcitypes.RequestListSnapshots) abcitypes.ResponseListSnapshots { 145 return abcitypes.ResponseListSnapshots{} 146 } 147 148 func (KVStoreApplication) OfferSnapshot(abcitypes.RequestOfferSnapshot) abcitypes.ResponseOfferSnapshot { 149 return abcitypes.ResponseOfferSnapshot{} 150 } 151 152 func (KVStoreApplication) LoadSnapshotChunk(abcitypes.RequestLoadSnapshotChunk) abcitypes.ResponseLoadSnapshotChunk { 153 return abcitypes.ResponseLoadSnapshotChunk{} 154 } 155 156 func (KVStoreApplication) ApplySnapshotChunk(abcitypes.RequestApplySnapshotChunk) abcitypes.ResponseApplySnapshotChunk { 157 return abcitypes.ResponseApplySnapshotChunk{} 158 } 159 ``` 160 161 Now I will go through each method explaining when it's called and adding 162 required business logic. 163 164 ### 1.3.1 CheckTx 165 166 When a new transaction is added to the Tendermint Core, it will ask the 167 application to check it (validate the format, signatures, etc.). 168 169 ```go 170 import "bytes" 171 172 func (app *KVStoreApplication) isValid(tx []byte) (code uint32) { 173 // check format 174 parts := bytes.Split(tx, []byte("=")) 175 if len(parts) != 2 { 176 return 1 177 } 178 179 key, value := parts[0], parts[1] 180 181 // check if the same key=value already exists 182 err := app.db.View(func(txn *badger.Txn) error { 183 item, err := txn.Get(key) 184 if err != nil && err != badger.ErrKeyNotFound { 185 return err 186 } 187 if err == nil { 188 return item.Value(func(val []byte) error { 189 if bytes.Equal(val, value) { 190 code = 2 191 } 192 return nil 193 }) 194 } 195 return nil 196 }) 197 if err != nil { 198 panic(err) 199 } 200 201 return code 202 } 203 204 func (app *KVStoreApplication) CheckTx(req abcitypes.RequestCheckTx) abcitypes.ResponseCheckTx { 205 code := app.isValid(req.Tx) 206 return abcitypes.ResponseCheckTx{Code: code, GasWanted: 1} 207 } 208 ``` 209 210 Don't worry if this does not compile yet. 211 212 If the transaction does not have a form of `{bytes}={bytes}`, we return `1` 213 code. When the same key=value already exist (same key and value), we return `2` 214 code. For others, we return a zero code indicating that they are valid. 215 216 Note that anything with non-zero code will be considered invalid (`-1`, `100`, 217 etc.) by Tendermint Core. 218 219 Valid transactions will eventually be committed given they are not too big and 220 have enough gas. To learn more about gas, check out ["the 221 specification"](https://docs.tendermint.com/master/spec/abci/apps.html#gas). 222 223 For the underlying key-value store we'll use 224 [badger](https://github.com/dgraph-io/badger), which is an embeddable, 225 persistent and fast key-value (KV) database. 226 227 ```go 228 import "github.com/dgraph-io/badger" 229 230 type KVStoreApplication struct { 231 db *badger.DB 232 currentBatch *badger.Txn 233 } 234 235 func NewKVStoreApplication(db *badger.DB) *KVStoreApplication { 236 return &KVStoreApplication{ 237 db: db, 238 } 239 } 240 ``` 241 242 ### 1.3.2 BeginBlock -> DeliverTx -> EndBlock -> Commit 243 244 When Tendermint Core has decided on the block, it's transfered to the 245 application in 3 parts: `BeginBlock`, one `DeliverTx` per transaction and 246 `EndBlock` in the end. DeliverTx are being transfered asynchronously, but the 247 responses are expected to come in order. 248 249 ```go 250 func (app *KVStoreApplication) BeginBlock(req abcitypes.RequestBeginBlock) abcitypes.ResponseBeginBlock { 251 app.currentBatch = app.db.NewTransaction(true) 252 return abcitypes.ResponseBeginBlock{} 253 } 254 255 ``` 256 257 Here we create a batch, which will store block's transactions. 258 259 ```go 260 func (app *KVStoreApplication) DeliverTx(req abcitypes.RequestDeliverTx) abcitypes.ResponseDeliverTx { 261 code := app.isValid(req.Tx) 262 if code != 0 { 263 return abcitypes.ResponseDeliverTx{Code: code} 264 } 265 266 parts := bytes.Split(req.Tx, []byte("=")) 267 key, value := parts[0], parts[1] 268 269 err := app.currentBatch.Set(key, value) 270 if err != nil { 271 panic(err) 272 } 273 274 return abcitypes.ResponseDeliverTx{Code: 0} 275 } 276 ``` 277 278 If the transaction is badly formatted or the same key=value already exist, we 279 again return the non-zero code. Otherwise, we add it to the current batch. 280 281 In the current design, a block can include incorrect transactions (those who 282 passed CheckTx, but failed DeliverTx or transactions included by the proposer 283 directly). This is done for performance reasons. 284 285 Note we can't commit transactions inside the `DeliverTx` because in such case 286 `Query`, which may be called in parallel, will return inconsistent data (i.e. 287 it will report that some value already exist even when the actual block was not 288 yet committed). 289 290 `Commit` instructs the application to persist the new state. 291 292 ```go 293 func (app *KVStoreApplication) Commit() abcitypes.ResponseCommit { 294 app.currentBatch.Commit() 295 return abcitypes.ResponseCommit{Data: []byte{}} 296 } 297 ``` 298 299 ### 1.3.3 Query 300 301 Now, when the client wants to know whenever a particular key/value exist, it 302 will call Tendermint Core RPC `/abci_query` endpoint, which in turn will call 303 the application's `Query` method. 304 305 Applications are free to provide their own APIs. But by using Tendermint Core 306 as a proxy, clients (including [light client 307 package](https://godoc.org/github.com/tendermint/tendermint/light)) can leverage 308 the unified API across different applications. Plus they won't have to call the 309 otherwise separate Tendermint Core API for additional proofs. 310 311 Note we don't include a proof here. 312 313 ```go 314 func (app *KVStoreApplication) Query(reqQuery abcitypes.RequestQuery) (resQuery abcitypes.ResponseQuery) { 315 resQuery.Key = reqQuery.Data 316 err := app.db.View(func(txn *badger.Txn) error { 317 item, err := txn.Get(reqQuery.Data) 318 if err != nil && err != badger.ErrKeyNotFound { 319 return err 320 } 321 if err == badger.ErrKeyNotFound { 322 resQuery.Log = "does not exist" 323 } else { 324 return item.Value(func(val []byte) error { 325 resQuery.Log = "exists" 326 resQuery.Value = val 327 return nil 328 }) 329 } 330 return nil 331 }) 332 if err != nil { 333 panic(err) 334 } 335 return 336 } 337 ``` 338 339 The complete specification can be found 340 [here](https://docs.tendermint.com/master/spec/abci/). 341 342 ## 1.4 Starting an application and a Tendermint Core instance in the same process 343 344 Put the following code into the "main.go" file: 345 346 ```go 347 package main 348 349 import ( 350 "flag" 351 "fmt" 352 "os" 353 "os/signal" 354 "path/filepath" 355 "syscall" 356 357 "github.com/dgraph-io/badger" 358 "github.com/spf13/viper" 359 360 abci "github.com/tendermint/tendermint/abci/types" 361 cfg "github.com/tendermint/tendermint/config" 362 tmflags "github.com/tendermint/tendermint/libs/cli/flags" 363 "github.com/tendermint/tendermint/libs/log" 364 nm "github.com/tendermint/tendermint/node" 365 "github.com/tendermint/tendermint/p2p" 366 "github.com/tendermint/tendermint/privval" 367 "github.com/tendermint/tendermint/proxy" 368 ) 369 370 var configFile string 371 372 func init() { 373 flag.StringVar(&configFile, "config", "$HOME/.tendermint/config/config.toml", "Path to config.toml") 374 } 375 376 func main() { 377 db, err := badger.Open(badger.DefaultOptions("/tmp/badger")) 378 if err != nil { 379 fmt.Fprintf(os.Stderr, "failed to open badger db: %v", err) 380 os.Exit(1) 381 } 382 defer db.Close() 383 app := NewKVStoreApplication(db) 384 385 flag.Parse() 386 387 node, err := newTendermint(app, configFile) 388 if err != nil { 389 fmt.Fprintf(os.Stderr, "%v", err) 390 os.Exit(2) 391 } 392 393 node.Start() 394 defer func() { 395 node.Stop() 396 node.Wait() 397 }() 398 399 c := make(chan os.Signal, 1) 400 signal.Notify(c, os.Interrupt, syscall.SIGTERM) 401 <-c 402 os.Exit(0) 403 } 404 405 func newTendermint(app abci.Application, configFile string) (*nm.Node, error) { 406 // read config 407 config := cfg.DefaultConfig() 408 config.RootDir = filepath.Dir(filepath.Dir(configFile)) 409 viper.SetConfigFile(configFile) 410 if err := viper.ReadInConfig(); err != nil { 411 return nil, fmt.Errorf("viper failed to read config file: %w", err) 412 } 413 if err := viper.Unmarshal(config); err != nil { 414 return nil, fmt.Errorf("viper failed to unmarshal config: %w", err) 415 } 416 if err := config.ValidateBasic(); err != nil { 417 return nil, fmt.Errorf("config is invalid: %w", err) 418 } 419 420 // create logger 421 logger := log.NewTMLogger(log.NewSyncWriter(os.Stdout)) 422 var err error 423 logger, err = tmflags.ParseLogLevel(config.LogLevel, logger, cfg.DefaultLogLevel) 424 if err != nil { 425 return nil, fmt.Errorf("failed to parse log level: %w", err) 426 } 427 428 // read private validator 429 pv := privval.LoadFilePV( 430 config.PrivValidatorKeyFile(), 431 config.PrivValidatorStateFile(), 432 ) 433 434 // read node key 435 nodeKey, err := p2p.LoadNodeKey(config.NodeKeyFile()) 436 if err != nil { 437 return nil, fmt.Errorf("failed to load node's key: %w", err) 438 } 439 440 // create node 441 node, err := nm.NewNode( 442 config, 443 pv, 444 nodeKey, 445 proxy.NewLocalClientCreator(app), 446 nm.DefaultGenesisDocProviderFunc(config), 447 nm.DefaultDBProvider, 448 nm.DefaultMetricsProvider(config.Instrumentation), 449 logger) 450 if err != nil { 451 return nil, fmt.Errorf("failed to create new Tendermint node: %w", err) 452 } 453 454 return node, nil 455 } 456 ``` 457 458 This is a huge blob of code, so let's break it down into pieces. 459 460 First, we initialize the Badger database and create an app instance: 461 462 ```go 463 db, err := badger.Open(badger.DefaultOptions("/tmp/badger")) 464 if err != nil { 465 fmt.Fprintf(os.Stderr, "failed to open badger db: %v", err) 466 os.Exit(1) 467 } 468 defer db.Close() 469 app := NewKVStoreApplication(db) 470 ``` 471 472 For **Windows** users, restarting this app will make badger throw an error as it requires value log to be truncated. For more information on this, visit [here](https://github.com/dgraph-io/badger/issues/744). 473 This can be avoided by setting the truncate option to true, like this: 474 475 ```go 476 db, err := badger.Open(badger.DefaultOptions("/tmp/badger").WithTruncate(true)) 477 ``` 478 479 Then we use it to create a Tendermint Core `Node` instance: 480 481 ```go 482 flag.Parse() 483 484 node, err := newTendermint(app, configFile) 485 if err != nil { 486 fmt.Fprintf(os.Stderr, "%v", err) 487 os.Exit(2) 488 } 489 490 ... 491 492 // create node 493 node, err := nm.NewNode( 494 config, 495 pv, 496 nodeKey, 497 proxy.NewLocalClientCreator(app), 498 nm.DefaultGenesisDocProviderFunc(config), 499 nm.DefaultDBProvider, 500 nm.DefaultMetricsProvider(config.Instrumentation), 501 logger) 502 if err != nil { 503 return nil, fmt.Errorf("failed to create new Tendermint node: %w", err) 504 } 505 ``` 506 507 `NewNode` requires a few things including a configuration file, a private 508 validator, a node key and a few others in order to construct the full node. 509 510 Note we use `proxy.NewLocalClientCreator` here to create a local client instead 511 of one communicating through a socket or gRPC. 512 513 [viper](https://github.com/spf13/viper) is being used for reading the config, 514 which we will generate later using the `tendermint init` command. 515 516 ```go 517 config := cfg.DefaultConfig() 518 config.RootDir = filepath.Dir(filepath.Dir(configFile)) 519 viper.SetConfigFile(configFile) 520 if err := viper.ReadInConfig(); err != nil { 521 return nil, fmt.Errorf("viper failed to read config file: %w", err) 522 } 523 if err := viper.Unmarshal(config); err != nil { 524 return nil, fmt.Errorf("viper failed to unmarshal config: %w", err) 525 } 526 if err := config.ValidateBasic(); err != nil { 527 return nil, fmt.Errorf("config is invalid: %w", err) 528 } 529 ``` 530 531 We use `FilePV`, which is a private validator (i.e. thing which signs consensus 532 messages). Normally, you would use `SignerRemote` to connect to an external 533 [HSM](https://kb.certus.one/hsm.html). 534 535 ```go 536 pv := privval.LoadFilePV( 537 config.PrivValidatorKeyFile(), 538 config.PrivValidatorStateFile(), 539 ) 540 541 ``` 542 543 `nodeKey` is needed to identify the node in a p2p network. 544 545 ```go 546 nodeKey, err := p2p.LoadNodeKey(config.NodeKeyFile()) 547 if err != nil { 548 return nil, fmt.Errorf("failed to load node's key: %w", err) 549 } 550 ``` 551 552 As for the logger, we use the build-in library, which provides a nice 553 abstraction over [go-kit's 554 logger](https://github.com/go-kit/kit/tree/master/log). 555 556 ```go 557 logger := log.NewTMLogger(log.NewSyncWriter(os.Stdout)) 558 var err error 559 logger, err = tmflags.ParseLogLevel(config.LogLevel, logger, cfg.DefaultLogLevel()) 560 if err != nil { 561 return nil, fmt.Errorf("failed to parse log level: %w", err) 562 } 563 ``` 564 565 Finally, we start the node and add some signal handling to gracefully stop it 566 upon receiving SIGTERM or Ctrl-C. 567 568 ```go 569 node.Start() 570 defer func() { 571 node.Stop() 572 node.Wait() 573 }() 574 575 c := make(chan os.Signal, 1) 576 signal.Notify(c, os.Interrupt, syscall.SIGTERM) 577 <-c 578 os.Exit(0) 579 ``` 580 581 ## 1.5 Getting Up and Running 582 583 We are going to use [Go modules](https://github.com/golang/go/wiki/Modules) for 584 dependency management. 585 586 ```bash 587 go mod init github.com/me/example 588 go get github.com/tendermint/tendermint/@v0.34.0 589 ``` 590 591 After running the above commands you will see two generated files, go.mod and go.sum. The go.mod file should look similar to: 592 593 ```go 594 module github.com/me/example 595 596 go 1.15 597 598 require ( 599 github.com/dgraph-io/badger v1.6.2 600 github.com/tendermint/tendermint v0.34.0 601 ) 602 ``` 603 604 Finally, we will build our binary: 605 606 ```sh 607 go build 608 ``` 609 610 To create a default configuration, nodeKey and private validator files, let's 611 execute `tendermint init`. But before we do that, we will need to install 612 Tendermint Core. Please refer to [the official 613 guide](https://docs.tendermint.com/master/introduction/install.html). If you're 614 installing from source, don't forget to checkout the latest release (`git checkout vX.Y.Z`). 615 616 ```bash 617 $ rm -rf /tmp/example 618 $ TMHOME="/tmp/example" tendermint init 619 620 I[2019-07-16|18:40:36.480] Generated private validator module=main keyFile=/tmp/example/config/priv_validator_key.json stateFile=/tmp/example2/data/priv_validator_state.json 621 I[2019-07-16|18:40:36.481] Generated node key module=main path=/tmp/example/config/node_key.json 622 I[2019-07-16|18:40:36.482] Generated genesis file module=main path=/tmp/example/config/genesis.json 623 ``` 624 625 We are ready to start our application: 626 627 ```bash 628 $ ./example -config "/tmp/example/config/config.toml" 629 630 badger 2019/07/16 18:42:25 INFO: All 0 tables opened in 0s 631 badger 2019/07/16 18:42:25 INFO: Replaying file id: 0 at offset: 0 632 badger 2019/07/16 18:42:25 INFO: Replay took: 695.227s 633 E[2019-07-16|18:42:25.818] Couldn't connect to any seeds module=p2p 634 I[2019-07-16|18:42:26.853] Executed block module=state height=1 validTxs=0 invalidTxs=0 635 I[2019-07-16|18:42:26.865] Committed state module=state height=1 txs=0 appHash= 636 ``` 637 638 Now open another tab in your terminal and try sending a transaction: 639 640 ```bash 641 $ curl -s 'localhost:26657/broadcast_tx_commit?tx="tendermint=rocks"' 642 { 643 "jsonrpc": "2.0", 644 "id": "", 645 "result": { 646 "check_tx": { 647 "gasWanted": "1" 648 }, 649 "deliver_tx": {}, 650 "hash": "1B3C5A1093DB952C331B1749A21DCCBB0F6C7F4E0055CD04D16346472FC60EC6", 651 "height": "128" 652 } 653 } 654 ``` 655 656 Response should contain the height where this transaction was committed. 657 658 Now let's check if the given key now exists and its value: 659 660 ```json 661 $ curl -s 'localhost:26657/abci_query?data="tendermint"' 662 { 663 "jsonrpc": "2.0", 664 "id": "", 665 "result": { 666 "response": { 667 "log": "exists", 668 "key": "dGVuZGVybWludA==", 669 "value": "cm9ja3M=" 670 } 671 } 672 } 673 ``` 674 675 "dGVuZGVybWludA==" and "cm9ja3M=" are the base64-encoding of the ASCII of 676 "tendermint" and "rocks" accordingly. 677 678 ## Outro 679 680 I hope everything went smoothly and your first, but hopefully not the last, 681 Tendermint Core application is up and running. If not, please [open an issue on 682 Github](https://github.com/tendermint/tendermint/issues/new/choose). To dig 683 deeper, read [the docs](https://docs.tendermint.com/master/).