github.com/kula/etcd@v0.2.1-0.20131226070625-e96234382ac0/README.md (about) 1 # etcd 2 3 README version 0.2.0 4 5 [![Build Status](https://travis-ci.org/coreos/etcd.png)](https://travis-ci.org/coreos/etcd) 6 7 A highly-available key value store for shared configuration and service discovery. 8 etcd is inspired by zookeeper and doozer, with a focus on: 9 10 * Simple: curl'able user facing API (HTTP+JSON) 11 * Secure: optional SSL client cert authentication 12 * Fast: benchmarked 1000s of writes/s per instance 13 * Reliable: Properly distributed using Raft 14 15 Etcd is written in Go and uses the [Raft][raft] consensus algorithm to manage a highly-available replicated log. 16 17 See [etcdctl][etcdctl] for a simple command line client. 18 Or feel free to just use curl, as in the examples below. 19 20 [raft]: https://github.com/coreos/go-raft 21 [etcdctl]: http://github.com/coreos/etcdctl/ 22 23 ## Contact 24 25 - Mailing list: http://coreos.com/lists/etcd-dev/ 26 - IRC: #coreos on irc.freenode.net 27 - Planning/Roadmap: https://trello.com/b/OiEbU547/etcd 28 - Bugs: https://github.com/coreos/etcd/issues 29 30 31 ## Getting Started 32 33 ### Getting etcd 34 35 The latest release and setup instructions are available at [Github][github-release]. 36 37 [github-release]: https://github.com/coreos/etcd/releases/ 38 39 40 ### Building 41 42 You can build etcd from source: 43 44 ```sh 45 git clone https://github.com/coreos/etcd 46 cd etcd 47 ./build 48 ``` 49 50 This will generate a binary in the base directory called `./etcd`. 51 52 _NOTE_: you need go 1.1+. Please check your installation with 53 54 ``` 55 go version 56 ``` 57 58 59 ### Running a single machine 60 61 These examples will use a single machine cluster to show you the basics of the etcd REST API. 62 Let's start etcd: 63 64 ```sh 65 ./etcd -data-dir machine0 -name machine0 66 ``` 67 68 This will bring up etcd listening on port 4001 for client communication and on port 7001 for server-to-server communication. 69 The `-data-dir machine0` argument tells etcd to write machine configuration, logs and snapshots to the `./machine0/` directory. 70 The `-name machine` tells the rest of the cluster that this machine is named machine0. 71 72 73 74 ## Usage 75 76 ### Setting the value to a key 77 78 Let’s set the first key-value pair to the datastore. 79 In this case the key is `/message` and the value is `Hello world`. 80 81 ```sh 82 curl -L http://127.0.0.1:4001/v2/keys/message -X PUT -d value="Hello world" 83 ``` 84 85 ```json 86 { 87 "action": "set", 88 "node": { 89 "createdIndex": 2, 90 "key": "/message", 91 "modifiedIndex": 2, 92 "value": "Hello world" 93 } 94 } 95 ``` 96 97 This response contains four fields. 98 We will introduce three more fields as we try more commands. 99 100 1. The action of the request; we set the value via a `PUT` request, thus the action is `set`. 101 102 2. The key of the request; we set `/message` to `Hello world`, so the key field is `/message`. 103 We use a file system like structure to represent the key-value pairs so each key starts with `/`. 104 105 3. The current value of the key; we set the value to`Hello world`. 106 107 4. Modified Index is a unique, monotonically incrementing index created for each change to etcd. 108 Requests that change the index include `set`, `delete`, `update`, `create` and `compareAndSwap`. 109 Since the `get` and `watch` commands do not change state in the store, they do not change the index. 110 You may notice that in this example the index is `2` even though it is the first request you sent to the server. 111 This is because there are internal commands that also change the state like adding and syncing servers. 112 113 ### Response Headers 114 115 etcd includes a few HTTP headers that provide global information about the etcd cluster that serviced a request: 116 117 ``` 118 X-Etcd-Index: 35 119 X-Raft-Index: 5398 120 X-Raft-Term: 0 121 ``` 122 123 - `X-Etcd-Index` is the current etcd index as explained above. 124 - `X-Raft-Index` is similar to the etcd index but is for the underlying raft protocol 125 - `X-Raft-Term` this number will increase when an etcd master election happens. If this number is increasing rapdily you may need to tune the election timeout. See the [tuning][tuning] section for details. 126 127 [tuning]: #tuning 128 129 ### Get the value of a key 130 131 We can get the value that we just set in `/message` by issuing a `GET` request: 132 133 ```sh 134 curl -L http://127.0.0.1:4001/v2/keys/message 135 ``` 136 137 ```json 138 { 139 "action": "get", 140 "node": { 141 "createdIndex": 2, 142 "key": "/message", 143 "modifiedIndex": 2, 144 "value": "Hello world" 145 } 146 } 147 ``` 148 149 150 ### Changing the value of a key 151 152 You can change the value of `/message` from `Hello world` to `Hello etcd` with another `PUT` request to the key: 153 154 ```sh 155 curl -L http://127.0.0.1:4001/v2/keys/message -XPUT -d value="Hello etcd" 156 ``` 157 158 ```json 159 { 160 "action": "set", 161 "node": { 162 "createdIndex": 3, 163 "key": "/message", 164 "modifiedIndex": 3, 165 "prevValue": "Hello world", 166 "value": "Hello etcd" 167 } 168 } 169 ``` 170 171 Notice that `node.prevValue` is set to the previous value of the key - `Hello world`. 172 It is useful when you want to atomically set a value to a key and get its old value. 173 174 175 ### Deleting a key 176 177 You can remove the `/message` key with a `DELETE` request: 178 179 ```sh 180 curl -L http://127.0.0.1:4001/v2/keys/message -XDELETE 181 ``` 182 183 ```json 184 { 185 "action": "delete", 186 "node": { 187 "createdIndex": 3, 188 "key": "/message", 189 "modifiedIndex": 4, 190 "prevValue": "Hello etcd" 191 } 192 } 193 ``` 194 195 196 ### Using key TTL 197 198 Keys in etcd can be set to expire after a specified number of seconds. 199 You can do this by setting a TTL (time to live) on the key when send a `PUT` request: 200 201 ```sh 202 curl -L http://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar -d ttl=5 203 ``` 204 205 ```json 206 { 207 "action": "set", 208 "node": { 209 "createdIndex": 5, 210 "expiration": "2013-12-04T12:01:21.874888581-08:00", 211 "key": "/foo", 212 "modifiedIndex": 5, 213 "ttl": 5, 214 "value": "bar" 215 } 216 } 217 ``` 218 219 Note the two new fields in response: 220 221 1. The `expiration` is the time that this key will expire and be deleted. 222 223 2. The `ttl` is the time to live for the key, in seconds. 224 225 _NOTE_: Keys can only be expired by a cluster leader so if a machine gets disconnected from the cluster, its keys will not expire until it rejoins. 226 227 Now you can try to get the key by sending a `GET` request: 228 229 ```sh 230 curl -L http://127.0.0.1:4001/v2/keys/foo 231 ``` 232 233 If the TTL has expired, the key will be deleted, and you will be returned a 100. 234 235 ```json 236 { 237 "cause": "/foo", 238 "errorCode": 100, 239 "index": 6, 240 "message": "Key Not Found" 241 } 242 ``` 243 244 ### Waiting for a change 245 246 We can watch for a change on a key and receive a notification by using long polling. 247 This also works for child keys by passing `recursive=true` in curl. 248 249 In one terminal, we send a get request with `wait=true` : 250 251 ```sh 252 curl -L http://127.0.0.1:4001/v2/keys/foo?wait=true 253 ``` 254 255 Now we are waiting for any changes at path `/foo`. 256 257 In another terminal, we set a key `/foo` with value `bar`: 258 259 ```sh 260 curl -L http://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar 261 ``` 262 263 The first terminal should get the notification and return with the same response as the set request. 264 265 ```json 266 { 267 "action": "set", 268 "node": { 269 "createdIndex": 7, 270 "key": "/foo", 271 "modifiedIndex": 7, 272 "value": "bar" 273 } 274 } 275 ``` 276 277 However, the watch command can do more than this. 278 Using the the index we can watch for commands that has happened in the past. 279 This is useful for ensuring you don't miss events between watch commands. 280 281 Let's try to watch for the set command of index 7 again: 282 283 ```sh 284 curl -L http://127.0.0.1:4001/v2/keys/foo?wait=true\&waitIndex=7 285 ``` 286 287 The watch command returns immediately with the same response as previous. 288 289 290 ### Atomically Creating In-Order Keys 291 292 Using the `POST` on a directory you can create keys with key names that are created in-order. 293 This can be used in a variety of useful patterns like implementing queues of keys that need to be processed in strict order. 294 An example use case is the [locking module][lockmod] which uses it to ensure clients get fair access to a mutex. 295 296 Creating an in-order key is easy 297 298 ```sh 299 curl -X POST http://127.0.0.1:4001/v2/keys/queue -d value=Job1 300 ``` 301 302 ```json 303 { 304 "action": "create", 305 "node": { 306 "createdIndex": 6, 307 "key": "/queue/6", 308 "modifiedIndex": 6, 309 "value": "Job1" 310 } 311 } 312 ``` 313 314 If you create another entry some time later it is guaranteed to have a key name that is greater than the previous key. 315 Also note the key names use the global etcd index so the next key can be more than `previous + 1`. 316 317 ```sh 318 curl -X POST http://127.0.0.1:4001/v2/keys/queue -d value=Job2 319 ``` 320 321 ```json 322 { 323 "action": "create", 324 "node": { 325 "createdIndex": 29, 326 "key": "/queue/29", 327 "modifiedIndex": 29, 328 "value": "Job2" 329 } 330 } 331 ``` 332 333 [lockmod]: #lock 334 335 336 ### Using a directory TTL 337 338 Like keys, directories in etcd can be set to expire after a specified number of seconds. 339 You can do this by setting a TTL (time to live) on a directory when it is created with a `PUT`: 340 341 ```sh 342 curl -L http://127.0.0.1:4001/v2/keys/dir -XPUT -d ttl=30 -d dir=true 343 ``` 344 345 ```json 346 { 347 "action": "set", 348 "node": { 349 "createdIndex": 17, 350 "dir": true, 351 "expiration": "2013-12-11T10:37:33.689275857-08:00", 352 "key": "/newdir", 353 "modifiedIndex": 17, 354 "ttl": 30 355 } 356 } 357 ``` 358 359 The directories TTL can be refreshed by making an update. 360 You can do this by making a PUT with `prevExist=true` and a new TTL. 361 362 ```sh 363 curl -L http://127.0.0.1:4001/v2/keys/dir -XPUT -d ttl=30 -d dir=true -d prevExist=true 364 ``` 365 366 Keys that are under this directory work as usual, but when the directory expires a watcher on a key under the directory will get an expire event: 367 368 ```sh 369 curl -X GET http://127.0.0.1:4001/v2/keys/dir/asdf\?consistent\=true\&wait\=true 370 ``` 371 372 ```json 373 { 374 "action": "expire", 375 "node": { 376 "createdIndex": 8, 377 "key": "/dir", 378 "modifiedIndex": 15 379 } 380 } 381 ``` 382 383 384 ### Atomic Compare-and-Swap (CAS) 385 386 Etcd can be used as a centralized coordination service in a cluster and `CompareAndSwap` is the most basic operation to build distributed lock service. 387 388 This command will set the value of a key only if the client-provided conditions are equal to the current conditions. 389 390 The current comparable conditions are: 391 392 1. `prevValue` - checks the previous value of the key. 393 394 2. `prevIndex` - checks the previous index of the key. 395 396 3. `prevExist` - checks existence of the key: if `prevExist` is true, it is a `update` request; if prevExist is `false`, it is a `create` request. 397 398 Here is a simple example. 399 Let's create a key-value pair first: `foo=one`. 400 401 ```sh 402 curl -L http://127.0.0.1:4001/v2/keys/foo -XPUT -d value=one 403 ``` 404 405 Let's try some invalid `CompareAndSwap` commands first. 406 407 Trying to set this existing key with `prevExist=false` fails as expected: 408 ```sh 409 curl -L http://127.0.0.1:4001/v2/keys/foo?prevExist=false -XPUT -d value=three 410 ``` 411 412 The error code explains the problem: 413 414 ```json 415 { 416 "cause": "/foo", 417 "errorCode": 105, 418 "index": 39776, 419 "message": "Already exists" 420 } 421 ``` 422 423 Now lets provide a `prevValue` parameter: 424 425 ```sh 426 curl -L http://127.0.0.1:4001/v2/keys/foo?prevValue=two -XPUT -d value=three 427 ``` 428 429 This will try to compare the previous value of the key and the previous value we provided. If they are equal, the value of the key will change to three. 430 431 ```json 432 { 433 "cause": "[two != one] [0 != 8]", 434 "errorCode": 101, 435 "index": 8, 436 "message": "Test Failed" 437 } 438 ``` 439 440 which means `CompareAndSwap` failed. 441 442 Let's try a valid condition: 443 444 ```sh 445 curl -L http://127.0.0.1:4001/v2/keys/foo?prevValue=one -XPUT -d value=two 446 ``` 447 448 The response should be 449 450 ```json 451 { 452 "action": "compareAndSwap", 453 "node": { 454 "createdIndex": 8, 455 "key": "/foo", 456 "modifiedIndex": 9, 457 "prevValue": "one", 458 "value": "two" 459 } 460 } 461 ``` 462 463 We successfully changed the value from "one" to "two" since we gave the correct previous value. 464 465 ### Creating Directories 466 467 In most cases directories for a key are automatically created. 468 But, there are cases where you will want to create a directory or remove one. 469 470 Creating a directory is just like a key only you cannot provide a value and must add the `dir=true` parameter. 471 472 ```sh 473 curl -L http://127.0.0.1:4001/v2/keys/dir -XPUT -d dir=true 474 ``` 475 ```json 476 { 477 "action": "set", 478 "node": { 479 "createdIndex": 30, 480 "dir": true, 481 "key": "/dir", 482 "modifiedIndex": 30 483 } 484 } 485 ``` 486 487 ### Listing a directory 488 489 In etcd we can store two types of things: keys and directories. 490 Keys store a single string value. 491 Directories store a set of keys and/or other directories. 492 493 In this example, let's first create some keys: 494 495 We already have `/foo=two` so now we'll create another one called `/foo_dir/foo` with the value of `bar`: 496 497 ```sh 498 curl -L http://127.0.0.1:4001/v2/keys/foo_dir/foo -XPUT -d value=bar 499 ``` 500 501 ```json 502 { 503 "action": "set", 504 "node": { 505 "createdIndex": 2, 506 "key": "/foo_dir/foo", 507 "modifiedIndex": 2, 508 "value": "bar" 509 } 510 } 511 ``` 512 513 Now we can list the keys under root `/`: 514 515 ```sh 516 curl -L http://127.0.0.1:4001/v2/keys/ 517 ``` 518 519 We should see the response as an array of items: 520 521 ```json 522 { 523 "action": "get", 524 "node": { 525 "dir": true, 526 "key": "/", 527 "nodes": [ 528 { 529 "createdIndex": 2, 530 "dir": true, 531 "key": "/foo_dir", 532 "modifiedIndex": 2 533 } 534 ] 535 } 536 } 537 ``` 538 539 Here we can see `/foo` is a key-value pair under `/` and `/foo_dir` is a directory. 540 We can also recursively get all the contents under a directory by adding `recursive=true`. 541 542 ```sh 543 curl -L http://127.0.0.1:4001/v2/keys/?recursive=true 544 ``` 545 546 ```json 547 { 548 "action": "get", 549 "node": { 550 "dir": true, 551 "key": "/", 552 "nodes": [ 553 { 554 "createdIndex": 2, 555 "dir": true, 556 "key": "/foo_dir", 557 "modifiedIndex": 2, 558 "nodes": [ 559 { 560 "createdIndex": 2, 561 "key": "/foo_dir/foo", 562 "modifiedIndex": 2, 563 "value": "bar" 564 } 565 ] 566 } 567 ] 568 } 569 } 570 ``` 571 572 573 ### Deleting a Directory 574 575 Now let's try to delete the directory `/foo_dir`. 576 577 You can remove an empty directory using the `DELETE` verb and the `dir=true` parameter. 578 579 ```sh 580 curl -L -X DELETE 'http://127.0.0.1:4001/v2/keys/dir?dir=true' 581 ``` 582 ```json 583 { 584 "action": "delete", 585 "node": { 586 "createdIndex": 30, 587 "dir": true, 588 "key": "/dir", 589 "modifiedIndex": 31 590 } 591 } 592 ``` 593 594 To delete a directory that holds keys, you must add `recursive=true`. 595 596 ```sh 597 curl -L http://127.0.0.1:4001/v2/keys/dir?recursive=true -XDELETE 598 ``` 599 600 ```json 601 { 602 "action": "delete", 603 "node": { 604 "createdIndex": 10, 605 "dir": true, 606 "key": "/dir", 607 "modifiedIndex": 11 608 } 609 } 610 ``` 611 612 613 ### Creating a hidden node 614 615 We can create a hidden key-value pair or directory by add a `_` prefix. 616 The hidden item will not be listed when sending a `GET` request for a directory. 617 618 First we'll add a hidden key named `/_message`: 619 620 ```sh 621 curl -L http://127.0.0.1:4001/v2/keys/_message -XPUT -d value="Hello hidden world" 622 ``` 623 624 ```json 625 { 626 "action": "set", 627 "node": { 628 "createdIndex": 3, 629 "key": "/_message", 630 "modifiedIndex": 3, 631 "value": "Hello hidden world" 632 } 633 } 634 ``` 635 636 637 Next we'll add a regular key named `/message`: 638 639 ```sh 640 curl -L http://127.0.0.1:4001/v2/keys/message -XPUT -d value="Hello world" 641 ``` 642 643 ```json 644 { 645 "action": "set", 646 "node": { 647 "createdIndex": 4, 648 "key": "/message", 649 "modifiedIndex": 4, 650 "value": "Hello world" 651 } 652 } 653 ``` 654 655 Now let's try to get a listing of keys under the root directory, `/`: 656 657 ```sh 658 curl -L http://127.0.0.1:4001/v2/keys/ 659 ``` 660 661 ```json 662 { 663 "action": "get", 664 "node": { 665 "dir": true, 666 "key": "/", 667 "nodes": [ 668 { 669 "createdIndex": 2, 670 "dir": true, 671 "key": "/foo_dir", 672 "modifiedIndex": 2 673 }, 674 { 675 "createdIndex": 4, 676 "key": "/message", 677 "modifiedIndex": 4, 678 "value": "Hello world" 679 } 680 ] 681 } 682 } 683 ``` 684 685 Here we see the `/message` key but our hidden `/_message` key is not returned. 686 687 ## Advanced Usage 688 689 ### Transport security with HTTPS 690 691 Etcd supports SSL/TLS and client cert authentication for clients to server, as well as server to server communication. 692 693 First, you need to have a CA cert `clientCA.crt` and signed key pair `client.crt`, `client.key`. 694 This site has a good reference for how to generate self-signed key pairs: 695 http://www.g-loaded.eu/2005/11/10/be-your-own-ca/ 696 697 For testing you can use the certificates in the `fixtures/ca` directory. 698 699 Let's configure etcd to use this keypair: 700 701 ```sh 702 ./etcd -f -name machine0 -data-dir machine0 -cert-file=./fixtures/ca/server.crt -key-file=./fixtures/ca/server.key.insecure 703 ``` 704 705 There are a few new options we're using: 706 707 * `-f` - forces a new machine configuration, even if an existing configuration is found. (WARNING: data loss!) 708 * `-cert-file` and `-key-file` specify the location of the cert and key files to be used for for transport layer security between the client and server. 709 710 You can now test the configuration using HTTPS: 711 712 ```sh 713 curl --cacert ./fixtures/ca/server-chain.pem https://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar -v 714 ``` 715 716 You should be able to see the handshake succeed. 717 718 **OSX 10.9+ Users**: curl 7.30.0 on OSX 10.9+ doesn't understand certificates passed in on the command line. 719 Instead you must import the dummy ca.crt directly into the keychain or add the `-k` flag to curl to ignore errors. 720 If you want to test without the `-k` flag run `open ./fixtures/ca/ca.crt` and follow the prompts. 721 Please remove this certificate after you are done testing! 722 If you know of a workaround let us know. 723 724 ``` 725 ... 726 SSLv3, TLS handshake, Finished (20): 727 ... 728 ``` 729 730 And also the response from the etcd server: 731 732 ```json 733 { 734 "action": "set", 735 "key": "/foo", 736 "modifiedIndex": 3, 737 "prevValue": "bar", 738 "value": "bar" 739 } 740 ``` 741 742 743 ### Authentication with HTTPS client certificates 744 745 We can also do authentication using CA certs. 746 The clients will provide their cert to the server and the server will check whether the cert is signed by the CA and decide whether to serve the request. 747 748 ```sh 749 ./etcd -f -name machine0 -data-dir machine0 -ca-file=./fixtures/ca/ca.crt -cert-file=./fixtures/ca/server.crt -key-file=./fixtures/ca/server.key.insecure 750 ``` 751 752 ```-ca-file``` is the path to the CA cert. 753 754 Try the same request to this server: 755 756 ```sh 757 curl --cacert ./fixtures/ca/server-chain.pem https://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar -v 758 ``` 759 760 The request should be rejected by the server. 761 762 ``` 763 ... 764 routines:SSL3_READ_BYTES:sslv3 alert bad certificate 765 ... 766 ``` 767 768 We need to give the CA signed cert to the server. 769 770 ```sh 771 curl --key ./fixtures/ca/server2.key.insecure --cert ./fixtures/ca/server2.crt --cacert ./fixtures/ca/server-chain.pem -L https://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar -v 772 ``` 773 774 You should able to see: 775 776 ``` 777 ... 778 SSLv3, TLS handshake, CERT verify (15): 779 ... 780 TLS handshake, Finished (20) 781 ``` 782 783 And also the response from the server: 784 785 ```json 786 { 787 "action": "set", 788 "node": { 789 "createdIndex": 12, 790 "key": "/foo", 791 "modifiedIndex": 12, 792 "prevValue": "two", 793 "value": "bar" 794 } 795 } 796 ``` 797 798 799 ## Clustering 800 801 ### Example cluster of three machines 802 803 Let's explore the use of etcd clustering. 804 We use Raft as the underlying distributed protocol which provides consistency and persistence of the data across all of the etcd instances. 805 806 Let start by creating 3 new etcd instances. 807 808 We use `-peer-addr` to specify server port and `-addr` to specify client port and `-data-dir` to specify the directory to store the log and info of the machine in the cluster: 809 810 ```sh 811 ./etcd -peer-addr 127.0.0.1:7001 -addr 127.0.0.1:4001 -data-dir machines/machine1 -name machine1 812 ``` 813 814 **Note:** If you want to run etcd on an external IP address and still have access locally, you'll need to add `-bind-addr 0.0.0.0` so that it will listen on both external and localhost addresses. 815 A similar argument `-peer-bind-addr` is used to setup the listening address for the server port. 816 817 Let's join two more machines to this cluster using the `-peers` argument: 818 819 ```sh 820 ./etcd -peer-addr 127.0.0.1:7002 -addr 127.0.0.1:4002 -peers 127.0.0.1:7001 -data-dir machines/machine2 -name machine2 821 ./etcd -peer-addr 127.0.0.1:7003 -addr 127.0.0.1:4003 -peers 127.0.0.1:7001 -data-dir machines/machine3 -name machine3 822 ``` 823 824 We can retrieve a list of machines in the cluster using the HTTP API: 825 826 ```sh 827 curl -L http://127.0.0.1:4001/v2/machines 828 ``` 829 830 We should see there are three machines in the cluster 831 832 ``` 833 http://127.0.0.1:4001, http://127.0.0.1:4002, http://127.0.0.1:4003 834 ``` 835 836 The machine list is also available via the main key API: 837 838 ```sh 839 curl -L http://127.0.0.1:4001/v2/keys/_etcd/machines 840 ``` 841 842 ```json 843 { 844 "action": "get", 845 "node": { 846 "createdIndex": 1, 847 "dir": true, 848 "key": "/_etcd/machines", 849 "modifiedIndex": 1, 850 "nodes": [ 851 { 852 "createdIndex": 1, 853 "key": "/_etcd/machines/machine1", 854 "modifiedIndex": 1, 855 "value": "raft=http://127.0.0.1:7001&etcd=http://127.0.0.1:4001" 856 }, 857 { 858 "createdIndex": 2, 859 "key": "/_etcd/machines/machine2", 860 "modifiedIndex": 2, 861 "value": "raft=http://127.0.0.1:7002&etcd=http://127.0.0.1:4002" 862 }, 863 { 864 "createdIndex": 3, 865 "key": "/_etcd/machines/machine3", 866 "modifiedIndex": 3, 867 "value": "raft=http://127.0.0.1:7003&etcd=http://127.0.0.1:4003" 868 } 869 ] 870 } 871 } 872 ``` 873 874 We can also get the current leader in the cluster: 875 876 ``` 877 curl -L http://127.0.0.1:4001/v2/leader 878 ``` 879 880 The first server we set up should still be the leader unless it has died during these commands. 881 882 ``` 883 http://127.0.0.1:7001 884 ``` 885 886 Now we can do normal SET and GET operations on keys as we explored earlier. 887 888 ```sh 889 curl -L http://127.0.0.1:4001/v2/keys/foo -XPUT -d value=bar 890 ``` 891 892 ```json 893 { 894 "action": "set", 895 "node": { 896 "createdIndex": 4, 897 "key": "/foo", 898 "modifiedIndex": 4, 899 "value": "bar" 900 } 901 } 902 ``` 903 904 905 ### Killing Nodes in the Cluster 906 907 Now if we kill the leader of the cluster, we can get the value from one of the other two machines: 908 909 ```sh 910 curl -L http://127.0.0.1:4002/v2/keys/foo 911 ``` 912 913 We can also see that a new leader has been elected: 914 915 ``` 916 curl -L http://127.0.0.1:4002/v2/leader 917 ``` 918 919 ``` 920 http://127.0.0.1:7002 921 ``` 922 923 or 924 925 ``` 926 http://127.0.0.1:7003 927 ``` 928 929 930 ### Testing Persistence 931 932 Next we'll kill all the machines to test persistence. 933 Type `CTRL-C` on each terminal and then rerun the same command you used to start each machine. 934 935 Your request for the `foo` key will return the correct value: 936 937 ```sh 938 curl -L http://127.0.0.1:4002/v2/keys/foo 939 ``` 940 941 ```json 942 { 943 "action": "get", 944 "node": { 945 "createdIndex": 4, 946 "key": "/foo", 947 "modifiedIndex": 4, 948 "value": "bar" 949 } 950 } 951 ``` 952 953 954 ### Using HTTPS between servers 955 956 In the previous example we showed how to use SSL client certs for client-to-server communication. 957 Etcd can also do internal server-to-server communication using SSL client certs. 958 To do this just change the `-*-file` flags to `-peer-*-file`. 959 960 If you are using SSL for server-to-server communication, you must use it on all instances of etcd. 961 962 ## Modules 963 964 etcd has a number of modules that are built on top of the core etcd API. 965 These modules provide things like dashboards, locks and leader election. 966 967 ### Dashboard 968 969 An HTML dashboard can be found at `http://127.0.0.1:4001/mod/dashboard/` 970 971 ### Lock 972 973 The Lock module implements a fair lock that can be used when lots of clients want access to a single resource. 974 A lock can be associated with a value. 975 The value is unique so if a lock tries to request a value that is already queued for a lock then it will find it and watch until that value obtains the lock. 976 If you lock the same value on a key from two separate curl sessions they'll both return at the same time. 977 978 Here's the API: 979 980 **Acquire a lock (with no value) for "customer1"** 981 982 ```sh 983 curl -X POST http://127.0.0.1:4001/mod/v2/lock/customer1?ttl=60 984 ``` 985 986 **Acquire a lock for "customer1" that is associated with the value "bar"** 987 988 ```sh 989 curl -X POST http://127.0.0.1:4001/mod/v2/lock/customer1?ttl=60 -d value=bar 990 ``` 991 992 **Renew the TTL on the "customer1" lock for index 2** 993 994 ```sh 995 curl -X PUT http://127.0.0.1:4001/mod/v2/lock/customer1?ttl=60 -d index=2 996 ``` 997 998 **Renew the TTL on the "customer1" lock for value "customer1"** 999 1000 ```sh 1001 curl -X PUT http://127.0.0.1:4001/mod/v2/lock/customer1?ttl=60 -d value=bar 1002 ``` 1003 1004 **Retrieve the current value for the "customer1" lock.** 1005 1006 ```sh 1007 curl http://127.0.0.1:4001/mod/v2/lock/customer1 1008 ``` 1009 1010 **Retrieve the current index for the "customer1" lock** 1011 1012 ```sh 1013 curl http://127.0.0.1:4001/mod/v2/lock/customer1?field=index 1014 ``` 1015 1016 **Delete the "customer1" lock with the index 2** 1017 1018 ```sh 1019 curl -X DELETE http://127.0.0.1:4001/mod/v2/lock/customer1?index=customer1 1020 ``` 1021 1022 **Delete the "customer1" lock with the value "bar"** 1023 1024 ```sh 1025 curl -X DELETE http://127.0.0.1:4001/mod/v2/lock/customer1?value=bar 1026 ``` 1027 1028 1029 ### Leader Election 1030 1031 The Leader Election module wraps the Lock module to allow clients to come to consensus on a single value. 1032 This is useful when you want one server to process at a time but allow other servers to fail over. 1033 The API is similar to the Lock module but is limited to simple strings values. 1034 1035 Here's the API: 1036 1037 **Attempt to set a value for the "order_processing" leader key:** 1038 1039 ```sh 1040 curl -X POST http://127.0.0.1:4001/mod/v2/leader/order_processing?ttl=60 -d name=myserver1.foo.com 1041 ``` 1042 1043 **Retrieve the current value for the "order_processing" leader key:** 1044 1045 ```sh 1046 curl http://127.0.0.1:4001/mod/v2/leader/order_processing 1047 myserver1.foo.com 1048 ``` 1049 1050 **Remove a value from the "order_processing" leader key:** 1051 1052 ```sh 1053 curl -X POST http://127.0.0.1:4001/mod/v2/leader/order_processing?name=myserver1.foo.com 1054 ``` 1055 1056 If multiple clients attempt to set the value for a key then only one will succeed. 1057 The other clients will hang until the current value is removed because of TTL or because of a `DELETE` operation. 1058 Multiple clients can submit the same value and will all be notified when that value succeeds. 1059 1060 To update the TTL of a value simply reissue the same `POST` command that you used to set the value. 1061 1062 1063 ## Contributing 1064 1065 See [CONTRIBUTING](https://github.com/coreos/etcd/blob/master/CONTRIBUTING.md) for details on submitting patches and contacting developers via IRC and mailing lists. 1066 1067 1068 ## Libraries and Tools 1069 1070 **Tools** 1071 1072 - [etcdctl](https://github.com/coreos/etcdctl) - A command line client for etcd 1073 1074 **Go libraries** 1075 1076 - [go-etcd](https://github.com/coreos/go-etcd) 1077 1078 **Java libraries** 1079 1080 - [justinsb/jetcd](https://github.com/justinsb/jetcd) 1081 - [diwakergupta/jetcd](https://github.com/diwakergupta/jetcd) 1082 1083 **Python libraries** 1084 1085 - [transitorykris/etcd-py](https://github.com/transitorykris/etcd-py) 1086 - [jplana/python-etcd](https://github.com/jplana/python-etcd) 1087 - [russellhaering/txetcd](https://github.com/russellhaering/txetcd) - a Twisted Python library 1088 1089 **Node libraries** 1090 1091 - [stianeikeland/node-etcd](https://github.com/stianeikeland/node-etcd) 1092 1093 **Ruby libraries** 1094 1095 - [iconara/etcd-rb](https://github.com/iconara/etcd-rb) 1096 - [jpfuentes2/etcd-ruby](https://github.com/jpfuentes2/etcd-ruby) 1097 - [ranjib/etcd-ruby](https://github.com/ranjib/etcd-ruby) 1098 1099 **C libraries** 1100 1101 - [jdarcy/etcd-api](https://github.com/jdarcy/etcd-api) 1102 1103 **Clojure libraries** 1104 1105 - [aterreno/etcd-clojure](https://github.com/aterreno/etcd-clojure) 1106 - [dwwoelfel/cetcd](https://github.com/dwwoelfel/cetcd) 1107 - [rthomas/clj-etcd](https://github.com/rthomas/clj-etcd) 1108 1109 **Erlang libraries** 1110 1111 - [marshall-lee/etcd.erl](https://github.com/marshall-lee/etcd.erl) 1112 1113 **Chef Integration** 1114 1115 - [coderanger/etcd-chef](https://github.com/coderanger/etcd-chef) 1116 1117 **Chef Cookbook** 1118 1119 - [spheromak/etcd-cookbook](https://github.com/spheromak/etcd-cookbook) 1120 1121 **BOSH Releases** 1122 1123 - [cloudfoundry-community/etcd-boshrelease](https://github.com/cloudfoundry-community/etcd-boshrelease) 1124 - [cloudfoundry/cf-release](https://github.com/cloudfoundry/cf-release/tree/master/jobs/etcd) 1125 1126 **Projects using etcd** 1127 1128 - [binocarlos/yoda](https://github.com/binocarlos/yoda) - etcd + ZeroMQ 1129 - [calavera/active-proxy](https://github.com/calavera/active-proxy) - HTTP Proxy configured with etcd 1130 - [derekchiang/etcdplus](https://github.com/derekchiang/etcdplus) - A set of distributed synchronization primitives built upon etcd 1131 - [go-discover](https://github.com/flynn/go-discover) - service discovery in Go 1132 - [gleicon/goreman](https://github.com/gleicon/goreman/tree/etcd) - Branch of the Go Foreman clone with etcd support 1133 - [garethr/hiera-etcd](https://github.com/garethr/hiera-etcd) - Puppet hiera backend using etcd 1134 - [mattn/etcd-vim](https://github.com/mattn/etcd-vim) - SET and GET keys from inside vim 1135 - [mattn/etcdenv](https://github.com/mattn/etcdenv) - "env" shebang with etcd integration 1136 - [kelseyhightower/confd](https://github.com/kelseyhightower/confd) - Manage local app config files using templates and data from etcd 1137 1138 1139 ## FAQ 1140 1141 ### What size cluster should I use? 1142 1143 Every command the client sends to the master is broadcast to all of the followers. 1144 The command is not committed until the majority of the cluster peers receive that command. 1145 1146 Because of this majority voting property, the ideal cluster should be kept small to keep speed up and be made up of an odd number of peers. 1147 1148 Odd numbers are good because if you have 8 peers the majority will be 5 and if you have 9 peers the majority will still be 5. 1149 The result is that an 8 peer cluster can tolerate 3 peer failures and a 9 peer cluster can tolerate 4 machine failures. 1150 And in the best case when all 9 peers are responding the cluster will perform at the speed of the fastest 5 machines. 1151 1152 1153 ### Why SSLv3 alert handshake failure when using SSL client auth? 1154 1155 The `crypto/tls` package of `golang` checks the key usage of the certificate public key before using it. 1156 To use the certificate public key to do client auth, we need to add `clientAuth` to `Extended Key Usage` when creating the certificate public key. 1157 1158 Here is how to do it: 1159 1160 Add the following section to your openssl.cnf: 1161 1162 ``` 1163 [ ssl_client ] 1164 ... 1165 extendedKeyUsage = clientAuth 1166 ... 1167 ``` 1168 1169 When creating the cert be sure to reference it in the `-extensions` flag: 1170 1171 ``` 1172 openssl ca -config openssl.cnf -policy policy_anything -extensions ssl_client -out certs/machine.crt -infiles machine.csr 1173 ``` 1174 1175 ### Tuning 1176 1177 The default settings in etcd should work well for installations on a local network where the average network latency is low. 1178 However, when using etcd across multiple data centers or over networks with high latency you may need to tweak the heartbeat and election timeout settings. 1179 1180 The underlying distributed consensus protocol relies on two separate timeouts to ensure that nodes can handoff leadership if one stalls or goes offline. 1181 The first timeout is called the *Heartbeat Timeout*. 1182 This is the frequency with which the leader will notify followers that it is still the leader. 1183 etcd batches commands together for higher throughput so this heartbeat timeout is also a delay for how long it takes for commands to be committed. 1184 By default, etcd uses a `50ms` heartbeat timeout. 1185 1186 The second timeout is the *Election Timeout*. 1187 This timeout is how long a follower node will go without hearing a heartbeat before attempting to become leader itself. 1188 By default, etcd uses a `200ms` election timeout. 1189 1190 Adjusting these values is a trade off. 1191 Lowering the heartbeat timeout will cause individual commands to be committed faster but it will lower the overall throughput of etcd. 1192 If your etcd instances have low utilization then lowering the heartbeat timeout can improve your command response time. 1193 1194 The election timeout should be set based on the heartbeat timeout and your network ping time between nodes. 1195 Election timeouts should be at least 10 times your ping time so it can account for variance in your network. 1196 For example, if the ping time between your nodes is 10ms then you should have at least a 100ms election timeout. 1197 1198 You should also set your election timeout to at least 4 to 5 times your heartbeat timeout to account for variance in leader replication. 1199 For a heartbeat timeout of 50ms you should set your election timeout to at least 200ms - 250ms. 1200 1201 You can override the default values on the command line: 1202 1203 ```sh 1204 # Command line arguments: 1205 $ etcd -peer-heartbeat-timeout=100 -peer-election-timeout=500 1206 1207 # Environment variables: 1208 $ ETCD_PEER_HEARTBEAT_TIMEOUT=100 ETCD_PEER_ELECTION_TIMEOUT=500 etcd 1209 ``` 1210 1211 Or you can set the values within the configuration file: 1212 1213 ```toml 1214 [peer] 1215 heartbeat_timeout = 100 1216 election_timeout = 100 1217 ``` 1218 1219 The values are specified in milliseconds. 1220 1221 1222 ## Project Details 1223 1224 ### Versioning 1225 1226 etcd uses [semantic versioning][semver]. 1227 New minor versions may add additional features to the API however. 1228 1229 You can get the version of etcd by issuing a request to /version: 1230 1231 ```sh 1232 curl -L http://127.0.0.1:4001/version 1233 ``` 1234 1235 During the pre-v1.0.0 series of releases we may break the API as we fix bugs and get feedback. 1236 1237 [semver]: http://semver.org/ 1238 1239 1240 ### License 1241 1242 etcd is under the Apache 2.0 license. See the [LICENSE][license] file for details. 1243 1244 [license]: https://github.com/coreos/etcd/blob/master/LICENSE