github.com/marineam/etcd@v0.1.2-0.20130821182615-9b7109b46686/README.md (about) 1 # etcd 2 3 [![Build Status](https://travis-ci.org/coreos/etcd.png)](https://travis-ci.org/coreos/etcd) 4 5 A highly-available key value store for shared configuration and service discovery. etcd is inspired by zookeeper and doozer, with a focus on: 6 7 * Simple: curl'able user facing API (HTTP+JSON) 8 * Secure: optional SSL client cert authentication 9 * Fast: benchmarked 1000s of writes/s per instance 10 * Reliable: Properly distributed using Raft 11 12 Etcd is written in Go and uses the [raft][raft] consensus algorithm to manage a highly-available replicated log. 13 14 See [etcdctl][etcdctl] for a simple command line client. Or feel free to just use curl, as in the examples below. 15 16 [raft]: https://github.com/coreos/go-raft 17 [etcdctl]: http://coreos.com/docs/etcdctl/ 18 19 ## Getting Started 20 21 ### Getting etcd 22 23 The latest release is available as a binary at [Github][github-release]. 24 25 [github-release]: https://github.com/coreos/etcd/releases/ 26 27 ### Building 28 29 You can build etcd from source: 30 31 ```sh 32 git clone https://github.com/coreos/etcd 33 ./build 34 ``` 35 36 This will generate a binary in the base directory called `./etcd`. 37 38 _NOTE_: you need go 1.1+. Please check your installation with 39 40 ``` 41 go version 42 ``` 43 44 ### Running a single node 45 46 These examples will use a single node cluster to show you the basics of the etcd REST API. Lets start etcd: 47 48 ```sh 49 ./etcd -d node0 -n node0 50 ``` 51 52 This will bring up an etcd node listening on port 4001 for client communication and on port 7001 for server-to-server communication. 53 The `-d node0` argument tells etcd to write node configuration, logs and snapshots to the `./node0/` directory. 54 The `-n node0` tells the rest of the cluster that this node is named node0. 55 56 ## Usage 57 58 ### Setting the value to a key 59 60 Let’s set the first key-value pair to the node. In this case the key is `/message` and the value is `Hello world`. 61 62 ```sh 63 curl -L http://127.0.0.1:4001/v1/keys/message -d value="Hello world" 64 ``` 65 66 ```json 67 {"action":"SET","key":"/message","value":"Hello world","newKey":true,"index":3} 68 ``` 69 70 This response contains five fields. We will introduce three more fields as we try more commands. 71 72 1. The action of the request; we set the value via a POST request, thus the action is `SET`. 73 74 2. The key of the request; we set `/message` to `Hello world!`, so the key field is `/message`. 75 Notice we use a file system like structure to represent the key-value pairs. So each key starts with `/`. 76 77 3. The current value of the key; we set the value to`Hello world`. 78 79 4. If we set a new key; `/message` did not exist before, so this is a new key. 80 81 5. Index is the unique internal log index of the set request. Requests that change the log index include `SET`, `DELETE` and `TESTANDSET`. The `GET`, `LIST` and `WATCH` commands do not change state in the store and so they do not change the index. You may notice that in this example the index is 3, although it is the first request you sent to the server. This is because there are internal commands that also change the state like adding and syncing servers. 82 83 ### Get the value of a key 84 85 Get the value that we just set in `/message` by issuing a GET: 86 87 ```sh 88 curl -L http://127.0.0.1:4001/v1/keys/message 89 ``` 90 91 ```json 92 {"action":"GET","key":"/message","value":"Hello world","index":3} 93 ``` 94 ### Change the value of a key 95 96 Change the value of `/message` from `Hello world` to `Hello etcd` with another POST to the key: 97 98 ```sh 99 curl -L http://127.0.0.1:4001/v1/keys/message -d value="Hello etcd" 100 ``` 101 102 ```json 103 {"action":"SET","key":"/message","prevValue":"Hello world","value":"Hello etcd","index":4} 104 ``` 105 106 Notice that the `prevValue` is set to `Hello world`. 107 108 ### Delete a key 109 110 Remove the `/message` key with a DELETE: 111 112 ```sh 113 curl -L http://127.0.0.1:4001/v1/keys/message -X DELETE 114 ``` 115 116 ```json 117 {"action":"DELETE","key":"/message","prevValue":"Hello etcd","index":5} 118 ``` 119 120 ### Using key TTL 121 122 Keys in etcd can be set to expire after a specified number of seconds. That is done by setting a TTL (time to live) on the key when you POST: 123 124 ```sh 125 curl -L http://127.0.0.1:4001/v1/keys/foo -d value=bar -d ttl=5 126 ``` 127 128 ```json 129 {"action":"SET","key":"/foo","value":"bar","newKey":true,"expiration":"2013-07-11T20:31:12.156146039-07:00","ttl":4,"index":6} 130 ``` 131 132 Note the last two new fields in response: 133 134 1. The expiration is the time that this key will expire and be deleted. 135 136 2. The ttl is the time to live of the key. 137 138 Now you can try to get the key by sending: 139 140 ```sh 141 curl -L http://127.0.0.1:4001/v1/keys/foo 142 ``` 143 144 If the TTL has expired, the key will be deleted, and you will be returned a 100. 145 146 ```json 147 {"errorCode":100,"message":"Key Not Found","cause":"/foo"} 148 ``` 149 150 ### Watching a prefix 151 152 We can watch a path prefix and get notifications if any key change under that prefix. 153 154 In one terminal, we send a watch request: 155 156 ```sh 157 curl -L http://127.0.0.1:4001/v1/watch/foo 158 ``` 159 160 Now, we are watching at the path prefix `/foo` and wait for any changes under this path. 161 162 In another terminal, we set a key `/foo/foo` to `barbar` to see what will happen: 163 164 ```sh 165 curl -L http://127.0.0.1:4001/v1/keys/foo/foo -d value=barbar 166 ``` 167 168 The first terminal should get the notification and return with the same response as the set request. 169 170 ```json 171 {"action":"SET","key":"/foo/foo","value":"barbar","newKey":true,"index":7} 172 ``` 173 174 However, the watch command can do more than this. Using the the index we can watch for commands that has happened in the past. This is useful for ensuring you don't miss events between watch commands. 175 176 Let's try to watch for the set command of index 6 again: 177 178 ```sh 179 curl -L http://127.0.0.1:4001/v1/watch/foo -d index=7 180 ``` 181 182 The watch command returns immediately with the same response as previous. 183 184 ### Atomic Test and Set 185 186 Etcd can be used as a centralized coordination service in a cluster and `TestAndSet` is the most basic operation to build distributed lock service. This command will set the value only if the client provided `prevValue` is equal the current key value. 187 188 Here is a simple example. Let's create a key-value pair first: `testAndSet=one`. 189 190 ```sh 191 curl -L http://127.0.0.1:4001/v1/keys/testAndSet -d value=one 192 ``` 193 194 Let's try an invaild `TestAndSet` command. 195 We can give another parameter prevValue to set command to make it a TestAndSet command. 196 197 ```sh 198 curl -L http://127.0.0.1:4001/v1/keys/testAndSet -d prevValue=two -d value=three 199 ``` 200 201 This will try to test if the previous of the key is two, it is change it to three. 202 203 ```json 204 {"errorCode":101,"message":"The given PrevValue is not equal to the value of the key","cause":"TestAndSet: one!=two"} 205 ``` 206 207 which means `testAndSet` failed. 208 209 Let us try a vaild one. 210 211 ```sh 212 curl -L http://127.0.0.1:4001/v1/keys/testAndSet -d prevValue=one -d value=two 213 ``` 214 215 The response should be 216 217 ```json 218 {"action":"SET","key":"/testAndSet","prevValue":"one","value":"two","index":10} 219 ``` 220 221 We successfully changed the value from “one” to “two”, since we give the correct previous value. 222 223 ### Listing a directory 224 225 Last we provide a simple List command to list all the keys under a prefix path. 226 227 Let us create some keys first. 228 229 We already have `/foo/foo=barbar` 230 231 We create another one `/foo/foo_dir/foo=barbarbar` 232 233 ```sh 234 curl -L http://127.0.0.1:4001/v1/keys/foo/foo_dir/bar -d value=barbarbar 235 ``` 236 237 Now list the keys under `/foo` 238 239 ```sh 240 curl -L http://127.0.0.1:4001/v1/keys/foo/ 241 ``` 242 243 We should see the response as an array of items 244 245 ```json 246 [{"action":"GET","key":"/foo/foo","value":"barbar","index":10},{"action":"GET","key":"/foo/foo_dir","dir":true,"index":10}] 247 ``` 248 249 which meas `foo=barbar` is a key-value pair under `/foo` and `foo_dir` is a directory. 250 251 ## Advanced Usage 252 253 ### Transport security with HTTPS 254 255 Etcd supports SSL/TLS and client cert authentication for clients to server, as well as server to server communication 256 257 First, you need to have a CA cert `clientCA.crt` and signed key pair `client.crt`, `client.key`. This site has a good reference for how to generate self-signed key pairs: 258 http://www.g-loaded.eu/2005/11/10/be-your-own-ca/ 259 260 For testing you can use the certificates in the `fixtures/ca` directory. 261 262 Next, lets configure etcd to use this keypair: 263 264 ```sh 265 ./etcd -n node0 -d node0 -clientCert=./fixtures/ca/server.crt -clientKey=./fixtures/ca/server.key.insecure -f 266 ``` 267 268 `-f` forces new node configuration if existing configuration is found (WARNING: data loss!) 269 `-clientCert` and `-clientKey` are the key and cert for transport layer security between client and server 270 271 You can now test the configuration using https: 272 273 ```sh 274 curl --cacert fixtures/ca/ca.crt https://127.0.0.1:4001/v1/keys/foo -F value=bar 275 ``` 276 277 You should be able to see the handshake succeed. 278 279 ``` 280 ... 281 SSLv3, TLS handshake, Finished (20): 282 ... 283 ``` 284 285 And also the response from the etcd server. 286 287 ```json 288 {"action":"SET","key":"/foo","value":"bar","newKey":true,"index":3} 289 ``` 290 291 ### Authentication with HTTPS client certificates 292 293 We can also do authentication using CA certs. The clients will provide their cert to the server and the server will check whether the cert is signed by the CA and decide whether to serve the request. 294 295 ```sh 296 ./etcd -n node0 -d node0 -clientCAFile=./fixtures/ca/ca.crt -clientCert=./fixtures/ca/server.crt -clientKey=./fixtures/ca/server.key.insecure -f 297 ``` 298 299 ```-clientCAFile``` is the path to the CA cert. 300 301 Try the same request to this server: 302 303 ```sh 304 curl --cacert fixtures/ca/ca.crt https://127.0.0.1:4001/v1/keys/foo -F value=bar 305 ``` 306 307 The request should be rejected by the server. 308 309 ``` 310 ... 311 routines:SSL3_READ_BYTES:sslv3 alert bad certificate 312 ... 313 ``` 314 315 We need to give the CA signed cert to the server. 316 317 ```sh 318 curl -L https://127.0.0.1:4001/v1/keys/foo -d value=bar -v --key myclient.key --cert myclient.crt -cacert clientCA.crt 319 ``` 320 321 You should able to see 322 ``` 323 ... 324 SSLv3, TLS handshake, CERT verify (15): 325 ... 326 TLS handshake, Finished (20) 327 ``` 328 329 And also the response from the server: 330 331 ```json 332 {"action":"SET","key":"/foo","value":"bar","newKey":true,"index":3} 333 ``` 334 335 ## Clustering 336 337 ### Example cluster of three machines 338 339 Let's explore the use of etcd clustering. We use go-raft as the underlying distributed protocol which provides consistency and persistence of the data across all of the etcd instances. 340 341 Let start by creating 3 new etcd instances. 342 343 We use -s to specify server port and -c to specify client port and -d to specify the directory to store the log and info of the node in the cluster 344 345 ```sh 346 ./etcd -s 127.0.0.1:7001 -c 127.0.0.1:4001 -d nodes/node1 -n node1 347 ``` 348 349 Let the join two more nodes to this cluster using the -C argument: 350 351 ```sh 352 ./etcd -c 127.0.0.1:4002 -s 127.0.0.1:7002 -C 127.0.0.1:7001 -d nodes/node2 -n node2 353 ./etcd -c 127.0.0.1:4003 -s 127.0.0.1:7003 -C 127.0.0.1:7001 -d nodes/node3 -n node3 354 ``` 355 356 Get the machines in the cluster: 357 358 ```sh 359 curl -L http://127.0.0.1:4001/v1/machines 360 ``` 361 362 We should see there are three nodes in the cluster 363 364 ``` 365 0.0.0.0:4001,0.0.0.0:4002,0.0.0.0:4003 366 ``` 367 368 The machine list is also available via this API: 369 370 ```sh 371 curl -L http://127.0.0.1:4001/v1/keys/_etcd/machines 372 ``` 373 374 ```json 375 [{"action":"GET","key":"/machines/node1","value":"0.0.0.0,7001,4001","index":4},{"action":"GET","key":"/machines/node3","value":"0.0.0.0,7002,4002","index":4},{"action":"GET","key":"/machines/node4","value":"0.0.0.0,7003,4003","index":4}] 376 ``` 377 378 The key of the machine is based on the ```commit index``` when it was added. The value of the machine is ```hostname```, ```raft port``` and ```client port```. 379 380 Also try to get the current leader in the cluster 381 382 ``` 383 curl -L http://127.0.0.1:4001/v1/leader 384 ``` 385 The first server we set up should be the leader, if it has not dead during these commands. 386 387 ``` 388 0.0.0.0:7001 389 ``` 390 391 Now we can do normal SET and GET operations on keys as we explored earlier. 392 393 ```sh 394 curl -L http://127.0.0.1:4001/v1/keys/foo -d value=bar 395 ``` 396 397 ```json 398 {"action":"SET","key":"/foo","value":"bar","newKey":true,"index":5} 399 ``` 400 401 ### Killing Nodes in the Cluster 402 403 Let's kill the leader of the cluster and get the value from the other machine: 404 405 ```sh 406 curl -L http://127.0.0.1:4002/v1/keys/foo 407 ``` 408 409 A new leader should have been elected. 410 411 ``` 412 curl -L http://127.0.0.1:4001/v1/leader 413 ``` 414 415 ``` 416 0.0.0.0:7002 or 0.0.0.0:7003 417 ``` 418 419 You should be able to see this: 420 421 ```json 422 {"action":"GET","key":"/foo","value":"bar","index":5} 423 ``` 424 425 It succeeded! 426 427 ### Testing Persistence 428 429 OK. Next let us kill all the nodes to test persistence. And restart all the nodes use the same command as before. 430 431 Your request for the `foo` key will return the correct value: 432 433 ```sh 434 curl -L http://127.0.0.1:4002/v1/keys/foo 435 ``` 436 437 ```json 438 {"action":"GET","key":"/foo","value":"bar","index":5} 439 ``` 440 441 ### Using HTTPS between servers 442 443 In the previous example we showed how to use SSL client certs for client to server communication. Etcd can also do internal server to server communication using SSL client certs. To do this just change the ```-client*``` flags to ```-server*```. 444 445 If you are using SSL for server to server communication, you must use it on all instances of etcd. 446 447 ## Libraries and Tools 448 449 **Tools** 450 451 - [etcdctl](https://github.com/coreos/etcdctl) - A command line client for etcd 452 453 **Go libraries** 454 455 - [go-etcd](https://github.com/coreos/go-etcd) 456 457 **Node libraries** 458 459 - [stianeikeland/node-etcd](https://github.com/stianeikeland/node-etcd) 460 461 **Ruby libraries** 462 463 - [iconara/etcd-rb](https://github.com/iconara/etcd-rb) 464 - [jpfuentes2/etcd-ruby](https://github.com/jpfuentes2/etcd-ruby) 465 - [ranjib/etcd-ruby](https://github.com/ranjib/etcd-ruby) 466 467 **Chef Cookbook** 468 469 - [spheromak/etcd-cookbook](https://github.com/spheromak/etcd-cookbook) 470 471 **Projects using etcd** 472 473 - [calavera/active-proxy](https://github.com/calavera/active-proxy) - HTTP Proxy configured with etcd 474 - [gleicon/goreman](https://github.com/gleicon/goreman/tree/etcd) - Branch of the Go Foreman clone with etcd support 475 - [garethr/hiera-etcd](https://github.com/garethr/hiera-etcd) - Puppet hiera backend using etcd 476 - [mattn/etcd-vim](https://github.com/mattn/etcd-vim) - SET and GET keys from inside vim 477 - [mattn/etcdenv](https://github.com/mattn/etcdenv) - "env" shebang with etcd integration 478 479 ## Project Details 480 481 ### Versioning 482 483 etcd uses [semantic versioning][semver]. 484 When we release v1.0.0 of etcd we will promise not to break the "v1" REST API. 485 New minor versions may add additional features to the API however. 486 487 You can get the version of etcd by requesting the root path of etcd: 488 489 ```sh 490 curl -L http://127.0.0.1:4001 491 ``` 492 493 During the v0 series of releases we may break the API as we fix bugs and get feedback. 494 495 [semver]: http://semver.org/ 496 497 ### License 498 499 etcd is under the Apache 2.0 license. See the [LICENSE][license] file for details. 500 501 [license]: https://github.com/coreos/etcd/blob/master/LICENSE