github.com/sym3tri/etcd@v0.2.1-0.20140422215517-a563d82f95d6/Documentation/debugging.md (about)

     1  # Debugging etcd
     2  
     3  Diagnosing issues in a distributed application is hard.
     4  etcd will help as much as it can - just enable these debug features using the CLI flag `-trace=*` or the config option `trace=*`.
     5  
     6  ## Logging
     7  
     8  Log verbosity can be increased to the max using either the `-vvv` CLI flag or the `very_very_verbose=true` config option.
     9  
    10  The only supported logging mode is to stdout.
    11  
    12  ## Metrics
    13  
    14  etcd itself can generate a set of metrics.
    15  These metrics represent many different internal data points that can be helpful when debugging etcd servers.
    16  
    17  #### Metrics reference
    18  
    19  Each individual metric name is prefixed with `etcd.<NAME>`, where \<NAME\> is the configured name of the etcd server.
    20  
    21  * `timer.appendentries.handle`: amount of time a peer takes to process an AppendEntriesRequest from the POV of the peer itself
    22  * `timer.peer.<PEER>.heartbeat`: amount of time a peer heartbeat operation takes from the POV of the leader that initiated that operation for peer \<PEER\>
    23  * `timer.command.<COMMAND>`: amount of time a given command took to be processed through the local server's raft state machine. This does not include time waiting on locks.
    24  
    25  #### Fetching metrics over HTTP
    26  
    27  Once tracing has been enabled on a given etcd server, all metric data is available at the server's `/debug/metrics` HTTP endpoint (i.e. `http://127.0.0.1:4001/debug/metrics`).
    28  Executing a GET HTTP command against the metrics endpoint will yield the current state of all metrics in the etcd server.
    29  
    30  #### Sending metrics to Graphite
    31  
    32  etcd supports [Graphite's Carbon plaintext protocol](https://graphite.readthedocs.org/en/latest/feeding-carbon.html#the-plaintext-protocol) - a TCP wire protocol designed for shipping metric data to an aggregator.
    33  To send metrics to a Graphite endpoint using this protocol, use of the `-graphite-host` CLI flag or the `graphite_host` config option (i.e. `graphite_host=172.17.0.19:2003`).
    34  
    35  See an [example graphite deploy script](https://github.com/coreos/etcd/contrib/graphite).
    36  
    37  #### Generating additional metrics with Collectd
    38  
    39  [Collectd](http://collectd.org/documentation.shtml) gathers metrics from the host running etcd.
    40  While these aren't metrics generated by etcd itself, it can be invaluable to compare etcd's view of the world to that of a separate process running next to etcd.
    41  
    42  See an [example collectd deploy script](https://github.com/coreos/etcd/contrib/collectd).
    43  
    44  ## Profiling
    45  
    46  etcd exposes profiling information from the Go pprof package over HTTP.
    47  The basic browsable interface is served by etcd at the `/debug/pprof` HTTP endpoint (i.e. `http://127.0.0.1:4001/debug/pprof`).
    48  For more information on using profiling tools, see http://blog.golang.org/profiling-go-programs.
    49  
    50  **NOTE**: In the following examples you need to ensure that the `./bin/etcd` is identical to the `./bin/etcd` that you are targeting (same git hash, arch, platform, etc).
    51  
    52  #### Heap memory profile
    53  
    54  ```
    55  go tool pprof ./bin/etcd http://127.0.0.1:4001/debug/pprof/heap
    56  ```
    57  
    58  #### CPU profile
    59  
    60  ```
    61  go tool pprof ./bin/etcd http://127.0.0.1:4001/debug/pprof/profile
    62  ```
    63  
    64  #### Blocked goroutine profile
    65  
    66  ```
    67  go tool pprof ./bin/etcd http://127.0.0.1:4001/debug/pprof/block
    68  ```
    69