github.com/ari-anchor/sei-tendermint@v0.0.0-20230519144642-dc826b7b56bb/docs/nodes/metrics.md (about) 1 --- 2 order: 4 3 --- 4 5 # Metrics 6 7 Tendermint can report and serve the Prometheus metrics, which in their turn can 8 be consumed by Prometheus collector(s). 9 10 This functionality is disabled by default. 11 12 To enable the Prometheus metrics, set `instrumentation.prometheus=true` in your 13 config file. Metrics will be served under `/metrics` on 26660 port by default. 14 Listen address can be changed in the config file (see 15 `instrumentation.prometheus\_listen\_addr`). 16 17 ## List of available metrics 18 19 The following metrics are available: 20 21 | **Name** | **Type** | **Tags** | **Description** | 22 |-----------------------------------------|-----------|-----------------|--------------------------------------------------------------------------------------------------------------------------------------------| 23 | abci_connection_method_timing | Histogram | method, type | Timings for each of the ABCI methods | 24 | consensus_height | Gauge | | Height of the chain | 25 | consensus_validators | Gauge | | Number of validators | 26 | consensus_validators_power | Gauge | | Total voting power of all validators | 27 | consensus_validator_power | Gauge | | Voting power of the node if in the validator set | 28 | consensus_validator_last_signed_height | Gauge | | Last height the node signed a block, if the node is a validator | 29 | consensus_validator_missed_blocks | Gauge | | Total amount of blocks missed for the node, if the node is a validator | 30 | consensus_missing_validators | Gauge | | Number of validators who did not sign | 31 | consensus_missing_validators_power | Gauge | | Total voting power of the missing validators | 32 | consensus_byzantine_validators | Gauge | | Number of validators who tried to double sign | 33 | consensus_byzantine_validators_power | Gauge | | Total voting power of the byzantine validators | 34 | consensus_block_interval_seconds | Histogram | | Time between this and last block (Block.Header.Time) in seconds | 35 | consensus_rounds | Gauge | | Number of rounds | 36 | consensus_num_txs | Gauge | | Number of transactions | 37 | consensus_total_txs | Gauge | | Total number of transactions committed | 38 | consensus_block_parts | Counter | peer_id | number of blockparts transmitted by peer | 39 | consensus_latest_block_height | gauge | | /status sync_info number | 40 | consensus_fast_syncing | gauge | | either 0 (not fast syncing) or 1 (syncing) | 41 | consensus_state_syncing | gauge | | either 0 (not state syncing) or 1 (syncing) | 42 | consensus_block_size_bytes | Gauge | | Block size in bytes | 43 | consensus_step_duration | Histogram | step | Histogram of durations for each step in the consensus protocol | 44 | consensus_block_gossip_receive_latency | Histogram | | Histogram of time taken to receive a block in seconds, measure between when a new block is first discovered to when the block is completed | 45 | consensus_block_gossip_parts_received | Counter | matches_current | Number of block parts received by the node | 46 | consensus_quorum_prevote_delay | Gauge | | Interval in seconds between the proposal timestamp and the timestamp of the earliest prevote that achieved a quorum | 47 | consensus_full_prevote_delay | Gauge | | Interval in seconds between the proposal timestamp and the timestamp of the latest prevote in a round where all validators voted | 48 | consensus_proposal_timestamp_difference | Histogram | | Difference between the timestamp in the proposal message and the local time of the validator at the time it received the message | 49 | consensus_vote_extension_receive_count | Counter | status | Number of vote extensions received | 50 | consensus_proposal_receive_count | Counter | status | Total number of proposals received by the node since process start | 51 | consensus_proposal_create_count | Counter | | Total number of proposals created by the node since process start | 52 | consensus_round_voting_power_percent | Gauge | vote_type | A value between 0 and 1.0 representing the percentage of the total voting power per vote type received within a round | 53 | consensus_late_votes | Counter | vote_type | Number of votes received by the node since process start that correspond to earlier heights and rounds than this node is currently in. | 54 | evidence_pool_num_evidence | Gauge | | Number of evidence in the evidence pool | 55 | p2p_peers | Gauge | | Number of peers node's connected to | 56 | p2p_peer_receive_bytes_total | Counter | peer_id, chID | number of bytes per channel received from a given peer | 57 | p2p_peer_send_bytes_total | Counter | peer_id, chID | number of bytes per channel sent to a given peer | 58 | p2p_peer_pending_send_bytes | Gauge | peer_id | number of pending bytes to be sent to a given peer | 59 | p2p_router_peer_queue_recv | Histogram | | The time taken to read off of a peer's queue before sending on the connection | 60 | p2p_router_peer_queue_send | Histogram | | The time taken to send on a peer's queue which will later be sent on the connection | 61 | p2p_router_channel_queue_send | Histogram | | The time taken to send on a p2p channel's queue which will later be consumed by the corresponding service | 62 | p2p_router_channel_queue_dropped_msgs | Counter | ch_id | The number of messages dropped from a peer's queue for a specific p2p channel | 63 | p2p_peer_queue_msg_size | Gauge | ch_id | The size of messages sent over a peer's queue for a specific p2p channel | 64 | mempool_size | Gauge | | Number of uncommitted transactions | 65 | mempool_tx_size_bytes | Histogram | | transaction sizes in bytes | 66 | mempool_failed_txs | Counter | | number of failed transactions | 67 | mempool_recheck_times | Counter | | number of transactions rechecked in the mempool | 68 | state_block_processing_time | Histogram | | time between BeginBlock and EndBlock in ms | 69 | state_consensus_param_updates | Counter | | number of consensus parameter updates returned by the application since process start | 70 | state_validator_set_updates | Counter | | number of validator set updates returned by the application since process start | 71 72 ## Useful queries 73 74 Percentage of missing + byzantine validators: 75 76 ```prometheus 77 ((consensus_byzantine_validators_power + consensus_missing_validators_power) / consensus_validators_power) * 100 78 ``` 79 80 Rate at which the application is responding to each ABCI method call. 81 ``` 82 sum(rate(tendermint_abci_connection_method_timing_count[5m])) by (method) 83 ``` 84 85 The 95th percentile response time for the application to the `deliver_tx` ABCI method call. 86 ``` 87 histogram_quantile(0.95, sum by(le) (rate(tendermint_abci_connection_method_timing_bucket{method="deliver_tx"}[5m]))) 88 ```