github.com/choria-io/go-choria@v0.28.1-0.20240416190746-b3bf9c7d5a45/docs/content/broker/monitoring/_index.md (about)

     1  +++
     2  title = "Monitoring"
     3  toc = true
     4  weight = 10
     5  pre = "<b>1. </b>"
     6  +++
     7  
     8  Choria Broker is just one process running in your server, the command `systemctl status choria-broker` will show a basic
     9  overview of the process running, you should have basic monitoring for the process in place.
    10  
    11  ## Prometheus Metrics
    12  
    13  By default, the broker does not expose any monitoring metrics, if you set the `plugin.choria.stats_port` configuration
    14  to a port number it will listen on that port. You can listen on non localhost by setting `plugin.choria.stats_address`.
    15  
    16  It will then serve a number of URLs, some from Choria and some from the embedded NATS Server - see [NATS Monitoring](https://docs.nats.io/running-a-nats-service/nats_admin/monitoring) for detail about those.
    17  
    18  | Path              | Description                              |
    19  |-------------------|------------------------------------------|
    20  | `/choria/`        | Build information, run-time resource use |
    21  | `/choria/metrics` | Prometheus format metrics                |
    22  
    23  ## System Account
    24  
    25  Monitoring NATS and Choria Streams require a System account enabled:
    26  
    27  ```ini
    28  plugin.choria.network.system.user = system
    29  plugin.choria.network.system.password = s3cret
    30  ```
    31  
    32  This should be set in the Broker configuration and any client who wish to access the broker.
    33  
    34  We have a [basic Dashboard](https://grafana.com/grafana/dashboards/12430-choria-broker/) you can use to view these.
    35  
    36  ## Included Checks
    37  
    38  {{% notice tip %}}
    39  All the `choria broker server` commands are from the embedded [NATS CLI](https://github.com/nats-io/natscli) and so can be an awkward fit within our CLI hierarchy
    40  {{% /notice %}}
    41  
    42  We include a number of checks in the binary that can be used to monitor various aspects of the service.
    43  
    44  | Command                                 | Description                                                                                |
    45  |-----------------------------------------|--------------------------------------------------------------------------------------------|
    46  | `choria broker server check connection` | Performs a basic network connection and round-trip test of the NATS service                |
    47  | `choria broker server check stream`     | Checks the health of individual [Choria Streams](https://choria.io/docs/streams/) Streams  |
    48  | `choria broker server check meta`       | Checks the health of the overall Choria Streams System                                     |
    49  | `choria broker server check jetstream`  | Checks Choria Streams usage limits                                                         |
    50  | `choria broker server check server`     | Checks the health of the embedded NATS Server                                              |
    51  | `choria broker server check kv`         | Checks the health of [Choria Key-Value](https://choria.io/docs/streams/key-value/) buckets |
    52  
    53  All of these Checks require the System Account to be enabled in your broker and the client configuration to have the same settings.
    54  A custom Choria Client configuration can be set using `--choria-config` on these commands.
    55  
    56  By default, these commands act like Nagios checks:
    57  
    58  ```nohighlight
    59  % choria broker server check js
    60  OK JetStream | memory=0B memory_pct=0%;75;90 storage=1942997776B storage_pct=0%;75;90 streams=13 streams_pct=0% consumers=21 consumers_pct=0%
    61  % echo $?
    62  0
    63  ```
    64  
    65  They can though also output `json`, `prometheus` and `text` formats:
    66  
    67  ```nohighlight
    68  % choria broker server check js --format text
    69  JetStream: OK
    70  
    71  Check Metrics
    72  
    73  ╭───────────────┬───────────────┬──────┬────────────────────┬───────────────────╮
    74  │ Metric        │ Value         │ Unit │ Critical Threshold │ Warning Threshold │
    75  ├───────────────┼───────────────┼──────┼────────────────────┼───────────────────┤
    76  │ memory        │ 0.00          │ B    │ 0.00               │ 0.00              │
    77  │ memory_pct    │ 0.00          │ %    │ 90.00              │ 75.00             │
    78  │ storage       │ 1942955289.00 │ B    │ 0.00               │ 0.00              │
    79  │ storage_pct   │ 0.00          │ %    │ 90.00              │ 75.00             │
    80  │ streams       │ 13.00         │      │ 0.00               │ 0.00              │
    81  │ streams_pct   │ 0.00          │ %    │ -1.00              │ -1.00             │
    82  │ consumers     │ 21.00         │      │ 0.00               │ 0.00              │
    83  │ consumers_pct │ 0.00          │ %    │ -1.00              │ -1.00             │
    84  ╰───────────────┴───────────────┴──────┴────────────────────┴───────────────────╯
    85  ```
    86  
    87  ## Reports
    88  
    89  Several run time reports are included that can show connection states and more, all of these require the System Account.
    90  
    91  ### List Brokers in the cluster
    92  
    93  This shows all the connected NATS Servers / Choria Brokers in your cluster and some basic information about them
    94  
    95  ```nohighlight
    96  $ choria broker server list
    97  ```
    98  
    99  ### Broker Connections
   100  
   101  You can view and search active connections to your brokers, here we limit it to the top-5 by subject, see `--help` for other options
   102  
   103  ```nohighlight
   104  % choria broker server report connections --top 5
   105  ```
   106  
   107  Add the `--account=provisioning` option to see connections waiting to be provisioned if enabled.
   108  
   109  ### Streams Report
   110  
   111  One can get a overview of Choria Streams backends:
   112  
   113  ```nohighlight
   114  % choria broker server report jetstream
   115  ```
   116  
   117  ### Details Broker Data
   118  
   119  A wealth of data is available in the Brokers about every connection and every subscription and more, run `choria broker server req --help` to see a full list.
   120  
   121  ## Golang Profiling
   122  
   123  As an advanced option, that should not be enabled by default, one can enable the [Golang PProf](https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/)
   124  port to facilitate deep debugging of memory allocations and more. How to use this is out of scope of this document and
   125  really only useful for developers.
   126  
   127  ```ini
   128  plugin.choria.network.pprof_port = 9090
   129  ```