github.com/choria-io/go-choria@v0.28.1-0.20240416190746-b3bf9c7d5a45/broker/network/README.md (about)

     1  # Choria Network Broker
     2  
     3  This is a [NATS](https://nats.io) compatible Network Broker for use by the Choria Orchestration System.
     4  
     5  Please review the official documentation at [choria.io](https://choria.io) for installation and usage.
     6  
     7  ## Motivation
     8  
     9  Running a middleware broker for Choria is quite the undertaking, while NATS is really easy to operate it does have a plethora of settings and using the wrong ones can adversely affect your network.
    10  
    11  The Choria Network Broker is a managed NATS broker that integrates into the `choria broker` command.  It ships as part of the normal Choria package and exist within the single binary.
    12  
    13  It sets up the NATS server in ways thats suitable for use by Choria with sane defaults enabled.
    14  
    15  Features:
    16  
    17  * Works by default without any broker specific configuration in your Choria broker
    18  * Secure by default - only accepts TLS connections with certificates signed by the known CA
    19  * Support NATS Accounts technology for large scale multi tenancy
    20  * Supports Clustering using a simple comma separated list of peers - TLS by default
    21  * Support Gateways enabling communication between NATS clusters - an alternative to Choria Federation
    22  * Support Leafnodes enabling joining older or unauthenticated clients to a secure multi tenant network
    23  * Exports statistics using the popular Prometheus format via the normal Choria statistics port
    24  
    25  ## Configuration
    26  
    27  The broker is configured using the Choria daemon configuration, below a reference of the settings it supports.
    28  
    29  ### Choria core settings that affect the broker:
    30  
    31  |Setting|Description|
    32  |-------|-----------|
    33  |`plugin.choria.broker_network`|Enables the network broker when running `choria broker run`|
    34  |`loglevel`|The logging level to use|
    35  |`plugin.choria.stats_port`|The port Choria listens on for metrics, when >0 the broker enables statistics|
    36  |`plugin.choria.stats_address`|The network address to listen on for metrics requests|
    37  
    38  It also uses the `build.maxBrokerClients` build time configuration in Choria to configure it's maximum connection limit, this defaults to 50 000.
    39  
    40  ### Basic Broker Settings
    41  
    42  |Setting|Description|Default|
    43  |-------|-----------|-------|
    44  |`plugin.choria.network.listen_address`|The network address to listen on|`::`|
    45  |`plugin.choria.network.client_port`|The port to listen on for network clients|`4222`|
    46  |`plugin.choria.network.write_deadline`|The time to allow for writes to network clients to complete before considering them slow|5s|
    47  |`plugin.choria.network.client_hosts`|List of hosts - ip addresses or cidrs - that are allowed to use clients|all|
    48  |`plugin.choria.network.client_tls_force_required`|Force TLS on for client connections regardless of build settings|`false`|
    49  |`plugin.choria.network.tls_timeout`|Sets the timeout for establishing TLS connections|`2`|
    50  
    51  ### Cluster Settings
    52  
    53  Network Clusters are suitable for creating a cluster of up to 5 nodes on a local LAN. These form a full Mesh and provides scalability and HA.
    54  
    55  They are based on NATS technology and you can read more about them [at NATS.io](https://nats-io.github.io/docs/nats_server/clustering.html)
    56  
    57  |Setting|Description|Default|
    58  |-------|-----------|-------|
    59  |`plugin.choria.network.peer_port`|The port to listen on for broker cluster peers|`5222`|
    60  |`plugin.choria.network.peer_user`|Username to connect to cluster peers with|unset|
    61  |`plugin.choria.network.peer_password`|Password to use when connecting to cluster peers|unset|
    62  |`plugin.choria.network.peers`|Comma separated List of cluster peers to connect to|unset|
    63  
    64  ### Gateway Settings
    65  
    66  Gateways allow you to combine multiple Clusters into a single large cluster.  This allow you to span your collective across multiple data centers without the need for the much harder to configure federation brokers.
    67  
    68  By default if the broker is compiled with TLS the Gateway will use the same TLS settings for the connection - you can customize it on a per remote basis.
    69  
    70  They are based on NATS technology and you can read more about them [at NATS.io](https://nats-io.github.io/docs/gateways/)
    71  
    72  |Setting|Description|Default|
    73  |-------|-----------|-------|
    74  |`plugin.choria.network.gateway_port`|The port to listen to for Gateway connections, disabled when 0|`0`|
    75  |`plugin.choria.network.gateway_name`|Unique name for the cluster listening on the port|`CHORIA`|
    76  |`plugin.choria.network.gateway_remotes`|A comma sep list of remote names to activate|`""`|
    77  |`plugin.choria.network.gateway_remote.C1.urls`|A comma sep list of `host:port` combinations to connect to for the remote `C1` cluster||
    78  |`plugin.choria.network.gateway_remote.C1.tls.cert`|Path to a custom certificate for this remote only||
    79  |`plugin.choria.network.gateway_remote.C1.tls.key`|Path to a custom private key for this remote only||
    80  |`plugin.choria.network.gateway_remote.C1.tls.ca`|Path to a custom ca for this remote only||
    81  |`plugin.choria.network.gateway_remote.C1.tls.disable`|Disables the TLS configuration that would have inherited from the Choria Security system|`false`|
    82  |`plugin.choria.network.gateway_remote.C1.tls.verify`|Disables full TLS verify for this remote only|`true`|
    83  
    84  ### Leafnode Settings
    85  
    86  Leafnodes exist to take unauthenticated or unsecured connections and forge them into a specific Account (see below). They allow older Choria agents and clients to take part of a multi tenant or account secured network.
    87  
    88  By default if the broker is compiled with TLS the leafnode will use the same TLS settings for the connection - you can customize it on a per remote basis.
    89  
    90  They are based on NATS technology and you can read more about them [at NATS.io](https://nats-io.github.io/docs/leafnodes/)
    91  
    92  |Setting|Description|Default|
    93  |-------|-----------|-------|
    94  |`plugin.choria.network.leafnode_port`|The port to listen to for Gateway connections, disabled when 0|`0`|
    95  |`plugin.choria.network.leafnode_remotes`|A comma sep list of remote names to activate|`""`|
    96  |`plugin.choria.network.leafnode_remote.C1.url`|A `host:port` combination to connect to for the remote `C1` leafnode||
    97  |`plugin.choria.network.leafnode_remote.C1.account`|The local account name to use when connecting to the remote||
    98  |`plugin.choria.network.leafnode_remote.C1.credential`|The local credential file to use when connecting to the remote||
    99  |`plugin.choria.network.leafnode_remote.C1.tls.cert`|Path to a custom certificate for this remote only||
   100  |`plugin.choria.network.leafnode_remote.C1.tls.key`|Path to a custom private key for this remote only||
   101  |`plugin.choria.network.leafnode_remote.C1.tls.ca`|Path to a custom ca for this remote only||
   102  |`plugin.choria.network.leafnode_remote.C1.tls.disable`|Disables the TLS configuration that would have inherited from the Choria Security system|`false`|
   103  |`plugin.choria.network.leafnode_remote.C1.tls.verify`|Disables full TLS verify for this remote only|`true`|
   104  
   105  ### Accounts
   106  
   107  Accounts are based on NATS technology, you can read more about them [at NATS.io](https://nats-io.github.io/docs/nats_server/accounts.html)
   108  
   109  |Setting|Description|Default|
   110  |-------|-----------|-------|
   111  |`plugin.choria.network.operator_account`|The operator account that is managing this cluster||
   112  |`plugin.choria.network.system_account`|The system account to use, when set enables server events||
   113  
   114  ## Statistics
   115  
   116  When Statistics are enabled in Choria by setting `plugin.choria.stats_port` to nonzero the Choria Broker expose the following Prometheus statistics:
   117  
   118  |Statistic|Description|
   119  |---------|-----------|
   120  |`choria_network_connections`|Current connections on the network broker|
   121  |`choria_network_total_connections`|Total connections received since start|
   122  |`choria_network_routes`|Current active routes to other brokers|
   123  |`choria_network_remotes`|Current active connections to other brokers|
   124  |`choria_network_leafnode_remotes`|Current active connections to leaf nodes|
   125  |`choria_network_in_msgs`|Messages received by the network broker|
   126  |`choria_network_out_msgs`|Messages sent by the network broker|
   127  |`choria_network_in_bytes`|Total size of messages received by the network broker|
   128  |`choria_network_out_bytes`|Total size of messages sent by the network broker|
   129  |`choria_network_slow_consumers`|Total number of clients who were considered slow consumers|
   130  |`choria_network_subscriptions`|Number of active subscriptions to subjects on this broker|
   131  |`choria_network_leafnode_rtt_ms`|RTT for the Leafnode connection|
   132  |`choria_network_leafnode_in_msgs`|Messages received over the leafnode connection|
   133  |`choria_network_leafnode_out_msgs`|Messages sent over the leafnode connection|
   134  |`choria_network_leafnode_in_bytes`|Bytes received over the leafnode connection|
   135  |`choria_network_leafnode_out_msgs`|Total size of messages sent over the leafnode connection|
   136  |`choria_network_leafnode_subscriptions`|Number of active subscriptions to subjects on this leafnode|