github.com/Jeffail/benthos/v3@v3.65.0/website/docs/guides/streams_mode/about.md (about)

     1  ---
     2  title: Streams Mode
     3  sidebar_label: About
     4  description: Find out about Benthos Streams mode
     5  ---
     6  
     7  A Benthos stream consists of four components; an input, an optional buffer, processor pipelines and an output. Under normal use a Benthos instance is a single stream, and these components are configured within the service config file.
     8  
     9  Alternatively, Benthos can be run in `streams` mode, where a single running Benthos instance is able to run multiple entirely isolated streams. Adding streams in this mode can be done in two ways:
    10  
    11  1. [Static configuration files][static-files] allows you to maintain a directory of static stream configuration files that will be traversed by Benthos.
    12  
    13  2. An [HTTP REST API][rest-api] allows you to dynamically create, read the status of, update, and delete streams at runtime.
    14  
    15  These two methods can be used in combination, i.e. it's possible to update and delete streams that were created with static files.
    16  
    17  When running Benthos in streams mode it is still necessary to provide a general service wide configuration with the `-c`/`--config` flag that specifies observability configuration such as the `metrics`, `logger` and `tracing` sections, as well the `http` section for configuring how the HTTP server should behave.
    18  
    19  You can import resources either in the general configuration, or using the `-r`/`--resources` flag, the same as when running Benthos in regular mode.
    20  
    21  ```sh
    22  benthos -r "./prod/*.yaml" -c ./config.yaml streams
    23  ```
    24  
    25  ## Resources
    26  
    27  When running Benthos in streams mode [resource components][resources] are shared across all streams. The streams mode HTTP API also provides an endpoint for modifying and adding resource configurations dynamically.
    28  
    29  ## Metrics
    30  
    31  Metrics from all streams are aggregated and exposed via the method specified in [the config][metrics] of the Benthos instance running in `streams` mode, with their metrics prefixed by their respective stream name.
    32  
    33  For example, a Benthos instance running in streams mode with the configured prefix `benthos` running a stream named `foo` would have metrics from `foo` registered with the prefix `benthos.foo`.
    34  
    35  This can cause problems if your streams are short lived and uniquely named as the number of metrics registered will continue to climb indefinitely. In order to avoid this you can use the `path_mapping` field to filter metric names.
    36  
    37  ```yaml
    38  # Only register metrics for the stream `foo`. Others will be ignored.
    39  metrics:
    40    prometheus:
    41      prefix: benthos
    42      path_mapping: if !this.has_prefix("foo") { deleted() }
    43  ```
    44  
    45  Or use it to rename prefixes in order to reduce the cardinality of names:
    46  
    47  ```yaml
    48  # Rename all stream metric prefixes of the form `foo_<uuid_v4>` to just `foo`.
    49  metrics:
    50    statsd:
    51      prefix: benthos
    52      address: localhost:8125
    53      flush_period: 100m
    54      path_mapping: this.re_replace("foo_[0-9\\-a-zA-Z]+\\.(.*)","foo.$1")
    55  ```
    56  
    57  [static-files]: /docs/guides/streams_mode/using_config_files
    58  [rest-api]: /docs/guides/streams_mode/using_rest_api
    59  [metrics]: /docs/components/metrics/about
    60  [resources]: /docs/configuration/resources