github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/about.md (about)

     1  ---
     2  title: Outputs
     3  sidebar_label: About
     4  ---
     5  
     6  An output is a sink where we wish to send our consumed data after applying an optional array of [processors][processors]. Only one output is configured at the root of a Benthos config. However, the output can be a [broker][output.broker] which combines multiple outputs under a chosen brokering pattern, or a [switch][output.switch] which is used to multiplex against different outputs.
     7  
     8  An output config section looks like this:
     9  
    10  ```yaml
    11  output:
    12    label: my_s3_output
    13  
    14    aws_s3:
    15      bucket: TODO
    16      path: '${! meta("kafka_topic") }/${! json("message.id") }.json'
    17  
    18    # Optional list of processing steps
    19    processors:
    20      - bloblang: '{"message":this,"meta":{"link_count":this.links.length()}}'
    21  ```
    22  
    23  ## Back Pressure
    24  
    25  Benthos outputs apply back pressure to components upstream. This means if your output target starts blocking traffic Benthos will gracefully stop consuming until the issue is resolved.
    26  
    27  ## Retries
    28  
    29  When a Benthos output fails to send a message the error is propagated back up to the input, where depending on the protocol it will either be pushed back to the source as a Noack (e.g. AMQP) or will be reattempted indefinitely with the commit withheld until success (e.g. Kafka).
    30  
    31  It's possible to instead have Benthos indefinitely retry an output until success with a [`retry`][output.retry] output. Some other outputs, such as the [`broker`][output.broker], might also retry indefinitely depending on their configuration.
    32  
    33  ## Dead Letter Queues
    34  
    35  It's possible to create fallback outputs for when an output target fails using a [`fallback`][output.fallback] output:
    36  
    37  ```yaml
    38  output:
    39    fallback:
    40      - aws_sqs:
    41          url: https://sqs.us-west-2.amazonaws.com/TODO/TODO
    42          max_in_flight: 20
    43  
    44      - http_client:
    45          url: http://backup:1234/dlq
    46          verb: POST
    47  ```
    48  
    49  ## Multiplexing Outputs
    50  
    51  There are a few different ways of multiplexing in Benthos, here's a quick run through:
    52  
    53  ### Interpolation Multiplexing
    54  
    55  Some output fields support [field interpolation][interpolation], which is a super easy way to multiplex messages based on their contents in situations where you are multiplexing to the same service.
    56  
    57  For example, multiplexing against Kafka topics is a common pattern:
    58  
    59  ```yaml
    60  output:
    61    kafka:
    62      addresses: [ TODO:6379 ]
    63      topic: ${! meta("target_topic") }
    64  ```
    65  
    66  Refer to the field documentation for a given output to see if it support interpolation.
    67  
    68  ### Switch Multiplexing
    69  
    70  A more advanced form of multiplexing is to route messages to different output configurations based on a query. This is easy with the [`switch` output][output.switch]:
    71  
    72  ```yaml
    73  output:
    74    switch:
    75      cases:
    76        - check: this.type == "foo"
    77          output:
    78            amqp_1:
    79              url: amqps://guest:guest@localhost:5672/
    80              target_address: queue:/the_foos
    81  
    82        - check: this.type == "bar"
    83          output:
    84            gcp_pubsub:
    85              project: dealing_with_mike
    86              topic: mikes_bars
    87  
    88        - output:
    89            redis_streams:
    90              url: tcp://localhost:6379
    91              stream: everything_else
    92            processors:
    93              - bloblang: |
    94                  root = this
    95                  root.type = this.type.not_null() | "unknown"
    96  ```
    97  
    98  ## Labels
    99  
   100  Outputs have an optional field `label` that can uniquely identify them in observability data such as metrics and logs. This can be useful when running configs with multiple outputs, otherwise their metrics labels will be generated based on their composition. For more information check out the [metrics documentation][metrics.about].
   101  
   102  import ComponentsByCategory from '@theme/ComponentsByCategory';
   103  
   104  ## Categories
   105  
   106  <ComponentsByCategory type="outputs"></ComponentsByCategory>
   107  
   108  import ComponentSelect from '@theme/ComponentSelect';
   109  
   110  <ComponentSelect type="outputs"></ComponentSelect>
   111  
   112  [processors]: /docs/components/processors/about
   113  [processor.bloblang]: /docs/components/processors/bloblang
   114  [output.broker]: /docs/components/outputs/broker
   115  [output.switch]: /docs/components/outputs/switch
   116  [output.retry]: /docs/components/outputs/retry
   117  [output.fallback]: /docs/components/outputs/fallback
   118  [interpolation]: /docs/configuration/interpolation
   119  [metrics.about]: /docs/components/metrics/about