github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/about.md (about) 1 --- 2 title: Outputs 3 sidebar_label: About 4 --- 5 6 An output is a sink where we wish to send our consumed data after applying an optional array of [processors][processors]. Only one output is configured at the root of a Benthos config. However, the output can be a [broker][output.broker] which combines multiple outputs under a chosen brokering pattern, or a [switch][output.switch] which is used to multiplex against different outputs. 7 8 An output config section looks like this: 9 10 ```yaml 11 output: 12 label: my_s3_output 13 14 aws_s3: 15 bucket: TODO 16 path: '${! meta("kafka_topic") }/${! json("message.id") }.json' 17 18 # Optional list of processing steps 19 processors: 20 - bloblang: '{"message":this,"meta":{"link_count":this.links.length()}}' 21 ``` 22 23 ## Back Pressure 24 25 Benthos outputs apply back pressure to components upstream. This means if your output target starts blocking traffic Benthos will gracefully stop consuming until the issue is resolved. 26 27 ## Retries 28 29 When a Benthos output fails to send a message the error is propagated back up to the input, where depending on the protocol it will either be pushed back to the source as a Noack (e.g. AMQP) or will be reattempted indefinitely with the commit withheld until success (e.g. Kafka). 30 31 It's possible to instead have Benthos indefinitely retry an output until success with a [`retry`][output.retry] output. Some other outputs, such as the [`broker`][output.broker], might also retry indefinitely depending on their configuration. 32 33 ## Dead Letter Queues 34 35 It's possible to create fallback outputs for when an output target fails using a [`fallback`][output.fallback] output: 36 37 ```yaml 38 output: 39 fallback: 40 - aws_sqs: 41 url: https://sqs.us-west-2.amazonaws.com/TODO/TODO 42 max_in_flight: 20 43 44 - http_client: 45 url: http://backup:1234/dlq 46 verb: POST 47 ``` 48 49 ## Multiplexing Outputs 50 51 There are a few different ways of multiplexing in Benthos, here's a quick run through: 52 53 ### Interpolation Multiplexing 54 55 Some output fields support [field interpolation][interpolation], which is a super easy way to multiplex messages based on their contents in situations where you are multiplexing to the same service. 56 57 For example, multiplexing against Kafka topics is a common pattern: 58 59 ```yaml 60 output: 61 kafka: 62 addresses: [ TODO:6379 ] 63 topic: ${! meta("target_topic") } 64 ``` 65 66 Refer to the field documentation for a given output to see if it support interpolation. 67 68 ### Switch Multiplexing 69 70 A more advanced form of multiplexing is to route messages to different output configurations based on a query. This is easy with the [`switch` output][output.switch]: 71 72 ```yaml 73 output: 74 switch: 75 cases: 76 - check: this.type == "foo" 77 output: 78 amqp_1: 79 url: amqps://guest:guest@localhost:5672/ 80 target_address: queue:/the_foos 81 82 - check: this.type == "bar" 83 output: 84 gcp_pubsub: 85 project: dealing_with_mike 86 topic: mikes_bars 87 88 - output: 89 redis_streams: 90 url: tcp://localhost:6379 91 stream: everything_else 92 processors: 93 - bloblang: | 94 root = this 95 root.type = this.type.not_null() | "unknown" 96 ``` 97 98 ## Labels 99 100 Outputs have an optional field `label` that can uniquely identify them in observability data such as metrics and logs. This can be useful when running configs with multiple outputs, otherwise their metrics labels will be generated based on their composition. For more information check out the [metrics documentation][metrics.about]. 101 102 import ComponentsByCategory from '@theme/ComponentsByCategory'; 103 104 ## Categories 105 106 <ComponentsByCategory type="outputs"></ComponentsByCategory> 107 108 import ComponentSelect from '@theme/ComponentSelect'; 109 110 <ComponentSelect type="outputs"></ComponentSelect> 111 112 [processors]: /docs/components/processors/about 113 [processor.bloblang]: /docs/components/processors/bloblang 114 [output.broker]: /docs/components/outputs/broker 115 [output.switch]: /docs/components/outputs/switch 116 [output.retry]: /docs/components/outputs/retry 117 [output.fallback]: /docs/components/outputs/fallback 118 [interpolation]: /docs/configuration/interpolation 119 [metrics.about]: /docs/components/metrics/about