github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/broker.md (about)

     1  ---
     2  title: broker
     3  type: output
     4  status: stable
     5  categories: ["Utility"]
     6  ---
     7  
     8  <!--
     9       THIS FILE IS AUTOGENERATED!
    10  
    11       To make changes please edit the contents of:
    12       lib/output/broker.go
    13  -->
    14  
    15  import Tabs from '@theme/Tabs';
    16  import TabItem from '@theme/TabItem';
    17  
    18  
    19  Allows you to route messages to multiple child outputs using a range of
    20  brokering [patterns](#patterns).
    21  
    22  
    23  <Tabs defaultValue="common" values={[
    24    { label: 'Common', value: 'common', },
    25    { label: 'Advanced', value: 'advanced', },
    26  ]}>
    27  
    28  <TabItem value="common">
    29  
    30  ```yaml
    31  # Common config fields, showing default values
    32  output:
    33    label: ""
    34    broker:
    35      pattern: fan_out
    36      outputs: []
    37      batching:
    38        count: 0
    39        byte_size: 0
    40        period: ""
    41        check: ""
    42  ```
    43  
    44  </TabItem>
    45  <TabItem value="advanced">
    46  
    47  ```yaml
    48  # All config fields, showing default values
    49  output:
    50    label: ""
    51    broker:
    52      copies: 1
    53      pattern: fan_out
    54      max_in_flight: 1
    55      outputs: []
    56      batching:
    57        count: 0
    58        byte_size: 0
    59        period: ""
    60        check: ""
    61        processors: []
    62  ```
    63  
    64  </TabItem>
    65  </Tabs>
    66  
    67  [Processors](/docs/components/processors/about) can be listed to apply across
    68  individual outputs or all outputs:
    69  
    70  ```yaml
    71  output:
    72    broker:
    73      pattern: fan_out
    74      outputs:
    75        - resource: foo
    76        - resource: bar
    77          # Processors only applied to messages sent to bar.
    78          processors:
    79            - resource: bar_processor
    80  
    81    # Processors applied to messages sent to all brokered outputs.
    82    processors:
    83      - resource: general_processor
    84  ```
    85  
    86  ## Fields
    87  
    88  ### `copies`
    89  
    90  The number of copies of each configured output to spawn.
    91  
    92  
    93  Type: `int`  
    94  Default: `1`  
    95  
    96  ### `pattern`
    97  
    98  The brokering pattern to use.
    99  
   100  
   101  Type: `string`  
   102  Default: `"fan_out"`  
   103  Options: `fan_out`, `fan_out_sequential`, `round_robin`, `greedy`.
   104  
   105  ### `max_in_flight`
   106  
   107  The maximum number of parallel message batches to have in flight at any given time. Note that if a child output has a higher `max_in_flight` then the switch output will automatically match it, therefore this value is the minimum `max_in_flight` to set in cases where the child values can't be inferred (such as when using resource outputs as children). Only relevant for `fan_out`, `fan_out_sequential` brokers.
   108  
   109  
   110  Type: `int`  
   111  Default: `1`  
   112  
   113  ### `outputs`
   114  
   115  A list of child outputs to broker.
   116  
   117  
   118  Type: `array`  
   119  Default: `[]`  
   120  
   121  ### `batching`
   122  
   123  Allows you to configure a [batching policy](/docs/configuration/batching).
   124  
   125  
   126  Type: `object`  
   127  
   128  ```yaml
   129  # Examples
   130  
   131  batching:
   132    byte_size: 5000
   133    count: 0
   134    period: 1s
   135  
   136  batching:
   137    count: 10
   138    period: 1s
   139  
   140  batching:
   141    check: this.contains("END BATCH")
   142    count: 0
   143    period: 1m
   144  ```
   145  
   146  ### `batching.count`
   147  
   148  A number of messages at which the batch should be flushed. If `0` disables count based batching.
   149  
   150  
   151  Type: `int`  
   152  Default: `0`  
   153  
   154  ### `batching.byte_size`
   155  
   156  An amount of bytes at which the batch should be flushed. If `0` disables size based batching.
   157  
   158  
   159  Type: `int`  
   160  Default: `0`  
   161  
   162  ### `batching.period`
   163  
   164  A period in which an incomplete batch should be flushed regardless of its size.
   165  
   166  
   167  Type: `string`  
   168  Default: `""`  
   169  
   170  ```yaml
   171  # Examples
   172  
   173  period: 1s
   174  
   175  period: 1m
   176  
   177  period: 500ms
   178  ```
   179  
   180  ### `batching.check`
   181  
   182  A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch.
   183  
   184  
   185  Type: `string`  
   186  Default: `""`  
   187  
   188  ```yaml
   189  # Examples
   190  
   191  check: this.type == "end_of_transaction"
   192  ```
   193  
   194  ### `batching.processors`
   195  
   196  A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op.
   197  
   198  
   199  Type: `array`  
   200  Default: `[]`  
   201  
   202  ```yaml
   203  # Examples
   204  
   205  processors:
   206    - archive:
   207        format: lines
   208  
   209  processors:
   210    - archive:
   211        format: json_array
   212  
   213  processors:
   214    - merge_json: {}
   215  ```
   216  
   217  ## Patterns
   218  
   219  The broker pattern determines the way in which messages are allocated and can be
   220  chosen from the following:
   221  
   222  ### `fan_out`
   223  
   224  With the fan out pattern all outputs will be sent every message that passes
   225  through Benthos in parallel.
   226  
   227  If an output applies back pressure it will block all subsequent messages, and if
   228  an output fails to send a message it will be retried continuously until
   229  completion or service shut down.
   230  
   231  Sometimes it is useful to disable the back pressure or retries of certain fan
   232  out outputs and instead drop messages that have failed or were blocked. In this
   233  case you can wrap outputs with a [`drop_on` output](/docs/components/outputs/drop_on).
   234  
   235  ### `fan_out_sequential`
   236  
   237  Similar to the fan out pattern except outputs are written to sequentially,
   238  meaning an output is only written to once the preceding output has confirmed
   239  receipt of the same message.
   240  
   241  ### `round_robin`
   242  
   243  With the round robin pattern each message will be assigned a single output
   244  following their order. If an output applies back pressure it will block all
   245  subsequent messages. If an output fails to send a message then the message will
   246  be re-attempted with the next input, and so on.
   247  
   248  ### `greedy`
   249  
   250  The greedy pattern results in higher output throughput at the cost of
   251  potentially disproportionate message allocations to those outputs. Each message
   252  is sent to a single output, which is determined by allowing outputs to claim
   253  messages as soon as they are able to process them. This results in certain
   254  faster outputs potentially processing more messages at the cost of slower
   255  outputs.
   256