github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/kafka.md (about)

     1  ---
     2  title: kafka
     3  type: output
     4  status: stable
     5  categories: ["Services"]
     6  ---
     7  
     8  <!--
     9       THIS FILE IS AUTOGENERATED!
    10  
    11       To make changes please edit the contents of:
    12       lib/output/kafka.go
    13  -->
    14  
    15  import Tabs from '@theme/Tabs';
    16  import TabItem from '@theme/TabItem';
    17  
    18  
    19  The kafka output type writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating it back to the input.
    20  
    21  
    22  <Tabs defaultValue="common" values={[
    23    { label: 'Common', value: 'common', },
    24    { label: 'Advanced', value: 'advanced', },
    25  ]}>
    26  
    27  <TabItem value="common">
    28  
    29  ```yaml
    30  # Common config fields, showing default values
    31  output:
    32    label: ""
    33    kafka:
    34      addresses:
    35        - localhost:9092
    36      topic: benthos_stream
    37      client_id: benthos_kafka_output
    38      target_version: 1.0.0
    39      key: ""
    40      partitioner: fnv1a_hash
    41      compression: none
    42      static_headers: {}
    43      metadata:
    44        exclude_prefixes: []
    45      max_in_flight: 1
    46      batching:
    47        count: 0
    48        byte_size: 0
    49        period: ""
    50        check: ""
    51  ```
    52  
    53  </TabItem>
    54  <TabItem value="advanced">
    55  
    56  ```yaml
    57  # All config fields, showing default values
    58  output:
    59    label: ""
    60    kafka:
    61      addresses:
    62        - localhost:9092
    63      tls:
    64        enabled: false
    65        skip_cert_verify: false
    66        enable_renegotiation: false
    67        root_cas: ""
    68        root_cas_file: ""
    69        client_certs: []
    70      sasl:
    71        mechanism: ""
    72        user: ""
    73        password: ""
    74        access_token: ""
    75        token_cache: ""
    76        token_key: ""
    77      topic: benthos_stream
    78      client_id: benthos_kafka_output
    79      target_version: 1.0.0
    80      rack_id: ""
    81      key: ""
    82      partitioner: fnv1a_hash
    83      partition: ""
    84      compression: none
    85      static_headers: {}
    86      metadata:
    87        exclude_prefixes: []
    88      inject_tracing_map: ""
    89      max_in_flight: 1
    90      ack_replicas: false
    91      max_msg_bytes: 1000000
    92      timeout: 5s
    93      retry_as_batch: false
    94      batching:
    95        count: 0
    96        byte_size: 0
    97        period: ""
    98        check: ""
    99        processors: []
   100      max_retries: 0
   101      backoff:
   102        initial_interval: 3s
   103        max_interval: 10s
   104        max_elapsed_time: 30s
   105  ```
   106  
   107  </TabItem>
   108  </Tabs>
   109  
   110  The config field `ack_replicas` determines whether we wait for acknowledgement from all replicas or just a single broker.
   111  
   112  Both the `key` and `topic` fields can be dynamically set using function interpolations described [here](/docs/configuration/interpolation#bloblang-queries).
   113  
   114  [Metadata](/docs/configuration/metadata) will be added to each message sent as headers (version 0.11+), but can be restricted using the field [`metadata`](#metadata).
   115  
   116  ### Strict Ordering and Retries
   117  
   118  When strict ordering is required for messages written to topic partitions it is important to ensure that both the field `max_in_flight` is set to `1` and that the field `retry_as_batch` is set to `true`.
   119  
   120  You must also ensure that failed batches are never rerouted back to the same output. This can be done by setting the field `max_retries` to `0` and `backoff.max_elapsed_time` to empty, which will apply back pressure indefinitely until the batch is sent successfully.
   121  
   122  However, this also means that manual intervention will eventually be required in cases where the batch cannot be sent due to configuration problems such as an incorrect `max_msg_bytes` estimate. A less strict but automated alternative would be to route failed batches to a dead letter queue using a [`fallback` broker](/docs/components/outputs/fallback), but this would allow subsequent batches to be delivered in the meantime whilst those failed batches are dealt with.
   123  
   124  ### Troubleshooting
   125  
   126  If you're seeing issues writing to or reading from Kafka with this component then it's worth trying out the newer [`kafka_franz` output](/docs/components/outputs/kafka_franz).
   127  
   128  - I'm seeing logs that report `Failed to connect to kafka: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)`, but the brokers are definitely reachable.
   129  
   130  Unfortunately this error message will appear for a wide range of connection problems even when the broker endpoint can be reached. Double check your authentication configuration and also ensure that you have [enabled TLS](#tlsenabled) if applicable.
   131  
   132  ## Performance
   133  
   134  This output benefits from sending multiple messages in flight in parallel for
   135  improved performance. You can tune the max number of in flight messages with the
   136  field `max_in_flight`.
   137  
   138  This output benefits from sending messages as a batch for improved performance.
   139  Batches can be formed at both the input and output level. You can find out more
   140  [in this doc](/docs/configuration/batching).
   141  
   142  ## Fields
   143  
   144  ### `addresses`
   145  
   146  A list of broker addresses to connect to. If an item of the list contains commas it will be expanded into multiple addresses.
   147  
   148  
   149  Type: `array`  
   150  Default: `["localhost:9092"]`  
   151  
   152  ```yaml
   153  # Examples
   154  
   155  addresses:
   156    - localhost:9092
   157  
   158  addresses:
   159    - localhost:9041,localhost:9042
   160  
   161  addresses:
   162    - localhost:9041
   163    - localhost:9042
   164  ```
   165  
   166  ### `tls`
   167  
   168  Custom TLS settings can be used to override system defaults.
   169  
   170  
   171  Type: `object`  
   172  
   173  ### `tls.enabled`
   174  
   175  Whether custom TLS settings are enabled.
   176  
   177  
   178  Type: `bool`  
   179  Default: `false`  
   180  
   181  ### `tls.skip_cert_verify`
   182  
   183  Whether to skip server side certificate verification.
   184  
   185  
   186  Type: `bool`  
   187  Default: `false`  
   188  
   189  ### `tls.enable_renegotiation`
   190  
   191  Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you're seeing the error message `local error: tls: no renegotiation`.
   192  
   193  
   194  Type: `bool`  
   195  Default: `false`  
   196  Requires version 3.45.0 or newer  
   197  
   198  ### `tls.root_cas`
   199  
   200  An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate.
   201  
   202  
   203  Type: `string`  
   204  Default: `""`  
   205  
   206  ```yaml
   207  # Examples
   208  
   209  root_cas: |-
   210    -----BEGIN CERTIFICATE-----
   211    ...
   212    -----END CERTIFICATE-----
   213  ```
   214  
   215  ### `tls.root_cas_file`
   216  
   217  An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate.
   218  
   219  
   220  Type: `string`  
   221  Default: `""`  
   222  
   223  ```yaml
   224  # Examples
   225  
   226  root_cas_file: ./root_cas.pem
   227  ```
   228  
   229  ### `tls.client_certs`
   230  
   231  A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both.
   232  
   233  
   234  Type: `array`  
   235  Default: `[]`  
   236  
   237  ```yaml
   238  # Examples
   239  
   240  client_certs:
   241    - cert: foo
   242      key: bar
   243  
   244  client_certs:
   245    - cert_file: ./example.pem
   246      key_file: ./example.key
   247  ```
   248  
   249  ### `tls.client_certs[].cert`
   250  
   251  A plain text certificate to use.
   252  
   253  
   254  Type: `string`  
   255  Default: `""`  
   256  
   257  ### `tls.client_certs[].key`
   258  
   259  A plain text certificate key to use.
   260  
   261  
   262  Type: `string`  
   263  Default: `""`  
   264  
   265  ### `tls.client_certs[].cert_file`
   266  
   267  The path to a certificate to use.
   268  
   269  
   270  Type: `string`  
   271  Default: `""`  
   272  
   273  ### `tls.client_certs[].key_file`
   274  
   275  The path of a certificate key to use.
   276  
   277  
   278  Type: `string`  
   279  Default: `""`  
   280  
   281  ### `sasl`
   282  
   283  Enables SASL authentication.
   284  
   285  
   286  Type: `object`  
   287  
   288  ### `sasl.mechanism`
   289  
   290  The SASL authentication mechanism, if left empty SASL authentication is not used. Warning: SCRAM based methods within Benthos have not received a security audit.
   291  
   292  
   293  Type: `string`  
   294  Default: `""`  
   295  
   296  | Option | Summary |
   297  |---|---|
   298  | `PLAIN` | Plain text authentication. NOTE: When using plain text auth it is extremely likely that you'll also need to [enable TLS](#tlsenabled). |
   299  | `OAUTHBEARER` | OAuth Bearer based authentication. |
   300  | `SCRAM-SHA-256` | Authentication using the SCRAM-SHA-256 mechanism. |
   301  | `SCRAM-SHA-512` | Authentication using the SCRAM-SHA-512 mechanism. |
   302  
   303  
   304  ### `sasl.user`
   305  
   306  A `PLAIN` username. It is recommended that you use environment variables to populate this field.
   307  
   308  
   309  Type: `string`  
   310  Default: `""`  
   311  
   312  ```yaml
   313  # Examples
   314  
   315  user: ${USER}
   316  ```
   317  
   318  ### `sasl.password`
   319  
   320  A `PLAIN` password. It is recommended that you use environment variables to populate this field.
   321  
   322  
   323  Type: `string`  
   324  Default: `""`  
   325  
   326  ```yaml
   327  # Examples
   328  
   329  password: ${PASSWORD}
   330  ```
   331  
   332  ### `sasl.access_token`
   333  
   334  A static `OAUTHBEARER` access token
   335  
   336  
   337  Type: `string`  
   338  Default: `""`  
   339  
   340  ### `sasl.token_cache`
   341  
   342  Instead of using a static `access_token` allows you to query a [`cache`](/docs/components/caches/about) resource to fetch `OAUTHBEARER` tokens from
   343  
   344  
   345  Type: `string`  
   346  Default: `""`  
   347  
   348  ### `sasl.token_key`
   349  
   350  Required when using a `token_cache`, the key to query the cache with for tokens.
   351  
   352  
   353  Type: `string`  
   354  Default: `""`  
   355  
   356  ### `topic`
   357  
   358  The topic to publish messages to.
   359  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   360  
   361  
   362  Type: `string`  
   363  Default: `"benthos_stream"`  
   364  
   365  ### `client_id`
   366  
   367  An identifier for the client connection.
   368  
   369  
   370  Type: `string`  
   371  Default: `"benthos_kafka_output"`  
   372  
   373  ### `target_version`
   374  
   375  The version of the Kafka protocol to use. This limits the capabilities used by the client and should ideally match the version of your brokers.
   376  
   377  
   378  Type: `string`  
   379  Default: `"1.0.0"`  
   380  
   381  ### `rack_id`
   382  
   383  A rack identifier for this client.
   384  
   385  
   386  Type: `string`  
   387  Default: `""`  
   388  
   389  ### `key`
   390  
   391  The key to publish messages with.
   392  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   393  
   394  
   395  Type: `string`  
   396  Default: `""`  
   397  
   398  ### `partitioner`
   399  
   400  The partitioning algorithm to use.
   401  
   402  
   403  Type: `string`  
   404  Default: `"fnv1a_hash"`  
   405  Options: `fnv1a_hash`, `murmur2_hash`, `random`, `round_robin`, `manual`.
   406  
   407  ### `partition`
   408  
   409  The manually-specified partition to publish messages to, relevant only when the field `partitioner` is set to `manual`. Must be able to parse as a 32-bit integer.
   410  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   411  
   412  
   413  Type: `string`  
   414  Default: `""`  
   415  
   416  ### `compression`
   417  
   418  The compression algorithm to use.
   419  
   420  
   421  Type: `string`  
   422  Default: `"none"`  
   423  Options: `none`, `snappy`, `lz4`, `gzip`, `zstd`.
   424  
   425  ### `static_headers`
   426  
   427  An optional map of static headers that should be added to messages in addition to metadata.
   428  
   429  
   430  Type: `object`  
   431  Default: `{}`  
   432  
   433  ```yaml
   434  # Examples
   435  
   436  static_headers:
   437    first-static-header: value-1
   438    second-static-header: value-2
   439  ```
   440  
   441  ### `metadata`
   442  
   443  Specify criteria for which metadata values are sent with messages as headers.
   444  
   445  
   446  Type: `object`  
   447  
   448  ### `metadata.exclude_prefixes`
   449  
   450  Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages.
   451  
   452  
   453  Type: `array`  
   454  Default: `[]`  
   455  
   456  ### `inject_tracing_map`
   457  
   458  EXPERIMENTAL: A [Bloblang mapping](/docs/guides/bloblang/about) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer.
   459  
   460  
   461  Type: `string`  
   462  Default: `""`  
   463  Requires version 3.45.0 or newer  
   464  
   465  ```yaml
   466  # Examples
   467  
   468  inject_tracing_map: meta = meta().merge(this)
   469  
   470  inject_tracing_map: root.meta.span = this
   471  ```
   472  
   473  ### `max_in_flight`
   474  
   475  The maximum number of parallel message batches to have in flight at any given time.
   476  
   477  
   478  Type: `int`  
   479  Default: `1`  
   480  
   481  ### `ack_replicas`
   482  
   483  Ensure that messages have been copied across all replicas before acknowledging receipt.
   484  
   485  
   486  Type: `bool`  
   487  Default: `false`  
   488  
   489  ### `max_msg_bytes`
   490  
   491  The maximum size in bytes of messages sent to the target topic.
   492  
   493  
   494  Type: `int`  
   495  Default: `1000000`  
   496  
   497  ### `timeout`
   498  
   499  The maximum period of time to wait for message sends before abandoning the request and retrying.
   500  
   501  
   502  Type: `string`  
   503  Default: `"5s"`  
   504  
   505  ### `retry_as_batch`
   506  
   507  When enabled forces an entire batch of messages to be retried if any individual message fails on a send, otherwise only the individual messages that failed are retried. Disabling this helps to reduce message duplicates during intermittent errors, but also makes it impossible to guarantee strict ordering of messages.
   508  
   509  
   510  Type: `bool`  
   511  Default: `false`  
   512  
   513  ### `batching`
   514  
   515  Allows you to configure a [batching policy](/docs/configuration/batching).
   516  
   517  
   518  Type: `object`  
   519  
   520  ```yaml
   521  # Examples
   522  
   523  batching:
   524    byte_size: 5000
   525    count: 0
   526    period: 1s
   527  
   528  batching:
   529    count: 10
   530    period: 1s
   531  
   532  batching:
   533    check: this.contains("END BATCH")
   534    count: 0
   535    period: 1m
   536  ```
   537  
   538  ### `batching.count`
   539  
   540  A number of messages at which the batch should be flushed. If `0` disables count based batching.
   541  
   542  
   543  Type: `int`  
   544  Default: `0`  
   545  
   546  ### `batching.byte_size`
   547  
   548  An amount of bytes at which the batch should be flushed. If `0` disables size based batching.
   549  
   550  
   551  Type: `int`  
   552  Default: `0`  
   553  
   554  ### `batching.period`
   555  
   556  A period in which an incomplete batch should be flushed regardless of its size.
   557  
   558  
   559  Type: `string`  
   560  Default: `""`  
   561  
   562  ```yaml
   563  # Examples
   564  
   565  period: 1s
   566  
   567  period: 1m
   568  
   569  period: 500ms
   570  ```
   571  
   572  ### `batching.check`
   573  
   574  A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch.
   575  
   576  
   577  Type: `string`  
   578  Default: `""`  
   579  
   580  ```yaml
   581  # Examples
   582  
   583  check: this.type == "end_of_transaction"
   584  ```
   585  
   586  ### `batching.processors`
   587  
   588  A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op.
   589  
   590  
   591  Type: `array`  
   592  Default: `[]`  
   593  
   594  ```yaml
   595  # Examples
   596  
   597  processors:
   598    - archive:
   599        format: lines
   600  
   601  processors:
   602    - archive:
   603        format: json_array
   604  
   605  processors:
   606    - merge_json: {}
   607  ```
   608  
   609  ### `max_retries`
   610  
   611  The maximum number of retries before giving up on the request. If set to zero there is no discrete limit.
   612  
   613  
   614  Type: `int`  
   615  Default: `0`  
   616  
   617  ### `backoff`
   618  
   619  Control time intervals between retry attempts.
   620  
   621  
   622  Type: `object`  
   623  
   624  ### `backoff.initial_interval`
   625  
   626  The initial period to wait between retry attempts.
   627  
   628  
   629  Type: `string`  
   630  Default: `"3s"`  
   631  
   632  ### `backoff.max_interval`
   633  
   634  The maximum period to wait between retry attempts.
   635  
   636  
   637  Type: `string`  
   638  Default: `"10s"`  
   639  
   640  ### `backoff.max_elapsed_time`
   641  
   642  The maximum period to wait before retry attempts are abandoned. If zero then no limit is used.
   643  
   644  
   645  Type: `string`  
   646  Default: `"30s"`  
   647  
   648