github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/inputs/kafka_franz.md (about) 1 --- 2 title: kafka_franz 3 type: input 4 status: experimental 5 categories: ["Services"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/input/kafka_franz.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 :::caution EXPERIMENTAL 19 This component is experimental and therefore subject to change or removal outside of major version releases. 20 ::: 21 An alternative Kafka input using the [Franz Kafka client library](https://github.com/twmb/franz-go). 22 23 Introduced in version 3.61.0. 24 25 26 <Tabs defaultValue="common" values={[ 27 { label: 'Common', value: 'common', }, 28 { label: 'Advanced', value: 'advanced', }, 29 ]}> 30 31 <TabItem value="common"> 32 33 ```yaml 34 # Common config fields, showing default values 35 input: 36 label: "" 37 kafka_franz: 38 seed_brokers: [] 39 topics: [] 40 consumer_group: "" 41 ``` 42 43 </TabItem> 44 <TabItem value="advanced"> 45 46 ```yaml 47 # All config fields, showing default values 48 input: 49 label: "" 50 kafka_franz: 51 seed_brokers: [] 52 topics: [] 53 consumer_group: "" 54 checkpoint_limit: 100 55 tls: 56 enabled: false 57 skip_cert_verify: false 58 enable_renegotiation: false 59 root_cas: "" 60 root_cas_file: "" 61 client_certs: [] 62 sasl: [] 63 ``` 64 65 </TabItem> 66 </Tabs> 67 68 Consumes one or more topics by balancing the partitions across any other connected clients with the same consumer group. 69 70 This input is new and experimental, and the existing `kafka` input is not going anywhere, but here's some reasons why it might be worth trying this one out: 71 72 - You like shiny new stuff 73 - You are experiencing issues with the existing `kafka` input 74 - Someone told you to 75 76 ### Metadata 77 78 This input adds the following metadata fields to each message: 79 80 ``` text 81 - kafka_key 82 - kafka_topic 83 - kafka_partition 84 - kafka_offset 85 - kafka_timestamp_unix 86 - All record headers 87 ``` 88 89 90 ## Fields 91 92 ### `seed_brokers` 93 94 A list of broker addresses to connect to in order to establish connections. If an item of the list contains commas it will be expanded into multiple addresses. 95 96 97 Type: `array` 98 99 ```yaml 100 # Examples 101 102 seed_brokers: 103 - localhost:9092 104 105 seed_brokers: 106 - foo:9092 107 - bar:9092 108 109 seed_brokers: 110 - foo:9092,bar:9092 111 ``` 112 113 ### `topics` 114 115 A list of topics to consume from, partitions are automatically shared across consumers sharing the consumer group. 116 117 118 Type: `array` 119 120 ### `consumer_group` 121 122 A consumer group to consume as. Partitions are automatically distributed across consumers sharing a consumer group, and partition offsets are automatically commited and resumed under this name. 123 124 125 Type: `string` 126 127 ### `checkpoint_limit` 128 129 Determines how many messages of the same partition can be processed in parallel before applying back pressure. When a message of a given offset is delivered to the output the offset is only allowed to be committed when all messages of prior offsets have also been delivered, this ensures at-least-once delivery guarantees. However, this mechanism also increases the likelihood of duplicates in the event of crashes or server faults, reducing the checkpoint limit will mitigate this. 130 131 132 Type: `int` 133 Default: `100` 134 135 ### `tls` 136 137 Custom TLS settings can be used to override system defaults. 138 139 140 Type: `object` 141 142 ### `tls.enabled` 143 144 Whether custom TLS settings are enabled. 145 146 147 Type: `bool` 148 Default: `false` 149 150 ### `tls.skip_cert_verify` 151 152 Whether to skip server side certificate verification. 153 154 155 Type: `bool` 156 Default: `false` 157 158 ### `tls.enable_renegotiation` 159 160 Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you're seeing the error message `local error: tls: no renegotiation`. 161 162 163 Type: `bool` 164 Default: `false` 165 Requires version 3.45.0 or newer 166 167 ### `tls.root_cas` 168 169 An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. 170 171 172 Type: `string` 173 Default: `""` 174 175 ```yaml 176 # Examples 177 178 root_cas: |- 179 -----BEGIN CERTIFICATE----- 180 ... 181 -----END CERTIFICATE----- 182 ``` 183 184 ### `tls.root_cas_file` 185 186 An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. 187 188 189 Type: `string` 190 Default: `""` 191 192 ```yaml 193 # Examples 194 195 root_cas_file: ./root_cas.pem 196 ``` 197 198 ### `tls.client_certs` 199 200 A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. 201 202 203 Type: `array` 204 205 ```yaml 206 # Examples 207 208 client_certs: 209 - cert: foo 210 key: bar 211 212 client_certs: 213 - cert_file: ./example.pem 214 key_file: ./example.key 215 ``` 216 217 ### `tls.client_certs[].cert` 218 219 A plain text certificate to use. 220 221 222 Type: `string` 223 Default: `""` 224 225 ### `tls.client_certs[].key` 226 227 A plain text certificate key to use. 228 229 230 Type: `string` 231 Default: `""` 232 233 ### `tls.client_certs[].cert_file` 234 235 The path to a certificate to use. 236 237 238 Type: `string` 239 Default: `""` 240 241 ### `tls.client_certs[].key_file` 242 243 The path of a certificate key to use. 244 245 246 Type: `string` 247 Default: `""` 248 249 ### `sasl` 250 251 Specify one or more methods of SASL authentication. SASL is tried in order; if the broker supports the first mechanism, all connections will use that mechanism. If the first mechanism fails, the client will pick the first supported mechanism. If the broker does not support any client mechanisms, connections will fail. 252 253 254 Type: `array` 255 256 ```yaml 257 # Examples 258 259 sasl: 260 - mechanism: SCRAM-SHA-512 261 password: bar 262 username: foo 263 ``` 264 265 ### `sasl[].mechanism` 266 267 The SASL mechanism to use. 268 269 270 Type: `string` 271 272 | Option | Summary | 273 |---|---| 274 | `OAUTHBEARER` | OAuth Bearer based authentication. | 275 | `PLAIN` | Plain text authentication. | 276 | `SCRAM-SHA-256` | SCRAM based authentication as specified in RFC5802. | 277 | `SCRAM-SHA-512` | SCRAM based authentication as specified in RFC5802. | 278 279 280 ### `sasl[].username` 281 282 A username to provide for PLAIN or SCRAM-* authentication. 283 284 285 Type: `string` 286 Default: `""` 287 288 ### `sasl[].password` 289 290 A password to provide for PLAIN or SCRAM-* authentication. 291 292 293 Type: `string` 294 Default: `""` 295 296 ### `sasl[].token` 297 298 The token to use for a single session's OAUTHBEARER authentication. 299 300 301 Type: `string` 302 Default: `""` 303 304 ### `sasl[].extensions` 305 306 Key/value pairs to add to OAUTHBEARER authentication requests. 307 308 309 Type: `object` 310 311