github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/kinesis.md (about) 1 --- 2 title: kinesis 3 type: output 4 status: deprecated 5 categories: ["Services","AWS"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/output/kinesis.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 :::warning DEPRECATED 19 This component is deprecated and will be removed in the next major version release. Please consider moving onto [alternative components](#alternatives). 20 ::: 21 22 Sends messages to a Kinesis stream. 23 24 25 <Tabs defaultValue="common" values={[ 26 { label: 'Common', value: 'common', }, 27 { label: 'Advanced', value: 'advanced', }, 28 ]}> 29 30 <TabItem value="common"> 31 32 ```yaml 33 # Common config fields, showing default values 34 output: 35 label: "" 36 kinesis: 37 stream: "" 38 partition_key: "" 39 max_in_flight: 1 40 batching: 41 count: 0 42 byte_size: 0 43 period: "" 44 check: "" 45 region: eu-west-1 46 ``` 47 48 </TabItem> 49 <TabItem value="advanced"> 50 51 ```yaml 52 # All config fields, showing default values 53 output: 54 label: "" 55 kinesis: 56 stream: "" 57 partition_key: "" 58 hash_key: "" 59 max_in_flight: 1 60 batching: 61 count: 0 62 byte_size: 0 63 period: "" 64 check: "" 65 processors: [] 66 region: eu-west-1 67 endpoint: "" 68 credentials: 69 profile: "" 70 id: "" 71 secret: "" 72 token: "" 73 role: "" 74 role_external_id: "" 75 max_retries: 0 76 backoff: 77 initial_interval: 1s 78 max_interval: 5s 79 max_elapsed_time: 30s 80 ``` 81 82 </TabItem> 83 </Tabs> 84 85 ## Alternatives 86 87 This output has been renamed to [`aws_kinesis`](/docs/components/outputs/aws_kinesis). 88 89 Both the `partition_key`(required) and `hash_key` (optional) 90 fields can be dynamically set using function interpolations described 91 [here](/docs/configuration/interpolation#bloblang-queries). When sending batched messages the 92 interpolations are performed per message part. 93 94 ### Credentials 95 96 By default Benthos will use a shared credentials file when connecting to AWS 97 services. It's also possible to set them explicitly at the component level, 98 allowing you to transfer data across accounts. You can find out more 99 [in this document](/docs/guides/cloud/aws). 100 101 ## Performance 102 103 This output benefits from sending multiple messages in flight in parallel for 104 improved performance. You can tune the max number of in flight messages with the 105 field `max_in_flight`. 106 107 This output benefits from sending messages as a batch for improved performance. 108 Batches can be formed at both the input and output level. You can find out more 109 [in this doc](/docs/configuration/batching). 110 111 ## Fields 112 113 ### `stream` 114 115 The stream to publish messages to. 116 117 118 Type: `string` 119 Default: `""` 120 121 ### `partition_key` 122 123 A required key for partitioning messages. 124 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 125 126 127 Type: `string` 128 Default: `""` 129 130 ### `hash_key` 131 132 A optional hash key for partitioning messages. 133 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 134 135 136 Type: `string` 137 Default: `""` 138 139 ### `max_in_flight` 140 141 The maximum number of messages to have in flight at a given time. Increase this to improve throughput. 142 143 144 Type: `int` 145 Default: `1` 146 147 ### `batching` 148 149 Allows you to configure a [batching policy](/docs/configuration/batching). 150 151 152 Type: `object` 153 154 ```yaml 155 # Examples 156 157 batching: 158 byte_size: 5000 159 count: 0 160 period: 1s 161 162 batching: 163 count: 10 164 period: 1s 165 166 batching: 167 check: this.contains("END BATCH") 168 count: 0 169 period: 1m 170 ``` 171 172 ### `batching.count` 173 174 A number of messages at which the batch should be flushed. If `0` disables count based batching. 175 176 177 Type: `int` 178 Default: `0` 179 180 ### `batching.byte_size` 181 182 An amount of bytes at which the batch should be flushed. If `0` disables size based batching. 183 184 185 Type: `int` 186 Default: `0` 187 188 ### `batching.period` 189 190 A period in which an incomplete batch should be flushed regardless of its size. 191 192 193 Type: `string` 194 Default: `""` 195 196 ```yaml 197 # Examples 198 199 period: 1s 200 201 period: 1m 202 203 period: 500ms 204 ``` 205 206 ### `batching.check` 207 208 A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. 209 210 211 Type: `string` 212 Default: `""` 213 214 ```yaml 215 # Examples 216 217 check: this.type == "end_of_transaction" 218 ``` 219 220 ### `batching.processors` 221 222 A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. 223 224 225 Type: `array` 226 Default: `[]` 227 228 ```yaml 229 # Examples 230 231 processors: 232 - archive: 233 format: lines 234 235 processors: 236 - archive: 237 format: json_array 238 239 processors: 240 - merge_json: {} 241 ``` 242 243 ### `region` 244 245 The AWS region to target. 246 247 248 Type: `string` 249 Default: `"eu-west-1"` 250 251 ### `endpoint` 252 253 Allows you to specify a custom endpoint for the AWS API. 254 255 256 Type: `string` 257 Default: `""` 258 259 ### `credentials` 260 261 Optional manual configuration of AWS credentials to use. More information can be found [in this document](/docs/guides/cloud/aws). 262 263 264 Type: `object` 265 266 ### `credentials.profile` 267 268 A profile from `~/.aws/credentials` to use. 269 270 271 Type: `string` 272 Default: `""` 273 274 ### `credentials.id` 275 276 The ID of credentials to use. 277 278 279 Type: `string` 280 Default: `""` 281 282 ### `credentials.secret` 283 284 The secret for the credentials being used. 285 286 287 Type: `string` 288 Default: `""` 289 290 ### `credentials.token` 291 292 The token for the credentials being used, required when using short term credentials. 293 294 295 Type: `string` 296 Default: `""` 297 298 ### `credentials.role` 299 300 A role ARN to assume. 301 302 303 Type: `string` 304 Default: `""` 305 306 ### `credentials.role_external_id` 307 308 An external ID to provide when assuming a role. 309 310 311 Type: `string` 312 Default: `""` 313 314 ### `max_retries` 315 316 The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. 317 318 319 Type: `int` 320 Default: `0` 321 322 ### `backoff` 323 324 Control time intervals between retry attempts. 325 326 327 Type: `object` 328 329 ### `backoff.initial_interval` 330 331 The initial period to wait between retry attempts. 332 333 334 Type: `string` 335 Default: `"1s"` 336 337 ### `backoff.max_interval` 338 339 The maximum period to wait between retry attempts. 340 341 342 Type: `string` 343 Default: `"5s"` 344 345 ### `backoff.max_elapsed_time` 346 347 The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. 348 349 350 Type: `string` 351 Default: `"30s"` 352 353