github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/aws_kinesis.md (about) 1 --- 2 title: aws_kinesis 3 type: output 4 status: stable 5 categories: ["Services","AWS"] 6 --- 7 8 <!-- 9 THIS FILE IS AUTOGENERATED! 10 11 To make changes please edit the contents of: 12 lib/output/aws_kinesis.go 13 --> 14 15 import Tabs from '@theme/Tabs'; 16 import TabItem from '@theme/TabItem'; 17 18 19 Sends messages to a Kinesis stream. 20 21 Introduced in version 3.36.0. 22 23 24 <Tabs defaultValue="common" values={[ 25 { label: 'Common', value: 'common', }, 26 { label: 'Advanced', value: 'advanced', }, 27 ]}> 28 29 <TabItem value="common"> 30 31 ```yaml 32 # Common config fields, showing default values 33 output: 34 label: "" 35 aws_kinesis: 36 stream: "" 37 partition_key: "" 38 max_in_flight: 1 39 batching: 40 count: 0 41 byte_size: 0 42 period: "" 43 check: "" 44 region: eu-west-1 45 ``` 46 47 </TabItem> 48 <TabItem value="advanced"> 49 50 ```yaml 51 # All config fields, showing default values 52 output: 53 label: "" 54 aws_kinesis: 55 stream: "" 56 partition_key: "" 57 hash_key: "" 58 max_in_flight: 1 59 batching: 60 count: 0 61 byte_size: 0 62 period: "" 63 check: "" 64 processors: [] 65 region: eu-west-1 66 endpoint: "" 67 credentials: 68 profile: "" 69 id: "" 70 secret: "" 71 token: "" 72 role: "" 73 role_external_id: "" 74 max_retries: 0 75 backoff: 76 initial_interval: 1s 77 max_interval: 5s 78 max_elapsed_time: 30s 79 ``` 80 81 </TabItem> 82 </Tabs> 83 84 Both the `partition_key`(required) and `hash_key` (optional) 85 fields can be dynamically set using function interpolations described 86 [here](/docs/configuration/interpolation#bloblang-queries). When sending batched messages the 87 interpolations are performed per message part. 88 89 ### Credentials 90 91 By default Benthos will use a shared credentials file when connecting to AWS 92 services. It's also possible to set them explicitly at the component level, 93 allowing you to transfer data across accounts. You can find out more 94 [in this document](/docs/guides/cloud/aws). 95 96 ## Performance 97 98 This output benefits from sending multiple messages in flight in parallel for 99 improved performance. You can tune the max number of in flight messages with the 100 field `max_in_flight`. 101 102 This output benefits from sending messages as a batch for improved performance. 103 Batches can be formed at both the input and output level. You can find out more 104 [in this doc](/docs/configuration/batching). 105 106 ## Fields 107 108 ### `stream` 109 110 The stream to publish messages to. 111 112 113 Type: `string` 114 Default: `""` 115 116 ### `partition_key` 117 118 A required key for partitioning messages. 119 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 120 121 122 Type: `string` 123 Default: `""` 124 125 ### `hash_key` 126 127 A optional hash key for partitioning messages. 128 This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries). 129 130 131 Type: `string` 132 Default: `""` 133 134 ### `max_in_flight` 135 136 The maximum number of messages to have in flight at a given time. Increase this to improve throughput. 137 138 139 Type: `int` 140 Default: `1` 141 142 ### `batching` 143 144 Allows you to configure a [batching policy](/docs/configuration/batching). 145 146 147 Type: `object` 148 149 ```yaml 150 # Examples 151 152 batching: 153 byte_size: 5000 154 count: 0 155 period: 1s 156 157 batching: 158 count: 10 159 period: 1s 160 161 batching: 162 check: this.contains("END BATCH") 163 count: 0 164 period: 1m 165 ``` 166 167 ### `batching.count` 168 169 A number of messages at which the batch should be flushed. If `0` disables count based batching. 170 171 172 Type: `int` 173 Default: `0` 174 175 ### `batching.byte_size` 176 177 An amount of bytes at which the batch should be flushed. If `0` disables size based batching. 178 179 180 Type: `int` 181 Default: `0` 182 183 ### `batching.period` 184 185 A period in which an incomplete batch should be flushed regardless of its size. 186 187 188 Type: `string` 189 Default: `""` 190 191 ```yaml 192 # Examples 193 194 period: 1s 195 196 period: 1m 197 198 period: 500ms 199 ``` 200 201 ### `batching.check` 202 203 A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. 204 205 206 Type: `string` 207 Default: `""` 208 209 ```yaml 210 # Examples 211 212 check: this.type == "end_of_transaction" 213 ``` 214 215 ### `batching.processors` 216 217 A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. 218 219 220 Type: `array` 221 Default: `[]` 222 223 ```yaml 224 # Examples 225 226 processors: 227 - archive: 228 format: lines 229 230 processors: 231 - archive: 232 format: json_array 233 234 processors: 235 - merge_json: {} 236 ``` 237 238 ### `region` 239 240 The AWS region to target. 241 242 243 Type: `string` 244 Default: `"eu-west-1"` 245 246 ### `endpoint` 247 248 Allows you to specify a custom endpoint for the AWS API. 249 250 251 Type: `string` 252 Default: `""` 253 254 ### `credentials` 255 256 Optional manual configuration of AWS credentials to use. More information can be found [in this document](/docs/guides/cloud/aws). 257 258 259 Type: `object` 260 261 ### `credentials.profile` 262 263 A profile from `~/.aws/credentials` to use. 264 265 266 Type: `string` 267 Default: `""` 268 269 ### `credentials.id` 270 271 The ID of credentials to use. 272 273 274 Type: `string` 275 Default: `""` 276 277 ### `credentials.secret` 278 279 The secret for the credentials being used. 280 281 282 Type: `string` 283 Default: `""` 284 285 ### `credentials.token` 286 287 The token for the credentials being used, required when using short term credentials. 288 289 290 Type: `string` 291 Default: `""` 292 293 ### `credentials.role` 294 295 A role ARN to assume. 296 297 298 Type: `string` 299 Default: `""` 300 301 ### `credentials.role_external_id` 302 303 An external ID to provide when assuming a role. 304 305 306 Type: `string` 307 Default: `""` 308 309 ### `max_retries` 310 311 The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. 312 313 314 Type: `int` 315 Default: `0` 316 317 ### `backoff` 318 319 Control time intervals between retry attempts. 320 321 322 Type: `object` 323 324 ### `backoff.initial_interval` 325 326 The initial period to wait between retry attempts. 327 328 329 Type: `string` 330 Default: `"1s"` 331 332 ### `backoff.max_interval` 333 334 The maximum period to wait between retry attempts. 335 336 337 Type: `string` 338 Default: `"5s"` 339 340 ### `backoff.max_elapsed_time` 341 342 The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. 343 344 345 Type: `string` 346 Default: `"30s"` 347 348