github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/aws_kinesis.md (about)

     1  ---
     2  title: aws_kinesis
     3  type: output
     4  status: stable
     5  categories: ["Services","AWS"]
     6  ---
     7  
     8  <!--
     9       THIS FILE IS AUTOGENERATED!
    10  
    11       To make changes please edit the contents of:
    12       lib/output/aws_kinesis.go
    13  -->
    14  
    15  import Tabs from '@theme/Tabs';
    16  import TabItem from '@theme/TabItem';
    17  
    18  
    19  Sends messages to a Kinesis stream.
    20  
    21  Introduced in version 3.36.0.
    22  
    23  
    24  <Tabs defaultValue="common" values={[
    25    { label: 'Common', value: 'common', },
    26    { label: 'Advanced', value: 'advanced', },
    27  ]}>
    28  
    29  <TabItem value="common">
    30  
    31  ```yaml
    32  # Common config fields, showing default values
    33  output:
    34    label: ""
    35    aws_kinesis:
    36      stream: ""
    37      partition_key: ""
    38      max_in_flight: 1
    39      batching:
    40        count: 0
    41        byte_size: 0
    42        period: ""
    43        check: ""
    44      region: eu-west-1
    45  ```
    46  
    47  </TabItem>
    48  <TabItem value="advanced">
    49  
    50  ```yaml
    51  # All config fields, showing default values
    52  output:
    53    label: ""
    54    aws_kinesis:
    55      stream: ""
    56      partition_key: ""
    57      hash_key: ""
    58      max_in_flight: 1
    59      batching:
    60        count: 0
    61        byte_size: 0
    62        period: ""
    63        check: ""
    64        processors: []
    65      region: eu-west-1
    66      endpoint: ""
    67      credentials:
    68        profile: ""
    69        id: ""
    70        secret: ""
    71        token: ""
    72        role: ""
    73        role_external_id: ""
    74      max_retries: 0
    75      backoff:
    76        initial_interval: 1s
    77        max_interval: 5s
    78        max_elapsed_time: 30s
    79  ```
    80  
    81  </TabItem>
    82  </Tabs>
    83  
    84  Both the `partition_key`(required) and `hash_key` (optional)
    85  fields can be dynamically set using function interpolations described
    86  [here](/docs/configuration/interpolation#bloblang-queries). When sending batched messages the
    87  interpolations are performed per message part.
    88  
    89  ### Credentials
    90  
    91  By default Benthos will use a shared credentials file when connecting to AWS
    92  services. It's also possible to set them explicitly at the component level,
    93  allowing you to transfer data across accounts. You can find out more
    94  [in this document](/docs/guides/cloud/aws).
    95  
    96  ## Performance
    97  
    98  This output benefits from sending multiple messages in flight in parallel for
    99  improved performance. You can tune the max number of in flight messages with the
   100  field `max_in_flight`.
   101  
   102  This output benefits from sending messages as a batch for improved performance.
   103  Batches can be formed at both the input and output level. You can find out more
   104  [in this doc](/docs/configuration/batching).
   105  
   106  ## Fields
   107  
   108  ### `stream`
   109  
   110  The stream to publish messages to.
   111  
   112  
   113  Type: `string`  
   114  Default: `""`  
   115  
   116  ### `partition_key`
   117  
   118  A required key for partitioning messages.
   119  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   120  
   121  
   122  Type: `string`  
   123  Default: `""`  
   124  
   125  ### `hash_key`
   126  
   127  A optional hash key for partitioning messages.
   128  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   129  
   130  
   131  Type: `string`  
   132  Default: `""`  
   133  
   134  ### `max_in_flight`
   135  
   136  The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
   137  
   138  
   139  Type: `int`  
   140  Default: `1`  
   141  
   142  ### `batching`
   143  
   144  Allows you to configure a [batching policy](/docs/configuration/batching).
   145  
   146  
   147  Type: `object`  
   148  
   149  ```yaml
   150  # Examples
   151  
   152  batching:
   153    byte_size: 5000
   154    count: 0
   155    period: 1s
   156  
   157  batching:
   158    count: 10
   159    period: 1s
   160  
   161  batching:
   162    check: this.contains("END BATCH")
   163    count: 0
   164    period: 1m
   165  ```
   166  
   167  ### `batching.count`
   168  
   169  A number of messages at which the batch should be flushed. If `0` disables count based batching.
   170  
   171  
   172  Type: `int`  
   173  Default: `0`  
   174  
   175  ### `batching.byte_size`
   176  
   177  An amount of bytes at which the batch should be flushed. If `0` disables size based batching.
   178  
   179  
   180  Type: `int`  
   181  Default: `0`  
   182  
   183  ### `batching.period`
   184  
   185  A period in which an incomplete batch should be flushed regardless of its size.
   186  
   187  
   188  Type: `string`  
   189  Default: `""`  
   190  
   191  ```yaml
   192  # Examples
   193  
   194  period: 1s
   195  
   196  period: 1m
   197  
   198  period: 500ms
   199  ```
   200  
   201  ### `batching.check`
   202  
   203  A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch.
   204  
   205  
   206  Type: `string`  
   207  Default: `""`  
   208  
   209  ```yaml
   210  # Examples
   211  
   212  check: this.type == "end_of_transaction"
   213  ```
   214  
   215  ### `batching.processors`
   216  
   217  A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op.
   218  
   219  
   220  Type: `array`  
   221  Default: `[]`  
   222  
   223  ```yaml
   224  # Examples
   225  
   226  processors:
   227    - archive:
   228        format: lines
   229  
   230  processors:
   231    - archive:
   232        format: json_array
   233  
   234  processors:
   235    - merge_json: {}
   236  ```
   237  
   238  ### `region`
   239  
   240  The AWS region to target.
   241  
   242  
   243  Type: `string`  
   244  Default: `"eu-west-1"`  
   245  
   246  ### `endpoint`
   247  
   248  Allows you to specify a custom endpoint for the AWS API.
   249  
   250  
   251  Type: `string`  
   252  Default: `""`  
   253  
   254  ### `credentials`
   255  
   256  Optional manual configuration of AWS credentials to use. More information can be found [in this document](/docs/guides/cloud/aws).
   257  
   258  
   259  Type: `object`  
   260  
   261  ### `credentials.profile`
   262  
   263  A profile from `~/.aws/credentials` to use.
   264  
   265  
   266  Type: `string`  
   267  Default: `""`  
   268  
   269  ### `credentials.id`
   270  
   271  The ID of credentials to use.
   272  
   273  
   274  Type: `string`  
   275  Default: `""`  
   276  
   277  ### `credentials.secret`
   278  
   279  The secret for the credentials being used.
   280  
   281  
   282  Type: `string`  
   283  Default: `""`  
   284  
   285  ### `credentials.token`
   286  
   287  The token for the credentials being used, required when using short term credentials.
   288  
   289  
   290  Type: `string`  
   291  Default: `""`  
   292  
   293  ### `credentials.role`
   294  
   295  A role ARN to assume.
   296  
   297  
   298  Type: `string`  
   299  Default: `""`  
   300  
   301  ### `credentials.role_external_id`
   302  
   303  An external ID to provide when assuming a role.
   304  
   305  
   306  Type: `string`  
   307  Default: `""`  
   308  
   309  ### `max_retries`
   310  
   311  The maximum number of retries before giving up on the request. If set to zero there is no discrete limit.
   312  
   313  
   314  Type: `int`  
   315  Default: `0`  
   316  
   317  ### `backoff`
   318  
   319  Control time intervals between retry attempts.
   320  
   321  
   322  Type: `object`  
   323  
   324  ### `backoff.initial_interval`
   325  
   326  The initial period to wait between retry attempts.
   327  
   328  
   329  Type: `string`  
   330  Default: `"1s"`  
   331  
   332  ### `backoff.max_interval`
   333  
   334  The maximum period to wait between retry attempts.
   335  
   336  
   337  Type: `string`  
   338  Default: `"5s"`  
   339  
   340  ### `backoff.max_elapsed_time`
   341  
   342  The maximum period to wait before retry attempts are abandoned. If zero then no limit is used.
   343  
   344  
   345  Type: `string`  
   346  Default: `"30s"`  
   347  
   348