github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/kinesis.md (about)

     1  ---
     2  title: kinesis
     3  type: output
     4  status: deprecated
     5  categories: ["Services","AWS"]
     6  ---
     7  
     8  <!--
     9       THIS FILE IS AUTOGENERATED!
    10  
    11       To make changes please edit the contents of:
    12       lib/output/kinesis.go
    13  -->
    14  
    15  import Tabs from '@theme/Tabs';
    16  import TabItem from '@theme/TabItem';
    17  
    18  :::warning DEPRECATED
    19  This component is deprecated and will be removed in the next major version release. Please consider moving onto [alternative components](#alternatives).
    20  :::
    21  
    22  Sends messages to a Kinesis stream.
    23  
    24  
    25  <Tabs defaultValue="common" values={[
    26    { label: 'Common', value: 'common', },
    27    { label: 'Advanced', value: 'advanced', },
    28  ]}>
    29  
    30  <TabItem value="common">
    31  
    32  ```yaml
    33  # Common config fields, showing default values
    34  output:
    35    label: ""
    36    kinesis:
    37      stream: ""
    38      partition_key: ""
    39      max_in_flight: 1
    40      batching:
    41        count: 0
    42        byte_size: 0
    43        period: ""
    44        check: ""
    45      region: eu-west-1
    46  ```
    47  
    48  </TabItem>
    49  <TabItem value="advanced">
    50  
    51  ```yaml
    52  # All config fields, showing default values
    53  output:
    54    label: ""
    55    kinesis:
    56      stream: ""
    57      partition_key: ""
    58      hash_key: ""
    59      max_in_flight: 1
    60      batching:
    61        count: 0
    62        byte_size: 0
    63        period: ""
    64        check: ""
    65        processors: []
    66      region: eu-west-1
    67      endpoint: ""
    68      credentials:
    69        profile: ""
    70        id: ""
    71        secret: ""
    72        token: ""
    73        role: ""
    74        role_external_id: ""
    75      max_retries: 0
    76      backoff:
    77        initial_interval: 1s
    78        max_interval: 5s
    79        max_elapsed_time: 30s
    80  ```
    81  
    82  </TabItem>
    83  </Tabs>
    84  
    85  ## Alternatives
    86  
    87  This output has been renamed to [`aws_kinesis`](/docs/components/outputs/aws_kinesis).
    88  
    89  Both the `partition_key`(required) and `hash_key` (optional)
    90  fields can be dynamically set using function interpolations described
    91  [here](/docs/configuration/interpolation#bloblang-queries). When sending batched messages the
    92  interpolations are performed per message part.
    93  
    94  ### Credentials
    95  
    96  By default Benthos will use a shared credentials file when connecting to AWS
    97  services. It's also possible to set them explicitly at the component level,
    98  allowing you to transfer data across accounts. You can find out more
    99  [in this document](/docs/guides/cloud/aws).
   100  
   101  ## Performance
   102  
   103  This output benefits from sending multiple messages in flight in parallel for
   104  improved performance. You can tune the max number of in flight messages with the
   105  field `max_in_flight`.
   106  
   107  This output benefits from sending messages as a batch for improved performance.
   108  Batches can be formed at both the input and output level. You can find out more
   109  [in this doc](/docs/configuration/batching).
   110  
   111  ## Fields
   112  
   113  ### `stream`
   114  
   115  The stream to publish messages to.
   116  
   117  
   118  Type: `string`  
   119  Default: `""`  
   120  
   121  ### `partition_key`
   122  
   123  A required key for partitioning messages.
   124  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   125  
   126  
   127  Type: `string`  
   128  Default: `""`  
   129  
   130  ### `hash_key`
   131  
   132  A optional hash key for partitioning messages.
   133  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   134  
   135  
   136  Type: `string`  
   137  Default: `""`  
   138  
   139  ### `max_in_flight`
   140  
   141  The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
   142  
   143  
   144  Type: `int`  
   145  Default: `1`  
   146  
   147  ### `batching`
   148  
   149  Allows you to configure a [batching policy](/docs/configuration/batching).
   150  
   151  
   152  Type: `object`  
   153  
   154  ```yaml
   155  # Examples
   156  
   157  batching:
   158    byte_size: 5000
   159    count: 0
   160    period: 1s
   161  
   162  batching:
   163    count: 10
   164    period: 1s
   165  
   166  batching:
   167    check: this.contains("END BATCH")
   168    count: 0
   169    period: 1m
   170  ```
   171  
   172  ### `batching.count`
   173  
   174  A number of messages at which the batch should be flushed. If `0` disables count based batching.
   175  
   176  
   177  Type: `int`  
   178  Default: `0`  
   179  
   180  ### `batching.byte_size`
   181  
   182  An amount of bytes at which the batch should be flushed. If `0` disables size based batching.
   183  
   184  
   185  Type: `int`  
   186  Default: `0`  
   187  
   188  ### `batching.period`
   189  
   190  A period in which an incomplete batch should be flushed regardless of its size.
   191  
   192  
   193  Type: `string`  
   194  Default: `""`  
   195  
   196  ```yaml
   197  # Examples
   198  
   199  period: 1s
   200  
   201  period: 1m
   202  
   203  period: 500ms
   204  ```
   205  
   206  ### `batching.check`
   207  
   208  A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch.
   209  
   210  
   211  Type: `string`  
   212  Default: `""`  
   213  
   214  ```yaml
   215  # Examples
   216  
   217  check: this.type == "end_of_transaction"
   218  ```
   219  
   220  ### `batching.processors`
   221  
   222  A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op.
   223  
   224  
   225  Type: `array`  
   226  Default: `[]`  
   227  
   228  ```yaml
   229  # Examples
   230  
   231  processors:
   232    - archive:
   233        format: lines
   234  
   235  processors:
   236    - archive:
   237        format: json_array
   238  
   239  processors:
   240    - merge_json: {}
   241  ```
   242  
   243  ### `region`
   244  
   245  The AWS region to target.
   246  
   247  
   248  Type: `string`  
   249  Default: `"eu-west-1"`  
   250  
   251  ### `endpoint`
   252  
   253  Allows you to specify a custom endpoint for the AWS API.
   254  
   255  
   256  Type: `string`  
   257  Default: `""`  
   258  
   259  ### `credentials`
   260  
   261  Optional manual configuration of AWS credentials to use. More information can be found [in this document](/docs/guides/cloud/aws).
   262  
   263  
   264  Type: `object`  
   265  
   266  ### `credentials.profile`
   267  
   268  A profile from `~/.aws/credentials` to use.
   269  
   270  
   271  Type: `string`  
   272  Default: `""`  
   273  
   274  ### `credentials.id`
   275  
   276  The ID of credentials to use.
   277  
   278  
   279  Type: `string`  
   280  Default: `""`  
   281  
   282  ### `credentials.secret`
   283  
   284  The secret for the credentials being used.
   285  
   286  
   287  Type: `string`  
   288  Default: `""`  
   289  
   290  ### `credentials.token`
   291  
   292  The token for the credentials being used, required when using short term credentials.
   293  
   294  
   295  Type: `string`  
   296  Default: `""`  
   297  
   298  ### `credentials.role`
   299  
   300  A role ARN to assume.
   301  
   302  
   303  Type: `string`  
   304  Default: `""`  
   305  
   306  ### `credentials.role_external_id`
   307  
   308  An external ID to provide when assuming a role.
   309  
   310  
   311  Type: `string`  
   312  Default: `""`  
   313  
   314  ### `max_retries`
   315  
   316  The maximum number of retries before giving up on the request. If set to zero there is no discrete limit.
   317  
   318  
   319  Type: `int`  
   320  Default: `0`  
   321  
   322  ### `backoff`
   323  
   324  Control time intervals between retry attempts.
   325  
   326  
   327  Type: `object`  
   328  
   329  ### `backoff.initial_interval`
   330  
   331  The initial period to wait between retry attempts.
   332  
   333  
   334  Type: `string`  
   335  Default: `"1s"`  
   336  
   337  ### `backoff.max_interval`
   338  
   339  The maximum period to wait between retry attempts.
   340  
   341  
   342  Type: `string`  
   343  Default: `"5s"`  
   344  
   345  ### `backoff.max_elapsed_time`
   346  
   347  The maximum period to wait before retry attempts are abandoned. If zero then no limit is used.
   348  
   349  
   350  Type: `string`  
   351  Default: `"30s"`  
   352  
   353