github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/s3.md (about)

     1  ---
     2  title: s3
     3  type: output
     4  status: deprecated
     5  categories: ["Services","AWS"]
     6  ---
     7  
     8  <!--
     9       THIS FILE IS AUTOGENERATED!
    10  
    11       To make changes please edit the contents of:
    12       lib/output/s3.go
    13  -->
    14  
    15  import Tabs from '@theme/Tabs';
    16  import TabItem from '@theme/TabItem';
    17  
    18  :::warning DEPRECATED
    19  This component is deprecated and will be removed in the next major version release. Please consider moving onto [alternative components](#alternatives).
    20  :::
    21  
    22  Sends message parts as objects to an Amazon S3 bucket. Each object is uploaded
    23  with the path specified with the `path` field.
    24  
    25  
    26  <Tabs defaultValue="common" values={[
    27    { label: 'Common', value: 'common', },
    28    { label: 'Advanced', value: 'advanced', },
    29  ]}>
    30  
    31  <TabItem value="common">
    32  
    33  ```yaml
    34  # Common config fields, showing default values
    35  output:
    36    label: ""
    37    s3:
    38      bucket: ""
    39      path: ${!count("files")}-${!timestamp_unix_nano()}.txt
    40      tags: {}
    41      content_type: application/octet-stream
    42      metadata:
    43        exclude_prefixes: []
    44      max_in_flight: 1
    45      batching:
    46        count: 0
    47        byte_size: 0
    48        period: ""
    49        check: ""
    50      region: eu-west-1
    51  ```
    52  
    53  </TabItem>
    54  <TabItem value="advanced">
    55  
    56  ```yaml
    57  # All config fields, showing default values
    58  output:
    59    label: ""
    60    s3:
    61      bucket: ""
    62      path: ${!count("files")}-${!timestamp_unix_nano()}.txt
    63      tags: {}
    64      content_type: application/octet-stream
    65      content_encoding: ""
    66      cache_control: ""
    67      content_disposition: ""
    68      content_language: ""
    69      website_redirect_location: ""
    70      metadata:
    71        exclude_prefixes: []
    72      storage_class: STANDARD
    73      kms_key_id: ""
    74      server_side_encryption: ""
    75      force_path_style_urls: false
    76      max_in_flight: 1
    77      timeout: 5s
    78      batching:
    79        count: 0
    80        byte_size: 0
    81        period: ""
    82        check: ""
    83        processors: []
    84      region: eu-west-1
    85      endpoint: ""
    86      credentials:
    87        profile: ""
    88        id: ""
    89        secret: ""
    90        token: ""
    91        role: ""
    92        role_external_id: ""
    93  ```
    94  
    95  </TabItem>
    96  </Tabs>
    97  
    98  ## Alternatives
    99  
   100  This output has been renamed to [`aws_s3`](/docs/components/outputs/aws_s3).
   101  
   102  In order to have a different path for each object you should use function
   103  interpolations described [here](/docs/configuration/interpolation#bloblang-queries), which are
   104  calculated per message of a batch.
   105  
   106  ### Metadata
   107  
   108  Metadata fields on messages will be sent as headers, in order to mutate these values (or remove them) check out the [metadata docs](/docs/configuration/metadata).
   109  
   110  ### Tags
   111  
   112  The tags field allows you to specify key/value pairs to attach to objects as tags, where the values support
   113  [interpolation functions](/docs/configuration/interpolation#bloblang-queries):
   114  
   115  ```yaml
   116  output:
   117    aws_s3:
   118      bucket: TODO
   119      path: ${!count("files")}-${!timestamp_unix_nano()}.tar.gz
   120      tags:
   121        Key1: Value1
   122        Timestamp: ${!meta("Timestamp")}
   123  ```
   124  
   125  ### Credentials
   126  
   127  By default Benthos will use a shared credentials file when connecting to AWS
   128  services. It's also possible to set them explicitly at the component level,
   129  allowing you to transfer data across accounts. You can find out more
   130  [in this document](/docs/guides/cloud/aws).
   131  
   132  ### Batching
   133  
   134  It's common to want to upload messages to S3 as batched archives, the easiest
   135  way to do this is to batch your messages at the output level and join the batch
   136  of messages with an
   137  [`archive`](/docs/components/processors/archive) and/or
   138  [`compress`](/docs/components/processors/compress) processor.
   139  
   140  For example, if we wished to upload messages as a .tar.gz archive of documents
   141  we could achieve that with the following config:
   142  
   143  ```yaml
   144  output:
   145    s3:
   146      bucket: TODO
   147      path: ${!count("files")}-${!timestamp_unix_nano()}.tar.gz
   148      batching:
   149        count: 100
   150        period: 10s
   151        processors:
   152          - archive:
   153              format: tar
   154          - compress:
   155              algorithm: gzip
   156  ```
   157  
   158  Alternatively, if we wished to upload JSON documents as a single large document
   159  containing an array of objects we can do that with:
   160  
   161  ```yaml
   162  output:
   163    s3:
   164      bucket: TODO
   165      path: ${!count("files")}-${!timestamp_unix_nano()}.json
   166      batching:
   167        count: 100
   168        processors:
   169          - archive:
   170              format: json_array
   171  ```
   172  
   173  ## Performance
   174  
   175  This output benefits from sending multiple messages in flight in parallel for
   176  improved performance. You can tune the max number of in flight messages with the
   177  field `max_in_flight`.
   178  
   179  ## Fields
   180  
   181  ### `bucket`
   182  
   183  The bucket to upload messages to.
   184  
   185  
   186  Type: `string`  
   187  Default: `""`  
   188  
   189  ### `path`
   190  
   191  The path of each message to upload.
   192  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   193  
   194  
   195  Type: `string`  
   196  Default: `"${!count(\"files\")}-${!timestamp_unix_nano()}.txt"`  
   197  
   198  ```yaml
   199  # Examples
   200  
   201  path: ${!count("files")}-${!timestamp_unix_nano()}.txt
   202  
   203  path: ${!meta("kafka_key")}.json
   204  
   205  path: ${!json("doc.namespace")}/${!json("doc.id")}.json
   206  ```
   207  
   208  ### `tags`
   209  
   210  Key/value pairs to store with the object as tags.
   211  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   212  
   213  
   214  Type: `object`  
   215  Default: `{}`  
   216  
   217  ```yaml
   218  # Examples
   219  
   220  tags:
   221    Key1: Value1
   222    Timestamp: ${!meta("Timestamp")}
   223  ```
   224  
   225  ### `content_type`
   226  
   227  The content type to set for each object.
   228  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   229  
   230  
   231  Type: `string`  
   232  Default: `"application/octet-stream"`  
   233  
   234  ### `content_encoding`
   235  
   236  An optional content encoding to set for each object.
   237  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   238  
   239  
   240  Type: `string`  
   241  Default: `""`  
   242  
   243  ### `cache_control`
   244  
   245  The cache control to set for each object.
   246  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   247  
   248  
   249  Type: `string`  
   250  Default: `""`  
   251  
   252  ### `content_disposition`
   253  
   254  The content disposition to set for each object.
   255  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   256  
   257  
   258  Type: `string`  
   259  Default: `""`  
   260  
   261  ### `content_language`
   262  
   263  The content language to set for each object.
   264  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   265  
   266  
   267  Type: `string`  
   268  Default: `""`  
   269  
   270  ### `website_redirect_location`
   271  
   272  The website redirect location to set for each object.
   273  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   274  
   275  
   276  Type: `string`  
   277  Default: `""`  
   278  
   279  ### `metadata`
   280  
   281  Specify criteria for which metadata values are attached to objects as headers.
   282  
   283  
   284  Type: `object`  
   285  
   286  ### `metadata.exclude_prefixes`
   287  
   288  Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages.
   289  
   290  
   291  Type: `array`  
   292  Default: `[]`  
   293  
   294  ### `storage_class`
   295  
   296  The storage class to set for each object.
   297  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   298  
   299  
   300  Type: `string`  
   301  Default: `"STANDARD"`  
   302  Options: `STANDARD`, `REDUCED_REDUNDANCY`, `GLACIER`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `DEEP_ARCHIVE`.
   303  
   304  ### `kms_key_id`
   305  
   306  An optional server side encryption key.
   307  
   308  
   309  Type: `string`  
   310  Default: `""`  
   311  
   312  ### `server_side_encryption`
   313  
   314  An optional server side encryption algorithm.
   315  
   316  
   317  Type: `string`  
   318  Default: `""`  
   319  
   320  ### `force_path_style_urls`
   321  
   322  Forces the client API to use path style URLs, which helps when connecting to custom endpoints.
   323  
   324  
   325  Type: `bool`  
   326  Default: `false`  
   327  
   328  ### `max_in_flight`
   329  
   330  The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
   331  
   332  
   333  Type: `int`  
   334  Default: `1`  
   335  
   336  ### `timeout`
   337  
   338  The maximum period to wait on an upload before abandoning it and reattempting.
   339  
   340  
   341  Type: `string`  
   342  Default: `"5s"`  
   343  
   344  ### `batching`
   345  
   346  Allows you to configure a [batching policy](/docs/configuration/batching).
   347  
   348  
   349  Type: `object`  
   350  
   351  ```yaml
   352  # Examples
   353  
   354  batching:
   355    byte_size: 5000
   356    count: 0
   357    period: 1s
   358  
   359  batching:
   360    count: 10
   361    period: 1s
   362  
   363  batching:
   364    check: this.contains("END BATCH")
   365    count: 0
   366    period: 1m
   367  ```
   368  
   369  ### `batching.count`
   370  
   371  A number of messages at which the batch should be flushed. If `0` disables count based batching.
   372  
   373  
   374  Type: `int`  
   375  Default: `0`  
   376  
   377  ### `batching.byte_size`
   378  
   379  An amount of bytes at which the batch should be flushed. If `0` disables size based batching.
   380  
   381  
   382  Type: `int`  
   383  Default: `0`  
   384  
   385  ### `batching.period`
   386  
   387  A period in which an incomplete batch should be flushed regardless of its size.
   388  
   389  
   390  Type: `string`  
   391  Default: `""`  
   392  
   393  ```yaml
   394  # Examples
   395  
   396  period: 1s
   397  
   398  period: 1m
   399  
   400  period: 500ms
   401  ```
   402  
   403  ### `batching.check`
   404  
   405  A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch.
   406  
   407  
   408  Type: `string`  
   409  Default: `""`  
   410  
   411  ```yaml
   412  # Examples
   413  
   414  check: this.type == "end_of_transaction"
   415  ```
   416  
   417  ### `batching.processors`
   418  
   419  A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op.
   420  
   421  
   422  Type: `array`  
   423  Default: `[]`  
   424  
   425  ```yaml
   426  # Examples
   427  
   428  processors:
   429    - archive:
   430        format: lines
   431  
   432  processors:
   433    - archive:
   434        format: json_array
   435  
   436  processors:
   437    - merge_json: {}
   438  ```
   439  
   440  ### `region`
   441  
   442  The AWS region to target.
   443  
   444  
   445  Type: `string`  
   446  Default: `"eu-west-1"`  
   447  
   448  ### `endpoint`
   449  
   450  Allows you to specify a custom endpoint for the AWS API.
   451  
   452  
   453  Type: `string`  
   454  Default: `""`  
   455  
   456  ### `credentials`
   457  
   458  Optional manual configuration of AWS credentials to use. More information can be found [in this document](/docs/guides/cloud/aws).
   459  
   460  
   461  Type: `object`  
   462  
   463  ### `credentials.profile`
   464  
   465  A profile from `~/.aws/credentials` to use.
   466  
   467  
   468  Type: `string`  
   469  Default: `""`  
   470  
   471  ### `credentials.id`
   472  
   473  The ID of credentials to use.
   474  
   475  
   476  Type: `string`  
   477  Default: `""`  
   478  
   479  ### `credentials.secret`
   480  
   481  The secret for the credentials being used.
   482  
   483  
   484  Type: `string`  
   485  Default: `""`  
   486  
   487  ### `credentials.token`
   488  
   489  The token for the credentials being used, required when using short term credentials.
   490  
   491  
   492  Type: `string`  
   493  Default: `""`  
   494  
   495  ### `credentials.role`
   496  
   497  A role ARN to assume.
   498  
   499  
   500  Type: `string`  
   501  Default: `""`  
   502  
   503  ### `credentials.role_external_id`
   504  
   505  An external ID to provide when assuming a role.
   506  
   507  
   508  Type: `string`  
   509  Default: `""`  
   510  
   511