github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/inputs/azure_blob_storage.md (about)

     1  ---
     2  title: azure_blob_storage
     3  type: input
     4  status: beta
     5  categories: ["Services","Azure"]
     6  ---
     7  
     8  <!--
     9       THIS FILE IS AUTOGENERATED!
    10  
    11       To make changes please edit the contents of:
    12       lib/input/azure_blob_storage.go
    13  -->
    14  
    15  import Tabs from '@theme/Tabs';
    16  import TabItem from '@theme/TabItem';
    17  
    18  :::caution BETA
    19  This component is mostly stable but breaking changes could still be made outside of major version releases if a fundamental problem with the component is found.
    20  :::
    21  
    22  Downloads objects within an Azure Blob Storage container, optionally filtered by
    23  a prefix.
    24  
    25  Introduced in version 3.36.0.
    26  
    27  
    28  <Tabs defaultValue="common" values={[
    29    { label: 'Common', value: 'common', },
    30    { label: 'Advanced', value: 'advanced', },
    31  ]}>
    32  
    33  <TabItem value="common">
    34  
    35  ```yaml
    36  # Common config fields, showing default values
    37  input:
    38    label: ""
    39    azure_blob_storage:
    40      storage_account: ""
    41      storage_access_key: ""
    42      storage_sas_token: ""
    43      storage_connection_string: ""
    44      container: ""
    45      prefix: ""
    46      codec: all-bytes
    47  ```
    48  
    49  </TabItem>
    50  <TabItem value="advanced">
    51  
    52  ```yaml
    53  # All config fields, showing default values
    54  input:
    55    label: ""
    56    azure_blob_storage:
    57      storage_account: ""
    58      storage_access_key: ""
    59      storage_sas_token: ""
    60      storage_connection_string: ""
    61      container: ""
    62      prefix: ""
    63      codec: all-bytes
    64      delete_objects: false
    65  ```
    66  
    67  </TabItem>
    68  </Tabs>
    69  
    70  Downloads objects within an Azure Blob Storage container, optionally filtered by a prefix.
    71  
    72  ## Downloading Large Files
    73  
    74  When downloading large files it's often necessary to process it in streamed parts in order to avoid loading the entire file in memory at a given time. In order to do this a [`codec`](#codec) can be specified that determines how to break the input into smaller individual messages.
    75  
    76  ## Metadata
    77  
    78  This input adds the following metadata fields to each message:
    79  
    80  ```
    81  - blob_storage_key
    82  - blob_storage_container
    83  - blob_storage_last_modified
    84  - blob_storage_last_modified_unix
    85  - blob_storage_content_type
    86  - blob_storage_content_encoding
    87  - All user defined metadata
    88  ```
    89  
    90  You can access these metadata fields using [function interpolation](/docs/configuration/interpolation#metadata).
    91  
    92  ## Fields
    93  
    94  ### `storage_account`
    95  
    96  The storage account to download blobs from. This field is ignored if `storage_connection_string` is set.
    97  
    98  
    99  Type: `string`  
   100  Default: `""`  
   101  
   102  ### `storage_access_key`
   103  
   104  The storage account access key. This field is ignored if `storage_connection_string` is set.
   105  
   106  
   107  Type: `string`  
   108  Default: `""`  
   109  
   110  ### `storage_sas_token`
   111  
   112  The storage account SAS token. This field is ignored if `storage_connection_string` or `storage_access_key` are set.
   113  
   114  
   115  Type: `string`  
   116  Default: `""`  
   117  Requires version 3.38.0 or newer  
   118  
   119  ### `storage_connection_string`
   120  
   121  A storage account connection string. This field is required if `storage_account` and `storage_access_key` / `storage_sas_token` are not set.
   122  
   123  
   124  Type: `string`  
   125  Default: `""`  
   126  
   127  ### `container`
   128  
   129  The name of the container from which to download blobs.
   130  
   131  
   132  Type: `string`  
   133  Default: `""`  
   134  
   135  ### `prefix`
   136  
   137  An optional path prefix, if set only objects with the prefix are consumed.
   138  
   139  
   140  Type: `string`  
   141  Default: `""`  
   142  
   143  ### `codec`
   144  
   145  The way in which the bytes of a data source should be converted into discrete messages, codecs are useful for specifying how large files or contiunous streams of data might be processed in small chunks rather than loading it all in memory. It's possible to consume lines using a custom delimiter with the `delim:x` codec, where x is the character sequence custom delimiter. Codecs can be chained with `/`, for example a gzip compressed CSV file can be consumed with the codec `gzip/csv`.
   146  
   147  
   148  Type: `string`  
   149  Default: `"all-bytes"`  
   150  
   151  | Option | Summary |
   152  |---|---|
   153  | `auto` | EXPERIMENTAL: Attempts to derive a codec for each file based on information such as the extension. For example, a .tar.gz file would be consumed with the `gzip/tar` codec. Defaults to all-bytes. |
   154  | `all-bytes` | Consume the entire file as a single binary message. |
   155  | `chunker:x` | Consume the file in chunks of a given number of bytes. |
   156  | `csv` | Consume structured rows as comma separated values, the first row must be a header row. |
   157  | `csv:x` | Consume structured rows as values separated by a custom delimiter, the first row must be a header row. The custom delimiter must be a single character, e.g. the codec `"csv:\t"` would consume a tab delimited file. |
   158  | `delim:x` | Consume the file in segments divided by a custom delimiter. |
   159  | `gzip` | Decompress a gzip file, this codec should precede another codec, e.g. `gzip/all-bytes`, `gzip/tar`, `gzip/csv`, etc. |
   160  | `lines` | Consume the file in segments divided by linebreaks. |
   161  | `multipart` | Consumes the output of another codec and batches messages together. A batch ends when an empty message is consumed. For example, the codec `lines/multipart` could be used to consume multipart messages where an empty line indicates the end of each batch. |
   162  | `regex:(?m)^\d\d:\d\d:\d\d` | Consume the file in segments divided by regular expression. |
   163  | `tar` | Parse the file as a tar archive, and consume each file of the archive as a message. |
   164  
   165  
   166  ```yaml
   167  # Examples
   168  
   169  codec: lines
   170  
   171  codec: "delim:\t"
   172  
   173  codec: delim:foobar
   174  
   175  codec: gzip/csv
   176  ```
   177  
   178  ### `delete_objects`
   179  
   180  Whether to delete downloaded objects from the blob once they are processed.
   181  
   182  
   183  Type: `bool`  
   184  Default: `false`  
   185  
   186