github.com/Jeffail/benthos/v3@v3.65.0/website/docs/components/outputs/azure_table_storage.md (about)

     1  ---
     2  title: azure_table_storage
     3  type: output
     4  status: beta
     5  categories: ["Services","Azure"]
     6  ---
     7  
     8  <!--
     9       THIS FILE IS AUTOGENERATED!
    10  
    11       To make changes please edit the contents of:
    12       lib/output/azure_table_storage.go
    13  -->
    14  
    15  import Tabs from '@theme/Tabs';
    16  import TabItem from '@theme/TabItem';
    17  
    18  :::caution BETA
    19  This component is mostly stable but breaking changes could still be made outside of major version releases if a fundamental problem with the component is found.
    20  :::
    21  
    22  Stores message parts in an Azure Table Storage table.
    23  
    24  Introduced in version 3.36.0.
    25  
    26  
    27  <Tabs defaultValue="common" values={[
    28    { label: 'Common', value: 'common', },
    29    { label: 'Advanced', value: 'advanced', },
    30  ]}>
    31  
    32  <TabItem value="common">
    33  
    34  ```yaml
    35  # Common config fields, showing default values
    36  output:
    37    label: ""
    38    azure_table_storage:
    39      storage_account: ""
    40      storage_access_key: ""
    41      storage_connection_string: ""
    42      table_name: ""
    43      partition_key: ""
    44      row_key: ""
    45      properties: {}
    46      max_in_flight: 1
    47      batching:
    48        count: 0
    49        byte_size: 0
    50        period: ""
    51        check: ""
    52  ```
    53  
    54  </TabItem>
    55  <TabItem value="advanced">
    56  
    57  ```yaml
    58  # All config fields, showing default values
    59  output:
    60    label: ""
    61    azure_table_storage:
    62      storage_account: ""
    63      storage_access_key: ""
    64      storage_connection_string: ""
    65      table_name: ""
    66      partition_key: ""
    67      row_key: ""
    68      properties: {}
    69      insert_type: INSERT
    70      max_in_flight: 1
    71      timeout: 5s
    72      batching:
    73        count: 0
    74        byte_size: 0
    75        period: ""
    76        check: ""
    77        processors: []
    78  ```
    79  
    80  </TabItem>
    81  </Tabs>
    82  
    83  Only one authentication method is required, `storage_connection_string` or `storage_account` and `storage_access_key`. If both are set then the `storage_connection_string` is given priority.
    84  
    85  In order to set the `table_name`,  `partition_key` and `row_key`
    86  you can use function interpolations described [here](/docs/configuration/interpolation#bloblang-queries), which are
    87  calculated per message of a batch.
    88  
    89  If the `properties` are not set in the config, all the `json` fields
    90  are marshaled and stored in the table, which will be created if it does not exist.
    91  
    92  The `object` and `array` fields are marshaled as strings. e.g.:
    93  
    94  The JSON message:
    95  ```json
    96  {
    97    "foo": 55,
    98    "bar": {
    99      "baz": "a",
   100      "bez": "b"
   101    },
   102    "diz": ["a", "b"]
   103  }
   104  ```
   105  
   106  Will store in the table the following properties:
   107  ```yml
   108  foo: '55'
   109  bar: '{ "baz": "a", "bez": "b" }'
   110  diz: '["a", "b"]'
   111  ```
   112  
   113  It's also possible to use function interpolations to get or transform the properties values, e.g.:
   114  
   115  ```yml
   116  properties:
   117    device: '${! json("device") }'
   118    timestamp: '${! json("timestamp") }'
   119  ```
   120  
   121  ## Performance
   122  
   123  This output benefits from sending multiple messages in flight in parallel for
   124  improved performance. You can tune the max number of in flight messages with the
   125  field `max_in_flight`.
   126  
   127  This output benefits from sending messages as a batch for improved performance.
   128  Batches can be formed at both the input and output level. You can find out more
   129  [in this doc](/docs/configuration/batching).
   130  
   131  ## Fields
   132  
   133  ### `storage_account`
   134  
   135  The storage account to upload messages to. This field is ignored if `storage_connection_string` is set.
   136  
   137  
   138  Type: `string`  
   139  Default: `""`  
   140  
   141  ### `storage_access_key`
   142  
   143  The storage account access key. This field is ignored if `storage_connection_string` is set.
   144  
   145  
   146  Type: `string`  
   147  Default: `""`  
   148  
   149  ### `storage_connection_string`
   150  
   151  A storage account connection string. This field is required if `storage_account` and `storage_access_key` are not set.
   152  
   153  
   154  Type: `string`  
   155  Default: `""`  
   156  
   157  ### `table_name`
   158  
   159  The table to store messages into.
   160  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   161  
   162  
   163  Type: `string`  
   164  Default: `""`  
   165  
   166  ```yaml
   167  # Examples
   168  
   169  table_name: ${!meta("kafka_topic")}
   170  ```
   171  
   172  ### `partition_key`
   173  
   174  The partition key.
   175  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   176  
   177  
   178  Type: `string`  
   179  Default: `""`  
   180  
   181  ```yaml
   182  # Examples
   183  
   184  partition_key: ${!json("date")}
   185  ```
   186  
   187  ### `row_key`
   188  
   189  The row key.
   190  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   191  
   192  
   193  Type: `string`  
   194  Default: `""`  
   195  
   196  ```yaml
   197  # Examples
   198  
   199  row_key: ${!json("device")}-${!uuid_v4()}
   200  ```
   201  
   202  ### `properties`
   203  
   204  A map of properties to store into the table.
   205  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   206  
   207  
   208  Type: `object`  
   209  Default: `{}`  
   210  
   211  ### `insert_type`
   212  
   213  Type of insert operation
   214  This field supports [interpolation functions](/docs/configuration/interpolation#bloblang-queries).
   215  
   216  
   217  Type: `string`  
   218  Default: `"INSERT"`  
   219  Options: `INSERT`, `INSERT_MERGE`, `INSERT_REPLACE`.
   220  
   221  ### `max_in_flight`
   222  
   223  The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
   224  
   225  
   226  Type: `int`  
   227  Default: `1`  
   228  
   229  ### `timeout`
   230  
   231  The maximum period to wait on an upload before abandoning it and reattempting.
   232  
   233  
   234  Type: `string`  
   235  Default: `"5s"`  
   236  
   237  ### `batching`
   238  
   239  Allows you to configure a [batching policy](/docs/configuration/batching).
   240  
   241  
   242  Type: `object`  
   243  
   244  ```yaml
   245  # Examples
   246  
   247  batching:
   248    byte_size: 5000
   249    count: 0
   250    period: 1s
   251  
   252  batching:
   253    count: 10
   254    period: 1s
   255  
   256  batching:
   257    check: this.contains("END BATCH")
   258    count: 0
   259    period: 1m
   260  ```
   261  
   262  ### `batching.count`
   263  
   264  A number of messages at which the batch should be flushed. If `0` disables count based batching.
   265  
   266  
   267  Type: `int`  
   268  Default: `0`  
   269  
   270  ### `batching.byte_size`
   271  
   272  An amount of bytes at which the batch should be flushed. If `0` disables size based batching.
   273  
   274  
   275  Type: `int`  
   276  Default: `0`  
   277  
   278  ### `batching.period`
   279  
   280  A period in which an incomplete batch should be flushed regardless of its size.
   281  
   282  
   283  Type: `string`  
   284  Default: `""`  
   285  
   286  ```yaml
   287  # Examples
   288  
   289  period: 1s
   290  
   291  period: 1m
   292  
   293  period: 500ms
   294  ```
   295  
   296  ### `batching.check`
   297  
   298  A [Bloblang query](/docs/guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch.
   299  
   300  
   301  Type: `string`  
   302  Default: `""`  
   303  
   304  ```yaml
   305  # Examples
   306  
   307  check: this.type == "end_of_transaction"
   308  ```
   309  
   310  ### `batching.processors`
   311  
   312  A list of [processors](/docs/components/processors/about) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op.
   313  
   314  
   315  Type: `array`  
   316  Default: `[]`  
   317  
   318  ```yaml
   319  # Examples
   320  
   321  processors:
   322    - archive:
   323        format: lines
   324  
   325  processors:
   326    - archive:
   327        format: json_array
   328  
   329  processors:
   330    - merge_json: {}
   331  ```
   332  
   333