github.com/Jeffail/benthos/v3@v3.65.0/website/docs/configuration/metadata.md (about)

     1  ---
     2  title: Metadata
     3  ---
     4  
     5  In Benthos each message has raw contents and metadata, which is a map of key/value pairs representing an arbitrary amount of complementary data.
     6  
     7  When an input protocol supports attributes or metadata they will automatically be added to your messages, refer to the respective input documentation for a list of metadata keys. When an output supports attributes or metadata any metadata key/value pairs in a message will be sent (subject to service limits).
     8  
     9  ## Editing Metadata
    10  
    11  Benthos allows you to add and remove metadata using the [`bloblang` processor][processors.bloblang]. For example, you can do something like this in your pipeline:
    12  
    13  ```yaml
    14  pipeline:
    15    processors:
    16    - bloblang: |
    17        # Remove all existing metadata from messages
    18        meta = deleted()
    19  
    20        # Add a new metadata field `time` from the contents of a JSON
    21        # field `event.timestamp`
    22        meta time = event.timestamp
    23  ```
    24  
    25  You can also use [Bloblang][guides.bloblang] to delete individual metadata keys with:
    26  
    27  ```coffee
    28  meta foo = deleted()
    29  ```
    30  
    31  Or do more interesting things like remove all metadata keys with a certain prefix:
    32  
    33  ```coffee
    34  meta = meta().filter(!this.key.has_prefix("kafka_"))
    35  ```
    36  
    37  ## Using Metadata
    38  
    39  Metadata values can be referenced in any field that supports [interpolation functions][interpolation]. For example, you can route messages to Kafka topics using interpolation of metadata keys:
    40  
    41  ```yaml
    42  output:
    43    kafka:
    44      addresses: [ TODO ]
    45      topic: ${! meta("target_topic") }
    46  ```
    47  
    48  Benthos also allows you to conditionally process messages based on their metadata with the [`switch` processor][processors.switch]:
    49  
    50  ```yaml
    51  pipeline:
    52    processors:
    53    - switch:
    54      - check: meta("doc_type") == "nested"
    55        processors:
    56          - sql_insert:
    57              driver: mysql
    58              dsn: foouser:foopassword@tcp(localhost:3306)/foodb
    59              table: footable
    60              columns: [ foo, bar, baz ]
    61              args_mapping: |
    62                root = [
    63                  this.document.foo,
    64                  this.document.bar,
    65                  meta("kafka_topic"),
    66                ]
    67  ```
    68  
    69  ## Restricting Metadata
    70  
    71  Outputs that support metadata, headers or some other variant of enriched fields on messages will attempt to send all metadata key/value pairs by default. However, sometimes it's useful to refer to metadata fields at the output level even though we do not wish to send them with our data. In this case it's possible to restrict the metadata keys that are sent with the field `metadata.exclude_prefixes` within the respective output config.
    72  
    73  For example, if we were sending messages to kafka using a metadata key `target_topic` to determine the topic but we wished to prevent that metadata key from being sent as a header we could use the following configuration:
    74  
    75  ```yaml
    76  output:
    77    kafka:
    78      addresses: [ TODO ]
    79      topic: ${! meta("target_topic") }
    80      metadata:
    81        exclude_prefixes:
    82          - target_topic
    83  ```
    84  
    85  And when the list of metadata keys that we do _not_ want to send is large it can be helpful to use a [Bloblang mapping][guides.bloblang] in order to give all of these "private" keys a common prefix:
    86  
    87  ```yaml
    88  pipeline:
    89    processors:
    90      # Has an explicit list of public metadata keys, and everything else is given
    91      # an underscore prefix.
    92      - bloblang: |
    93          let allowed_meta = [
    94            "foo",
    95            "bar",
    96            "baz",
    97          ]
    98          meta = meta().map_each_key(key -> if !$allowed_meta.contains(key) {
    99            "_" + key
   100          })
   101  
   102  output:
   103    kafka:
   104      addresses: [ TODO ]
   105      topic: ${! meta("_target_topic")
   106      metadata:
   107        exclude_prefixes: [ "_" ]
   108  ```
   109  
   110  [interpolation]: /docs/configuration/interpolation
   111  [processors.switch]: /docs/components/processors/switch
   112  [processors.bloblang]: /docs/components/processors/bloblang
   113  [guides.bloblang]: /docs/guides/bloblang/about