github.com/Jeffail/benthos/v3@v3.65.0/website/cookbooks/filtering.md (about)

     1  ---
     2  id: filtering
     3  title: Filtering and Sampling
     4  description: Configure Benthos to conditionally drop messages.
     5  ---
     6  
     7  Events are like eyebrows, sometimes it's best to just get rid of them. Filtering events in Benthos is both easy and flexible, this cookbook demonstrates a few different types of filtering you can do. All of these examples make use of the [`bloblang` processor][processors.bloblang] but shouldn't require any prior knowledge.
     8  
     9  ## The Basic Filter
    10  
    11  Dropping events with [Bloblang][guides.bloblang] is done by mapping the function `deleted()` to the `root` of the mapped document. To remove all events indiscriminantly you can simply do:
    12  
    13  ```yaml
    14  pipeline:
    15    processors:
    16    - bloblang: root = deleted()
    17  ```
    18  
    19  But that's most likely not what you want. We can instead only delete an event under certain conditions with a [`match`][bloblang.match] or [`if`][bloblang.if] expression:
    20  
    21  ```yaml
    22  pipeline:
    23    processors:
    24    - bloblang: |
    25        root = if meta("topic").or("") == "foo" ||
    26          this.doc.type == "bar" ||
    27          this.doc.urls.contains("https://www.benthos.dev/").catch(false) {
    28          deleted()
    29        }
    30  ```
    31  
    32  The above config removes any events where:
    33  
    34  - The metadata field `topic` is equal to `foo`
    35  - The event field `doc.type` (a string) is equal to `bar`
    36  - The event field `doc.urls` (an array) contains the string `https://www.benthos.dev/`
    37  
    38  Events that do not match any of these conditions will remain unchanged.
    39  
    40  ## Sample Events
    41  
    42  Another type of filter we might want is a sampling filter, we can do that with a random number generator:
    43  
    44  ```yaml
    45  pipeline:
    46    processors:
    47    - bloblang: |
    48        # Drop 50% of documents randomly
    49        root = if random_int() % 2 == 0 { deleted() }
    50  ```
    51  
    52  We can also do this in a deterministic way by hashing events and filtering by that hash value:
    53  
    54  ```yaml
    55  pipeline:
    56    processors:
    57    - bloblang: |
    58        # Drop ~10% of documents deterministically (same docs filtered each run)
    59        root = if content().hash("xxhash64").slice(-8).number() % 10 == 0 {
    60           deleted()
    61        }
    62  ```
    63  
    64  [processors.bloblang]: /docs/components/processors/bloblang
    65  [bloblang.match]: /docs/guides/bloblang/about#pattern-matching
    66  [bloblang.if]: /docs/guides/bloblang/about#conditional-mapping
    67  [guides.bloblang]: /docs/guides/bloblang/about