github.com/Jeffail/benthos/v3@v3.65.0/website/cookbooks/filtering.md (about) 1 --- 2 id: filtering 3 title: Filtering and Sampling 4 description: Configure Benthos to conditionally drop messages. 5 --- 6 7 Events are like eyebrows, sometimes it's best to just get rid of them. Filtering events in Benthos is both easy and flexible, this cookbook demonstrates a few different types of filtering you can do. All of these examples make use of the [`bloblang` processor][processors.bloblang] but shouldn't require any prior knowledge. 8 9 ## The Basic Filter 10 11 Dropping events with [Bloblang][guides.bloblang] is done by mapping the function `deleted()` to the `root` of the mapped document. To remove all events indiscriminantly you can simply do: 12 13 ```yaml 14 pipeline: 15 processors: 16 - bloblang: root = deleted() 17 ``` 18 19 But that's most likely not what you want. We can instead only delete an event under certain conditions with a [`match`][bloblang.match] or [`if`][bloblang.if] expression: 20 21 ```yaml 22 pipeline: 23 processors: 24 - bloblang: | 25 root = if meta("topic").or("") == "foo" || 26 this.doc.type == "bar" || 27 this.doc.urls.contains("https://www.benthos.dev/").catch(false) { 28 deleted() 29 } 30 ``` 31 32 The above config removes any events where: 33 34 - The metadata field `topic` is equal to `foo` 35 - The event field `doc.type` (a string) is equal to `bar` 36 - The event field `doc.urls` (an array) contains the string `https://www.benthos.dev/` 37 38 Events that do not match any of these conditions will remain unchanged. 39 40 ## Sample Events 41 42 Another type of filter we might want is a sampling filter, we can do that with a random number generator: 43 44 ```yaml 45 pipeline: 46 processors: 47 - bloblang: | 48 # Drop 50% of documents randomly 49 root = if random_int() % 2 == 0 { deleted() } 50 ``` 51 52 We can also do this in a deterministic way by hashing events and filtering by that hash value: 53 54 ```yaml 55 pipeline: 56 processors: 57 - bloblang: | 58 # Drop ~10% of documents deterministically (same docs filtered each run) 59 root = if content().hash("xxhash64").slice(-8).number() % 10 == 0 { 60 deleted() 61 } 62 ``` 63 64 [processors.bloblang]: /docs/components/processors/bloblang 65 [bloblang.match]: /docs/guides/bloblang/about#pattern-matching 66 [bloblang.if]: /docs/guides/bloblang/about#conditional-mapping 67 [guides.bloblang]: /docs/guides/bloblang/about