github.com/Jeffail/benthos/v3@v3.65.0/website/docs/guides/bloblang/walkthrough.md (about) 1 --- 2 title: Bloblang Walkthrough 3 sidebar_label: Walkthrough 4 description: A step by step introduction to Bloblang 5 --- 6 7 Bloblang is the most advanced mapping language that you'll learn from this walkthrough (probably). It is designed for readability, the power to shape even the most outrageous input documents, and to easily make erratic schemas bend to your will. Bloblang is the native mapping language of Benthos, but it has been designed as a general purpose technology ready to be adopted by other tools. 8 9 In this walkthrough you'll learn how to make new friends by mapping their documents, and lose old friends as they grow jealous and bitter of your mapping abilities. There are a few ways to execute Bloblang but the way we'll do it in this guide is to pull a Benthos docker image and run the command `benthos blobl server`, which opens up an interactive Bloblang editor: 10 11 ```sh 12 docker pull jeffail/benthos:latest 13 docker run -p 4195:4195 --rm jeffail/benthos blobl server --no-open --host 0.0.0.0 14 ``` 15 16 :::note Alternatives 17 For alternative Benthos installation options check out the [getting started guide][guides.getting_started]. 18 ::: 19 20 Next, open your browser at `http://localhost:4195` and you should see an app with three panels, the top-left is where you paste an input document, the bottom is your Bloblang mapping and on the top-right is the output. 21 22 ## Your first assignment 23 24 The primary goal of a Bloblang mapping is to construct a brand new document by using an input document as a reference, which we achieve through a series of assignments. Bloblang is traditionally used to map JSON documents and that's mostly what we'll be doing in this walkthrough. The first mapping you'll see when you open the editor is a single assignment: 25 26 ```coffee 27 root = this 28 ``` 29 30 On the left-hand side of the assignment is our assignment target, where `root` is a keyword referring to the root of the new document being constructed. On the right-hand side is a query which determines the value to be assigned, where `this` is a keyword that refers to the context of the mapping which begins as the root of the input document. 31 32 As you can see the input document in the editor begins as a JSON object `{"message":"hello world"}`, and the output panel should show the result as: 33 34 ```json 35 { 36 "message": "hello world" 37 } 38 ``` 39 40 Which is a (neatly formatted) replica of the input document. This is the result of our mapping because we assigned the entire input document to the root of our new thing. However, you won't get far in life by trapping yourself in the past, let's create a brand new document by assigning a fresh object to the root: 41 42 ```coffee 43 root = {} 44 root.foo = this.message 45 ``` 46 47 Bloblang supports a bunch of [literal types][blobl.literals], and the first line of this mapping assigns an empty object literal to the root. The second line then creates a new field `foo` on that object by assigning it the value of `message` from the input document. You should see that our output has changed to: 48 49 ```json 50 { 51 "foo": "hello world" 52 } 53 ``` 54 55 In Bloblang, when the path that we assign to contains fields that are themselves unset then they are created as empty objects. This rule also applies to `root` itself, which means the mapping: 56 57 ```coffee 58 root.foo.bar = this.message 59 root.foo."buz me".baz = "I like mapping" 60 ``` 61 62 Will automatically create the objects required to produce the output document: 63 64 ```json 65 { 66 "foo": { 67 "bar": "hello world", 68 "buz me": { 69 "baz": "I like mapping" 70 } 71 } 72 } 73 ``` 74 75 Also note that we can use quotes in order to express path segments that contain symbols or whitespace. Great, let's move on quick before our self-satisfaction gets in the way of progress. 76 77 ## Basic Methods and Functions 78 79 Nothing is ever good enough for you, why should the input document be any different? Usually in our mappings it's necessary to mutate values whilst we map them over, this is almost always done with methods, of which [there are many][blobl.methods]. To demonstrate we're going to change our mapping to [uppercase][blobl.methods.uppercase] the field `message` from our input document: 80 81 ```coffee 82 root.foo.bar = this.message.uppercase() 83 root.foo."buz me".baz = "I like mapping" 84 ``` 85 86 As you can see the syntax for a method is similar to many languages, simply add a dot on the target value followed by the method name and arguments within brackets. With this method added our output document should look like this: 87 88 ```json 89 { 90 "foo": { 91 "bar": "HELLO WORLD", 92 "buz me": { 93 "baz": "I like mapping" 94 } 95 } 96 } 97 ``` 98 99 Since the result of any Bloblang query is a value you can use methods on anything, including other methods. For example, we could expand our mapping of `message` to also replace `WORLD` with `EARTH` using the [`replace` method][blobl.methods.replace]: 100 101 ```coffee 102 root.foo.bar = this.message.uppercase().replace("WORLD", "EARTH") 103 root.foo."buz me".baz = "I like mapping" 104 ``` 105 106 As you can see this method required some arguments. Methods support both nameless (like above) and named arguments, which are often literal values but can also be queries themselves. For example try out the following mapping using both named style and a dynamic argument: 107 108 ```coffee 109 root.foo.bar = this.message.uppercase().replace(old: "WORLD", new: this.message.capitalize()) 110 root.foo."buz me".baz = "I like mapping" 111 ``` 112 113 Woah, I think that's the plot to Inception, let's move onto functions. Functions are just boring methods that don't have a target, and there are [plenty of them as well][blobl.functions]. Functions are often used to extract information unrelated to the input document, such as [environment variables][blobl.functions.env], or to generate data such as [timestamps][blobl.functions.now] or [UUIDs][blobl.functions.uuid_v4]. 114 115 Since we're completionists let's add one to our mapping: 116 117 ```coffee 118 root.foo.bar = this.message.uppercase().replace("WORLD", "EARTH") 119 root.foo."buz me".baz = "I like mapping" 120 root.foo.id = uuid_v4() 121 ``` 122 123 Now I can't tell you what the output looks like since it will be different each time it's mapped, how fun! 124 125 ### Deletions 126 127 Everything in Bloblang is an expression to be assigned, including deletions, which is a [function `deleted()`][blobl.functions.deleted]. To illustrate let's create a field we want to delete by changing our input to the following: 128 129 ```json 130 { 131 "name": "fooman barson", 132 "age": 7, 133 "opinions": ["trucks are cool","trains are cool","chores are bad"] 134 } 135 ``` 136 137 If we wanted a full copy of this document without the field `name` then we can assign `deleted()` to it: 138 139 ```coffee 140 root = this 141 root.name = deleted() 142 ``` 143 144 And it won't be included in the output: 145 146 ```json 147 { 148 "age": 7, 149 "opinions": [ 150 "trucks are cool", 151 "trains are cool", 152 "chores are bad" 153 ] 154 } 155 ``` 156 157 An alternative way to delete fields is the [method `without`][blobl.methods.without], our above example could be rewritten as a single assignment `root = this.without("name")`. However, `deleted()` is generally more powerful and will come into play more later on. 158 159 ## Variables 160 161 Sometimes it's necessary to capture a value for later, but we might not want it to be added to the resulting document. In Bloblang we can achieve this with variables which are created using the `let` keyword, and can be referenced within subsequent queries with a dollar sign prefix: 162 163 ```coffee 164 let id = uuid_v4() 165 root.id_sha1 = $id.hash("sha1").encode("hex") 166 root.id_md5 = $id.hash("md5").encode("hex") 167 ``` 168 169 Variables can be assigned any value type, including objects and arrays. 170 171 ## Unstructured and Binary Data 172 173 So far in all of our examples both the input document and our newly mapped document are structured, but this does not need to be so. Try assigning some literal value types directly to the `root`, such as a string `root = "hello world"`, or a number `root = 5`. 174 175 You should notice that when a value type is assigned to the root the output is the raw value, and therefore strings are not quoted. This is what makes it possible to output data of any format, including encrypted, encoded or otherwise binary data. 176 177 Unstructured mapping is not limited to the output. Rather than referencing the input document with `this`, where it must be structured, it is possible to reference it as a binary string with the [function `content`][blobl.functions.content], try changing your mapping to: 178 179 ```coffee 180 root = content().uppercase() 181 ``` 182 183 And then put any old gibberish in the input panel, the output panel should be the same gibberish but all uppercase. 184 185 ## Conditionals 186 187 In order to play around with conditionals let's set our input to something structured: 188 189 ```json 190 { 191 "pet": { 192 "type": "cat", 193 "is_cute": true, 194 "treats": 5, 195 "toys": 3 196 } 197 } 198 ``` 199 200 In Bloblang all conditionals are expressions, this is a core principal of Bloblang and will be important later on when we're mapping deeply nested structures. 201 202 ### If Expression 203 204 The simplest conditional is the `if` expression, where the boolean condition does not need to be in parentheses. Let's create a map that modifies the number of treats our pet receives based on a field: 205 206 ```coffee 207 root = this 208 root.pet.treats = if this.pet.is_cute { 209 this.pet.treats + 10 210 } 211 ``` 212 213 Try that mapping out and you should see the number of treats in the output increased to 15. Now try changing the input field `pet.is_cute` to `false` and the output treats count should go back to the original 5. 214 215 When a conditional expression doesn't have a branch to execute then the assignment is skipped entirely, which means when the pet is not cute the value of `pet.treats` is unchanged (and remains the value set in the `root = this` assignment). 216 217 We can add an `else` block to our `if` expression to remove treats entirely when the pet is not cute: 218 219 ```coffee 220 root = this 221 root.pet.treats = if this.pet.is_cute { 222 this.pet.treats + 10 223 } else { 224 deleted() 225 } 226 ``` 227 228 This is possible because field deletions are expressed as assigned values created with the `deleted()` function. This is cool but also in poor taste, treats should be allocated based on need, not cuteness! 229 230 ### Match Expression 231 232 Another conditional expression is `match` which allows you to list many branches consisting of a condition and a query to execute separated with `=>`, where the first condition to pass is the one that is executed: 233 234 ```coffee 235 root = this 236 root.pet.toys = match { 237 this.pet.treats > 5 => this.pet.treats - 5, 238 this.pet.type == "cat" => 3, 239 this.pet.type == "dog" => this.pet.toys - 3, 240 this.pet.type == "horse" => this.pet.toys + 10, 241 _ => 0, 242 } 243 ``` 244 245 Try executing that mapping with different values for `pet.type` and `pet.treats`. Match expressions can also specify a new context for the keyword `this` which can help reduce some of the boilerplate in your boolean conditions. The following mapping is equivalent to the previous: 246 247 ```coffee 248 root = this 249 root.pet.toys = match this.pet { 250 this.treats > 5 => this.treats - 5, 251 this.type == "cat" => 3, 252 this.type == "dog" => this.toys - 3, 253 this.type == "horse" => this.toys + 10, 254 _ => 0, 255 } 256 ``` 257 258 Your boolean conditions can also be expressed as value types, in which case the context being matched will be compared to the value: 259 260 ```coffee 261 root = this 262 root.pet.toys = match this.pet.type { 263 "cat" => 3, 264 "dog" => 5, 265 "rabbit" => 8, 266 "horse" => 20, 267 _ => 0, 268 } 269 ``` 270 271 ## Error Handling 272 273 Are you feeling relaxed? Well don't, because in the world of mapping anything can happen, at ANY TIME, and there are plenty of ways that a mapping can fail due to variations in the input data. Are you feeling stressed? Well don't, because Bloblang makes handling errors easy. 274 275 First, let's take a look at what happens when errors _aren't_ handled, change your input to the following: 276 277 ```json 278 { 279 "palace_guards": 10, 280 "angry_peasants": "I couldn't be bothered to ask them" 281 } 282 ``` 283 284 And change your mapping to something simple like a number comparison: 285 286 ```coffee 287 root.in_trouble = this.angry_peasants > this.palace_guards 288 ``` 289 290 Uh oh! It looks like our canvasser was too lazy and our `angry_peasants` count was incorrectly set for this document. You should see an error in the output window that mentions something like `cannot compare types string (from field this.angry_peasants) and number (from field this.palace_guards)`, which means the mapping was abandoned. 291 292 So what if we want to try and map something, but don't care if it fails? In this case if we are unable to compare our angry peasants with palace guards then I would still consider us in trouble just to be safe. 293 294 For that we have a special [method `catch`][blobl.methods.catch], which if we add to any query allows us to specify an argument to be returned when an error occurs. Since methods can be added to any query we can surround our arithmetic with brackets and catch the whole thing: 295 296 ```coffee 297 root.in_trouble = (this.angry_peasants > this.palace_guards).catch(true) 298 ``` 299 300 Now instead of an error we should see an output with `in_trouble` set to `true`. Try changing to value of `angry_peasants` to a few different values, including some numbers. 301 302 One of the powerful features of `catch` is that when it is added at the end of a series of expressions and methods it will capture errors at any part of the series, allowing you to capture errors at any granularity. For example, the mapping: 303 304 ```coffee 305 root.abort_mission = if this.mission.type == "impossible" { 306 !this.user.motives.contains("must clear name") 307 } else { 308 this.mission.difficulty > 10 309 }.catch(false) 310 ``` 311 312 Will catch errors caused by: 313 314 - `this.mission.type` not being a string 315 - `this.user.motives` not being an array 316 - `this.mission.difficulty` not being a number 317 318 But will always return `false` if any of those errors occur. Try it out with this input and play around by breaking some of the fields: 319 320 ```json 321 { 322 "mission": { 323 "type": "impossible", 324 "difficulty": 5 325 }, 326 "user": { 327 "motives": ["must clear name"] 328 } 329 } 330 ``` 331 332 Now try out this mapping: 333 334 ```coffee 335 root.abort_mission = if (this.mission.type == "impossible").catch(true) { 336 !this.user.motives.contains("must clear name").catch(false) 337 } else { 338 (this.mission.difficulty > 10).catch(true) 339 } 340 ``` 341 342 This version is more granular and will capture each of the errors individually, with each error given a unique `true` or `false` fallback. 343 344 ## Validation 345 346 I'm worried that I've turned you into some sort of error hating thug, hell-bent on eliminating all errors from existence. However, sometimes errors are what we want. Failing a mapping with an error allows us to handle the bad document in other ways, such as routing it to a dead-letter queue or filtering it entirely. 347 348 You can read about common Benthos error handling patterns for bad data in the [error handling guide][configuration.error_handling], but the first step is to create the error. Luckily, Bloblang has a range of ways of creating errors under certain circumstances, which can be used in order to validate the data being mapped. 349 350 There are [a few helper methods][blobl.methods.coercion] that make validating and coercing fields nice and easy, try this mapping out: 351 352 ```coffee 353 root.foo = this.foo.number() 354 root.bar = this.bar.not_null() 355 root.baz = this.baz.not_empty() 356 ``` 357 358 With some of these sample inputs: 359 360 ```json 361 {"foo":"nope","bar":"hello world","baz":[1,2,3]} 362 {"foo":5,"baz":[1,2,3]} 363 {"foo":10,"bar":"hello world","baz":[]} 364 ``` 365 366 However, these methods don't cover all use cases. The general purpose error throwing technique is the [`throw` function][blobl.functions.throw], which takes an argument string that describes the error. When it's called it will throw a mapping error that abandons the mapping (unless it's caught, psych!) 367 368 For example, we can check the type of a field with the [method `type`][blobl.methods.type], and then throw an error if it's not the type we expected: 369 370 ```coffee 371 root.foos = if this.user.foos.type() == "array" { 372 this.user.foos 373 } else { 374 throw("foos must be an array, but it ain't, what gives?") 375 } 376 ``` 377 378 Try this mapping out with a few sample inputs: 379 380 ```json 381 {"user":{"foos":[1,2,3]}} 382 {"user":{"foos":"1,2,3"}} 383 ``` 384 385 ## Context 386 387 In Bloblang, when we refer to the context we're talking about the value returned with the keyword `this`. At the beginning of a mapping the context starts off as a reference to the root of a structured input document, which is why the mapping `root = this` will result in the same document coming out as you put in. 388 389 However, in Bloblang there are mechanisms whereby the context might change, we've already seen how this can happen within a `match` expression. Another useful way to change the context is by adding a bracketed query expression as a method to a query, which looks like this: 390 391 ```coffee 392 root = this.foo.bar.(this.baz + this.buz) 393 ``` 394 395 Within the bracketed query expression the context becomes the result of the query that it's a method of, so within the brackets in the above mapping the value of `this` points to the result of `this.foo.bar`, and the mapping is therefore equivalent to: 396 397 ```coffee 398 root = this.foo.bar.baz + this.foo.bar.buz 399 ``` 400 401 With this handy trick the `throw` mapping from the validation section above could be rewritten as: 402 403 ```coffee 404 root.foos = this.user.foos.(if this.type() == "array" { this } else { 405 throw("foos must be an array, but it ain't, what gives?") 406 }) 407 ``` 408 409 ### Naming the Context 410 411 Shadowing the keyword `this` with new contexts can look confusing in your mappings, and it also limits you to only being able to reference one context at any given time. As an alternative, Bloblang supports context capture expressions that look similar to lambda functions from other languages, where you can name the new context with the syntax `<context name> -> <query>`, which looks like this: 412 413 ```coffee 414 root = this.foo.bar.(thing -> thing.baz + thing.buz) 415 ``` 416 417 Within the brackets we now have a new field `thing`, which returns the context that would have otherwise been captured as `this`. This also means the value returned from `this` hasn't changed and will continue to return the root of the input document. 418 419 ## Coalescing 420 421 Being able to open up bracketed query expressions on fields leads us onto another cool trick in Bloblang referred to as coalescing. It's very common in the world of document mapping that due to structural deviations a value that we wish to obtain could come from one of multiple possible paths. 422 423 To illustrate this problem change the input document to the following: 424 425 ```json 426 { 427 "thing": { 428 "article": { 429 "id": "foo", 430 "contents": "Some people did some stuff" 431 } 432 } 433 } 434 ``` 435 436 Let's say we wish to flatten this structure with the following mapping: 437 438 ```coffee 439 root.contents = this.thing.article.contents 440 ``` 441 442 But articles are only one of many document types we expect to receive, where the field `contents` remains the same but the field `article` could instead be `comment` or `share`. In this case we could expand our map of `contents` to use a `match` expression where we check for the existence of `article`, `comment`, etc in the input document. 443 444 However, a much cleaner way of approaching this is with the pipe operator (`|`), which in Bloblang can be used to join multiple queries, where the first to yield a non-null result is selected. Change your mapping to the following: 445 446 ```coffee 447 root.contents = this.thing.article.contents | this.thing.comment.contents 448 ``` 449 450 And now try changing the field `article` in your input document to `comment`. You should see that the value of `contents` remains as `Some people did some stuff` in the output document. 451 452 Now, rather than write out the full path prefix `this.thing` each time we can use a bracketed query expression to change the context, giving us more space for adding other fields: 453 454 ```coffee 455 root.contents = this.thing.(this.article | this.comment | this.share).contents 456 ``` 457 458 And by the way, the keyword `this` within queries can be omitted and made implicit, which allows us to reduce this even further: 459 460 ```coffee 461 root.contents = this.thing.(article | comment | share).contents 462 ``` 463 464 Finally, we can also add a pipe operator at the end to fallback to a literal value when none of our candidates exists: 465 466 ```coffee 467 root.contents = this.thing.(article | comment | share).contents | "nothing" 468 ``` 469 470 Neat. 471 472 ## Advanced Methods 473 474 Congratulations for making it this far, but if you take your current level of knowledge to a map-off you'll be laughed off the stage. What happens when you need to map all of the elements of an array? Or filter the keys of an object by their values? What if the fellowship just used the eagles to fly to mount doom? 475 476 Bloblang offers a bunch of advanced methods for [manipulating structured data types][blobl.methods.object-array-manipulation], let's take a quick tour of some of the cooler ones. Set your input document to this list of things: 477 478 ```json 479 { 480 "num_friends": 5, 481 "things": [ 482 { 483 "name": "yo-yo", 484 "quantity": 10, 485 "is_cool": true 486 }, 487 { 488 "name": "dish soap", 489 "quantity": 50, 490 "is_cool": false 491 }, 492 { 493 "name": "scooter", 494 "quantity": 1, 495 "is_cool": true 496 }, 497 { 498 "name": "pirate hat", 499 "quantity": 7, 500 "is_cool": true 501 } 502 ] 503 } 504 ``` 505 506 Let's say we wanted to reduce the `things` in our input document to only those that are cool and where we have enough of them to share with our friends. We can do this with a [`filter` method][blobl.methods.filter]: 507 508 ```coffee 509 root = this.things.filter(thing -> thing.is_cool && thing.quantity > this.num_friends) 510 ``` 511 512 Try running that mapping and you'll see that the output is reduced. What is happening here is that the `filter` method takes an argument that is a query, and that query will be mapped for each individual element of the array (where the context is changed to the element itself). We have captured the context into a field `thing` which allows us to continue referencing the root of the input with `this`. 513 514 The `filter` method requires the query parameter to resolve to a boolean `true` or `false`, and if it resolves to `true` the element will be present in the resulting array, otherwise it is removed. 515 516 Being able to express a query argument to be applied to a range in this way is one of the more powerful features of Bloblang, and when mapping complex structured data these advanced methods will likely be a common tool that you'll reach for. 517 518 Another such method is [`map_each`][blobl.methods.map_each], which allows you to mutate each element of an array, or each value of an object. Change your input document to the following: 519 520 ```json 521 { 522 "talking_heads": [ 523 "1:E.T. is a bad film,Pokemon corrupted an entire generation", 524 "2:Digimon ripped off Pokemon,Cats are boring", 525 "3:I'm important", 526 "4:Science is just made up,The Pokemon films are good,The weather is good" 527 ] 528 } 529 ``` 530 531 Here we have an array of talking heads, where each element is a string containing an identifer, a colon, and a comma separated list of their opinions. We wish to map each string into a structured object, which we can do with the following mapping: 532 533 ```coffee 534 root = this.talking_heads.map_each(raw -> { 535 "id": raw.split(":").index(0), 536 "opinions": raw.split(":").index(1).split(",") 537 }) 538 ``` 539 540 The argument to `map_each` is a query where the context is the element, which we capture into the field `raw`. The result of the query argument will become the value of the element in the resulting array, and in this case we return an object literal. 541 542 In order to separate the identifier from opinions we perform a `split` by colon on the raw string element and get the first substring with the `index` method. We then do the split again and extract the remainder, and split that by comma in order to extract all of the opinions to an array field. 543 544 However, one problem with this mapping is that the split by colon is written out twice and executed twice. A more efficient way of performing the same thing is with the bracketed query expressions we've played with before: 545 546 ```coffee 547 root = this.talking_heads.map_each(raw -> raw.split(":").(split_string -> { 548 "id": split_string.index(0), 549 "opinions": split_string.index(1).split(",") 550 })) 551 ``` 552 553 :::note Challenge! 554 Try updating that map so that only opinions that mention Pokemon are kept 555 ::: 556 557 558 Cool. To find more methods for manipulating structured data types check out the [methods page][blobl.methods.object-array-manipulation]. 559 560 ## Reusable Mappings 561 562 Bloblang has cool methods, sure, but there's nothing cooler than methods you've made yourself. When the going gets tough in the mapping world the best solution is often to create a named mapping, which you can do with the keyword `map`: 563 564 ```coffee 565 map parse_talking_head { 566 let split_string = this.split(":") 567 568 root.id = $split_string.index(0) 569 root.opinions = $split_string.index(1).split(",") 570 } 571 572 root = this.talking_heads.map_each(raw -> raw.apply("parse_talking_head")) 573 ``` 574 575 The body of a named map, encapsulated with squiggly brackets, is a totally isolated mapping where `root` now refers to a new value being created for each invocation of the map, and `this` refers to the root of the context provided to the map. 576 577 Named maps are executed with the [method `apply`][blobl.methods.apply], which has a string parameter identifying the map to execute, this means it's possible to dynamically select the target map. 578 579 As you can see in the above example we were able to use a custom map in order to create our talking head objects without the object literal. Within a named map we can also create variables that exist only within the scope of the map. 580 581 A cool feature of named mappings is that they can invoke themselves recursively, allowing you to define mappings that walk deeply nested structures. The following mapping will scrub all values from a document that contain the word "Voldemort" (case insensitive): 582 583 ```coffee 584 map remove_naughty_man { 585 root = match { 586 this.type() == "object" => this.map_each(item -> item.value.apply("remove_naughty_man")), 587 this.type() == "array" => this.map_each(ele -> ele.apply("remove_naughty_man")), 588 this.type() == "string" => if this.lowercase().contains("voldemort") { deleted() }, 589 this.type() == "bytes" => if this.lowercase().contains("voldemort") { deleted() }, 590 _ => this, 591 } 592 } 593 594 root = this.apply("remove_naughty_man") 595 ``` 596 597 Try running that mapping with the following input document: 598 599 ```json 600 { 601 "summer_party": { 602 "theme": "the woman in black", 603 "guests": [ 604 "Emma Bunton", 605 "the seal I spotted in Trebarwith", 606 "Voldemort", 607 "The cast of Swiss Army Man", 608 "Richard" 609 ], 610 "notes": { 611 "lisa": "I don't think voldemort eats fish", 612 "monty": "Seals hate dance music" 613 } 614 }, 615 "crushes": [ 616 "Richard is nice but he hates pokemon", 617 "Victoria Beckham but I think she's taken", 618 "Charlie but they're totally into Voldemort" 619 ] 620 } 621 ``` 622 623 Charlie will be upset but at least we'll be safe. 624 625 ## Unit Testing 626 627 You are truly a champion of mappings, and you're probably feeling pretty confident right now. Maybe you even have a mapping that you're particularly proud of. Well, I'm sorry to inform you that your mapping is DOOMED, as a mapping without unit tests is like a Twitter session, with the progression of time it will inevitably descend into madness. 628 629 However, if you act now there is still time to spare your mapping from this fate, as Benthos has it's own [unit testing capabilities][configuration.unit_testing] that you can also use for your mappings. To start with save a mapping into a file called something like `naughty_man.blobl`, we can use the example above from the reusable mappings section: 630 631 ```coffee 632 map remove_naughty_man { 633 root = match { 634 this.type() == "object" => this.map_each(item -> item.value.apply("remove_naughty_man")), 635 this.type() == "array" => this.map_each(ele -> ele.apply("remove_naughty_man")), 636 this.type() == "string" => if this.lowercase().contains("voldemort") { deleted() }, 637 this.type() == "bytes" => if this.lowercase().contains("voldemort") { deleted() }, 638 _ => this, 639 } 640 } 641 642 root = this.apply("remove_naughty_man") 643 ``` 644 645 Next, we can define our unit tests in an accompanying YAML file in the same directory, let's call this `naughty_man_test.yaml`: 646 647 ```yaml 648 tests: 649 - name: test naughty man scrubber 650 target_mapping: './naughty_man.blobl' 651 environment: {} 652 input_batch: 653 - content: | 654 { 655 "summer_party": { 656 "theme": "the woman in black", 657 "guests": [ 658 "Emma Bunton", 659 "the seal I spotted in Trebarwith", 660 "Voldemort", 661 "The cast of Swiss Army Man", 662 "Richard" 663 ] 664 } 665 } 666 output_batches: 667 - 668 - json_equals: { 669 "summer_party": { 670 "theme": "the woman in black", 671 "guests": [ 672 "Emma Bunton", 673 "the dolphin I spotted in Trebarwith", 674 "The cast of Swiss Army Man", 675 "Richard" 676 ] 677 } 678 } 679 ``` 680 681 As you can see we've defined a single test, where we point to our mapping file which will be executed in our test. We then specify an input message which is a reduced version of the document we tried out before, and finally we specify output predicates, which is a JSON comparison against the output document. 682 683 We can execute these tests with `benthos test ./naught_man_test.yaml`, Benthos will also automatically find our tests if you simply run `benthos test ./...`. You should see an output something like: 684 685 ```text 686 Test 'naughty_man_test.yaml' failed 687 688 Failures: 689 690 --- naughty_man_test.yaml --- 691 692 test naughty man scrubber [line 2]: 693 batch 0 message 0: json_equals: JSON content mismatch 694 { 695 "summer_party": { 696 "guests": [ 697 "Emma Bunton", 698 "the seal I spotted in Trebarwith" => "the dolphin I spotted in Trebarwith", 699 "The cast of Swiss Army Man", 700 "Richard" 701 ], 702 "theme": "the woman in black" 703 } 704 } 705 ``` 706 707 Because in actual fact our expected output is wrong, I'll leave it to you to spot the error. Once the test is fixed you should see: 708 709 ```text 710 Test 'naughty_man_test.yaml' succeeded 711 ``` 712 713 And now our mapping, should we need to expand it in the future, is better protected against regressions. You can read more about the Benthos unit test specification, including alternative output predicates, in [this document][configuration.unit_testing]. 714 715 ## Final Words 716 717 That's it for this walkthrough, if you're hungry for more then I suggest you re-evaluate your priorities in life. If you have feedback then please [get in touch][community], despite being terrible people the Benthos community are very welcoming. 718 719 [guides.getting_started]: /docs/guides/getting_started 720 [blobl.methods]: /docs/guides/bloblang/methods 721 [blobl.methods.uppercase]: /docs/guides/bloblang/methods#uppercase 722 [blobl.methods.replace]: /docs/guides/bloblang/methods#replace 723 [blobl.methods.catch]: /docs/guides/bloblang/methods#catch 724 [blobl.methods.without]: /docs/guides/bloblang/methods#without 725 [blobl.methods.type]: /docs/guides/bloblang/methods#type 726 [blobl.methods.coercion]: /docs/guides/bloblang/methods#type-coercion 727 [blobl.methods.object-array-manipulation]: /docs/guides/bloblang/methods#object--array-manipulation 728 [blobl.methods.filter]: /docs/guides/bloblang/methods#filter 729 [blobl.methods.map_each]: /docs/guides/bloblang/methods#map_each 730 [blobl.methods.apply]: /docs/guides/bloblang/methods#apply 731 [blobl.functions]: /docs/guides/bloblang/functions 732 [blobl.functions.deleted]: /docs/guides/bloblang/functions#deleted 733 [blobl.functions.content]: /docs/guides/bloblang/functions#content 734 [blobl.functions.env]: /docs/guides/bloblang/functions#env 735 [blobl.functions.now]: /docs/guides/bloblang/functions#now 736 [blobl.functions.uuid_v4]: /docs/guides/bloblang/functions#uuid_v4 737 [blobl.functions.throw]: /docs/guides/bloblang/functions#throw 738 [blobl.literals]: /docs/guides/bloblang/about#literals 739 [configuration.error_handling]: /docs/configuration/error_handling 740 [configuration.unit_testing]: /docs/configuration/unit_testing 741 [community]: /community