github.com/Jeffail/benthos/v3@v3.65.0/website/blog/2020-08-30-improved-workflows.md (about)

     1  ---
     2  title: Powered Up Workflows
     3  author: Ashley Jeffs
     4  author_url: https://github.com/Jeffail
     5  author_image_url: /img/ash.jpg
     6  description: Available in v3.26.0
     7  keywords: [
     8      "benthos",
     9      "workflows",
    10      "go",
    11      "golang",
    12      "stream processor",
    13      "enrichments",
    14  ]
    15  tags: [ "Workflows" ]
    16  ---
    17  
    18  For the last few weeks I've been working on improving the workflow story in Benthos. That means reducing the number of processors, simplifying them, and at the same time making them more powerful than before. The new functionality outlined here can be used in the latest release [v3.26.0](https://github.com/Jeffail/benthos/releases/tag/v3.26.0).
    19  
    20  <!--truncate-->
    21  
    22  ## The Motivation
    23  
    24  After similar efforts to [improve the mapping story][post.bloblang-beta] in Benthos it seemed sensible to target workflows. Specifically, I've added a new [`branch` processor][processor.branch] for wrapping child processors in request/result maps, and have reworked the [`workflow` processor][processor.workflow] to use them.
    25  
    26  If you haven't used workflows in Benthos then there's a section in the new [`workflow` processor][processor.workflow.why] page outlining why they're useful. In short, when performing multiple integrations within a pipeline such as hitting HTTP services, lambdas, caches, etc, it's best to perform them in parallel when possible in order to reduce the processing latency of messages, organizing these integrations into a topology with a workflow makes it easier to manage their interdependencies and ensure they're executed in the right order.
    27  
    28  In the old world you could use the `process_dag` processor which has child `process_map` processors, where the mappings were a series of clunky to/from [dot paths][configuration.field_paths], separated into optional and non-optional mappings. There was no way to manually specify the dependency tree, and conditional flows required a separate list of conditions which didn't factor into dependency resolution.
    29  
    30  Having such complex and brittle mapping capabilities meant these processors were difficult to document and more so to understand and use.
    31  
    32  ## Leaning into Bloblang
    33  
    34  Thankfully, with [Bloblang][guides.bloblang] now finished it was pretty easy to replace most of the complexity of the workflow mappings for the language itself.
    35  
    36  For example, when mapping the request payload for an integration you can express a bunch of different patterns...
    37  
    38  Empty request body:
    39  
    40  ```yaml
    41  request_map: root = ""
    42  ```
    43  
    44  Sub-object (`foo`) as request body, if the sub-object doesn't exist (or is null) the integration is abandoned:
    45  
    46  ```yaml
    47  request_map: root = this.foo.not_null()
    48  ```
    49  
    50  Sub-object as request body which can be obtained from one of a number of possible paths:
    51  
    52  ```yaml
    53  request_map: root = this.(foo | bar | baz).doc.not_null()
    54  ```
    55  
    56  Conditional integration applies when the `type` is `foo`, with an unmodified message as request body:
    57  
    58  ```yaml
    59  request_map: |
    60    root = if this.type != "foo" {
    61      deleted()
    62    }
    63  ```
    64  
    65  Conditional integration applies when the `type` is `foo`, with a sub-object as the request body:
    66  
    67  ```yaml
    68  request_map: |
    69    root = if this.type == "foo" {
    70      this.foo.not_null()
    71    } else {
    72      deleted()
    73    }
    74  ```
    75  
    76  Similarly, it's possible to express a bunch of things in the result mapping...
    77  
    78  Discard the result (the original message is unchanged):
    79  
    80  ```yaml
    81  result_map: ""
    82  ```
    83  
    84  Place the entire result at a path:
    85  
    86  ```yaml
    87  result_map: root.foo = this
    88  ```
    89  
    90  Place the result in a metadata field:
    91  
    92  ```yaml
    93  result_map: meta foo = this
    94  ```
    95  
    96  If you want to see what it looks like there is an [enrichment cookbook][cookbook.enrichment] that demonstrates workflows in action, but there are also smaller examples on the [workflow page][processor.workflow.examples] such as the following snippet:
    97  
    98  ```yaml
    99  pipeline:
   100    processors:
   101      - workflow:
   102          meta_path: meta.workflow
   103          branches:
   104            foo:
   105              request_map: 'root = ""'
   106              processors:
   107                - http:
   108                    url: TODO
   109              result_map: 'root.foo = this'
   110  
   111            bar:
   112              request_map: 'root = this.body'
   113              processors:
   114                - lambda:
   115                    function: TODO
   116              result_map: 'root.bar = this'
   117  
   118            baz:
   119              request_map: |
   120                root.fooid = this.foo.id
   121                root.barstuff = this.bar.content
   122              processors:
   123                - cache:
   124                    resource: TODO
   125                    operator: set
   126                    key: ${! json("fooid") }
   127                    value: ${! json("barstuff") }
   128  ```
   129  
   130  ## Conclusion
   131  
   132  The docs have been updated to use these new goodies. Obviously the old processors are still being maintained but in a mostly dormant state. The workflow and branch processors are currently labelled as `beta`, but their general behavior is stable with the only exceptions being odd edge cases that might arise.
   133  
   134  With the behavior of these processors being dramatically simplified I've also been able to simplify the documentation for them, which also means using more space on the page for example configs.
   135  
   136  If you have feedback then [get the absolute heck in the chat you utter recluse][community].
   137  
   138  [processor.workflow]: /docs/components/processors/workflow/
   139  [processor.branch]: /docs/components/processors/branch/
   140  [processor.workflow.why]: /docs/components/processors/workflow/#why-use-a-workflow
   141  [processor.workflow.examples]: /docs/components/processors/workflow/#examples
   142  [post.bloblang-beta]: /blog/2020/05/10/bloblang-beta/
   143  [configuration.field_paths]: /docs/configuration/field_paths/
   144  [cookbook.enrichment]: /cookbooks/enrichments/
   145  [guides.bloblang]: /docs/guides/bloblang/about/
   146  [community]: /community/