github.com/Jeffail/benthos/v3@v3.65.0/website/blog/2021-06-02-new-plugins-and-templates.md (about)

     1  ---
     2  title: 'Preview: Go Plugins V2 and Config Templates'
     3  author: Ashley Jeffs
     4  author_url: https://github.com/Jeffail
     5  author_image_url: /img/ash.jpg
     6  description: It's ready, now we need test subjects
     7  keywords: [
     8      "go",
     9      "golang",
    10      "stream processor",
    11      "ETL",
    12  ]
    13  tags: [ "v4", "plugins", "templates", "roadmap" ]
    14  ---
    15  
    16  I need help, attention and affirmation, and therefore it's time for a development update. Around five months ago I posted a [roadmap for Benthos v4](/blog/2021/01/04/v4-roadmap) that included some utterly unattainable goals that only a super human could achieve.
    17  
    18  Now that most of those features are ready to test, namely a new plugins API and config templating, I'm looking for people to try them out and give feedback. Please read on if that sounds like fun to you, or also if it doesn't sound fun but you intend to do it anyway.
    19  
    20  <!--truncate-->
    21  
    22  ## Config Templates
    23  
    24  The new config templates functionality allows you to define parameterised templates for Benthos configuration snippets. These templates can then be imported with a cli flag and used in Benthos configs like native Benthos components.
    25  
    26  This is going to be super useful in situations where you have commonly used configuration patterns with small differences that prevent you from using resources.
    27  
    28  The current state of templates is that they'll be included in the next release as an experimental feature, meaning any aspect of this functionality is subject to change outside of major version releases. This includes the config spec of templates, how they work, and so on.
    29  
    30  Defining a template looks roughly like this:
    31  
    32  ```yaml
    33  name: log_message
    34  type: processor
    35  summary: Print a log line that shows the contents of a message.
    36  
    37  fields:
    38    - name: level
    39      description: The level to log at.
    40      type: string
    41      default: INFO
    42  
    43  mapping: |
    44    root.log.level = this.level
    45    root.log.message = "${! content() }"
    46    root.log.fields.metadata = "${! meta() }"
    47    root.log.fields.error = "${! error() }"
    48  ```
    49  
    50  And you're able to import templates with the `-t` flag:
    51  
    52  ```sh
    53  benthos -t ./templates/foo.yaml -c ./config.yaml
    54  ```
    55  
    56  And using it in a config looks like any other component:
    57  
    58  ```yaml
    59  pipeline:
    60    processors:
    61      - log_message:
    62          level: ERROR
    63  ```
    64  
    65  To find out more about configuration templates, including how to try them out, check out [the new templates page][configuration.templating]. More importantly, you can give feedback on them [in this Github discussion][templates-feedback-thread].
    66  
    67  ## The V2 Go Plugins API
    68  
    69  Benthos has had Go plugins for a while now and they're fairly well received. However, they can sometimes be confusing as they expose Benthos internals that aren't necessary to understand as plugin authors.
    70  
    71  It was also an issue for me as a maintainer that the current plugin APIs hook directly into Benthos packages that have no business being public. This makes it extra difficult to improve the service without introducing breaking changes.
    72  
    73  The new APIs are simpler, more powerful (in the ways that matter), add milk after the water, and most importantly are air-gapped from Benthos internals so that they can evolve independently. Here's a sneaky glance of what a processor plugin looks like:
    74  
    75  ```go
    76  type ReverseProcessor struct {
    77  	logger *service.Logger
    78  }
    79  
    80  func (r *ReverseProcessor) Process(ctx context.Context, m *service.Message) ([]*service.Message, error) {
    81  	bytesContent, err := m.AsBytes()
    82  	if err != nil {
    83  		return nil, err
    84  	}
    85  
    86  	newBytes := make([]byte, len(bytesContent))
    87  	for i, b := range bytesContent {
    88  		newBytes[len(newBytes)-i-1] = b
    89  	}
    90  
    91  	if bytes.Equal(newBytes, bytesContent) {
    92  		r.logger.Infof("Woah! This is like totally a palindrome: %s", bytesContent)
    93  	}
    94  
    95  	m.SetBytes(newBytes)
    96  	return []*service.Message{m}, nil
    97  }
    98  
    99  func (r *ReverseProcessor) Close(ctx context.Context) error {
   100  	return nil
   101  }
   102  
   103  func main() {
   104  	err := service.RegisterProcessor(
   105  		"reverse", service.NewConfigSpec(),
   106  		func(conf *service.ParsedConfig, mgr *service.Resources) (service.Processor, error) {
   107  			return &ReverseProcessor{logger: mgr.Logger()}, nil
   108  		})
   109  	if err != nil {
   110  		panic(err)
   111  	}
   112  
   113  	service.RunCLI()
   114  }
   115  ```
   116  
   117  You can play around with these APIs right now by pulling the latest commit with:
   118  
   119  ```sh
   120  go get -u github.com/Jeffail/benthos/v3@master
   121  ```
   122  
   123  And you can find more examples along with the API docs at [pkg.go.dev][plugins.api].
   124  
   125  The package will remain in an experimental state under `public/x/service` for a month or so, and once it's "ready" (I'm personally happy with it) then it'll be moved to `public/service` and will be considered stable.
   126  
   127  The goal is to allow everyone to migrate to the new APIs whilst still supporting the old ones, and then when Benthos V4 is tagged the old ones will vanish and we're no longer blocked on them.
   128  
   129  Similar to the templates there is [a Github discussion open for feedback][plugins-feedback-thread]. Be honest, be brutal.
   130  
   131  ## Join the Community
   132  
   133  I've been babbling on for months so if this stuff is news to you then you're clearly out of the loop. Worry not, for you can remedy the situation by joining one or more of our [glorious community spaces][community].
   134  
   135  [community]: /community
   136  [configuration.templating]: /docs/configuration/templating
   137  [plugins.api]: https://pkg.go.dev/github.com/Jeffail/benthos/v3/public/service
   138  [templates-feedback-thread]: https://github.com/Jeffail/benthos/discussions/785
   139  [plugins-feedback-thread]: https://github.com/Jeffail/benthos/discussions/754