github.com/hashicorp/go-plugin@v1.6.0/README.md (about)

     1  # Go Plugin System over RPC
     2  
     3  `go-plugin` is a Go (golang) plugin system over RPC. It is the plugin system
     4  that has been in use by HashiCorp tooling for over 4 years. While initially
     5  created for [Packer](https://www.packer.io), it is additionally in use by
     6  [Terraform](https://www.terraform.io), [Nomad](https://www.nomadproject.io),
     7  [Vault](https://www.vaultproject.io),
     8  [Boundary](https://www.boundaryproject.io),
     9  and [Waypoint](https://www.waypointproject.io).
    10  
    11  While the plugin system is over RPC, it is currently only designed to work
    12  over a local [reliable] network. Plugins over a real network are not supported
    13  and will lead to unexpected behavior.
    14  
    15  This plugin system has been used on millions of machines across many different
    16  projects and has proven to be battle hardened and ready for production use.
    17  
    18  ## Features
    19  
    20  The HashiCorp plugin system supports a number of features:
    21  
    22  **Plugins are Go interface implementations.** This makes writing and consuming
    23  plugins feel very natural. To a plugin author: you just implement an
    24  interface as if it were going to run in the same process. For a plugin user:
    25  you just use and call functions on an interface as if it were in the same
    26  process. This plugin system handles the communication in between.
    27  
    28  **Cross-language support.** Plugins can be written (and consumed) by
    29  almost every major language. This library supports serving plugins via
    30  [gRPC](http://www.grpc.io). gRPC-based plugins enable plugins to be written
    31  in any language.
    32  
    33  **Complex arguments and return values are supported.** This library
    34  provides APIs for handling complex arguments and return values such
    35  as interfaces, `io.Reader/Writer`, etc. We do this by giving you a library
    36  (`MuxBroker`) for creating new connections between the client/server to
    37  serve additional interfaces or transfer raw data.
    38  
    39  **Bidirectional communication.** Because the plugin system supports
    40  complex arguments, the host process can send it interface implementations
    41  and the plugin can call back into the host process.
    42  
    43  **Built-in Logging.** Any plugins that use the `log` standard library
    44  will have log data automatically sent to the host process. The host
    45  process will mirror this output prefixed with the path to the plugin
    46  binary. This makes debugging with plugins simple. If the host system
    47  uses [hclog](https://github.com/hashicorp/go-hclog) then the log data
    48  will be structured. If the plugin also uses hclog, logs from the plugin
    49  will be sent to the host hclog and be structured.
    50  
    51  **Protocol Versioning.** A very basic "protocol version" is supported that
    52  can be incremented to invalidate any previous plugins. This is useful when
    53  interface signatures are changing, protocol level changes are necessary,
    54  etc. When a protocol version is incompatible, a human friendly error
    55  message is shown to the end user.
    56  
    57  **Stdout/Stderr Syncing.** While plugins are subprocesses, they can continue
    58  to use stdout/stderr as usual and the output will get mirrored back to
    59  the host process. The host process can control what `io.Writer` these
    60  streams go to to prevent this from happening.
    61  
    62  **TTY Preservation.** Plugin subprocesses are connected to the identical
    63  stdin file descriptor as the host process, allowing software that requires
    64  a TTY to work. For example, a plugin can execute `ssh` and even though there
    65  are multiple subprocesses and RPC happening, it will look and act perfectly
    66  to the end user.
    67  
    68  **Host upgrade while a plugin is running.** Plugins can be "reattached"
    69  so that the host process can be upgraded while the plugin is still running.
    70  This requires the host/plugin to know this is possible and daemonize
    71  properly. `NewClient` takes a `ReattachConfig` to determine if and how to
    72  reattach.
    73  
    74  **Cryptographically Secure Plugins.** Plugins can be verified with an expected
    75  checksum and RPC communications can be configured to use TLS. The host process
    76  must be properly secured to protect this configuration.
    77  
    78  ## Architecture
    79  
    80  The HashiCorp plugin system works by launching subprocesses and communicating
    81  over RPC (using standard `net/rpc` or [gRPC](http://www.grpc.io)). A single
    82  connection is made between any plugin and the host process. For net/rpc-based
    83  plugins, we use a [connection multiplexing](https://github.com/hashicorp/yamux)
    84  library to multiplex any other connections on top. For gRPC-based plugins,
    85  the HTTP2 protocol handles multiplexing.
    86  
    87  This architecture has a number of benefits:
    88  
    89    * Plugins can't crash your host process: A panic in a plugin doesn't
    90      panic the plugin user.
    91  
    92    * Plugins are very easy to write: just write a Go application and `go build`.
    93      Or use any other language to write a gRPC server with a tiny amount of
    94      boilerplate to support go-plugin.
    95  
    96    * Plugins are very easy to install: just put the binary in a location where
    97      the host will find it (depends on the host but this library also provides
    98      helpers), and the plugin host handles the rest.
    99  
   100    * Plugins can be relatively secure: The plugin only has access to the
   101      interfaces and args given to it, not to the entire memory space of the
   102      process. Additionally, go-plugin can communicate with the plugin over
   103      TLS.
   104  
   105  ## Usage
   106  
   107  To use the plugin system, you must take the following steps. These are
   108  high-level steps that must be done. Examples are available in the
   109  `examples/` directory.
   110  
   111    1. Choose the interface(s) you want to expose for plugins.
   112  
   113    2. For each interface, implement an implementation of that interface
   114       that communicates over a `net/rpc` connection or over a
   115       [gRPC](http://www.grpc.io) connection or both. You'll have to implement
   116       both a client and server implementation.
   117  
   118    3. Create a `Plugin` implementation that knows how to create the RPC
   119       client/server for a given plugin type.
   120  
   121    4. Plugin authors call `plugin.Serve` to serve a plugin from the
   122       `main` function.
   123  
   124    5. Plugin users use `plugin.Client` to launch a subprocess and request
   125       an interface implementation over RPC.
   126  
   127  That's it! In practice, step 2 is the most tedious and time consuming step.
   128  Even so, it isn't very difficult and you can see examples in the `examples/`
   129  directory as well as throughout our various open source projects.
   130  
   131  For complete API documentation, see [GoDoc](https://godoc.org/github.com/hashicorp/go-plugin).
   132  
   133  ## Roadmap
   134  
   135  Our plugin system is constantly evolving. As we use the plugin system for
   136  new projects or for new features in existing projects, we constantly find
   137  improvements we can make.
   138  
   139  At this point in time, the roadmap for the plugin system is:
   140  
   141  **Semantic Versioning.** Plugins will be able to implement a semantic version.
   142  This plugin system will give host processes a system for constraining
   143  versions. This is in addition to the protocol versioning already present
   144  which is more for larger underlying changes.
   145  
   146  ## What About Shared Libraries?
   147  
   148  When we started using plugins (late 2012, early 2013), plugins over RPC
   149  were the only option since Go didn't support dynamic library loading. Today,
   150  Go supports the [plugin](https://golang.org/pkg/plugin/) standard library with
   151  a number of  limitations. Since 2012, our plugin system has stabilized 
   152  from tens of millions of users using it, and has many benefits we've come to 
   153  value greatly.
   154  
   155  For example, we use this plugin system in
   156  [Vault](https://www.vaultproject.io) where dynamic library loading is
   157  not acceptable for security reasons. That is an extreme
   158  example, but we believe our library system has more upsides than downsides
   159  over dynamic library loading and since we've had it built and tested for years,
   160  we'll continue to use it.
   161  
   162  Shared libraries have one major advantage over our system which is much
   163  higher performance. In real world scenarios across our various tools,
   164  we've never required any more performance out of our plugin system and it
   165  has seen very high throughput, so this isn't a concern for us at the moment.