gvisor.dev/gvisor@v0.0.0-20240520182842-f9d4d51c7e0f/tools/go_marshal/README.md (about)

     1  This package implements the go_marshal utility.
     2  
     3  # Overview
     4  
     5  `go_marshal` is a code generation utility similar to `go_stateify` for
     6  marshalling go data structures to and from memory.
     7  
     8  `go_marshal` attempts to improve on `binary.Write` and the sentry's
     9  `binary.Marshal` by moving the expensive use of reflection from runtime to
    10  compile-time.
    11  
    12  `go_marshal` automatically generates implementations for `marshal.Marshallable`
    13  interface. Data structures that require custom serialization can be accomodated
    14  through a manual implementation this interface.
    15  
    16  Data structures can be flagged for code generation by adding a struct-level
    17  comment `// +marshal`. For additional details and options, see the documentation
    18  for the `marshal.Marshallable` interface.
    19  
    20  # Usage
    21  
    22  See `defs.bzl`: a new rule is provided, `go_marshal`.
    23  
    24  Under the hood, the `go_marshal` rule is used to generate a file that will
    25  appear in a Go target; the output file should appear explicitly in a srcs list.
    26  For example (note that the above is the preferred method):
    27  
    28  ```
    29  load("<PKGPATH>/gvisor/tools/go_marshal:defs.bzl", "go_marshal")
    30  
    31  go_marshal(
    32      name = "foo_abi",
    33      srcs = ["foo.go"],
    34      out = "foo_abi.go",
    35      package = "foo",
    36  )
    37  
    38  go_library(
    39      name = "foo",
    40      srcs = [
    41          "foo.go",
    42          "foo_abi.go",
    43      ],
    44      ...
    45  )
    46  ```
    47  
    48  As part of the interface generation, `go_marshal` also generates some tests for
    49  sanity checking the struct definitions for potential alignment issues, and a
    50  simple round-trip test through Marshal/Unmarshal to verify the implementation.
    51  These tests use reflection to verify properties of the ABI struct, and should be
    52  considered part of the generated interfaces (but are too expensive to execute at
    53  runtime). Ensure these tests run at some point.
    54  
    55  # Restrictions
    56  
    57  Not all valid go type definitions can be used with `go_marshal`. `go_marshal` is
    58  intended for ABI structs, which have these additional restrictions:
    59  
    60  -   At the moment, `go_marshal` only supports struct declarations.
    61  
    62  -   Structs are marshalled as packed types. This means no implicit padding is
    63      inserted between fields shorter than the platform register size. For
    64      alignment, manually insert padding fields.
    65  
    66  -   Structs used with `go_marshal` must have a compile-time static size. This
    67      means no dynamically sizes fields like slices or strings. Use statically
    68      sized array (byte arrays for strings) instead.
    69  
    70  -   No pointers, channel, map or function pointer fields, and no fields that are
    71      arrays of these types. These don't make sense in an ABI data structure.
    72  
    73  -   We could support opaque pointers as `uintptr`, but this is currently not
    74      implemented. Implementing this would require handling the architecture
    75      dependent native pointer size.
    76  
    77  -   Fields must either be a primitive integer type (`byte`,
    78      `[u]int{8,16,32,64}`), or of a type that implements `marshal.Marshallable`.
    79  
    80  -   `int` and `uint` fields are not allowed. Use an explicitly-sized numeric
    81      type.
    82  
    83  -   `float*` fields are currently not supported, but could be if necessary.
    84  
    85  # Appendix
    86  
    87  ## Working with Non-Packed Structs
    88  
    89  ABI structs must generally be packed types, meaning they should have no implicit
    90  padding between short fields. However, if a field is tagged
    91  `marshal:"unaligned"`, `go_marshal` will fall back to a safer but slower
    92  mechanism to deal with potentially unaligned fields.
    93  
    94  Note that the non-packed property is inheritted by any other struct that embeds
    95  this struct, since the `go_marshal` tool currently can't reason about alignments
    96  for embedded structs that are not aligned.
    97  
    98  Because of this, it's generally best to avoid using `marshal:"unaligned"` and
    99  insert explicit padding fields instead.
   100  
   101  ## Working with dynamically sized structs
   102  
   103  While `go_marshal` seamlessly supports statically sized structs (which most ABI
   104  structs are), it can also used for other uses cases where marshalling is
   105  required. There is some provision to partially support dynamically sized structs
   106  that may not be ABI structs. A user can define a dynamic struct and define
   107  `SizeBytes()`, `MarshalBytes(dst)` and `UnmarshalBytes(src)` for it. Then user
   108  can then add a comment above the struct like `// +marshal dynamic` while will
   109  make `go_marshal` autogenerate the remaining methods required to complete the
   110  `Marshallable` interface. This feature is currently only available for structs
   111  and can not be used alongside the Slice API.
   112  
   113  ## Modifying the `go_marshal` Tool
   114  
   115  The following are some guidelines for modifying the `go_marshal` tool:
   116  
   117  -   The `go_marshal` tool currently does a single pass over all types requesting
   118      code generation, in arbitrary order. This means the generated code can't
   119      directly obtain information about embedded marshallable types at
   120      compile-time. One way to work around this restriction is to add a new
   121      Marshallable interface method providing this piece of information, and
   122      calling it from the generated code. Use this sparingly, as we want to rely
   123      on compile-time information as much as possible for performance.
   124  
   125  -   No runtime reflection in the code generated for the marshallable interface.
   126      The entire point of the tool is to avoid runtime reflection. The generated
   127      tests may use reflection.
   128  
   129  ## Debugging
   130  
   131  To enable debugging output from the go-marshal tool, use one of the following
   132  options, depending on how go-marshal is being invoked:
   133  
   134  -   Pass `--define gomarshal=verbose` to the bazel command. Note that this can
   135      generate a lot of output depending on what's being compiled, as this will
   136      enable debugging for all packages built by the command.
   137  
   138  -   Set `marshal_debug = True` on the top-level `go_library` BUILD rule.
   139  
   140  -   Set `debug = True` on the `go_marshal` BUILD rule.
   141  
   142  -   Pass `-debug` to the go-marshal tool invocation.
   143  
   144  If bazel complains about stdout output being too large, set a larger value
   145  through `--experimental_ui_max_stdouterr_bytes`, or `-1` for unlimited output.