go.fuchsia.dev/infra@v0.0.0-20240507153436-9b593402251b/cmd/recipe_wrapper/README.md (about)

     1  # Recipe Wrapping
     2  
     3  The recipe wrapping tool reads a build.proto from stdin, resolves the
     4  appropriate recipe version and properties to use, and executes the build.
     5  
     6  The tool implements the
     7  [luciexe protocol](https://pkg.go.dev/go.chromium.org/luci/luciexe). Fuchsia
     8  Buildbucket builders which require recipe versioning should configure the tool
     9  as their executable. See the
    10  [lucicfg documentation](https://chromium.googlesource.com/infra/luci/luci-go/+/main/lucicfg/doc/#luci.executable)
    11  for more information.
    12  
    13  ## Behavior
    14  
    15  ### Builder properties resolution
    16  
    17  recipe_wrapper will resolve the build's input properties by reading properties
    18  from a file in integration.git, at a revision determined by the input
    19  build.proto.
    20  
    21  The path to the properties file is calculated based on the builder information
    22  in the input build.proto, specifically
    23  `infra/config/generated/<project>/properties/<bucket>/<builder>.json`,
    24  where `<project>`, `<bucket>`, and `<builder>` each come from the corresponding
    25  field of the input build.proto under the top-level `builder` field.
    26  
    27  This makes it possible to "version" builder properties – i.e. property changes
    28  can be tested in CQ, and properties don't need to be kept backwards compatible
    29  with old versions of code on release branches.
    30  
    31  However, some properties cannot be versioned because they're needed by
    32  recipe_wrapper to resolve the integration repository and revision to checkout.
    33  See the "Properties" section for a full listing.
    34  
    35  ### Recipe version resolution
    36  
    37  The "recipe version" is a recipes.git revision, which is resolved through a
    38  decision tree on build.proto. See [main file](main.go).
    39  
    40  The recipe version is normally read from a file in the same integration.git
    41  checkout that the properties file comes from. However, if the build input
    42  contains a change to the recipes repository, recipe_wrapper will check out the
    43  recipes repository at the input change, rather than at the version pinned in
    44  integration. That allows us to test recipe changes in presubmit using regular
    45  tryjobs.
    46  
    47  Pinning a recipe version in integration.git makes it possible to "version"
    48  recipes by using correspondingly old versions of recipes on old release
    49  branches. That way, HEAD of recipes does not need to maintain backwards
    50  compatibility with every integration.git release branch we wish to build and
    51  test, making recipe development much easier.
    52  
    53  ### Recipe checkout & execution
    54  
    55  After recipe_wrapper has resolved a recipe version, is checks out recipes.git
    56  at the resolved revision and execs `recipes.py luciexe` within the recipes
    57  checkout to take over the remainder of the build execution.
    58  
    59  ## Properties
    60  
    61  recipe_wrapper reads several optional builder properties to determine its
    62  behavior:
    63  
    64  ### `recipe_integration_remote`
    65  
    66  Example: "https://fuchsia-internal.googlesource.com/integration"
    67  
    68  If a build is triggered by a cron schedule rather than by a commit (via LUCI
    69  Scheduler) or CL (via Commit Queue), then it won't have a `gitiles_commit` or
    70  `gerrit_change` in its input.
    71  
    72  For public builders it's safe to fall back to using the public integration
    73  repository. However, many internal builders assume access to the internal
    74  integration repository, so they need to fall back to using internal
    75  integration instead.
    76  
    77  `recipe_integration_remote` lets internal builders that may be triggered on a
    78  cron job fall back to checking out the specified integration repository
    79  (generally the internal integration repository) rather than the default.
    80  
    81  ### `recipes_host_override`
    82  
    83  Example: "fuchsia.googlesource.com"
    84  
    85  Some builders are triggered by commits in Git-on-Borg repositories that are
    86  not on one of the Fuchsia-owned Git-on-Borg hosts. Only Fuchsia-owned hosts
    87  have integration repositories, so for these builds it's not valid to try to
    88  checkout the integration repository that's on the same host and ref as the
    89  triggering commit.
    90  
    91  To avoid that, a builder that's triggered by a non-Fuchsia-owned repo can
    92  set `recipes_host_override` to hostname of the Fuchsia-owned Git-on-Borg
    93  instance that holds the desired integration repository. If `recipes_host_override`
    94  is set, we use the `refs/heads/main` ref for checkout because there's no
    95  guarantee for non-Fuchsia-owned repositories that the triggering ref will
    96  correspond to a ref of the integration repository.
    97  
    98  ### `recipes_integration_project_override`
    99  
   100  Example: "alternate-integration"
   101  
   102  Some builders may checkout an alternate, integration-like Git-on-Borg project
   103  which nevertheless contains valid recipe versioning and property files. This
   104  property overrides the default "integration" project with the given project for
   105  resolution of said files.
   106  
   107  ### `recipes_integration_ref_override`
   108  
   109  Example: "refs/heads/main"
   110  
   111  This property overrides the integration.git ref that recipe_wrapper resolves
   112  in the case where a resolved integration base revision isn't already specified
   113  in the build input.
   114  
   115  Similar to `recipes_host_override`, this is useful for builders that are
   116  triggered by non-Fuchsia-owned repositories, for which we can't assume that the
   117  triggering ref will correspond to a valid integration.git ref.
   118  
   119  ### `recipes_integration_ref_mapping`
   120  
   121  Example: `{"refs/heads/foo": "refs/heads/bar"}`
   122  
   123  This property also overrides the integration.git ref that recipe_wrapper
   124  resolves, but only depending on the original resolved ref (the ref from the
   125  build input). Given the above example property value, if the build is triggered
   126  on a change to the `foo` branch, recipe_wrapper would check out the `bar` branch
   127  of the integration repository.
   128  
   129  This is useful for builders that checkout integration repositories that use
   130  different branch naming schemes from global integration, and that still need to
   131  run against changes to global integration-based repositories.
   132  
   133  ## Testing with led
   134  
   135  recipe_wrapper has unit tests, and in general as much logic as possible should
   136  be covered by unit tests. However, because it has a large interface with
   137  Buildbucket and depends on services like Git-on-Borg, unit tests aren't
   138  sufficient to guarantee correctness of recipe_wrapper changes.
   139  
   140  The easiest way to integration-test recipe_wrapper changes is with
   141  [`led`](http://go/luci-how-to-led). Testing recipe_wrapper changes with `led`
   142  is slightly more involved than testing recipe changes, because by default `led`
   143  jobs run `recipes.py` directly instead of using `recipe_wrapper`.
   144  
   145  To circumvent this behavior, you need to build recipe_wrapper locally and
   146  upload the resulting executable to a file with the special name `luciexe` in the
   147  led job's isolated input. This work is handled by
   148  `scripts/led-edit-recipe-wrapper.py`, which can be pipelined with other led
   149  commands. For example:
   150  
   151  ```sh
   152  led get-build -real-build 8841535313446777120 | ./scripts/led-edit-recipe-wrapper.py | led launch
   153  ```