go.chromium.org/luci@v0.0.0-20240309015107-7cdc2e660f33/luciexe/doc.go

// Copyright 2019 The LUCI Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Package luciexe documents the "LUCI Executable" protocol, and contains
// constants which are part of this protocol.
//
// # Summary
//
// A LUCI Executable ("luciexe") is a binary which implements a protocol to:
//
//   - Pass the initial state of the 'build' from a parent process to the luciexe.
//   - Understand the build's local system contracts (like the location of cached data).
//   - Asynchronously update the state of the build as it runs.
//
// This protocol is recursive; A luciexe can run another luciexe such that the
// child's updates are reflected on the parent's output.
//
// The protocol has 3 parts:
//
//   - Host Application - This sits at the top of the luciexe process
//     hierarchy and sets up singleton environmental requirements for the whole
//     tree.
//   - Invocation of a luciexe binary - This invocation process occurs both
//     for the topmost luciexe, as well as all internal invocations of other
//     luciexe's within the process hierarchy.
//   - The luciexe binary - The binary has a couple of responsibilities to be
//     compatible with this protocol. Once the binary has fulfilled it's
//     responsibilities it's free to do what it wants (i.e. actually do its
//     task).
//
// In general, we strive where possible to minimize the complexity of the
// luciexe binary. This is because we expect to have a small number of 'host
// application' implementations, and a relatively large number of 'luciexe'
// implementations.
//
// # The Host Application
//
// At the root of every tree of luciexe invocations there is a 'host'
// application which sets up an manages all environmental singletons (like the
// Logdog 'butler' service, the LUCI ambient authentication service, etc.). This
// Host Application is responsible for intercepting and merging all
// 'build.proto' streams emitted within this tree of luciexes (see "Recursive
// Invocation"). The Host Application may choose what happens to these
// intercepted build.proto messages.
//
// The Host Application MUST:
//   - Run a logdog butler service and expose all relevant LOGDOG_* environment
//     variables such that the following client libraries can stream log data:
//   - Golang: go.chromium.org/luci/logdog/client/butlerlib/bootstrap
//   - Python: infra_libs.logdog.bootstrap
//   - Hook the butler to intercept and merge build.proto streams into a single
//     build.proto (zlib-compressed) stream.
//   - Set up a local LUCI ambient authentication service which luciexe's can
//     use to mint auth tokens.
//   - Prepare an empty directory which will house tempdirs and workdirs for
//     all luciexe invocations. The Host Application MAY clean this directory
//     up, but it may be useful to leak it for debugging. It's permissible for
//     the Host Application to defer this cleanup to an external process (e.g.
//     buildbucket's agent may defer this to swarming).
//
// The Host Application MAY hook additional streams for debugging/logging; it is
// frequently convenient to hook the stderr/stdout streams from the top level
// luciexe and tee them to the Host Application's stdout/stderr.
//
// For example: the `go.chromium.org/luci/buildbucket/cmd/agent` binary forwards
// these merged build.proto messages to the Buildbucket service, and also
// uploads all streams to them to the Logdog cloud service. Other host
// implementations may instead choose to write all streams to disk, send them to
// /dev/null or render them as html. However, from the point of view of the
// luciexe that they run, this is transparent.
//
// Host Applications MAY implement 'backpressure' on the luciexe binaries by
// throttling the rate at which the Logdog butler accepts data on its various
// streams. However, doing this could introduce timing issues in the luciexe
// binaries as they try to run, so this should be done thoughtfully.
//
// If a Host Application detects a protocol violation from a luciexe within its
// purview, it SHOULD report the violation (in a manner of its choosing) and
// MUST consider the entire Build status to be INFRA_FAILURE. In addition the
// Host Application SHOULD attempt to kill (via process group SIGTERM/SIGKILL on
// *nix, and CTRL+BREAK/Terminate on windows) the luciexe hierarchy. The host
// application MAY provide a window of time between the initial "please stop"
// signal and the "you die now" signal, but this comes with the usual caveats of
// cleanups and deadlines (notably: best-effort clean up is just that:
// best-effort. It cannot be relied on to run to completion (or to run
// completely)).
//
// # Invocation
//
// When invoking a luciexe, the parent process has a couple responsibilities. It
// must:
//
//   - Set $TEMPDIR, $TMPDIR, $TEMP, $TMP and $MAC_CHROMIUM_TMPDIR to all point
//     to the same, empty directory.
//     This directory MUST be located on the same file system as CWD.
//     This directory MUST NOT be the same as CWD.
//
//   - Set $LUCI_CONTEXT["luciexe"]["cache_dir"] to a cache dir which makes sense
//     for the luciexe.
//     The cache dir MAY persist/be shared between luciexe invocations.
//     The cache dir MAY NOT be on the same filesystem as CWD.
//
//   - Set the $LOGDOG_NAMESPACE to a prefix which namespaces all logdog streams
//     generated from the luciexe.
//
//   - Set the Status of initial buildbucket.v2.Build message to `STARTED`.
//
//   - Set the CreateTime and StartTime of initial buildbucket.v2.Build message.
//
//   - Clear following fields in the initial buildbucket.v2.Build message.
//
//   - EndTime
//
//   - Output
//
//   - StatusDetails
//
//   - Steps
//
//   - SummaryMarkdown
//
//   - UpdateTime
//
// The CWD is up to your application. Some contexts (like Buildbucket) will
// guarantee an empty CWD, but others (like recursive invocation) may explicitly
// share CWD between multiple luciexe's.
//
// The tempdir and workdir paths SHOULD NOT be cleaned up by the invoking
// process. Instead, the invoking process should defer to the Host Application
// to provide this cleanup, since the Host Application may be configured to leak
// these for debugging purposes.
//
// The invoker MUST attach the stdout/stderr to the logdog butler as text
// streams. These MUST be located at `$LOGDOG_NAMESPACE/std{out,err}`.
// Typical luciexe implementations will use these for debug logging and output,
// but are not required to do so.
//
// The invoker MUST write a binary-encoded buildbucket.v2.Build to the stdin of
// the luciexe which contains all the input parameters that the luciexe needs to
// know to run successfully.
//
// # The luciexe binary
//
// Once running, the luciexe MUST read a binary-encoded buildbucket.v2.Build
// message from stdin until EOF.
//
// As per the invoker's responsibility, the luciexe binary MAY expect the
// status of the initial Build message to be `STARTED` and CreateTime and
// StartTime are populated. It MAY also expect following fields to be empty.
// It MUST NOT assume other fields in the Build message are set. However, the
// Host Application or invoker MAY fill in other fields they think are useful.
//
//	EndTime
//	Output
//	StatusDetails
//	Steps
//	SummaryMarkdown
//	Tags
//	UpdateTime
//
// As per the Host Application's responsibilities, the luciexe binary MAY expect
// the "luciexe" and "local_auth" sections of LUCI_CONTEXT to be filled. Other
// sections of LUCI_CONTEXT MAY also be filled. See the LUCI_CONTEXT docs:
// https://chromium.googlesource.com/infra/luci/luci-py/+/HEAD/client/LUCI_CONTEXT.md
//
//	!!NOTE!! The paths supplied to the luciexe MUST NOT be considered stable
//	across invocations. Do not hard-code these, and try not to rely on their
//	consistency (e.g. for build reproducibility).
//
// # The luciexe binary - Updating the Build state
//
// A luciexe MAY update the Build state by writing to a "build.proto" Logdog
// stream named "$LOGDOG_NAMESPACE/build.proto". A "build.proto" Logdog stream
// is defined as:
//
//	Content-Type: "application/luci+proto; message=buildbucket.v2.Build"
//	Type: Datagram
//
// Additionally, a build.proto stream MAY append "; encoding=zlib" to the
// Content-Type (and compress each message accordingly). This is useful for when
// you potentially have very large builds.
//
// Each datagram MUST be a valid binary-encoded buildbucket.v2.Build message.
// The state of the build is defined as the last Build message sent on this
// stream. There's no implicit accumulation between sent Build messages.
//
// The Step.Log.Url field in the emitted Build messages can be either absolute
// (has a valid url scheme) or relative. If an absolute `Url` is provided, the
// luciexe is also responsible for supplying corresponding `ViewUrl` and the
// host application won't validate or make any adjustments to those urls. This
// allows luciexe to include logs from other sources into this build. If a
// relative `Url` is provided, the host application will namespace the `Url`
// with $LOGDOG_NAMESPACE of the build.proto stream. For example, if the host
// application is parsing a Build.proto in a stream named
// "logdog://host/project/prefix/+/something/build.proto", then a Log with a Url
// of "hello/world/stdout" will be transformed into:
//
//	Url:     logdog://host/project/prefix/+/something/hello/world/stdout
//	ViewUrl: <implementation defined>
//
// The `ViewUrl` field in this case SHOULD be left empty, and will be filled in
// by the host application running the luciexe (if supplied it will be
// overwritten).
//
// The following Build fields will be read from the luciexe-controlled
// build.proto stream:
//
//	EndTime
//	Output
//	Status
//	StatusDetails
//	Steps
//	SummaryMarkdown
//	Tags
//	UpdateTime
//
// # The luciexe binary - Reporting final status
//
// A luciexe MUST report its success/failure by sending a Build message with
// a terminal `status` value before exiting. If the luciexe exits before sending
// a Build message with a terminal Status, the invoking application MUST
// interpret this as an INFRA_FAILURE status. The exit code of the luciexe
// SHOULD be ignored, except for advisory (logging/reporting) purposes. The host
// application MUST detect this case and fill in a final status of
// INFRA_FAILURE, but MUST NOT terminate the process hierarchy in this case.
//
// # Recursive Invocation
//
// To support recursive invocation, a luciexe MUST accept the flag:
//
//	--output=path/to/file.{pb,json,textpb}
//
// The value of this flag MUST be an absolute path to a non-existent file in
// an existing directory. The extension of the file dictates the data format
// (binary, json or text protobuf). The luciexe MUST write it's final Build
// message to this file in the correct format. If `--output` is specified,
// but no Build message (or an invalid/improperly formatted Build message)
// is written, the caller MUST interpret this as an INFRA_FAILURE status.
//
//	NOTE: JSON outputs SHOULD be written with the original proto field names,
//	not the lowerCamelCase names; downstream users may not be using jsonpb
//	unmarshallers to interpret the JSON data.
//
//	This may need to be revised in a subsequent version of this API
//	specification.
//
// LUCI Executables MAY invoke other LUCI Executables as sub-steps and have the
// Steps from the child luciexe show in the parent's Build updates. This is one
// of the responsibilities of the Host Application.
//
// The parent can achieve this by recording a Step S (with no children), and
// a Step.MergeBuild.from_logdog_stream which points to a "build.proto" stream
// (see "Updating the Build State"). This is called a "Merge Step", and is
// a directive for the host to merge Build messages from in the
// from_logdog_stream log here. Note that this stream name should follow the
// same provisions specified for "Step.Log.Url". It's the invoker's
// responsibility to populate $LOGDOG_NAMESPACE with the new, full, namespace.
// Failure to do this will result in missing logs/step data.
//
// (SIDENOTE: There's an internal proposal to improve the Logdog "butler" API so
// that application code only needs to handle the relative namespaces as well,
// which would make this much less confusing).
//
// The Host Application MUST append all steps from the child build.proto
// stream to the parent build as substeps of step S and copy the following
// fields of the child Build to the equivalent fields of step S *only if*
// step S has *non-final* status. It is the caller's responsibility to populate
// rest of the fields of step S if the caller explicitly marks the step
// status as final.
//
//	SummaryMarkdown
//	Status
//	EndTime
//	Output.Logs (appended)
//
// This rule applies recursively, i.e. the child build MAY have Merge Step(s).
//
// Each luciexe's step names should be emitted as relative names. e.g. say
// a build runs a sub-luciexe with the name "a|b". This sub-luciexe then runs
// a step "x|y|z". The top level build.proto stream will show the step
// "a|b|x|y|z".
//
// The graph of datagram streams MUST be a tree. This follows from the
// namespacing rules of the Log.Url fields; Since Log.Url fields are relative to
// their build's namespace, it's only possible to have a merge step point
// further 'down' the tree, making it impossible to create a cycle.
//
// # Recursive Invocation - Legacy ChromeOS style
//
// Note that there's an option in MergeBuild called 'legacy_global_namespace'.
// This mode is NOT RECOMMENDED, and was only added to support legacy ChromeOS
// builders which relied on this functionality prior to luciexe.
//
// When this option is turned on, output properties on the build stream will be
// merged into the top-level build containing the Merge Step, and it will also
// de-namespace the step names (i.e. a MergeStep called "Parent" whose
// build.proto stream contains a name "Step" would normally show on the build as
// "Parent|Step", will instead simply show as "Step" in this mode. This also
// means that these merge step processes must make sure not to emit a step which
// duplicates the name of one emitted in the parent build (otherwise the overall
// build will be invalid)).
//
// Using this functionality has the downside that properties emitted by the
// parent build WILL BE OVERWRITTEN by the merged build stream, if written to
// the same top-level property keys. In ChromeOS's case this is "OK" because the
// legacy recipes do a short bootstrap followed by the main payload which does
// all the actual stuff.
//
// There's a version of this feature which would be more-supportable, which
// would be to introduce namespaced outputs so that the top-level build could
// reflect the outputs from a given step (i.e. allow Step to actually have its
// own Output message). If you feel like you need this functionality, please
// contact ChOps Foundation team to discuss, rather than trying to use the
// legacy_global_namespace option.
//
// # Related Libraries
//
// For implementation-level details, please refer to the following:
//
//   - "go.chromium.org/luci/luciexe" - low-level protocol details (this module).
//   - "go.chromium.org/luci/luciexe/host" - the Host Application library.
//   - "go.chromium.org/luci/luciexe/invoke" - luciexe invocation.
//   - "go.chromium.org/luci/luciexe/exe" - luciexe binary helper library.
//
// # Other Client Implementations
//
// Python Recipes (https://chromium.googlesource.com/infra/luci/recipes-py)
// implement the LUCI Executable protocol using the "luciexe" subcommand.
//
//	TODO(iannucci): Implement a luciexe binary helper in `infra_libs` analogous
//	to go.chromium.org/luci/luciexe/exe and implement Recipes' support in terms
//	of this.
//
// # LUCI Executables on Buildbucket
//
// Buildbucket accepts LUCI Executables as CIPD packages containing the
// luciexe to run with the fixed name of "luciexe".
//
// On Windows, this may be named "luciexe.exe" or "luciexe.bat". Buildbucket
// will prefer the first of these which it finds.
//
// Note that the `recipes.py bundle` command generates a "luciexe" wrapper
// script for compatibility with Buildbucket.
package luciexe