golang.org/x/build@v0.0.0-20240506185731-218518f32b70/README.md (about)

     1  # Go Build Tools
     2  
     3  [![Go Reference](https://pkg.go.dev/badge/golang.org/x/build.svg)](https://pkg.go.dev/golang.org/x/build)
     4  
     5  This repository holds the source for various packages and tools that support
     6  Go's build system and the development of the Go programming language.
     7  
     8  **Warning:** Packages here are internal to Go's build system and its needs.
     9  Some may one day be promoted to another `golang.org/x` repository,
    10  or they may be modified arbitrarily or even disappear altogether.
    11  In short, code in this repository is not subject to the Go 1 compatibility
    12  promise nor the [Release Policy](https://go.dev/doc/devel/release#policy).
    13  
    14  ## Report Issues / Send Patches
    15  
    16  This repository uses Gerrit for code changes. To contribute, see
    17  https://golang.org/doc/contribute.html.
    18  
    19  The main issue tracker for the blog is located at
    20  https://github.com/golang/go/issues. Prefix your issue with
    21  "`x/build/DIR: `" in the subject line.
    22  
    23  ## Overview
    24  
    25  The main components of the Go build system are:
    26  
    27  * The **coordinator**, in
    28    [cmd/coordinator/](https://dev.golang.org/dir/build/cmd/coordinator/),
    29    serves https://farmer.golang.org/ and https://build.golang.org/.
    30    It runs on GKE and coordinates the whole build system. It finds work
    31    to do (both pre-submit "TryBot" work, and post-submit work) and executes
    32    builds, allocating machines to run the builds. It is the owner of all machines.
    33    It holds the state for which builds passed or failed, and the build logs.
    34  
    35  * The Go package in [buildenv/](https://dev.golang.org/dir/build/buildenv/)
    36    contains constants for where the dashboard and coordinator run, for prod,
    37    staging, and local development.
    38  
    39  * The **buildlet**, in
    40    [cmd/buildlet/](https://dev.golang.org/dir/build/cmd/buildlet/), is the
    41    HTTP server that runs on each worker machine to execute builds on the
    42    coordinator's behalf. This runs on every possible GOOS/GOARCH value. The
    43    buildlet binaries are stored on Google Cloud Storage and fetched
    44    per-build, so we can update the buildlet binary independently of the
    45    underlying machine images. The buildlet is the most insecure server
    46    possible: it has HTTP handlers to read & write arbitrary content to disk,
    47    and to execute any file on disk. It also has an SSH tunnel handler. The
    48    buildlet must never be exposed to the Internet. The coordinator provisions
    49    buildlets in one of three ways:
    50  
    51    1. by creating VMs on Google Compute Engine (GCE) with custom images
    52    configured to fetch & run the buildlet on boot, listening on port 80 in a
    53    private network.
    54  
    55    2. by running Linux containers (on either Google Kubernetes Engine
    56    or GCE with the Container-Optimized OS image), with the container
    57    images configured to fetch & run the buildlet on start, also
    58    listening on port 80 in a private network.
    59  
    60    3. by taking buildlets out of a pool of connected, dedicated machines. The
    61    buildlet can run in either *listen mode* (as on GCE and GKE) or in
    62    *reverse mode*. In reverse mode, the buildlet connects out to
    63    https://farmer.golang.org/ and registers itself with the coordinator. The
    64    TCP connection is then logically reversed (using
    65    [revdial](https://dev.golang.org/dir/build/revdial/) and when the
    66    coordinator needs to do a build, it makes HTTP requests to the coordinator
    67    over the already-open TCP connection.
    68  
    69    These three pools can be viewed at the coordinator's
    70    https://farmer.golang.org/#pools.
    71  
    72  
    73  * The [env/](https://dev.golang.org/dir/build/env/) directory describes
    74    build environments. It contains scripts to create VM images, Dockerfiles
    75    to create Kubernetes containers, and instructions and tools for dedicated
    76    machines.
    77  
    78  
    79  * **maintner** in [maintner/](https://dev.golang.org/dir/build/maintner) is
    80    a library for slurping all of Go's GitHub and Gerrit state into memory.
    81    The daemon **maintnerd** in
    82    [maintner/maintnerd/](https://dev.golang.org/dir/build/maintner/maintnerd)
    83    runs on GKE and serves https://maintner.golang.org/. The daemon watches
    84    GitHub and Gerrit and appends to a mutation log whenever it sees new
    85    activity. The logs are stored on GCS and served to clients.
    86  
    87  
    88  * The [godata package](https://godoc.org/golang.org/x/build/maintner/godata)
    89    in [maintner/godata/](https://dev.golang.org/dir/build/maintner/godata)
    90    provides a trivial API to let anybody write programs against
    91    Go's maintner corpus (all of our GitHub and Gerrit history), live up
    92    to the second. It takes a few seconds to load into memory and a few hundred
    93    MB of RAM after it downloads the mutation log from the network.
    94  
    95  
    96  * **pubsubhelper** in
    97    [cmd/pubsubhelper/](https://dev.golang.org/dir/build/cmd/pubsubhelper/) is
    98    a dependency of maintnerd. It runs on GKE, is available at
    99    https://pubsubhelper.golang.org/, and runs an HTTP server to receive
   100    Webhook updates from GitHub on new activity and an SMTP server to receive
   101    new activity emails from Gerrit. It then is a pubsub system for maintnerd
   102    to subscribe to.
   103  
   104  
   105  * The **gitmirror** server in
   106    [cmd/gitmirror/](https://dev.golang.org/dir/build/cmd/gitmirror) mirrors
   107    Gerrit to GitHub, and also serves a mirror of the Gerrit code to the
   108    coordinator for builds, so we don't overwhelm Gerrit and blow our quota.
   109  
   110  
   111  * The Go **gopherbot** bot logic runs on GKE. The code is in
   112    [cmd/gopherbot](https://dev.golang.org/dir/build/cmd/gopherbot). It
   113    depends on maintner via the godata package.
   114  
   115  
   116  * The **developer dashboard** at https://dev.golang.org/ runs on GKE.
   117    Its code is in [devapp/](https://dev.golang.org/dir/build/devapp/).
   118    It also depends on maintner via the godata package.
   119  
   120  
   121  * **cmd/retrybuilds**: a Go client program to delete build results from the
   122      dashboard
   123  
   124  
   125  * The **perfdata** server, in
   126    [perfdata/appengine](https://dev.golang.org/dir/build/perfdata/appengine)
   127    serves https://perfdata.golang.org/. It runs on App Engine and
   128    serves the benchmark result storage system.
   129  
   130  
   131  * The **perf** server, in
   132    [perf/appengine](https://dev.golang.org/dir/build/perf/appengine)
   133    serves https://perf.golang.org/. It runs on App Engine and serves
   134    the benchmark result analysis system. See [its
   135    README](perf/README.md) for how to start a local testing instance.
   136  
   137  
   138  ### Adding a Go Builder
   139  
   140  If you wish to run a Go builder, please email
   141  [golang-dev@googlegroups.com](mailto:golang-dev@googlegroups.com) first. There
   142  is documentation at https://golang.org/wiki/DashboardBuilders, but depending
   143  on the type of builder, we may want to run it ourselves, after you prepare an
   144  environment description (resulting in a VM image) of it. See the env directory.
   145