github.com/GoogleContainerTools/skaffold/v2@v2.13.2/docs-v2/design_proposals/ko-builder-auto-sync.md (about)

     1  # ko builder: Hot reloading of the Go binary in dev mode
     2  
     3  * Author(s): Halvard Skogsrud (@halvards)
     4  * Design Shepherd: \<skaffold-core-team-member\>
     5  * Date: 2022-09-26
     6  * Status: Draft
     7  * go/skaffold-ko-devmode
     8  
     9  ## tl;dr
    10  
    11  Make Go development on a Kubernetes cluster feel as fast and responsive as on
    12  a developer workstation. Iterate on your code, and let Skaffold and ko rebuild
    13  and reload the running binary, with latencies that are indistinguishable from
    14  local development.
    15  
    16  ## Proposal
    17  
    18  Implement
    19  [auto sync](https://skaffold.dev/docs/filesync/#auto-sync-mode)
    20  for the
    21  [Skaffold ko builder](https://skaffold.dev/docs/builders/ko/) in
    22  [`dev`](https://skaffold.dev/docs/references/cli/#skaffold-dev) mode.
    23  
    24  On source code changes, Skaffold rebuilds the Go binary locally using ko, and
    25  copies it directly to the running container. Once copied, a process
    26  supervisor + file watcher
    27  ([`watchexec`](https://github.com/watchexec/watchexec)) reloads the
    28  application.
    29  
    30  The prototype implementation can reload a hello-world web application in two
    31  seconds on a GKE cluster, and in about one second on a local Minikube cluster.
    32  
    33  ## Motivation
    34  
    35  Local development can be setup to afford fast feedback with rapid rebuilding of
    36  application code. However, when the code needs to run in a Kubernetes cluster,
    37  this feedback loop slows down. Reasons for having to run on a Kubernetes cluster
    38  during development and testing include having to interact with dependent
    39  applications, and with other infrastructure components that are too heavyweight
    40  or cumbersome to run on a developer workstation.
    41  
    42  Skaffold has a `dev` mode that rebuilds and reloads the application running on a
    43  Kubernetes cluster. In most situations, this involves:
    44  
    45  1. Rebuilding the container image.
    46  2. Pushing the image to a registry.
    47  3. Creating a new pod with the updated image, replacing the existing pod.
    48  
    49  These steps involve multiple network hops (developer workstation pushing to
    50  registry, cluster node pulling from the registry) and the overhead of creating a
    51  new image and a new pod. These constraints put a lower bound on the latency of
    52  having the new code running, which in turn impacts an engineer's flow.
    53  
    54  Avoiding the cost of creating a new pod can be especially beneficial for some
    55  applications. An example is applications that pre-populate a cache on startup,
    56  either via application logic, or in an `emptyDir` volume using an init
    57  container.
    58  
    59  ## Background
    60  
    61  The Skaffold
    62  [Buildpacks builder](https://skaffold.dev/docs/builders/buildpacks/)
    63  supports
    64  [hot reloading](https://skaffold.dev/docs/filesync/#buildpacks)
    65  with
    66  [Google Cloud Buildpacks](https://github.com/GoogleCloudPlatform/buildpacks).
    67  
    68  Skaffold watches for source file changes on the developer workstation and
    69  copies them (as a tarball) to the running container. Another file watcher
    70  (`watchexec`) runs in the container, and when it detects the copied source
    71  files, it
    72  [recompiles and relaunches the binary](https://gist.github.com/halvards/a3c1f9a48adc931a2dcdd9db083350c4).
    73  
    74  This Buildpacks builder feature provides a great user experience for Node.js and
    75  JVM development. However, for Go development,
    76  [ko enables faster image builds than Cloud Native Buildpacks](https://cloud.google.com/blog/topics/developers-practitioners/ship-your-go-applications-faster-cloud-run-ko),
    77  and ko doesn't require Docker.
    78  
    79  The Buildpacks feature only sends source file changes over the network, but
    80  the binary rebuilds in-container are constrained by the resources available to
    81  the pod.
    82  
    83  ## Implementation, briefly
    84  
    85  User runs `skaffold dev` for an artifact configured to use the ko builder, and
    86  `sync.auto` is `true`:
    87  
    88  1.  During the image build, check if `watchexec` exists in the
    89      [`kodata`](https://ko.build/features/static-assets/) directory (under the
    90      workspace). If it doesn't, download a release for the  tarball, extract
    91      the binary to the `kodata` directory (creating it if it doesn't exist).
    92      Also, add `watchexec` as an entry in a `.gitignore` file in the `kodata`
    93      directory.
    94  
    95  2.  Rewrite the Kubernetes pod spec and specify watchexec as the container
    96      `command`. The original `command` and `args` will be added to `args`. To
    97      implement this, repurpose existing manifest rewriting logic from the
    98      Skaffold debug manifest rewriting implementation.
    99  
   100  3.  When a change event takes place for local source code files, determine
   101      the platform for the rebuilt binary. Use the field from `skaffold.yaml` or
   102      `--platform` flag if present, if not, default to `linux/<host arch>`. Using
   103      the host architecture as the default helps Minikube and KinD users.
   104  
   105  4.  Set the [`KOCACHE`](https://github.com/ko-build/ko/issues/264)
   106      environment variable if unset, so that ko builds the binary in a
   107      deterministic location. A clear contract from ko on this behavior would be
   108      helpful.
   109  
   110  5.  When constructing the
   111      [ko build options](https://github.com/ko-build/ko/blob/main/pkg/build/options.go),
   112      ensure that ko doesn't download the base image again. Skaffold is only
   113      rebuilding the binary, so the base image isn't required. To achieve this,
   114      provide an
   115      [empty image](https://github.com/google/go-containerregistry/tree/main/pkg/v1/empty)
   116      as the ko base image.
   117  
   118  6.  Use ko to build the new binary
   119      [src](https://github.com/ko-build/ko/blob/5e0452ad67230076340d0e28dd8488e4370675c2/pkg/build/gobuild.go#L967).
   120  
   121  7.  Use the existing Skaffold sync feature to sync the rebuilt binary from
   122      the local file `$KOCACHE/bin/<import path>/linux/<arch>/out` to the
   123      container destination `/ko-app/<base name of import path>`.
   124  
   125  ## Open questions
   126  
   127  1.  Should `sync.auto` default to `true` for the ko builder, as it does for
   128      the Buildpacks and Jib builders?
   129  
   130      Users must specify a default base image that works with watchexec, and
   131      the default ko base image doesn't. This means that if we default to
   132      `true`, `skaffold dev` will fail for ko users who don't specify a
   133      compatible base image.
   134  
   135      Options:
   136  
   137      -   Default to `true` for consistency with other builders. If
   138          skaffold dev fails, print an error message suggesting a compatible base
   139          image (e.g., `gcr.io/distroless/cc:debug`)
   140  
   141      -   Default to `false`, and print a message suggesting to set it to
   142          `true` when they run `skaffold dev`.
   143  
   144  2.  Should the URL used to download `watchexec` be exposed as a field in the
   145      Skaffold schema?
   146  
   147      This can be helpful for two reasons:
   148  
   149      -   Users can specify a different version of `watchexec`, or the
   150          musl-based binary instead of the default glibc one.
   151  
   152      -   Some users work in environments with restricted internet access,
   153          and this would allow them to specify an internal HTTP server as an
   154          alternative.
   155  
   156      However, an alternative that we can document is to ask users to download
   157      `watchexec` out-of-band, and place it in the `kodata` directory before
   158      running `skaffold dev`. Skaffold checks for the presence of the
   159      `watchexec` binary in this directory and skips the download if it is
   160      present. This option avoids adding yet another field to the schema.
   161  
   162  3.  Should the feature also cover the Docker deployer?
   163  
   164  ## Alternative implementation steps
   165  
   166  1.  Download the `watchexec` binary in the container at startup (using `curl`).
   167      This avoids polluting the local filesystem with the `watchexec` binary,
   168      but it adds time to each image build (in `dev` mode). It also requires a
   169      wrapper script as the entrypoint. The pod requires network access to the
   170      location where `watchexec` can be downloaded.