github.com/GoogleContainerTools/skaffold/v2@v2.13.2/docs-v2/content/en/docs/workflows/debug.md (about)

     1  ---
     2  title: "Debugging With Skaffold"
     3  linkTitle: "Debugging"
     4  weight: 30
     5  featureId: debug
     6  aliases: [/docs/how-tos/debug]
     7  ---
     8  
     9  Skaffold lets you set breakpoints and step through your application, even when deployed to remote Kubernetes clusters, as if the code were running locally on your machine. Skaffold detects the language runtimes of your container images, reconfigures the pods for debugging, forwards debugging ports, and then monitors for when debuggable containers come online.  IDE integrations like Cloud Code leverage Skaffold's events to automatically set up debugging sessions.
    10  
    11  Debugging is currently supported for five language runtimes.
    12  
    13    - Go 1.13+ (runtime ID: `go`) using [Delve](https://github.com/go-delve/delve)
    14    - NodeJS (runtime ID: `nodejs`) using the NodeJS Inspector (Chrome DevTools)
    15    - Java and JVM languages (runtime ID: `jvm`) using JDWP
    16    - Python 3.5-3.10 runtimes (runtime ID: `python`) using `debugpy` (Debug Adapter Protocol) or `pydevd`
    17      - NOTE: Python 3.11 support is planned and coming soon but is currently still in progress
    18    - .NET Core (runtime ID: `netcore`) using `vsdbg` (only for VS Code)
    19  
    20  Skaffold can usually detect the correct language runtime if present. However if you encounter difficulties then checkout the [Supported Language Runtimes]({{< relref "#supported-language-runtimes">}}) section for the exact heuristics that Skaffold uses and you can modify your application accordingly, or read about [how you can manually configure your container image]({{< relref "#can-images-be-debugged-without-the-runtime-support-images" >}}).
    21  
    22  ## (Recommended) Debugging using Cloud Code
    23  
    24  The easiest way to debug using Skaffold is by using the [Cloud Code IDE extension]({{< relref "../install/#managed-ide">}}) for Visual Studio Code, JetBrains IDEs, and Cloud Shell. Cloud Code will automatically configure container images for debugging so you can debug Kubernetes services just like how you would debug a local service in your IDE.
    25  
    26  For more information, see the corresponding documentation for [VS Code](https://cloud.google.com/code/docs/vscode/debug), [IntelliJ](https://cloud.google.com/code/docs/intellij/kubernetes-debugging) and [Cloud Shell](https://cloud.google.com/code/docs/shell/debug).
    27  
    28  ## Detailed debugger configuration and setup
    29  
    30  This section describes how Skaffold configures the container image for debugging for the supported language runtimes, and how to setup with certain IDEs.
    31  
    32  Note that many debuggers may require additional information for the location of source files.
    33  We are looking for ways to identify this information and to pass it back if found.
    34  
    35  {{% tabs %}}
    36  {{% tab "GO" %}}
    37  
    38  #### Go (runtime: `go`, protocols: `dlv`)
    39  
    40  Go-based applications are configured to run under
    41  [Delve](https://github.com/go-delve/delve) in its headless-server mode.
    42  
    43  On recognizing a Go-based container image, `debug` rewrites the container image's
    44  entrypoint to invoke your application using `dlv`:
    45  
    46  ```
    47  dlv exec --headless --continue --accept-multiclient --listen=:56268 --api-version=2 <app> -- <args> ...
    48  ```
    49  
    50  ##### Skaffold debug using the JetBrains GoLand and IntelliJ Ultimate
    51  
    52  Debugging is only supported in JetBrains IDEs for Go applications built using Go Modules.
    53  The IDE settings must also be explicitly configured to use Go Modules.  Errors like
    54  `cannot find debugger path` indicate misconfiguration.
    55  
    56  
    57  ##### Skaffold debug using the VS Code Go extension
    58  
    59  If you use the debug functionality of the
    60  [VS Code Go extension](https://github.com/golang/vscode-go) to connect to
    61  `dlv`, you may need to configure _local_ and _remote_ source paths, via
    62  either the `cwd` and `remotePath` properties, or the `substitutePath` array
    63  property
    64  ([reference](https://github.com/golang/vscode-go/blob/master/docs/debugging-legacy.md#launch-configurations)).
    65  
    66  The `cwd` or `from` properties should be the absolute path of the top-level
    67  directory of your source files and should generally match the artifact's
    68  `context` directory in `skaffold.yaml`. You can represent this using the
    69  `${workspaceFolder}` variable.
    70  
    71  The `remotePath` or `to` properties should be set to the absolute path of the
    72  source location _during compilation_. For example, the `golang` images, which
    73  are
    74  [often used in multi-stage builds](https://github.com/GoogleContainerTools/skaffold/tree/main/examples/getting-started/Dockerfile),
    75  have a default working directory of `/go`. The following
    76  [configuration](https://github.com/golang/vscode-go/blob/master/docs/debugging-legacy.md#remote-debugging)
    77  works in this case:
    78  
    79  ```json
    80  {
    81      "name": "Skaffold Debug",
    82      "type": "go",
    83      "request": "attach",
    84      "mode": "remote",
    85      "host": "localhost",
    86      "port": 56268,
    87      "cwd": "${workspaceFolder}",
    88      "remotePath": "/go"
    89  }
    90  ```
    91  
    92  You can use the `substitutePath` property instead of `cwd` and `remotePath`:
    93  
    94  ```json
    95  {
    96      "name": "Skaffold Debug",
    97      "type": "go",
    98      "request": "attach",
    99      "mode": "remote",
   100      "host": "localhost",
   101      "port": 56268,
   102      "substitutePath": [
   103          {
   104              "from": "${workspaceFolder}",
   105              "to": "/go",
   106          },
   107      ],
   108  }
   109  ```
   110  
   111  If you use the `ko` local builder, the source path during compilation matches
   112  the source path in your IDE. In this case, the value for `remotePath` will
   113  match `cwd`. If the values of both `cwd` and `remotePath` are
   114  `${workspaceFolder}`, you can omit these properties.
   115  
   116  {{% /tab %}}
   117  {{% tab "NODEJS" %}}
   118  
   119  #### NodeJS (runtime: `nodejs`, protocols: `devtools`)
   120  
   121  NodeJS applications are configured to use the Chrome DevTools inspector via the `--inspect` argument.
   122  
   123  On recognizing a NodeJS-based container image, `debug` rewrites the container image's
   124  entrypoint to invoke your application with `--inspect`:
   125  ```
   126  node --inspect=9229 <app.js>
   127  ```
   128  
   129  {{< alert title="Note" >}}
   130  Many applications use NodeJS-based tools as part of their launch, like <tt>npm</tt>, rather than
   131  invoke <tt>node</tt> directly.  These intermediate <tt>node</tt> instances may interpret the
   132  <tt>--inspect</tt> arguments.  Skaffold introduces a <tt>node</tt> wrapper that
   133  only invokes the real <tt>node</tt> with <tt>--inspect</tt> if running an application script,
   134  and skips scripts located in <tt>node_modules</tt>.  For more details see the
   135  <a href="https://github.com/GoogleContainerTools/container-debug-support/pull/34">associated PR</a>.
   136  {{< /alert >}}
   137  
   138  A debugging client must first obtain [the inspector UUID](https://github.com/nodejs/node/issues/9185#issuecomment-254872466).
   139  
   140  {{% /tab %}}
   141  {{% tab "JAVA" %}}
   142  
   143  #### Java and Other JVM Languages (runtime: `jvm`, protocols: `jdwp`)
   144  
   145  Java/JVM applications are configured to expose the JDWP agent using the `JAVA_TOOL_OPTIONS`
   146  environment variable.
   147  Note that the use of `JAVA_TOOL_OPTIONS` causes extra debugging output from the JVM on launch.
   148  
   149  On recognizing a JVM-based container image, `debug` rewrites the container image's
   150  environment to set:
   151  ```
   152  JAVA_TOOL_OPTIONS=-agentlib:jdwp=transport=dt_socket,server=y,address=5005,suspend=n,quiet=y
   153  ```
   154  {{% /tab %}}
   155  {{% tab "PYTHON" %}}
   156  
   157  #### Python (runtime: `python`, protocols: `dap` or `pydevd`)
   158  
   159  Python applications are configured to use either  [`debugpy`](https://github.com/microsoft/debugpy/) or
   160  [`pydevd`](https://github.com/fabioz/PyDev.Debugger).  `debugpy` uses the
   161  [_debug adapter protocol_ (DAP)](https://microsoft.github.io/debug-adapter-protocol/) which
   162  is supported by Visual Studio Code, [Eclipse LSP4e](https://projects.eclipse.org/projects/technology.lsp4e),
   163  [and other editors and IDEs](https://microsoft.github.io/debug-adapter-protocol/implementors/tools/).
   164  
   165  On recognizing a Python-based container image, `debug` rewrites the container image's
   166  entrypoint to invoke Python using either the `pydevd` or `debugpy` modules:
   167  ```
   168  python -m debugpy --listen 5678 <app>
   169  ```
   170  
   171  or
   172  ```
   173  python -m pydevd --server --port 5678 <app.py>
   174  ```
   175  
   176  {{< alert title="Note" >}}
   177  As many Python web frameworks use launcher scripts, like `gunicorn`, Skaffold now uses
   178  a debug launcher that examines the app command-line.
   179  {{< /alert >}}
   180  
   181  {{% /tab %}}
   182  {{% tab ".NETCORE" %}}
   183  
   184  #### .NET Core (runtime: `dotnet`, protocols: `vsdbg`)
   185  
   186  .NET Core applications are configured to be deployed along with `vsdbg`
   187  for VS Code.
   188  
   189  {{< alert title="JetBrains Rider" >}}
   190  This set up does not yet work automatically with [Cloud Code for IntelliJ in JetBrains Rider](https://github.com/GoogleCloudPlatform/cloud-code-intellij/issues/2903).
   191  There is [a manual workaround](https://github.com/GoogleCloudPlatform/cloud-code-intellij/wiki/Manual-set-up-for-remote-debugging-in-Rider).
   192  {{< /alert >}}
   193  
   194  {{< alert title="Omnisharp for VS Code" >}}
   195  For users of [VS Code's debug adapter for C#](https://github.com/OmniSharp/omnisharp-vscode):**
   196  the following configuration can be used to debug a container. It assumes that your code is deployed
   197  in `/app` or `/src` folder in the container. If that is not the case, the `sourceFileMap` property
   198  should be changed to match the correct folder. `processId` is usually 1 but might be different if you
   199  have an unusual entrypoint. You can also use `"${command:pickRemoteProcess}"` instead if supported by
   200  your base image.  (`//` comments must be stripped.)
   201  ```json
   202  {
   203      "name": "Skaffold Debug",
   204      "type": "coreclr",
   205      "request": "attach",
   206      "processId" : 1,
   207      "justMyCode": true, // set to `true` in debug configuration and `false` in release configuration
   208      "pipeTransport": {
   209          "pipeProgram": "kubectl",
   210          "pipeArgs": [
   211              "exec",
   212              "-i",
   213              "<NAME OF YOUR POD>", // name of the pod you debug.
   214              "--"
   215          ],
   216          "pipeCwd": "${workspaceFolder}",
   217          "debuggerPath": "/dbg/netcore/vsdbg", // location where vsdbg binary installed.
   218          "quoteArgs": false
   219      },
   220      "sourceFileMap": {
   221          // Change this mapping if your app in not deployed in /src or /app in your docker image
   222          "/src": "${workspaceFolder}",
   223          "/app": "${workspaceFolder}"
   224          // May also be like this, depending of your repository layout
   225          // "/src": "${workspaceFolder}/src",
   226          // "/app": "${workspaceFolder}/src/<YOUR PROJECT TO DEBUG>"
   227      }
   228  }
   229  ```
   230  {{< /alert >}}
   231  
   232  {{% /tab %}}
   233  {{% /tabs %}}
   234  
   235  ## Supported Language Runtimes
   236  
   237  The language runtime for the artifacts can be specified in the `skaffold.yaml` file.
   238  {{% readfile file="samples/debug/runtime-type.yaml" %}}
   239  
   240  Otherwise Skaffold `debug` recognizes the language runtime used in a
   241  container image for specific language runtimes using standard heuristics described below.
   242  
   243  {{% tabs %}}
   244  {{% tab "GO" %}}
   245  #### Go (runtime: `go`)
   246  
   247  Go-based container images are recognized by:
   248  
   249  - the presence of one of the
   250    [standard Go runtime environment variables](https://godoc.org/runtime):
   251    `GODEBUG`, `GOGC`, `GOMAXPROCS`, or `GOTRACEBACK`, or
   252  - the presence of the
   253    [`KO_DATA_PATH` environment variable](https://github.com/google/ko#static-assets)
   254    in container images built by
   255    [`ko`]({{< relref "/docs/builders/builder-types/ko" >}}), or
   256  - is launching using `dlv`.
   257  
   258  Unless you built your container image using `ko`, set one of the standard Go
   259  environment variables to allow Skaffold to detect your container as Go.
   260  `GOTRACEBACK=single` is the default setting for Go, so you can add this
   261  environment variable without changing behavior. Alternatively, `GOTRACEBACK=all`
   262  is a useful setting in many situtions.
   263  
   264  Your application should be built with the `-gcflags='all=-N -l'` flag to
   265  disable optimizations and inlining. If you do not add this flag, debugging can
   266  becoming confusing due to seemingly-random execution jumps that happen due to
   267  statement reordering and inlining. Skaffold configures Docker builds with a
   268  `SKAFFOLD_GO_GCFLAGS` build argument that you should use in your `Dockerfile`:
   269  
   270  ```
   271  FROM golang
   272  ENV GOTRACEBACK=all
   273  COPY . .
   274  ARG SKAFFOLD_GO_GCFLAGS
   275  RUN go build -gcflags="${SKAFFOLD_GO_GCFLAGS}" -o /app .
   276  ```
   277  
   278  Note that the Alpine Linux-based `golang` container images do not include a
   279  C compiler which is required to build with `-gcflags='all=-N -l'`.
   280  
   281  The ko builder adds `-gcflags='all=-N -l'` automatically when you use
   282  `skaffold debug`.
   283  
   284  We recommend that you do _not_ add the `-trimpath` flag when debugging, as this
   285  flag removes information from the resulting binary that aids debugging.
   286  
   287  {{% /tab %}}
   288  {{% tab "JAVA" %}}
   289  
   290  #### Java and Other JVM Languages (runtime: `jvm`)
   291  
   292  JVM application are recognized by:
   293  - the presence of a `JAVA_VERSION` or `JAVA_TOOL_OPTIONS` environment variable, or
   294  - the container command-line invokes `java`.
   295  
   296  {{% /tab %}}
   297  {{% tab "NODEJS" %}}
   298  
   299  #### NodeJS (runtime: `nodejs`)
   300  
   301  NodeJS images are recognized by:
   302  - the presence of a `NODE_VERSION`, `NODEJS_VERSION`, or `NODE_ENV` environment variable, or
   303  - the container command-line invokes `node` or `npm`.
   304  
   305  {{% /tab %}}
   306  {{% tab "PYTHON" %}}
   307  
   308  #### Python (runtime: `python`)
   309  
   310  Python application are recognized by:
   311  - the presence of a standard Python environment variable:
   312    `PYTHON_VERSION`, `PYTHONVERBOSE`, `PYTHONINSPECT`, `PYTHONOPTIMIZE`,
   313    `PYTHONUSERSITE`, `PYTHONUNBUFFERED`, `PYTHONPATH`, `PYTHONUSERBASE`,
   314    `PYTHONWARNINGS`, `PYTHONHOME`, `PYTHONCASEOK`, `PYTHONIOENCODING`,
   315    `PYTHONHASHSEED`, `PYTHONDONTWRITEBYTECODE`, or
   316  - the container command-line invokes `python`, `python2`, or `python3`.
   317  
   318  {{% /tab %}}
   319  {{% tab ".NETCORE" %}}
   320  
   321  #### .NET Core (runtime: `dotnet`)
   322  
   323  .NET Core application are recognized by:
   324  - the presence of a standard .NET environment variable:
   325    `ASPNETCORE_URLS`, `DOTNET_RUNNING_IN_CONTAINER`,
   326    `DOTNET_SYSTEM_GLOBALIZATION_INVARIANT`, or
   327  - the container command-line invokes the [dotnet](https://github.com/dotnet/sdk) cli
   328  
   329  Furthermore, your app must be built with the `--configuration Debug` options to disable optimizations.
   330  
   331  {{% /tab %}}
   332  {{% /tabs %}}
   333  
   334  ## How It works
   335  
   336  `skaffold debug` acts like `skaffold dev`, but it configures containers in pods
   337  for debugging as required for each container's runtime technology.
   338  The associated debugging ports are exposed and labelled so that they can be port-forwarded to the
   339  local machine.
   340  One notable difference from `skaffold dev` is that `debug` disables image rebuilding and
   341  syncing as it leads to users accidentally terminating debugging sessions by saving file changes.
   342  These behaviours can be re-enabled with the `--auto-build`, `--auto-deploy`, and `--auto-sync`
   343  flags, or triggering a [devloop iteration]({{< relref "/docs/design/api#control-api" >}}) using the Skaffold API.
   344  
   345  Enabling debugging has two phases:
   346  
   347  1. **Configuring:** Skaffold automatically examines each built container image and
   348     attempts to recognize the underlying language runtime.  Container images can be
   349     explicitly configured too.
   350  3. **Monitoring:** Skaffold watches the cluster to detect when debuggable containers
   351     start execution.
   352  
   353  ### Configuring container images for debugging
   354  
   355  `skaffold debug` examines the *built artifacts* to determine the underlying language runtime technology.
   356  Kubernetes manifests that reference these artifacts are transformed on-the-fly to enable the
   357  language runtime's debugging functionality.  These transforms add or alter environment variables
   358  and entrypoints, and more.
   359  
   360  Some language runtimes require additional support files to enable debugging.
   361  For these languages, a special set of [runtime-specific images](https://github.com/GoogleContainerTools/container-debug-support)
   362  are configured as _init-containers_ to populate a shared-volume that is mounted into
   363  each of the appropriate containers.  These images are hosted at
   364  `gcr.io/k8s-skaffold/skaffold-debug-support`; alternative locations can be
   365  specified in [Skaffold's global configuration]({{< relref "/docs/design/global-config.md" >}}).
   366  
   367  For images that are successfully recognized, Skaffold adds a `debug.cloud.google.com/config`
   368  annotation to the corresponding Kubernetes pod-spec that encode the debugging parameters.
   369  
   370  ### Monitoring for debuggable containers
   371  
   372  Once the application is deployed, `debug` monitors the cluster looking for debuggable pods with a
   373  `debug.cloud.google.com/config` annotation.  For each new debuggable pod,  Skaffold emits
   374  an event that can be used by tools like IDEs to establish a debug session.
   375  
   376  ### Additional changes
   377  
   378  `debug` makes some other adjustments to simplify the debug experience:
   379  
   380    - *Replica Counts*: `debug` rewrites  the replica counts to 1 for
   381      deployments, replica sets, and stateful sets.  This results in
   382      requests being serialized so that one request is processed at a time.
   383  
   384    - *Kubernetes Probes*:  `debug` changes the timeouts on HTTP-based
   385      [liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)
   386      to 600 seconds (10 minutes) from the default of 1 second.
   387      This change allows probes to be debugged, and avoids negative
   388      consequences from blocked probes when the app is already suspended
   389      during a debugging session.
   390      Failed liveness probes in particular result in the container
   391      being terminated and restarted.
   392  
   393  	The probe timeout value can be set on a per-podspec basis by setting
   394  	a `debug.cloud.google.com/probe/timeouts` annotation on the podspec's metadata
   395  	with a valid duration (see [Go's time.ParseDuration()](https://pkg.go.dev/time#ParseDuration)).
   396      This probe timeout-rewriting can be skipped entirely by using `skip`.  For example:
   397      ```yaml
   398      metadata:
   399        annotations:
   400          debug.cloud.google.com/probe/timeouts: skip
   401      spec: ...
   402      ```
   403  
   404  ## Troubleshooting
   405  
   406  ### My container is not being made debuggable?
   407  
   408  **Was this image built by Skaffold?**
   409  `debug` only works for images that were built by
   410   Skaffold so as to avoid affecting system- or infrastructure-level containers such as proxy sidecars.
   411  
   412  **Was Skaffold able to recognize the image?**
   413  `debug` emits a warning when it is unable to configure an image for debugging:
   414  ```
   415  WARN[0005] Image "image-name" not configured for debugging: unable to determine runtime for "image-name"
   416  ```
   417  
   418  See the language runtime section details on how container images are recognized.
   419  
   420  **Is your container command-line using shell constructs?**
   421  
   422  Automatic debugging of Go and Python applications requires that Skaffold
   423  adjust the command-line to use a specific launcher (`dlv` for Go, or our
   424  custom Python launcher to configure the appropriate debugging module).
   425  These launchers require that the container command-line invokes the
   426  binary or launch script (i.e., a `pip`-generated launcher script such as `gunicorn`).
   427  Skaffold cannot debug Go and Python applications that use user-created
   428  shell scripts, or that use shell constructs like `exec` or `eval`.
   429  Either rewrite your container image command-line or
   430  [manually configure your container for debugging]({{< relref "#can-images-be-debugged-without-the-runtime-support-images" >}}).
   431  For example:
   432  ```
   433  ❌ CMD launch.sh
   434  ❌ CMD exec gunicorn app:app
   435  ✅ CMD gunicorn app:app
   436  ✅ CMD [gunicorn, app:app]
   437  ✅ CMD [python, -m, gunicorn, app:app]
   438  ```
   439  
   440  
   441  ### Why aren't my breakpoints being hit?
   442  
   443  **Missing required language extension.**
   444  Go, Python, and JavaScript debugging in IDEA Ultimate requires the
   445  corresponding language plugin to be installed. If it is not installed,
   446  the debugger will not attach and breakpoints will not be hit.
   447  
   448  **Breakpoints are not hit during program startup.**
   449  Skaffold configures the containers to run-on-start (sometimes called
   450  _continue_ mode), such that the containers do not wait for the
   451  debugger to connect.  The container will have likely finished its
   452  startup by the time the debugger is able to connect and configure
   453  the breakpoints, and so breakpoints in the normal startup path are
   454  unlikely to be hit.
   455  
   456  **File mapping misconfiguration between local and container filesystems.**
   457  Some language runtimes require configuring the IDE with a _source map_
   458  to map files in the container to their corresponding local files.
   459  If this mapping is incorrect, then the remote debugging runtime
   460  will not be able to find the source file for the breakpoint. The Cloud Code plugins generally prompt the user when required.
   461  
   462  - Some builds use multi-stage dockerfiles to compile the binary using
   463    a build-time image and then copy the binary into a smaller image
   464    intended for deployment.  In these cases, the source mapping must
   465    use the build-time paths.  For example, many Go builds use a `golang`
   466    image for building, where the source files are typically copied into `/go`.
   467  - JetBrains GoLand and IDEA Ultimate only support remote breakpoints
   468    for Go applications built using Go Modules.  The IDE settings must
   469    also be explicitly configured to use Go Modules.
   470  - JVM languages do not require the mapping as the JVM uses class
   471    names rather than file paths.
   472  
   473  ### Can images be debugged without the runtime support images?
   474  
   475  The special [runtime-support images](https://github.com/GoogleContainerTools/container-debug-support)
   476  are provided as a convenience for automatic configuration.  You can
   477  manually configure your images for debugging by one of the following
   478  means:
   479  
   480  - `skaffold debug` usually recognizes already-configured container images
   481    that use the appropriate debugger.  For example, you could use a
   482    multi-stage Dockerfile with a `debug` stage that invokes the
   483    application using the debugger.
   484  - Use a `debug.cloud.google.com/config` workload annotation on the
   485    pod-spec to describe the debug configuration of each container
   486    image in the pod, as described in [_Workload Annotations_](#workload-annotations).
   487  
   488  ## Limitations
   489  
   490  `skaffold debug` has some limitations.
   491  
   492  ### Unsupported Container Entrypoints
   493  
   494  `skaffold debug` requires being able to examine and alter the
   495  command-line used in the container entrypoint.  This transformation
   496  will not work with images that use intermediate launch scripts or
   497  binaries.
   498  
   499  ### Supported Deployers
   500  
   501  `skaffold debug` is only supported with the `kubectl`, `kustomize`, and `helm` deployers.
   502  
   503  {{< alert title="Note" >}}
   504  Helm support requires using Helm v3.1.0 or greater.
   505  {{< /alert >}}
   506  
   507  
   508  ### Deprecated Workload API Objects
   509  
   510  `skaffold debug` does not support deprecated versions of Workload API objects:
   511  
   512    - `extensions/v1beta1` and `apps/v1beta1` was [deprecated in Kubernetes 1.8](https://github.com/kubernetes/kubernetes/blob/HEAD/CHANGELOG/CHANGELOG-1.8.md#other-notable-changes-16)
   513      and [removed in Kubernetes 1.16](https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/).
   514    - `apps/v1beta2` was [deprecated in Kubernetes 1.9](https://github.com/kubernetes/kubernetes/blob/HEAD/CHANGELOG/CHANGELOG-1.9.md#apps)
   515      and [removed in Kubernetes 1.16](https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/).
   516  
   517  Applications should transition to the `apps/v1` APIs,
   518  [introduced in Kubernetes 1.9](https://kubernetes.io/blog/2017/12/kubernetes-19-workloads-expanded-ecosystem/#workloads-api-ga).
   519  
   520  ----
   521  
   522  ## Appendix: IDE Support via Events and Metadata {#metadata-events}
   523  
   524  `debug` provides additional support for IDEs to detect the debuggable containers and to determine
   525  appropriate configuration parameters.
   526  
   527  ### Workload Annotations
   528  
   529  Each transformed workload object carries a `debug.cloud.google.com/config` annotation with
   530  a JSON object describing the debug configurations for the pod's containers (linebreaks for readability):
   531  ```
   532  	debug.cloud.google.com/config={
   533  		"<containerName>":{"runtime":"<runtimeId>",...},
   534  		"<containerName>":{"runtime":"<runtimeId>",...},
   535  		}
   536  ```
   537  
   538  For example the following annotation indicates that the container named `web` is a Go application
   539  that is being debugged by a headless Delve session on port `56268` (linebreaks for readability):
   540  ```
   541  debug.cloud.google.com/config={
   542    "web":{
   543      "artifact":"gcr.io/random/image",
   544      "runtime":"go",
   545      "ports":{"dlv":56268},
   546      "workingDir":"/some/path"}}
   547  ```
   548  
   549  `artifact` is the corresponding artifact's image name in the `skaffold.yaml`.
   550  `runtime` is the language runtime detected (one of: `go`, `jvm`, `nodejs`, `python`).
   551  `ports` is a list of debug ports keyed by the language runtime debugging protocol.
   552  `workingDir` is the working directory (if not an empty string).
   553  
   554  
   555  ### API: Events
   556  
   557  Each debuggable container being started or stopped raises a _debug-container-event_ through
   558  Skaffold's event mechanism ([gRPC]({{< ref "/docs/references/api/grpc#debuggingcontainerevent" >}}),
   559  [REST]({{< ref "/docs/references/api/swagger#/SkaffoldService/Events" >}})).
   560  
   561  <details>
   562  <summary>`/v1/events` stream of `skaffold debug` within `examples/jib`</summary>
   563  
   564  In this example, we do a `skaffold debug`, and then kill the deployed pod.  The deployment starts a new pod.
   565  We get a terminated event for the container for the killed pod.
   566  
   567  ```json
   568  {
   569    "result": {
   570      "timestamp": "2020-02-05T03:27:30.114354Z",
   571      "event": {
   572        "debuggingContainerEvent": {
   573          "status": "Started",
   574          "podName": "web-f6d56bcc5-6csgs",
   575          "containerName": "web",
   576          "namespace": "default",
   577          "artifact": "skaffold-jib",
   578          "runtime": "jvm",
   579          "debugPorts": {
   580            "jdwp": 5005
   581          }
   582        }
   583      },
   584      "entry": "Debuggable container started pod/web-f6d56bcc5-6csgs:web (default)"
   585    }
   586  }
   587  ```
   588  
   589  </details>
   590  
   591  
   592  
   593  ### API: State
   594  
   595  The API's _state_ ([gRPC]({{< relref "/docs/references/api/grpc#skaffoldservice" >}}),
   596  [REST]({{< relref "/docs/references/api/swagger#/SkaffoldService/GetState" >}})) also includes a list of debuggable containers.
   597  
   598  <details>
   599  <summary>The `/v1/state` listing debugging containers</summary>
   600  
   601  ```json
   602  {
   603    "buildState": {
   604      "artifacts": {
   605        "skaffold-jib": "Complete"
   606      }
   607    },
   608    "deployState": {
   609      "status": "Complete"
   610    },
   611    "forwardedPorts": {
   612      "5005": {
   613        "localPort": 5005,
   614        "remotePort": 5005,
   615        "podName": "web-f6d56bcc5-6csgs",
   616        "containerName": "web",
   617        "namespace": "default",
   618        "portName": "jdwp",
   619        "resourceType": "pod",
   620        "resourceName": "web-f6d56bcc5-6csgs",
   621        "address": "127.0.0.1"
   622      },
   623      "8080": {
   624        "localPort": 8080,
   625        "remotePort": 8080,
   626        "namespace": "default",
   627        "resourceType": "service",
   628        "resourceName": "web",
   629        "address": "127.0.0.1"
   630      },
   631      "8081": {
   632        "localPort": 8081,
   633        "remotePort": 8080,
   634        "podName": "web-f6d56bcc5-6csgs",
   635        "containerName": "web",
   636        "namespace": "default",
   637        "resourceType": "pod",
   638        "resourceName": "web-f6d56bcc5-6csgs",
   639        "address": "127.0.0.1"
   640      }
   641    },
   642    "statusCheckState": {
   643      "status": "Not Started"
   644    },
   645    "fileSyncState": {
   646      "status": "Not Started"
   647    },
   648    "debuggingContainers": [
   649      {
   650        "status": "Started",
   651        "podName": "web-f6d56bcc5-6csgs",
   652        "containerName": "web",
   653        "namespace": "default",
   654        "artifact": "skaffold-jib",
   655        "runtime": "jvm",
   656        "debugPorts": {
   657          "jdwp": 5005
   658        }
   659      }
   660    ]
   661  }
   662  
   663  ```
   664  
   665  </details>
   666