github.com/GoogleContainerTools/skaffold/v2@v2.13.2/docs-v1/content/en/docs/workflows/debug.md (about)

     1  ---
     2  title: "Debugging With Skaffold"
     3  linkTitle: "Debugging"
     4  weight: 30
     5  featureId: debug
     6  aliases: [/docs/how-tos/debug]
     7  ---
     8  
     9  Skaffold lets you set breakpoints and step through your application, even when deployed to remote Kubernetes clusters, as if the code were running locally on your machine. Skaffold detects the language runtimes of your container images, reconfigures the pods for debugging, forwards debugging ports, and then monitors for when debuggable containers come online.  IDE integrations like Cloud Code leverage Skaffold's events to automatically set up debugging sessions.
    10  
    11  Debugging is currently supported for five language runtimes.
    12  
    13    - Go 1.13+ (runtime ID: `go`) using [Delve](https://github.com/go-delve/delve)
    14    - NodeJS (runtime ID: `nodejs`) using the NodeJS Inspector (Chrome DevTools)
    15    - Java and JVM languages (runtime ID: `jvm`) using JDWP
    16    - Python 3.5+ (runtime ID: `python`) using `debugpy` (Debug Adapter Protocol) or `pydevd`
    17    - .NET Core (runtime ID: `netcore`) using `vsdbg` (only for VS Code)
    18  
    19  Skaffold can usually detect the correct language runtime if present. However if you encounter difficulties then checkout the [Supported Language Runtimes]({{< relref "#supported-language-runtimes">}}) section for the exact heuristics that Skaffold uses and you can modify your application accordingly, or read about [manually configuring your container for debugging](#manually-configuring-your-container-for-debugging).
    20  
    21  ## (Recommended) Debugging using Cloud Code
    22  
    23  The easiest way to debug using Skaffold is by using the [Cloud Code IDE extension]({{< relref "../install/#managed-ide">}}) for Visual Studio Code, JetBrains IDEs, and Cloud Shell. Cloud Code will automatically configure container images for debugging so you can debug Kubernetes services just like how you would debug a local service in your IDE.
    24  
    25  For more information, see the corresponding documentation for [VS Code](https://cloud.google.com/code/docs/vscode/debug), [IntelliJ](https://cloud.google.com/code/docs/intellij/kubernetes-debugging) and [Cloud Shell](https://cloud.google.com/code/docs/shell/debug).
    26  
    27  ## Detailed debugger configuration and setup
    28  
    29  This section describes how Skaffold configures the container image for debugging for the supported language runtimes, and how to setup with certain IDEs.
    30  
    31  Note that many debuggers may require additional information for the location of source files.
    32  We are looking for ways to identify this information and to pass it back if found.
    33  
    34  {{% tabs %}}
    35  {{% tab "GO" %}}
    36  
    37  #### Go (runtime: `go`, protocols: `dlv`)
    38  
    39  Go-based applications are configured to run under
    40  [Delve](https://github.com/go-delve/delve) in its headless-server mode.
    41  
    42  On recognizing a Go-based container image, `debug` rewrites the container image's
    43  entrypoint to invoke your application using `dlv`:
    44  
    45  ```
    46  dlv exec --headless --continue --accept-multiclient --listen=:56268 --api-version=2 <app> -- <args> ...
    47  ```
    48  
    49  ##### Skaffold debug using the JetBrains GoLand and IntelliJ Ultimate
    50  
    51  Debugging is only supported in JetBrains IDEs for Go applications built using Go Modules.
    52  The IDE settings must also be explicitly configured to use Go Modules.  Errors like
    53  `cannot find debugger path` indicate misconfiguration.
    54  
    55  
    56  ##### Skaffold debug using the VS Code Go extension
    57  
    58  If you use the debug functionality of the
    59  [VS Code Go extension](https://github.com/golang/vscode-go) to connect to
    60  `dlv`, you may need to configure _local_ and _remote_ source paths, via
    61  either the `cwd` and `remotePath` properties, or the `substitutePath` array
    62  property
    63  ([reference](https://github.com/golang/vscode-go/blob/master/docs/debugging-legacy.md#launch-configurations)).
    64  
    65  The `cwd` or `from` properties should be the absolute path of the top-level
    66  directory of your source files and should generally match the artifact's
    67  `context` directory in `skaffold.yaml`. You can represent this using the
    68  `${workspaceFolder}` variable.
    69  
    70  The `remotePath` or `to` properties should be set to the absolute path of the
    71  source location _during compilation_. For example, the `golang` images, which
    72  are
    73  [often used in multi-stage builds](https://github.com/GoogleContainerTools/skaffold/tree/main/examples/getting-started/Dockerfile),
    74  have a default working directory of `/go`. The following
    75  [configuration](https://github.com/golang/vscode-go/blob/master/docs/debugging-legacy.md#remote-debugging)
    76  works in this case:
    77  
    78  ```json
    79  {
    80      "name": "Skaffold Debug",
    81      "type": "go",
    82      "request": "attach",
    83      "mode": "remote",
    84      "host": "localhost",
    85      "port": 56268,
    86      "cwd": "${workspaceFolder}",
    87      "remotePath": "/go"
    88  }
    89  ```
    90  
    91  You can use the `substitutePath` property instead of `cwd` and `remotePath`:
    92  
    93  ```json
    94  {
    95      "name": "Skaffold Debug",
    96      "type": "go",
    97      "request": "attach",
    98      "mode": "remote",
    99      "host": "localhost",
   100      "port": 56268,
   101      "substitutePath": [
   102          {
   103              "from": "${workspaceFolder}",
   104              "to": "/go",
   105          },
   106      ],
   107  }
   108  ```
   109  
   110  If you use the `ko` local builder, the source path during compilation matches
   111  the source path in your IDE. In this case, the value for `remotePath` will
   112  match `cwd`. If the values of both `cwd` and `remotePath` are
   113  `${workspaceFolder}`, you can omit these properties.
   114  
   115  {{% /tab %}}
   116  {{% tab "NODEJS" %}}
   117  
   118  #### NodeJS (runtime: `nodejs`, protocols: `devtools`)
   119  
   120  NodeJS applications are configured to use the Chrome DevTools inspector via the `--inspect` argument.
   121  
   122  On recognizing a NodeJS-based container image, `debug` rewrites the container image's
   123  entrypoint to invoke your application with `--inspect`:
   124  ```
   125  node --inspect=9229 <app.js>
   126  ```
   127  
   128  {{< alert title="Note" >}}
   129  Many applications use NodeJS-based tools as part of their launch, like <tt>npm</tt>, rather than
   130  invoke <tt>node</tt> directly.  These intermediate <tt>node</tt> instances may interpret the
   131  <tt>--inspect</tt> arguments.  Skaffold introduces a <tt>node</tt> wrapper that
   132  only invokes the real <tt>node</tt> with <tt>--inspect</tt> if running an application script,
   133  and skips scripts located in <tt>node_modules</tt>.  For more details see the
   134  <a href="https://github.com/GoogleContainerTools/container-debug-support/pull/34">associated PR</a>.
   135  {{< /alert >}}
   136  
   137  A debugging client must first obtain [the inspector UUID](https://github.com/nodejs/node/issues/9185#issuecomment-254872466).
   138  
   139  {{% /tab %}}
   140  {{% tab "JAVA" %}}
   141  
   142  #### Java and Other JVM Languages (runtime: `jvm`, protocols: `jdwp`)
   143  
   144  Java/JVM applications are configured to expose the JDWP agent using the `JAVA_TOOL_OPTIONS`
   145  environment variable.
   146  Note that the use of `JAVA_TOOL_OPTIONS` causes extra debugging output from the JVM on launch.
   147  
   148  On recognizing a JVM-based container image, `debug` rewrites the container image's
   149  environment to set:
   150  ```
   151  JAVA_TOOL_OPTIONS=-agentlib:jdwp=transport=dt_socket,server=y,address=5005,suspend=n,quiet=y
   152  ```
   153  {{% /tab %}}
   154  {{% tab "PYTHON" %}}
   155  
   156  #### Python (runtime: `python`, protocols: `dap` or `pydevd`)
   157  
   158  Python applications are configured to use either  [`debugpy`](https://github.com/microsoft/debugpy/) or
   159  [`pydevd`](https://github.com/fabioz/PyDev.Debugger).  `debugpy` uses the
   160  [_debug adapter protocol_ (DAP)](https://microsoft.github.io/debug-adapter-protocol/) which
   161  is supported by Visual Studio Code, [Eclipse LSP4e](https://projects.eclipse.org/projects/technology.lsp4e),
   162  [and other editors and IDEs](https://microsoft.github.io/debug-adapter-protocol/implementors/tools/).
   163  
   164  On recognizing a Python-based container image, `debug` rewrites the container image's
   165  entrypoint to invoke Python using either the `pydevd` or `debugpy` modules:
   166  ```
   167  python -m debugpy --listen 5678 <app>
   168  ```
   169  
   170  or
   171  ```
   172  python -m pydevd --server --port 5678 <app.py>
   173  ```
   174  
   175  {{< alert title="Note" >}}
   176  As many Python web frameworks use launcher scripts, like `gunicorn`, Skaffold now uses
   177  a debug launcher that examines the app command-line.
   178  {{< /alert >}}
   179  
   180  {{% /tab %}}
   181  {{% tab ".NETCORE" %}}
   182  
   183  #### .NET Core (runtime: `dotnet`, protocols: `vsdbg`)
   184  
   185  .NET Core applications are configured to be deployed along with `vsdbg`
   186  for VS Code.
   187  
   188  {{< alert title="JetBrains Rider" >}}
   189  This set up does not yet work automatically with [Cloud Code for IntelliJ in JetBrains Rider](https://github.com/GoogleCloudPlatform/cloud-code-intellij/issues/2903).
   190  There is [a manual workaround](https://github.com/GoogleCloudPlatform/cloud-code-intellij/wiki/Manual-set-up-for-remote-debugging-in-Rider).
   191  {{< /alert >}}
   192  
   193  {{< alert title="Omnisharp for VS Code" >}}
   194  For users of [VS Code's debug adapter for C#](https://github.com/OmniSharp/omnisharp-vscode):**
   195  the following configuration can be used to debug a container. It assumes that your code is deployed
   196  in `/app` or `/src` folder in the container. If that is not the case, the `sourceFileMap` property
   197  should be changed to match the correct folder. `processId` is usually 1 but might be different if you
   198  have an unusual entrypoint. You can also use `"${command:pickRemoteProcess}"` instead if supported by
   199  your base image.  (`//` comments must be stripped.)
   200  ```json
   201  {
   202      "name": "Skaffold Debug",
   203      "type": "coreclr",
   204      "request": "attach",
   205      "processId" : 1,
   206      "justMyCode": true, // set to `true` in debug configuration and `false` in release configuration
   207      "pipeTransport": {
   208          "pipeProgram": "kubectl",
   209          "pipeArgs": [
   210              "exec",
   211              "-i",
   212              "<NAME OF YOUR POD>", // name of the pod you debug.
   213              "--"
   214          ],
   215          "pipeCwd": "${workspaceFolder}",
   216          "debuggerPath": "/dbg/netcore/vsdbg", // location where vsdbg binary installed.
   217          "quoteArgs": false
   218      },
   219      "sourceFileMap": {
   220          // Change this mapping if your app in not deployed in /src or /app in your docker image
   221          "/src": "${workspaceFolder}",
   222          "/app": "${workspaceFolder}"
   223          // May also be like this, depending of your repository layout
   224          // "/src": "${workspaceFolder}/src",
   225          // "/app": "${workspaceFolder}/src/<YOUR PROJECT TO DEBUG>"
   226      }
   227  }
   228  ```
   229  {{< /alert >}}
   230  
   231  {{% /tab %}}
   232  {{% /tabs %}}
   233  
   234  ## Supported Language Runtimes
   235  
   236  This section describes how `debug` recognizes the language runtime used in a
   237  container image for specific language runtimes.
   238  
   239  {{% tabs %}}
   240  {{% tab "GO" %}}
   241  #### Go (runtime: `go`)
   242  
   243  Go-based container images are recognized by:
   244  
   245  - the presence of one of the
   246    [standard Go runtime environment variables](https://godoc.org/runtime):
   247    `GODEBUG`, `GOGC`, `GOMAXPROCS`, or `GOTRACEBACK`, or
   248  - the presence of the
   249    [`KO_DATA_PATH` environment variable](https://github.com/google/ko#static-assets)
   250    in container images built by
   251    [`ko`]({{< relref "/docs/pipeline-stages/builders/ko" >}}), or
   252  - is launching using `dlv`.
   253  
   254  Unless you built your container image using `ko`, set one of the standard Go
   255  environment variables to allow Skaffold to detect your container as Go.
   256  `GOTRACEBACK=single` is the default setting for Go, so you can add this
   257  environment variable without changing behavior. Alternatively, `GOTRACEBACK=all`
   258  is a useful setting in many situtions.
   259  
   260  Your application should be built with the `-gcflags='all=-N -l'` flag to
   261  disable optimizations and inlining. If you do not add this flag, debugging can
   262  become confusing due to seemingly-random execution jumps that happen due to
   263  statement reordering and inlining. Skaffold configures Docker builds with a
   264  `SKAFFOLD_GO_GCFLAGS` build argument that you should use in your `Dockerfile`:
   265  
   266  ```
   267  FROM golang
   268  ENV GOTRACEBACK=all
   269  COPY . .
   270  ARG SKAFFOLD_GO_GCFLAGS
   271  RUN go build -gcflags="${SKAFFOLD_GO_GCFLAGS}" -o /app .
   272  ```
   273  
   274  Note that the Alpine Linux-based `golang` container images do not include a
   275  C compiler which is required to build with `-gcflags='all=-N -l'`.
   276  
   277  The ko builder adds `-gcflags='all=-N -l'` automatically when you use
   278  `skaffold debug`.
   279  
   280  We recommend that you do _not_ add the `-trimpath` flag when debugging, as this
   281  flag removes information from the resulting binary that aids debugging.
   282  
   283  {{% /tab %}}
   284  {{% tab "JAVA" %}}
   285  
   286  #### Java and Other JVM Languages (runtime: `jvm`)
   287  
   288  JVM application are recognized by:
   289  - the presence of a `JAVA_VERSION` or `JAVA_TOOL_OPTIONS` environment variable, or
   290  - the container command-line invokes `java`.
   291  
   292  {{% /tab %}}
   293  {{% tab "NODEJS" %}}
   294  
   295  #### NodeJS (runtime: `nodejs`)
   296  
   297  NodeJS images are recognized by:
   298  - the presence of a `NODE_VERSION`, `NODEJS_VERSION`, or `NODE_ENV` environment variable, or
   299  - the container command-line invokes `node` or `npm`.
   300  
   301  {{% /tab %}}
   302  {{% tab "PYTHON" %}}
   303  
   304  #### Python (runtime: `python`)
   305  
   306  Python application are recognized by:
   307  - the presence of a standard Python environment variable:
   308    `PYTHON_VERSION`, `PYTHONVERBOSE`, `PYTHONINSPECT`, `PYTHONOPTIMIZE`,
   309    `PYTHONUSERSITE`, `PYTHONUNBUFFERED`, `PYTHONPATH`, `PYTHONUSERBASE`,
   310    `PYTHONWARNINGS`, `PYTHONHOME`, `PYTHONCASEOK`, `PYTHONIOENCODING`,
   311    `PYTHONHASHSEED`, `PYTHONDONTWRITEBYTECODE`, or
   312  - the container command-line invokes `python`, `python2`, or `python3`.
   313  
   314  {{% /tab %}}
   315  {{% tab ".NETCORE" %}}
   316  
   317  #### .NET Core (runtime: `dotnet`)
   318  
   319  .NET Core application are recognized by:
   320  - the presence of a standard .NET environment variable:
   321    `ASPNETCORE_URLS`, `DOTNET_RUNNING_IN_CONTAINER`,
   322    `DOTNET_SYSTEM_GLOBALIZATION_INVARIANT`, or
   323  - the container command-line invokes the [dotnet](https://github.com/dotnet/sdk) cli
   324  
   325  Furthermore, your app must be built with the `--configuration Debug` options to disable optimizations.
   326  
   327  {{% /tab %}}
   328  {{% /tabs %}}
   329  
   330  ## How It works
   331  
   332  `skaffold debug` acts like `skaffold dev`, but it configures containers in pods
   333  for debugging as required for each container's runtime technology.
   334  The associated debugging ports are exposed and labelled so that they can be port-forwarded to the
   335  local machine.
   336  One notable difference from `skaffold dev` is that `debug` disables image rebuilding and
   337  syncing as it leads to users accidentally terminating debugging sessions by saving file changes.
   338  These behaviours can be re-enabled with the `--auto-build`, `--auto-deploy`, and `--auto-sync`
   339  flags, or triggering a [devloop iteration]({{< relref "/docs/design/api#control-api" >}}) using the Skaffold API.
   340  
   341  Enabling debugging has two phases:
   342  
   343  1. **Configuring:** Skaffold automatically examines each built container image and
   344     attempts to recognize the underlying language runtime.  Container images can be
   345     explicitly configured too.
   346  3. **Monitoring:** Skaffold watches the cluster to detect when debuggable containers
   347     start execution.
   348  
   349  ### Configuring container images for debugging
   350  
   351  `skaffold debug` examines the *built artifacts* to determine the underlying language runtime technology.
   352  Kubernetes manifests that reference these artifacts are transformed on-the-fly to enable the
   353  language runtime's debugging functionality.  These transforms add or alter environment variables
   354  and entrypoints, and more.
   355  
   356  Some language runtimes require additional support files to enable debugging.
   357  For these languages, a special set of [runtime-specific images](https://github.com/GoogleContainerTools/container-debug-support)
   358  are configured as _init-containers_ to populate a shared-volume that is mounted into
   359  each of the appropriate containers.  These images are hosted at
   360  `gcr.io/k8s-skaffold/skaffold-debug-support`; alternative locations can be
   361  specified in [Skaffold's global configuration]({{< relref "/docs/design/global-config.md" >}}).
   362  
   363  For images that are successfully recognized, Skaffold adds a `debug.cloud.google.com/config`
   364  annotation to the corresponding Kubernetes pod-spec that encode the debugging parameters.
   365  
   366  ### Monitoring for debuggable containers
   367  
   368  Once the application is deployed, `debug` monitors the cluster looking for debuggable pods with a
   369  `debug.cloud.google.com/config` annotation.  For each new debuggable pod,  Skaffold emits
   370  an event that can be used by tools like IDEs to establish a debug session.
   371  
   372  ### Additional changes
   373  
   374  `debug` makes some other adjustments to simplify the debug experience:
   375  
   376    - *Replica Counts*: `debug` rewrites  the replica counts to 1 for
   377      deployments, replica sets, and stateful sets.  This results in
   378      requests being serialized so that one request is processed at a time.
   379  
   380    - *Kubernetes Probes*:  `debug` changes the timeouts on HTTP-based
   381      [liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)
   382      to 600 seconds (10 minutes) from the default of 1 second.
   383      This change allows probes to be debugged, and avoids negative
   384      consequences from blocked probes when the app is already suspended
   385      during a debugging session.
   386      Failed liveness probes in particular result in the container
   387      being terminated and restarted.
   388  
   389  	The probe timeout value can be set on a per-podspec basis by setting
   390  	a `debug.cloud.google.com/probe/timeouts` annotation on the podspec's metadata
   391  	with a valid duration (see [Go's time.ParseDuration()](https://pkg.go.dev/time#ParseDuration)).
   392      This probe timeout-rewriting can be skipped entirely by using `skip`.  For example:
   393      ```yaml
   394      metadata:
   395        annotations:
   396          debug.cloud.google.com/probe/timeouts: skip
   397      spec: ...
   398      ```
   399  
   400  ## Troubleshooting
   401  
   402  ### My container is not being made debuggable?
   403  
   404  **Was this image built by Skaffold?**
   405  `debug` only works for images that were built by
   406   Skaffold so as to avoid affecting system- or infrastructure-level containers such as proxy sidecars.
   407  
   408  **Was Skaffold able to recognize the image?**
   409  `debug` emits a warning when it is unable to configure an image for debugging:
   410  ```
   411  WARN[0005] Image "image-name" not configured for debugging: unable to determine runtime for "image-name"
   412  ```
   413  
   414  See the language runtime section details on how container images are recognized.
   415  
   416  **Is your container command-line using shell constructs?**
   417  
   418  Automatic debugging of Go and Python applications requires that Skaffold
   419  adjust the command-line to use a specific launcher (`dlv` for Go, or our
   420  custom Python launcher to configure the appropriate debugging module).
   421  These launchers require that the container command-line invokes the
   422  binary or launch script (i.e., a `pip`-generated launcher script such as `gunicorn`).
   423  Skaffold cannot debug Go and Python applications that use user-created
   424  shell scripts, or that use shell constructs like `exec` or `eval`.
   425  Either rewrite your container image command-line or
   426  [manually configure your container for debugging](#manually-configuring-your-container-for-debugging).
   427  For example:
   428  ```
   429  ❌ CMD launch.sh
   430  ❌ CMD exec gunicorn app:app
   431  ✅ CMD gunicorn app:app
   432  ✅ CMD [gunicorn, app:app]
   433  ✅ CMD [python, -m, gunicorn, app:app]
   434  ```
   435  
   436  
   437  ### Why aren't my breakpoints being hit?
   438  
   439  **Missing required language extension.**
   440  Go, Python, and JavaScript debugging in IDEA Ultimate requires the
   441  corresponding language plugin to be installed. If it is not installed,
   442  the debugger will not attach and breakpoints will not be hit.
   443  
   444  **Breakpoints are not hit during program startup.**
   445  Skaffold configures the containers to run-on-start (sometimes called
   446  _continue_ mode), such that the containers do not wait for the
   447  debugger to connect.  The container will have likely finished its
   448  startup by the time the debugger is able to connect and configure
   449  the breakpoints, and so breakpoints in the normal startup path are
   450  unlikely to be hit.
   451  
   452  **File mapping misconfiguration between local and container filesystems.**
   453  Some language runtimes require configuring the IDE with a _source map_
   454  to map files in the container to their corresponding local files.
   455  If this mapping is incorrect, then the remote debugging runtime
   456  will not be able to find the source file for the breakpoint. The Cloud Code plugins generally prompt the user when required.
   457  
   458  - Some builds use multi-stage dockerfiles to compile the binary using
   459    a build-time image and then copy the binary into a smaller image
   460    intended for deployment.  In these cases, the source mapping must
   461    use the build-time paths.  For example, many Go builds use a `golang`
   462    image for building, where the source files are typically copied into `/go`.
   463  - JetBrains GoLand and IDEA Ultimate only support remote breakpoints
   464    for Go applications built using Go Modules.  The IDE settings must
   465    also be explicitly configured to use Go Modules.
   466  - JVM languages do not require the mapping as the JVM uses class
   467    names rather than file paths.
   468  
   469  ### Manually configuring your container for debugging
   470  
   471  The special [runtime-support images](https://github.com/GoogleContainerTools/container-debug-support)
   472  are provided as a convenience for automatic configuration.  You can
   473  manually configure your images for debugging by one of the following
   474  means:
   475  
   476  - `skaffold debug` usually recognizes already-configured container images
   477    that use the appropriate debugger.  For example, you could use a
   478    multi-stage Dockerfile with a `debug` stage that invokes the
   479    application using the debugger.
   480  - Use a `debug.cloud.google.com/config` workload annotation on the
   481    pod-spec to describe the debug configuration of each container
   482    image in the pod, as described in [_Workload Annotations_](#workload-annotations).
   483  
   484  ## Limitations
   485  
   486  `skaffold debug` has some limitations.
   487  
   488  ### Unsupported Container Entrypoints
   489  
   490  `skaffold debug` requires being able to examine and alter the
   491  command-line used in the container entrypoint.  This transformation
   492  will not work with images that use intermediate launch scripts or
   493  binaries.
   494  
   495  ### Supported Deployers
   496  
   497  `skaffold debug` is only supported with the `kubectl`, `kustomize`, and `helm` deployers.
   498  
   499  {{< alert title="Note" >}}
   500  Helm support requires using Helm v3.1.0 or greater.
   501  {{< /alert >}}
   502  
   503  
   504  ### Deprecated Workload API Objects
   505  
   506  `skaffold debug` does not support deprecated versions of Workload API objects:
   507  
   508    - `extensions/v1beta1` and `apps/v1beta1` was [deprecated in Kubernetes 1.8](https://github.com/kubernetes/kubernetes/blob/HEAD/CHANGELOG/CHANGELOG-1.8.md#other-notable-changes-16)
   509      and [removed in Kubernetes 1.16](https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/).
   510    - `apps/v1beta2` was [deprecated in Kubernetes 1.9](https://github.com/kubernetes/kubernetes/blob/HEAD/CHANGELOG/CHANGELOG-1.9.md#apps)
   511      and [removed in Kubernetes 1.16](https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/).
   512  
   513  Applications should transition to the `apps/v1` APIs,
   514  [introduced in Kubernetes 1.9](https://kubernetes.io/blog/2017/12/kubernetes-19-workloads-expanded-ecosystem/#workloads-api-ga).
   515  
   516  ----
   517  
   518  ## Appendix: IDE Support via Events and Metadata {#metadata-events}
   519  
   520  `debug` provides additional support for IDEs to detect the debuggable containers and to determine
   521  appropriate configuration parameters.
   522  
   523  ### Workload Annotations
   524  
   525  Each transformed workload object carries a `debug.cloud.google.com/config` annotation with
   526  a JSON object describing the debug configurations for the pod's containers (linebreaks for readability):
   527  ```
   528  	debug.cloud.google.com/config={
   529  		"<containerName>":{"runtime":"<runtimeId>",...},
   530  		"<containerName>":{"runtime":"<runtimeId>",...},
   531  		}
   532  ```
   533  
   534  For example the following annotation indicates that the container named `web` is a Go application
   535  that is being debugged by a headless Delve session on port `56268` (linebreaks for readability):
   536  ```
   537  debug.cloud.google.com/config={
   538    "web":{
   539      "artifact":"gcr.io/random/image",
   540      "runtime":"go",
   541      "ports":{"dlv":56268},
   542      "workingDir":"/some/path"}}
   543  ```
   544  
   545  `artifact` is the corresponding artifact's image name in the `skaffold.yaml`.
   546  `runtime` is the language runtime detected (one of: `go`, `jvm`, `nodejs`, `python`).
   547  `ports` is a list of debug ports keyed by the language runtime debugging protocol.
   548  `workingDir` is the working directory (if not an empty string).
   549  
   550  
   551  ### API: Events
   552  
   553  Each debuggable container being started or stopped raises a _debug-container-event_ through
   554  Skaffold's event mechanism ([gRPC](../references/api/grpc/#debuggingcontainerevent),
   555  [REST](../references/api/swagger/#/SkaffoldService/Events)).
   556  
   557  <details>
   558  <summary>`/v1/events` stream of `skaffold debug` within `examples/jib`</summary>
   559  
   560  In this example, we do a `skaffold debug`, and then kill the deployed pod.  The deployment starts a new pod.  We get a terminated event for the container for the killed pod.
   561  
   562  ```json
   563  {
   564    "result": {
   565      "timestamp": "2020-02-05T03:27:30.114354Z",
   566      "event": {
   567        "debuggingContainerEvent": {
   568          "status": "Started",
   569          "podName": "web-f6d56bcc5-6csgs",
   570          "containerName": "web",
   571          "namespace": "default",
   572          "artifact": "skaffold-jib",
   573          "runtime": "jvm",
   574          "debugPorts": {
   575            "jdwp": 5005
   576          }
   577        }
   578      },
   579      "entry": "Debuggable container started pod/web-f6d56bcc5-6csgs:web (default)"
   580    }
   581  }
   582  ```
   583  
   584  </details>
   585  
   586  
   587  
   588  ### API: State
   589  
   590  The API's _state_ ([gRPC](../references/api/grpc/#skaffoldservice), [REST](../references/api/swagger/#/SkaffoldService/GetState)) also includes a list of debuggable containers.
   591  
   592  <details>
   593  <summary>The `/v1/state` listing debugging containers</summary>
   594  
   595  ```json
   596  {
   597    "buildState": {
   598      "artifacts": {
   599        "skaffold-jib": "Complete"
   600      }
   601    },
   602    "deployState": {
   603      "status": "Complete"
   604    },
   605    "forwardedPorts": {
   606      "5005": {
   607        "localPort": 5005,
   608        "remotePort": 5005,
   609        "podName": "web-f6d56bcc5-6csgs",
   610        "containerName": "web",
   611        "namespace": "default",
   612        "portName": "jdwp",
   613        "resourceType": "pod",
   614        "resourceName": "web-f6d56bcc5-6csgs",
   615        "address": "127.0.0.1"
   616      },
   617      "8080": {
   618        "localPort": 8080,
   619        "remotePort": 8080,
   620        "namespace": "default",
   621        "resourceType": "service",
   622        "resourceName": "web",
   623        "address": "127.0.0.1"
   624      },
   625      "8081": {
   626        "localPort": 8081,
   627        "remotePort": 8080,
   628        "podName": "web-f6d56bcc5-6csgs",
   629        "containerName": "web",
   630        "namespace": "default",
   631        "resourceType": "pod",
   632        "resourceName": "web-f6d56bcc5-6csgs",
   633        "address": "127.0.0.1"
   634      }
   635    },
   636    "statusCheckState": {
   637      "status": "Not Started"
   638    },
   639    "fileSyncState": {
   640      "status": "Not Started"
   641    },
   642    "debuggingContainers": [
   643      {
   644        "status": "Started",
   645        "podName": "web-f6d56bcc5-6csgs",
   646        "containerName": "web",
   647        "namespace": "default",
   648        "artifact": "skaffold-jib",
   649        "runtime": "jvm",
   650        "debugPorts": {
   651          "jdwp": 5005
   652        }
   653      }
   654    ]
   655  }
   656  
   657  ```
   658  
   659  </details>
   660