github.com/SagerNet/gvisor@v0.0.0-20210707092255-7731c139d75c/g3doc/user_guide/debugging.md (about)

     1  # Debugging
     2  
     3  [TOC]
     4  
     5  To enable debug and system call logging, add the `runtimeArgs` below to your
     6  [Docker](../quick_start/docker/) configuration (`/etc/docker/daemon.json`):
     7  
     8  ```json
     9  {
    10      "runtimes": {
    11          "runsc": {
    12              "path": "/usr/local/bin/runsc",
    13              "runtimeArgs": [
    14                  "--debug-log=/tmp/runsc/",
    15                  "--debug",
    16                  "--strace"
    17              ]
    18         }
    19      }
    20  }
    21  ```
    22  
    23  > Note: the last `/` in `--debug-log` is needed to interpret it as a directory.
    24  > Then each `runsc` command executed will create a separate log file. Otherwise,
    25  > log messages from all commands will be appended to the same file.
    26  
    27  You may also want to pass `--log-packets` to troubleshoot network problems. Then
    28  restart the Docker daemon:
    29  
    30  ```bash
    31  sudo systemctl restart docker
    32  ```
    33  
    34  Run your container again, and inspect the files under `/tmp/runsc`. The log file
    35  ending with `.boot` will contain the strace logs from your application, which
    36  can be useful for identifying missing or broken system calls in gVisor. If you
    37  are having problems starting the container, the log file ending with `.create`
    38  may have the reason for the failure.
    39  
    40  ## Stack traces
    41  
    42  The command `runsc debug --stacks` collects stack traces while the sandbox is
    43  running which can be useful to troubleshoot issues or just to learn more about
    44  gVisor. It connects to the sandbox process, collects a stack dump, and writes it
    45  to the console. For example:
    46  
    47  ```bash
    48  docker run --runtime=runsc --rm -d alpine sh -c "while true; do echo running; sleep 1; done"
    49  63254c6ab3a6989623fa1fb53616951eed31ac605a2637bb9ddba5d8d404b35b
    50  
    51  sudo runsc --root /var/run/docker/runtime-runsc/moby debug --stacks 63254c6ab3a6989623fa1fb53616951eed31ac605a2637bb9ddba5d8d404b35b
    52  ```
    53  
    54  > Note: `--root` variable is provided by docker and is normally set to
    55  > `/var/run/docker/runtime-[runtime-name]/moby`. If in doubt, `--root` is logged
    56  > to `runsc` logs.
    57  
    58  ## Debugger
    59  
    60  You can debug gVisor like any other Golang program. If you're running with
    61  Docker, you'll need to find the sandbox PID and attach the debugger as root.
    62  Here is an example:
    63  
    64  Install a runsc with debug symbols (you can also use the
    65  [nightly release](../install/#nightly)):
    66  
    67  ```bash
    68  make dev BAZEL_OPTIONS="-c dbg"
    69  ```
    70  
    71  Start the container you want to debug using the runsc runtime with debug
    72  options:
    73  
    74  ```bash
    75  docker run --runtime=$(git branch --show-current)-d --rm --name=test -p 8080:80 -d nginx
    76  ```
    77  
    78  Find the PID and attach your favorite debugger:
    79  
    80  ```bash
    81  sudo dlv attach $(docker inspect test | grep Pid | head -n 1 | grep -oe "[0-9]*")
    82  ```
    83  
    84  Set a breakpoint for accept:
    85  
    86  ```bash
    87  break gvisor.dev/gvisor/pkg/sentry/socket/netstack.(*SocketOperations).Accept
    88  continue
    89  ```
    90  
    91  In a different window connect to nginx to trigger the breakpoint:
    92  
    93  ```bash
    94  curl http://localhost:8080/
    95  ```
    96  
    97  ## Profiling
    98  
    99  `runsc` integrates with Go profiling tools and gives you easy commands to
   100  profile CPU and heap usage. First you need to enable `--profile` in the command
   101  line options before starting the container:
   102  
   103  ```json
   104  {
   105      "runtimes": {
   106          "runsc-prof": {
   107              "path": "/usr/local/bin/runsc",
   108              "runtimeArgs": [
   109                  "--profile"
   110              ]
   111         }
   112      }
   113  }
   114  ```
   115  
   116  > Note: Enabling profiling loosens the seccomp protection added to the sandbox,
   117  > and should not be run in production under normal circumstances.
   118  
   119  Then restart docker to refresh the runtime options. While the container is
   120  running, execute `runsc debug` to collect profile information and save to a
   121  file. Here are the options available:
   122  
   123  *   **--profile-heap:** Generates heap profile to the speficied file.
   124  *   **--profile-cpu:** Enables CPU profiler, waits for `--duration` seconds and
   125      generates CPU profile to the speficied file.
   126  
   127  For example:
   128  
   129  ```bash
   130  docker run --runtime=runsc-prof --rm -d alpine sh -c "while true; do echo running; sleep 1; done"
   131  63254c6ab3a6989623fa1fb53616951eed31ac605a2637bb9ddba5d8d404b35b
   132  
   133  sudo runsc --root /var/run/docker/runtime-runsc-prof/moby debug --profile-heap=/tmp/heap.prof 63254c6ab3a6989623fa1fb53616951eed31ac605a2637bb9ddba5d8d404b35b
   134  sudo runsc --root /var/run/docker/runtime-runsc-prof/moby debug --profile-cpu=/tmp/cpu.prof --duration=30s 63254c6ab3a6989623fa1fb53616951eed31ac605a2637bb9ddba5d8d404b35b
   135  ```
   136  
   137  The resulting files can be opened using `go tool pprof` or [pprof][]. The
   138  examples below create image file (`.svg`) with the heap profile and writes the
   139  top functions using CPU to the console:
   140  
   141  ```bash
   142  go tool pprof -svg /usr/local/bin/runsc /tmp/heap.prof
   143  go tool pprof -top /usr/local/bin/runsc /tmp/cpu.prof
   144  ```
   145  
   146  [pprof]: https://github.com/google/pprof/blob/master/doc/README.md
   147  
   148  ### Docker Proxy
   149  
   150  When forwarding a port to the container, Docker will likely route traffic
   151  through the [docker-proxy][]. This proxy may make profiling noisy, so it can be
   152  helpful to bypass it. Do so by sending traffic directly to the container IP and
   153  port. e.g., if the `docker0` IP is `192.168.9.1`, the container IP is likely a
   154  subsequent IP, such as `192.168.9.2`.
   155  
   156  [docker-proxy]: https://windsock.io/the-docker-proxy/