github.com/inspektor-gadget/inspektor-gadget@v0.28.1/docs/builtin-gadgets/trace/oomkill.md (about)

     1  ---
     2  title: 'Using trace oomkill'
     3  weight: 20
     4  description: >
     5    Trace when OOM killer is triggered and kills a process.
     6  ---
     7  
     8  The trace oomkill gadget is used to monitor when out-of-memory killer kills a process.
     9  
    10  ### On Kubernetes
    11  
    12  First, we need to create one pod with limited amount of memory:
    13  
    14  ```bash
    15  $ kubectl apply -f docs/examples/limited-memory-pod.yaml
    16  namespace/oomkill-demo created
    17  pod/test-pod created
    18  ```
    19  
    20  You can now use the gadget, but output will be empty:
    21  
    22  ```bash
    23  $ kubectl gadget trace oomkill -n oomkill-demo
    24  K8S.NODE         K8S.NAMESPACE    K8S.POD          K8S.CONTAINER    KPID   KCOMM            PAGES  TPID             TCOMM
    25  ```
    26  
    27  The gadget is waiting for the OOM killer to get triggered and kill a process in `oomkill-demo` namespace (alternatively, we could use `-A` and get out-of-memory killer events in all namespaces).
    28  
    29  To trigger the OOM killer, in *another terminal*, `exec` a container and run this command to exhaust the memory:
    30  
    31  ```bash
    32  $ kubectl get pod -n oomkill-demo
    33  NAME       READY   STATUS    RESTARTS   AGE
    34  test-pod   1/1     Running   0          52s
    35  $ kubectl exec -n oomkill-demo -ti test-pod -- tail /dev/zero
    36  command terminated with exit code 137
    37  ```
    38  
    39  Go back to *the first terminal* and see:
    40  
    41  ```bash
    42  K8S.NODE         K8S.NAMESPACE    K8S.POD          K8S.CONTAINER    KPID   KCOMM            PAGES  TPID             TCOMM
    43  minikube         oomkill-demo     test-pod         test-container   11507  tail             32768  11507            tail
    44  ```
    45  
    46  The printed lined corresponds to the killing of the `tail` process by the OOM killer.
    47  Here is the full legend of all the fields:
    48  
    49  * `KPID`: The PID of the process killed by the OOM killer (KilledPID).
    50  * `KCOMM`: The command of the process killed by the OOM killer (KilledCommand).
    51  * `PAGES`: The number of pages the killed process had. A page is 4096 bytes on majority of operating system.
    52  * `TPID`: The PID of the process which triggered the OOM killer (TriggeredPID).
    53  * `TCOMM`: The command of the process which triggered the OOM killer (TriggeredCommand).
    54  
    55  The line shown above can also be read like this: "The tail command, with PID 11507, running inside the test-container container, in the test-pod pod, in the oomkill-demo namespace, on the minikube node, was killed by the OOM killer because it allocated 32768 pages. The OOM killer was triggered by tail with PID 11507."
    56  
    57  Note that, in this case, the command which was killed by the OOM killer is the same which triggered it, **this is not always the case**.
    58  
    59  #### Clean everything
    60  
    61  Congratulations! You reached the end of this guide!
    62  You can now delete the resource we created:
    63  
    64  ```bash
    65  $ kubectl delete -f docs/examples/limited-memory-pod.yaml
    66  namespace "oomkill-demo" deleted
    67  pod "test-pod" deleted
    68  ```
    69  
    70  ### With `ig`
    71  
    72  Let's start the gadget in a terminal:
    73  
    74  ```bash
    75  $ sudo ig trace oomkill -c test-trace-oomkill
    76  RUNTIME.CONTAINERNAME                                                                           KPID       KCOMM            PAGES               TPID       TCOMM
    77  ```
    78  
    79  Run a container that will be killed by the OOM killer:
    80  
    81  ```bash
    82  $ docker run --name test-trace-oomkill -m 512M -it --rm busybox tail /dev/zero
    83  ```
    84  
    85  The tool will show the killed process:
    86  
    87  ```bash
    88  $ sudo ig trace oomkill -c test-trace-oomkill
    89  RUNTIME.CONTAINERNAME                                                                           KPID       KCOMM            PAGES               TPID       TCOMM
    90  test-trace-oomkill                                                                              85862      tail             262144              85862      tail
    91  ```