github.com/inspektor-gadget/inspektor-gadget@v0.28.1/docs/builtin-gadgets/top/ebpf.md (about) 1 --- 2 title: 'Using top ebpf' 3 weight: 20 4 description: > 5 Periodically report ebpf runtime stats. 6 --- 7 8 The top ebpf gadget is used to visualize the usage and performance of eBPF programs. 9 10 ### On Kubernetes 11 12 This guide will let you run another gadget (we're using `top file` in this example) and see its metrics 13 using `top ebpf`. 14 15 So first, start `top ebpf` in a terminal. You should see something like: 16 17 ```bash 18 $ kubectl gadget top ebpf 19 K8S.NODE PROGID TYPE NAME PID COMM RUNTIME RUNCOUNT MAPMEMORY MAPCOUNT 20 minikube 503 Tracing ig_top_ebpf_it 573222 gadgettracerman 54.09µs 1069 12B 1 21 minikube 187 CGroupDevice 2.292µs 1 0B 0 22 minikube 13 CGroupSKB 0s 0 0B 0 23 ... 24 ``` 25 26 The output should already contain the `top ebpf` gadget itself. 27 28 Now, in a second terminal, start `top file` on the _gadget_ namespace to get some output. 29 30 ```bash 31 $ kubectl gadget top file -n gadget 32 K8S.NODE K8S.NAMESPACE K8S.POD K8S.CONTAINER PID COMM READS WRITES R_KB W_KB T FILE 33 minikube gadget gadget-k2mvp gadget 575955 runc:[2:INIT] 1 0 0 0 R cap_last_cap 34 minikube gadget gadget-k2mvp gadget 575955 runc:[2:INIT] 2 0 8 0 R group 35 minikube gadget gadget-k2mvp gadget 575955 gadgettracerman 2 0 8 0 R UTC 36 ... 37 ``` 38 39 While the `top file` gadget is still running, go back to the first terminal. 40 Some eBPF programs of type `Kprobe` should pop up, including their runtime and run count in the current interval 41 (default is 1s): 42 43 ```bash 44 $ kubectl gadget top ebpf 45 K8S.NODE PROGID TYPE NAME PID COMM RUNTIME RUNCOUNT MAPMEMORY MAPCOUNT 46 minikube 506 Kprobe ig_topfile_rd_e 573222 gadgettracerman 824.589µs 2076 40.95MiB 4 47 minikube 505 Tracing ig_top_ebpf_it 573222 gadgettracerman 47.171µs 1103 12B 1 48 minikube 507 Kprobe ig_topfile_wr_e 573222 gadgettracerman 609.645µs 836 40.95MiB 4 49 minikube 187 CGroupDevice 4.417µs 2 0B 0 50 minikube 13 CGroupSKB 0s 0 0B 0 51 minikube 8 CGroupSKB 0s 0 0B 0 52 ... 53 ``` 54 55 So in this case for example, in the past second `vfs_write_entry` has been called 647 times, which took 614.455µs. 56 The program references 4 maps that have a total maximum size of 40.953 MB (see below for more information on MapMemory). 57 58 If you want to get the cumulative runtime and run count of the eBPF programs starting from the beginning of the trace, 59 you can call the gadget with the columns option and specify the cumulruntime and cumulruncount columns. 60 Combined with the `--sort cumulruntime` and `--timeout 60` parameters, you can for example measure the time spent 61 over a minute: 62 63 ```bash 64 $ kubectl-gadget top ebpf -o columns=k8s.node,progid,type,name,pid,comm,cumulruntime,cumulruncount --sort cumulruntime --timeout 60 65 K8S.NODE PROGID TYPE NAME PID COMM CUMULRUNTIME CUMULRUNCOUNT 66 minikube 509 Tracing ig_top_ebpf_it 573222 gadgettracerman 1.265693ms 15879 67 minikube 187 CGroupDevice 40.795µs 48 68 minikube 256 CGroupDevice 5.834µs 2 69 minikube 13 CGroupSKB 0s 0 70 minikube 7 CGroupSKB 0s 0 71 minikube 8 CGroupSKB 0s 0 72 ... 73 ``` 74 75 ### With `ig` 76 77 Run any tool that uses eBPF, for instance, [iovisor/bcc `execsnoop`](https://github.com/iovisor/bcc/blob/v0.26.0/libbpf-tools/execsnoop.bpf.c): from BCC. Please note that it can be run on the host directly. 78 79 80 ```bash 81 $ sudo ./execsnoop 82 PCOMM PID PPID RET ARGS 83 runc 167851 142428 0 /usr/bin/runc --version 84 docker-init 167857 142428 0 /usr/bin/docker-init --version 85 ... 86 ``` 87 88 Start the `top ebpf` gadget in another terminal. It'll show the programs installed by the tool above. 89 90 ```bash 91 $ sudo ig top ebpf 92 PROGID TYPE NAME PID COMM RUNTIME RUNCOUNT MAPMEMORY MAPCOUNT 93 1102 Tracing ig_top_ebpf_it 167925 ig 299.054µs 5534 4KiB 1 94 1097 TracePoint tracepoint__sys 167850 execsnoop 25.055µs 2 75.48MiB 3 95 1099 TracePoint tracepoint__sys 167850 execsnoop 23.629µs 2 75.48MiB 4 96 ``` 97 98 ### A note about memory usage of maps 99 100 The shown value for MapMemory is read from `/proc/<pid>/fdinfo/<map_id>`. 101 This is the maximum size the map can have, but it doesn't necessarily reflect its current memory allocation. Additionally, maps can 102 be used by more than one program and would account towards the MapMemory of all those programs. 103 104 Also note: 105 * BPF_MAP_TYPE_PERF_EVENT_ARRAY: value_size is not counting the ring buffers, but only their file descriptors (i.e. sizeof(int) = 4 bytes) 106 * BPF_MAP_TYPE_{HASH,ARRAY}_OF_MAPS: value_size is not counting the inner maps, but only their file descriptors (i.e. sizeof(int) = 4 bytes) 107 108 ### A note about CPU usage 109 110 There are two types of cpu usage metrics available in top ebpf gadget: 111 * TotalCPUUsage: It means all ebpf progs' share of the elapsed CPU time since the last update, expressed as a percentage of total CPU time. This value can be >100% because it summed up cpu time in all cpu cores, similar to linux `top`. 112 * PerCPUUsage: This value is `TotalCPUUsage / CPUCores` and `CPUCores` is the number of cpu cores in a node. PerCPUUsage will be always be <=100%. When using ebpf top gadget to check cpu usage for many nodes, it's suggested to use PerCPUUsage. 113 114 To show them, you can run the gadget with `percpu` and `totalcpu` specified in `-o columns` option.