github.com/inspektor-gadget/inspektor-gadget@v0.28.1/docs/builtin-gadgets/traceloop.md (about) 1 --- 2 title: 'Using traceloop' 3 weight: 30 4 description: > 5 Get strace-like logs of a container from the past. 6 --- 7 8 The `traceloop` gadget is used to trace system calls issued by containers. 9 10 ### On Kubernetes 11 12 #### Start traceloop 13 14 Traceloop is disabled by default from version 0.4.0. It can be enabled by using: 15 16 ```bash 17 $ kubectl gadget traceloop start 18 ``` 19 20 ###### Multiplication demo 21 22 Let's run a pod to compute an important multiplication: 23 24 ```bash 25 $ kubectl create ns test 26 $ kubectl run --restart=Never -n test -ti --image=busybox mypod -- sh -c 'RANDOM=output ; echo "3*7*2" | bc > /tmp/file-$RANDOM ; cat /tmp/file-$RANDOM' 27 cat: can't open '/tmp/file-3240': No such file or directory 28 pod default/mypod terminated (Error) 29 $ kubectl delete pod -n test mypod 30 pod "mypod" deleted 31 ``` 32 33 Oh no! We made a mistake in the shell script: we opened the wrong file. Is the 34 result lost forever? Let's check with the traceloop gadget: 35 36 ```bash 37 $ kubectl gadget traceloop list 38 K8S.NODE K8S.NAMESPACE K8S.POD K8S.CONTAINER CONTAINERID 39 minikube kube-system kube-scheduler-minikube kube-scheduler 2b63eb745ce2cf 40 ... 41 minikube test mypod mypod ef6f2d3f44b555 42 ``` 43 44 Let's inspect the traceloop log: 45 46 ```bash 47 $ kubectl gadget traceloop show ef6f2d3f44b555 48 CPU PID COMM NAME PARAMS RET 49 ... 50 1 337825 sh write fd=1, buf=34716864 3*7*2 51 , count=6 6 52 ... 53 1 337826 sh open filename=34717192 /tmp/file-24581, flags=577, mode=438 3 54 ... 55 1 337826 sh execve filename=34717344 /bin/bc, argv=34717176, envp=34717224 0 56 ... 57 1 337826 bc read fd=0, buf=5355360 3*7*2 58 , count=4096 6 59 1 337826 bc write fd=1, buf=5359456 42 60 , count=3 3 61 ... 62 0 337813 cat open filename=140736754175569 /tmp/file-3240, flags=0, mode=0 -2 63 0 337813 cat write fd=2, buf=140736754168912 cat: can't open '/tmp/file-3240': No such file or directory 64 , coun… 60 65 0 337813 cat exit_group error_code=1 X 66 ``` 67 68 Thanks to the traceloop gadget, we can recover the result of the 69 multiplication: 42. And we can understand the mistake in the shell script: the 70 result was saved in `/tmp/file-24581` but we attempted to open 71 `/tmp/file-3240`. 72 Note that, return value of `exit_group()` is `X` as this syscall never returns. 73 The same applies for `exit()` and `rt_sigreturn()`. 74 75 You can now remove the trace associated to this container: 76 77 ```bash 78 $ kubectl gadget traceloop delete ef6f2d3f44b555 79 ``` 80 81 And if you no more need the `traceloop` gadget, you can stop it: 82 83 ```bash 84 $ kubectl gadget traceloop stop 85 ``` 86 87 ##### Listing files demo 88 89 With traceloop, we can strace pods in the past, even after they terminated. 90 91 Example: let's list the programs in /bin: 92 93 ```bash 94 $ kubectl run -n test --restart=Never -ti --image=debian mypod -- sh -c 'ls -l /bin | grep mv' 95 -rwxr-xr-x 1 root root 147080 Sep 24 2020 mv 96 $ kubectl delete pod mypod 97 pod "mypod" deleted 98 ``` 99 100 Because of the `grep mv`, we only see one entry. But traceloop can recover other entries: 101 102 ```bash 103 $ kubectl gadget traceloop list 104 ... 105 minikube test mypod mypod 9c691a53cd43a0 106 $ kubectl gadget traceloop show 9c691a53cd43a0 107 ... 108 1 97851 ls lgetxattr pathname=140729093707632 /bin/more, name=139950439515987 security.selinux, value=94534240126… -61 109 1 97851 ls getxattr pathname=140729093707632 /bin/more, name=94534234714099 system.posix_acl_access, value=0, si… -61 110 1 97851 ls statx dfd=4294967196, filename=140729093707632 /bin/vdir, flags=256, mask=606, buffer=140729093707… 0 111 1 97851 ls lgetxattr pathname=140729093707632 /bin/vdir, name=139950439515987 security.selinux, value=94534240126… -61 112 1 97851 ls getxattr pathname=140729093707632 /bin/vdir, name=94534234714099 system.posix_acl_access, value=0, si… -61 113 1 97851 ls statx dfd=4294967196, filename=140729093707632 /bin/rmdir, flags=256, mask=606, buffer=14072909370… 0 114 1 97851 ls lgetxattr pathname=140729093707632 /bin/rmdir, name=139950439515987 security.selinux, value=9453424012… -61 115 1 97851 ls getxattr pathname=140729093707632 /bin/rmdir, name=94534234714099 system.posix_acl_access, value=0, s… -61 116 1 97851 ls statx dfd=4294967196, filename=140729093707632 /bin/tempfile, flags=256, mask=606, buffer=14072909… 0 117 1 97851 ls lgetxattr pathname=140729093707632 /bin/tempfile, name=139950439515987 security.selinux, value=9453424… -61 118 1 97851 ls getxattr pathname=140729093707632 /bin/tempfile, name=94534234714099 system.posix_acl_access, value=0… -61 119 1 97851 ls statx dfd=4294967196, filename=140729093707632 /bin/zfgrep, flags=256, mask=606, buffer=1407290937… 0 120 1 97851 ls lgetxattr pathname=140729093707632 /bin/zfgrep, name=139950439515987 security.selinux, value=945342401… -61 121 1 97851 ls getxattr pathname=140729093707632 /bin/zfgrep, name=94534234714099 system.posix_acl_access, value=0, … -61 122 1 123 ... 124 ``` 125 126 ### With `ig` 127 128 Start a container in interactive mode: 129 130 ```bash 131 $ docker run -it --rm --name test-traceloop busybox /bin/sh 132 ``` 133 134 135 Start traceloop in another terminal: 136 137 ```bash 138 $ sudo ig traceloop -c test-traceloop 139 ``` 140 141 Run a command inside the container 142 143 ```bash 144 $ docker run -it --rm --name test-traceloop busybox /bin/sh 145 / # ls 146 147 ``` 148 149 Press Ctrl+C on the gadget terminal, it'll print the systemcalls performed by the container: 150 151 ```bash 152 $ sudo ig traceloop -c test-traceloop 153 Tracing syscalls... Hit Ctrl-C to end 154 ^C 155 CPU PID COMM NAME PARAMS RET 156 ... 157 6 150829 sh execve filename=18759352 /bin/ls, argv=18759280, envp=18759296 0 158 6 150829 ls brk brk=0 36… 159 6 150829 ls brk brk=36440320 36… 160 ... 161 6 150829 ls write fd=1, buf=5355360 bin dev etc home pro… 158 162 6 150829 ls exit_group error_code=0 ... 163 ``` 164 165 ### Filter by syscalls 166 167 If you are only interested by specific syscalls, you can use the `--syscall-filters` flag: 168 169 ```bash 170 $ kubectl gadget traceloop start --syscall-filters openat,write,execve 171 $ kubectl create ns test 172 $ kubectl run --restart=Never -n test --rm -ti --image=busybox mypod -- sh -c 'RANDOM=output ; echo "3*7*2" | bc > /tmp/file-$RANDOM ; cat /tmp/file-$RANDOM' 173 cat: can't open '/tmp/file-3240': No such file or directory 174 pod default/mypod terminated (Error) 175 $ kubectl delete pod -n test mypod 176 pod "mypod" deleted 177 $ kubectl gadget traceloop list 178 NODE NAMESPACE POD CONTAINER CONTAINERID 179 ... 180 minikube test mypod mypod 50f702a9e3f0cd2 181 $ kubectl gadget traceloop show 50f702a9e3f0cd2 182 CPU PID COMM SYSCALL PARAMS RET 183 ... 184 1 118483 sh execve filename="/bin/bc", argv=0x5576c8219668, envp=0x5576c8219698 0 185 ... 186 5 118470 sh execve filename="/bin/cat", argv=0x5576c8219688, envp=0x5576c82196a0 0 187 ... 188 1 10685 cat openat dfd=4294967196, filename="/tmp/file-3240", flags=0, mode=0 -1 (… 189 1 10685 cat write fd=2, buf="cat: can't open '/tmp/file-3240': No such file or director… 60 190 ``` 191 192 This options is also available with `ig`: 193 194 ```bash 195 $ sudo ig traceloop -c test-traceloop --syscall-filters openat,write 196 RUNTIME.CONTAINERNAME CPU PID COMM SYSCALL PARAMS RET 197 # Run a container in a second terminal 198 $ docker run --rm --name test-traceloop busybox ls 199 bin 200 dev 201 ... 202 # Go back to first terminal 203 ^C 204 ... 205 test-traceloop 1 135771 ls openat dfd=4294967196, filename=".", flags=591872,… 3 206 test-traceloop 1 135771 ls write fd=1, buf="bin\ndev\netc\nhome\nlib\nlib64\… 53 207 f 208 ```