github.com/inspektor-gadget/inspektor-gadget@v0.28.1/docs/builtin-gadgets/trace/tcpconnect.md (about) 1 --- 2 title: 'Using trace tcpconnect' 3 weight: 20 4 description: > 5 Trace connect system calls. 6 --- 7 8 The trace tcpconnect gadget traces TCP connect calls. 9 10 ### On Kubernetes 11 12 In this guide, we will use this gadget to define a restrictive policy for outgoing connections. 13 14 Before we start a demo pod that connects to a public HTTP server, we already begin to trace 15 the outgoing connections of our future pod (don't terminate it with Ctrl-C for now). 16 17 ```bash 18 $ kubectl gadget trace tcpconnect --podname mypod 19 ``` 20 21 When we run the pod in a new terminal, we see the output `ok` since the public HTTP server was reached. 22 23 ```bash 24 $ kubectl run --restart=Never -ti --image=busybox mypod -- sh -c 'wget -q -O /dev/null -T 3 http://1.1.1.1 && echo ok || echo failed' 25 ok 26 ``` 27 28 In our trace tcpconnect gadget terminal we can now see the logged connection: 29 30 ```bash 31 $ kubectl gadget trace tcpconnect --podname mypod 32 K8S.NODE K8S.NAMESPACE K8S.POD K8S.CONTAINER PID COMM IP SRC DST 33 minikube-docker default mypod mypod 2011630 wget 4 p/default/mypod:46779 r/1.1.1.1:80 34 minikube-docker default mypod mypod 2011630 wget 4 p/default/mypod:21731 r/1.1.1.1:443 35 ``` 36 37 If the pod was started as part of a deployment, the name of the pod is not known 38 in advance since random characters will be added as suffix. 39 In that case, it is still possible to trace the connections. We would just 40 use `kubectl gadget trace tcpconnect --selector key=value` to filter the pods by 41 labels instead of names. 42 43 There was a HTTP redirect to HTTPS, so we need to allow both ports for our pod. 44 Don't terminate it yet, we will have another look later. 45 46 Since we now know which network accesses our pod does, we can define and apply a very 47 restrictive network policy: 48 49 ```bash 50 $ cat docs/examples/network-policy.yaml 51 apiVersion: networking.k8s.io/v1 52 kind: NetworkPolicy 53 metadata: 54 name: restrictive-network-policy 55 namespace: default 56 spec: 57 podSelector: 58 matchLabels: 59 run: mypod 60 policyTypes: 61 - Ingress 62 - Egress 63 ingress: 64 - from: 65 - ipBlock: 66 cidr: 1.1.1.1/32 67 egress: 68 - to: 69 - ipBlock: 70 cidr: 1.1.1.1/32 71 ports: 72 - protocol: TCP 73 port: 80 74 - protocol: TCP 75 port: 443 76 77 $ kubectl apply -f docs/examples/network-policy.yaml 78 networkpolicy.networking.k8s.io/restrictive-network-policy created 79 ``` 80 81 Let's test if the pod still works as expected: 82 83 ```bash 84 $ kubectl delete pod mypod 85 $ kubectl run --restart=Never -ti --image=busybox mypod -- sh -c 'wget -q -O /dev/null -T 3 http://1.1.1.1 && echo ok || echo failed' 86 ok 87 88 ``` 89 90 Switching to the gadget trace tcpconnnect terminal, we see the same connections again 91 (but now with a new PID since it's a new pod): 92 93 ```bash 94 $ kubectl gadget trace tcpconnect --podname mypod # (still running in old terminal) 95 K8S.NODE K8S.NAMESPACE K8S.POD K8S.CONTAINER PID COMM IP SRC DST 96 minikube-docker default mypod mypod 2011630 wget 4 p/default/mypod:46779 r/1.1.1.1:80 # (previous output) 97 minikube-docker default mypod mypod 2011630 wget 4 p/default/mypod:21731 r/1.1.1.1:443 # (previous output) 98 minikube-docker default mypod mypod 2011630 wget 4 p/default/mypod:40676 r/1.1.1.1:80 99 minikube-docker default mypod mypod 2011630 wget 4 p/default/mypod:40630 r/1.1.1.1:443 100 ``` 101 102 But what if the pod would connect to other IP addresses which we disallowed? 103 Let's modify our pod to connect to a different address to verify that the connection fails. 104 105 ```bash 106 $ kubectl delete pod mypod 107 $ kubectl run --restart=Never -ti --image=busybox mypod -- sh -c 'wget -q -O /dev/null -T 3 http://1.0.0.1 && echo ok || echo failed' 108 wget: download timed out 109 failed 110 ``` 111 112 Indeed the network policy was applied and we can also see in the gadget output which 113 connection the pod wanted to make in the last line. Since connecting to port 80 failed 114 there is no redirect visible to port 443: 115 116 ```bash 117 $ kubectl gadget trace tcpconnect --podname mypod # (still running in old terminal) 118 K8S.NODE K8S.NAMESPACE K8S.POD K8S.CONTAINER PID COMM IP SRC DST 119 minikube-docker default mypod mypod 2011630 wget 4 p/default/mypod:46779 r/1.1.1.1:80 # (previous output) 120 minikube-docker default mypod mypod 2011630 wget 4 p/default/mypod:21731 r/1.1.1.1:443 # (previous output) 121 minikube-docker default mypod mypod 2011630 wget 4 p/default/mypod:40676 r/1.1.1.1:80 # (previous output) 122 minikube-docker default mypod mypod 2011630 wget 4 p/default/mypod:40630 r/1.1.1.1:443 # (previous output) 123 minikube-docker default mypod mypod 2011630 wget 4 p/default/mypod:17418 r/1.0.0.1:80 124 ``` 125 126 We created a tailored network policy for our (original) demo pod by observing its connection behavior :) 127 Finally, we should delete the demo pod and network policy again: 128 129 ```bash 130 $ kubectl delete pod mypod 131 pod "mypod" deleted 132 $ kubectl delete -f docs/examples/network-policy.yaml 133 networkpolicy.networking.k8s.io "restrictive-network-policy" deleted 134 ``` 135 136 ### With `ig` 137 138 Start the gadget on a terminal. 139 140 ```bash 141 $ sudo ig trace tcpconnect -c test-tcp-connect 142 ``` 143 144 Then, create a gadget that performs a TCP connection. 145 146 ```bash 147 $ docker run -it --rm --name test-tcp-connect busybox /bin/sh -c "wget 1.1.1.1" 148 Connecting to 1.1.1.1 (1.1.1.1:80) 149 Connecting to 1.1.1.1 (1.1.1.1:443) 150 wget: note: TLS certificate validation not implemented 151 saving to 'index.html' 152 index.html 100% |********************************| 54361 0:00:00 ETA 153 'index.html' saved 154 ``` 155 156 The gadget will show the connection and related information to it. 157 158 ```bash 159 $ sudo ig trace tcpconnect -c test-tcp-connect 160 RUNTIME.CONTAINERNAME PID COMM IP SRC DST 161 test-tcp-connect 2021739 wget 4 172.17.0.2:4784 1.1.1.1:80 162 test-tcp-connect 2021739 wget 4 172.17.0.2:14023 1.1.1.1:443 163 ``` 164 165 ### Calculating the latency of a connection 166 167 This tools provides a `--latency` option to show the latency (time) for the connection as measured 168 from the TCP client perspective: the time from SYN sent to the response packet. 169 170 TCP connection latency is a useful performance measure showing the time taken to establish a 171 connection. This typically involves kernel TCP/IP processing and the network round trip time, and 172 not application runtime. 173 174 This measures the time from any connection to the response packet, even if the response is a RST 175 (port closed). 176 177 When this option is used, the event is only shown when the server replies or the socket is removed. 178 179 #### On Kubernetes 180 181 Start the gadget on a terminal: 182 183 ```bash 184 $ kubectl gadget trace tcpconnect --latency 185 RUNTIME.CONTAINERNAME PID COMM IP SRC DST LATENCY``` 186 187 In another terminal, create a nginx service and a pod to send some http requests: 188 189 ```bash 190 $ kubectl create service nodeport nginx --tcp=80:80 191 $ kubectl create deployment nginx --image=nginx 192 $ kubectl run -ti --privileged --image wbitt/network-multitool myclientpod -- bash 193 ``` 194 195 Send some requests to the server: 196 197 ```bash 198 # curl nginx 199 # curl nginx 200 ... 201 ``` 202 203 The first terminal show all those connections and their latency. In my case both pods are running on 204 the same node, so it's very low: 205 206 ```bash 207 K8S.NODE K8S.NAMESPACE K8S.POD K8S.CONTAINER PID COMM IP SRC DST LATENCY 208 minikube-docker default myclientpod myclientpod 2054329 curl 4 p/default/myclientpod:50306 s/default/nginx:80 47.069µs 209 minikube-docker default myclientpod myclientpod 2054338 curl 4 p/default/myclientpod:53378 s/default/nginx:80 120.017µs 210 ``` 211 212 Now, let's use the [network emulator](https://wiki.linuxfoundation.org/networking/netem) to 213 introduce some random delay to the packets on the container interface and send some more requests to 214 the server again: 215 216 ```bash 217 # tc qdisc add dev eth0 root netem delay 50ms 50ms 25% 218 # curl nginx 219 # curl nginx 220 ``` 221 222 Now the latency is a lot higher and has some variance because of the emulation configuration: 223 224 ```bash 225 K8S.NODE K8S.NAMESPACE K8S.POD K8S.CONTAINER PID COMM IP SRC DST LATENCY 226 ... 227 minikube-docker default myclientpod myclientpod 2056697 curl 4 p/default/myclientpod:32415 s/default/nginx:80 7.820966ms 228 minikube-docker default myclientpod myclientpod 2056832 curl 4 p/default/myclientpod:32927 s/default/nginx:80 64.388825ms 229 minikube-docker default myclientpod myclientpod 2056905 curl 4 p/default/myclientpod:46811 s/default/nginx:80 39.244112ms 230 ``` 231 232 #### With `ig` 233 234 Start the trace tcpconnect gadget on a first terminal: 235 236 ```bash 237 $ sudo ig trace tcpconnect --latency 238 RUNTIME.CONTAINERNAME PID COMM IP SRC DST LATENCY 239 ``` 240 241 Then, start a container and download a web page: 242 243 ```bash 244 $ docker run -ti --rm --cap-add NET_ADMIN --name=netem wbitt/network-multitool -- /bin/bash 245 # wget 1.1.1.1 246 ``` 247 248 The first terminal will show the connections created and their latency: 249 250 ```bash 251 RUNTIME.CONTAINERNAME PID COMM IP SRC DST LATENCY 252 netem 2036550 wget 4 172.17.0.2:47250 1.1.1.1:80 14.149828ms 253 netem 2036550 wget 4 172.17.0.2:44734 1.1.1.1:443 15.025666ms 254 ``` 255 256 In this case, it can be seen that two connections were made by the `curl` command and their latency 257 was around 50ms for each of them. 258 259 Now, let's configure the network emulator to add a delay of one second to all packets in the 260 container's interface and then download the web page again: 261 262 ```bash 263 # tc qdisc add dev eth0 root netem delay 1000ms 10ms 100% 264 # wget 1.1.1.1 265 ``` 266 267 In this case the `wget` command takes way longer to complete the request. We can check that the 268 latency for those connections is more than one second as expected: 269 270 ```bash 271 RUNTIME.CONTAINERNAME PID COMM IP SRC DST LATENCY 272 ... 273 netem 2037935 wget 4 172.17.0.2:38587 1.1.1.1:80 1.006814808s 274 ... 275 netem 2037935 wget 4 172.17.0.2:10469 1.1.1.1:443 1.010320064s 276 ```