github.com/inspektor-gadget/inspektor-gadget@v0.28.1/docs/builtin-gadgets/profile/tcprtt.md (about) 1 --- 2 title: 'Using profile tcprtt' 3 weight: 20 4 description: > 5 Analyze TCP connections through an Round-Trip Time (RTT) distribution 6 --- 7 8 The profile tcprtt gadget generates a histogram distribution of the TCP 9 connections' Round-Trip Time (RTT). The RTT values used to create the histogram 10 are collected from [the smoothed 11 RTT](https://elixir.bootlin.com/linux/v5.11.22/source/include/linux/tcp.h#L258) 12 information already provided by the Linux kernel for the TCP sockets. 13 14 The histogram considers only the TCP connections that have been already 15 established, so it does not take into account the connection phase (3-way TCP 16 Handshake). If it is what you are looking for, please check the latency 17 information the trace tcpconnect gadget provides. See further information 18 [here](../trace/tcpconnect.md#calculating-the-latency-of-a-connection). 19 20 By default, the profile tcprtt gadget generates one single histogram per host, 21 and one per node in case of Kubernetes. However, it also provides multiple ways 22 to analyze specific connections. For instance, we can generate multiple 23 histograms separated by local IP addresses to explore all the connections 24 generated by each local address. And the same can be done for remote IP 25 addresses. In addition, it is also possible to filter by a specific local and/or 26 remote address to isolate the analysis. 27 28 ### On Kubernetes 29 30 First of all, let's start the gadget on a terminal: 31 32 ```bash 33 kubectl gadget profile tcprtt 34 ``` 35 36 In another terminal, create a server using nginx: 37 38 ```bash 39 kubectl create service nodeport nginx --tcp=80:80 40 kubectl create deployment nginx --image=nginx 41 ``` 42 43 And then, create a pod to generate some traffic with the server: 44 45 ```bash 46 $ kubectl run -ti --privileged --image wbitt/network-multitool myclientpod -- bash 47 # curl nginx 48 # curl nginx 49 ``` 50 51 If we move back to the first terminal and stop the gadget, it will generate the histograms: 52 53 ```bash 54 All Addresses = ****** [AVG 1211.066824] 55 µs : count distribution 56 0 -> 1 : 0 | | 57 2 -> 3 : 0 | | 58 4 -> 7 : 0 | | 59 8 -> 15 : 114 |*********** | 60 16 -> 31 : 397 |****************************************| 61 32 -> 63 : 182 |****************** | 62 64 -> 127 : 49 |**** | 63 128 -> 255 : 48 |**** | 64 256 -> 511 : 107 |********** | 65 512 -> 1023 : 108 |********** | 66 1024 -> 2047 : 86 |******** | 67 2048 -> 4095 : 31 |*** | 68 4096 -> 8191 : 111 |*********** | 69 8192 -> 16383 : 28 |** | 70 16384 -> 32767 : 11 |* | 71 ``` 72 73 Take into account that the generated histogram considers all TCP connections for 74 each node, not only the ones we established. Notice that we used a cluster with 75 a single node for this guide. 76 77 So, let's repeat the test but this time filtering by remote address so that we 78 can analyse the traffic we are generating toward our nginx service: 79 80 ```bash 81 $ kubectl get service nginx -o jsonpath={.spec.clusterIP} 82 10.0.38.234 83 $ kubectl gadget profile tcprtt --raddr 10.0.38.234 84 All Addresses = ****** [AVG 2087.000000] 85 µs : count distribution 86 0 -> 1 : 0 | | 87 2 -> 3 : 0 | | 88 4 -> 7 : 0 | | 89 8 -> 15 : 0 | | 90 16 -> 31 : 0 | | 91 32 -> 63 : 0 | | 92 64 -> 127 : 0 | | 93 128 -> 255 : 0 | | 94 256 -> 511 : 0 | | 95 512 -> 1023 : 0 | | 96 1024 -> 2047 : 3 |****************************************| 97 2048 -> 4095 : 3 |****************************************| 98 99 ``` 100 101 Now, let's use the [network 102 emulator](https://wiki.linuxfoundation.org/networking/netem) to introduce some 103 random delay to the packets and increase indirectly the RTT: 104 105 ```bash 106 # tc qdisc add dev eth0 root netem delay 50ms 50ms 25% 107 # curl nginx 108 # curl nginx 109 ``` 110 111 Now the average RTT value of the new histogram is clearly higher: 112 113 ```bash 114 $ kubectl gadget profile tcprtt --raddr 10.0.38.234 115 All Addresses = ****** [AVG 68973.833333] 116 µs : count distribution 117 0 -> 1 : 0 | | 118 2 -> 3 : 0 | | 119 4 -> 7 : 0 | | 120 8 -> 15 : 0 | | 121 16 -> 31 : 0 | | 122 32 -> 63 : 0 | | 123 64 -> 127 : 0 | | 124 128 -> 255 : 0 | | 125 256 -> 511 : 0 | | 126 512 -> 1023 : 0 | | 127 1024 -> 2047 : 0 | | 128 2048 -> 4095 : 0 | | 129 4096 -> 8191 : 0 | | 130 8192 -> 16383 : 0 | | 131 16384 -> 32767 : 0 | | 132 32768 -> 65535 : 3 |****************************************| 133 65536 -> 131071 : 3 |****************************************| 134 135 ``` 136 137 ### With `ig` 138 139 Start the profile tcprtt gadget on a first terminal: 140 141 ```bash 142 sudo ig profile tcprtt 143 ``` 144 145 Then, start a container and download a web page: 146 147 ```bash 148 $ docker run -ti --rm --cap-add NET_ADMIN --name=netem wbitt/network-multitool -- /bin/bash 149 # wget 1.1.1.1 150 ``` 151 152 Moving back to the first terminal and stopping the gadget, it will generate the histograms: 153 154 ```bash 155 All Addresses = ****** [AVG 2343.510333] 156 µs : count distribution 157 0 -> 1 : 0 | | 158 2 -> 3 : 0 | | 159 4 -> 7 : 25 | | 160 8 -> 15 : 226 |** | 161 16 -> 31 : 777 |******** | 162 32 -> 63 : 1532 |**************** | 163 64 -> 127 : 2822 |******************************* | 164 128 -> 255 : 2254 |************************ | 165 256 -> 511 : 3305 |************************************ | 166 512 -> 1023 : 2863 |******************************* | 167 1024 -> 2047 : 1284 |************** | 168 2048 -> 4095 : 1456 |**************** | 169 4096 -> 8191 : 3612 |****************************************| 170 8192 -> 16383 : 167 |* | 171 16384 -> 32767 : 0 | | 172 32768 -> 65535 : 14 | | 173 65536 -> 131071 : 75 | | 174 131072 -> 262143 : 0 | | 175 262144 -> 524287 : 0 | | 176 524288 -> 1048575 : 8 | | 177 ``` 178 179 This histogram represents the distribution of the RTT for all the TCP 180 connections established in the host and not only the TCP connections we 181 established in the test container. 182 183 Let's repeat the test but this time filtering by remote address so that we can 184 analyse the traffic we are generating: 185 186 ```bash 187 $ sudo ig profile tcprtt --raddr 1.1.1.1 188 All Addresses = ****** [AVG 7359.357143] 189 µs : count distribution 190 0 -> 1 : 0 | | 191 2 -> 3 : 0 | | 192 4 -> 7 : 0 | | 193 8 -> 15 : 0 | | 194 16 -> 31 : 0 | | 195 32 -> 63 : 0 | | 196 64 -> 127 : 0 | | 197 128 -> 255 : 0 | | 198 256 -> 511 : 0 | | 199 512 -> 1023 : 0 | | 200 1024 -> 2047 : 0 | | 201 2048 -> 4095 : 0 | | 202 4096 -> 8191 : 9 |****************************************| 203 8192 -> 16383 : 5 |********************** | 204 ``` 205 206 Now, let's introduce some random delay to the packets to increase indirectly the 207 RTT using the [network 208 emulator](https://wiki.linuxfoundation.org/networking/netem): 209 210 ```bash 211 # tc qdisc add dev eth0 root netem delay 50ms 50ms 25% 212 # wget 1.1.1.1 213 ``` 214 215 And, regenerate the histogram to see the change in the RTT: 216 217 ```bash 218 $ sudo ig profile tcprtt --raddr 1.1.1.1 219 All Addresses = ****** [AVG 72278.307692] 220 µs : count distribution 221 0 -> 1 : 0 | | 222 2 -> 3 : 0 | | 223 4 -> 7 : 0 | | 224 8 -> 15 : 0 | | 225 16 -> 31 : 0 | | 226 32 -> 63 : 0 | | 227 64 -> 127 : 0 | | 228 128 -> 255 : 0 | | 229 256 -> 511 : 0 | | 230 512 -> 1023 : 0 | | 231 1024 -> 2047 : 0 | | 232 2048 -> 4095 : 0 | | 233 4096 -> 8191 : 0 | | 234 8192 -> 16383 : 0 | | 235 16384 -> 32767 : 0 | | 236 32768 -> 65535 : 0 | | 237 65536 -> 131071 : 13 |****************************************| 238 ``` 239 240 We can see how the average RTT passed from 7359.357143 to 72278.307692.