github.com/inspektor-gadget/inspektor-gadget@v0.28.1/docs/builtin-gadgets/trace/tcpdrop.md (about)

     1  ---
     2  title: 'Using trace tcpdrop'
     3  weight: 20
     4  description: >
     5      Trace TCP kernel-dropped packets/segments.
     6  ---
     7  
     8  The trace tcpdrop gadget traces TCP packets dropped by the kernel.
     9  
    10  ### On Kubernetes
    11  
    12  In terminal 1, start the trace tcpdrop gadget:
    13  
    14  ```bash
    15  $ kubectl gadget trace tcpdrop
    16  K8S.NODE         K8S.NAMESPACE  K8S.POD K8s.CONTAINER  PID     COMM  IP SRC                    DST                        STATE        TCPFLAGS  REASON
    17  ```
    18  
    19  In terminal 2, start a pod and configure the network emulator to drop 25% of the packets:
    20  
    21  ```bash
    22  $ kubectl create service nodeport nginx --tcp=80:80
    23  $ kubectl create deployment nginx --image=nginx
    24  $ kubectl run --rm -ti --privileged --image ubuntu shell -- bash
    25  root@shell:/# apt-get update
    26  root@shell:/# apt install -y iproute2 curl
    27  root@shell:/# tc qdisc add dev eth0 root netem drop 25%
    28  root@shell:/# curl nginx
    29  ```
    30  
    31  The results in terminal 1 will show that some packets are dropped by the network emulator qdisc:
    32  
    33  ```
    34  K8S.NODE         K8S.NAMESPACE  K8S.POD K8s.CONTAINER  PID     COMM  IP SRC                    DST                        STATE        TCPFLAGS  REASON
    35  minikube-docker  default        shell   shell          0             4  p/default/shell:45979  s/kube-system/kube-dns:53  ESTABLISHED  FIN       QDISC_DROP
    36  minikube-docker  default        shell   shell          406293  curl  4  p/default/shell:34482  s/default/nginx:80         ESTABLISHED  ACK       QDISC_DROP
    37  ```
    38  
    39  The network emulator uses a random generator to drop 25% of the packets.
    40  The results may vary.
    41  
    42  The gadget tries its best to link the dropped packets to the process which generated it.
    43  In some cases, this information might be missing.
    44  
    45  The source and destination addresses are written in condensed form.
    46  It is possible to see more detailed information by reading specific columns or using the json or yaml ouput:
    47  
    48  ```
    49  $ kubectl gadget trace tcpdrop \
    50      -o columns=k8s.node,k8s.namespace,k8s.pod,k8s.container,pid,comm,ip,src.addr,src.port,src.kind,src.ns,src.name,dst.addr,dst.port,dst.kind,dst.ns,dst.name,state,tcpflags,reason
    51  ```
    52  
    53  ```
    54  $ kubectl gadget trace tcpdrop -o yaml
    55  ---
    56  comm: curl
    57  container: shell
    58  dst:
    59    addr: 10.101.116.61
    60    kind: svc
    61    namespace: default
    62    podlabels:
    63      app: nginx
    64    podname: nginx
    65    port: 80
    66  gid: 0
    67  ipversion: 4
    68  mountnsid: 4026533845
    69  namespace: default
    70  netnsid: 4026533672
    71  node: minikube-docker
    72  pid: 412491
    73  pod: shell
    74  reason: QDISC_DROP
    75  src:
    76    addr: 10.244.0.91
    77    kind: pod
    78    namespace: default
    79    podlabels:
    80      run: shell
    81    podname: shell
    82    port: 35802
    83  state: ESTABLISHED
    84  tcpflags: ACK
    85  timestamp: 1681911565379499967
    86  type: normal
    87  uid: 0
    88  ```
    89  
    90  ### With `ig`
    91  
    92  In terminal 1, start the trace tcpdrop gadget:
    93  
    94  ```bash
    95  $ sudo ig trace tcpdrop -r docker
    96  CONTAINER  PID     COMM  IP SRC               DST          STATE        TCPFLAGS  REASON
    97  ```
    98  
    99  In terminal 2, start a container, configure the network emulator to drop 25% of the packets, and download a web page:
   100  
   101  ```bash
   102  $ docker run -ti --rm --cap-add NET_ADMIN --name=netem wbitt/network-multitool -- /bin/bash
   103  # tc qdisc add dev eth0 root netem drop 25%
   104  # wget 1.1.1.1
   105  ```
   106  
   107  The container needs NET_ADMIN capability to manage network interfaces
   108  
   109  The results in terminal 1 will show that some packets are dropped by the network emulator qdisc:
   110  
   111  ```
   112  CONTAINER  PID     COMM  IP SRC               DST          STATE        TCPFLAGS  REASON
   113  netem      456426  wget  4  172.17.0.2:35790  1.1.1.1:443  ESTABLISHED  ACK       QDISC_DROP
   114  ```
   115  
   116  The following section tells us that QDISC_DROP means the packet was "dropped by qdisc when packet outputting (failed to enqueue to current qdisc)".
   117  
   118  ### List of drop reasons
   119  
   120  The drop reason enum is not stable and may change between kernel versions.
   121  The tcpdrop gadget needs BTF information to decode the drop reason.
   122  The following table shows the list of drop reasons for Linux 6.2.
   123  
   124  <!-- markdown-link-check-disable -->
   125  
   126  | Name    | Documentation |
   127  |---------|---------------|
   128  | [SKB_NOT_DROPPED_YET](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_NOT_DROPPED_YET&type=code) | skb is not dropped yet (used for no-drop case) |
   129  | [SKB_CONSUMED](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_CONSUMED&type=code) | packet has been consumed |
   130  | [SKB_DROP_REASON_NOT_SPECIFIED](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_NOT_SPECIFIED&type=code) | drop reason is not specified |
   131  | [SKB_DROP_REASON_NO_SOCKET](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_NO_SOCKET&type=code) | socket not found |
   132  | [SKB_DROP_REASON_PKT_TOO_SMALL](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_PKT_TOO_SMALL&type=code) | packet size is too small |
   133  | [SKB_DROP_REASON_TCP_CSUM](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_CSUM&type=code) | TCP checksum error |
   134  | [SKB_DROP_REASON_SOCKET_FILTER](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_SOCKET_FILTER&type=code) | dropped by socket filter |
   135  | [SKB_DROP_REASON_UDP_CSUM](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_UDP_CSUM&type=code) | UDP checksum error |
   136  | [SKB_DROP_REASON_NETFILTER_DROP](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_NETFILTER_DROP&type=code) | dropped by netfilter |
   137  | [SKB_DROP_REASON_OTHERHOST](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_OTHERHOST&type=code) | packet don't belong to current host (interface is in promisc mode) |
   138  | [SKB_DROP_REASON_IP_CSUM](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_IP_CSUM&type=code) | IP checksum error |
   139  | [SKB_DROP_REASON_IP_INHDR](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_IP_INHDR&type=code) | there is something wrong with IP header (see IPSTATS_MIB_INHDRERRORS) |
   140  | [SKB_DROP_REASON_IP_RPFILTER](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_IP_RPFILTER&type=code) | IP rpfilter validate failed. see the document for rp_filter in ip-sysctl.rst for more information |
   141  | [SKB_DROP_REASON_UNICAST_IN_L2_MULTICAST](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_UNICAST_IN_L2_MULTICAST&type=code) | destination address of L2 is multicast, but L3 is unicast. |
   142  | [SKB_DROP_REASON_XFRM_POLICY](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_XFRM_POLICY&type=code) | xfrm policy check failed |
   143  | [SKB_DROP_REASON_IP_NOPROTO](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_IP_NOPROTO&type=code) | no support for IP protocol |
   144  | [SKB_DROP_REASON_SOCKET_RCVBUFF](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_SOCKET_RCVBUFF&type=code) | socket receive buff is full |
   145  | [SKB_DROP_REASON_PROTO_MEM](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_PROTO_MEM&type=code) | proto memory limition, such as udp packet drop out of udp_memory_allocated. |
   146  | [SKB_DROP_REASON_TCP_MD5NOTFOUND](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_MD5NOTFOUND&type=code) | no MD5 hash and one expected, corresponding to LINUX_MIB_TCPMD5NOTFOUND |
   147  | [SKB_DROP_REASON_TCP_MD5UNEXPECTED](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_MD5UNEXPECTED&type=code) | MD5 hash and we're not expecting one, corresponding to LINUX_MIB_TCPMD5UNEXPECTED |
   148  | [SKB_DROP_REASON_TCP_MD5FAILURE](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_MD5FAILURE&type=code) | MD5 hash and its wrong, corresponding to LINUX_MIB_TCPMD5FAILURE |
   149  | [SKB_DROP_REASON_SOCKET_BACKLOG](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_SOCKET_BACKLOG&type=code) | failed to add skb to socket backlog ( see LINUX_MIB_TCPBACKLOGDROP) |
   150  | [SKB_DROP_REASON_TCP_FLAGS](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_FLAGS&type=code) | TCP flags invalid |
   151  | [SKB_DROP_REASON_TCP_ZEROWINDOW](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_ZEROWINDOW&type=code) | TCP receive window size is zero, see LINUX_MIB_TCPZEROWINDOWDROP |
   152  | [SKB_DROP_REASON_TCP_OLD_DATA](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_OLD_DATA&type=code) | the TCP data reveived is already received before (spurious retrans may happened), see LINUX_MIB_DELAYEDACKLOST |
   153  | [SKB_DROP_REASON_TCP_OVERWINDOW](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_OVERWINDOW&type=code) | the TCP data is out of window, the seq of the first byte exceed the right edges of receive window |
   154  | [SKB_DROP_REASON_TCP_OFOMERGE](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_OFOMERGE&type=code) | the data of skb is already in the ofo queue, corresponding to LINUX_MIB_TCPOFOMERGE |
   155  | [SKB_DROP_REASON_TCP_RFC7323_PAWS](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_RFC7323_PAWS&type=code) | PAWS check, corresponding to LINUX_MIB_PAWSESTABREJECTED |
   156  | [SKB_DROP_REASON_TCP_INVALID_SEQUENCE](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_INVALID_SEQUENCE&type=code) | Not acceptable SEQ field |
   157  | [SKB_DROP_REASON_TCP_RESET](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_RESET&type=code) | Invalid RST packet |
   158  | [SKB_DROP_REASON_TCP_INVALID_SYN](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_INVALID_SYN&type=code) | Incoming packet has unexpected SYN flag |
   159  | [SKB_DROP_REASON_TCP_CLOSE](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_CLOSE&type=code) | TCP socket in CLOSE state |
   160  | [SKB_DROP_REASON_TCP_FASTOPEN](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_FASTOPEN&type=code) | dropped by FASTOPEN request socket |
   161  | [SKB_DROP_REASON_TCP_OLD_ACK](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_OLD_ACK&type=code) | TCP ACK is old, but in window |
   162  | [SKB_DROP_REASON_TCP_TOO_OLD_ACK](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_TOO_OLD_ACK&type=code) | TCP ACK is too old |
   163  | [SKB_DROP_REASON_TCP_ACK_UNSENT_DATA](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_ACK_UNSENT_DATA&type=code) | TCP ACK for data we haven't sent yet |
   164  | [SKB_DROP_REASON_TCP_OFO_QUEUE_PRUNE](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_OFO_QUEUE_PRUNE&type=code) | pruned from TCP OFO queue |
   165  | [SKB_DROP_REASON_TCP_OFO_DROP](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TCP_OFO_DROP&type=code) | data already in receive queue |
   166  | [SKB_DROP_REASON_IP_OUTNOROUTES](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_IP_OUTNOROUTES&type=code) | route lookup failed |
   167  | [SKB_DROP_REASON_BPF_CGROUP_EGRESS](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_BPF_CGROUP_EGRESS&type=code) | dropped by BPF_PROG_TYPE_CGROUP_SKB eBPF program |
   168  | [SKB_DROP_REASON_IPV6DISABLED](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_IPV6DISABLED&type=code) | IPv6 is disabled on the device |
   169  | [SKB_DROP_REASON_NEIGH_CREATEFAIL](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_NEIGH_CREATEFAIL&type=code) | failed to create neigh entry |
   170  | [SKB_DROP_REASON_NEIGH_FAILED](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_NEIGH_FAILED&type=code) | neigh entry in failed state |
   171  | [SKB_DROP_REASON_NEIGH_QUEUEFULL](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_NEIGH_QUEUEFULL&type=code) | arp_queue for neigh entry is full |
   172  | [SKB_DROP_REASON_NEIGH_DEAD](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_NEIGH_DEAD&type=code) | neigh entry is dead |
   173  | [SKB_DROP_REASON_TC_EGRESS](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TC_EGRESS&type=code) | dropped in TC egress HOOK |
   174  | [SKB_DROP_REASON_QDISC_DROP](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_QDISC_DROP&type=code) | dropped by qdisc when packet outputting ( failed to enqueue to current qdisc) |
   175  | [SKB_DROP_REASON_CPU_BACKLOG](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_CPU_BACKLOG&type=code) | failed to enqueue the skb to the per CPU backlog queue. This can be caused by backlog queue full (see netdev_max_backlog in net.rst) or RPS flow limit |
   176  | [SKB_DROP_REASON_XDP](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_XDP&type=code) | dropped by XDP in input path |
   177  | [SKB_DROP_REASON_TC_INGRESS](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TC_INGRESS&type=code) | dropped in TC ingress HOOK |
   178  | [SKB_DROP_REASON_UNHANDLED_PROTO](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_UNHANDLED_PROTO&type=code) | protocol not implemented or not supported |
   179  | [SKB_DROP_REASON_SKB_CSUM](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_SKB_CSUM&type=code) | sk_buff checksum computation error |
   180  | [SKB_DROP_REASON_SKB_GSO_SEG](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_SKB_GSO_SEG&type=code) | gso segmentation error |
   181  | [SKB_DROP_REASON_SKB_UCOPY_FAULT](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_SKB_UCOPY_FAULT&type=code) | failed to copy data from user space, e.g., via zerocopy_sg_from_iter() or skb_orphan_frags_rx() |
   182  | [SKB_DROP_REASON_DEV_HDR](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_DEV_HDR&type=code) | device driver specific header/metadata is invalid |
   183  | [SKB_DROP_REASON_DEV_READY](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_DEV_READY&type=code) | the device is not ready to xmit/recv due to any of its data structure that is not up/ready/initialized, e.g., the IFF_UP is not set, or driver specific tun->tfiles[txq] is not initialized |
   184  | [SKB_DROP_REASON_FULL_RING](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_FULL_RING&type=code) | ring buffer is full |
   185  | [SKB_DROP_REASON_NOMEM](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_NOMEM&type=code) | error due to OOM |
   186  | [SKB_DROP_REASON_HDR_TRUNC](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_HDR_TRUNC&type=code) | failed to trunc/extract the header from networking data, e.g., failed to pull the protocol header from frags via pskb_may_pull() |
   187  | [SKB_DROP_REASON_TAP_FILTER](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TAP_FILTER&type=code) | dropped by (ebpf) filter directly attached to tun/tap, e.g., via TUNSETFILTEREBPF |
   188  | [SKB_DROP_REASON_TAP_TXFILTER](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_TAP_TXFILTER&type=code) | dropped by tx filter implemented at tun/tap, e.g., check_filter() |
   189  | [SKB_DROP_REASON_ICMP_CSUM](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_ICMP_CSUM&type=code) | ICMP checksum error |
   190  | [SKB_DROP_REASON_INVALID_PROTO](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_INVALID_PROTO&type=code) | the packet doesn't follow RFC 2211, such as a broadcasts ICMP_TIMESTAMP |
   191  | [SKB_DROP_REASON_IP_INADDRERRORS](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_IP_INADDRERRORS&type=code) | host unreachable, corresponding to IPSTATS_MIB_INADDRERRORS |
   192  | [SKB_DROP_REASON_IP_INNOROUTES](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_IP_INNOROUTES&type=code) | network unreachable, corresponding to IPSTATS_MIB_INADDRERRORS |
   193  | [SKB_DROP_REASON_PKT_TOO_BIG](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_PKT_TOO_BIG&type=code) | packet size is too big (maybe exceed the MTU) |
   194  | [SKB_DROP_REASON_DUP_FRAG](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_DUP_FRAG&type=code) | duplicate fragment |
   195  | [SKB_DROP_REASON_FRAG_REASM_TIMEOUT](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_FRAG_REASM_TIMEOUT&type=code) | fragment reassembly timeout |
   196  | [SKB_DROP_REASON_FRAG_TOO_FAR](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_FRAG_TOO_FAR&type=code) | ipv4 fragment too far. (/proc/sys/net/ipv4/ipfrag_max_dist) |
   197  | [SKB_DROP_REASON_MAX](https://github.com/search?q=repo%3Atorvalds%2Flinux%20SKB_DROP_REASON_MAX&type=code) | the maximum of drop reason, which shouldn't be used as a real 'reason' |
   198  
   199  <!-- markdown-link-check-enable -->
   200  
   201  This table can be generated with:
   202  
   203  ```bash
   204  $ go run ./pkg/gadgets/trace/tcpdrop/tracer/dropreasongen/...
   205  ```
   206  
   207  ### Other tools showing dropped packets
   208  
   209  The following tools can be used to show dropped packets but they are not focused on containers or Kubernetes:
   210  
   211  * [dropwatch](https://github.com/nhorman/dropwatch): interactive tool using [Netlink Devlink Trap](https://www.kernel.org/doc/html/latest/networking/devlink/devlink-trap.html) to see drops packets by the NIC.
   212  * [BCC's tcpdrop](https://github.com/iovisor/bcc/blob/master/tools/tcpdrop_example.txt): tool using eBPF and kprobes/tracepoints to show when a socket buffer is released by the kernel.
   213  * [Retis' skb-drop collector](https://github.com/retis-org/retis): tool using various collectors (skb-drop, nftables, ovs) to show the flow of packets in the kernel.