How to capture "dropped packets" in tcpdump

Question

I have a problem with my networking performance. I am using Ubuntu 16.04 on VMware Cloud Server with NIC E1000. But I see some packets dropped in sections of ifconfig command:

root@ubuntu:~# ifconfig ens192
ens192    Link encap:Ethernet  HWaddr 00:50:56:03:25:14  
          inet addr:192.16.1.100  Bcast:192.16.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:574749 errors:0 dropped:83 overruns:0 frame:0
          TX packets:76478 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:44109471 (44.1 MB)  TX bytes:19484534 (19.4 MB)

Although it just some packets dropped but my server is running a real-time game online, so you know it impacts my clients that are connecting to it.

I have done in some researching and exploring information on Google, after that I tried to change config file for buffer ring, max windows size, and so on. But it still drops my packets.

So, now I want to capture packets that dropped for analyzing what type of packets exactly it is.

I also tried this capture for my view in wireshark:

sudo tcpdump -i ens192 -n -w /var/www/html/logs.pcap -C 1 -Z root

But I don't think I can see what packets are dropped! I think packets dropped is ignored before going to the filter of tcpdump.

Can you suggest me what method to capture "dropped packets" above (dropped:83)?

Thanks in advance!

you probably need to hit up `iptables` for that information. see https://serverfault.com/a/126078 and https://serverfault.com/q/385937 — quixotic, May 23 '17 at 17:51
Read [this answer](https://unix.stackexchange.com/a/205290/108618). I can't verify it's true but if it is, then you cannot capture those packets at all because they never get to the kernel. My understanding is: if you could capture them, they wouldn't be considered "dropped" it the first place. — Kamil Maciorowski, May 23 '17 at 19:12
The problem can be with tcpdump itself: If it doesn't respond quickly enough then old packets will be overwritten with new ones. You should maybe get tcpdump to go faster. Why are you limiting the log file size? You may find much more advice [here](https://stackoverflow.com/a/41903056/165358). — harrymc, May 23 '17 at 20:14
Consider taking a look at `ethtool -S ens192` based on [this answer](https://serverfault.com/a/561132/171566). There can be any number of reasons your interface is dropping packets. May want to also check your outbound switchport to see if it is also detecting dropped/error packets in or out. Malformed and packets that fail checks can be dropped. — Will, May 29 '17 at 22:21
Do you have access to VMware infrastructure? If yes, you can install one more vm, just for packet capturing, and make port mirroring into it. After, you can capture packet in 2 vm in same time. And after, you will need to compare it. If you don't have access to VMware, probably,you can ask your hoster to capture traffic for you. Technically, they can do it. But it can be non-technical problems — Mihail Kuznesov, May 30 '17 at 05:43
@Michael Kuznetsov: I have tested by creating another VM but not thing changed at all. It stills drop packets. — Joey, May 30 '17 at 11:58
No, you can mirror port in Vmware switch, and start capturing packet in second vm. — Mihail Kuznesov, May 30 '17 at 12:02
https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.hostclient.doc%2FGUID-8D1768B0-074D-4F06-9931-2BE4777D35F8.html — Mihail Kuznesov, May 30 '17 at 12:03
And start tcpdump in your two vm. After, you can compare dumps for diff — Mihail Kuznesov, May 30 '17 at 12:04

score 4 · Answer 1 · answered May 28 '17 at 07:47

The problem can be with tcpdump itself: If it doesn't respond quickly enough then old packets will be overwritten with new ones, which means that they are dropped.

If you capture all the bytes of each packet, it's very easy to overrun the kernel's packet capture buffer. The symptoms of this overrun are that your packet trace program will report that it dropped packets.

In the case of tcpdump, it prints a summary of how many packets were captured, filtered, and dropped when you stop the capture. For example:

$ sudo tcpdump -i en0 -w trace.pcap
tcpdump: listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C
94 packets captured
177 packets received by filter
0 packets dropped by kernel

If the dropped count is non-zero, you need to increase the packet capture buffer size by passing the -B option to tcpdump. Try it also without a capture file, to see if this improves the capture ratio.

Thanks for your comment, but TCPDUMP is just the way we troubleshoot for "What's exactly with "RX: dropped:83"? I want to know why my ubuntu dropped that packets, so I have to know what packets are they. :( — Joey, May 29 '17 at 00:51
Have you tried increasing `-B`? What is your output for tcpdump? — harrymc, May 29 '17 at 06:07

score 2 · Answer 2 · answered May 29 '17 at 22:50

Your questions seems to pertain to going back and finding out what might have caused a previous packet to drop rather than attempting to capture real time packets to try to find one that drops. For the latter, harrymc's answer should be able to eventually capture one that you're looking for.

To go back and look into what might be happening at that interface, you have to understand that those headers you're seeing for RX packets:574749 errors:0 dropped:83 overruns:0 frame:0 are simply summaries of underlying counters.

From my comment, I would recommend using the ethtool command/package from this answer to analyze some of the counters to see if you can find something that appears out of place.
ethtool -S ens192

From the broadcom drivers source file, we see tp->rx_dropped++; for a few separate occasions in the file. 1 2 3 Any one of these or more (depending on your exact NIC and underlying drivers) contribute to what could be causing dropped packets.

To ease your mind, your server is dropping less than 0.015% of packets received based on your output above. Your clients wouldn't notice any server interruption or even jitter until your drop rate/error rate climbs above 1%. Even then, it would be hardly noticeable. TCP will take care of any of the re-transmissions required.

For a real-time online game, 1% drops is sufficient to cause a noticeable lagging of game feel. — Joey.Z, Nov 01 '18 at 06:47
Sorry but seems the broadcom driver source had changed. There's only a single line with `rx_dropped++` but it is now labelled as `drop_it_no_recycle:` and there are three `goto`s to it by the following conditions: 1) failure of `build_skb(data, frag_size)`; 2) failure of `netdev_alloc_skb(tp->dev, len + TG3_RAW_IP_ALIGN)`; 3) length is bigger than MTU. I think we can't just ignore them in some cases. As well as even a single packet lost can spoil the picture of video codec like H.264 or VP8 when you use RTP protocol. — kay27, Apr 30 '20 at 07:12

How to capture "dropped packets" in tcpdump

2 Answers2