1

I have a 2960g cisco switch and 2 servers with 4 NICs each, configured in lacp mode. I have enabled jumbo frames (9000 bytes) on the network. I have tested a file transfer, and it runs at full speed (100Mb/s compared to about 80Mb/s without jumbo frames. I know tcp transfers from one IP to another will only use 1 link out of the 4 available with lacp.)

I did a capture on one of the nics and was pretty surprised to see frames of 17966, 26914, and even 44810 bytes... Most frames are only 9018 bytes, as they should. But how/why/what are these mega-jumbo frames of 40kb?? I didn't think it was even possible..

Is it a side effect of lacp? Note that i see these frames whether i capture them from the bonded interface, or the slave interface that's actually transmitting the data. Or is it an artefact that somehow got in through tcpdump? Or do i need to change my glasses?

Thanks,

Vincent

Hennes
  • 64,768
  • 7
  • 111
  • 168
user338930
  • 13
  • 2

1 Answers1

2

It was probably TCP Transmit Segmentation Offload (TSO), possibly in conjunction with Large Receive Offload (LRO).

You captured on one of the machines doing the test, rather than on an independent observer machine. So you didn't really see what was on the wire, you saw what went between your host's network stack and its Ethernet NIC. And when a NIC is providing services like TSO and LRO to its host, the packets between the host and its NIC are much bigger than what the NIC is actually sending/receiving on the wire.

If it's too inconvenient to set up a separate sniffer machine and port mirroring, you can probably disable TSO and LRO so that you see something more like what would actually be on the wire.

For example, if your servers are running OS X, you can use these sysctls to disable TSO and LRO:

sudo sysctl -w net.inet.tcp.tso=0 net.inet.tcp.lro=0

Of course, you'll probably get higher CPU utilization and lower throughput if you do this, but at least your packet capture will seem more sane.

Spiff
  • 101,729
  • 17
  • 175
  • 229
  • Thanks, TSO was causing this behavior. With TSO off, all the frames have correct lengts. It's actually documented on the wireshark wiki: http://wiki.wireshark.org/CaptureSetup/Offloading On centos, you can get an interface's settings with: ethtool -K eth3 and disable TSO with: ethtool -K eth3 tso off (Although there's no reason to turn tso off on a prod system) – user338930 Jun 27 '14 at 00:44