0

I purchased a Netgear GS110EMX switch and am trying to configure all 4 ports of an Intel gigabit quad port interface card for link aggregation. I downloaded and installed the Intel Network Adapter Driver for Windows. I configured a LAG on the GS110EMX by selecting the corresponding 4 ports with a LAG type of LACP. I then created an adapter team with the Intel driver on my computer using the team type IEEE 802.3ad dynamic link aggregation. The LAG status on the GS110EMX changed to UP so I think I did it right.

The problem is I don't seem to get any improvement in my speed. Using LAN Speed Test before setting up the LAG I got about 733 Mbps writing and 854 Mbps reading on all 4 ports. After I get 758 Mbps writing and 637 Mbps reading on the single LAG port. So the writing speed has increased slightly but the reading speed has decreased significantly. I'm sure this isn't supposed to be how it works. So what am I doing wrong?

Additional information that might help is I am testing the speed to a NAS which is connected to the switch with a 10gbe link. So the other side of the connection is not causing a bottleneck. The actual speeds vary with the packet size I select in LAN Speed Test, but the result is always the same - slightly faster write speeds and much slower read speeds.

user5820228
  • 3
  • 1
  • Separately from LAG, I would actually say those are surprisingly low numbers for a gigabit link. I can easily get 850–900 Mbps between two old Xeon servers running iperf3, _without any tuning whatsoever,_ and I typically see 900+ Mbps for file copies over SMBv3 as well (again just basic install)... – u1686_grawity Oct 04 '21 at 06:40

1 Answers1

0

L2 link aggregation (LACP) doesn't give you a "real" 4 Gbps link; although it indeed acts as a single interface at MAC layer (top half of L2), below that it's still four independent 1 Gbps links that the endpoints just have freedom to automatically select from.

The difference is that for certain reasons (e.g. to make sure the packets belonging to a given stream do not get reordered), LACP aggregation always tries to map a stream to a single physical link only. To fully make use of a 4x aggregated port, you would need 4 separate packet streams.

The "hashing policy" of each LACP endpoint configures how it chooses which physical link to send the packets through. For example, a "L3" policy chooses purely based on IP addresses, therefore all packets from host X to Y are considered a single stream. So if you test against only a single host, and the policy is set to L3 on the sending side, then you'll always be using only a single link as "same (src_ip, dst_ip) ⇒ same port".

Meanwhile, the (somewhat non-standard!) "L3+L4" policy also includes the TCP ports in the consideration, so if you do a multi-connection test (e.g. iperf3 with --parallel 4) then the total may in fact be 4 Gbps. (But if there is no L3+L4 option in the sending side, then testing via IPv6 may work regardless, as it has "flow labels" as part of the L3 header without the need to do deeper L4 inspection.)

(Note that SMBv3 in particular has a "Multichannel" feature that does link aggregation by itself if it recognizes that the server has multiple NICs attached to the same network. It also recognizes the Windows built-in "NIC Teaming" LACP feature and creates multiple connections over the LAG in that case, however, it might not recognize the LACP feature that is provided by NIC drivers.)

u1686_grawity
  • 426,297
  • 64
  • 894
  • 966
  • To some extent, I'm just repeating what I learned recently in Networking.SE, so try that site for deeper network architecture questions. – u1686_grawity Oct 04 '21 at 06:27
  • Comment 1: Thanks. I thought that could be the case because it works that way for other aggregate configurations, but I thought 802.3ad might be different. So does that mean there is no method that will split a single stream over the four ports? Also, any idea why the write speed would be faster and the read speed slower? The slower speed is actually more understandable since everything is still going through a single link and the LAG would just add overhead to slow that down. But I don't understand the write speed increase, which is small but definitely real. – user5820228 Oct 04 '21 at 06:45
  • Comment 2: I forgot to mention this in my original post but I kept the settings on the GS110EMX at their default values, but I don't know if that's optimal. The options I have are "LACP System Priority" which is set to 32768 from a range of 1 to 65535, "LACP Priority" which is 128 for all ports, and "Timeout" which is Long. – user5820228 Oct 04 '21 at 06:45
  • Comment 3: I guess the final question is, is this even worth doing? I suspect a lot of my traffic would be single stream. The only time I can think I might have parallel transfers would be downloading something from the internet and transferring a file. But the internet download wouldn't be limited by the 1 gbps connection anyway. So the rare times I would get an improvement in throughput might not make it worth the more common case when my single stream transfer gets slowed down by the overhead of the LAG. – user5820228 Oct 04 '21 at 06:54
  • When you say "read/write" that makes me think a disk is involved. Are those the terms your speed test program uses? Normally I'd expect it to say "send/receive" or "transmit/receive" if it's a pure network-only test. Regarding #1 the difference in both directions, I don't know a definite answer, but it could easily be at least partially caused by the OS and/or the CPU of the sending machine (sending data takes a little bit more CPU than receiving). One NAS that I have was manufactured with an Atom CPU that is *just* barely enough for transferring files from disk at full gigabit, but not more. – u1686_grawity Oct 04 '21 at 07:05
  • It could be a difference in Ethernet adapters, in their drivers (some are more efficient than others), in the "hardware offload" settings on the sending side, OS in general... However, this being said, LAG only adds *processing* overhead (and I expect it would be incredibly small) – it does not actually add any *transmission* overhead, i.e. it is still exactly the same amount of data that gets transferred, so if there's any impact it should still be nowhere near what you're seeing... – u1686_grawity Oct 04 '21 at 07:09
  • Re #2: Those should have no effect on performance, they only control how the LAG is established. I'd maybe set timeout to fast but all it does is control how fast the LAG will rearrange itself if one of the links suddenly gets cut off, e.g. cable unplugged. (LACP is thus also useful as a link failover/reliability mechanism, not only for increasing speed.) And re #3, not sure, but I'm indeed tempted to say that if you want speed you shouldn't bother with LAGs and should get 10Gbps Ethernet instead. – u1686_grawity Oct 04 '21 at 07:13
  • I am using LAN Speed Test (https://totusoft.com/lanspeed). It does two separate transfers and reports the results under the headings "Writing (Upload)" and "Reading (Download)" – user5820228 Oct 04 '21 at 07:41
  • Ah, so it specifically tests access to a "shared folder" over SMB (which goes to the server's disk and back), not raw network throughput. Then it highly depends on the NAS's storage, on the amount of RAM it has for cache, on both systems' SMB implementations... far more than e.g. iperf3 or another network speed test. – u1686_grawity Oct 04 '21 at 10:29