Linux uses a wrong network interface than the intended one

Question

I use an Odroid device with Debian 11 and kernel 6.1 as my NAS. I tried older kernel versions (5.10 and such) with no difference.

I recently connected a second network interface, so now I have got:

eth0 (the built-in 1Gb/s Ethernet port), IP: 192.168.1.155.
eth1 (USB 3.0 1Gb/s Ethernet card), IP: 192.168.1.156.

In netplan, devices are configured to use a single IP address, eth0:

addresses:
  - 192.168.1.155/32

eth1:

addresses:
  - 192.168.1.156/32

ifconfig:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 192.168.1.155  netmask 255.255.255.255  broadcast 0.0.0.0
    inet6 X  prefixlen 64  scopeid 0x20<link>
    ether X  txqueuelen 1000  (Ethernet)
    RX packets 23767007  bytes 1991691527 (1.8 GiB)
    RX errors 0  dropped 4484  overruns 0  frame 0
    TX packets 39239638  bytes 58168353216 (54.1 GiB)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    device interrupt 24
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 192.168.1.156  netmask 255.255.255.255  broadcast 0.0.0.0
    inet6 X  prefixlen 64  scopeid 0x20<link>
    ether X  txqueuelen 1000  (Ethernet)
    RX packets 2532258  bytes 202799128 (193.4 MiB)
    RX errors 1  dropped 2821  overruns 0  frame 0
    TX packets 1164474  bytes 16121299674 (15.0 GiB)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ip route:

default via 192.168.1.1 dev eth0 proto dhcp src 192.168.1.155 metric 100
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.156
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.155
192.168.1.1 dev eth0 proto dhcp scope link src 192.168.1.155 metric 100

It seems like my USB3.0 eth1 interface is slower (around 830Mb/s max outgoing speed) vs eth0 (around 980Mb/s).

I'm always accessing my NAS by 192.168.1.155 (so it should go to eth0 on NAS), but I noticed reduced speeds. After checking ipconfig, I noticed that TX packets increase on the wrong interface.

This seems to be very random. Generally, I can fix all the reduced speed issues by shutting down eth1: sudo ip link set eth1 down.

Sometimes, when I am accessing NAS by 192.168.1.156 it also links to the wrong interface. Instead of getting reduced speed, I get the full one of eth0, instead of eth1.

I am using Windows SMB, and Filezilla's SFTP to test these speeds.

So my question is, what I'm doing wrong? Why linux just selects a single interface (randomly), and uses it for my requests on both IPs: *.155 and *.156?

EDIT:

I think I solved the problem. I found some useful information here: https://unix.stackexchange.com/questions/4420/reply-on-same-interface-as-incoming

Step 1:

sysctl -w net.ipv4.conf.default.arp_filter=1
sysctl -w net.ipv4.conf.all.arp_filter=1

sysctl -w net.ipv4.conf.default.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_ignore=1

Step 2:

reboot

Step 3:

I created a .sh script, based on the original/default output from ip route:

#!/bin/bash
# crontab -e
# @reboot sleep 120; /home/mona/ip_route_rules_for_separate_eth0_eth1.sh

if grep -Fxq '252     ispa' "/etc/iproute2/rt_tables"; then
    echo ""
else
    echo '252     ispa' >> "/etc/iproute2/rt_tables"
fi
if grep -Fxq '251     ispb' "/etc/iproute2/rt_tables"; then
    echo ""
else
    echo '251     ispb' >> "/etc/iproute2/rt_tables"
fi

#Eth1:
ip rule add from 192.168.1.156 table ispa prio 1
ip route add 192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.156 table ispa
ip route add 192.168.1.1 dev eth1 proto dhcp scope link src 192.168.1.156 metric 100 table ispa
ip route add default via 192.168.1.1 dev eth1 proto dhcp src 192.168.1.156 metric 100 table ispa

#Eth0:
ip rule add from 192.168.1.155 table ispb prio 1
ip route add 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.155 table ispb
ip route add 192.168.1.1 dev eth0 proto dhcp scope link src 192.168.1.155 metric 100 table ispb
ip route add default via 192.168.1.1 dev eth0 proto dhcp src 192.168.1.155 metric 100 table ispb

I'm using the croon job on reboot, but this is probably wrong, since interface restarts wouldn't apply rules.

Does anyone know how can I rewrite these commands to be executed by netplan?

For what reason do you use 2 network cards connected to the same network segment? Could you check the ARP table if the MAC addresses match? — mashuptwice, Dec 23 '22 at 23:01
Agree with @mushuptwice - Would it not make sense to use Linux Load Balancing here? Also, Linux would seem to be acting appropriately - but what does your routing table show? "sudo ip route" - My guess is you can play with metrics or netmasks to prefer one interface over the other for outgoing traffic - or specifiy a preferred device for outbound traffic. — davidgo, Dec 23 '22 at 23:10
I'm doing it, because I've ordered a 2.5Gb/s switch, and it will arrive in a few days. Then, I'll connect a 2.5Gb/s USB Ethernet to the NAS, to have better speeds. It still feels nice to be able to access the 1Gb/s port, e.g. from a different device, so in total I'd be able to use the speed of 3.5Gb/s (at max). I updated my question with `ip route`. — Mona, Dec 23 '22 at 23:23
The `arp` command only shows 1 entry for `eth1` now... I'm not really good at these networking things. — Mona, Dec 23 '22 at 23:29
@mashuptwice and davidgo. I edited my question and added a solution that worked for me. The only thing left is to make it work with netplan :) — Mona, Dec 24 '22 at 04:24
@Mona please do not add a solution into your question. Instead you can simply answer your own question! Nice that you found something that worked out for you. — mashuptwice, Dec 25 '22 at 01:58
@mashuptwice Yes, you're right, but I feel like it's not complete yet. Maybe once I test it works as expected, I'll move it to the answer? Or it should be moved to an answer already? — Mona, Dec 26 '22 at 06:53

score 1 · Answer 1 · answered Dec 24 '22 at 04:37

1

Instead of arp_filter = 1, which requires you to set up source based routing, consider arp_ignore = 1:

1 - reply only if the target IP address is local address configured on the incoming interface

Ref.: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

answered Dec 24 '22 at 04:37

Tom Yan

9,075
2
17
36

Thanks. It seems to work too :) I set `arp_filter` back to `0` and then added `arp_filter=1`. Do you maybe know how to rewrite these `ip rule`, `ip route` commands to work with netplan instead? – Mona Dec 24 '22 at 05:20
No, but the point of this answer is that it works without those extra configurations. – Tom Yan Dec 24 '22 at 06:19
Do you mean that I wouldn't need to set up `ip rule`, `ip route` once I set `arp_ignore = 1`? Or by "extra configurations" you mean the `arp_filter`? – Mona Dec 24 '22 at 21:49
1

Yes for the former. `source based routing` above refers to the setup you've done with `ip rule` / `route`. Also see (again) the quoted description above for `1` in `arp_ignore` to understand how it works. – Tom Yan Dec 25 '22 at 03:39

Linux uses a wrong network interface than the intended one

1 Answers1