Netplan with bonded NICs and VLANS and bridges not working

Question

I have a netplan issue. Config is below. Bond seems to be working but I cannot connect to the network VLAN128 (br128) which is bridged on bond. The hardware and bonds worked before the server upgrade to 20.04 (was not through netplan before). It also works if I just use one Ethernet NIC i.e. move VLAN128 to the single non bonded NIC.

What was missing on bonded interface was all the routing which seems odd as on a single NIC automagically gets generated. So I have no idea what is going on. Anyone an idea what I am missing?

Single NIC is on-board Realtec 22:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

Bonded NIC is a dual port Intel 21:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02) 21:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)

Netplan:

    version: 2
    renderer: networkd

    ethernets:
        enp34s0:
            dhcp4: false
            dhcp6: false
        eno1:
            dhcp4: false
            dhcp6: false
        enp33s0f1:
            dhcp4: false
            dhcp6: false

    bonds:
      bond0:
        dhcp4: false
        dhcp6: false
        interfaces: [eno1, enp33s0f1]
        parameters:
            mode: 802.3ad
            lacp-rate: fast
            transmit-hash-policy: layer2+3
            mii-monitor-interval: 100

    vlans:
      vlan88:
        accept-ra:        no
        id:               88
        link:             enp34s0
      vlan89:
        accept-ra:        no
        id:               89
        link:             enp34s0
      vlan94:
        accept-ra:        no
        id:               94
        link:             enp34s0
      vlan128:
        accept-ra:        no
        id:               128
        link:             bond0
      vlan129:
        accept-ra:        no
        id:               129
        link:             enp34s0

    bridges:
      br0:
        dhcp4:            yes
        dhcp4-overrides:
          hostname:       defiant89
        interfaces:
          - vlan89
      br88:
        dhcp4:            yes
        dhcp4-overrides:
          hostname:       defiant
        interfaces:
          - vlan88
      br94:
        dhcp4:            yes
        dhcp4-overrides:
          hostname:       defiant94
        interfaces:
          - vlan94
      br128:
        dhcp4:            yes
        dhcp4-overrides:
          hostname:       defiant128
        routes:
          - to: 192.168.128.0/24
            via: 0.0.0.0
          - to: 0.0.0.0/0
            via: 192.168.128.1
            metric: 100
          - to: 192.168.128.1/32
            via: 0.0.0.0
            metric: 100
        interfaces:
          - vlan128
      br129:
        dhcp4:            yes
        dhcp4-overrides:
          hostname:       defiant129
        interfaces:
          - vlan129

Bonded interface:


Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2+3 (2)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 42:31:...
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 9
        Partner Key: 2
        Partner Mac Address: 90:ef:...

Slave Interface: enp33s0f1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 6a:c1:01:8a:1f:a9
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: 42:31:...
    port key: 9
    port priority: 255
    port number: 1
    port state: 63
details partner lacp pdu:
    system priority: 65535
    system mac address: 90:ef:....
    oper key: 2
    port priority: 1
    port number: 3
    port state: 61

Slave Interface: eno1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 6a:c1:....
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: 42:31:...
    port key: 9
    port priority: 255
    port number: 2
    port state: 63
details partner lacp pdu:
    system priority: 65535
    system mac address: 90:ef:...
    oper key: 2
    port priority: 1
    port number: 4
    port state: 61

Routing:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.128.1   0.0.0.0         UG    100    0        0 br128
0.0.0.0         192.168.94.1    0.0.0.0         UG    100    0        0 br94
0.0.0.0         192.168.88.1    0.0.0.0         UG    100    0        0 br88
0.0.0.0         192.168.129.1   0.0.0.0         UG    100    0        0 br129
0.0.0.0         192.168.89.1    0.0.0.0         UG    100    0        0 br0
192.168.88.0    0.0.0.0         255.255.255.0   U     0      0        0 br88
192.168.88.1    0.0.0.0         255.255.255.255 UH    100    0        0 br88
192.168.89.0    0.0.0.0         255.255.255.0   U     0      0        0 br0
192.168.89.1    0.0.0.0         255.255.255.255 UH    100    0        0 br0
192.168.94.0    0.0.0.0         255.255.255.0   U     0      0        0 br94
192.168.94.1    0.0.0.0         255.255.255.255 UH    100    0        0 br94
192.168.128.0   0.0.0.0         255.255.255.0   U     0      0        0 br128
192.168.128.1   0.0.0.0         255.255.255.255 UH    100    0        0 br128
192.168.129.0   0.0.0.0         255.255.255.0   U     0      0        0 br129
192.168.129.1   0.0.0.0         255.255.255.255 UH    100    0        0 br129

Modprobe:

~# lsmod | grep bond
bonding               167936  0

Hello olivernz, thank you for this very good explanation of the problem with all the details! What I do not fully understand, yet: Which routes are you missing exactly? You're defining 3 static routes for `br128` which all seem to be there in your routing table. Cloud you please show the difference between "routing missing on bond0" and "routing automagically generated on a single NIC"? — Slyon, Oct 20 '20 at 08:30
As you can see the routing for br128 is in the YAML file. The others are not in Netplan but are in routing table. If I take out the routing stuff in br128 there is nothing in the routing table for br128. Hope that explains it. But the problem is although the br128 looks all good it isn't working at all. The bond seems ok but nothing else is there. — olivernz, Oct 21 '20 at 08:30
I see... so the routes for br128 are actually missing, there you're adding them manually. Whereas they should be provided by DHCP. Is DHCP working at all on br128, i.e. does it get an IP address? Could you show the output of `ip a`? — Slyon, Oct 21 '20 at 09:46
Ah so that's why THAT doesn't work! It doesn't get an IP. Thee problem is deeper in the stack then. before we get to routing. It means that it can't reach the gateway and dhcp. So what tests can I run? I am thinking bond looks ok. How do I test VLAN and Bridge? — olivernz, Oct 21 '20 at 11:30
Correct. You could try to configure DHCP directly on `bond0` via `dhcp4: true` and see if that gets an IP for itself (maybe disabling/commenting the br128 and vlan128 for the test) — Slyon, Oct 21 '20 at 12:54
Sorry can't do that as that interface cannot work without VLAN tags. I can try and just get an IP over VLAN without the bridge. — olivernz, Oct 21 '20 at 20:06
Sh*t. Thanks you helped a LOT but not in the way i would have thought. Basically was a tagging and trunking VLAN setting on switch. :-( — olivernz, Oct 21 '20 at 21:55
So basically I got the VLANS accross the trunk and all started working (duh). So remove the routing info ablve from the YAML and you'll have a correct example for Bond-->VLAN-->Bridge — olivernz, Oct 21 '20 at 21:57
Having another odd issue now....I'd expect all bridges to be the same but I see differences. Example: 9: **br88**: ... link/ether **1e:ae:ef:61:35:84** brd ... inet 192.168.88.214/24 brd 192.168.88.255 scope global dynamic br88 valid_lft 5557sec preferred_lft 5557sec 10: **br94**: ... link/ether **42:31:56:3e:9d:9b** brd ... inet 192.168.94.7/24 brd 192.168.94.255 scope global dynamic br94 valid_lft 5548sec preferred_lft 5548sec Id expect br88 have the same MAC as br95 (real MAC) but br88 constantly changes MAC on reboot. — olivernz, Oct 22 '20 at 02:53

score 0 · Answer 1 · answered Oct 22 '20 at 09:51

to solve your 2nd problem, you should be able to set the mac address explicitly via the macaddress setting, e.g. by setting it to the same MAC as your physical interface:

[...]
      br88:
        macaddress:       DE:AD:BE:EF:CA:FE
        dhcp4:            yes
        dhcp4-overrides:
          hostname:       defiant
        interfaces:
          - vlan88
      br94:
        macaddress:       DE:AD:BE:EF:CA:FE
        dhcp4:            yes
        dhcp4-overrides:
          hostname:       defiant94
        interfaces:
          - vlan94
[...]

Netplan with bonded NICs and VLANS and bridges not working

1 Answers1