No connection to client network via OpenVPN site-to-site connection

Question

I am trying to connect two of my homes (and maybe more than that later) via an OpenVPN site-to-site connection. My goal is so that all clients on all sites are able to talk to eachother.

But for some reason I cannot manage to get a connection from Site A to the clients in Site B. Vice versa it works perfectly fine.

Do I have an error in my OpenVPN config?
Did I forgot to push any routes?

Any help is greatly appreciated.

The situation

Here is how the setup looks like:

Network Map

Site A

ISP Modem/Router
- OpenVPN port 1194 is forwarded to Unifi USG
Unifi USG as Router / DHCP
- Network: 192.168.200.1/24
- connected via WAN port to ISP Modem/Router
- OpenVPN port 1194 is forwarded to Synology NAS
- Static route 192.168.100.1/24 to Synology NAS
Synology NAS with fixed IP 192.168.200.200
- OpenVPN server using the Synology VPN Server Package
- OpenVPN server network: 10.0.10.0/24

Site B

ISP Modem/Router
ASUS RT-AC68U router with merlin firmware
- Network: 192.168.100.1/24
- connected via WAN port to ISP Modem/Router
OpenVPN client connects to Site A/OpenVPN server
- Client IP 10.0.10.6 in this example. I tried fixed cleint IPs as well, but that did not change the outcome

Additional devices

more clients can connect via OpenVPN and will be assigned IPs in the range of 10.0.10.x

Problem

✅ OpenVPN connection is established successfully
✅ Site B Router (192.168.100.1) can ping Site A OpenVPN server (192.168.200.200)
✅ Site B Router (192.168.100.1) can ping Site A router (192.168.200.1)
✅ Site B Router (192.168.100.1) can ping Site A clients (eg 192.168.200.6)
✅ Site B Client (eg 192.168.100.129) can ping Site A OpenVPN server (192.168.200.200)
✅ Site B Client (eg 192.168.100.129) can ping Site A router (192.168.200.1)
✅ Site B Client (eg 192.168.100.129) can ping Site A clients (eg 192.168.200.6)
✅ Site A Router (192.168.200.1) can ping Site B router (192.168.100.1)
❌ Site A Router (192.168.200.1) cannnot ping Site B clients (eg 192.168.100.10)
✅ Site A Client (eg 192.168.200.6) can ping Site B router (192.168.100.1)
❌ Site A Client (eg 192.168.200.6) cannnot ping Site B clients (eg 192.168.100.129)

I already tried to run tcpdump -eni any icmp on site B router.

When I ping site B router (192.168.100.1) from site A client (192.168.200.6), it works:

21:33:04.416012  In ethertype IPv4 (0x0800), length 100: 192.168.200.6 > 192.168.100.1: ICMP echo request, id 1038, seq 1, length 64
21:33:04.416165 Out ethertype IPv4 (0x0800), length 100: 192.168.100.1 > 192.168.200.6: ICMP echo reply, id 1038, seq 1, length 64
21:33:05.418067  In ethertype IPv4 (0x0800), length 100: 192.168.200.6 > 192.168.100.1: ICMP echo request, id 1038, seq 2, length 64
21:33:05.418147 Out ethertype IPv4 (0x0800), length 100: 192.168.100.1 > 192.168.200.6: ICMP echo reply, id 1038, seq 2, length 64
21:33:06.419075  In ethertype IPv4 (0x0800), length 100: 192.168.200.6 > 192.168.100.1: ICMP echo request, id 1038, seq 3, length 64
21:33:06.419153 Out ethertype IPv4 (0x0800), length 100: 192.168.100.1 > 192.168.200.6: ICMP echo reply, id 1038, seq 3, length 64

However when I ping site B client (192.168.100.129) from site A client (192.168.200.6), it fails:

21:37:08.038559  In ethertype IPv4 (0x0800), length 100: 192.168.200.6 > 192.168.100.129: ICMP echo request, id 1039, seq 1, length 64
21:37:09.070567  In ethertype IPv4 (0x0800), length 100: 192.168.200.6 > 192.168.100.129: ICMP echo request, id 1039, seq 2, length 64
21:37:10.110696  In ethertype IPv4 (0x0800), length 100: 192.168.200.6 > 192.168.100.129: ICMP echo request, id 1039, seq 3, length 64
21:37:11.150865  In ethertype IPv4 (0x0800), length 100: 192.168.200.6 > 192.168.100.129: ICMP echo request, id 1039, seq 4, length 64

Vice versa, when I ping site A client (192.168.200.6) from site B client (192.168.100.129), it works:

21:39:11.565543  In bc:5f:f4:63:a9:46 ethertype IPv4 (0x0800), length 76: 192.168.100.129 > 192.168.200.6: ICMP echo request, id 1, seq 230, length 40
21:39:11.565543  In bc:5f:f4:63:a9:46 ethertype IPv4 (0x0800), length 76: 192.168.100.129 > 192.168.200.6: ICMP echo request, id 1, seq 230, length 40
21:39:11.565665 Out ethertype IPv4 (0x0800), length 76: 10.0.10.6 > 192.168.200.6: ICMP echo request, id 1, seq 230, length 40
21:39:11.573657  In ethertype IPv4 (0x0800), length 76: 192.168.200.6 > 10.0.10.6: ICMP echo reply, id 1, seq 230, length 40
21:39:11.573715 Out 60:45:cb:59:fd:d0 ethertype IPv4 (0x0800), length 76: 192.168.200.6 > 192.168.100.129: ICMP echo reply, id 1, seq 230, length 40
21:39:11.573736 Out 60:45:cb:59:fd:d0 ethertype IPv4 (0x0800), length 76: 192.168.200.6 > 192.168.100.129: ICMP echo reply, id 1, seq 230, length 40
21:39:12.584996  In bc:5f:f4:63:a9:46 ethertype IPv4 (0x0800), length 76: 192.168.100.129 > 192.168.200.6: ICMP echo request, id 1, seq 231, length 40
21:39:12.584996  In bc:5f:f4:63:a9:46 ethertype IPv4 (0x0800), length 76: 192.168.100.129 > 192.168.200.6: ICMP echo request, id 1, seq 231, length 40
21:39:12.585099 Out ethertype IPv4 (0x0800), length 76: 10.0.10.6 > 192.168.200.6: ICMP echo request, id 1, seq 231, length 40
21:39:12.592692  In ethertype IPv4 (0x0800), length 76: 192.168.200.6 > 10.0.10.6: ICMP echo reply, id 1, seq 231, length 40
21:39:12.592751 Out 60:45:cb:59:fd:d0 ethertype IPv4 (0x0800), length 76: 192.168.200.6 > 192.168.100.129: ICMP echo reply, id 1, seq 231, length 40
21:39:12.592777 Out 60:45:cb:59:fd:d0 ethertype IPv4 (0x0800), length 76: 192.168.200.6 > 192.168.100.129: ICMP echo reply, id 1, seq 231, length 40

OpenVPN Server Config

(Synology NAS /usr/syno/etc/packages/VPNCenter/openvpn/openvpn.conf)

dev tun

# make all clients aware of local site
push "route 192.168.200.0 255.255.255.0"
push "route 10.0.10.0 255.255.255.0"

# forward routes to other sites
route 192.168.100.0 255.255.255.0
push "route 192.168.100.0 255.255.255.0"

management /var/run/openvpn.sock unix

server 10.0.10.0 255.255.255.0

client-config-dir /usr/syno/etc/packages/VPNCenter/userIPs/
client-to-client

dh /var/packages/VPNCenter/target/etc/openvpn/keys/dh3072.pem
ca /var/packages/VPNCenter/target/etc/openvpn/keys/ca.crt
cert /var/packages/VPNCenter/target/etc/openvpn/keys/server.crt
key /var/packages/VPNCenter/target/etc/openvpn/keys/server.key

max-clients 5

comp-lzo

persist-tun
persist-key

verb 3

#log-append /var/log/openvpn.log

keepalive 10 60
reneg-sec 0

plugin /var/packages/VPNCenter/target/lib/radiusplugin.so /var/packages/VPNCenter/target/etc/openvpn/radiusplugin.cnf
verify-client-cert none
username-as-common-name
duplicate-cn

status /tmp/ovpn_status_2_result 30
status-version 2
proto udp6
mssfix 1450
port 1194
cipher AES-256-CBC
auth SHA512

I have a client config dir with a file for my vpn user:

(Synology NAS /usr/syno/etc/packages/VPNCenter/userIPs/<vpn user name for site b>)

# set static ip address for this client
#ifconfig-push 10.0.10.100 255.255.255.0

# make server aware that the 192.168.100.0/24 address range is handled by this client
iroute 192.168.100.0 255.255.255.0

OpenVPN Client File

dev tun
tls-client

remote <site A dynamic host name> 1194

float

#redirect-gateway def1

dhcp-option DNS 192.168.200.1

pull

proto udp

script-security 2


comp-lzo

reneg-sec 0

cipher AES-256-CBC

auth SHA512

auth-user-pass

setenv CLIENT_CERT 0

<ca>
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
</ca>

Update: This is how the USG (Router on Site A) is configured to route traffic to the Synology

Update: Routing Table on the Synology. I did not add any routes here, They were automatically added by the OpenVPN server.

You might want to add your routing tables. I've not gone through this in any depth, but I wonder how the synology is seeing the local devices through the USG. I suspect the problem is here - ie if the USG is acting like a switch between the devices and synology the devices won't know to route VPN traffic to it, and if the SG is routing their is likely a hairpin nat issue lurking there. — davidgo, Jul 29 '22 at 06:46
If there was an award for "Best question asked by a new user", I'd give it to you. — mtak, Jul 29 '22 at 13:08
hey @mtak. nice to hear that :-). I tried my best to give you as much information as possible. Still, any help is appreciated. Maybe I am reading the `tcpdump` results wrong. Can somebody explain me what they can read from the data I posted? — whitespace, Jul 31 '22 at 08:52