1

I've got a virtual network setup inside proxmox. I've installed OPNsense as the router to connect traffic from my regular network to the virtual network. The LAN side of OPNsense is on a bridge network in proxmox assigned to the 10.0.42.0/24 network with a statically assigned IP. The WAN side is connected to the virtual bridge bound to the physical interface on the proxmox server. The WAN interface is assigned an IP via DHCP from my internet router.

The virtual network is able to receive IPs from OPNsense with no problem. Those same systems are able to access the internet through OPNsense without problems. I can access the OPNsense web interface from my desktop with no problems.

Inside the virtual network I've got portainer up and running. I'm not able to access the management page for this portainer system from my desktop. I've confirmed that the portainer address was accessible from inside the network.

I've tried adding port forwarding rules on OPNsense. I've used the following pages to try and get a port forwarding rule in place. Most of the pages have instructions very similar to each other

When I attempt to access the portainer page from the desktop, I see the following in the logs of the OPNsense firewall:

Source: 10.0.42.11:9443
Destination: [desktop IP]:51918
Proto: tcp
Label: Default deny / state violation rule

Firewall rule I've tried: Interface: WAN TCP/IP Version: IPv4 Protocol: TCP Source: any Source port range: any Destination: WAN Address Destination port range: any Redirect target IP: portainer IP Redirect target port: 9443 NAT Reflection: tried both enable and disable Filter rule association: Tried both Pass and Add associated filter rule

I've also tried using the reverse of the above with no change to the errors in the firewall logs.

Also tried adding WAN rules. I've used portainer as the source IP/port and WAN Address as the destination with all ports. I've also tried adding in a reverse rule as well.

If I disable the firewall completely, I'm able to access the portainer web interface, but the virtual machines are no longer able to access the internet. This is due to NAT being disabled along with the firewall.

EDIT

The configuration for the networks/devices are as follows:

Modem->(public IP) Netgear router (192.168.x.x)->TP-link Switch->Desktop
Modem->(public IP) Netgear router (192.168.x.x)->TP-link Switch->Proxmox->(192.168.x.x) OPNsense (10.0.42.x)->Portainer

I've created a second bridged interface in proxmox that is not bound to the physical interface. That bridge is the gateway for packets headed to proxmox to be forwarded to OPNsense for accessing the virtual network. Achieved by adding a new ip table and rule on proxmox. I've added the bridge bound to the physical interface as the WAN interface in OPNsense. The unbound bridge is assigned to the LAN interface in OPNsense. On the windows desktop, I've added a route to point 10.0.42.x traffic to the IP of the virtual network bridge on proxmox.

This all works to allow me to access the management webpage for OPNsense, due to it's anti-lockout rule passing all traffic to port 80 on itself. I've tried mimicking that rule to pass all traffic from my desktop, but still getting blocked by the firewall based on log entries.

Ultimate goal is to have a virtual network separate from the LAN that I can play around in as the beginnings of a homelab. Inside the homelab, I'm planning to install truenas, jellyfin, and bind among other tools. Most of these will need to be accessible from the physical LAN at some point.

Jonathan Heady
  • 3,671
  • 22
  • 16
  • It sounds like something along the lines of trying to access a private network from the Internet. I expect Proxmox bridging etc. adds to the complexity. Don't you have dual NAT issues etc? If you want to allow WAN side traffic you probably want to use a reverse proxy (haproxy, nginx, etc.) on your router to simplify management. By default there are a lot of barriers to communicating between the Internet and private subnets. What are you hoping to achieve with this type of configuration? What is the other firewall/router on the WAN (LAN) side of your box? – Blind Spots Jan 11 '23 at 02:57
  • So the OPNsense router WAN (VM), the Proxmox server (VM host) and the desktop are in the same network (IP subnet & broadcast domain), right? Is `desktop IP` a private IP? Or do all these hosts get a public IP? – Tom Yan Jan 11 '23 at 03:12
  • @TomYan Yes they are all same subnet/broadcast with private IPs. I've added an edit that shows the setup. – Jonathan Heady Jan 11 '23 at 14:59

1 Answers1

0

I've figured out how to solve the problem for portainer. I logged into portainer via the console in proxmox. Then I added a route into the routing table to pass traffic bound for 192.168.0.0/16 via the 10.0.42.x port on proxmox.

ip route add 192.168.0.0/16 via 10.0.42.x

Once that route was in place, access to portainer's web interface was successful from my desktop. This means that for every server inside the virtual LAN I want to access, a route needs to be in place for the "WAN" network.

I've tested adding the same route on a fedora VM inside the virtual network. My desktop is able to ping and connect to that VM now as well. This is without a firewall rule in place. So it looks like adding that route bypasses the firewall completely. In essence, turning the proxmox server into a router as well. I suspect OPNsense needs to remain in place to provide NAT so the VMs can access the internet.

I'm sure there's a way to achieve this same result without needing to add the route on each individual server. That will probably require an enterprise grade router instead of a firewall appliance like OPNsense or my home router. I'll also need to figure out where to add the route command into the startup files so that it'll persist upon reboot.

Jonathan Heady
  • 3,671
  • 22
  • 16
  • You shouldn't need the route if you have a `default` route with `10.0.42.x` (I assume it refers to the OPNsense) being the gateway / nexthop. If the OPNsense is not the "default gateway", then it's quite natural that you need the above route. In the typical WAN-LAN scenario, traffics from the WAN side to the LAN side will not be source NAT'd. You can probably set up such SNAT too, but obviously in that case, the LAN hosts will see all traffics from the WAN side being originated from the OPNsense itself. – Tom Yan Jan 11 '23 at 17:03
  • I added the route to point to the proxmox side of the virtual interface on the 10.0.42.0/24 subnet. Which is also the interface that is bound to the LAN interface in OPNsense. My understanding is that the default route only knows how to handle the IP range it's part of. Thus the 192 subnet destined packet has no idea where to go to access the systems it's heading to. Based on your comment, I think I'm wrong. I don't know enough networking (yet) to know why I'm wrong, or why the configuration works as desired. – Jonathan Heady Jan 11 '23 at 20:53