Routing a public subnet through a GRE tunnel using OpenBSD, WireGuard and rdomain(4)

Introduction

Goal

We want to terminate a public IPv4* subnet, offered through a GRE tunnel, on an OpenBSD machine, and expose the available IP addresses through WireGuard to another machine.

* The setup for IPv6 should be analogous, but we will cover that in another post. Yes, IPv6 should have priority, I know, but due to circumstances, etc. etc.

Setup

  • OpenBSD machine
    • Endpoint for GRE tunnel that receives all traffic for the public subnet
    • Routes IP addresses from the public subnet through WireGuard to the Linux machine
  • Linux machine
    • ‘Endpoint’ for all available IP addresses in the public subnet
    • This machine may further decide where to route the IP addresses, for example on a VLAN

We reserve three** addresses from the public subnet to make this setup possible.

We use the following prefixes for documentation purposes (taken from RFC5737):

  • WAN_BSD endpoint: 203.0.113.113/24 (default router: 203.0.113.254)
  • WAN_LIN endpoint: 192.0.2.2/27 (default router: 192.0.2.1)
  • WAN_GRE endpoint: 192.0.2.192
  • Routed, public subnet: 198.51.100.232/29

** Perhaps two would suffice as well.

Layout

WAN_GRE --.
          |
          |- GRE-tunnel (routes 198.51.100.232/29)
          |
          |   .-- gre232 (198.51.100.232/32)
WAN_BSD --+-- [ OpenBSD ]
          |   `-- wg232 (198.51.100.238/31)
          |
          |- WireGuard tunnel
          |
          |   .-- wg232 (198.51.100.239/31)
WAN_LIN --+-- [  Linux  ]
              `-- vlan232 (198.51.100.233/32)
                 `-- host (198.51.100.234/32)
                 `-- host (198.51.100.235/32)
                 `-- host (198.51.100.236/32)
                 `-- host (198.51.100.237/32)

Configuration

Before you start typing commands as they appear in this document, please pause for a moment, and take the following into consideration:

  • Setup a restrictive firewall on both OpenBSD and Linux before continuing. These machines will receive internet traffic in the end, and it’s better to carefully open up the firewall where needed, than to bluntly accept everything the world may send at you;
    • Yes, you will get traffic on port 23 (Telnet)
    • Yes, you will get traffic on port 445 (SMB/CIFS, or Samba)
    • Yes, you will get traffic on port 5060 (SIP)
    • Or even broadcasts from your GRE tunnel provider
  • Pay attention when restricting ICMP traffic. Or better: do not restrict it other than rate-limiting where appropriate. It will help you with traceroutes, path MTU, and generally detecting if your setup works both ways.
  • We assume the OpenBSD and Linux machine have a working internet connection. Preferably, you also have a way to interact with the machine when this connection gets interrupted.

OpenBSD

In order to properly route traffic of the public subnet to/from to the GRE tunnel without messing with the ‘regular’ routing of our OpenBSD machine, we use rdomain(4), or routing domain, to separate this traffic. This functionality is similar to Linux’ network namespaces (on Linux: man 7 network_namespaces).

We chose rdomain 232, because that’s the first address in our public /29 subnet. In short, everything that has an address of the public /29 subnet needs to be in that rdomain to completely* separate it from our default routing domain. In some cases, we will see this setting on the interface part, and in some cases on the tunnel part. If we do not set rdomain (or equivalent for the tunnel part), rdomain 0 is implied: the default routing domain.

* It is still possible to transfer traffic from/to routing domains, but this is set through configuration in pf.

Interface definitions

/etc/hostname.vio0 (our connection to BSD_WAN, rdomain 0):

inet 203.0.113.113/24
!route add default 203.0.113.254

/etc/hostname.gre232 (our connection to GRE_WAN; the tunnel is setup in rdomain 0 through vio0’s default gateway, but the traffic inside the tunnel will be put in rdomain 232):

tunnel 203.0.113.113 192.0.2.192 tunnelttl 225 rdomain 232
inet 198.51.100.232/32

/etc/hostname.wg232 (our connection to the Linux machine; the tunnel is setup in rdomain 0 through vio0's default gateway, but the traffic _inside_ the tunnel will be put in rdomain 232`):

wgkey <openbsd-wg-private-key> wgport 51232 rdomain 232
inet 198.51.100.238/31
wgpeer <linux-wg-public-key> wgaip 0.0.0.0/0
!route -T232 add 198.51.100.233 198.51.100.239
!route -T232 add 198.51.100.234 198.51.100.239
!route -T232 add 198.51.100.235 198.51.100.239
!route -T232 add 198.51.100.236 198.51.100.239
!route -T232 add 198.51.100.237 198.51.100.239

Note: we do not set wgendpoint because our Linux machine “connects” to the OpenBSD machine. Otherwise, you could append wgendpoint 192.0.2.2 51820 to the wgpeer stanza to set the endpoint.

Configuring the OpenBSD end

sh /etc/netstart wg232 setups the WireGuard interface with the corresponding interface configuration. After that, you could run ifconfig wg232 and/or wg to inspect what happened. This could look similar to this:

$ ifconfig wg232
wg232: flags=80c3<UP,BROADCAST,RUNNING,NOARP,MULTICAST> rdomain 232 mtu 1392
        index 6 priority 0 llprio 3
        wgport 51232
        wgpubkey <openbsd-wg-public-key>
        wgpeer <linux-wg-public-key>
                tx: 0, rx: 0
                wgaip 0.0.0.0/0
        groups: wg
        inet 198.51.100.238 netmask 0xfffffffe

$ wg
interface: wg232
  public key: <openbsd-wg-public-key>
  private key: (hidden)
  listening port: 51232

peer: <linux-wg-public-key>
  allowed ips: 0.0.0.0/0

Identically, this works for gre232 (sh /etc/netstart gre232)

pf

We provide a /etc/pf.conf that should give you the ; you can load it using pfctl -f /etc/pf.conf.

ext_if = "vio0"
prod_if = "gre232"

# Bogons: these address ranges should not occur on the internet. Beware that these ranges could change (e.g. 224.0.0.0/4 is
# currently reserved for multicast).
#
# Note that you should exclude 198.51.100.0/24, 203.0.113.0/24 when running this documentation setup, and due to how pf.conf works,
# you should delete those lines; commenting them on this multi-line statement would also comment the last two ranges.
table <bogons_ext> const { 0.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, \
        127.0.0.0/8, 169.254.0.0/16, 172.16.0.0/12, 192.0.2.0/24, \
        192.88.99.0/24, 192.168.0.0/24, 198.18.0.0/15, \
        198.51.100.0/24, \
        203.0.113.0/24, \
        224.0.0.0/4, 240.0.0.0/4 }

table <prod_usable> const { 198.51.100.233, \
        198.51.100.234, 198.51.100.235, 198.51.100.236, \
        198.51.100.237, 198.51.100.238, 198.51.100.239 }

set skip on lo

# default block rule
block log all

# failsafe: make sure we do not send from/to bogons to the internet
block out log quick on $prod_if from <bogons_ext>
block out log quick on $prod_if to <bogons_ext>
block out log quick on $ext_if from <bogons_ext>
block out log quick on $ext_if to <bogons_ext>

pass in quick on $ext_if proto { icmp ipv6-icmp }
pass in quick on $ext_if proto tcp to port ssh
pass in quick on $ext_if proto udp to port 51232     # Linux->BSD WireGuard

# Allow GRE traffic from/to our GRE tunnel provider
pass in quick on $ext_if proto gre from 192.0.2.192
pass out quick on $ext_if proto gre to 192.0.2.192

# Allow ICMP to this machine on the GRE tunnel interface
pass in log on $prod_if proto icmp from any to 198.51.100.232/32

# Allow traffic to flow freely from/to GRE and WireGuard tunnels for our usable range.
# Note: you could build in restrictions here as well.
pass on wg232 from <eipprod_usable>
pass on wg232 to <eipprod_usable>
pass on $prod_if from <eipprod_usable>
pass on $prod_if to <eipprod_usable>

block in log quick from urpf-failed

pass out on $ext_if proto { tcp udp icmp ipv6-icmp } all modulate state

sysctl

Set the following in /etc/sysctl.conf:

net.inet.gre.allow=1
net.inet.ip.forwarding=1

Next, run sysctl for each line, using the line itself as argument, e.g. sysctl net.inet.gre.allow=1. (Or reboot the machine).

Linux

systemd-networkd

Yeah, sorry.

WireGuard

Intricacies

One of the key points that did not really come forward in the available documentation on wg(4) was the wgrtable n option for hostname.if(5). rdomain n exists for putting the tunneled part of the wg(4) interface into a certain routing domain. However, in our case, we wanted to have the setup of the tunnel in a certain routing domain, whereas the tunneled part should be in rdomain 0.

Fortunately, ifconfig(8) has a WireGuard section that provides us with this info (although it took me a while to find where it’s defined). In our setup, the endpoint of the tunnel is actually part of the public subnet we want to route to another machine, in another rdomain than where the GRE tunnel terminates.

References