Tiernan's Comms Closet

Day 31 of #100daysofhomelab and I am going through the config from my CHR to bring over to my RB5009, and, well, I have no idea what I was doing when I built the original config… Now to try and figure out what the config did, since I want to document it here so I know what I was thinking, but to also possibly help someone else… Mind you, at this stage, it won’t be much help… I also need to figure out how to add my Zerotier Bridge into the mix.

So, as trying to get a high level overview of how this works, lets start with this:

• The cable modem comes in at 1Gb/s down, 50Mb/s up. It hands off at 1Gb ethernet and plugs into a switch on VLAN 900. Anything on VLAN 900 can get a public IP from that modem (statically assigned, I have 5 usages, the first being the modem to act as a gateway).
• FTTH comes in and goes to my small quad 2.5Gb box, which then, using CHR (we call this DUB1-BK01), hands off a /29 to VLAN 905. Again, any devices on VLAN 905 can get a public IP from there, and use BK01 as a gateway.
• For the current CHR (DUB1-BGP01) it being a VM has currently got 3 connections: eth1 is connected to VLAN900, eth2 is connected to VLAN905 and eth3 is connected to VLAN901. VLAN901 has a /27 from my block of /24 addresses, and anything on that VLAN can use an IP from that pool and the IP from DUB1-BGP01 as its gateway.
• DUB1-BGP01 does some BGP routing to my upstream servers. lon1, which is based in Vultr London, and fra3, which is based in M&M Networks in Frankfurt Germany. lon1 has transit from Vultr and fra3 gets transit from M&M Networks, but also connects to multiple Internet Exchanges: DE-CIX Frankfurt, DE-CIX Dusseldorf, DE-CIX Hamburg, DE-CIX Munich, KleyReX, LocIX and LocIX Dusseldorf. More details of the network and peers, etc, are available on as204994.net.
• DUB1-BGP01 connects to both lon1 and fra3 over WireGuard connections. All traffic to lon1 is sent over the Cable Modem link. All traffic to fra3 is sent over the FTTH link. Currently, there is no automatic failover if one link dies… This is where (hopefully) Zerotier comes into play.
• I have a VM running on my i7 2.5Gb box that has connections to both VLAN900 and VLAN905, along with VLAN911. I have a bridge on that box that connects VLAN911 to a Zerotier network which is used only for internal peering. It has a /28 Public IP Range and anything on that bridge can use an IP from that network and talk to other machines. Currently that bridge is directly connected to my UDM Pro, and it gets a public IP and uses fra3 as a gateway. Sometimes traffic goes though fra3 but comes back over lon1 (due to asymmetric routing). But because of the way the network is working, all traffic can flow without issues.
• The plan is to use that VLAN along with the 2 WireGuard links and give me 2 connections to lon1 and fra3. In theory, if one connection goes down, the traffic should be able to flow the other way…

So, at least that is the theory… How well this will work is anyone’s guess… But more messing with configs is required.