Tiernan's Comms Closet

Geek, Programmer, Photographer, network egineer…

Currently Viewing Posts Tagged mikrotik

Day 31 of #100daysofhomelab

Day 31 of and I am going through the config from my CHR to bring over to my RB5009, and, well, I have no idea what I was doing when I built the original config… Now to try and figure out what the config did, since I want to document it here so I know what I was thinking, but to also possibly help someone else… Mind you, at this stage, it won’t be much help… I also need to figure out how to add my Zerotier Bridge into the mix.

So, as trying to get a high level overview of how this works, lets start with this:

  • The cable modem comes in at 1Gb/s down, 50Mb/s up. It hands off at 1Gb ethernet and plugs into a switch on VLAN 900. Anything on VLAN 900 can get a public IP from that modem (statically assigned, I have 5 usages, the first being the modem to act as a gateway).
  • FTTH comes in and goes to my small quad 2.5Gb box, which then, using CHR (we call this DUB1-BK01), hands off a /29 to VLAN 905. Again, any devices on VLAN 905 can get a public IP from there, and use BK01 as a gateway.
  • For the current CHR (DUB1-BGP01) it being a VM has currently got 3 connections: eth1 is connected to VLAN900, eth2 is connected to VLAN905 and eth3 is connected to VLAN901. VLAN901 has a /27 from my block of /24 addresses, and anything on that VLAN can use an IP from that pool and the IP from DUB1-BGP01 as its gateway.
  • DUB1-BGP01 does some BGP routing to my upstream servers. lon1, which is based in Vultr London, and fra3, which is based in M&M Networks in Frankfurt Germany. lon1 has transit from Vultr and fra3 gets transit from M&M Networks, but also connects to multiple Internet Exchanges: DE-CIX Frankfurt, DE-CIX Dusseldorf, DE-CIX Hamburg, DE-CIX Munich, KleyReX, LocIX and LocIX Dusseldorf. More details of the network and peers, etc, are available on as204994.net.
  • DUB1-BGP01 connects to both lon1 and fra3 over WireGuard connections. All traffic to lon1 is sent over the Cable Modem link. All traffic to fra3 is sent over the FTTH link. Currently, there is no automatic failover if one link dies… This is where (hopefully) Zerotier comes into play.
  • I have a VM running on my i7 2.5Gb box that has connections to both VLAN900 and VLAN905, along with VLAN911. I have a bridge on that box that connects VLAN911 to a Zerotier network which is used only for internal peering. It has a /28 Public IP Range and anything on that bridge can use an IP from that network and talk to other machines. Currently that bridge is directly connected to my UDM Pro, and it gets a public IP and uses fra3 as a gateway. Sometimes traffic goes though fra3 but comes back over lon1 (due to asymmetric routing). But because of the way the network is working, all traffic can flow without issues.
  • The plan is to use that VLAN along with the 2 WireGuard links and give me 2 connections to lon1 and fra3. In theory, if one connection goes down, the traffic should be able to flow the other way…

So, at least that is the theory… How well this will work is anyone’s guess… But more messing with configs is required.

Day 30 of #100daysofhomelab

Day 30 of and I tried to look into getting my RB5009 setup, and well… it has the wrong power supply! EU, not UK/Ireland… More messing is required! [Update] Found the right supply, but fell asleep watching TV… more messing tomorrow…


Day 29 of #100daysofhomelab

Day 29 of #100daysofhomelab and my RB5009 finally arrived! The bad news is I am up to my eyes with some out-of-hours updates for my $DayJob… So, it will probably be tomorrow or Friday before I get to it… It’s been that kind of a day. I am OOF from Friday to next Wednesday, so I should have plenty of time to play with. I also started playing around with Tailscale Funnel. I got my hands on an invite, and it looks like I can invite other people to it… If you are interested, leave a comment. I have not actually done much with it, mostly reading the docs and testing it before i make it public… But should be interesting. Anyway, now for some links.

Day 26 of #100daysofhomelab

Day 26 of #100daysofhomelab and I have been trying to figure out why my internet has been unstable today… it up and down a few times… well, parts of it are… Zerotier seems to be sorting out my main network, it’s smaller parts that are going wonky… I am half thinking of leaving it till next weekend since my RB5009 arrives next week… This should help me sort out my network…

Also, spending time upgrading my WordPress site too… just making sure all is working correctly… Fun times…

[Update]: I have managed to upgrade to PHP 8.2, the latest Nginx and now have Varnish in front of the site… Let’s see what breaks…

Day 22 of #100daysofhomelab

Day 22 of #100daysofhomelab and I have been planning out my network update for when my RB5009 arrives… Not ready to share, yet, but it should be here on the 2nd Feb, so I will have a plan (maybe) by the weekend… Other than that, it’s a link dump for today:

Ok, I kind of got the following diagram, but it only makes sense in my head, and I’m not even sure it makes sense there… I’ll leave this here without further explanation, till maybe the weekend…

Day 21 of #100daysofhomelab

Day 21 (slightly late, forgot to post this last night) of and its a links day.

on a more different note, my Mikrotik RB5009UG+S+IN is finally on its way! Hopefully will have it next week! Happy days!

Day 19 of #100daysofhomelab

Day 19 of #100daysofhomelab and not done a lot today, so its mostly links…

Day 17 of #100daysofhomelab

day 17 of #100daysofhomelab, and I haven’t done much, so its a link roundup today:

Day 14 of #100daysofhomelab

Day 14 of #100daysofhomelab and I have been thinking about future upgrades if I had the money… So, I have my CloudShed in the back garden. Currently, I only have an HP Micro Server and a (not currently in production) Dell R720, along with a Ubiquiti Edge Switch 48 Lite. Between the Shed and the house is a fibre link purchased through FS.com, with 6 pairs. Currently, only 1 pair is in use, giving me a 10Gb/s between the house and shed, and with the easy option to upgrade to 20Gb. But I have been thinking bigger.

I have been looking at the Mikrotik CCR2004-1G-2XS-PCIe (a bit of a mouthful…) SmartNIC. It’s a full MikroTik router on a PCIe Card. It has a Quad-core ARM Processor, 4GB RAM, some storage and 2X25Gbit/s Interfaces… Well, technically, 4… there are 2 front connectors and 4 that the host server sees… If I am reading the diagram below correctly, it looks like all ports are seen by the Host, but 2 go through the bridge and 2 go direct… I haven’t played with one yet (Mikrotik, if you are listening, hint, hint!) so not sure how it would work… A review from Alyx Wijers says that on the Linux box they tried, the 10Gb SFP+ module they had shows in passthrough and the other 2 are connected to the bridge… Ideally, for the ideas i have for this, I passthrough would be handy for stuff like storage, but i would want the rest of my traffic going over that bridge interface… Or at least i think thats how it would work…

The card has 2X25Gb ports (SFP28 ports) that connect to the rest of your network, for example to a CRS504-4XQ-IN switch (4x100Gb ports, which can be broken out into 4x25Gb ports each…). If you go through the bridge, you get all the features of RouterOS, like firewall rules, VXLAN, etc, all in the NIC. The switch then doesn’t need to do as much, letting it do the switching and leaving everything else at a NIC level. If you use passthrough, you, essentially, bypass the router/firewall rules… I think…

So, what would my plan for the upgrade be? Well, this is where things get expensive… I would need 2 of the switches (one in the house, one in the shed linked with a single 100Gb fibre). Then, I would need 8 of the SmartNICs (GodboxV3, GodBoxV2, 1 for each of the R720s (second one coming soon), 1 for each of the R620s (coming soon), one for the HP DL380 G8 (also pending) and 1 for the big storage box… again, pending). The plan would be that GodBoxV2 and V3 would be in the house, and both connect to the house switch at 25Gb a sec. there would be uplinks to internal 10Gb switches along with the UDM Pro.

In the shed, the 6 servers would each connect to the switch at 25Gb, using 10 of the renaming 12 ports. The R720s, Storage Box and HP will probably get 2x25Gb connections. In theory, the R620s could also connect at 50Gb but I would have no extra room later… Might not be a major issue, mind you. There would be spare ports in the house… I could, in theory, get a second 100Gb switch for the shed! 😛

But, what would this cost? Well, current prices are showing that the cards are around 200EUR a pop and the switch is just shy of 800 quid… so, for a little under €3200, I could get 2 switches and 8 NICs. I would need break-out cables, 100Gb Optics, and some other bits, so, say 500 quid for that… So, just under 4k? One of these days, hey! I can dream!

MikroTik CRS504-4XQ-IN Review Momentus 4x 100GbE and 25GbE Desktop Switch (servethehome.com)

Review: A Dive into Mikrotik’s Weird SmartNIC (CCR2004-1G-2XS-PCIe) // Alyx Wijers

RouterOS Using Host names in Firewall Rules

As a follow-up to yesterday’s post on RouterOS Blocking Machine access to all but one IP, I thought I would show how to add extra IPs to that list, without having a shedload of firewall filters.

  • First things first, get your list of IPs you allow access to. In my case, I just did an NSLOOKUP on the name and got the IPs.
  • Create an “Address List” in RouterOS. This can be done on the Web Interface by going to IP / Firewall / Address List and clicking Add. I had none previously, so I created a new rule, naming it ExpressVPN (the lads I use for VPN access) and added the first address.
  • this is where things get interesting. for extra IP (for ExpressVPN, I have 4) you create a new address with the SAME name, but different IP.
  • in your firewall rule, you should have either an src address or a dst address. in my case, I had both, but this was a change for the dst address. I removed the address from the rule, and I added it as a dst address list entry. If you have multiple address lists, you will see them here.

to do this at the command prompt:

this will block any traffic, other than the IPs in the expressVPN address list, for the machine