Tiernan's Comms Closet

Geek, Programmer, Photographer, network egineer…

Currently Viewing Posts in Networking

Unifi Network Update 7.1.61

A few weeks back, Ubiquiti released a pre-release update for the Unifi Network Controller, version 7.1.61. It got installed on my UDM and I noticed a few interesting bits that you might find handy… First, you will need to be signed up for Unifi Early Access before you can download or even read the release notes, but this is just a quick update based on my findings so far.

The first thing to note: You can see the list of devices connected to switches on the Overview Tab. I can’t remember exactly when that was added, but I think it’s new…

Under the ports tab, you now have a ports insight option:

Clicking this give you:

You can also select multiple ports and make changes at a bulk level:

You can also see a bit more info about each port:

Screenshot 2022-04-25 at 22.35.59

Teleport VPN is also now added. This makes giving someone access to your network a LOT easier than usual. They will need the WifiMan software on Android, iOS or Mac to join. Not sure what happens on a Windows machine… Maybe it’s coming soon? To use it, just generate a new link and send it to your user. Not sure how to remove them afterwards (if you want to give them temp access for example…)

Final Interesting part, and something I have been waiting for for a while, under Traffic Management, you can now create custom traffic rules:

You can set it based on destination Domain Name, IP or even the full internet:

And you can set the Source to be All Devices, group of devices (network) or individual (or multiple targeted) devices.

Finally, you can set the output internet connection.

If you had multiple internet connections, and one had better speeds for stuff like Netflix, or you wanted to send bulk data over a different link, you can do this using this feature. Very cool stuff.

So, still testing, but looking good so far.

Running a Raspberry Pi in a car and backing up dashcam footage

A few months back (well, November 2020) I wrote about connecting to my car with Zerotier. In this post, I mentioned using a TP-Link router running OpenWRT and a Huawei LTE dongle to connect to the internet, which allowed me to then connect to my Blackvue Dashcam and watch remotely… But it had some issues I wanted to fix:

  • The Huawei Wingle was a little slower on 4G than I would have hoped…
  • When the power in the car went out, everything stopped working immediately (12V sockets in the car run for about 20 min after the engine shut off)
  • It did not connect to the WiFi in the house when parked
  • No option for backing up Video…

So, I went digging to find some alternatives… and I realized I had a load of them floating around the house: the Raspberry Pi. Specifically, the 4GB Pi 4. I got my hands on a Pi UPS Hat, a couple of 18650 Cells and an SSD Expansion board with a 512GB (overkill I know) SSD. I also got a BlackVue Power Magic Battery, B112, which will power the Dash Cam (a BlackVue DR750S-2CH). It has 2 USB ports, which allows me to run both the Pi and the new WIFI router, a Netgear Nighthawk M1.

When the car starts, it powers, via the 12V socket in the boot of the car (trunk for my American friends), which powers the Blackvue Battery. Cables run from there to the front of the car where the front camera is. (there is also a rear-facing camera in the boot too… more cables!) This then also starts the Pi and starts charging the 2 18650 batteries. Finally, well, at the same time really, the Nighthawk starts running too. Because the batter on this was running hot, the battery is removed from this.

The Pi is hooked to the Nighthawk via ethernet and the WIFI is set to connect to the house when it sees it. The BlackVue uses the WIFI from the Nighthawk for its internet requirements. When the pi boots, it connects to Zerotier for management via SSH or VNC (I use VNC to remote into the box and watch the live video when the car is parked or when someone else is driving).

There is also a python script that is scheduled to run every 15 min that downloads the videos from the Dashcam. It also downloads any GPS and other info. The folder these files are downloaded to is on the SSD and is shared with my machine at home via Resilio Sync. To make sure I don’t use all my LTE usage, the machine at home is set to only download what I want to download. So, if the car is somewhere else, I can download specific files when I want, or when at home, I can download full days, if required.

It’s been running for a few weeks now, and so far, so good. I haven’t had to do any clean up of the SSD, yet, but I would guess that eventually, I will need to look into that… With the 4G connection and Zerotier, I can then connect to my car and watch the live video whenever it is online, and whenever it is driving, within 15 min it will start downloading videos. I could, in theory, do a LOT more with the Pi in the car… Some ideas that come to mind:

  • Turn WIFI off on the Nighthawk and use the Pi as a Router, probably adding a second WIFI adapter to get better range… This could then have PiHole running on it for monitoring DNS traffic…
  • Since I have access to the GPS files in (somewhat) real-time, use it to map the car in somewhat real-time. Though, I do this already using Ruhavik and a Teltonika FMC-001.
  • Connecting to the car’s OBDII port (On-Board Diagnostics) and getting data from the car… Technically, again, the FMC001 does most of this, but in theory, it could be replaced with something else…

Keep an eye on the blog for future possible projects with this… Not sure where this project will get me, but we will figure it out at some stage… Leave a comment if you have questions!

Ubiquiti UDM Pro Fail over to Speedify

So, this has been a blog post in the making for a while now but never got around to fully writing it up, so here goes nothing…

I run a UDM Pro in the house. It has 2 WAN Links: 1 1Gb link and 1 10Gb Link. I also run AS204994, my own ASN with its own Transit and Peering connections, mostly in Europe. There is a VM in the house which acts as a connection to AS204994, which gives me a full connection to the Internet through my own ASN. More details on my AS204994 blog are here.

That connection is hooked up to the 10Gb Link on the UDM Pro, which is listed as the primary internet link. Details on how these works were uploaded in this video on YouTube:

In the video above, I was using OpenMPTCPRouter to connect to the internet, but it’s been causing some issues lately, I decided to try something else.

The new setup is an Intel Nuc (i3 with 32GB RAM and 2x512GB SSDs… VERY OVERKILL for the job at hand) running Ubuntu Linux. It has a USB Hub with 3 USB Ports and an Ethernet port connected, giving me 2 Ethernet ports on the box in total. 2 of the USB Ports are connected to USB 4G Modems from Huawei and the external ethernet port is directly connected to my cable modem.

USB Hub with 1 Huawei Modem and connection to second

Both modems and the ethernet port are connected to the NUC with full internet connections (The Huawei boxes give up NATed IPs, but the Cable modem is a full public IP) and then Speedify takes those 3 connections and does some bonding magic. Speedify is a handy little VPN service that does connection bonding. You can use it to make sure your internet is rock solid using multiple links, make sure streams are stable, etc. It can bond Wifi Links, LTE modems, Cable Modems, DSL, etc. Anything that can connect and be bonded. The only issue I have with it, compared to OpenMPTCPRouter is that you don’t control the upstream server…

Speedify is set in shared mode, so the internal port on the NUC is set to share the internet connection. This is hooked to the 1Gb WAN Port on the UDM Pro. This is set for failover only (currently the only option on a UDM Pro) so if my AS204994 link goes down (VM reboots, VM host dies, Cable modem connection goes out, etc) I will still have a connection. If the cable goes out, it will use just the 4G links, but if everything is running, I get all 3 connections.

Connecting to my car over ZeroTier

I use ZeroTier on my network for a good few things, including internal network peering between BGP VMs, management of machines, and now, connecting to my car over LTE. This is one of those posts that sounds silly, but is very handy! First, the parts list:

  • Car…
  • 3G/4G/5G modem of some sort. I am using a Huawei Wingle… Can be used without the Router below, but I wanted Zerotier, so I have it in modem only mode…
  • A router that supports Zerotier. I am using a modified TP-Link TL-WR703N upgraded to 16MB ROM and 64MB RAM. This is required for newer OpenWRT builds
  • a dashcam that connects over Wifi. I am using a BlackVue DR750S-2CH
  • Latest ROOter software from Of Modems and Men
  • Patients…

After installing the the latest copy of ROOter on the TPLink (or router of your choice) and getting the modem configured correctly (this took a while) you need to install the Zerotier software though the dashboard. Once installed, I joined my Zerotier network using the CLI (SSH into the router) and the approved it though the my.zerotier.com dashboard. Once its approved and connected, you can now go to the Zerotier IP and get to the router directly. From here, you can either setup a route in Zerotier to point at the internal network behind the router, or, in my case, setup a  SSH tunnel to the dashcam. I found the IP given to the dashcam and used SSH forwarding to get to it. Finally, i used the URLs from Digital-Nebula’s hackview repo to get to the different URLs. I use this to download stuff like GPS logs, emergency videos, etc. I have to clean up some scripts at some stage for this, and plan to upload them at some stage.

If anyone has any questions, leave a comment!

Domain Joining a machine over VPN and Password Resets/Changes with Azure AD

With the whole Work From Home thing probably becoming more and more normal in the years to come (I can count on 2 hands how many times I have physically been in my main office in the last 7 months) there are a couple of certainties in that people will come up against. One is passwords expiring and needing to be changed, one is password resets being required and finally laptops or desktops needing to be domain joined or connected to the domain before they can be fully provisioned. As the (currently only) IT guy in our office, I have had to deal with these first hand, and decide to write this post, helping both my fellow employees, and possibly other IT Admins stuck in this challenge.

So, as the IT person, there are a couple of assumptions:

  • You have on premises AD
  • You have Azure AD (P1 and above seems to be required if users are mixed AD and on prem. Free allows just Cloud users).
  • Azure AD Sync installed and enabled

If all above are set, you will need to follow the steps to Enable Azure Active Directory Self Service Password Reset. I have enabled this on our domain. Next, you need to get your users to setup their secondary authentication for backup. All our users have a 2FA requirement, so most of them had that already. New users need to go though those setups. Finally, if a user needs to change or reset their password, they can do so though https://aka.ms/sspr. If all is done well, that reduces the amount of support calls I (and you) get.

Now, the next task: domain joining over VPN. This is a bit more “fun” to play with.

First, you need a VPN connection. We use Meraki gear using Active Directory for RADIUS auth. I wont go into too much details on setting that part up, but the script we use to build the VPN connections for users is below. This will probably be different for different VPNs, but this is our starting point.

Lines you need to change are at 8, 9, 10 and 47. Line 39 can also be modified to change from Split Tunneling (only sending traffic to internal subnets) or full Tunneling (all traffic over VPN). If you have multiple internal subnets, Line 49 can be copied with more.

The most important part we need though is line 34. The -AllUserConnection allows the connection to be available to all users on the machine, but also on the start screen. This is important.

So, with all that in place, you will need to connect to the VPN

you should now be able to join the domain as if you where on your local network.

Enter Domain details and change name of machine if required
when asked enter your domain username and password
You will be welcomed to the domain
and then asked to reboot

reboot your machine as usual and when it boots, you should see a new option on the login screen

VPN login option

Click this icon and if you only have one VPN connection the screen below will show up. If you have more than one, you will be given a list of options to use.

Login to VPN at the login screen

Enter your domain credentials. Since our AD and VPN use the same credentials, it will automatically log you in aswell.

Machine is now domain joined and logged in, and in my case, finishing setup

So, there you have it. How to domain join a machine outside the network. Now, in reality, Azure Active Directory and Intune would probably be the better option, but that’s future work…

Network Update Info April 2019

So, this post has been a long time coming! A load of different things to talk about, so lets get started!

GodBox V3

So, for a long time, I have been thinking about GodBoxV3, the replacement to GodBoxV2. And when planning this, i had some ideas of what it should be:

  • Minimum of 2×16 cores (double godboxv2)
  • About the same RAM, if not more
  • FAST STORAGE!
  • Is able to run my twin 30" 4K monitors
  • Would like 10Gb/s NICs

Well, It finally happened! I got the machine, built it and, well, its impressive! How did i do with specs? Well…

All is good! Photos, more details and benchmarks coming soon… stay tuned!

Finally 10Gb/s Networking!

Since GodBoxV3 had a few 10Gb nics, i needed to upgrade the network to support it. I ended up with a Ubiquiti Networks EdgeSwitch-XG. 16 ports (12 SFP+ and 4 RJ45). The SubperMicro board has 2xRJ45 ports. Due to lack of RJ45 ports, GodBoxV3 is connected to 1, GodBoxV2 is getting a 10Gb card soon, which will be connected to 1 port, and a new Sun Microsystems server (details below) will be getting the last 2… Of the SFP+ ports, 2 are connected to the EdgeSwitch Lite, 2 to the Synology (it got a 10Gig NIC reciently too!) and 2 to the new NAS (again, more details below!)

Good bye Mikrotik, Hello EdgeRouter 4

Since i was going all Ubiquiti gear (Wifi is Unifi gear) i got rid of the old Microtik and replaced it with a Ubiquiti ER4. Happy days! Got some plans for this, more details coming soon…

Updates to BGP Stuff, including IPv6

I lost one VPS in London, but replaced it with a new one from HostUS. I still use Vultr, Packet and VServer.Site as providers too. I am also adding more and more IPv6 stuff too… There is a post on AS204994 explaining a lot of this.

New NAS and more storage!

New NAS got purchased: QNAP TS-932X. I have 5X8TB spinny disks (shucked from 5 WD My Book 8TBs) + 4 X 500GB WD Blue SSDs.

New Servers and cooling updates

Moved lots of stuff around the room… Servers run cooler, and less noisy! happy days! I also got my hands on a very nice looking Sun Server X3-2. Its a Dual Xeon E5 (currently got quad cores, going to upgrade it to 8 cores) and i think its got 16GB ram and 4x300GB SAS Disks. It also has 4X10Gb nics! ESXi will probably go on here!

VMWare in the house

Up till recently, I ran Hyper-V all round. Its still on GodBox V2 and V3 (v1 has a HDD issue, so its off…), but the main VM hosts (the C6100’s) are being migrated to VMWare ESXi… Why? Its a learning exercise… We see how it goes…

So, long update… Any questions, comments, etc… shout!

Adding a Netgear LB2120 to the homelab

A few months back, Three Ireland came out with an LTE broadband offer: Unlimited* LTE broadband for EUR30 per month. It did come with a 18 month contract, but I pulled the trigger and got it as a backup link. I picked this up in the local Three store, and they had a couple of options for modems: a couple of Huawei mobile Wifi hotspots (E5573 or E5577) or a Huawei B525 Modem, which is designed for home use. Alternatively, there was a Sim only option, but given the modem was free with the contact, i went with the B525.

The B525 is not a bad router, don’t get me wrong, but its a Router… i already have a few of them, including my Mikrotik RouterBoard CCR1016-12G, an EdgeRouter POE, a Ubiquiti USG 3 and some virtual ones too… yea, don’t ask… What i wanted was a modem; no WiFi, no routing, and give me a full, non NATted IP to the internet. There was some mention of some of the Huawei modems being able to be put into bridge mode, but i could not find out how to do it… That’s where the Netgear LB2120 comes in.

The LB2120 (there are a few different models, but mine is the Europe edition) has a Micro SIM Slot,a WAN and a LAN port (both GigE), Power in, Power button and 2 inputs for Aerials.

The home page is fairly basic, and gives you all you need: how much data you’ve used, how much you have left, when the data plan resets, etc.

There is also an alerts option, so you can get it to send you an SMS when something happens:

But the relay handy stuff is under Advanced setting/LAN:

You have the option of using it as a router, or using it in a Bridge. Needless to say, i bridged it and got a fully public IP. Currently, mine is hooked up to the second WAN port of the USG, and is currently serving about 30-40% of the traffic on that network (mostly media devices, IOT stuff, etc). Speed wise, its not bad.

Not as good as a hardwired connection, but its only getting 4 bars, and its about 1km to the nearest cell tower. I do want to get some external aerials for it, to see if i can boost the download/upload speed, but we will see. Also, i plan on changing out the Mikrotik for something else… lets see what i end up with!

*Unlimited is mobile network speak for 750GB per month… which does not sound very unlimited to me… but, anyway…

AS204994, Own IP Space and Anycast

So, if you are reading this page, it is being delivered with the magic of Anycast… Well, technically, it was before, since i used Cloudflare, and it still is because of Cloudflare, but also because of my own ASN (As204994), some servers in different locations, and some magic, which i will explain a bit of in this post.

This all started late last year when i got my hands on an ASN and a /48 block of IPv6 addresses. I had been reading stuff about BGP, routing, etc, and decided to go all in. it was quite cheap with the help of HostUS. All in, it was about $50 for the year. As part of the process, i needed 2 upstream providers to say they would accept my announcement. They were Hurricane Electric though their Tunnel Broker service, and Vultr using a few of their VPSs.

After i got my space and ASN, i started to announce the V6 addresses over Vultr and Hurricane Electric, and all was good. I had 2 Vultr servers: 1 in London, UK, and one in New Jersey, USA. I had my home machine announce to HE, and then also link to both Vultr servers using Zerotier. All worked well, but due to some family issues, i never got around to putting it into production… till now.

Those 3 servers now share an IPv6 address on the loop-back port. When you (well, Cloudflare) asks for that IP, the closet (network) with that IP responds, and the NGinx server on that box sends back the contents of the site. This site is hosted on each box, since its fully static, but both AS204994 and TiernanOToole.net are hosted in Ghost, so Dublin (my machine in the house) serves them, and both Lon1 and Nyc1 do proxying. so, most requests from the US are hitting the box in NYC and the ones in Europe share either Dub1 or Lon1. I have some tweaks to do with which servers will be running where, and may add more, but currently its working well.

So, how do you figure out what server responded? Simple. Open your Dev tools on your browser, go to network tab, refresh, and see the response headers for anything on this domain. You should see something like below.

Over the next while, i will be updating tiernanotoole.net with more details on how this works, and more stuff will end up on AS204994.net too. If anyone notices any weird and wonderful issues, shout. If you have more questions, shout.

Zerotier and Minio Followup

in a previous post, I talked about setting up a distributed S3 like data storage system using Minio and ZeroTier. Well, this week, the ZeroTier guys tweeted about this.

A few people then started asking questions, and looking for a follow up, so here it is…

First, a quick recap. I had 4 machines, all running Linux. Three of them were in 1 time zone (GMT+1) and one was in another (GMT). Looking at the Distributed Minio Quickstart Guide again, there is a mention of times being in sync… which is probably why this did not work as planned… and by “not work as planed”, I mean that Minio would crash, or not be responsive, or not write data in the place it should have… which was a pain. But looking at the documentation again, they do mention that Windows support is “experimental” which means, hopefully, some day it will be not so experimental, and might work… Given that most of my machines in house are Windows boxes, this would be a nice feature.

Now, what about ZeroTier? Given they posted it to their twitter? Well, it worked. it did the inter connect stuff well, and, given bandwidth limitations on a home broadband connection, it was still quite fast.

So, the question is, how fast? Well, on my Surface Book on a WiFi connection in the house, behind a Meraki MX64 firewall, connecting to the GodBoxV2 over FTP though ZeroTier, i get the following result:

the same download over FTP direct (no ZeroTier) does the following:

So, direct over FTP is faster… in this instance by about 70%, but, over the download, it did get slower (seen it hit 12 at one stage) and because its over WiFi, those are a bit wonky…

I did get one last screen shot:

as you can see, the Zerotier network adapter is showing 77.3Mbps, but the main network adapter is showing 80.8Mbps. There would be other traffic there, but if we assume there is nothing but ZeroTier traffic being sent, there is about 5% of an overhead.

So, to wrap up: Minio and its distributed storage system over ZeroTier needs more testing. Ideally, all hosts need to be in the same time zone, or at least have the same time… Will try work on that soon. As for ZeroTier? I am extremely happy with them. Its fast, easy to setup, and easy to configure. What more could you ask for? Oh, and free, unless you need a pro account!

Distributed S3 data storage using Minio (and Zerotier)

So, something i have been looking into in recient times has been Distributed Storage, and, more specifically, how to use the storage in my many, many machines to protect data, and also increese my usable space… There are a few projects on the market that do this (Ceph, NooBaa and Gluster all spring to mind) but some are more painful to setup than others… which brings me nicely to Minio. Minio is a 20ish MB executable you download from their site, mark it as executable (on Linux or Mac Boxes) and run… and you have yourself a S3 compatable storage server… Simples!

“But Wait!” i here you screem! “thats not distributed!”. Well, yes… but, it can be! Their Distributed Quick Start Guide, which is where i started with this, allows you to run a distributed copy of your data. I will let their documentation explain more, but this is what i did:

  • download the minio server (single executable file) on a minimum of 4 machines.
  • on each machine, run a command like the following:

replacing accesskey and secretkey with keys (check minio documentation to get these) and foldertoexport with, well, the folder you want to export!

For me, i have 4 servers currently clustered. 2 are in online.net (one in Paris, one in Amsterdam), 1 in OVH.NET (France, somewhere) and one in Dublin (GodBoxV2 currently). They are all interconnected using ZeroTier (I will explain that later) and so far, so good… only ran some basic tests, but with it, i could loose 2 machines and still have data… Not bad for free! I will run some speed tests soon.