Tiernan's Comms Closet

Geek, Programmer, Photographer, network egineer…

Currently Viewing Posts in Internet

Unifi Network Update 7.1.61

A few weeks back, Ubiquiti released a pre-release update for the Unifi Network Controller, version 7.1.61. It got installed on my UDM and I noticed a few interesting bits that you might find handy… First, you will need to be signed up for Unifi Early Access before you can download or even read the release notes, but this is just a quick update based on my findings so far.

The first thing to note: You can see the list of devices connected to switches on the Overview Tab. I can’t remember exactly when that was added, but I think it’s new…

Under the ports tab, you now have a ports insight option:

Clicking this give you:

You can also select multiple ports and make changes at a bulk level:

You can also see a bit more info about each port:

Screenshot 2022-04-25 at 22.35.59

Teleport VPN is also now added. This makes giving someone access to your network a LOT easier than usual. They will need the WifiMan software on Android, iOS or Mac to join. Not sure what happens on a Windows machine… Maybe it’s coming soon? To use it, just generate a new link and send it to your user. Not sure how to remove them afterwards (if you want to give them temp access for example…)

Final Interesting part, and something I have been waiting for for a while, under Traffic Management, you can now create custom traffic rules:

You can set it based on destination Domain Name, IP or even the full internet:

And you can set the Source to be All Devices, group of devices (network) or individual (or multiple targeted) devices.

Finally, you can set the output internet connection.

If you had multiple internet connections, and one had better speeds for stuff like Netflix, or you wanted to send bulk data over a different link, you can do this using this feature. Very cool stuff.

So, still testing, but looking good so far.

Raspberry Pi in a car, part 2

For the last few weeks, I have been running a Raspberry Pi in my car, along with a small UPS and a Wifi Access point, allowing me to download videos from my dash cam and back them up to my NAS in the house. But I have had some teething issues, and I am currently thinking my way through some fixes…

  • First, the Pi is connected to both the network in the car (via ethernet) and network in the house (via Wifi). It seems that when the car is parked outside, sometimes the Pi can’t talk to the internet, and sometimes it can’t talk to the dashcam… It’s a routing issue, and it’s starting to annoy me…
  • I thought the onboard Wifi on the Pi was a little weak… it wasn’t getting much more than about 2-3Mbytes/s (16-24MBit/s) when downloading from the Pi to the House. Given the Pi was serving content from an SSD (not the internal MicroSD) I would have hoped for faster. I tried swapping in an external Wifi dongle with an aerial, but the same kind of speed… must be having issues getting through the metal and glass in the car, plus the metal, glass and brick in the house…
  • I started running out of disk space on the SSD on the Pi after about 3 or 4 weeks of video… so, I needed to tweak the command for the download script to only keep 14 days on the pi. Resilio Sync, the app I use to sync back to the house, has a “keep deleted files in an archive” folder option, so when the pi does delete the files, they are still stored on the Pi… I would like to find a way of automating that…

While trying to figure out how to fix part 1, I came up with an idea: I have an older Mikrotik RB951G that can be powered via a 12v adapter for the car. I am going to use that, along with a Huawei 4G dongle to act as an internet connection. The onboard Wifi will be in client mode, so when it’s near the house, it will connect to the main network and send traffic through that to the internet (or internal NAS) and when away, use the LTE modem. Then, using the Wifi dongle on the Raspberry Pi, use that as a Wifi AP.

Anything in the car that needs Wifi will connect to the Pi, which will act as a bridge to the Mikrotik. When the script needs to download files from the dashcam, it should have a direct connection to it, plus (hopefully) will be faster… then the Pi is connected to the internet through the Mikrotik. The Pi has both Tailscale and Zerotier on it for remote management, and the Mikrotik can be configured to use Wireguard to connect back to the house directly if required.

I have some of this working on a bench in the house, but it will be a while before I manage to get this running fully… Hopefully, I will have some more stuff sorted this weekend…

Running a Raspberry Pi in a car and backing up dashcam footage

A few months back (well, November 2020) I wrote about connecting to my car with Zerotier. In this post, I mentioned using a TP-Link router running OpenWRT and a Huawei LTE dongle to connect to the internet, which allowed me to then connect to my Blackvue Dashcam and watch remotely… But it had some issues I wanted to fix:

  • The Huawei Wingle was a little slower on 4G than I would have hoped…
  • When the power in the car went out, everything stopped working immediately (12V sockets in the car run for about 20 min after the engine shut off)
  • It did not connect to the WiFi in the house when parked
  • No option for backing up Video…

So, I went digging to find some alternatives… and I realized I had a load of them floating around the house: the Raspberry Pi. Specifically, the 4GB Pi 4. I got my hands on a Pi UPS Hat, a couple of 18650 Cells and an SSD Expansion board with a 512GB (overkill I know) SSD. I also got a BlackVue Power Magic Battery, B112, which will power the Dash Cam (a BlackVue DR750S-2CH). It has 2 USB ports, which allows me to run both the Pi and the new WIFI router, a Netgear Nighthawk M1.

When the car starts, it powers, via the 12V socket in the boot of the car (trunk for my American friends), which powers the Blackvue Battery. Cables run from there to the front of the car where the front camera is. (there is also a rear-facing camera in the boot too… more cables!) This then also starts the Pi and starts charging the 2 18650 batteries. Finally, well, at the same time really, the Nighthawk starts running too. Because the batter on this was running hot, the battery is removed from this.

The Pi is hooked to the Nighthawk via ethernet and the WIFI is set to connect to the house when it sees it. The BlackVue uses the WIFI from the Nighthawk for its internet requirements. When the pi boots, it connects to Zerotier for management via SSH or VNC (I use VNC to remote into the box and watch the live video when the car is parked or when someone else is driving).

There is also a python script that is scheduled to run every 15 min that downloads the videos from the Dashcam. It also downloads any GPS and other info. The folder these files are downloaded to is on the SSD and is shared with my machine at home via Resilio Sync. To make sure I don’t use all my LTE usage, the machine at home is set to only download what I want to download. So, if the car is somewhere else, I can download specific files when I want, or when at home, I can download full days, if required.

It’s been running for a few weeks now, and so far, so good. I haven’t had to do any clean up of the SSD, yet, but I would guess that eventually, I will need to look into that… With the 4G connection and Zerotier, I can then connect to my car and watch the live video whenever it is online, and whenever it is driving, within 15 min it will start downloading videos. I could, in theory, do a LOT more with the Pi in the car… Some ideas that come to mind:

  • Turn WIFI off on the Nighthawk and use the Pi as a Router, probably adding a second WIFI adapter to get better range… This could then have PiHole running on it for monitoring DNS traffic…
  • Since I have access to the GPS files in (somewhat) real-time, use it to map the car in somewhat real-time. Though, I do this already using Ruhavik and a Teltonika FMC-001.
  • Connecting to the car’s OBDII port (On-Board Diagnostics) and getting data from the car… Technically, again, the FMC001 does most of this, but in theory, it could be replaced with something else…

Keep an eye on the blog for future possible projects with this… Not sure where this project will get me, but we will figure it out at some stage… Leave a comment if you have questions!

Ubiquiti UDM Pro Fail over to Speedify

So, this has been a blog post in the making for a while now but never got around to fully writing it up, so here goes nothing…

I run a UDM Pro in the house. It has 2 WAN Links: 1 1Gb link and 1 10Gb Link. I also run AS204994, my own ASN with its own Transit and Peering connections, mostly in Europe. There is a VM in the house which acts as a connection to AS204994, which gives me a full connection to the Internet through my own ASN. More details on my AS204994 blog are here.

That connection is hooked up to the 10Gb Link on the UDM Pro, which is listed as the primary internet link. Details on how these works were uploaded in this video on YouTube:

In the video above, I was using OpenMPTCPRouter to connect to the internet, but it’s been causing some issues lately, I decided to try something else.

The new setup is an Intel Nuc (i3 with 32GB RAM and 2x512GB SSDs… VERY OVERKILL for the job at hand) running Ubuntu Linux. It has a USB Hub with 3 USB Ports and an Ethernet port connected, giving me 2 Ethernet ports on the box in total. 2 of the USB Ports are connected to USB 4G Modems from Huawei and the external ethernet port is directly connected to my cable modem.

USB Hub with 1 Huawei Modem and connection to second

Both modems and the ethernet port are connected to the NUC with full internet connections (The Huawei boxes give up NATed IPs, but the Cable modem is a full public IP) and then Speedify takes those 3 connections and does some bonding magic. Speedify is a handy little VPN service that does connection bonding. You can use it to make sure your internet is rock solid using multiple links, make sure streams are stable, etc. It can bond Wifi Links, LTE modems, Cable Modems, DSL, etc. Anything that can connect and be bonded. The only issue I have with it, compared to OpenMPTCPRouter is that you don’t control the upstream server…

Speedify is set in shared mode, so the internal port on the NUC is set to share the internet connection. This is hooked to the 1Gb WAN Port on the UDM Pro. This is set for failover only (currently the only option on a UDM Pro) so if my AS204994 link goes down (VM reboots, VM host dies, Cable modem connection goes out, etc) I will still have a connection. If the cable goes out, it will use just the 4G links, but if everything is running, I get all 3 connections.

Connecting to my car over ZeroTier

I use ZeroTier on my network for a good few things, including internal network peering between BGP VMs, management of machines, and now, connecting to my car over LTE. This is one of those posts that sounds silly, but is very handy! First, the parts list:

  • Car…
  • 3G/4G/5G modem of some sort. I am using a Huawei Wingle… Can be used without the Router below, but I wanted Zerotier, so I have it in modem only mode…
  • A router that supports Zerotier. I am using a modified TP-Link TL-WR703N upgraded to 16MB ROM and 64MB RAM. This is required for newer OpenWRT builds
  • a dashcam that connects over Wifi. I am using a BlackVue DR750S-2CH
  • Latest ROOter software from Of Modems and Men
  • Patients…

After installing the the latest copy of ROOter on the TPLink (or router of your choice) and getting the modem configured correctly (this took a while) you need to install the Zerotier software though the dashboard. Once installed, I joined my Zerotier network using the CLI (SSH into the router) and the approved it though the my.zerotier.com dashboard. Once its approved and connected, you can now go to the Zerotier IP and get to the router directly. From here, you can either setup a route in Zerotier to point at the internal network behind the router, or, in my case, setup a  SSH tunnel to the dashcam. I found the IP given to the dashcam and used SSH forwarding to get to it. Finally, i used the URLs from Digital-Nebula’s hackview repo to get to the different URLs. I use this to download stuff like GPS logs, emergency videos, etc. I have to clean up some scripts at some stage for this, and plan to upload them at some stage.

If anyone has any questions, leave a comment!

AS204994, Own IP Space and Anycast

So, if you are reading this page, it is being delivered with the magic of Anycast… Well, technically, it was before, since i used Cloudflare, and it still is because of Cloudflare, but also because of my own ASN (As204994), some servers in different locations, and some magic, which i will explain a bit of in this post.

This all started late last year when i got my hands on an ASN and a /48 block of IPv6 addresses. I had been reading stuff about BGP, routing, etc, and decided to go all in. it was quite cheap with the help of HostUS. All in, it was about $50 for the year. As part of the process, i needed 2 upstream providers to say they would accept my announcement. They were Hurricane Electric though their Tunnel Broker service, and Vultr using a few of their VPSs.

After i got my space and ASN, i started to announce the V6 addresses over Vultr and Hurricane Electric, and all was good. I had 2 Vultr servers: 1 in London, UK, and one in New Jersey, USA. I had my home machine announce to HE, and then also link to both Vultr servers using Zerotier. All worked well, but due to some family issues, i never got around to putting it into production… till now.

Those 3 servers now share an IPv6 address on the loop-back port. When you (well, Cloudflare) asks for that IP, the closet (network) with that IP responds, and the NGinx server on that box sends back the contents of the site. This site is hosted on each box, since its fully static, but both AS204994 and TiernanOToole.net are hosted in Ghost, so Dublin (my machine in the house) serves them, and both Lon1 and Nyc1 do proxying. so, most requests from the US are hitting the box in NYC and the ones in Europe share either Dub1 or Lon1. I have some tweaks to do with which servers will be running where, and may add more, but currently its working well.

So, how do you figure out what server responded? Simple. Open your Dev tools on your browser, go to network tab, refresh, and see the response headers for anything on this domain. You should see something like below.

Over the next while, i will be updating tiernanotoole.net with more details on how this works, and more stuff will end up on AS204994.net too. If anyone notices any weird and wonderful issues, shout. If you have more questions, shout.

Business Class Broadband… finally here….

So, after many (MANY) years messing with dual cable modems, struggling to get them working together, to get websites to even allow me in, having to use hacks and kluges to get it to work at all… I have given up. It has been a struggle getting two modems working properly. Load blanching kind of works… but it’s messy at best. Some sites kick you out every now and again because your IP changes. Some sites wont let you login at all… Mind you, some sites work grand and don’t ask questions…

And the whole idea of multiple modems, to allow you to download things faster, doesn’t work for everything… Anything you download in the browser is single threaded, so its limited to one modem… you can use use download accelerators, and they do work, but its an extra step, and some sites don’t work for that either (MSDN for example).

So, i have given up, bit the bullet, and moved to business class broadband from Virgin Media. It’s actually cheaper than the two residential lines i had, but it is also slower than the two combined: previously, it was two x 360/36mb/s. Now i am am on a single 400/40mb/s modem. That being said, there are definite advantages:

  • Static IPs pretty much as standard, and option of either one or five (no unbeaten!) Guess which one i went with? Its technically a /29 range, but the first usable IP is given to the modem, which acts as a gateway, so i end up with five usable.
  • Proper business class SLA. Any issues, someone who knows what they are talking about can help
  • Phone lines on a separate modem. So, i got phone lines with them, and they give a separate modem for those lines, so as not to interfere with the internet. that modem has no internet connection and is just for calls. They are also working on a VoIP/SIP offering, which is something i am interested in.
  • Guaranteed speed! They guarantee a minimum speed to the modem at all time. Business customers have a priority on the network, which is nice. And, during testing, so far, i am getting the advertised speed most of the time. I needed to download a Windows 10 ISO yesterday from the MSDN, and it came in at between 45 and 48MBytes/s!

So, only had it installed a week, and so far, so good. I have one IP given to my PFSesne box, and the rest given to a VyOS VM. The plan is to use the VyOS box for all network traffic, but first i need to do some testing and learning… Expect some posts on this soon!

double speed Internet Part 8 – Routing Around

[NOTE] This part 8 in a series of posts. The rest can be found here.

At the end of my last post I asked the question about routing traffic to different servers based on thier distances, etc… Well, after a bit of messing, i can say it kind of works! here is a quick over view:

  • server in the house has now got multiple OpenVPN connections (2 to Hetzner, 1 to OVH (with a plan to double), 1 to Digital Ocean (again, to be doubled) and i am planning 2 to Azure as well).
  • Quagga/Zebra has static routes (currently static, planing on dynamic soon… more eventually) to different servers depending on where they are. for example, all traffic to the hetzner network (including their Storage Boxes) go though the hetzner link. Hubic traffic goes though OVH, Azure (currently) and AWS traffic, aswell as some CDNs go direct over either WAN1 or WAN2 in the house, and some other stuff (CrashPlan currently) goes though Digital Ocean. Everything that has no static route goes though Hetzner…
  • Ideally, the static side of things should be removed, and a more dynamic setup done. How that works, i have no idea… Spotify have 2 posts about their SDN Internet Router (part 1 and part 2) which is an interesting idea… More digging and research is required.

So, there you have it. Everything currently seems to be working, mostly, and tweaks can be made easily… I have a couple posts i have in my head, including something to do with automating bringing up new machines (probably with Ansible or something like it), more monitoring, and some other stuff too… Any questions, leave a comment, and i will get back.

[UPDATE] I wrote a quick and dirty app called WhoIsToZebraConfig which takes an AS Number, looks up the info in the Merit RADb (with the help of some code from Coder Buddy) and outputs what you need to put into your Zebra Config… should save me some time, and it might save you time too… shout if you have questions!

double speed Internet Part 7 – ECMP (kind of)

[NOTE] This part 7 in a series of posts. The rest can be found here.

In the last post I mentioned I am now using Hetzner for hosting a dedicated box. Thats still live, and going well. I have a /29 IP range (6 usable) and also 2 other IPs. So far, so good… But because i was using a Socks Server, I was not fully able to use the /29 ips… I use something like as follows:

essentially, for each public IP i have that i want to map to an internal IP, i have a POST and PRE ROUTING rule, plus the required forward rules… But, if socks are used, then that goes out the Window, since TCP traffic will look like its coming from the socks server… So, i killed the socks server, removed the IPTables rule, and then realized that while outgoing traffic was being balanced somewhat (2 default rules on the internal box pointing at the OpenVPN IPs from the Hetzner box) incoming was a problem. Hetzner knew how to get to my internal network, but only though one ip… enter Quagga and Zebra…

Quagga is a routing software suite, which can do protocols like OSPF, BGP and RIP, and Zebra is the component that does static routing. using their documentation on static routes, I created a static route to my internal network with 2 next hops, the OVPN IPs from the internal box… and, after restarting Quagga, all works! happy days! now i can forward ips from outside the network to inside the network correctly, and they look like they are the public ip!

So, whats next then? well, I now have a server in Germany (Hetzner) and one in France (OVH), and can spin one up in the UK or the US (Digital Ocean). Given that i have Quagga running on the box, i am now thinking of trying to see if its possible to route traffic depending on distance or something similar… If i am trying to hit a server in Hetzner’s DC, i should go though Germany. If its in Digital Ocean, go though either US or UK servers, same with OVH. Then figure out who has the fastest links to, say, Amazon, Azure, Netflix, BBC, Dropbox, etc, and add either static or dynamic links to the router… essentially, thats the theory… lets see how that works…

double speed Internet Part 6 – Hetzner Edition

[NOTE] This part 6 in a series of posts. The rest can be found here.

Its been a while, since I posted, and there are some, well, pretty major changes since the last time… Lets start are the beginning.

Last time I was using Digital Ocean for my hosting provider. I was using their $20 a month server (2 cores, 2GB RAM, 40GB SSD, 3TB transfer), and it was all good… But I noticed that every now and again I would need to reboot the box. I also noticed that when transferring large files or using higher bandwidth (400mb/s+) the 100% of both cores were being used. So, I wanted to move to something with more power…

I also was limited to IP addresses. Yes, Digital Ocean do offer IPv6, but I could only get 1 IPv4 address… and I wanted more…

So, I went back to some old friends of mine, Hetzner, and bought a dedicated server with a Quad Core Xeon E3, 32GB RAM, 4*3 TB HDDs, 1Gbit/s network connection with 30TB transfer per month and a KVMoIP plugged in. I also got 2 extra standard IPs and a /29 (8 IPs, 6 usable). I will explain that next. I installed Debian 8.4 on the first disk, and I am planning on using the other 3 for storage of some sort. I then installed the MPTCP kernel, OpenVPN, Squid and a Socks server (same as the Digital Ocean box) and reconfigured the home machine… All good! Now when browsing the web, everywhere thinks I am in Germany, but so far, so good… Speed tests are about the same, but I have my theory about ECMP to try this weekend.

Because of the extra IPs, I am working on doing full IP forwarding, not just port forwarding. One of my IPs in pointing directly at my Meraki MX64 in house (a post on the Meraki stuff is coming, eventually…) and another at the Proliant box, and I plan on pointing other IPs at machines in the DMZ or a firewall of some sort. the /29 is routed though the IP pointing at the Proliant, so that makes life easier. The original IP is only used for SSH and OVPN from the house. it should not do much else. All network traffic in house is coming from other of the other IPs.

Again, so far, so good. Hopefully the ECMP stuff works correctly, so I will do an updated post soon.