Tiernan's Comms Closet

Geek, Programmer, Photographer, network egineer…

Ubiquiti UDM Pro Fail over to Speedify

So, this has been a blog post in the making for a while now, but never got around to fully writing it up, so here goes nothing…

I run a UDM Pro in the house. It has 2 WAN Links: 1 1Gb link and 1 10Gb Link. I also run AS204994, my own ASN with its own Transit and Peering connections, mostly in Europe. There is a VM in the house which acts as a connection to AS204994, which gives me a full connection to the Internet though my own ASN. More details on my AS204994 blog here.

That connection is hooked up to the 10Gb Link on the UDM Pro, which is listed as the primary internet link. Details on how this works was uploaded in this video on YouTube:

In the video above, I was using OpenMPTCPRouter to connect to the internet, but its been causing some issues latey, i decided to try something else.

The new setup is an Intel Nuc (i3 with 32GB RAM and 2x512GB SSDs… VERY OVERKILL for the job at hand) running Ubuntu Linux. It has a USB Hub with 3 USB Ports and an Ethernet port connected, giving me 2 Ethernet ports on the box in total. 2 of the USB Ports are connected to USB 4G Modems from Huawei and the external ethernet port is directly connected to my cable modem.

USB Hub with 1 Huawei Modem and connection to second

Both modems and the ethernet port are connected to the NUC with full internet connections (The Huawei boxes give up NATed IPs, but the Cable modem is a full public IP) and then Speedify takes those 3 connections and does some bonding magic. Speedify is a handy little VPN service that does connection bonding. You can use it to make sure your internet is rock solid using multiple links, make sure streams are stable, etc. It can bond Wifi Links, LTE modems, Cable Modems, DSL, etc. Anything that can connect and be bonded. Only issue i have with it, compaired to OpenMPTCPRouter is that you dont control the upstream server…

Speedify is set in shared mode, so the internal port on the NUC is set to share the internet connect. This is hooked to the 1Gb WAN Port on the UDM Pro. This is set for fail over only (currently the only option on a UDM Pro) so if my AS204994 link goes down (VM reboots, VM host dies, Cable modem connection goes out, etc) i will still have a connection. If the cable goes out, it will use just the 4G links, but if everything is running, i get all 3 connections.

Connecting to my car over ZeroTier

I use ZeroTier on my network for a good few things, including internal network peering between BGP VMs, management of machines, and now, connecting to my car over LTE. This is one of those posts that sounds silly, but is very handy! First, the parts list:

  • Car…
  • 3G/4G/5G modem of some sort. I am using a Huawei Wingle… Can be used without the Router below, but I wanted Zerotier, so I have it in modem only mode…
  • A router that supports Zerotier. I am using a modified TP-Link TL-WR703N upgraded to 16MB ROM and 64MB RAM. This is required for newer OpenWRT builds
  • a dashcam that connects over Wifi. I am using a BlackVue DR750S-2CH
  • Latest ROOter software from Of Modems and Men
  • Patients…

After installing the the latest copy of ROOter on the TPLink (or router of your choice) and getting the modem configured correctly (this took a while) you need to install the Zerotier software though the dashboard. Once installed, I joined my Zerotier network using the CLI (SSH into the router) and the approved it though the my.zerotier.com dashboard. Once its approved and connected, you can now go to the Zerotier IP and get to the router directly. From here, you can either setup a route in Zerotier to point at the internal network behind the router, or, in my case, setup a  SSH tunnel to the dashcam. I found the IP given to the dashcam and used SSH forwarding to get to it. Finally, i used the URLs from Digital-Nebula’s hackview repo to get to the different URLs. I use this to download stuff like GPS logs, emergency videos, etc. I have to clean up some scripts at some stage for this, and plan to upload them at some stage.

If anyone has any questions, leave a comment!

Backups, Backups, Backups!

I have posted about backups a few times on this site in recent years, and its still something I make tweaks to every now and again. The latest setup is probably over the top, but I will give you a walk though on it and some of it could be useful to some of you.

I have a couple of different machines and storage devices running that need backups. Some need daily backups, some could get away with weekly. The list is as follows:

  • GodBoxV1 (2X4 Core Xeon, 82GB RAM, Fedora, 512GB Boot SSD, 5x4TB HDD in ZFS RAIDZ1)
  • GodBoxV3 (2×20 Core Xeon, 192GB RAM, Ubuntu, 2x512GB NVMe SSD in RAID 0 for booth, 4X512GB NVMe SSD ZFS stripe for FAST storage, 8x8TB HDD in ZFS RAIDZ2 for bulk storage)
  • Docker Box (VM, runs a LOT of different containers on the network)
  • Synology DS1817+ (8x8TB HDD in SMR with 48TB usable, 2x10GB + 4x1Gb NICs)
  • QNAP TS-932X (5X8TB HDD in RAID 6 along with 4X512GB SSDs in RAID 5, 2X10Gb NICs)

2020-11-04_10-15-00_IMG_2342

GodBoxV2 and the 4 C6100 boxes are running Widows Server 2019, and I have 4 new C6220s which, when in production, may be either running Server 2019 or VMWare ESXi. More on this in a future post. GodBoxV1 and V3 are being backed up with Borg/Borgmatic, and the Server2019 boxes are running Hyper-V and the VMs are not backed up on a nightly basis, but that is planned in the future…

Borgmatic is basically a very nice and handy wrapper for Borg itself. It allows you to easily configure a YAML file with what you want to backup, what you want to exclude, where you want it backed up to (multiple locations if required) and details on retention, etc. It also allows you to send notices when something completes or fails. I have 3 main machines which are backed up using Borgmatic, but will probably add more at some stage. These three backup to 3 different locations; Local ZFS Storage in house (currently on GodBoxV1), RSync.NET and Hetzner’s Storage Box. [Note: Hetzner have 2 types of storage: Storage Box and Storage Share. Storage Share seems to be NextCloud and does not have BorgBackup installed. Storage Box can be used with BorgBackup though]

[Note: RSync.net have an offer for Borg Storage: 1.5c per Gig. So, 100Gb a year costs only $18. On their signup page, if you enter referral code 2019-09-13_05-27-04, I get some extra storage for backups on my end, and you can help me continue writing random stuff here!]

Nightly, Borgmatic runs and backs up everything important on GodBoxV1, V3 and the Docker Box, to all three locations. Then, on GodBoxV3, we backup some larger files (photos, video and other large data from my cameras) to Hetzner. I also plan on setting up a backup of those larger files to either my Synology or QNAP boxes. The reason the large files are only backed up to one current location is size; they currently weigh in at around 300GB, give or take, and I currently have around 200Gb of usable space with RSync.NET. My plan is to use the QNAP or Synology box as a secondary backup for this storage at some stage.

On a nightly basis, the Synology runs backups to both Backblaze B2, Wasabi and Hetzner using Hyper-Backup. Finally, on a weekly basis, some folders on the Synology are backed up to AWS Glacier.

This gives me a fairly good set of backup options, but there are some tweaks I want to make:

  • Important VMs on the Hyper-V Cluster should be backed up. Daily backup to local storage (QNAP, Synology, ZFS) and one weekly backup external (Hetzner, B2, RSync.net)
  • Large media files backed up to a second location, either local or remote.
  • Intel Nuc, Home Laptop and Mac Mini should also be backed up. 99% of the time they use storage from the ZFS pool or the NAS devices, but they still have local storage.
  • Look into backing up iPhones, Android Phones, iPads, etc, to local storage also. I do use PhotoSync to copy photos from my iPhone to the ZFS storage, which is backed up, but having something to backup the rest of the data, other than iCloud, would be handy.

So, thats my 2020 backup plan. Any comments, questions, etc, shout in the comments section.

Domain Joining a machine over VPN and Password Resets/Changes with Azure AD

With the whole Work From Home thing probably becoming more and more normal in the years to come (I can count on 2 hands how many times I have physically been in my main office in the last 7 months) there are a couple of certainties in that people will come up against. One is passwords expiring and needing to be changed, one is password resets being required and finally laptops or desktops needing to be domain joined or connected to the domain before they can be fully provisioned. As the (currently only) IT guy in our office, I have had to deal with these first hand, and decide to write this post, helping both my fellow employees, and possibly other IT Admins stuck in this challenge.

So, as the IT person, there are a couple of assumptions:

  • You have on premises AD
  • You have Azure AD (P1 and above seems to be required if users are mixed AD and on prem. Free allows just Cloud users).
  • Azure AD Sync installed and enabled

If all above are set, you will need to follow the steps to Enable Azure Active Directory Self Service Password Reset. I have enabled this on our domain. Next, you need to get your users to setup their secondary authentication for backup. All our users have a 2FA requirement, so most of them had that already. New users need to go though those setups. Finally, if a user needs to change or reset their password, they can do so though https://aka.ms/sspr. If all is done well, that reduces the amount of support calls I (and you) get.

Now, the next task: domain joining over VPN. This is a bit more “fun” to play with.

First, you need a VPN connection. We use Meraki gear using Active Directory for RADIUS auth. I wont go into too much details on setting that part up, but the script we use to build the VPN connections for users is below. This will probably be different for different VPNs, but this is our starting point.

Lines you need to change are at 8, 9, 10 and 47. Line 39 can also be modified to change from Split Tunneling (only sending traffic to internal subnets) or full Tunneling (all traffic over VPN). If you have multiple internal subnets, Line 49 can be copied with more.

The most important part we need though is line 34. The -AllUserConnection allows the connection to be available to all users on the machine, but also on the start screen. This is important.

So, with all that in place, you will need to connect to the VPN

you should now be able to join the domain as if you where on your local network.

Enter Domain details and change name of machine if required
when asked enter your domain username and password
You will be welcomed to the domain
and then asked to reboot

reboot your machine as usual and when it boots, you should see a new option on the login screen

VPN login option

Click this icon and if you only have one VPN connection the screen below will show up. If you have more than one, you will be given a list of options to use.

Login to VPN at the login screen

Enter your domain credentials. Since our AD and VPN use the same credentials, it will automatically log you in aswell.

Machine is now domain joined and logged in, and in my case, finishing setup

So, there you have it. How to domain join a machine outside the network. Now, in reality, Azure Active Directory and Intune would probably be the better option, but that’s future work…

Apple event October 2020

[NOTE] This post was done entirely on iPhone XS Max and a iPad Pro. Photos taken on the iPhone. Some edited on iPhone, some on the iPad. I have edited some text on the iPad with the keyboard, but if i missed anything, all was written mostly live, so apologies… Will add extra links to places like Engadget, etc, below.

Homepod mini. $99 available 16 November. The feature of intercom sounds good… When they mentioned the list of extra service, Spotify was very missing… [NOTE] I missed some stuff on this cause I was in a late meeting… This does look cool though.

iPhones. 5g available. 5g ultra wide band. 4gb down and 250mbs down ideal conditions. MmWave Support. Low latency support. But that’s normal for 5g. Verizon expanding their network to 60 cities by year end for ultra wide and and all cities for normal 5g. And it’s avail be on ALL models. Not just the high end. Very handy. Rumours had suggested it would be limited to either high end, or that mmWave would be available only on pro.

IPhone 12. First one announced. 5g support. New design. Looks very iphone 4 like. Bigger camera bump with 2 cameras. 6.1inch display. Smaller border. Super Renta XDR display. 2 million to 1 contrast ratio… 460ppi. Dolby vision hdr10 and hgl support too. 1200 nits. Ceramic shield on the screen to increase toughness. Tougher than any smartphone scree.

Most 5g bands in any smartphone. Even iOS core is modified to make 5g faster. When lower speeds will do, it can drop to LTE. Has been tested and gets up to 3.5Gb/s max and best conditions. 4Gb/s down on mmWave and best conditions and 1Gb/s in normal conditions.

A14 bionic. 5nm process. 11.8 billion transistors. 6 cores. 4 core gpu. Neural engine goes from 8 to 1y cores and 11 trillion operations per second.

Gaming stuff. Something called league of legends. I’m not a gamer, so… Hmm…. [I took this time to try upload photos for this post…]

Camera looks very cool. Larger aperture for better low light photos. video looks cool too…

MagSafe for iPhone. Qi charging with magets. 15w charger. NFC support too… New cases and wallet. And charger has magnet. Apple has a duo charger for both iPhone and Watch. Belkin have a car dock and a multi device charger too. I like the sound of the car dock, and a duo charger for iPhone and Apple Watch could be useful…

Recycling stuff. Lots of important stuff here… But very big words for trying to type live. But they are removing chargers and headphones from the box. Smaller box, which means they can get more on a shiping pallet, which reduces CO2. And by removing the headphones and charger, they can save 2 million metric tones of CO2 or 450k cars off the road. USB C to lightning cable included in the box.

Iphone 12 mini. Same spec as the full 12, just smaller.

12 mini starts at $699. 12 non mini is $799. More details of availability later in this post.

“There is simply nothing like iPhone 12”… Think that’s about to change now…

Pro line. They… Multiple… 12 pro. Still reminds me of the 4…

Pro camera also looks very cool. 12 pro max has better camera.

Pro raw option. Raw with some processing. Available later in the year. Works on all 4 cameras. Flexibility of raw with apples computational photography. Edit photos in photos app or in other professional apps. Wonder when light room gets it.

Pro video. Hdr shooting. Dolby vision Hdr recording in camera too. And the internet just went missing… Give me a sec…

Shoots the Hdr video at 4k 60fps. And it can be edited on the phone… Nice.

Lidar scanner. Interesting for ar objects but could be interesting. It was in the iPad pro. It can see in the dark too… 6x faster auto focus.

To finish up, a quick Gallery of the photos taken.

ESXi on Arm (and Raspberry Pi!)

A few days back (October 6th 2020) VMWare announced a new “Fling”: ESXi Arm Edition. Not completely sure what a Fling is, but anyway, I started reading, liked the idea and managed to download a copy for testing. I have 2 Pi 4s in the house, both 4Gb Models, and I wanted to play around with the new tech.

So, after some messing with UEFI stuff, formatting Micro SD cards correctly, copying files and some limitations, I managed to get 2 new ESXi servers running on Raspberry Pi!

There is a walk though Video showing everything I did to get up and running. Its embedded below. Some of the hardware I used is also mentioned below.

Equipment list:

  • 2 x 4G Raspberry Pi 4s
  • 2 x 16Gb Micro SD Cards (you could probably get away with 1Gb cards… You only need a small 256MB partition for the UEFI stuff)
  • 2 x 64GB Kingston DataTravler USB 3 Sticks (This is where ESXi is installed, plus the rest of the storage, if configured correctly, can be used for VMs).
  • 2 X POE to USB C Splitters. I used these so I can power both Pi’s though POE and can reboot them using the switch. You could use a USB Power Adapter like the Anker PowerPort 60W which would give you 6 ports to run your Raspberry Pi’s. I would probably limit it to running 4 Pi’s though, since the Pi 4 needs a bit more power…
  • Some way of installing the ISO to the Pi. I used an iodd Mini 256Gb for the task. I also did a video review of that here.
  • About an hour of your time.

As mentioned above, the USB key is used for storing ESXi when its installed. It can also be used for storing VMs. There is a command you run when installing to partition the drive in 2: 8GB for ESXi and the rest for storage. I managed to run this correctly on one, but missed it on the second. I might reinstall that Pi and get it up and running again soon. You also have the option of installing to iSCSI. That might be useful too…

Storage wise, VMWare recommend using usb3 or fast iscsi or nfs storage for vms. I’m using nfs on my workstation which seems to work OK. but you are still limited to 1Gb/s of the Raspberry Pi. They say it is possible to use extra USB network cards. Could be interesting to try that out.

So far i have managed to install a single VM on one of the Pis. I plan on migrating from a Physical PiHole instance to a virtual one. I also plan on getting a few 8Gb Pis and see where this rabbit hole gets me. It can also be managed with VSphere. Let’s see if I can get that working… Stay tuned!

If anyone has any questions, comments, etc., just shout. And if your interested in videos like these, subscribe and like the video!

Nexdock Touch Videos

A few months back, I pre ordered a Nexdock Touch. The Nexdock Touch is a laptop without the laptop components… its essentially a screen (1920×1080 touch) with a keyboard, battery, touch pad, a 3 USB C ports (one for charging, one for phones only and one for connecting other devices) a Full USB A port (for plugging in other stuff, more on that in a sec), a Micro SD Card and a full HDMI port. Interestingly, the HDMI port is not for output, like you would think it is, but for input.

This is the Nexdock’s party piece: plug in a compatible phone (I have a Samsung Galaxy A90 5G that works), Raspberry Pi (I tried with a Pi 4) or any other device that takes USB input and HDMI output (I also tried with an Intel Nuc) and that machine becomes a laptop… Well, within reason; the Phone and the Pi will both get charged or powered by the Nexdock’s built in batter, but for the Nuc, it needs to be powered externally.

I have recorded some videos and uploaded them to YouTube. There are some unboxing videos, showing you it working with Samsung Dex and the Galaxy A90 5G, a Raspberry Pi 4 and also the Intel Nuc. The full playlist is embedded below, or you can visit the playlist on Youtube here.

I am planning on releasing more Videos in the same kind of format over the next while, so, as they say “Like and Subscribe” on YouTube if your interested!

Back running WordPress

I have moved my blog back over to WordPress. It is running in house, on one of my workstations, using Cloudflare’s Argo tunnel to protect it on the internet. You might be asking “why?!” Well, its a couple of things.

  • Easier to blog and post from anywhere in the world.
  • I can blog on pretty much anything
  • No having to worry about upgrading my copy of Hugo breaking my site…

That last one is the reason I haven’t blogged in a while. Seems there was a major change in the versioning of Hugo, somewhere between the release I was on (0.55.6) and the latest one I tried (0.73.0 or something… 0.76.3 is out now) and my index.html pages just would not create, and I got many warnings when building… I spent a few hours trying to figure it out, but in the end, I gave up.

I ended up using Chris Salzman’s blob post explaining how he moved from Hugo to WordPress, spent a hour or so tweaking the imported files, built a Docker-Compose file (I will post this somewhere soon, if anyone wants it) and was off to the races. Few tweaks later, a copy of CloudflareD and some DNS tweaks, and everything was back online.

There are some disadvantages to WordPress:

  • Comment Spam
  • Performance
  • Maintenance
  • Security

But even so, I am willing to worry about these and be able to blog easier.

Fixing CID (Caller ID) on incoming calls with 3CX

In a previous post i talked about going all in on VoIP in the house. Its been nearly a year now, and other than some minor issues related to the VoIP Server being turned off accidentally, or a screw up on my end, all is going well. But, one thing i did notice was related to incoming calls and caller Id, specifically on my SIP2SIM card. Essentially, the country code was wrong: for example: Incoming calls from the Virgin Media trunk just show as local numbers (for Dublin, for example, it would so 01xxxxxxx). Using the CID reformatting feature in 3CX, I managed to change this.

All calls that come in starting with 0 are “fixed” and changed to +353 without the 0. When the call comes in though the SIP2SIM card, it does no longer show as a call from the UK, but now shows as a call in Ireland, or where it is coming from, so all the contact details show correctly! Happy days!

Network Update Info April 2019

So, this post has been a long time coming! A load of different things to talk about, so lets get started!

GodBox V3

So, for a long time, I have been thinking about GodBoxV3, the replacement to GodBoxV2. And when planning this, i had some ideas of what it should be:

  • Minimum of 2×16 cores (double godboxv2)
  • About the same RAM, if not more
  • FAST STORAGE!
  • Is able to run my twin 30" 4K monitors
  • Would like 10Gb/s NICs

Well, It finally happened! I got the machine, built it and, well, its impressive! How did i do with specs? Well…

All is good! Photos, more details and benchmarks coming soon… stay tuned!

Finally 10Gb/s Networking!

Since GodBoxV3 had a few 10Gb nics, i needed to upgrade the network to support it. I ended up with a Ubiquiti Networks EdgeSwitch-XG. 16 ports (12 SFP+ and 4 RJ45). The SubperMicro board has 2xRJ45 ports. Due to lack of RJ45 ports, GodBoxV3 is connected to 1, GodBoxV2 is getting a 10Gb card soon, which will be connected to 1 port, and a new Sun Microsystems server (details below) will be getting the last 2… Of the SFP+ ports, 2 are connected to the EdgeSwitch Lite, 2 to the Synology (it got a 10Gig NIC reciently too!) and 2 to the new NAS (again, more details below!)

Good bye Mikrotik, Hello EdgeRouter 4

Since i was going all Ubiquiti gear (Wifi is Unifi gear) i got rid of the old Microtik and replaced it with a Ubiquiti ER4. Happy days! Got some plans for this, more details coming soon…

Updates to BGP Stuff, including IPv6

I lost one VPS in London, but replaced it with a new one from HostUS. I still use Vultr, Packet and VServer.Site as providers too. I am also adding more and more IPv6 stuff too… There is a post on AS204994 explaining a lot of this.

New NAS and more storage!

New NAS got purchased: QNAP TS-932X. I have 5X8TB spinny disks (shucked from 5 WD My Book 8TBs) + 4 X 500GB WD Blue SSDs.

New Servers and cooling updates

Moved lots of stuff around the room… Servers run cooler, and less noisy! happy days! I also got my hands on a very nice looking Sun Server X3-2. Its a Dual Xeon E5 (currently got quad cores, going to upgrade it to 8 cores) and i think its got 16GB ram and 4x300GB SAS Disks. It also has 4X10Gb nics! ESXi will probably go on here!

VMWare in the house

Up till recently, I ran Hyper-V all round. Its still on GodBox V2 and V3 (v1 has a HDD issue, so its off…), but the main VM hosts (the C6100’s) are being migrated to VMWare ESXi… Why? Its a learning exercise… We see how it goes…

So, long update… Any questions, comments, etc… shout!