Tiernan's Comms Closet

Geek, Programmer, Photographer, network egineer…

Currently Viewing Posts in Homelab

Bulk updating Tasmota Devices over MQTT #100daysofhomelab

I have a load of these Smart Plugs from GoSund around the house (currently around 11, but more are still in boxes). The handy part of these is they can be re-programmed using Tuya Convert and using the following config you can get power usage and an on/off switch. I have mine hooked up to an MQTT server, and with the MQTT plug-in to Home Assistant, I get all the details about power usage and can control each device I need to (hence the 11 of them!).

But MQTT can be used for more than monitoring. I can send commands to the devices. Given that all of them are on a locked-down network, and only have access to the NTP server, internal DNS and the MQTT box, I needed to figure out how to get OTA updates to the box. Luckily, you can change the OTA update URL on the web interface and download it from a local endpoint… But, I am a Lazy Git, so I needed to figure out an automated way. Enter MQTT again.

First, you will need to log in to your Tasmota device, go to configuration, MQTT and Enter your MQTT host. Also, get your topic name while you are there and keep it handy.

Hit save and wait a few seconds for it to update. You need to watch the MQTT messages going through. I am using MQTT Explorer to see all messages.

For me, the tele topic has all my devices listed, plus stats around power, status, etc.

On the Tasmota site, they have documentation on sending commands over MQTT. I then installed the MQTT CLI on the mac (so I could automate this later) and ran the following commands:

mqtt pub -t cmnd/tasmota_<deviceid>/OtaUrl -m "<internal url hosting tasmota updates>/tamota.bin" -h <mqtt host> -p <mqtt port>

mqtt pub -t cmnd/tasmota_<deviceid>/Upgrade -m "1" -h <mqtt host> -p <mqtt port>

update <deviceid>, URLs, host and ports as required. for the internal URL, I just have a small copy of Nginx running in docker, and serving a folder with copies of the latest OTA files from the Tasmota Release page. I just wanted all the files and put them in the folder shared by NGinx. I need to automate it a bit better… maybe next time there is a new release?

the first one tells the device where the latest OTA file is. the second command kicks off the update. If your devices are not segregated, you can just leave the existing OTA Url there and kick off the upgrade task on its own… I wasn’t that brave…

Within a couple of seconds, you will start to see messages showing up in MQTT Explorer. After a couple of minutes, all devices will have been upgraded and rebooted (no power down, luckily) and all is good!

DNSControl and Github Actions #100daysofhomelab

I am participating in the #100daysofhomelab challenge and have been posting a lot on Twitter as @tiernano, but some posts and tasks I am doing will require longer-form write-ups. So, some updates will include either Videos (which will be published on my Youtube Channel) or blog posts, which will go here. This is the first of the blob posts.

DNSControl is a tool written by the Stackoverflow lads (when they called themselves StackExchange). It is designed to update DNS records and can work with DNS providers and registrars. I use it to update records in Cloudflare and Route53, but many providers are available. I wrote an article a while back about how to create reverse DNS records for IP space with Route53 and DNSControl, but most of it is still relevant, and the main documentation site for DNSControl has a lot of useful tips.

Up till this morning, if I wanted to update a record, I checked out the DNS records from my private Github repo, made the change, and ran the DNSControl commands on my machine (check for syntax checking the file, preview to show what will change at the provider level, and push to make the changes). But I wanted to have some automation for this. So, enter Github Actions.

I did a bit of digging and found a Github Action from koenrh called dnscontrol-action. The docs on this are quite simple to go through, so I created 2 action files for my Repo: preview and push. a Gist for Preview is below:

and the one for push is as follows:

The important parts are as follows:

In both preview and push, the check command does a syntax check of your DNS config file. Then preview will check the providers to see if any records need an update. When push runs, it will make the changes.

All my required secrets are set in the Github repo as secrets, so when the action is run, it will pull the required keys out. These are put into the environment variables. I use name.com and a registrar for some domains (though most have now moved to Cloudflare, and some, like my .ie domains, are with Blacknight, who are not supported on DNSControl). Cloudflare is used by the majority of my domains, and Route53 is used for 2 domains currently. There are around 53 domains current managed by this, and the plan is to add more. I also plan on getting some more automation around checking configs and sending alerts if anything changes.

So, enough “How it works” and show us it working!

Right. Let’s update my zt.tiernanotoole.net domain, which is used for Zerotier IPs internal to my network. It’s been a while since I did this, so most will be removed and a few adds… first, I create a new branch, called zt-update, and check it out in VSCode. I made my changes, git committed and git pushed to the branch.

at this stage, the actions have NOT run, since this is neither checked in to master, nor a PR for master.

I go into the create PR section, and I can see the changes I have made. in my case, I removed a load of unused records and added extras:

I now create my PR and wait for the checks to complete:

within a short time, I get an alert that all checks have passed, and I can find the results of the changes in the build (It was meant to add a note to the PR with the details, but I might be missing something in my config…)

Also, not sure why it is redacting part of my name here…

I check the rest of the list, and other than the deletes and creates in route53 for this domain, there are no other changes. So, being happy with that, I click the Merge Pull Request and the code is checked into master, and the DNSControl push command runs:

If i now go into Route53, i can see the records on the site:

Happy days! Next challenges to fix:

  • fix the PR to include the output of check and preview
  • only run a check and push on the master branch, and no need to run preview again…
  • run preview once a week and send alerts if anything has changed

Till next time, good luck!

Unifi Network Update 7.1.61

A few weeks back, Ubiquiti released a pre-release update for the Unifi Network Controller, version 7.1.61. It got installed on my UDM and I noticed a few interesting bits that you might find handy… First, you will need to be signed up for Unifi Early Access before you can download or even read the release notes, but this is just a quick update based on my findings so far.

The first thing to note: You can see the list of devices connected to switches on the Overview Tab. I can’t remember exactly when that was added, but I think it’s new…

Under the ports tab, you now have a ports insight option:

Clicking this give you:

You can also select multiple ports and make changes at a bulk level:

You can also see a bit more info about each port:

Teleport VPN is also now added. This makes giving someone access to your network a LOT easier than usual. They will need the WifiMan software on Android, iOS or Mac to join. Not sure what happens on a Windows machine… Maybe it’s coming soon? To use it, just generate a new link and send it to your user. Not sure how to remove them afterwards (if you want to give them temp access for example…)

Final Interesting part, and something I have been waiting for for a while, under Traffic Management, you can now create custom traffic rules:

You can set it based on destination Domain Name, IP or even the full internet:

And you can set the Source to be All Devices, group of devices (network) or individual (or multiple targeted) devices.

Finally, you can set the output internet connection.

If you had multiple internet connections, and one had better speeds for stuff like Netflix, or you wanted to send bulk data over a different link, you can do this using this feature. Very cool stuff.

So, still testing, but looking good so far.

Raspberry Pi in a car, part 2

For the last few weeks, I have been running a Raspberry Pi in my car, along with a small UPS and a Wifi Access point, allowing me to download videos from my dash cam and back them up to my NAS in the house. But I have had some teething issues, and I am currently thinking my way through some fixes…

  • First, the Pi is connected to both the network in the car (via ethernet) and network in the house (via Wifi). It seems that when the car is parked outside, sometimes the Pi can’t talk to the internet, and sometimes it can’t talk to the dashcam… It’s a routing issue, and it’s starting to annoy me…
  • I thought the onboard Wifi on the Pi was a little weak… it wasn’t getting much more than about 2-3Mbytes/s (16-24MBit/s) when downloading from the Pi to the House. Given the Pi was serving content from an SSD (not the internal MicroSD) I would have hoped for faster. I tried swapping in an external Wifi dongle with an aerial, but the same kind of speed… must be having issues getting through the metal and glass in the car, plus the metal, glass and brick in the house…
  • I started running out of disk space on the SSD on the Pi after about 3 or 4 weeks of video… so, I needed to tweak the command for the download script to only keep 14 days on the pi. Resilio Sync, the app I use to sync back to the house, has a “keep deleted files in an archive” folder option, so when the pi does delete the files, they are still stored on the Pi… I would like to find a way of automating that…

While trying to figure out how to fix part 1, I came up with an idea: I have an older Mikrotik RB951G that can be powered via a 12v adapter for the car. I am going to use that, along with a Huawei 4G dongle to act as an internet connection. The onboard Wifi will be in client mode, so when it’s near the house, it will connect to the main network and send traffic through that to the internet (or internal NAS) and when away, use the LTE modem. Then, using the Wifi dongle on the Raspberry Pi, use that as a Wifi AP.

Anything in the car that needs Wifi will connect to the Pi, which will act as a bridge to the Mikrotik. When the script needs to download files from the dashcam, it should have a direct connection to it, plus (hopefully) will be faster… then the Pi is connected to the internet through the Mikrotik. The Pi has both Tailscale and Zerotier on it for remote management, and the Mikrotik can be configured to use Wireguard to connect back to the house directly if required.

I have some of this working on a bench in the house, but it will be a while before I manage to get this running fully… Hopefully, I will have some more stuff sorted this weekend…

Ubiquiti UDM Pro Fail over to Speedify

So, this has been a blog post in the making for a while now but never got around to fully writing it up, so here goes nothing…

I run a UDM Pro in the house. It has 2 WAN Links: 1 1Gb link and 1 10Gb Link. I also run AS204994, my own ASN with its own Transit and Peering connections, mostly in Europe. There is a VM in the house which acts as a connection to AS204994, which gives me a full connection to the Internet through my own ASN. More details on my AS204994 blog are here.

That connection is hooked up to the 10Gb Link on the UDM Pro, which is listed as the primary internet link. Details on how these works were uploaded in this video on YouTube:

In the video above, I was using OpenMPTCPRouter to connect to the internet, but it’s been causing some issues lately, I decided to try something else.

The new setup is an Intel Nuc (i3 with 32GB RAM and 2x512GB SSDs… VERY OVERKILL for the job at hand) running Ubuntu Linux. It has a USB Hub with 3 USB Ports and an Ethernet port connected, giving me 2 Ethernet ports on the box in total. 2 of the USB Ports are connected to USB 4G Modems from Huawei and the external ethernet port is directly connected to my cable modem.

USB Hub with 1 Huawei Modem and connection to second

Both modems and the ethernet port are connected to the NUC with full internet connections (The Huawei boxes give up NATed IPs, but the Cable modem is a full public IP) and then Speedify takes those 3 connections and does some bonding magic. Speedify is a handy little VPN service that does connection bonding. You can use it to make sure your internet is rock solid using multiple links, make sure streams are stable, etc. It can bond Wifi Links, LTE modems, Cable Modems, DSL, etc. Anything that can connect and be bonded. The only issue I have with it, compared to OpenMPTCPRouter is that you don’t control the upstream server…

Speedify is set in shared mode, so the internal port on the NUC is set to share the internet connection. This is hooked to the 1Gb WAN Port on the UDM Pro. This is set for failover only (currently the only option on a UDM Pro) so if my AS204994 link goes down (VM reboots, VM host dies, Cable modem connection goes out, etc) I will still have a connection. If the cable goes out, it will use just the 4G links, but if everything is running, I get all 3 connections.

ESXi on Arm (and Raspberry Pi!)

A few days back (October 6th 2020) VMWare announced a new “Fling”: ESXi Arm Edition. Not completely sure what a Fling is, but anyway, I started reading, liked the idea and managed to download a copy for testing. I have 2 Pi 4s in the house, both 4Gb Models, and I wanted to play around with the new tech.

So, after some messing with UEFI stuff, formatting Micro SD cards correctly, copying files and some limitations, I managed to get 2 new ESXi servers running on Raspberry Pi!

There is a walk though Video showing everything I did to get up and running. Its embedded below. Some of the hardware I used is also mentioned below.

Equipment list:

  • 2 x 4G Raspberry Pi 4s
  • 2 x 16Gb Micro SD Cards (you could probably get away with 1Gb cards… You only need a small 256MB partition for the UEFI stuff)
  • 2 x 64GB Kingston DataTravler USB 3 Sticks (This is where ESXi is installed, plus the rest of the storage, if configured correctly, can be used for VMs).
  • 2 X POE to USB C Splitters. I used these so I can power both Pi’s though POE and can reboot them using the switch. You could use a USB Power Adapter like the Anker PowerPort 60W which would give you 6 ports to run your Raspberry Pi’s. I would probably limit it to running 4 Pi’s though, since the Pi 4 needs a bit more power…
  • Some way of installing the ISO to the Pi. I used an iodd Mini 256Gb for the task. I also did a video review of that here.
  • About an hour of your time.

As mentioned above, the USB key is used for storing ESXi when its installed. It can also be used for storing VMs. There is a command you run when installing to partition the drive in 2: 8GB for ESXi and the rest for storage. I managed to run this correctly on one, but missed it on the second. I might reinstall that Pi and get it up and running again soon. You also have the option of installing to iSCSI. That might be useful too…

Storage wise, VMWare recommend using usb3 or fast iscsi or nfs storage for vms. I’m using nfs on my workstation which seems to work OK. but you are still limited to 1Gb/s of the Raspberry Pi. They say it is possible to use extra USB network cards. Could be interesting to try that out.

So far i have managed to install a single VM on one of the Pis. I plan on migrating from a Physical PiHole instance to a virtual one. I also plan on getting a few 8Gb Pis and see where this rabbit hole gets me. It can also be managed with VSphere. Let’s see if I can get that working… Stay tuned!

If anyone has any questions, comments, etc., just shout. And if your interested in videos like these, subscribe and like the video!

Network Update Info April 2019

So, this post has been a long time coming! A load of different things to talk about, so lets get started!

GodBox V3

So, for a long time, I have been thinking about GodBoxV3, the replacement to GodBoxV2. And when planning this, i had some ideas of what it should be:

  • Minimum of 2×16 cores (double godboxv2)
  • About the same RAM, if not more
  • FAST STORAGE!
  • Is able to run my twin 30" 4K monitors
  • Would like 10Gb/s NICs

Well, It finally happened! I got the machine, built it and, well, its impressive! How did i do with specs? Well…

All is good! Photos, more details and benchmarks coming soon… stay tuned!

Finally 10Gb/s Networking!

Since GodBoxV3 had a few 10Gb nics, i needed to upgrade the network to support it. I ended up with a Ubiquiti Networks EdgeSwitch-XG. 16 ports (12 SFP+ and 4 RJ45). The SubperMicro board has 2xRJ45 ports. Due to lack of RJ45 ports, GodBoxV3 is connected to 1, GodBoxV2 is getting a 10Gb card soon, which will be connected to 1 port, and a new Sun Microsystems server (details below) will be getting the last 2… Of the SFP+ ports, 2 are connected to the EdgeSwitch Lite, 2 to the Synology (it got a 10Gig NIC reciently too!) and 2 to the new NAS (again, more details below!)

Good bye Mikrotik, Hello EdgeRouter 4

Since i was going all Ubiquiti gear (Wifi is Unifi gear) i got rid of the old Microtik and replaced it with a Ubiquiti ER4. Happy days! Got some plans for this, more details coming soon…

Updates to BGP Stuff, including IPv6

I lost one VPS in London, but replaced it with a new one from HostUS. I still use Vultr, Packet and VServer.Site as providers too. I am also adding more and more IPv6 stuff too… There is a post on AS204994 explaining a lot of this.

New NAS and more storage!

New NAS got purchased: QNAP TS-932X. I have 5X8TB spinny disks (shucked from 5 WD My Book 8TBs) + 4 X 500GB WD Blue SSDs.

New Servers and cooling updates

Moved lots of stuff around the room… Servers run cooler, and less noisy! happy days! I also got my hands on a very nice looking Sun Server X3-2. Its a Dual Xeon E5 (currently got quad cores, going to upgrade it to 8 cores) and i think its got 16GB ram and 4x300GB SAS Disks. It also has 4X10Gb nics! ESXi will probably go on here!

VMWare in the house

Up till recently, I ran Hyper-V all round. Its still on GodBox V2 and V3 (v1 has a HDD issue, so its off…), but the main VM hosts (the C6100’s) are being migrated to VMWare ESXi… Why? Its a learning exercise… We see how it goes…

So, long update… Any questions, comments, etc… shout!

Finally going all in on VoIP

After many years, I am finally trying to move to a proper VoIP system for the house. This post will explain what I am using, how I am setting it up, and some other details you might (or might not) find useful.

First, backstory. I have been interested in VoIP for many years. The first post I wrote about Ito this site was here back in 2012, but I had posted about it on my other site back in 2008. It got my attention years ago as a way of saving money on calls, but in recent times, that has changed a little, mainly because most providers gives you calls for free (my mobile and land lines both come with unlimited calls and with my mobile, I can make them anywhere in Europe). The new reason I am interesting in VoIP is consolidation: I currently have 3 mobile phone numbers, at least 1 landline dedicated to me in the house, plus a work landline. I want to be able to pick up any phone and make a call, and it show as coming from my main number. Or a call comes in and i can pick it up from any of my phones… And that is what i am trying to do here… I (will) have some of it working, but some parts are still missing…

The parts I have (or will have) working are as follows:

  • my land line number in the house is being ported to Virgin Media’s VoIP service. So, thats not stuck in an analog world any more!
  • The house phone now has a VoIP adapter allowing the standard analog phone make VoIP Calls

  • There is a company in the Netherlands called ZeroPlex who have a VoIP over GSM service. Essentially, the SIM they give is connected to your own SIP trunk. You can set it up to allow all calls to go though your SIP trunk, only incoming or only out going. I found their contact though Reddit but they may be able to help if you drop them an email.
  • All VoIP traffic in the house is routed though 3CX.

  • I have a couple of SIP trunks hooked up to 3CX: Virgin Media, Zeroplex (they redirect the NL number is sent over this, and i can make calls though this trunk too), Twilio, which i use for transient numbers, and Sip Discount which offers really cheap calls.
  • Phone wise, i use a Ubiquiti UVP-Executive desk phone, the SIM card, and the 3CX client on mobile (Either iPhone or Android).

So, all in, Im about 50% of the way there… As of the time of this post, the SIM is still in the mail and the phone numbers are not ported to Virgin Media… yet… Tomorrow they should be, and over the next few days there will be some tweaking to get it working correctly… I will probably have some updates over the coming week…

AS204994, Own IP Space and Anycast

So, if you are reading this page, it is being delivered with the magic of Anycast… Well, technically, it was before, since i used Cloudflare, and it still is because of Cloudflare, but also because of my own ASN (As204994), some servers in different locations, and some magic, which i will explain a bit of in this post.

This all started late last year when i got my hands on an ASN and a /48 block of IPv6 addresses. I had been reading stuff about BGP, routing, etc, and decided to go all in. it was quite cheap with the help of HostUS. All in, it was about $50 for the year. As part of the process, i needed 2 upstream providers to say they would accept my announcement. They were Hurricane Electric though their Tunnel Broker service, and Vultr using a few of their VPSs.

After i got my space and ASN, i started to announce the V6 addresses over Vultr and Hurricane Electric, and all was good. I had 2 Vultr servers: 1 in London, UK, and one in New Jersey, USA. I had my home machine announce to HE, and then also link to both Vultr servers using Zerotier. All worked well, but due to some family issues, i never got around to putting it into production… till now.

Those 3 servers now share an IPv6 address on the loop-back port. When you (well, Cloudflare) asks for that IP, the closet (network) with that IP responds, and the NGinx server on that box sends back the contents of the site. This site is hosted on each box, since its fully static, but both AS204994 and TiernanOToole.net are hosted in Ghost, so Dublin (my machine in the house) serves them, and both Lon1 and Nyc1 do proxying. so, most requests from the US are hitting the box in NYC and the ones in Europe share either Dub1 or Lon1. I have some tweaks to do with which servers will be running where, and may add more, but currently its working well.

So, how do you figure out what server responded? Simple. Open your Dev tools on your browser, go to network tab, refresh, and see the response headers for anything on this domain. You should see something like below.

Over the next while, i will be updating tiernanotoole.net with more details on how this works, and more stuff will end up on AS204994.net too. If anyone notices any weird and wonderful issues, shout. If you have more questions, shout.

Zerotier and Minio Followup

in a previous post, I talked about setting up a distributed S3 like data storage system using Minio and ZeroTier. Well, this week, the ZeroTier guys tweeted about this.

A few people then started asking questions, and looking for a follow up, so here it is…

First, a quick recap. I had 4 machines, all running Linux. Three of them were in 1 time zone (GMT+1) and one was in another (GMT). Looking at the Distributed Minio Quickstart Guide again, there is a mention of times being in sync… which is probably why this did not work as planned… and by “not work as planed”, I mean that Minio would crash, or not be responsive, or not write data in the place it should have… which was a pain. But looking at the documentation again, they do mention that Windows support is “experimental” which means, hopefully, some day it will be not so experimental, and might work… Given that most of my machines in house are Windows boxes, this would be a nice feature.

Now, what about ZeroTier? Given they posted it to their twitter? Well, it worked. it did the inter connect stuff well, and, given bandwidth limitations on a home broadband connection, it was still quite fast.

So, the question is, how fast? Well, on my Surface Book on a WiFi connection in the house, behind a Meraki MX64 firewall, connecting to the GodBoxV2 over FTP though ZeroTier, i get the following result:

the same download over FTP direct (no ZeroTier) does the following:

So, direct over FTP is faster… in this instance by about 70%, but, over the download, it did get slower (seen it hit 12 at one stage) and because its over WiFi, those are a bit wonky…

I did get one last screen shot:

as you can see, the Zerotier network adapter is showing 77.3Mbps, but the main network adapter is showing 80.8Mbps. There would be other traffic there, but if we assume there is nothing but ZeroTier traffic being sent, there is about 5% of an overhead.

So, to wrap up: Minio and its distributed storage system over ZeroTier needs more testing. Ideally, all hosts need to be in the same time zone, or at least have the same time… Will try work on that soon. As for ZeroTier? I am extremely happy with them. Its fast, easy to setup, and easy to configure. What more could you ask for? Oh, and free, unless you need a pro account!