Tiernan's Comms Closet

Geek, Programmer, Photographer, network egineer…

Currently Viewing Posts in Homelab

ESXi on Arm (and Raspberry Pi!)

A few days back (October 6th 2020) VMWare announced a new “Fling”: ESXi Arm Edition. Not completely sure what a Fling is, but anyway, I started reading, liked the idea and managed to download a copy for testing. I have 2 Pi 4s in the house, both 4Gb Models, and I wanted to play around with the new tech.

So, after some messing with UEFI stuff, formatting Micro SD cards correctly, copying files and some limitations, I managed to get 2 new ESXi servers running on Raspberry Pi!

There is a walk though Video showing everything I did to get up and running. Its embedded below. Some of the hardware I used is also mentioned below.

Equipment list:

  • 2 x 4G Raspberry Pi 4s
  • 2 x 16Gb Micro SD Cards (you could probably get away with 1Gb cards… You only need a small 256MB partition for the UEFI stuff)
  • 2 x 64GB Kingston DataTravler USB 3 Sticks (This is where ESXi is installed, plus the rest of the storage, if configured correctly, can be used for VMs).
  • 2 X POE to USB C Splitters. I used these so I can power both Pi’s though POE and can reboot them using the switch. You could use a USB Power Adapter like the Anker PowerPort 60W which would give you 6 ports to run your Raspberry Pi’s. I would probably limit it to running 4 Pi’s though, since the Pi 4 needs a bit more power…
  • Some way of installing the ISO to the Pi. I used an iodd Mini 256Gb for the task. I also did a video review of that here.
  • About an hour of your time.

As mentioned above, the USB key is used for storing ESXi when its installed. It can also be used for storing VMs. There is a command you run when installing to partition the drive in 2: 8GB for ESXi and the rest for storage. I managed to run this correctly on one, but missed it on the second. I might reinstall that Pi and get it up and running again soon. You also have the option of installing to iSCSI. That might be useful too…

Storage wise, VMWare recommend using usb3 or fast iscsi or nfs storage for vms. I’m using nfs on my workstation which seems to work OK. but you are still limited to 1Gb/s of the Raspberry Pi. They say it is possible to use extra USB network cards. Could be interesting to try that out.

So far i have managed to install a single VM on one of the Pis. I plan on migrating from a Physical PiHole instance to a virtual one. I also plan on getting a few 8Gb Pis and see where this rabbit hole gets me. It can also be managed with VSphere. Let’s see if I can get that working… Stay tuned!

If anyone has any questions, comments, etc., just shout. And if your interested in videos like these, subscribe and like the video!

Network Update Info April 2019

So, this post has been a long time coming! A load of different things to talk about, so lets get started!

GodBox V3

So, for a long time, I have been thinking about GodBoxV3, the replacement to GodBoxV2. And when planning this, i had some ideas of what it should be:

  • Minimum of 2×16 cores (double godboxv2)
  • About the same RAM, if not more
  • FAST STORAGE!
  • Is able to run my twin 30" 4K monitors
  • Would like 10Gb/s NICs

Well, It finally happened! I got the machine, built it and, well, its impressive! How did i do with specs? Well…

All is good! Photos, more details and benchmarks coming soon… stay tuned!

Finally 10Gb/s Networking!

Since GodBoxV3 had a few 10Gb nics, i needed to upgrade the network to support it. I ended up with a Ubiquiti Networks EdgeSwitch-XG. 16 ports (12 SFP+ and 4 RJ45). The SubperMicro board has 2xRJ45 ports. Due to lack of RJ45 ports, GodBoxV3 is connected to 1, GodBoxV2 is getting a 10Gb card soon, which will be connected to 1 port, and a new Sun Microsystems server (details below) will be getting the last 2… Of the SFP+ ports, 2 are connected to the EdgeSwitch Lite, 2 to the Synology (it got a 10Gig NIC reciently too!) and 2 to the new NAS (again, more details below!)

Good bye Mikrotik, Hello EdgeRouter 4

Since i was going all Ubiquiti gear (Wifi is Unifi gear) i got rid of the old Microtik and replaced it with a Ubiquiti ER4. Happy days! Got some plans for this, more details coming soon…

Updates to BGP Stuff, including IPv6

I lost one VPS in London, but replaced it with a new one from HostUS. I still use Vultr, Packet and VServer.Site as providers too. I am also adding more and more IPv6 stuff too… There is a post on AS204994 explaining a lot of this.

New NAS and more storage!

New NAS got purchased: QNAP TS-932X. I have 5X8TB spinny disks (shucked from 5 WD My Book 8TBs) + 4 X 500GB WD Blue SSDs.

New Servers and cooling updates

Moved lots of stuff around the room… Servers run cooler, and less noisy! happy days! I also got my hands on a very nice looking Sun Server X3-2. Its a Dual Xeon E5 (currently got quad cores, going to upgrade it to 8 cores) and i think its got 16GB ram and 4x300GB SAS Disks. It also has 4X10Gb nics! ESXi will probably go on here!

VMWare in the house

Up till recently, I ran Hyper-V all round. Its still on GodBox V2 and V3 (v1 has a HDD issue, so its off…), but the main VM hosts (the C6100’s) are being migrated to VMWare ESXi… Why? Its a learning exercise… We see how it goes…

So, long update… Any questions, comments, etc… shout!

Finally going all in on VoIP

After many years, I am finally trying to move to a proper VoIP system for the house. This post will explain what I am using, how I am setting it up, and some other details you might (or might not) find useful.

First, backstory. I have been interested in VoIP for many years. The first post I wrote about Ito this site was here back in 2012, but I had posted about it on my other site back in 2008. It got my attention years ago as a way of saving money on calls, but in recent times, that has changed a little, mainly because most providers gives you calls for free (my mobile and land lines both come with unlimited calls and with my mobile, I can make them anywhere in Europe). The new reason I am interesting in VoIP is consolidation: I currently have 3 mobile phone numbers, at least 1 landline dedicated to me in the house, plus a work landline. I want to be able to pick up any phone and make a call, and it show as coming from my main number. Or a call comes in and i can pick it up from any of my phones… And that is what i am trying to do here… I (will) have some of it working, but some parts are still missing…

The parts I have (or will have) working are as follows:

  • my land line number in the house is being ported to Virgin Media’s VoIP service. So, thats not stuck in an analog world any more!
  • The house phone now has a VoIP adapter allowing the standard analog phone make VoIP Calls

  • There is a company in the Netherlands called ZeroPlex who have a VoIP over GSM service. Essentially, the SIM they give is connected to your own SIP trunk. You can set it up to allow all calls to go though your SIP trunk, only incoming or only out going. I found their contact though Reddit but they may be able to help if you drop them an email.
  • All VoIP traffic in the house is routed though 3CX.

  • I have a couple of SIP trunks hooked up to 3CX: Virgin Media, Zeroplex (they redirect the NL number is sent over this, and i can make calls though this trunk too), Twilio, which i use for transient numbers, and Sip Discount which offers really cheap calls.
  • Phone wise, i use a Ubiquiti UVP-Executive desk phone, the SIM card, and the 3CX client on mobile (Either iPhone or Android).

So, all in, Im about 50% of the way there… As of the time of this post, the SIM is still in the mail and the phone numbers are not ported to Virgin Media… yet… Tomorrow they should be, and over the next few days there will be some tweaking to get it working correctly… I will probably have some updates over the coming week…

AS204994, Own IP Space and Anycast

So, if you are reading this page, it is being delivered with the magic of Anycast… Well, technically, it was before, since i used Cloudflare, and it still is because of Cloudflare, but also because of my own ASN (As204994), some servers in different locations, and some magic, which i will explain a bit of in this post.

This all started late last year when i got my hands on an ASN and a /48 block of IPv6 addresses. I had been reading stuff about BGP, routing, etc, and decided to go all in. it was quite cheap with the help of HostUS. All in, it was about $50 for the year. As part of the process, i needed 2 upstream providers to say they would accept my announcement. They were Hurricane Electric though their Tunnel Broker service, and Vultr using a few of their VPSs.

After i got my space and ASN, i started to announce the V6 addresses over Vultr and Hurricane Electric, and all was good. I had 2 Vultr servers: 1 in London, UK, and one in New Jersey, USA. I had my home machine announce to HE, and then also link to both Vultr servers using Zerotier. All worked well, but due to some family issues, i never got around to putting it into production… till now.

Those 3 servers now share an IPv6 address on the loopback port. When you (well, Cloudflare) asks for that IP, the closet (network) with that IP responds, and the NGinx server on that box sends back the contents of the site. This site is hosted on each box, since its fully static, but both AS204994 and TiernanOToole.net are hosted in Ghost, so Dublin (my machine in the house) serves them, and both Lon1 and Nyc1 do proxying. so, most requests from the US are hitting the box in NYC and the ones in Europe share either Dub1 or Lon1. I have some tweaks to do with which servers will be running where, and may add more, but currently its working well.

So, how do you figure out what server responded? Simple. Open your Dev tools on your browser, go to network tab, refresh, and see the response headers for anything on this domain. You should see something like below.

Over the next while, i will be updating tiernanotoole.net with more details on how this works, and more stuff will end up on AS204994.net too. If anyone notices any weird and wonderful issues, shout. If you have more questions, shout.

Zerotier and Minio Followup

in a previous post, I talked about setting up a distributed S3 like data storage system using Minio and ZeroTier. Well, this week, the ZeroTier guys tweeted about this.

A few people then started asking questions, and looking for a follow up, so here it is…

First, a quick recap. I had 4 machines, all running Linux. Three of them were in 1 time zone (GMT+1) and one was in another (GMT). Looking at the Distributed Minio Quickstart Guide again, there is a mention of times being in sync… which is probably why this did not work as planned… and by “not work as planed”, I mean that Minio would crash, or not be responsive, or not write data in the place it should have… which was a pain. But looking at the documentation again, they do mention that Windows support is “experimental” which means, hopefully, some day it will be not so experimental, and might work… Given that most of my machines in house are Windows boxes, this would be a nice feature.

Now, what about ZeroTier? Given they posted it to their twitter? Well, it worked. it did the inter connect stuff well, and, given bandwidth limitations on a home broadband connection, it was still quite fast.

So, the question is, how fast? Well, on my Surface Book on a WiFi connection in the house, behind a Meraki MX64 firewall, connecting to the GodBoxV2 over FTP though ZeroTier, i get the following result:

the same download over FTP direct (no ZeroTier) does the following:

So, direct over FTP is faster… in this instance by about 70%, but, over the download, it did get slower (seen it hit 12 at one stage) and because its over WiFi, those are a bit wonky…

I did get one last screen shot:

as you can see, the Zerotier network adapter is showing 77.3Mbps, but the main network adapter is showing 80.8Mbps. There would be other traffic there, but if we assume there is nothing but ZeroTier traffic being sent, there is about 5% of an overhead.

So, to wrap up: Minio and its distributed storage system over ZeroTier needs more testing. Ideally, all hosts need to be in the same time zone, or at least have the same time… Will try work on that soon. As for ZeroTier? I am extremely happy with them. Its fast, easy to setup, and easy to configure. What more could you ask for? Oh, and free, unless you need a pro account!

Business Class Broadband… finally here….

So, after many (MANY) years messing with dual cable modems, struggling to get them working together, to get websites to even allow me in, having to use hacks and kluges to get it to work at all… I have given up. It has been a struggle getting two modems working properly. Load blanching kind of works… but it’s messy at best. Some sites kick you out every now and again because your IP changes. Some sites wont let you login at all… Mind you, some sites work grand and don’t ask questions…

And the whole idea of multiple modems, to allow you to download things faster, doesn’t work for everything… Anything you download in the browser is single threaded, so its limited to one modem… you can use use download accelerators, and they do work, but its an extra step, and some sites don’t work for that either (MSDN for example).

So, i have given up, bit the bullet, and moved to business class broadband from Virgin Media. It’s actually cheaper than the two residential lines i had, but it is also slower than the two combined: previously, it was two x 360/36mb/s. Now i am am on a single 400/40mb/s modem. That being said, there are definite advantages:

  • Static IPs pretty much as standard, and option of either one or five (no unbeaten!) Guess which one i went with? Its technically a /29 range, but the first usable IP is given to the modem, which acts as a gateway, so i end up with five usable.
  • Proper business class SLA. Any issues, someone who knows what they are talking about can help
  • Phone lines on a separate modem. So, i got phone lines with them, and they give a separate modem for those lines, so as not to interfere with the internet. that modem has no internet connection and is just for calls. They are also working on a VoIP/SIP offering, which is something i am interested in.
  • Guaranteed speed! They guarantee a minimum speed to the modem at all time. Business customers have a priority on the network, which is nice. And, during testing, so far, i am getting the advertised speed most of the time. I needed to download a Windows 10 ISO yesterday from the MSDN, and it came in at between 45 and 48MBytes/s!

So, only had it installed a week, and so far, so good. I have one IP given to my PFSesne box, and the rest given to a VyOS VM. The plan is to use the VyOS box for all network traffic, but first i need to do some testing and learning… Expect some posts on this soon!

Distributed S3 data storage using Minio (and Zerotier)

So, something i have been looking into in recient times has been Distributed Storage, and, more specifically, how to use the storage in my many, many machines to protect data, and also increese my usable space… There are a few projects on the market that do this (Ceph, NooBaa and Gluster all spring to mind) but some are more painful to setup than others… which brings me nicely to Minio. Minio is a 20ish MB executable you download from their site, mark it as executable (on Linux or Mac Boxes) and run… and you have yourself a S3 compatable storage server… Simples!

“But Wait!” i here you screem! “thats not distributed!”. Well, yes… but, it can be! Their Distributed Quick Start Guide, which is where i started with this, allows you to run a distributed copy of your data. I will let their documentation explain more, but this is what i did:

  • download the minio server (single executable file) on a minimum of 4 machines.
  • on each machine, run a command like the following:

replacing accesskey and secretkey with keys (check minio documentation to get these) and foldertoexport with, well, the folder you want to export!

For me, i have 4 servers currently clustered. 2 are in online.net (one in Paris, one in Amsterdam), 1 in OVH.NET (France, somewhere) and one in Dublin (GodBoxV2 currently). They are all interconnected using ZeroTier (I will explain that later) and so far, so good… only ran some basic tests, but with it, i could loose 2 machines and still have data… Not bad for free! I will run some speed tests soon.

Meraki and Ubiquiti networks gear Update

In part 6 of my Double Internet Series I mentioned i was running a Meraki MX64 in the network, and said i would write up about it. I am taking this opportunity to also write up about the Ubiquiti networks gear in the house also.

  • First on the list is my older Ubiquiti Edgerouter POE. It currently in the process of being decommissioned, or used for something else. It was the main edge router for the network: it had both internet connections connected, and did routing, firewalls, etc, but with the Proliant taking over as a router, it is not required as much any more… Its still on, mainly because its still a DHCP server, but not much else.
  • There are 2 Meraki MS220-8 switches next. GodBox1 and Godbox2 both connect in here, and are bonded, as is everything else on the network. The MS220-8 has 8 GigE ports, but also has 2 SFP ports. I bought 4 SFP Ethernet adapters and have a short calbe running between the switches. That uplink is also bonded. All going well so far!
  • All Meraki hardware can be managed though the Meraki dashboard. check out their site for more details and examples of how to use it.
  • I bought one of the MS220’s from eBay a few months back, and loved it. Then i realized that you can get your hands on free gear, the MX64, an MS220 and a Wi-Fi Access point if you attend their webinars. Terms and conditions apply, but check them out!
  • I have 2 Ubiquiti UniFi APs, one in the front of the house, one in the back. They are connected to one of the MS220’s, but dont work with its POE (maybe the EdgeRouter could do that, since its POE…) so there are injectors for them. Anyway, the network ports on there are VLANed to the MX64 (more on that later) and the default traffic is going to a management VLAN.
  • The MX64 has a static internal IP on my DMZ network, and uses the Proliant as an upstream connection. Upstream on the Hetzner server, all traffic coming from the MX64 ip uses one of my /29 ip block. all traffic to that ip is also forwarded directly to the MX64.
  • I has 2 small, unmanaged switches (a cheap 8 port Linksys and a 8 port TP Link) which are used for separate things: the Linksys has 4 Raspberry Pi’s, which run a GlusterFS cluster, plugged into it and the TP Link connects to my printers.
  • I also have a Mikrotik CRS226-24G-2S+IN which has 2 10Gbit SFP+ Ports, and plan on using this for higher speed networking soon, aswell as a Cisco 48 port 3560 which also has 4 SFP ports (GigE) and may come in handy for something soon…

So, thats the network currently. any questions, please leave a comment.

double speed Internet Part 8 – Routing Around

[NOTE] This part 8 in a series of posts. The rest can be found here.

At the end of my last post I asked the question about routing traffic to different servers based on thier distances, etc… Well, after a bit of messing, i can say it kind of works! here is a quick over view:

  • server in the house has now got multiple OpenVPN connections (2 to Hetzner, 1 to OVH (with a plan to double), 1 to Digital Ocean (again, to be doubled) and i am planning 2 to Azure as well).
  • Quagga/Zebra has static routes (currently static, planing on dynamic soon… more eventually) to different servers depending on where they are. for example, all traffic to the hetzner network (including their Storage Boxes) go though the hetzner link. Hubic traffic goes though OVH, Azure (currently) and AWS traffic, aswell as some CDNs go direct over either WAN1 or WAN2 in the house, and some other stuff (CrashPlan currently) goes though Digital Ocean. Everything that has no static route goes though Hetzner…
  • Ideally, the static side of things should be removed, and a more dynamic setup done. How that works, i have no idea… Spotify have 2 posts about their SDN Internet Router (part 1 and part 2) which is an interesting idea… More digging and research is required.

So, there you have it. Everything currently seems to be working, mostly, and tweaks can be made easily… I have a couple posts i have in my head, including something to do with automating bringing up new machines (probably with Ansible or something like it), more monitoring, and some other stuff too… Any questions, leave a comment, and i will get back.

[UPDATE] I wrote a quick and dirty app called WhoIsToZebraConfig which takes an AS Number, looks up the info in the Merit RADb (with the help of some code from Coder Buddy) and outputs what you need to put into your Zebra Config… should save me some time, and it might save you time too… shout if you have questions!

double speed Internet Part 7 – ECMP (kind of)

[NOTE] This part 7 in a series of posts. The rest can be found here.

In the last post I mentioned I am now using Hetzner for hosting a dedicated box. Thats still live, and going well. I have a /29 IP range (6 usable) and also 2 other IPs. So far, so good… But because i was using a Socks Server, I was not fully able to use the /29 ips… I use something like as follows:

essentially, for each public IP i have that i want to map to an internal IP, i have a POST and PRE ROUTING rule, plus the required forward rules… But, if socks are used, then that goes out the Window, since TCP traffic will look like its coming from the socks server… So, i killed the socks server, removed the IPTables rule, and then realized that while outgoing traffic was being balanced somewhat (2 default rules on the internal box pointing at the OpenVPN IPs from the Hetzner box) incoming was a problem. Hetzner knew how to get to my internal network, but only though one ip… enter Quagga and Zebra…

Quagga is a routing software suite, which can do protocols like OSPF, BGP and RIP, and Zebra is the component that does static routing. using their documentation on static routes, I created a static route to my internal network with 2 next hops, the OVPN IPs from the internal box… and, after restarting Quagga, all works! happy days! now i can forward ips from outside the network to inside the network correctly, and they look like they are the public ip!

So, whats next then? well, I now have a server in Germany (Hetzner) and one in France (OVH), and can spin one up in the UK or the US (Digital Ocean). Given that i have Quagga running on the box, i am now thinking of trying to see if its possible to route traffic depending on distance or something similar… If i am trying to hit a server in Hetzner’s DC, i should go though Germany. If its in Digital Ocean, go though either US or UK servers, same with OVH. Then figure out who has the fastest links to, say, Amazon, Azure, Netflix, BBC, Dropbox, etc, and add either static or dynamic links to the router… essentially, thats the theory… lets see how that works…