Day 16 of #100daysofhomelab and not much going on. Busy with work. I am running a test though. I seen the following tweet a few days back:
Given some of my smaller boxes are running 90%+ memory usage, i have decided to move the VMs from my NUC to my QNAP storage box. Its going to take a while to move them over, but we see what RAM usage is like after.
Just for reference, this is the before:
Day 15 of #100daysofhomelab and i have been playing with Portainer a bit, including the Pi-Hosted templates. So far, i have installed Rust Desk and Your Spotify.
Day 14 of #100daysofhomelab and I have been thinking about future upgrades if I had the money… So, I have my CloudShed in the back garden. Currently, I only have an HP Micro Server and a (not currently in production) Dell R720, along with a Ubiquiti Edge Switch 48 Lite. Between the Shed and the house is a fibre link purchased through FS.com, with 6 pairs. Currently, only 1 pair is in use, giving me a 10Gb/s between the house and shed, and with the easy option to upgrade to 20Gb. But I have been thinking bigger.
I have been looking at the Mikrotik CCR2004-1G-2XS-PCIe (a bit of a mouthful…) SmartNIC. It’s a full MikroTik router on a PCIe Card. It has a Quad-core ARM Processor, 4GB RAM, some storage and 2X25Gbit/s Interfaces… Well, technically, 4… there are 2 front connectors and 4 that the host server sees… If I am reading the diagram below correctly, it looks like all ports are seen by the Host, but 2 go through the bridge and 2 go direct… I haven’t played with one yet (Mikrotik, if you are listening, hint, hint!) so not sure how it would work… A review from Alyx Wijers says that on the Linux box they tried, the 10Gb SFP+ module they had shows in passthrough and the other 2 are connected to the bridge… Ideally, for the ideas i have for this, I passthrough would be handy for stuff like storage, but i would want the rest of my traffic going over that bridge interface… Or at least i think thats how it would work…
The card has 2X25Gb ports (SFP28 ports) that connect to the rest of your network, for example to a CRS504-4XQ-IN switch (4x100Gb ports, which can be broken out into 4x25Gb ports each…). If you go through the bridge, you get all the features of RouterOS, like firewall rules, VXLAN, etc, all in the NIC. The switch then doesn’t need to do as much, letting it do the switching and leaving everything else at a NIC level. If you use passthrough, you, essentially, bypass the router/firewall rules… I think…
So, what would my plan for the upgrade be? Well, this is where things get expensive… I would need 2 of the switches (one in the house, one in the shed linked with a single 100Gb fibre). Then, I would need 8 of the SmartNICs (GodboxV3, GodBoxV2, 1 for each of the R720s (second one coming soon), 1 for each of the R620s (coming soon), one for the HP DL380 G8 (also pending) and 1 for the big storage box… again, pending). The plan would be that GodBoxV2 and V3 would be in the house, and both connect to the house switch at 25Gb a sec. there would be uplinks to internal 10Gb switches along with the UDM Pro.
In the shed, the 6 servers would each connect to the switch at 25Gb, using 10 of the renaming 12 ports. The R720s, Storage Box and HP will probably get 2x25Gb connections. In theory, the R620s could also connect at 50Gb but I would have no extra room later… Might not be a major issue, mind you. There would be spare ports in the house… I could, in theory, get a second 100Gb switch for the shed! 😛
But, what would this cost? Well, current prices are showing that the cards are around 200EUR a pop and the switch is just shy of 800 quid… so, for a little under €3200, I could get 2 switches and 8 NICs. I would need break-out cables, 100Gb Optics, and some other bits, so, say 500 quid for that… So, just under 4k? One of these days, hey! I can dream!
MikroTik CRS504-4XQ-IN Review Momentus 4x 100GbE and 25GbE Desktop Switch (servethehome.com)
Review: A Dive into Mikrotik’s Weird SmartNIC (CCR2004-1G-2XS-PCIe) // Alyx Wijers
Day 13 of #100daysofhomelab and it’s mostly a rest and update day. I got my Plex Server back online, so ended up watching a load of stuff on that. Also, I upgraded my OpnSense box to OPNsense 23.1.beta. There are not many machines behind it, but I will keep my eye on it and see how things go. Hope to be back to some normality tomorrow.
Day 12 of #100daysofhomelab and I am still battling with my ZFS pool on my Plex Server… So, that has taken all my time today… ugh…
update to day 11 of #100daysofhomelab, and I thought this needed its own dedicated post. I managed to fix my ZFS pool and got it imported into Ubuntu, so all is good, but I found these links and this is cool!
So…. running MacOS and Android inside Docker is pretty cool! Could be handy for building, well, build servers for developers that need MacOS. and the Android stuff is handy for dev/testing too. Very cool.
Day 11 of #100daysofhomelab and i am trying to fix my Plex Server… Seems when i moved from Fedora to Ubuntu, my ZFS pool did not import. I did not notice this, since mostly what is on it was temp files and logs… Well, the main drive is running out of space, so I checked the ZFS and it was failing because the version of OpenZFS I was running on Fedora (from master in their GitHub repo) is not compatible with the one on Ubuntu… (facepalm) So, have to rebuild and install OpenZFS from code… hopefully this works… [Edit… it did not work… ugh] [Edit 2: This did work though: Installing ZFS on Ubuntu (uptrace.dev) especially the part of building from code].
In other news, some links below.
Day 10 of #100daysofhomelab and its mostly updates and monitoring.
I think my next plan for the Kubernetes cluster is to rebuild the VMs and start from scratch. Currently, they ranged from 2-4 cores and 4-8Gb RAM. They also had a single disk on them and used an Ubuntu 22.04 cloud image. I think the plan going forward is to make sure each has similar RAM and Cores, none are going on the smaller VM Hosts I have, and I will be adding a new disk just for storage. Looks like Minio might work for me… More testing and reading are required though.
Well, day 9 of #100daysofhomelab is about Disaster Recovery… Well, at least the disaster part… Recovery not so much… My Kubernetes cluster, how do I put this… shat the bed… It’s been up and down all day and then the Longhorn storage failed and took my WordPress install with it… I lost yesterday’s post (which isn’t the end of the world) but it’s a pain in the ass… I ended up using the old docker copy of WordPress, so at least that’s online.
So, going to shut down the full cluster and start again… Might be looking at something other than Longhorn for storage… but giving up for the day… I will be back tomorrow.
Day 7 of #100daysofhomelab and just a quick update for today: this site is now running on my Kubernetes Cluster! I am using Cloudflare tunnels for the ingress controller (more on that later) and so far, so good… Most of this was done yesterday, and it was a swap over of the DNS stuff today… been sick most of the day, so that’s all I got in me for day 7…
