Tiernan's Comms Closet

Geek, Programmer, Photographer, network egineer…

Currently Viewing Posts in Homelab

Day 61 of #100daysofhomelab – swapping disks in a Hetzner Dedicated Machine

It’s been a while… So, for Day 61 of , I thought I should write up how to swap a disk in a Hetzner Dedicated Machine.

I have a dedicated server I rent from Hetzner in Germany. It has an Xeon E5-1650 V2 processor (6 cores, 12 threads, 3.5Gz base, 3.9Gz turbo), 128Gb RAM, and a pretty impressive 15 6Tb HDD. All drives are hooked to a Mega RAID controller, but because I am running ProxMox, I left it in JBOD mode and set up the 15 drives in RAIDZ-2. All 15 drives are in a single pool (probably not ideal, but it works for me). Now and again, I get a message from ProxMox telling me about bad blocks… and every time it happens, I have to remember what to do to find the bad drive, report it to Hetzner, wait for them to replace the drive and then add it back to the pool… Today, it happened, so I thought I better document it, to help future me, and hopefully someone else out there…

First, we need to find the drive in question. Usually, I’m my alerts, I get the Serial number of the drive causing problems. So, I ran the following command:

megacli -PDList -aAll | egrep "Enclosure Device ID:|Slot Number:|Inquiry Data:|Error Count:|state"

This gives me a full list of drives along with the Slot Number (needed when sending to Hetzner) and the Serial Number. the data output starts with the “Enclosure Device ID:” so when you find the Serial number, look above it for the Slot Number… so, my issue is with the disk in Slot 10. I opened a support ticket with Hetzner requesting a replacement disk. It can take an hour or more for this, but sometimes faster. Depends on their load…

Once you get a confirmation that the disk is done, you now need to swap it into the zpool.

first, we must check if the new drive is set up correctly. Run the following:

megacli -PDList -a0 | grep Firmware

We are looking for “Firmware status: Online, Spun Up”. If we have anything marked as configured, we need to run the following:

megacli -CfgForeign -Scan -a0

This shows us any foreign configurations. If that’s more than 0, we run:

megacli -CfgForeign -Clear -a0

This clears out that configuration. Next, we need the Enclosure ID and Slot number for the new drive from:

megacli -PDList -aAll | egrep "Enclosure Device ID:|Slot Number:|Inquiry Data:|Error Count:|state"

cause we need to run:

megacli -PDMakeGood -PhysDrv [<enclosure>:<slot>] -a0

Finally, run:

megacli -CfgEachDskRaid0 WB RA Direct CachedBadBBU -a0

Note: If that fails with a message about cache data, you may need to run:

megacli -DiscardPreservedCache -L"10" -a0

This will clear the cache and then you can run the CfgEachDskRaid0. This will mark all new disks as JBOD disks… used for ZFS. If you have something different, check the docs from Hetzner below.

Next, we need to swap disks in ZFS. Run

zpool status

to get the info about the missing disks. the missing disk will show as unavailable. Next, find the ID of the disk that was added.

cd /dev/disk/by-id/

ls

find the new disk (usually wont have any partitions on it). Now, its a matter of running the following:

zpool replace rpool /dev/disk/by-id/scsi-3600605b008f498802aa37da51674ea7e-part3 /dev/disk/by-id/wwn-0x600605b008f498802b2a3a683752e088

swap the scsi-36xxx and wwn-0x6xxx parts for the ones you found and rpool with your ZFS pool name.

finally, run

zpool status

to see the status, run:

zpool status -v -1

shows you the status with more info and refreshes every second. ZFS is now running in the background resilvering the drives and swapping out the old ones. since the old one is missing, it will wait till the new drive is sorted then remove the old one. This can take some time, depending on your disks and data size.

Hopefully, this helps someone!

Some links for info:

LSI RAID Controller – Hetzner Docs

Day 60 of #100daysofhomelab

Day 60 of and I have been sick for most of the last 2 weeks, so that’s why I haven’t been posting much… Today is going to be links only too…

Day 59 of #100daysofhomelab – Proxmox Updates, LTT Hacked, New Framework Laptops

Day 59 of and Proxmox released 7.4 of their Virtual Environment. I have not upgraded any of my machines to it, just yet, but that’s the plan for the weekend. Other than that, some links:

Day 58 of #100daysofhomelab

Day 58 of and today is mostly a retrospective of what I did over the last few days, with some links thrown in for good measure…

Given I am going to keep GodBoxV3 running Windows Server 2022 for the foreseeable future, I installed Veeam Availability Suite (through their NFR program) and got it to backup up my Hyper-V VMs, along with my ESXi VMs to both local and Backblaze B2 storage. So far, so good.

Also, Ubiquiti released Unifi OS 3.0 for the UDM Pro, which I upgraded this morning. Links for that are below. Some nice bits in here, like:

  • Added Wireguard VPN Server support.
  • Added VPN Client Routing.
  • Added Ad-blocking feature.
  • Added support for OpenVPN tunnel in Traffic Routes.
  • Allow adding multiple VPN Clients.

the 2.5 release OS had the VPN Client option, but ALL traffic went over the VPN, whether you wanted it to or not. This release gives you the option to say that traffic from a given host, network or even traffic to a given IP or range, goes over the VPN link. The Ad Block feature is nice too, but I have not tried it yet (still using PiHole for the moment) and the Wireguard VPN option is going to be VERY handy. More testing coming soon…

Anyway, on to the links.

Day 57 of #100daysofhomelab

Day 57 of and its a link dump for today:

Day 55 of #100daysofhomelab

Day 55 of and it’s going to mostly link dump today… Still working on my TrueNAS stuff, hopefully, I can get a couple of blog posts about it soon enough… Anyway, on to the links.

Day 54 of #100daysofhomelab

Day 54 of and it’s going to be a very quick one… My head is wrecked with TrueNAS… Swapped TrueNAS Core (FreeBSD) to TrueNAS Scale (Linux). Trying to get Resilio Sync to work on it, but getting permissions issues… It’s after 2 am here, so giving up for the moment, but hopefully, I can figure it out tomorrow… On a different note, I ordered a load of storage upgrades (Another Hyper M.2 x16 card, some new NVMe drives, and some other stuff) for GodBoxV3… More details soon…

Day 53 of #100daysofhomelab

Day 53 of and It’s been a busy week… ish… I’ve been battling with Vertigo on and off this week, so haven’t don’t a lot. I did, however, fix some issues with the network, set up a proper failover WAN connection using SmoothWAN and my Quad 2.5Gb Box, and have started making major changes to GodBoxV3.

Originally, GodBoxV3 had all spinning disks (8 8Tb drives shucked from WD My Book 8TBs or 8TB Seagate IronWolf) in a single RAID 5 pool in Windows Storage Spaces. Then the NVMe drives were a second pool (5 of them, 4 Force MP510 480Gb NVMe SSDs on a Hyper M.2 x16 card and a 5th unbranded one of a 1X PCI-E add-in card) and a third pool of 2 960Gb IronWolf SSDs.

I deleted the RAID5 and NVMe arrays, and now, for testing, I have spun up a TrueNAS Core VM on the 2 SSDs and passed the NVMe and HDDs into that VM. Windows can still “see” them, but they are marked as offline, but Hard Disk Sentinel and CrystalDiskInfo can both see the SMART status of them (TrueNAS cant, weirdly…). Then, I have 7 of the 8 drives added to a single pool (one is failing so I left it out, this is for testing currently, anyway) and then the 5 NVMes are added to a second pool (the plan is to use the 4 Force MP510s or replacement drives as a single pool, then the other NVMe (or maybe even 2) as a Cache or Log for the Spinning disk pool).

So far, doesn’t matter if I am using the NVMe or HDD pool, but speeds from my Mac (with a 10Gb Thunderbolt adapter) are around the same… Might be a config issue, might be the odd NVNe drive slowing the rest down… but I am happy with the speeds so far… I have seen 3-400Mb/s Writes and 900+ Reads on both NVMe and HDD… Most of that is probably cached… the VM has 64Gb RAM given to it, and the test file was only 5Gb (BlackMagic Disk Speed Test). More testing is required though.

ZFS over multiple DVD/BD-R images

A couple of days back, I started thinking about archiving and backup software. I kind of have backups “sorted”, with my MacBook Pro using BackBlaze to backup to the cloud, Time Machine backing it up to my Synology, my VMs on Proxmox being backed up to Proxmox Backup Server off-site, my Synology and QNAPs being backup to B2 and Hetzner and some other bits and bobs… But for the Archiving stuff, I am not really set up… So, I went looking for archiving software. Couldn’t find anything, so asked on r/DataHoarder. Still no options, at the time of posting, but someone did reply with the idea of using DVDs (or Blu Rays) for ZFS...

Ok, that’s just crazy, but in a kind of a good way… kind of like the floppy RAID stuff I have seen… It does help with the storage of data, plus allows for potential loss of data… but it needs some automation to get it fully perfect…

Assuming you are using this for archiving, you could automate building 5 ISOs, just shy of 100Gb each, once a month, create ZFS ZRAID 2 or 3 (depending on how paranoid you are) and then write your data to it. ZRAID lets you lose 1 disk, giving you around 400GB of usage. Z2 brings that up to 2 losable disks, and 300Gb and Z3 is 3 disks and 200GB. I think Z2 would be your best bet, especially if you are using something like MDisk and are storing them safely.

Once finished, unmount and send an email saying you need to write the ISOs to disk. Label each disk with a unique serial number (this is where the archiving software would be handy) plus the set details and number (so, March 2023 Disk 1/5).

If you need something from that backup you stick it in the drives… You can do it with multiple drives, so with 5 disks and ZRAID, you need to mount a minimum of 4 of them. ZRAID2 needs 3 and ZRAID3 needs a minimum of 2… Ideally, you would want 5 of them, allowing you to check all disks (ZFS Scrub) and then get your files off.

A year of archiving would require 5 drives (say 100 quid a pop, USB makes things easier... Internal is possibly cheaper) and 60 disks (I Found 25 100Gb MDisks disks on Amazon for around 500 EUR) costing a total of maybe 2k, with 15 extra disks…

Follow-up questions:

  • Does ZFS allow the mounting of read-only?
  • Could you do this with Rewritable BluRay disks? Could they be mounted directly and written to? Leave them in the drives for the month, let writes do their thing and then archive them once a month? It’s archived, so it doesn’t need to be fast…