All is good! Photos, more details and benchmarks coming soon… stay tuned!
Finally 10Gb/s Networking!
Since GodBoxV3 had a few 10Gb nics, i needed to upgrade the network to support it. I ended up with a Ubiquiti Networks EdgeSwitch-XG. 16 ports (12 SFP+ and 4 RJ45). The SubperMicro board has 2xRJ45 ports. Due to lack of RJ45 ports, GodBoxV3 is connected to 1, GodBoxV2 is getting a 10Gb card soon, which will be connected to 1 port, and a new Sun Microsystems server (details below) will be getting the last 2… Of the SFP+ ports, 2 are connected to the EdgeSwitch Lite, 2 to the Synology (it got a 10Gig NIC reciently too!) and 2 to the new NAS (again, more details below!)
Good bye Mikrotik, Hello EdgeRouter 4
Since i was going all Ubiquiti gear (Wifi is Unifi gear) i got rid of the old Microtik and replaced it with a Ubiquiti ER4. Happy days! Got some plans for this, more details coming soon…
Moved lots of stuff around the room… Servers run cooler, and less noisy! happy days! I also got my hands on a very nice looking Sun Server X3-2. Its a Dual Xeon E5 (currently got quad cores, going to upgrade it to 8 cores) and i think its got 16GB ram and 4x300GB SAS Disks. It also has 4X10Gb nics! ESXi will probably go on here!
VMWare in the house
Up till recently, I ran Hyper-V all round. Its still on GodBox V2 and V3 (v1 has a HDD issue, so its off…), but the main VM hosts (the C6100’s) are being migrated to VMWare ESXi… Why? Its a learning exercise… We see how it goes…
So, long update… Any questions, comments, etc… shout!
So, a few days back I got an email from Packet.net about a promotion they and AMD where running. Essentially, they gave me some credit for their service (i am an existing customer) to play with one of their c2.medium machines. A c2.medium comes with an AMD EPYC 7401P which consists of 24 physical cores clocked at 2Gz with an all core boost at 2.8Gb and a max clock of 3Gz, 48 threads, 64GB ECC Memory, 2x120GB SSDs for boot and 2x480GB SSDs for main storage. It also has a 20Gb network link (2x10gb bonded) and can run pretty much any OS you can think of (Windows is not on the list officially, but you can boot off your own ISO, so you could probably get it on there… might not be supported, but it might be possible). all this for $1 per hour! And did i mention they are bare metal machines?
This was the perfect opportunity to play with the new AMD processors. My current and previoius generation workstations (GodBoxv1 and Godboxv2) are both running Intel Xeon processors. the machine previous to this, the mac pro, is also running a Xeon processor. But previous to both of them, my first 2 major workstations ran AMD… the first ran 2 AMD Athlon MP processors. These were old school processors that were single core, and i cant even remember their speeds, but i do know there were 32bit only and the machine maxed out at about 1.25GB RAM (well, i had it maxed out at that). the second AMD workstation ran 2 AMD Opterons… again, single core machines, but this time, they ran 64 bit and IIRC maxed out at 8GB ram. This was a limitation of the board, not the processor…
I have been thinking about GodboxV.next, and the AMD processors, specicially the Threadrippers and Epycs, are contenders for the next machine… so, this test allows me to check them out before i buy!Why would i say no?!
So, i spun a box up in New Jersey running Ubuntu 17.10 to play with it, and here are my findings…
First, i ran lscpu on the box to see what i was playing with:
I then ran ‘fdisk -l’ to see what disks i had to play with. on my machine sda and sdb where the 480gb SSDs, sdc was a 120gb that was empty and sdd was the boot drive… i installed the ‘btrfs-progs’ and then formatted sda and sdb as a RAID0 array, which i then mounted to /mnt. this gave me just under 900gb to play with…
So, my first test is the usual test: building the Linux Kernel. I know that this is something that the lads at ServeTheHome do a lot but its something i wanted to try my self… So, first i installed git and build essential, then bison, flex and ncurses-dev, then i cloned Linus' git repo at git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git. First things first: this machine has a twin 10gb link, a shead load of cores and some very fast storage. How long did it take to clone? it download 1.02 GiB at 35.32MiB/s (about 30 seconds and about 280Mbit/s) and all in, took 2 min 55 seconds to clone. I then ran time make -j 49 to see how long it would take… hmmm… no config file… make menuconfig and just hit save… defaults are grand… time make -j 49 again… and more errors… after a bit of googling, i find the page from Ubuntu showing what i need to do to build the kernel. i follow that… download a LOT more stuff using their instructions, and finally, we get to build… Time: 6 min 12 seconds… this is a FULL default build of the kernel…
Same build on a VM on GodboxV2 (which was given 32GB RAM and 16 thread, so a full Xeon E5-4620) took 8min 27s to clone (8.18MiB/s. or about 64Mbit/s) and 36 min to build… yea, that is 3x less cores, 2x less memory, slower storage (This is on Spinny Disk, not SSD), slower network and it is also a VM VS bare metal, still, to be essentially 6 times slower? interesting… I might, at some stage, boot the machine off a live Linux USB and run some more tests, but not tonight…
So, all this is because i was holding out for the main event… Photo processing… I wanted to do something “real life”, which for me would be development and photo processing… the kernel build gives an idea of a large project build built, the image processing gives an idea of multimedia work…
so, i devised a test: Export a bunch of photos (mix of photos taken on my 5Ds, 5D MKII, iPhone 6 Plus and iPhone 7Plus) that are stored in light room as full and run them though a basic .NET Core app i wrote. the code for the app is available here. The app fully utilises the machine by using multiple threads, and because its 64 bit, it will use as much memory as it can get its hands on. It just does some basic processing: open the file, resize to 1024X1024 and then save it… the 1024X1024 part is just a test… i was a bit under the gun on time, so couldn’t spend as much time working on it as i wanted to…
In total, there was 1546 photos exported, and the total file size was 15Gb. First obstacle was to get them uploaded to the Packet machine, which took a while (my upload speed is currently 40Mbit/s)… Once up, i downloaded a copy of dotnet core 2.0 SDK, cloned the repo with the project, built and ran… and man, its fast! 4 min 43 seconds. And it used all the cores.
Running the same code on GodBoxV2 on the bare metal (no VM this time), i got 17 min 35 seconds of a run… Now, GodBoxV2 has other things running in the back ground, but not that much… I also noticed that, on average, photos were being processed in 3-5 seconds on Epyc, but nearly 13-15, and sometimes 20 and 25 seconds on GodBoxV2. I also noticed that on Epyc, the dotnet process took nearly 45GB of RAM… to run… On GodBoxV2, it took over 70!
So, there you have it. Some starting tests with these processors. I am well impressed with these processors, and would have no issue getting one for the next GodBox… And with names like Epyc and Threadripper, why not?!
So, a few months back I bought myself an iPad Pro. I got a 10.5" with 64GB Storage and the Smart Keyboard. Since then, i have been mostly using it for playing around: watching YouTube, Netflix, surfing on the couch, etc. but i started to wonder how “Pro” this was…so i went and did some testing, and in the end nearly all of this post is being written on it…
Panic’s Prompt works well too… again, with ZeroTier, i can SSH into boxes and remote manage them. Handy for checking on docker instances…
Panic also have Coda for iOS. its a very nice (if somewhat expensive at $25) editor for the iPad. This post is being written on there now.
for Git stuff, i am using an app called Working Copy. Its free, to an extend, but if you need to do stuff like push changes, which is kind of important, then you need to pay a fee.
Coda and Working Copy work together with some magic built into Working Copy. It can act as a WebDav server, which Coda can the connect to. you open, edit, change and create docs, and Working Copy keeps note. then you swap to them and checkin. You need to have both apps on the screen at the same time (the docking feature works well for that) since iOS seems to kill some background tasks.
Unrelated to blogging, but i have also tried editing photos using Lightroom, and so far so good. I have used the Apple SD Card adapter to download 50MP photos (upwards of 60MB) from my Canon 5Ds quick enough, add them to Lightroom, make some changes and send them to Twitter, Facebook (not Instagram just yet…) and it works well. I have managed to hook it to my Gnarbox too.
keyboard takes a bit of getting used to. the Stupid “Global” button to swap keyboards (from
English to Emoji) is in the place you would expect to find CTRL. and CTRL, Option and Cmd (remember, this is a “Mac” style keyboard) and all shifted one place… I would have preferred if they moved that somewhere else, or removed it altogether…
a mouse would be very handy! I have tried pairing a bluetooth mouse to it, and no luck… it would be handy especially for editing documents and code, since touchscreen is “handy” some of the time, but not all the time…
So, there you have it. Blogging on an iPad. Would i give up my daily driver of my Surface Book and GodBoxV2 and just uses an iPad? Hell No… for basic stuff, it works well. Basic photo editing, blogging, surfing, etc, yes. But there is a reason my workstation has 16 processor cores and 160GB RAM: I need it. I have multiple copies of Visual Studio running, SQL Server, multiple VMs running different tasks, multiple web browsers, multiple monitors, etc. the iPad can do a good chunk of stuff, but not the major stuff… not yet. Don’t get me wrong: Word, Excel, Power Point, Outlook. all the major office tools work grand. But Visual Studio? SQL Management Studio? just not there yet…
So, what did i not do on the iPad, and ended up doing on the PC? Well, so far, nothing… using Coda and Working Copy, i wrote the text, previewed it and checked it into GitHub. Then, Prompt is being used to build it on my docker box, check into the static site and push, which will then publish. unless you see an update below, all went as planned and all was done on an iPad…
Both EC2 and Azure now have GPU enabled VMs to spin up and use
NVidia have released NVidia Grid cards, specifically for the “cloud” or “remote” desktop services.
Windows Continuum now allows you to use your Windows Phone with a keyboard, monitor and mouse, allowing you to replace your desktop. basically, plug your continuum enabled phone into a docking station, and use it as a Windows Desktop. Email, Browsing, Office Apps, etc. anything that supports UWP.
With all the increased bandwidth for mobile devices (4 and 5G, expanding wifi, etc) the idea of having your desktop live in the cloud is getting nearer… interesting times, my friend… interesting times…
[Update] Thinking a bit more about this, and if this was to work correctly, your phone could be everything required. Get up in the morning, check your emails on your phone, calander items, and news. head off, head to the coffee shop and plug your phone into a “laptop” style device like a NexDock and catch up on some emails, checking more news sites, etc. When you get into the office, plug your phone into the docking station and Remote Desktop in to your cloud desktop to do your development work, or whatever needs to be done. basic Office apps and Email can be run direct from the phone. When you get home, you can use the Microsoft Wireless Display adapter to watch videos on the big TV, or show web pages. And if your cloud desktop is available outside of your work network, you could work anywhere also…
First on the list is my older Ubiquiti Edgerouter POE. It currently in the process of being decommissioned, or used for something else. It was the main edge router for the network: it had both internet connections connected, and did routing, firewalls, etc, but with the Proliant taking over as a router, it is not required as much any more… Its still on, mainly because its still a DHCP server, but not much else.
There are 2 Meraki MS220-8 switches next. GodBox1 and Godbox2 both connect in here, and are bonded, as is everything else on the network. The MS220-8 has 8 GigE ports, but also has 2 SFP ports. I bought 4 SFP Ethernet adapters and have a short calbe running between the switches. That uplink is also bonded. All going well so far!
All Meraki hardware can be managed though the Meraki dashboard. check out their site for more details and examples of how to use it.
I bought one of the MS220’s from eBay a few months back, and loved it. Then i realized that you can get your hands on free gear, the MX64, an MS220 and a Wi-Fi Access point if you attend their webinars. Terms and conditions apply, but check them out!
I have 2 Ubiquiti UniFi APs, one in the front of the house, one in the back. They are connected to one of the MS220’s, but dont work with its POE (maybe the EdgeRouter could do that, since its POE…) so there are injectors for them. Anyway, the network ports on there are VLANed to the MX64 (more on that later) and the default traffic is going to a management VLAN.
The MX64 has a static internal IP on my DMZ network, and uses the Proliant as an upstream connection. Upstream on the Hetzner server, all traffic coming from the MX64 ip uses one of my /29 ip block. all traffic to that ip is also forwarded directly to the MX64.
I has 2 small, unmanaged switches (a cheap 8 port Linksys and a 8 port TP Link) which are used for separate things: the Linksys has 4 Raspberry Pi’s, which run a GlusterFS cluster, plugged into it and the TP Link connects to my printers.
I also have a Mikrotik CRS226-24G-2S+IN which has 2 10Gbit SFP+ Ports, and plan on using this for higher speed networking soon, aswell as a Cisco 48 port 3560 which also has 4 SFP ports (GigE) and may come in handy for something soon…
So, thats the network currently. any questions, please leave a comment.
So, after some messing, tweaking, and thinking, I have made some progress with the home lab… or at least broken some stuff… I mentioned previously that i had a Ubiqititi networks EdgeRouter POE in the home lab. Originally, the plan was to use a Virtual PFSense box for my core router… Given the power usage of the current PfSense Box (I have 2 MPower Pro’s watching power in the lab) I am now thinking of moving to just the EdgeRouter for, well, edge routing… below is the usage of the ProLiant for the last 12 hours or so:
for the same period, here is the usage for the Edge Router:
I am also setting up a DMZ for front facing services, and then a LAN for inside facing machines. There will be a firewall (currently thinking Sophos UTM or similar) between the DMZ and the network. Some machines will be able to access the DMZ, and there may be machines allowed into the LAN, but only some things… not even sure if that would be done…
I also need to work out the VLAN side of things. I have currently though of the following VLAN setup:
WAN 1 (connected directly to the Cable modem)
WAN 2 (again, direct to cable modem)
IOT (stuff for running the house, like Nest, the MPower devices or the like)
Media Network (Plex, Roku, Apple TV, Chrome Cast, etc. Not sure if i need to separate this, but it might be done…)
The current Cisco 3560G switch should do all that, without problems, so no new switch needed… lets see what i can break over the next while…
After a few months of running Sabayon Linux on the GodboxV2, i am going back to Windows Server. Back around October of last year, i installed Windows 10 Preview on the GodBoxV2, and, well, there where issues with graphics drivers, etc. Then, some time after, i cant remember off hand when, i moved to Sabayon Linux. Its based on Gentoo but has a lot of the components pre-built. Gentoo is a “Build from scratch” sort of OS. You get a basic kernel and a basic set of components, but you build everything else from scratch… including rebuilding the kernel if you want. Sabayon, on the other hand has all that mostly prebuilt, though you can still use Gentoo’s Portage to build stuff yourself.
Anyway, for the last few months, all was going mostly well… but I miss Windows. And, given i have pretty much always ran a server OS on my main workstations, I am heading back to Server 2012R2. I was tempted by 2016, but its still very early days… Maybe i will run it as a VM for a while, but we will see…
total of 12 Gigabit network cards (of which 4 are currently used…)
I have been playing with some networking in the house and have managed to build some VLANs. The modems are connected both directly to the Router
and to a dedicated switch port for a given VLAN. The plan for the upgrade, which i hope to complete sooner than the rest of the network is as
get the ML110 running ESXi and visualize PFSense. Give it 2gb of RAM and some processor.
take some of the network cards out of the box. It does not need 12 ports, but maybe leave the 2 quad ports in there. They should be connected
to the main switch trunked. 8 may be overkill, but i never do things by half.
the PFSEnse VM should be connected to all 3 WAN VlANs (900, 901 and 902) and should also have at least one port to the LAN. There may also be
other ports for other internal VLANs.
with the spare processor, i can then add other (small) VMs to this machine.
So, in my last post i talked about the requirements for the home lab, and in this post, im going to talk about a few more updates i have made in the last few weeks.
First, the processors: in the first post, i talked about Xeon D or Xeon E3… Well, i missed one… The Xeon E5. I have 2 of these in GodBox 2, and you can get them into a microATX board. There does seem to be some limits with the microatx boards, but hopefully enough searching will find me what i am looking for. Ideally, i want it to take “normal” DDR3/4 memory (not SODIMMs like the ASRock one above) and also take enough of them to run 64 or 128Gb of ram (thinking 8 would do the job!). Also, i would like to have 4 GigE ports onboard and 1 management port. 4 onboard is not a hard requirement: If i can get one with 2 ports, i can always get a 4 port card for the PCI-Express slot… Finally, i would like it to have at least 6 SATA ports and possibly an MSATA port. Thinking Boot off MSATA (Windows Server 2016 Nano Server would be used), 2 SSDs and 4 HDDs. Using Storage Spaces, use the 2 SSDs as “Fast” storage for the pool.
I also think i moved off the idea of 10Gb. I like the idea of it, but given a small 10Gb switch costs upwards of a grand, and the plan is to build a machine for that price, i would prefer a fifth machine and using my existing Cisco 48 port switch and leave 10Gb as a future upgrade.
Also, changed from last time round is machine count. Originally i was saying 3-4 machines… now i am thinking 6-7… 5-6 of them should be Hyper-V boxes and the last one would be a Media Box.
I also think the Synology or SAN requirement is out… Hyper-V can be setup to do replication between hosts, and with a 4Gb link to the LAN, i think i should be OK. Also, if i have the media box separate, i should be ok there too. I will detail the media center in a later post.
So, any suggestions or thoughts on what should and shouldn’t be looked at?
So, its that time again… HomeLab upgrade time… Or at least the planning for it. I am in the process of rebuilding my home lab, which involves pull all old servers out of the rack and replacing them with new ones… It also means rewriting the network, possibly upgrading some existing gear and hopefully getting the whole lot done on a budget of some sort…
So, why? Well, biggest reason for all this is currently heat and power usage. We use about 4-6x more electricity than the average house here in Ireland, which means our electricity bill is fairly high. It also means that the lab, which is also my office/bedroom, gets quite warm and uncomfortable during the summer month. There is an Air-Con unit in the room, and, well, that’s costing the most on electricity!
So, what I got is a basic overview of what I want from the homelab and hopefully in the next post, I will have an idea of what it will look like..
3-4 machines running a Hyper Visor (HyperV, VMWare ESXi or other). Leaning more towards Hyper-V purely because its what I got currently and its what we use in our main office.
each machine should be connected to at least 2 networks: one for storage and migration, one for “public” to the LAN. There may be more VLANs for other networks, but 2 is a start.
ideally, 10Gb connections would be nice, but multiple 1Gb connections would also work.
shared storage (iSCSI, SMB3, etc) would also be a nice to have, but may bump up the server count (not actually a problem) but would increase power and cooling costs. An off the shelf box, like a Synology could do the job…
Lower power usage and less heat produced is also a major requirement. Most of the boxes I am decommissioning are older Xeon hardware (5000 series upto a 5200 series process and even an older Xeon P4!). The newer Xeon E3 and the even newer Xeon D are a lot more efficient, use less power, produce less heat and are way faster than what I currently have. The E3 can use up to 32Gb of RAM and the Xeon D top out at 128Gb… Me being me would like more than 32Gb RAM… 🙂
smaller machines would also be nice. I have been looking at both Xeon D and Xeon E3 Mini-ITX boards and cases for them. I do have a half height Dell Rack, which I host these machines, and ideally, these machines should be rack mountable, but micro ATX cases could work. 2 per shelf would work grand.
Onboard IPMI and KVM support is something I want too… I do have a KVMoIP switch in the house, and it works, most of the time, but getting a box that has this embedded into the board would be ideal… A lot of the server boards had it as standard or allowed it to be speced, so that’s all good.
I am also thinking of upgrading the router to a similar spec board… Possibly a Xeon E3, or even an i5…. Ideally it should have IPMI and KVMoIP on board and should produce less heat. Biggest issues is getting enough network cards into the box…
These are my requirements at a high level overview. Over time things may change, but lets see how we get on…