Tiernan's Comms Closet

Geek, Programmer, Photographer, network egineer…

Zerotier and Minio Followup

May 5, 2017
0 Comments

in a previous post, I talked about setting up a distributed S3 like data storage system using Minio and ZeroTier. Well, this week, the ZeroTier guys tweeted about this.

A few people then started asking questions, and looking for a follow up, so here it is…

First, a quick recap. I had 4 machines, all running Linux. Three of them were in 1 time zone (GMT+1) and one was in another (GMT). Looking at the Distributed Minio Quickstart Guide again, there is a mention of times being in sync… which is probably why this did not work as planned… and by “not work as planed”, I mean that Minio would crash, or not be responsive, or not write data in the place it should have… which was a pain. But looking at the documentation again, they do mention that Windows support is “experimental” which means, hopefully, some day it will be not so experimental, and might work… Given that most of my machines in house are Windows boxes, this would be a nice feature.

Now, what about ZeroTier? Given they posted it to their twitter? Well, it worked. it did the inter connect stuff well, and, given bandwidth limitations on a home broadband connection, it was still quite fast.

So, the question is, how fast? Well, on my Surface Book on a WiFi connection in the house, behind a Meraki MX64 firewall, connecting to the GodBoxV2 over FTP though ZeroTier, i get the following result:

the same download over FTP direct (no ZeroTier) does the following:

So, direct over FTP is faster… in this instance by about 70%, but, over the download, it did get slower (seen it hit 12 at one stage) and because its over WiFi, those are a bit wonky…

I did get one last screen shot:

as you can see, the Zerotier network adapter is showing 77.3Mbps, but the main network adapter is showing 80.8Mbps. There would be other traffic there, but if we assume there is nothing but ZeroTier traffic being sent, there is about 5% of an overhead.

So, to wrap up: Minio and its distributed storage system over ZeroTier needs more testing. Ideally, all hosts need to be in the same time zone, or at least have the same time… Will try work on that soon. As for ZeroTier? I am extremely happy with them. Its fast, easy to setup, and easy to configure. What more could you ask for? Oh, and free, unless you need a pro account!

Business Class Broadband… finally here….

March 9, 2017
0 Comments

So, after many (MANY) years messing with dual cable modems, struggling to get them working together, to get websites to even allow me in, having to use hacks and kluges to get it to work at all… I have given up. It has been a struggle getting two modems working properly. Load blanching kind of works… but it’s messy at best. Some sites kick you out every now and again because your IP changes. Some sites wont let you login at all… Mind you, some sites work grand and don’t ask questions…

And the whole idea of multiple modems, to allow you to download things faster, doesn’t work for everything… Anything you download in the browser is single threaded, so its limited to one modem… you can use use download accelerators, and they do work, but its an extra step, and some sites don’t work for that either (MSDN for example).

So, i have given up, bit the bullet, and moved to business class broadband from Virgin Media. It’s actually cheaper than the two residential lines i had, but it is also slower than the two combined: previously, it was two x 360/36mb/s. Now i am am on a single 400/40mb/s modem. That being said, there are definite advantages:

  • Static IPs pretty much as standard, and option of either one or five (no unbeaten!) Guess which one i went with? Its technically a /29 range, but the first usable IP is given to the modem, which acts as a gateway, so i end up with five usable.
  • Proper business class SLA. Any issues, someone who knows what they are talking about can help
  • Phone lines on a separate modem. So, i got phone lines with them, and they give a separate modem for those lines, so as not to interfere with the internet. that modem has no internet connection and is just for calls. They are also working on a VoIP/SIP offering, which is something i am interested in.
  • Guaranteed speed! They guarantee a minimum speed to the modem at all time. Business customers have a priority on the network, which is nice. And, during testing, so far, i am getting the advertised speed most of the time. I needed to download a Windows 10 ISO yesterday from the MSDN, and it came in at between 45 and 48MBytes/s!

So, only had it installed a week, and so far, so good. I have one IP given to my PFSesne box, and the rest given to a VyOS VM. The plan is to use the VyOS box for all network traffic, but first i need to do some testing and learning… Expect some posts on this soon!

Distributed S3 data storage using Minio (and Zerotier)

January 19, 2017
0 Comments

So, something i have been looking into in recient times has been Distributed Storage, and, more specifically, how to use the storage in my many, many machines to protect data, and also increese my usable space… There are a few projects on the market that do this (Ceph, NooBaa and Gluster all spring to mind) but some are more painful to setup than others… which brings me nicely to Minio. Minio is a 20ish MB executable you download from their site, mark it as executable (on Linux or Mac Boxes) and run… and you have yourself a S3 compatable storage server… Simples!

“But Wait!” i here you screem! “thats not distributed!”. Well, yes… but, it can be! Their Distributed Quick Start Guide, which is where i started with this, allows you to run a distributed copy of your data. I will let their documentation explain more, but this is what i did:

  • download the minio server (single executable file) on a minimum of 4 machines.
  • on each machine, run a command like the following:

replacing accesskey and secretkey with keys (check minio documentation to get these) and foldertoexport with, well, the folder you want to export!

For me, i have 4 servers currently clustered. 2 are in online.net (one in Paris, one in Amsterdam), 1 in OVH.NET (France, somewhere) and one in Dublin (GodBoxV2 currently). They are all interconnected using ZeroTier (I will explain that later) and so far, so good… only ran some basic tests, but with it, i could loose 2 machines and still have data… Not bad for free! I will run some speed tests soon.

Docker Jekyll and Mr ngrok

November 15, 2016
0 Comments

See what i did with the title?! Anyway, in my last post, i explained how i was building this site with Docker running on Windows 10 with the Anniversary update. Today, i am going to show you how to host it using Nginx and ngrok.

So, first, you should know what Nginx is at this stage… If not, check out their site. Next ngrok is basically a way of tunneling your localhost to the web. So, how do we build the whole lot together and serve your site to the internet? Well, this is what i have so far:

First, build your site in jekyll. for me, the command is

docker run --rm -v "$(pwd):/src" -w /src ruby sh -c 'bundle install --path vendor/bundle && exec jekyll build -s www.tiernanotoole.ie/ -d www.tiernanotoole.ie/_site/'

next, run an nginx server with that output folder:

docker run --name tiernanotoolenginx -v "$(pwd)/www.tiernanotoole.ie/_site/:/usr/share/nginx/html:ro" -d -p 8881:80 nginx

the docker container is called tiernanotoolenginx, since i could have multiple ones, and port 8881 is being redirected to port 80 on that container, but technically, it might not be needed due to the next command:

docker run --rm -it --link tiernanotoolenginx wernight/ngrok ngrok http tiernanotoolenginx:80

essentially, what we are doing here is running ngrok and pointing it at post 80 on the nginx container… you see i did not point at 8881, since we are using the continer directly… it might be different if you were not…

when that command runs, you get a screen telling you the URL of your site with some basic stats. your site is now hosted publically, via an ngrok tunnel! you could run that container as a daemon, and leave it running, but for me, i wanted to do some minor testing, so i can kill it when i want…

So, all is good with the world!

Building Jekyll sites with Docker on Windows

November 2, 2016
0 Comments

As some of you probably know (or based on the footer of the site) this site is built with Jekyll. Jekyll is a static web site builder, written in Ruby, and is a bit of a pain to build on Windows. Earlier on this year, I wrote up a post explaining how to use Jekyll on Windows using Bash on Ubuntu on Windows… It was a bit complicated, and, well, worked a few times, but was not too successfull… So, were do we go next? Well, Docker to the rescue!

I am running the Windows 10 Anniversary edtion witch has container and docker support. using the repo for this site and the scripts (specifically build-tiernanotooleie and geekphotographer.com) i can build the docker site on my local Windows machine and upload the sites as required (I host on NFSN and upload via RSync). The docker image i build from is a Linux docker image, do i need a Linux container running (and the docker tooling). I also use Bash on Ubuntu on Windows to upload using RSync. All is going well so far…

Cloud Desktop becoming a reality

September 14, 2016
0 Comments

I have talked about the theory of the “Cloud desktop” twice on my older blog (Rackspace’s Hosted Virtual Desktop and More on the desktop in the cloud) way back since 2011. Since then, a few things have changed:

With all the increased bandwidth for mobile devices (4 and 5G, expanding wifi, etc) the idea of having your desktop live in the cloud is getting nearer… interesting times, my friend… interesting times…

[Update] Thinking a bit more about this, and if this was to work correctly, your phone could be everything required. Get up in the morning, check your emails on your phone, calander items, and news. head off, head to the coffee shop and plug your phone into a “laptop” style device like a NexDock and catch up on some emails, checking more news sites, etc. When you get into the office, plug your phone into the docking station and Remote Desktop in to your cloud desktop to do your development work, or whatever needs to be done. basic Office apps and Email can be run direct from the phone. When you get home, you can use the Microsoft Wireless Display adapter to watch videos on the big TV, or show web pages. And if your cloud desktop is available outside of your work network, you could work anywhere also…

double speed Internet Part 9 – Going Back

July 20, 2016
0 Comments

[NOTE] This part 9 in a series of posts. The rest can be found here

Well, the double internet experiment is about ready to be finished… After 9 posts, 4 months, lots of sweating, many painful nights trying to figure out why something stopped using, shouting when Netflix did not work, wondering why my internet connection was so slow, and many, many other problems, i have decided to wind down the project. in the last 9 posts, i have learned a lot, and i hope i have helped someone figure out some stuff on their end. Even though this is a wind up of the project, there are still new things i have to share.

  • I found another project that has potential for speeding up the internet: VTrunkd. after some testing, i does seem to manage to speed up the connection, but either limits on hardware i have in house, or limits of hardware in the cloud, or even the software, stopped me in my tracks… i did see 400mb/s out of it at one stage, using 200mb/s from each modem… its close, but its not the full 720…
  • messing with Quagga/Zebra as mentioned in the previous post has been, well, interesting… I did manage to get all OVH traffic sent though their server, Digital Ocean traffic sent over that box, and everything else over Hetzner. I added an Azure box to the mix for a while, aswell as a Vultr box, but it got very messey, very quickly. if i had something automated, it would be better.
  • the idea of having a /29 IP range in Hetzner and forwarding it though the tunnels back to the house did work. My Meraki MX64 had one IP address, i had a mail server on a second, everything else on a third, and was planning on using more… but its just, well, again, messy. So, i will be going back to the idea of 2 IP addresses, and hoping whatever i put infront of the network can figure stuff out…

So, what am i moving to? well, thats a question… Currently, i have the Meraki MX64 plugged directly into the modems, and protecting my LAN. So far, so good, but due to hardware limits, it maxes out at around 260mb/s. So, thats out of the question for the main network! I did at one stage have Sophos UTM Home edition running. Sophos also have their XG firewall available for home use, so i might try that… There is also PFSense which i used before also… And there may be more… Maybe there will be a new series reviewing these home firewalls? we will see…

Meraki and Ubiquiti networks gear Update

June 20, 2016
0 Comments

In part 6 of my Double Internet Series I mentioned i was running a Meraki MX64 in the network, and said i would write up about it. I am taking this opportunity to also write up about the Ubiquiti networks gear in the house also.

  • First on the list is my older Ubiquiti Edgerouter POE. It currently in the process of being decommissioned, or used for something else. It was the main edge router for the network: it had both internet connections connected, and did routing, firewalls, etc, but with the Proliant taking over as a router, it is not required as much any more… Its still on, mainly because its still a DHCP server, but not much else.
  • There are 2 Meraki MS220-8 switches next. GodBox1 and Godbox2 both connect in here, and are bonded, as is everything else on the network. The MS220-8 has 8 GigE ports, but also has 2 SFP ports. I bought 4 SFP Ethernet adapters and have a short calbe running between the switches. That uplink is also bonded. All going well so far!
  • All Meraki hardware can be managed though the Meraki dashboard. check out their site for more details and examples of how to use it.
  • I bought one of the MS220’s from eBay a few months back, and loved it. Then i realized that you can get your hands on free gear, the MX64, an MS220 and a Wi-Fi Access point if you attend their webinars. Terms and conditions apply, but check them out!
  • I have 2 Ubiquiti UniFi APs, one in the front of the house, one in the back. They are connected to one of the MS220’s, but dont work with its POE (maybe the EdgeRouter could do that, since its POE…) so there are injectors for them. Anyway, the network ports on there are VLANed to the MX64 (more on that later) and the default traffic is going to a management VLAN.
  • The MX64 has a static internal IP on my DMZ network, and uses the Proliant as an upstream connection. Upstream on the Hetzner server, all traffic coming from the MX64 ip uses one of my /29 ip block. all traffic to that ip is also forwarded directly to the MX64.
  • I has 2 small, unmanaged switches (a cheap 8 port Linksys and a 8 port TP Link) which are used for separate things: the Linksys has 4 Raspberry Pi’s, which run a GlusterFS cluster, plugged into it and the TP Link connects to my printers.
  • I also have a Mikrotik CRS226-24G-2S+IN which has 2 10Gbit SFP+ Ports, and plan on using this for higher speed networking soon, aswell as a Cisco 48 port 3560 which also has 4 SFP ports (GigE) and may come in handy for something soon…

So, thats the network currently. any questions, please leave a comment.

double speed Internet Part 8 – Routing Around

June 8, 2016
0 Comments

[NOTE] This part 8 in a series of posts. The rest can be found here.

At the end of my last post I asked the question about routing traffic to different servers based on thier distances, etc… Well, after a bit of messing, i can say it kind of works! here is a quick over view:

  • server in the house has now got multiple OpenVPN connections (2 to Hetzner, 1 to OVH (with a plan to double), 1 to Digital Ocean (again, to be doubled) and i am planning 2 to Azure as well).
  • Quagga/Zebra has static routes (currently static, planing on dynamic soon… more eventually) to different servers depending on where they are. for example, all traffic to the hetzner network (including their Storage Boxes) go though the hetzner link. Hubic traffic goes though OVH, Azure (currently) and AWS traffic, aswell as some CDNs go direct over either WAN1 or WAN2 in the house, and some other stuff (CrashPlan currently) goes though Digital Ocean. Everything that has no static route goes though Hetzner…
  • Ideally, the static side of things should be removed, and a more dynamic setup done. How that works, i have no idea… Spotify have 2 posts about their SDN Internet Router (part 1 and part 2) which is an interesting idea… More digging and research is required.

So, there you have it. Everything currently seems to be working, mostly, and tweaks can be made easily… I have a couple posts i have in my head, including something to do with automating bringing up new machines (probably with Ansible or something like it), more monitoring, and some other stuff too… Any questions, leave a comment, and i will get back.

[UPDATE] I wrote a quick and dirty app called WhoIsToZebraConfig which takes an AS Number, looks up the info in the Merit RADb (with the help of some code from Coder Buddy) and outputs what you need to put into your Zebra Config… should save me some time, and it might save you time too… shout if you have questions!

double speed Internet Part 7 – ECMP (kind of)

May 31, 2016
0 Comments

[NOTE] This part 7 in a series of posts. The rest can be found here.

In the last post I mentioned I am now using Hetzner for hosting a dedicated box. Thats still live, and going well. I have a /29 IP range (6 usable) and also 2 other IPs. So far, so good… But because i was using a Socks Server, I was not fully able to use the /29 ips… I use something like as follows:

essentially, for each public IP i have that i want to map to an internal IP, i have a POST and PRE ROUTING rule, plus the required forward rules… But, if socks are used, then that goes out the Window, since TCP traffic will look like its coming from the socks server… So, i killed the socks server, removed the IPTables rule, and then realized that while outgoing traffic was being balanced somewhat (2 default rules on the internal box pointing at the OpenVPN IPs from the Hetzner box) incoming was a problem. Hetzner knew how to get to my internal network, but only though one ip… enter Quagga and Zebra…

Quagga is a routing software suite, which can do protocols like OSPF, BGP and RIP, and Zebra is the component that does static routing. using their documentation on static routes, I created a static route to my internal network with 2 next hops, the OVPN IPs from the internal box… and, after restarting Quagga, all works! happy days! now i can forward ips from outside the network to inside the network correctly, and they look like they are the public ip!

So, whats next then? well, I now have a server in Germany (Hetzner) and one in France (OVH), and can spin one up in the UK or the US (Digital Ocean). Given that i have Quagga running on the box, i am now thinking of trying to see if its possible to route traffic depending on distance or something similar… If i am trying to hit a server in Hetzner’s DC, i should go though Germany. If its in Digital Ocean, go though either US or UK servers, same with OVH. Then figure out who has the fastest links to, say, Amazon, Azure, Netflix, BBC, Dropbox, etc, and add either static or dynamic links to the router… essentially, thats the theory… lets see how that works…

  • My Mastodon Instance

    Buy Me A Coffee

  • Recent Posts

  • Archives

    • DevDmBootstrap4 Created by Danny Machal (DevDm.com)