Tiernan's Comms Closet

Geek, Programmer, Photographer, network egineer…

Currently Viewing Posts in Internet

Connecting to my car over ZeroTier

I use ZeroTier on my network for a good few things, including internal network peering between BGP VMs, management of machines, and now, connecting to my car over LTE. This is one of those posts that sounds silly, but is very handy! First, the parts list:

  • Car…
  • 3G/4G/5G modem of some sort. I am using a Huawei Wingle… Can be used without the Router below, but I wanted Zerotier, so I have it in modem only mode…
  • A router that supports Zerotier. I am using a modified TP-Link TL-WR703N upgraded to 16MB ROM and 64MB RAM. This is required for newer OpenWRT builds
  • a dashcam that connects over Wifi. I am using a BlackVue DR750S-2CH
  • Latest ROOter software from Of Modems and Men
  • Patients…

After installing the the latest copy of ROOter on the TPLink (or router of your choice) and getting the modem configured correctly (this took a while) you need to install the Zerotier software though the dashboard. Once installed, I joined my Zerotier network using the CLI (SSH into the router) and the approved it though the my.zerotier.com dashboard. Once its approved and connected, you can now go to the Zerotier IP and get to the router directly. From here, you can either setup a route in Zerotier to point at the internal network behind the router, or, in my case, setup a  SSH tunnel to the dashcam. I found the IP given to the dashcam and used SSH forwarding to get to it. Finally, i used the URLs from Digital-Nebula’s hackview repo to get to the different URLs. I use this to download stuff like GPS logs, emergency videos, etc. I have to clean up some scripts at some stage for this, and plan to upload them at some stage.

If anyone has any questions, leave a comment!

AS204994, Own IP Space and Anycast

So, if you are reading this page, it is being delivered with the magic of Anycast… Well, technically, it was before, since i used Cloudflare, and it still is because of Cloudflare, but also because of my own ASN (As204994), some servers in different locations, and some magic, which i will explain a bit of in this post.

This all started late last year when i got my hands on an ASN and a /48 block of IPv6 addresses. I had been reading stuff about BGP, routing, etc, and decided to go all in. it was quite cheap with the help of HostUS. All in, it was about $50 for the year. As part of the process, i needed 2 upstream providers to say they would accept my announcement. They were Hurricane Electric though their Tunnel Broker service, and Vultr using a few of their VPSs.

After i got my space and ASN, i started to announce the V6 addresses over Vultr and Hurricane Electric, and all was good. I had 2 Vultr servers: 1 in London, UK, and one in New Jersey, USA. I had my home machine announce to HE, and then also link to both Vultr servers using Zerotier. All worked well, but due to some family issues, i never got around to putting it into production… till now.

Those 3 servers now share an IPv6 address on the loop-back port. When you (well, Cloudflare) asks for that IP, the closet (network) with that IP responds, and the NGinx server on that box sends back the contents of the site. This site is hosted on each box, since its fully static, but both AS204994 and TiernanOToole.net are hosted in Ghost, so Dublin (my machine in the house) serves them, and both Lon1 and Nyc1 do proxying. so, most requests from the US are hitting the box in NYC and the ones in Europe share either Dub1 or Lon1. I have some tweaks to do with which servers will be running where, and may add more, but currently its working well.

So, how do you figure out what server responded? Simple. Open your Dev tools on your browser, go to network tab, refresh, and see the response headers for anything on this domain. You should see something like below.

Over the next while, i will be updating tiernanotoole.net with more details on how this works, and more stuff will end up on AS204994.net too. If anyone notices any weird and wonderful issues, shout. If you have more questions, shout.

Business Class Broadband… finally here….

So, after many (MANY) years messing with dual cable modems, struggling to get them working together, to get websites to even allow me in, having to use hacks and kluges to get it to work at all… I have given up. It has been a struggle getting two modems working properly. Load blanching kind of works… but it’s messy at best. Some sites kick you out every now and again because your IP changes. Some sites wont let you login at all… Mind you, some sites work grand and don’t ask questions…

And the whole idea of multiple modems, to allow you to download things faster, doesn’t work for everything… Anything you download in the browser is single threaded, so its limited to one modem… you can use use download accelerators, and they do work, but its an extra step, and some sites don’t work for that either (MSDN for example).

So, i have given up, bit the bullet, and moved to business class broadband from Virgin Media. It’s actually cheaper than the two residential lines i had, but it is also slower than the two combined: previously, it was two x 360/36mb/s. Now i am am on a single 400/40mb/s modem. That being said, there are definite advantages:

  • Static IPs pretty much as standard, and option of either one or five (no unbeaten!) Guess which one i went with? Its technically a /29 range, but the first usable IP is given to the modem, which acts as a gateway, so i end up with five usable.
  • Proper business class SLA. Any issues, someone who knows what they are talking about can help
  • Phone lines on a separate modem. So, i got phone lines with them, and they give a separate modem for those lines, so as not to interfere with the internet. that modem has no internet connection and is just for calls. They are also working on a VoIP/SIP offering, which is something i am interested in.
  • Guaranteed speed! They guarantee a minimum speed to the modem at all time. Business customers have a priority on the network, which is nice. And, during testing, so far, i am getting the advertised speed most of the time. I needed to download a Windows 10 ISO yesterday from the MSDN, and it came in at between 45 and 48MBytes/s!

So, only had it installed a week, and so far, so good. I have one IP given to my PFSesne box, and the rest given to a VyOS VM. The plan is to use the VyOS box for all network traffic, but first i need to do some testing and learning… Expect some posts on this soon!

double speed Internet Part 8 – Routing Around

[NOTE] This part 8 in a series of posts. The rest can be found here.

At the end of my last post I asked the question about routing traffic to different servers based on thier distances, etc… Well, after a bit of messing, i can say it kind of works! here is a quick over view:

  • server in the house has now got multiple OpenVPN connections (2 to Hetzner, 1 to OVH (with a plan to double), 1 to Digital Ocean (again, to be doubled) and i am planning 2 to Azure as well).
  • Quagga/Zebra has static routes (currently static, planing on dynamic soon… more eventually) to different servers depending on where they are. for example, all traffic to the hetzner network (including their Storage Boxes) go though the hetzner link. Hubic traffic goes though OVH, Azure (currently) and AWS traffic, aswell as some CDNs go direct over either WAN1 or WAN2 in the house, and some other stuff (CrashPlan currently) goes though Digital Ocean. Everything that has no static route goes though Hetzner…
  • Ideally, the static side of things should be removed, and a more dynamic setup done. How that works, i have no idea… Spotify have 2 posts about their SDN Internet Router (part 1 and part 2) which is an interesting idea… More digging and research is required.

So, there you have it. Everything currently seems to be working, mostly, and tweaks can be made easily… I have a couple posts i have in my head, including something to do with automating bringing up new machines (probably with Ansible or something like it), more monitoring, and some other stuff too… Any questions, leave a comment, and i will get back.

[UPDATE] I wrote a quick and dirty app called WhoIsToZebraConfig which takes an AS Number, looks up the info in the Merit RADb (with the help of some code from Coder Buddy) and outputs what you need to put into your Zebra Config… should save me some time, and it might save you time too… shout if you have questions!

double speed Internet Part 7 – ECMP (kind of)

[NOTE] This part 7 in a series of posts. The rest can be found here.

In the last post I mentioned I am now using Hetzner for hosting a dedicated box. Thats still live, and going well. I have a /29 IP range (6 usable) and also 2 other IPs. So far, so good… But because i was using a Socks Server, I was not fully able to use the /29 ips… I use something like as follows:

essentially, for each public IP i have that i want to map to an internal IP, i have a POST and PRE ROUTING rule, plus the required forward rules… But, if socks are used, then that goes out the Window, since TCP traffic will look like its coming from the socks server… So, i killed the socks server, removed the IPTables rule, and then realized that while outgoing traffic was being balanced somewhat (2 default rules on the internal box pointing at the OpenVPN IPs from the Hetzner box) incoming was a problem. Hetzner knew how to get to my internal network, but only though one ip… enter Quagga and Zebra…

Quagga is a routing software suite, which can do protocols like OSPF, BGP and RIP, and Zebra is the component that does static routing. using their documentation on static routes, I created a static route to my internal network with 2 next hops, the OVPN IPs from the internal box… and, after restarting Quagga, all works! happy days! now i can forward ips from outside the network to inside the network correctly, and they look like they are the public ip!

So, whats next then? well, I now have a server in Germany (Hetzner) and one in France (OVH), and can spin one up in the UK or the US (Digital Ocean). Given that i have Quagga running on the box, i am now thinking of trying to see if its possible to route traffic depending on distance or something similar… If i am trying to hit a server in Hetzner’s DC, i should go though Germany. If its in Digital Ocean, go though either US or UK servers, same with OVH. Then figure out who has the fastest links to, say, Amazon, Azure, Netflix, BBC, Dropbox, etc, and add either static or dynamic links to the router… essentially, thats the theory… lets see how that works…

double speed Internet Part 6 – Hetzner Edition

[NOTE] This part 6 in a series of posts. The rest can be found here.

Its been a while, since I posted, and there are some, well, pretty major changes since the last time… Lets start are the beginning.

Last time I was using Digital Ocean for my hosting provider. I was using their $20 a month server (2 cores, 2GB RAM, 40GB SSD, 3TB transfer), and it was all good… But I noticed that every now and again I would need to reboot the box. I also noticed that when transferring large files or using higher bandwidth (400mb/s+) the 100% of both cores were being used. So, I wanted to move to something with more power…

I also was limited to IP addresses. Yes, Digital Ocean do offer IPv6, but I could only get 1 IPv4 address… and I wanted more…

So, I went back to some old friends of mine, Hetzner, and bought a dedicated server with a Quad Core Xeon E3, 32GB RAM, 4*3 TB HDDs, 1Gbit/s network connection with 30TB transfer per month and a KVMoIP plugged in. I also got 2 extra standard IPs and a /29 (8 IPs, 6 usable). I will explain that next. I installed Debian 8.4 on the first disk, and I am planning on using the other 3 for storage of some sort. I then installed the MPTCP kernel, OpenVPN, Squid and a Socks server (same as the Digital Ocean box) and reconfigured the home machine… All good! Now when browsing the web, everywhere thinks I am in Germany, but so far, so good… Speed tests are about the same, but I have my theory about ECMP to try this weekend.

Because of the extra IPs, I am working on doing full IP forwarding, not just port forwarding. One of my IPs in pointing directly at my Meraki MX64 in house (a post on the Meraki stuff is coming, eventually…) and another at the Proliant box, and I plan on pointing other IPs at machines in the DMZ or a firewall of some sort. the /29 is routed though the IP pointing at the Proliant, so that makes life easier. The original IP is only used for SSH and OVPN from the house. it should not do much else. All network traffic in house is coming from other of the other IPs.

Again, so far, so good. Hopefully the ECMP stuff works correctly, so I will do an updated post soon.

(Mad) Max Speed – The Road Warrior (Internet connection) (double speed internet Part 5)

[NOTE] This part 5 in a series of posts. The rest can be found here.

This post is going to be an update and theoretical post. probably very little “new” stuff going on here, mostly updates, and what I am planning on doing later on.

This week, I have been OOF sick, so I have not done much work, but I have been surfing the web, watching videos, downloading stuff, etc., so I have an idea of how things are going. First, as mentioned in the previous post I have MPTCP, Squid, Socks Servers, OpenVPN and IPTables doing their magic. 2 OpenVPN tunnels between the house and Digital Ocean. All TCP Traffic (bar port 80) is sent over socks to the box in the cloud using RedSocks. All UDP traffic is sent direct over OpenVPN. Since MPTCP is in the mix, all socks traffic is actually split over the 2 connections. All port 80 traffic, and 443 (if the client is using local Squid as their proxy) is sent round-robin between the 2 upstream IPs to Squid (2 OpenVPN end points).

Things I have noticed:

  • Every now and again, RedSocks crashes… just full on dies. It’s just a matter of starting again, but it’s a pain…
  • I have had to restart squid a couple of times… not too often though
  • there was a power outage in the house a few days back… so, when everything came back online, it was a bit of a pain bringing all connections back to life. I do have to figure out a better plan

I still have to read more on this ECMP stuff. Hopefully it will do what I am hoping.

Now for the theoretical stuff. I started thinking, could this work outside the house? Could you build this into something smaller, like a Raspberry Pi, and stick 2 or more USB Modems in, connect it back to a server in the cloud, setup P2P OpenVPN connections and then get more than a single modem speed download? The problems I can see are around MPTCP. I am not sure if it has been ported to ARM to run on a Raspberry Pi. Second, the max you could ever get out of it is 100Mbit/s, given the 10/100mb network port on board… and you may need extra power for the USB dongles. Also, getting P2P connections may be complicated, given the non-static IPs on the modems, though, in theory, non-P2P OpenVPN could work… Again, it’s a theory. I had the though and that’s where the title came from… anyway, throwing it out there…

2 Cable Modems = Double Speed? Part 4

[NOTE] This part 4 in a series of posts. The rest can be found here.

So, this week I went in a completely different direction that I have been thinking recently…

So, the basic theory is as follows:

  • I am still using MPTCP kernels on both upstream and local machine
  • now have 2 P2P UDP OpenVPN tunnels between house and cloud. Example config is here
  • all TCP traffic (bar port 80) that hits the router in house is redirected to RedSocks
  • RedSocks uses a socks server, Dante, as an upstream server on the cloud box
  • since the socks traffic is over TCP (inside the UDP OpenVPN tunnel) it uses MPTCP
  • having socks running, gives me quite the download speed, turning it off does not, hence the following tweet

  • I am also noticing that I am starting to hit the limits of my upstream VM. If downloading or uploading at speed, the processor cores (2 in the case of the box I am currently running) are pegged at pretty much 100% full… Well, 80ish, but that because the other 20% is being used by Dante. I am noticing I can hit a full 72Mbit/s up, but the max currently downloading is about 400, maybe 450… Need a faster box now…
  • I mentioned port 80 not being set over socks. That’s because its redirected to Squid. Squid (in house) then uses Squid (in cloud) as a parent. There are 2 round-robin parents for squid, one on each OpenVPN connection IP address.
  • all other traffic (UDP, ICMP, etc.) are sent over the OpenVPN connection… currently only one is picked, but I have a cunning plan…

The cunning plan? Well, if I am reading the internet correctly, and I would like to think I am, I think ECMP, or Equal Cost Multi-Path Routing, could help… Again, it’s a fledgling idea currently, and I am still reading the documentation, but if it works… Well… I not sure… let’s see…

2 Cable modems = Double speed? Part 3

[NOTE] This part 3 in a series of posts. The rest can be found here.

In Part 1 of this series I explained the why and what I wanted to do for this “project”. In Part 2 I did some basic testing of both MPTCP and MLVPN. I also mentioned trying MMPPP using vtund but it has been a while since I did that testing, and it had not been on bare metal. So, this post is a follow up, where I am using bare metal.

So, first, the setup:

  • ProLiant box is running Debian 8.3 x64, and has both vtund and ppp installed
  • Digtial Ocean box also has Debian 8.3, vtund and ppp installed
  • walked though the guide from John Lewis and made some changes to the configs. the main ones are mentioned below

Once done, i installed both iperf and iftop on both boxes, and ran

iperf -s

on the Digital Ocean box and

iperf -c 192.168.10.1 -d

on the local box. And, well, the results where not as expected. Pretty poor actually:

First, using Squid installed on the DO box, i tried using WGET to download a file using it. If I did this on the DO box itself, i was getting 100MBytes/s… When I ran it over the MLPPP box, well, under 7 was achieved.

Then i though it might have been Squid. So, since the file had been downloaded to DO, i SFTPed into the box over the MLPPP link, and tried again… Again, pretty poor result. I think i seen it hit about 7MB a sec at one stage.

Here is what is showing on the DO box when running the SFTP download. You can see 2 connections from the 2 WAN links at home hitting the box, and they are balanced. Its just nowhere near the speed they are capable of.

I did not get a screen shot of this, but when I tried with iperf, thinking it might have been overhead of SFTP or Squid, I was getting results matching what I was seeing with SFTP. Downloads in the 55-60mbit/s range for download and 40ish for upload. 40 is still faster than 1 link, mind you…

I mentioned that I had made some minor tweaks to the configs from what John had written. Well, mostly it was config changes to how routing was done. In Johns case, he is bonding a DSL and a HSDPA connection, so he had setup to do for logging into his PPP modem and connecting. Also, when he setup the interfaces, he routing tables in there. I have mine setup in a single config file, like as follows:

I have changed the names from adsl1 and 2 to WAN1 and 2, and the IPs are changed from internal IPs to my public IPs. I manually run this when setting up my connection.

Nothing else on his config files have changed. I did not do any of the masquerading stuff, mainly cause this was testing. I just want a tunnel to start with. When reading the vtund.conf file, you can see that encryption and compression are both turned off, and and the same in the ppp configuration. I also don’t think the issue is to do with the CPU performance, since these are the screenshots of top running on both boxes:

in both cases, CPU usage is sub 6% for VTUN and SSH seems to be using less than 10%. So, now, I’m baffled as to why this is not performing as expected… More testing required!

[update 4/4/2016] – fixing images so they are clickable…

MPTCP, SSH, Squid, OpenVPN (and 2 Cable modems) = Double Speed? Not quite… Part 2

[NOTE] This part 2 in a series of posts. The rest can be found here.

In my previous post I explained what i was trying to do… This post explains what i have been working on recently, and performance results…
So, first, what have i tried… There are 3 different things i have tried, and here are some of their details. Some will need to be updated (other parts of this series), and others i will try get back to eventually.

Hardware and servers used

To test this, i am using my HP Proliant ML110 G5 running either Ubuntu or Debian Linux, with 2 GigE connections directly to the cable modems, and 1 connection to the LAN (for SSH and testing). The LAN has no gateway set, and the 2 WAN connections have DHCP enabled. They get fully public IP addresses. Upstream, I am using either Digital Ocean or ScaleWay VPS boxes.

Digital Ocean has the advantage of allowing different Kernels, so i have been using them for testing MPTCP. As for ScaleWay, well, their BareMetal C2S/M/L boxes have between 4 and 8 cores (4 for the S, 8 for the M and L) and between 8 and 32Gb RAM (S=8, M=16, L=32GB). The L model also comes with 256GB SSD (plus the boot disk, which seems to be a network disk of some sort) and they all come with lots of bandwidth (i use the L because its got about 800MBit/s to the internet).

Ping wise, Digital Ocean is about 20-30ms away from the house (I picked London to host the servers) and Scaleway is a little further at about 50ms (They are based in France).

MPTCP (MultiPath TCP)

MPTCP (their site is a bit wonky as of writing, so bare with me…) is a Linux Kernel patch that allows TCP connections to use multiple paths… Essentially, if you have Wifi and 4G in a phone, and MPTCP is enabled, it should allow you to use both connections for TCP traffic, as long as the server upstream supports it. It also allows for easy fail over if, say, you lose your wifi connection. There is an example video of it on YouTube which should show the fail over parts and this video shows how they managed to get 50Gbit/s out of a 6 10Gb Ethernet connections.

When i was using MPTCP, I had a copy of Squid on both boxes, and told Squid locally to use Squid on the upstream box (over a SSH tunnel, which was over the MPTCP link) as a parent cache. Using this method, i could see (using iftop) that both connections were being used. When trying proper performace testing, I setup a RAM disk on both machines and copied a Linux ISO to the Digtial Ocean Box. Then, using wget and Axel I downloaded the files using Nginx on the server, and checked the results. I can max out 1 single connection, plus use about 60-80mbit/s from the second. about 420-440mbit/s total. Disk was not the bottleneck, since I was writing to RAM, so more tests are required.

MLVPN

MLVPN is a pretty interesting project that caught my eye. The idea is quite simple: you configure the local box and server, as mentioned in their example guide and run the MLVPN program on the server, then the client. It creates 2 VPN tunnels between the 2 boxes, and bonds them… In my case, i was given an IP of 10.42.42.1 on my box in house and 10.42.42.2 on the server. Any traffic over that tunnel is bonded… Problem is, it seems to be quite processor intensive: my Digital Ocean box was showing one cpu core (out of 2) maxing out at around 80% and my Proliant in house maxing around the 70% mark… all while transferring data at around 100mbit/s. I tried iperf and got the following:

getting 50mbit/s upload is good, in reality, since in theory my max speed would be 72, without overhead. but 116mb/s down is less than a third the max speed of a single connection. So, I tried just uploads and downloads…

Upload Only (from local machine to server)

Download Only (from server to local machine)

As you can see, the download speed has increased a little, to 176Mbit/s, but the upload speed is now at over 60MBit/s!

Still.. download is as important as upload, and given I haven’t managed to get it to max out one connection, never mind 2, even more testing is required…

MLPPP (using VTUN)

This is one i need to come back to… Used the guide from John Lewis but was only managing to get about 100Mbit/s… I was originally using a VM (so disk may have been the issue) and also had the connection behind my EdgeRouter, so it might have been firewall rules causing a slow down. But I do need to come back to this soon… Watch this space.

Conclusions?

Well, at the moment, all I can conclude is that there is more testing required. Upload wise, i can somewhat use most of my bandwidth with MLVPN, and I did see promising results with MPTCP. I gave up a bit too early with MLPPP, so more testing is required with that. Also, all tests are using just iperf between boxes. I did use squid with the MPTCP box for a while, but not for proper performance testing. So, even once this is all sorted out, i will need to turn this into a proper “router” too… So, conclusion? this was originally meant to be a 2 parter… now it looks like I will require a lot more parts… Watch this space…