Tiernan's Comms Closet

Geek, Programmer, Photographer, network egineer…

Currently Viewing Posts in Backups

New Backup Plans

So, a few weeks back, CrashPlan, one of my chosen backup services, decided to kill their consumer backup plans. Which made me have to rethink my backup plan for the house…

Note: This is how I am backing up files, and may or may not work for you. Some of this is already in “production” as of today, but others are planned… Any questions, comments, etc, leave them in the comments.

So, first, as mentioned above, CrashPlan’s consumer backup options are now dead. They are giving you the option of either upgrading to their Small Business Plan for free till the end of your current contract + 2 months, or moving to Carbonite. For me, i just moved to their Small Business option, since its free, and meant that most of my backups just migrated over… i did not, how ever, look too much into Carbonite.

Just checking their site now, i would have to go for a minimum of the Plus plan, and that only allows me to backup 1 machine… I have 2 (currently) that get backed up… and probably more at a later date… and, given their yearly price for Plus is $75 per year, thats $150 for 2 current machines, and an extra $75 per machine after… Granted, they are saying its unlimited storage. but the extra fee per machine puts it out of contention…

For CrashPlans Small Business Plan, their plan is $10 per machine per month… So, my 2 machines would cost $240 per year to backup. They are giving me the next few months for free, and then for the first year, they are offering a discount of 75%, but we will see what happens when renewal comes along…

So, where does that leave me? Well, its looking like i will be using a mix of BackBlaze and Hubic. The current idea is as follows:

  • GodBoxV2 and the Mac Mini both have Drobos plugged into them: the GodBoxV2 has a Drobo 5D, and the mini has a Drobo mini. There is also a Drobo FS on the network. I know they are not backup solutions, but its better than having stuff on single drives…

  • For Windows boxes, I am backing up files to the cloud using Duplicati. These are backed up to both Hubic and BackBlaze B2. Photos and importing files are backed up to both, and stuff i want backed up, but not in 2 places, is just backed up on Hubic.

  • For my Mac mini, i use a a mix of Duplicati and Arq Backup. Arq got B2 support recently, so its very handy for that.

  • For some of my Linux Boxes, i use Duplicity. You may need to walk though the steps i have here to get it working with Hubic, and it also works with B2.

Some further notes:

  • Hubic are charging about EUR5 a month for 10Tb of storage. If you refer people, that can go up. I have maxed mine out at 12.5Tb.

  • Hubic is limited to 10mb/s for uploads and downloads. I never got anywhere close to that speed with CrashPlan (I’m based in Europe, they are in the US, so speed of light and what have you… maxed most of the time at 4ish). That being said, even those its “limited” to 10mb/s, i have seen higher.,.. might be 10 per thread…

  • I did, for a while, use the BackBlaze app and service. I just didn’t renew because of the move to B2. If you don’t want to mess with stuff, their App works great.And if you use this referal link you get a free month… Also, its FAST AF! it actually manged to max out my upload connection to their server! wont complain about that!

  • B2 is cheap. 0.5c per gig stored… So, it works out at about $5 per TB per month, minus your usual bandwidth fees…

  • Duplicati and Duplicity both have options to send emails on success or failure, or the like. I recommend having the emails on failure turned on at a minimum… also, on headless boxes, like my Linux box, i recommend checking the backup every week…

So, there we have it. Any questions?

Hubic and Duplicity

I mentioned HubiC in my last post, and in it i said that you could use Duplicity for backups. Well, this is how you get it to work…

First, i am using Ubuntu 14.04 (i think…). I use Ubuntu in house for a few things:

  • its running Tiernan’s Comms Closet, GeekPhotographer and Tiernan’s Podcast all in house, aswell as being used to build this site. The Web Server and MySQL Server are seperated, MySQL running on Windows, web on Ubuntu… but thats a different story…
  • I have a couple of proxy servers running Ubuntu also
  • Other general servers running Ubuntu… dont ask, cause i cant remember what they do half the time…

So, Duplicity is a backup application. From their website:

What is it?

Duplicity backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.

The duplicity package also includes the rdiffdir utility. Rdiffdir is an extension of librsync’s rdiff to directories—it can be used to produce signatures and deltas of directories as well as regular files. These signatures and deltas are in GNU tar format.

So, how do we get it working? Well, givin that i am on Ubuntu, these are the steps i needed to do:

  • first, we need some credentials and API keys… If you havent signed up for HubiC Do so now… That url gets you an extra 5Gb if you sign up for free (usually 25Gb) or if you pay 1EUR a month, you get 110Gb (usually 100Gb) and 5EUR a month gets you a staggering 10TB (yup! Terabytes!).
  • Login to Hubic, and in the menu go to ‘My Account’, ‘Developers’. in here, create a new application (name and URL to redirect to… http://localhost seems to work correctly). Get the Client ID and Secret ID that was given to you.
  • take the contents of the following gist and replace your own details… I know, i am not a fan of sticking my password in a txt file… but it should be your local machine…
  • that file should be in your home directory and should be called .hubic_credentials.
  • add the duplicity PPA project (https://launchpad.net/~duplicity-team/+archive/ubuntu/ppa) to ubuntu using the add-apt-repository command (details on the link above, under the link ‘read about installing’). for me, i just called ‘sudo add-apt-repository ppa:duplicity-team/ppa’
  • install duplicity by doing ‘sudo apt-get install duplicity’. Dont forget (its in the tutorial above!) to do an ‘sudo apt-get update’ first!
  • When i ran that, there where a few extra Python packages to be installed, so i was asked did i want to install them… Say, yes.
  • Now, to run a backup we run the following command:

duplicity ~/ cf+hubic://location

  • cf+hubic is the backend to use, ~/ is the url to backup (my home directory in this case) and location is where on Hubic we want it stored. If this doesent exist, not a problem… it will create it.
  • after we run this we… ahhh… i get an error:

BackendException: This backend requires the pyrax library available from Rackspace.

  • right… pyrax library is from Rackspace and is available to download though pip…
  • I seem to have python and a few other bits installed on this machine, so running ‘sudo pip install pyrax’ works… Your millage may vary… [eg, this is out of scope for this tutorial! your on your own!]
  • Other problem… I got a load of weird and wondering errors like this:

AttributeError: 'Module_six_moves_urllib_parse' object has no attribute 'SplitResult'

  • I fixed these by running:

sudo pip install furl --upgrade

  • FINALLY! ITS ALIVE!!! by default, it asks you for a key for the GnuPG encryption… and its all good! the first backup creates the directories, required files, etc. the next time you run the command, it will only upload changes. it will also ask for your GnuPG code you entered, so remember it!

And thats all folks! Any questions, leave them in the comments!

Symform – P2P Backup

I have previously posted about CrashPlan as my Backup System. I also, a long time ago, talked about Backing up SQL, MySQL and other stuff on my other blog. Well, CrashPlan is all good, but there are 2 “niggly” bits with it…

  • Its not FREE (well, this year i got it Free on Black Friday…) but it is cheap ($120 a year to backup 10 machines to the cloud aint bad.)
  • Its NOT FAST! The CrashPlan Datacenters all live in the US, and my servers live in Europe (either Dublin or Germany). So, bandwidth is limited… Getting less than 1Mbit/s most times, but have seen it reach 3… I have 20Mbits/s upload… even half that would be nice…

So, thats where Symform comes in. Symform is a P2P Backup Service, which runs on Windows, Linux and MacOSX. In theory, it should run anywhere that has a Mono runtime since its written in .NET. Anyway, you start with 10Gb of free storage, and you can increese that by one of 2 ways:

  • Pay money: for $0.15 per month, you get 1Gb of storage in the cloud
  • Pay Bytes: For every 2Gb “Contributed” (which is actually more like a pledge than a contribution… more on that later) you get 1Gb storage in the cloud.

It works very well, and is nice a fast too. I have a few machines in house which are contribting stoage, a total of about 2Tb, and I have been given 1Tb storage in “The Cloud”. There is a lot more on how this works on their “How Symform Works” section of their site.

I mentioned the “Contribution” VS “Pledge” up above… I have a machine in the house where i have Pledged 1Tb of storage. In reality, Symform can use the full 1Tb of storage, if it needs to, but is currently only using 168Gb. Now, that could just be that the machine is still getting files, and it will end up using the full 1Tb eventually, but either way, its all good.

Also, as a couple of notes on Contribution and Backups:

  • The machine needs to be online and accessable on the internet at least 80% of the time, but 24/7 is ideal. If you drop below the 80%, your account can be suspended.
  • your machine needs to be publically accessable, meaning port forwarded. I have a couple contribution machines in house, so they each have seperate ports forwarded to them.
  • Given the P2P nature of the software, lots of connections to different machines are made… if you are behind a firewall, you may need to allow all or most outgoing connections. If you are on a really restrictive firewall, you may want to stick a contribution box in your DMZ and probably use the Turbo Seeding feature.
  • Turbo Seeding is a handy feature, especially for Laptops… only problem is its Windows Only… So, importing and exporting does not work on Linux or OSX.
  • The software can managed Work and Non Work hours, and will limit the upload and download speed during this time. Also a nice feature…

So far, so good. Very happy with the software, but would like a nicer interface to see whats going on. At the moment, you are either limited to using the web interface, which aint bad, but not great, or watching the log files… I would also like the ability to prioritize certain files or folders, so, for example, upload my documents folder before anything else, and if anything changes in there, even if its uploading from somewhere else, pause and upload the documents folder… Just a thought…

Crashplan Backups

I have been running CrashPlan for a while now, and, other than some minor issues (backup speed to their central location is the big one), everything has been going grand. I use it to backup about 600GB of photos and videos, 500GB+ of VMs, documents, source code and a fair whack of other stuff… In total, about 2TB of data.

Anyway, here are some tips i have figured out over the last while for making Crashplan work a little better…

I am also in the process of doing some further tests with iSCSI drives linked to the Cloud, and other weird and wonderful things… once I have finished, i will upload extra stuff here.