Tiernan's Comms Closet

Geek, Programmer, Photographer, network egineer…

Currently Viewing Posts in Tools

Auto deploying to multiple servers with GitHub and Webhooks

In yesterdays post, i mentioned that i wanted to try get an auto deploy working for this site. It already builds automagically using Forestry and puts the static HTML into a Github repo, but i needed to manually update the servers hosting the site… Well, not any more!

using the magic of Github’s Web hooks, the Webhook project and a small piece of bash shell script, i have managed to get this auto deploying…

First, Download the Webhook project (its a Go application, so it works pretty much anywhere). Copy it somewhere on your machine. Next, you need a config. I used the Github sample config from the project site and made tweaks to what script to run and what i was passing in.

next, the script to pull from Github was simple enough:

The repo should already be cloned into the folder, /var/www/localfolder and your web server should be pointing at that also. Then, its just a matter of running the command:

./webhook --hooks github.json --verbose

The --verbose tag gives you lots of info, so its handy for testing. and then your app is running and listening on the default port, 9000.

next, head over to your project on Github and go to settings:

select webhooks and add new web hook

Fill in the required details on the page, and click save.

Github will go out and have a chat with the webhook and verify it can send and recieve stuff from the hook. You can see this in the deliveries section:

Clicking on these will show you the headers that were sent, along with the payload, and you can also see the response from your server. Finally, you have the option of resending the payload, just in case anything goes wrong.

So, there you have it. A complete automated deploy across multiple servers! Any questions, leave a comment below!

[UPDATE] yesterday i mentioned i had to modify the sample that was included on the webhook site. Well, i noticed something this morning. The reason i needed it modified was the trigger rule was checking the header and the reference for the branch, but any time i ran it, it would not trigger… The reason was simple: the webhook app is expecting application/json but i had it set to application/x-www-form-urlencoded which is the default… the webhook app then couldn’t parse it correctly… changing that fixes the problem! happy days!

Testing Forestry

So, as you probably know, this site is built with Jekyll. Jekyll is a Static Site Generator, basically taking an input of a load of text files (see the source repo for this site on Github here) and generating a load more HTML (the static HTML is hosted on Github here, which auto publishes to Azure App Service).

In previous posts, i have talked about using the likes of Visual Studio Code and Mark Down Monster to build the site. Well, a few days back, i found Forestry.io. Its a web application which, in my case, is linked with my GitHub repo (the Jekyll source one) and allows me to make changes to the code easily. Because the way i build my site is a little different, i manually build the site and push to the destination GitHub project, but they have features allowing you to push directly to SFTP or FTP servers, GitHub, or some other options.

The interface is nice and easy to use, and you can use drafts, etc. Mind you, even with drafts, because files are written into a public repo, they are not fully private… I suppose I could just make the source repo private… Anyway, they are free for single user sites (like this) or they have paid plans for teams (say, your business blog with more than 1 user updating it, for example).

VSCode and Markdown Monster with Powershell

A few years back, i created a post showing you how to add an Alias to PowerShell to easily start Sublime Text from a PowerShell command line . This worked well, but this is 2017 (that post is from 2012!) and my daily text editor has changed. I have moved to Visual Studio Code for most of my daily work. It works well 95% of the time. I still use Visual Studio Pro for C# Development, but for quick fixes and work on, say Go or smaller edits, Code is great. For blogging, on the other hand, I am trying out MarkDown Monster but code still has some nice features. We will see how tests go.

Anyway, to upgrade the post, and to be able to use VS Code and MarkDown Monster from your PowerShell command line, I added the following:

Set-Alias code "C:\Program Files (x86)\Microsoft VS Code\Code.exe"
Set-Alias mm "C:\Program Files (x86)\Markdown Monster\MarkdownMonster.exe"

You can either run this each time you open PowerShell (a bit of pain) or you can add it to your Microsoft.PowerShell_profile.ps1 which lives in your Document\WindowsPowerShell folder. If you dont have one, just create the file, add that piece of text, and next time you open PowerShell, you are good to go.

In my case, i am actually in the Visual Studio Code Insiders group, so my alias is:

Set-Alias code "C:\Program Files (x86)\Microsoft VS Code Insiders\Code - Insiders.exe"

When these are set (and you have restarted your PowerShell windows), you can now run the following commands:

code filename.txt
code folder
mm filename.txt

VSCode will open either files or folders, but it seems Markdown Monster only opens files.

Docker Jekyll and Mr ngrok

See what i did with the title?! Anyway, in my last post, i explained how i was building this site with Docker running on Windows 10 with the Anniversary update. Today, i am going to show you how to host it using Nginx and ngrok.

So, first, you should know what Nginx is at this stage… If not, check out their site. Next ngrok is basically a way of tunneling your localhost to the web. So, how do we build the whole lot together and serve your site to the internet? Well, this is what i have so far:

First, build your site in jekyll. for me, the command is

docker run --rm -v "$(pwd):/src" -w /src ruby sh -c 'bundle install --path vendor/bundle && exec jekyll build -s www.tiernanotoole.ie/ -d www.tiernanotoole.ie/_site/'

next, run an nginx server with that output folder:

docker run --name tiernanotoolenginx -v "$(pwd)/www.tiernanotoole.ie/_site/:/usr/share/nginx/html:ro" -d -p 8881:80 nginx

the docker container is called tiernanotoolenginx, since i could have multiple ones, and port 8881 is being redirected to port 80 on that container, but technically, it might not be needed due to the next command:

docker run --rm -it --link tiernanotoolenginx wernight/ngrok ngrok http tiernanotoolenginx:80

essentially, what we are doing here is running ngrok and pointing it at post 80 on the nginx container… you see i did not point at 8881, since we are using the continer directly… it might be different if you were not…

when that command runs, you get a screen telling you the URL of your site with some basic stats. your site is now hosted publically, via an ngrok tunnel! you could run that container as a daemon, and leave it running, but for me, i wanted to do some minor testing, so i can kill it when i want…

So, all is good with the world!

Building Jekyll sites with Docker on Windows

As some of you probably know (or based on the footer of the site) this site is built with Jekyll. Jekyll is a static web site builder, written in Ruby, and is a bit of a pain to build on Windows. Earlier on this year, I wrote up a post explaining how to use Jekyll on Windows using Bash on Ubuntu on Windows… It was a bit complicated, and, well, worked a few times, but was not too successfull… So, were do we go next? Well, Docker to the rescue!

I am running the Windows 10 Anniversary edtion witch has container and docker support. using the repo for this site and the scripts (specifically build-tiernanotooleie and geekphotographer.com) i can build the docker site on my local Windows machine and upload the sites as required (I host on NFSN and upload via RSync). The docker image i build from is a Linux docker image, do i need a Linux container running (and the docker tooling). I also use Bash on Ubuntu on Windows to upload using RSync. All is going well so far…

Useful Web and Desktop Apps 2016 edition

I have decided to do a post on some of my favourite tools to use for development, administration, etc. It’s kind of like Hanselman’s Ultimate Tools list, but not as popular and about 2 years newer… Anyway, the list is available here, and will be updated over time, much like my Daily Carry and Computers pages. If you are interested, you add links though GitHub by editing the toolslist.yml data file.

Bash on Ubuntu on Windows

Microsoft Build 2016 is on this week, and there were a lot of interesting developments yesterday, but the one that interested me the most is Bash on Ubuntu on Windows. Dustin from Ubuntu has a more details, and Scott Hanselman has posted a technical video about this. This is very interesting, and I CANT WAIT TO GET MY HANDS ON IT! But, I do have some questions, which I thought I would put down in blog format:

  • Based on the post by Dustin, it seems that low level Linux calls are being handled and translated to Windows system calls. Which makes me think, could any Linux Distro work? Could Arch Linux, RedHat or Centos work in the same way?
  • Will this Work on Windows Server 2016 when it launches?
  • Given that it calling down to a low level, could GUI applications work too?
  • Shut up and take my money! I WANT IT NOW!

So, there are my questions… This is very cool, and I cannot wait to get my hands on this. Just wondering if this will be available to Windows Insiders sooner, rather than later?

Hubic, OpenStack Swift and Curl

HubiC is an online storage site, built by the guys at OVH. They are currently offering 30Gb free (if you use the link above) or if you pay, you get 110Gb (insted of the usual 100Gb) for EUR1 a month, or 10.5TB (yup… TERABYTES!) for EUR5 a month… Thats a crazy amount of storage for a not crazy amount of money!

So, while playing around with different things, I found they have an API, so other than the usual apps to play with (like the Hubic Apps for iPhone, Android, Windows Phone, Windows Desktop and OSX, Duplicity for backing up *nix boxes, and a few others) you can build your own…

But first, i needed to figure out how… So, after a lot of arsing around in Linux shells with curl i finally got some stuff working!

First, i used the Hubic sandbox to get the keys… its quite simple to walk though… this gets you your Access Token (see step 3). next, we need to get the Endpoint from Hubic: This GIST shows more:

Quick walkthough:

the first CURL request is to the HubiC API to get the credentials… this gives you a JSON response with a token and a endpoint URL aswell with an expire time…

The next request gets you a list of all files (or at least a load of files in my case) of whats in your folder. the default name here is my folder… I think its what everyone starts out with in HubiC… if you remove it, you will see all your top level folders.

next request i tried was to upload a file… the filename part is where you want it to be stored. this must exist on your local machine.

finally, downloading of a file… pass in the location of the file on the server (listing files will give you the location) and then -o in curl shows the output location…

Simples! now to get this working in c#… Full OpenStack Swift API is available to show how to do more… hopefully it will help in my C# coding…

Symform – P2P Backup

I have previously posted about CrashPlan as my Backup System. I also, a long time ago, talked about Backing up SQL, MySQL and other stuff on my other blog. Well, CrashPlan is all good, but there are 2 “niggly” bits with it…

  • Its not FREE (well, this year i got it Free on Black Friday…) but it is cheap ($120 a year to backup 10 machines to the cloud aint bad.)
  • Its NOT FAST! The CrashPlan Datacenters all live in the US, and my servers live in Europe (either Dublin or Germany). So, bandwidth is limited… Getting less than 1Mbit/s most times, but have seen it reach 3… I have 20Mbits/s upload… even half that would be nice…

So, thats where Symform comes in. Symform is a P2P Backup Service, which runs on Windows, Linux and MacOSX. In theory, it should run anywhere that has a Mono runtime since its written in .NET. Anyway, you start with 10Gb of free storage, and you can increese that by one of 2 ways:

  • Pay money: for $0.15 per month, you get 1Gb of storage in the cloud
  • Pay Bytes: For every 2Gb “Contributed” (which is actually more like a pledge than a contribution… more on that later) you get 1Gb storage in the cloud.

It works very well, and is nice a fast too. I have a few machines in house which are contribting stoage, a total of about 2Tb, and I have been given 1Tb storage in “The Cloud”. There is a lot more on how this works on their “How Symform Works” section of their site.

I mentioned the “Contribution” VS “Pledge” up above… I have a machine in the house where i have Pledged 1Tb of storage. In reality, Symform can use the full 1Tb of storage, if it needs to, but is currently only using 168Gb. Now, that could just be that the machine is still getting files, and it will end up using the full 1Tb eventually, but either way, its all good.

Also, as a couple of notes on Contribution and Backups:

  • The machine needs to be online and accessable on the internet at least 80% of the time, but 24/7 is ideal. If you drop below the 80%, your account can be suspended.
  • your machine needs to be publically accessable, meaning port forwarded. I have a couple contribution machines in house, so they each have seperate ports forwarded to them.
  • Given the P2P nature of the software, lots of connections to different machines are made… if you are behind a firewall, you may need to allow all or most outgoing connections. If you are on a really restrictive firewall, you may want to stick a contribution box in your DMZ and probably use the Turbo Seeding feature.
  • Turbo Seeding is a handy feature, especially for Laptops… only problem is its Windows Only… So, importing and exporting does not work on Linux or OSX.
  • The software can managed Work and Non Work hours, and will limit the upload and download speed during this time. Also a nice feature…

So far, so good. Very happy with the software, but would like a nicer interface to see whats going on. At the moment, you are either limited to using the web interface, which aint bad, but not great, or watching the log files… I would also like the ability to prioritize certain files or folders, so, for example, upload my documents folder before anything else, and if anything changes in there, even if its uploading from somewhere else, pause and upload the documents folder… Just a thought…

More Jekyll Stuff

Couple of bits and pieces on Jekyll stuff today… I am tweaking the outline of the site, so i am surfing around finding stuff… here is what i have found

  • Host a static site on Amazon S3: Interesting idea, and something i would look into eventually… And with the help of CloudFront you could host your whole blog on a CDN!
  • Rake tasks for Jekyll: Rake is the Ruby version of make… and a RakeFile can have tasks, which are in Ruby… They can do, from what i can gather, pretty much anything… So, some examples of what you can do with them are linked here… I especially like the New Post generator… very handy!
  • Jekyll Plugins: Various different plugins for Jekyll… I am interested in a few of these, mainly the Generate_projects one, which generates a page for your projects based on your GitHub projects… very cool stuff…
  • Strictly speaking, this is not just a Jekyll how to, but Migrating from WordPress to Jekyll is a handy read. my main blog, my podcast and photography blog both run WordPress. migrating them to Jekyll would mean i could move them directly to a CDN and make things a lot faster… Maybe something i plan doing soon…

If you have any tips or tricks, why not leave a comment and i can add them to the post.