Blog

A link between DigitalOcean, Packer and Terraform

In the coming months, I will run several workshops about Jenkins. Those workshops will be hands-on, so people will bring their own laptops to hack on Jenkins instances.

For that, I expect lots of them will simply run Jenkins on their laptops, or use an external laptop. But, as a backup, or a primary solution, I plan to provision multiple instances of Jenkins, ready to be used, in the cloud.

Amongst the cloud providers, DigitalOcean is great for this purpose: while it is feature limited, it is simple. Its pricing model is simple too.

In the cloud, you run images. Like I will spin up a lot of them, I want to have those images ready, so I just need to boot them and configure them. The tool for that is Packer. Packer builds images. For DigitalOcean, Docker, AWS, Virtualbox.. It has a great list of providers.

That part was easy. I’ve got my snapshots built quickly.

Then enters Terraform. Terraform can deploy the snapshots built by Packer. Just like this:

resource "digitalocean_droplet" "jenkins" {
  image = "1028374"
  count  = "${var.count}"
  name = "${format("jenkins%02d", count.index + 1)}"
  ssh_keys = [ "${var.do_ssh_key}" ]
  region = "${var.do_datacenter}"
  private_networking = "true",
  size = "512mb"
}

That configuration works but has a problem: the image parameter is an ID. For Snapshots, you can not use name, slugs, you have to use id numbers. This is very annoying, because I to not want the image 1028374, I want the image “jenkins-1.0.0”. Which is the name I defined in Packer.

The workaround

Like lots of problems with open-source software, there is a workaround. Terraform has an external datasource plugin. It can run any script. Here is mine:

#!/bin/bash -e
set -o pipefail
snapshots="$(eval $(jq -r '@sh "doctl -t=\(.api_token) compute snapshot list
\(.id) -o json"'))"
nr="$(echo "$snapshots"|jq '.|length')"
if [[ "$nr" -ne 1 ]]
then
    echo "Expected 1 snapshot, found $nr" >&2
    exit 1
fi
id="$(echo "$snapshots"|jq -r .[0].id)"
jq -r -n --arg id "$id" '{"id":$id}'

In Terraform:

data "external" "jenkins_snapshot" {
  program = ["./do_snapshot_id"]
  query {
    id = "jenkins-${var.do_jenkins_droplet_version}"
    api_token = "${var.do_api_token}"
  }
}

resource "digitalocean_droplet" "jenkins" {
  image = "${data.external.jenkins_snapshot.result.id}"
  count  = "${var.count}"
  name = "${format("jenkins%02d", count.index + 1)}"
  ssh_keys = [ "${var.do_ssh_key}" ]
  region = "${var.do_datacenter}"
  private_networking = "true",
  size = "512mb"
}

That is already better. I can now use the same name as the one I use in Packer. No more meaningless ID’s.

But this approach has multiple problems:

  • Bash scripting is unreadable; it is just 10 lines but is a mess.
  • It introduces jq and doctl as dependencies. So it will not work for everyone.
  • It delegates the digitalocean credentials to multiple processes. I do not like that.

The Solution

I ended up spending some time on an implementation in go. It is now merged en Terraform, which means I had to write documentation and tests, so everyone can use it now. Here is the DigitalOcean image datasource:

data "digitalocean_image" "jenkins" {
  name = "jenkins-${var.do_jenkins_droplet_version}"
}

resource "digitalocean_droplet" "jenkins" {
  image  = "${data.digitalocean_image.jenkins.image}"
  count  = "${var.count}"
  name = "${format("jenkins%02d", count.index + 1)}"
  ssh_keys = [ "${var.do_ssh_key}" ]
  region = "${var.do_datacenter}"
  private_networking = "true",
  size = "512mb"
}

It is a lot better. No external dependencies, bash scripts, and it is available for everyone. And it is shorter and easier to understand.

I hope you will enjoy it. It will be available in the next release of Terraform (0.9.4).

Permalink. Category: cloud. Tags: linux hashicorp.
First published on Fri 21 April 2017.

Running Nightly jobs with Jenkins

Jenkins can spread the load of Jobs by using H instead of * in the cron fields. It means that:

H 3 * * *

Means: Run between 3 and 4 am.

The minute will be decided by Jenkins, by applying a hash function over the job name.

What about this one:

H H * * *

Means: Run once a day. The moment will be calculated by a Jenkins based on the job name.

But what if I have hundreds of jobs, I want to run them once a day, but during night? Something like:

H H(0-5) * * *

Means: Run the job once everyday between 12am and 6am.

But when you scale up your Jenkins, you want the jobs to run between e.g. 7pm and 6am. Because you also want to use the hours before midnight.,

There is a BAD WAY to do it:

H H(0-6),H(19-23) H/2 * *

That would run the jobs in the morning and the evening, every 2 days. It is complex and will not behave correctly at the end of months.

The GOOD WAY to do it is to set the timezone in the cron expression, something which is not documented yet. It is there since Jenkins 1.615 so you probably have it in your Jenkins:

TZ=GMT+7
H H(0-10) * * *

What does this mean? In the timezone GMT+7, run the jobs once between 12am and 11 am. Which means between 7pm and 6am in my timezone.

$ date -d '0:00 GMT+7'
Wed Mar 29 19:00:00 CEST 2017
$ date -d '11:00 GMT+7'
Thu Mar 30 06:00:00 CEST 2017

It is a lot more simple syntax and is more reliable. Please note that the validation (preview) below the Cron settings is not using that TZ (I opened JENKINS-43228 for this).

Permalink. Category: Automation. Tags: Jenkins Testing Planet-inuits.
First published on Thu 30 March 2017.

Setting a Socks Proxy in Firefox Webdriver with Robot Framework

Here are the Robot Framework keywords needed nowadays to setup a socks5 proxy (e.g ssh -ND 9050 bastion.example.com):

*** Settings ***
Documentation     Open a Web Page using a socks 5 proxy (demo)
Library           Selenium2Library

*** Test Cases ***
Create Webdriver and Open Page
    ${profile}=   Evaluate   sys.modules['selenium.webdriver'].FirefoxProfile()
sys
    Call Method   ${profile}   set_preference   network.proxy.socks   127.0.0.1
    Call Method   ${profile}   set_preference   network.proxy.socks_port
${9060}
    Call Method   ${profile}   set_preference   network.proxy.socks_remote_dns
${True}
    Call Method   ${profile}   set_preference   network.proxy.type   ${1}
    Create WebDriver   Firefox   firefox_profile=${profile}
    Go To    http://internal.example.com
Permalink. Category: Linux. Tags: Open-Source Testing Automation planet-inuits.
First published on Thu 30 March 2017.

My first Open Source contribution

The first Open-Source contribution I remember happened on December 2005. It happened on Gaim, which is now known as Pidgin.

It was not a big contribution: I did not like the arrows shown when moving discussion tabs. Those arrows where plain red. I created new ones (using The Gimp), with a gradient. It did not take a long time to make them, but it was definitively an improvement.

Here it is: my first contribution

It was quickly merged (< 2 hours). And it was important for me, because at that moment I understood that I could improve the tools I was using, and that everyone would get those improvements. That was the beginning of 12 years of Open Source contributions.

Permalink. Category: Open-Source. Tags: open-source gpl pidgin.
First published on Wed 22 March 2017.

Augeas resource for mgmt

Last week, I joined the mgmt hackathon, just after Config Management Camp Ghent. It helped me understanding how mgmt actually works and that helped me to introduce two improvements in the codebase: prometheus support, and an augeas resource.

I will blog later about the prometheus support, today I will focus about the Augeas resource.

Defining a resource

Currently, mgmt does not have a DSL, it only uses plain yaml.

Here is how you define an Augeas resource:

---
graph: mygraph
resources:
  augeas:
  - name: sshd_config
    lens: Sshd.lns
    file: "/etc/ssh/sshd_config"
    sets:
      - path: X11Forwarding
        value: no
edges:

As you can see, the augeas resource takes several parameters:

  • lens: the lens file to load
  • file: the path to the file that we want to change
  • sets: the paths/values that we want to change

Setting file will create a Watcher, which means that each time you change that file, mgmt will check if it is still aligned with what you want.

Code

The code can be found there: https://github.com/purpleidea/mgmt/pull/128/files

We are using go bindings for Augeas: https://github.com/dominikh/go-augeas/ Unfortunately, those bindings only support a recent version of Augeas. It was needed to vendor it to make it build on travis.

Future plans

Future plans regarding this resource is to add some parameters, probably a parameter to use as Puppet “onlyif” parameter, and a “rm” parameter.

Permalink. Category: Linux. Tags: automation planet-inuits.
First published on Tue 14 February 2017.