Blog

Setting a Socks Proxy in Firefox Webdriver with Robot Framework

Here are the Robot Framework keywords needed nowadays to setup a socks5 proxy (e.g ssh -ND 9050 bastion.example.com):

*** Settings ***
Documentation     Open a Web Page using a socks 5 proxy (demo)
Library           Selenium2Library

*** Test Cases ***
Create Webdriver and Open Page
    ${profile}=   Evaluate   sys.modules['selenium.webdriver'].FirefoxProfile()
sys
    Call Method   ${profile}   set_preference   network.proxy.socks   127.0.0.1
    Call Method   ${profile}   set_preference   network.proxy.socks_port
${9060}
    Call Method   ${profile}   set_preference   network.proxy.socks_remote_dns
${True}
    Call Method   ${profile}   set_preference   network.proxy.type   ${1}
    Create WebDriver   Firefox   firefox_profile=${profile}
    Go To    http://internal.example.com
Permalink. Category: Linux. Tags: Open-Source Testing Automation planet-inuits.
First published on Thu 30 March 2017.

My first Open Source contribution

The first Open-Source contribution I remember happened on December 2005. It happened on Gaim, which is now known as Pidgin.

It was not a big contribution: I did not like the arrows shown when moving discussion tabs. Those arrows where plain red. I created new ones (using The Gimp), with a gradient. It did not take a long time to make them, but it was definitively an improvement.

Here it is: my first contribution

It was quickly merged (< 2 hours). And it was important for me, because at that moment I understood that I could improve the tools I was using, and that everyone would get those improvements. That was the beginning of 12 years of Open Source contributions.

Permalink. Category: Open-Source. Tags: open-source gpl pidgin.
First published on Wed 22 March 2017.

Augeas resource for mgmt

Last week, I joined the mgmt hackathon, just after Config Management Camp Ghent. It helped me understanding how mgmt actually works and that helped me to introduce two improvements in the codebase: prometheus support, and an augeas resource.

I will blog later about the prometheus support, today I will focus about the Augeas resource.

Defining a resource

Currently, mgmt does not have a DSL, it only uses plain yaml.

Here is how you define an Augeas resource:

---
graph: mygraph
resources:
  augeas:
  - name: sshd_config
    lens: Sshd.lns
    file: "/etc/ssh/sshd_config"
    sets:
      - path: X11Forwarding
        value: no
edges:

As you can see, the augeas resource takes several parameters:

  • lens: the lens file to load
  • file: the path to the file that we want to change
  • sets: the paths/values that we want to change

Setting file will create a Watcher, which means that each time you change that file, mgmt will check if it is still aligned with what you want.

Code

The code can be found there: https://github.com/purpleidea/mgmt/pull/128/files

We are using go bindings for Augeas: https://github.com/dominikh/go-augeas/ Unfortunately, those bindings only support a recent version of Augeas. It was needed to vendor it to make it build on travis.

Future plans

Future plans regarding this resource is to add some parameters, probably a parameter to use as Puppet “onlyif” parameter, and a “rm” parameter.

Permalink. Category: Linux. Tags: automation planet-inuits.
First published on Tue 14 February 2017.

mgmt

At Config Management Camp, James was once again presenting mgmt. He presented the project one year ago, on his blog. There are multiple ideas behind mgmt (as you can read on his blog):

  • Parallel execution
  • Event driven
  • Distributed topology

And all of that makes a lot of sense. I really like it. I do really think that this might become a huge deal in the future. But there are a couple of concerns I would like to raise.

It is James’ project

James has been developing it since 2013. His ideas are great, however the project is still at its very beginning, not usable yet, even for the more brave out there. The fact that it is so early and that even early adopters have a lot to do, makes it difficult to contribute. Even if there are 20 code contributors to the project, James is the one that does most of the job.

I do like the fact that James knows Puppet very well. It means we speak the same languages, he knows the problem Puppet users face, and is one of the best people to challenge the config management community.

How to improve that: get involved, code, try… We need to bring this to a more stable and usable shape before we can drag more and more contributors.

It is scary because it is fast

With traditional configuration management, when you screw up some thing, you still have the time to fix your configuration before it is applied everywhere. Puppet running every 30 minutes is a good example. You have a chance to fix some things in emergency before everyone gets them.

Even ansible is an order of magnitude slower than mgmt. And that makes it scary.

How to improve that: we might be able to tell mgmt to do canary releases. That would be awesome, something really great for the future of mgmt. It is quite hard to do so with traditional config management, but it might become native in the future with mgmt. That would be awesome.

It is hard to understand

mgmt internals are not as easy to understand as other tools. You can be scary at first sight. It is using directed acyclic graphes, which enables it do do things in parallel. Even if other config management use something close to that, in mgmt you need to have a deep understanding of them, especially when you want to contribute (which is the only thing you can do now). It is not hard, it is just a price we do not have to pay with the other tools.

How to improve that: continue improving mgmt, remove bugs in that graph so we are not badly affected by bugs in that graph, and we do not need to understand everything to contribute (e.g to write new resource types or fix bugs in resources). You can deal with Puppet, and know nothing about its internals.

Golang is challenging

mgmt is written in go. Whether it is a great language or not is not what matters here. What matters is that in the Ops world, go is not so popular. I am afraid that go is not as extensible as Ruby, and the fact that you need to recompile it to extend it is also a pain.

How to improve that: nothing mgmt can do. The core open-source infrastructure tools nowadays are mainly written in go. Sysadmins need to evolve here.

The AGPL license

I love copyleft. However, most of the world does not. This part of mgmt will be one of the things that will slow down its adoption. Ansible is GPL-Licensed, but the difference with Ansible and mgmt is that the last one is written in go, as a library. You could use mgmt just as a lib, backed into other softwares. But the licencing makes it too hard.

For a lot of other projects, the license will be the first thing they will see. They will not ever look deeper to see if mgmt firs their needs or not.

For now I am afraid mgmt would just become a thing in the Red Hat world, where by default lots of things are copyleft, so there would be no problem with including mgmt.

How to improve that: we need to speak with the community, and decide if we want to speed up the adoption, or to try to get more people to contribute.

The name

Naming things is hard. But mgmt is probably not the best name. It does not sell a dream, it does not stand out. It is impossible to do some internet searches about it.

How to improve that: we need to start thinking about a better name, that would reflect the community.

containers

I am not sure about what will be the future of mgmt in a container world. Not everything is moving to that world nowadays, but a lot of things actually is. I just don’t know. Need to think further about it.

But, is there anything positive?

You can read the list above as bad things. But I see that list a my initial ideas about how we can bring the project to the next level, how we can making it more appealing for external contributors.

mgmt is one of the projects that I would qualify as unique and really creative. Creativity is a gift nowadays. It is just awesome, just give it a bit more time.

We look forward to see it more stable, and to see a language that will take advantage of it, and enable us to easily write mgmt catalogs (currently it is all yaml). Stay tuned!

Permalink. Category: Linux. Tags: cfgmgmt open-source planet-inuits.
First published on Thu 9 February 2017.

Gitslave for Jenkins Pipelines

At customer, we are using GitSlave, as a way to build and release multiple releases at the same time. Gitslave is like git submodules, except that you always get the latest version of the submodules (no commits needed in the super repository).

For CI, we are using Jenkins. A Jenkins plugin was developed for it. Unfortunately, it was written in a way that would make the work of opensourcing that module too big (IP was all over the place in the testsuite). In addition to that, the integration with Jenkins was not as best as with the Git plugin.

The multi scm plugin is not an option as the list of repos to clone is in a file; and that it sometimes changes.

Fortunately, we will eventually drop that proprietary module, switching to the new Pipeline plugin. I would then like to share a Jenkinsfile script that takes the .gitslave configuration file and clones every repository. Please note that it takes some parameters: gitBaseUrl, and superRepo. It will clone the repositories in multiple threads.

lock('checkout') {
    def checkouts = [:]
    def threads = 8

    stage('Super Repo') {
        checkout([$class: 'GitSCM',
                branches: [[name: branch]],
                doGenerateSubmoduleConfigurations: false,
                extensions: [[$class: 'CleanCheckout'], [$class: 'ScmName', name: 'super']],
                submoduleCfg: [],
                userRemoteConfigs: [[url: "${gitBaseUrl}/${superRepo}"]]])

        def repos = readFile('.gitslave')
        def reposLines = repos.readLines()
        for (i = 0; i < threads; i++){
            checkouts["Thread ${i}"] = []
        }
        def idx = 0
        for (line in reposLines) {
            def repoInfo = line.split(' ')
            def repoUrl = repoInfo[0]
            def repoPath = repoInfo[1]
            def curatedRepoUrl = repoUrl.substring(4, repoUrl.length()-1)
            def curatedRepoPath = repoPath.substring(1, repoPath.length()-1)
            echo("Thread ${idx%threads}")
            checkouts["Thread ${idx%threads}"] << [path: curatedRepoPath, url: "${gitBaseUrl}/${curatedRepoUrl}"]
            idx++
        }
    }
    stage('GitSlave Repos') {
        def doCheckouts = [:]
        for (i = 0; i < threads; i++){
            def j = i
            doCheckouts["Thread ${j}"] = {
                for (co in checkouts["Thread ${j}"]) {
                    retry(3) {
                        checkout([$class: 'GitSCM',
                                branches: [[name: branch]],
                                doGenerateSubmoduleConfigurations: false,
                                extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: co.path], [$class: 'CleanCheckout'], [$class: 'ScmName', name: co.path]],
                                submoduleCfg: [],
                                userRemoteConfigs: [[url: co.url]]])
                    }
                }
            }
        }
        parallel doCheckouts
    }
}

That script is under CC0.

Why do we run in threads and not everything in parallel? First, Blue Ocean can only display 100 threads. Running several operations in all the repos would end up with more that 100 threads. Plus it is probably not clever to run such a big amount of parallelism.

What is still to be done? We do not delete repositories that disappear from the .gitslave file.

Here is the result:

GitSlave in Jenkins Pipeline

PS: GitSlave is named GitSlave. I do not like the name – and kudos to the Jenkins team to drop the Master/Slave terminology as part of the Jenkins 2.0 release.

Permalink. Category: Linux. Tags: jenkins planet-inuits.
First published on Fri 18 November 2016.