Fixing a GlusterFS split-brain

I have found some time to test GlusterFS, in order to use it to replicate data between several servers.

I have created two vagrant boxes to set up a basic cluster.

And, during my tests, I have shut down the network interfaces and started writing different things on the same file, to create a split-brain.

srv01$ echo good > /mnt/test
srv02$ echo bad > /mnt/test

When the interfaces were up again, the split-brain was obviously present:

srv02$ cat /mnt/test
cat: /mnt/test: Input/output error

In the logs, I have found the following sentence:

[2013-07-17 13:57:54.156318] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log]
0-gv0-replicate-0: Unable to self-heal contents of '<gfid:470a742c-b0d6-4846-9ab3-2483c3a0c8da>'
(possible split-brain). Please delete the file from all but the preferred subvolume.-
Pending matrix:  [ [ 0 1 ] [ 1 0 ] ]

So I tried to delete the file in the node that was “wrong” to me:

srv02$ cat /export/brick1/sdb1/test
srv02$ sudo rm /export/brick1/sdb1/test

But it was obviously not working:

srv02$ cat /mnt/test
cat: /mnt/test: Input/output error

Even more, the file /export/brick1/sdb1/test was recreated by Gluster!

srv02$ cat /export/brick1/sdb1/test

The solution was the following: Gluster is creating hard links in a .gluster directory, and you have to delete all the hard links to the file to get rid of it. Please note that I am working on the ‘brick’.

srv02$ sudo find /export/brick1/sdb1/ -samefile /export/brick1/sdb1/test -print -delete

And it worked!

srv02$ cat /mnt/test
srv02$ cat /export/brick1/sdb1/test

Additional notes

Only the files that were “brain-splitted” were unreadable.

You can have the list of these files by running the following command:

$ gluster volume heal gv0 info
Number of entries: 1

Number of entries: 1

To avoid split-brains, you can enable quorums.

Permalink. Category: Linux. Tags: GlusterFS planet-inuits.
First published on Wed 17 July 2013.

Fill your Belgian tax form with Linux

Tax-on-web is the online service that enable you to fill in your Belgian tax form.

In order to be able to read you beid card, and so to connect to the online website, follow the following instructions.

First, get a card reader, and install the relevant driver. The following will often be enough (if you are using debian/ubuntu):

$ sudo apt-get install libacr38u pcscd

Then, do not install the software provided by the government.

Instead, install opensc.

$ sudo apt-get install opensc

Open Firefox, and go to Edit > Preferences > Certificates > Security devices.

Click on Load then enter “Beid” as module name (it does not matter), and put /usr/lib/ as second field. You might need to select it with the Browse button. If the file does not exist, try /usr/lib64/

You might need to start openscd:

$ sudo service openscd start

Restart Firefox, and go to tax-on-web.

You are done.

Permalink. Category: Linux. Tags: belgium planet-inuits.
First published on Mon 15 July 2013.

Rattaching a running process to a screen

Reptyr is a small project written in C that enable you to “re-pty” your process. It means you can, for example, attach a running process inside a screen.

For example, you launched irssi outside your screen, and you do not want to relaunch it.

Install reptyr, then, in a screen, run the following commands:

$ pidof irssi
$ reptyr 27518

That’s it. The process will be rattached to Init but you will be able to use it directly from your screen, without notifying the difference.

Tested and approved to rattach a 6-hour process launched outside a screen.

  • Reptyr: Tested and approved
  • Retty: Alternative, more limitations and only for x86
Permalink. Category: Linux. Tags: sysadmin screen planet-inuits.
First published on Tue 25 June 2013.

Grok and httpd

When you use the LogFormat parameter in you apache configuration, you may want to use the same logformat in logstash.

I made a small table that will enable you to “translate” your custom log format into a grok pattern.

httpd grok
%T %{NUMBER:duration_seconds}
%D %{NUMBER:duration_microseconds}
%h %{IPORHOST:client}
%l %{USERNAME:remotelogname}
%u %{USERNAME:username}
%t \[%{HTTPDATE:timestamp}\]
%r %{USERNAME:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}
%s %{NUMBER:response}
%B %{NUMBER:bytes}
%b (?:%{NUMBER:bytes}|-)
\"%{Referer}i\" %{QS:referer}
\"%{User-Agent}i\" %{QS:useragent}
Permalink. Category: Monitoring. Tags: grok logstash planet-inuits.
First published on Mon 17 June 2013.