Peering a server to a GlusterFS cluster

Some notes about how to add server to a GlusterFS cluster and to replicata data to it.

srv03$ sudo gluster peer probe
peer probe: failed: is already part of another cluster

It is logical that anyone may not add a server to a gluster cluster.

srv01$ sudo gluster peer probe
peer probe: success

It is successful.

srv01$ sudo gluster peer status
Number of Peers: 2

Uuid: 99f74112-cca9-439e-802e-a7822da446c9
State: Peer in Cluster (Connected)

Port: 24007
Uuid: 9099c59c-a667-4bb4-b5ea-fb6bc08e1e9a
State: Peer in Cluster (Connected)

And, first try to add the new node to the cluster will be a failure:

srv02$ sudo gluster volume add-brick gv0
volume add-brick: failed: Incorrect number of bricks supplied 1 with count 2

When I created the volume, I set the replica value to 2, so I guess Gluster wants me to add “bricks” two by two. Instead, I am going to update the replica value.

srv02$ sudo gluster volume add-brick gv0 replica 3
volume add-brick: success

Are the file replicated on the third node? Not yet.

srv03$ ls /export/brick1/sdb1/ | wc -l

But, the first time I list the files in the mounted volume, the files are replicated:

srv01$ ls /mnt|wc -l

srv03$ ls /export/brick1/sdb1/ | wc -l

And I am done.

Permalink. Category: Linux. Tags: GlusterFS planet-inuits.
First published on Thu 18 July 2013.

Fixing a GlusterFS split-brain

I have found some time to test GlusterFS, in order to use it to replicate data between several servers.

I have created two vagrant boxes to set up a basic cluster.

And, during my tests, I have shut down the network interfaces and started writing different things on the same file, to create a split-brain.

srv01$ echo good > /mnt/test
srv02$ echo bad > /mnt/test

When the interfaces were up again, the split-brain was obviously present:

srv02$ cat /mnt/test
cat: /mnt/test: Input/output error

In the logs, I have found the following sentence:

[2013-07-17 13:57:54.156318] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log]
0-gv0-replicate-0: Unable to self-heal contents of '<gfid:470a742c-b0d6-4846-9ab3-2483c3a0c8da>'
(possible split-brain). Please delete the file from all but the preferred subvolume.-
Pending matrix:  [ [ 0 1 ] [ 1 0 ] ]

So I tried to delete the file in the node that was “wrong” to me:

srv02$ cat /export/brick1/sdb1/test
srv02$ sudo rm /export/brick1/sdb1/test

But it was obviously not working:

srv02$ cat /mnt/test
cat: /mnt/test: Input/output error

Even more, the file /export/brick1/sdb1/test was recreated by Gluster!

srv02$ cat /export/brick1/sdb1/test

The solution was the following: Gluster is creating hard links in a .gluster directory, and you have to delete all the hard links to the file to get rid of it. Please note that I am working on the ‘brick’.

srv02$ sudo find /export/brick1/sdb1/ -samefile /export/brick1/sdb1/test -print -delete

And it worked!

srv02$ cat /mnt/test
srv02$ cat /export/brick1/sdb1/test

Additional notes

Only the files that were “brain-splitted” were unreadable.

You can have the list of these files by running the following command:

$ gluster volume heal gv0 info
Number of entries: 1

Number of entries: 1

To avoid split-brains, you can enable quorums.

Permalink. Category: Linux. Tags: GlusterFS planet-inuits.
First published on Wed 17 July 2013.

Fill your Belgian tax form with Linux

Tax-on-web is the online service that enable you to fill in your Belgian tax form.

In order to be able to read you beid card, and so to connect to the online website, follow the following instructions.

First, get a card reader, and install the relevant driver. The following will often be enough (if you are using debian/ubuntu):

$ sudo apt-get install libacr38u pcscd

Then, do not install the software provided by the government.

Instead, install opensc.

$ sudo apt-get install opensc

Open Firefox, and go to Edit > Preferences > Certificates > Security devices.

Click on Load then enter “Beid” as module name (it does not matter), and put /usr/lib/ as second field. You might need to select it with the Browse button. If the file does not exist, try /usr/lib64/

You might need to start openscd:

$ sudo service openscd start

Restart Firefox, and go to tax-on-web.

You are done.

Permalink. Category: Linux. Tags: belgium planet-inuits.
First published on Mon 15 July 2013.

Rattaching a running process to a screen

Reptyr is a small project written in C that enable you to “re-pty” your process. It means you can, for example, attach a running process inside a screen.

For example, you launched irssi outside your screen, and you do not want to relaunch it.

Install reptyr, then, in a screen, run the following commands:

$ pidof irssi
$ reptyr 27518

That’s it. The process will be rattached to Init but you will be able to use it directly from your screen, without notifying the difference.

Tested and approved to rattach a 6-hour process launched outside a screen.

  • Reptyr: Tested and approved
  • Retty: Alternative, more limitations and only for x86
Permalink. Category: Linux. Tags: sysadmin screen planet-inuits.
First published on Tue 25 June 2013.

Grok and httpd

When you use the LogFormat parameter in you apache configuration, you may want to use the same logformat in logstash.

I made a small table that will enable you to “translate” your custom log format into a grok pattern.

httpd grok
%T %{NUMBER:duration_seconds}
%D %{NUMBER:duration_microseconds}
%h %{IPORHOST:client}
%l %{USERNAME:remotelogname}
%u %{USERNAME:username}
%t \[%{HTTPDATE:timestamp}\]
%r %{USERNAME:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}
%s %{NUMBER:response}
%B %{NUMBER:bytes}
%b (?:%{NUMBER:bytes}|-)
\"%{Referer}i\" %{QS:referer}
\"%{User-Agent}i\" %{QS:useragent}
Permalink. Category: Monitoring. Tags: grok logstash planet-inuits.
First published on Mon 17 June 2013.