The first thing that you’ll want to do is to add in bricks. Bricks are similar to physical volumes a la LVM. The thing to bear in mind is that depending on what type of cluster you have (replicated / striped), you will need to add a certain number of blocks at a time.

Once you have a initialised the nodes, to add in a set of bricks, you need the following command which adds two more bricks to a cluster which keeps two replicas.

Once you have done this, you will need to rebalance the cluster, which involves redistributing the files across all the bricks. There are two steps to this process, the “fixing” of the layout changes and the rebalancing of the data itself. You can perform both tasks together.

As a starting point, to view the status of a rebalance, you can use:

You can also stop / pause a rebalance with

To “fix” the layout changes, you need to run:

Rebalancing the volume to migrate the data is easy and can be done using a similar command:

To complete both in one command, you just need:

Easy right?

With this mechanism, you have the ability to have storage that can be expanded on the fly by using additional hardware. You can also remove existing bricks using:

This means that you can remove a brick with smaller hard drives, upgrade the harddrives, and re-integrate into the cluster with bigger hard drives. This means that you have a cloud like storage solution which you can easily grow as necessary without worrying about resizing underlying filesystems or hotswapping hardisks or any of that hassle.

On the bright side, there are a few new things that have been worked on so hopefully plenty of things to cover soon.

We manage a number of servers and all over the place and all of them require to be backed up. We also have a number of desktops all with mirrored disks also getting backed up.

I like things to be all nicely efficient and its annoying when one server / desktop runs out of space when another two (or ten) has plenty of space. We grew to dislike NFS particularly due to the single point of failure and there were few other options.

We had tried glusterfs a few years ago (think it was at version 1.3 or something) and there were various issues particularly around small files and configuration was an absolute nightmare.

With high hopes that version 3.2 was exactly what we were looking for, we set up three basic machines for testing

Previously, glusterfs required all the configuration to be completed manually and with text files. It also required a fairly detailed knowledge of what they called translators and a lot of tweaking and fiddling with parameters.

I am very happy to report that this is no longer the case with 3.2.

The three servers(cserver[1-3]) are running Ubuntu and was updated to Natty (11.10) to get access to glusterfs 3.2 (11.04 only had 3.0). One thing to bear in mind is that the glusterfs website seemed to only have the 64 bit version but Ubuntu 11.10 also has the 32bit version.

Installing the server part of glusterfs was simple and straightfoward

Once this was done all three servers (terminator is a godsend when doing these things across a number of servers), adding the servers into a “cluster” was easy enough.

The thing to note is that these probe statements are two way. In other words, all three servers are now part of the same cluster.

The thing is that if you type in an incorrect hostname, the probe will keep trying to connect to it. I haven’t left it running long enough to know if it every returns.

Thankfully, removing host does-not-exist is simple enough

Creating a volume is straightforward. There are a number of different types of volumes which you can find out from the documentation. In this particular instance, we are creating a distributed replicated

The reason I have not included cserver3 in here is that the volume needs a multiple of the replica number of bricks. In the case, the there needs to be a muliple of 2 number of bricks.

Additionally, you could use rdma instead of tcp if you are using infiniband

Starting the volume is simple enough

this volume is now accessible from all the boxes in the cluster

Mounting this from another box is easy

If you get the error of “endpoint not connected” when listing the content of the mount, it is likely because the volume is not started.

If you are curious, check the gdata folders in the bricks after copying some files into the mount and you’ll find them show up intact and on both bricks in the above example.