Re: [Gluster-devel] What does features/index xlator do? Can I move indices dir for online servers to avoid xfs shutdown

[符永涛]

Dear Amar and glusterfs experts,

We use glusterfs 3.3.

xfs related debug info can be found in the following two thread.
http://oss.sgi.com/archives/xfs/2013-04/msg00327.html
http://www.spinics.net/lists/xfs/msg17893.html

The xfs shutdown due to potential racing unlink in xfs kernel on the following files:
directory ./glusterfs/indices/xattrop

and files under this directory

The racing cause xfs unlinked list memory corrupt and then xfs shutdown.

You can see the significant heavy remove/link pattern on a single
inode int the following kernel trace.
https://docs.google.com/file/d/0B7n2C4T5tfNCTlZGUVpnZENrZ3M/edit?usp=sharing

finally
<...>-6908  [001]  8739.967623: xfs_iunlink: dev 8:16 ino 0x83a8b mode
0100000, flags 0x0
<...>-6909  [001]  8739.970252: xfs_iunlink: dev 8:16 ino 0x83a8b mode
0100000, flags 0x0

0x83a8b and 0x1b20b both hash to unlinked list bucket 11.

the shutdown then happens because of corrupt unlinked list.

Now we move the indices directory out of xfs volume to an ext4 volume and the crash not happens again.

This issue always happen when we rebalance glusterfs volumes(for example to growth the cluster).
The worst case ever happened is one day 8 of our servers xfs shutdown at the same time.

The xfs experts is now working on this and before there're any progresses I also want to ask help and sugesstions from you.

Thank you very much.

2013/4/22 Amar Tumballi <address@hidden>

On 04/22/2013 08:58 AM, 符永涛 wrote:
> Dear Amar,
> Sorry to bother but I want to ask some help related to glusterfs.
> Recently we have encountered a xfs shutdown issue related to glusterfs
> rebalance. Finally we found it is related to fops related to
> ./glusterfs/indices/xattrop directory.
> I find it is related to glusterfs features/index xlator.
> My question is what does this xlator do? I find the indices dir can be
> configured in server-side volume files. Is it safe to move is somewhere
> else for online servers?
>

features/index translator keeps track of files which needs self-heal
(with respect to replicate xlator). About changing the volume files (ie,
moving the xlator position in volfile directly), we don't recommend it
as the default setup is the only one which gets tested both by QE and
almost all the user base.

Anyways, couple of questions to you on the issue you are facing.

* Please specify the version of GlusterFS you are using.
* when you say we have encountered a xfs shutdown [1] related "glusterfs
rebalance", can you post glusterfs rebalance logs?

Regards,
Amar

> Thank you very much.
>
> BTW, here's the xfs shutdown issue caused by glusterfs rebalance.
> http://www.marshut.com/irunt/xfs-iunlink-remove-xfs-inotobp-returned-error-22-debugging.html
>
>

--
符永涛