[Bug-tar] [PATCH] improved sparse file detection

[Bernd Schubert]

Hi all,

for improved stat() performance the Lustre filesystem uses entirely empty 
sparse files on its metadata target (MDT). Now with hundredes of millions of 
sparse file of huge sizes, creating a backup of of the MDT using vanilla 
gnu-tar is basically impossible, as it needs far too much time to detect 
sparse files.

Attached is a patch to improve sparse file detection, if the entire file is 
sparse. All credits go to my colleague Kit Westneat as it was his idea.

Example: Directory with a single 4 TiB  sparse file:

default-tar:

address@hidden@bathl:/tmpa/tests$ time tar cvfS sparse.tar sparse
sparse/
^C

real    0m14.710s
user    0m7.100s
sys     0m7.450s

(I aborted with ctrl-c, tar was running at 99% CPU time)

Improved tar:

address@hidden@bathl:/tmpa/tests$ time 
/tmpa/devel/Lustre/tar/tar-1.22-13.el6/tar-1.22/src/tar cvfS sparse.tar sparse
sparse/
sparse/ost2.img

real    0m0.006s
user    0m0.000s
sys     0m0.000s


The patch still has a TODO comment, as tar_sparse_scan(file, ...) may call
file->optab->scan_block() and simply did not figure out yet where the 
scan_block() method is assigned (if at all, dead code?).

Any comment is appreciated.


Thanks,
Bernd

PS: I'm used to linux-style indentation and I'm not sure if I did it the right 
way. If it is wrong, please complain and I will try to reformat it.

-- 
Bernd Schubert
DataDirect Networks

improve_sparse_detection.patch
Description: Text Data