bug#6131: [PATCH]: fiemap support for efficient sparse file copy

[Jim Meyering]

Tao Ma wrote:
> Hi Jim,
> On 05/29/2010 12:44 AM, Jim Meyering wrote:
>> Tao Ma wrote:
>>
>>> Hi Jim
>>>
>>> On 05/27/2010 06:30 PM, Jim Meyering wrote:
>>>> jeff.liu wrote:
>>>>> This is the revised version, it fixed the fiemap-start offset calculation
>>>>> approach to remove it out
>>>>> of the 'for (i = 0; i<   fiemap->fm_mapped_extents; i++)' loop.
>>>>
>>>> Hi Jeff,
>>>>
>>>> I've included below the state of my local changes.
>>>> Unfortunately, with that 5-patch series, there is always a test failure
>>>> on F13/ext4.  Maybe someone who knows more about extents can provide an
>>>> explanation?
>>> Just want to clarify why ocfs2 didn't work here. I guess the reason
>>> also works for ext4 since both ext4 and ocfs2 use block group to
>>> organize their blocks in the volume.
>>
>> Hi Tao,
>>
>> Thank you for the explanation.
>> I'm glad to hear that there is no underlying problem.
>>
>>> I checked the perl test script to create sparse src file, it will
>>> create contiguous bytes(around 20-24k) at an interval of around 40k.So
>>> in general, these 20-24k should be contiguous. But that does exist
>>> some scenario that they could be separately into 2 extents. Consider
>>> one block group is used to allocate blocks to this file, when the
>>> block group only has 10K left while you are requiring 20K, it will use
>>> the left 10K in this group and allocate 10K from another block
>>> group. That would become 2 extents since they can't be contiguous.
>>
>>> So I guess the right step is to check the holes by using filefrag if
>>> you prefer this tool and want to make sure cp doesn't copy holes(I get
>>
>> Do you know of a tool other than filefrag that I could use?
> nope.
>>
>> It looks like a small script could filter filefrag -v output, detect
>> split extents and rewrite to make the output match what's expected.
>> Probably not worth it, though, since this is already a very fragile test.

I went ahead and did it, after all.
Here's the script, filefrag-extent-compare.
With it, this test should pass when run on any of those four
file system types.


eval '(exit $?0)' && eval 'exec perl -wS "$0" ${1+"$@"}'
  & eval 'exec perl -wS "$0" $argv:q'
    if 0;
# Determine whether two files have the same extents by comparing
# the logical block numbers and lengths from filefrag -v for each.

# Invoke like this:
# This helper function, f, extracts logical block number and lengths.
# f() { awk '/^ *[0-9]/ {printf "%d %d ",$2,NF<5?$NF:$5} END {print ""}'; }
# { filefrag -v j1 | f; filefrag -v j2 | f; } | ./filefrag-extent-compare

use warnings;
use strict;
(my $ME = $0) =~ s|.*/||;

my @line = <>;
my $n_lines = @line;
$n_lines == 2
  or die "$ME: expected exactly two input lines; got $n_lines\n";

my @A = split ' ', $line[0];
my @B = split ' ', $line[1];
@A % 2 || @B % 2
  and die "$ME: unexpected input: odd number of numbers; expected even\n";

my @a;
my @b;
foreach my $i (address@hidden/2-1) { $a[$i] = { L_BLK => $A[2*$i], LEN => 
$A[2*$i+1] } };
foreach my $i (address@hidden/2-1) { $b[$i] = { L_BLK => $B[2*$i], LEN => 
$B[2*$i+1] } };

my $i = 0;
my $j = 0;
while (1)
  {
    !defined $a[$i] && !defined $b[$j]
      and exit 0;
    defined $a[$i] && defined $b[$j]
      or die "address@hidden and address@hidden have different lengths, even 
after adjustment\n";
    ($a[$i]->{L_BLK} == $b[$j]->{L_BLK}
     && $a[$i]->{LEN} == $b[$j]->{LEN})
      and next;
    ($a[$i]->{LEN} < $b[$j]->{LEN}
     && exists $a[$i+1] && $a[$i]->{LEN} + $a[$i+1]->{LEN} == $b[$j]->{LEN})
      and ++$i, next;
    exists $b[$j+1] && $a[$i]->{LEN} == $b[$i]->{LEN} + $b[$i+1]->{LEN}
      and ++$j, next;
    die "differing extent:\n"
      . "  [$i]=$a[$i]->{L_BLK} $a[$i]->{LEN}\n"
      . "  [$j]=$b[$j]->{L_BLK} $b[$j]->{LEN}\n"
  }
continue
  {
    ++$i;
    ++$j;
  }