Re: [PATCH] tests: fix false du failure on newer XFS

[Bernhard Voelker]

On 09/16/2014 11:40 AM, Philipp Thomas wrote:
> * Pádraig Brady (address@hidden) [20140916 02:44]:
> 
>> What was the actual allocation reported by du?
> 
> As I was the one witnessing the original bug I can answer that :) Du thought
> the file being 4 gig in size which made the test fail unexpectantly.
> 
> And as the testsuite is run with every build of the coreutils package
> (without the tests requiring root since building as root in the open build
> service is more or less forbidden), a failing test suite results in a failed
> build of coreutils.


I can confirm this locally, too.
I created 2 loopback-mounted file systems for ext4 and xfs, and mounted them
for a VPATH build:

  $ dd if=/dev/zero of=/mnt/xfs.img  bs=1G count=6
  $ dd if=/dev/zero of=/mnt/ext4.img bs=1G count=6

  $ mkfs.ext4 /mnt/ext4.img
  $ mkfs.xfs  /mnt/xfs.img

  $ mount -o loop /mnt/ext4.img /media/sdb5/berny/git/coreutils/build-ext4
  $ mount -o loop /mnt/xfs.img /media/sdb5/berny/git/coreutils/build-xfs

  $ chown berny /media/sdb5/berny/git/coreutils/build-ext4
  $ chown berny /media/sdb5/berny/git/coreutils/build-xfs

Then I did the usual combo in both directories:

  $ ../configure --quiet && make

Before running the test, I modified it like this;

diff --git a/tests/du/2g.sh b/tests/du/2g.sh
index f766d4d..33147a9 100755
--- a/tests/du/2g.sh
+++ b/tests/du/2g.sh
@@ -43,6 +43,7 @@ test $min_kb -lt $free_kb ||

 big=big
 rm -f $big
+df --out --block-size=1 .
 test -t 1 || printf 'creating a 2GB file...\n'
 for i in $(seq 100); do
   # Note: 2147483648 == 2^31. Print floor(2^31/100) per iteration.
@@ -52,6 +53,12 @@ for i in $(seq 100); do
   test -t 1 && printf 'creating a 2GB file: %d%% complete\r' $i
 done
 echo
+df --out --block-size=1 .
+
+stat $big
+ls -l $big
+du --block-size=1 $big
+du --block-size=1 --apparent-size $big

 du -k $big > out1 || fail=1
 rm -f $big

Then I ran the test on both the ext4 and the xfs file system:

  $ make check-very-expensive SUBDIRS=. V=1 TESTS=tests/du/2g.sh

This is the difference between the logs:

...
@@ -78,15 +78,15 @@
 + test yes '!=' yes
 ++ df -k --output=avail .
 ++ tail -n1
-+ free_kb=5651196
++ free_kb=6196824
 + case "$free_kb" in
 + min_kb=3000000
-+ test 3000000 -lt 5651196
++ test 3000000 -lt 6196824
 + big=big
 + rm -f big
 + df --out --block-size=1 .
-Filesystem     Type Inodes IUsed  IFree IUse%  1B-blocks     Used      Avail 
Use% File Mounted on
-/dev/loop2     ext4 393216  1623 391593    1% 6207111168 81387520 5786824704   
2% .
/media/sdb5/berny/git/coreutils/build-ext4
+Filesystem     Type  Inodes IUsed   IFree IUse%  1B-blocks     Used      Avail 
Use% File Mounted on
+/dev/loop1     xfs  6291456  1615 6289841    1% 6431965184 86417408 6345547776 
  2% .
/media/sdb5/berny/git/coreutils/build-xfs
 + test -t 1
 + printf 'creating a 2GB file...\n'
 creating a 2GB file...
@@ -495,21 +495,21 @@
 + echo

 + df --out --block-size=1 .
-Filesystem     Type Inodes IUsed  IFree IUse%  1B-blocks       Used      Avail 
Use% File Mounted on
-/dev/loop2     ext4 393216  1624 391592    1% 6207111168 2214813696 3653398528 
 38% .
/media/sdb5/berny/git/coreutils/build-ext4
+Filesystem     Type  Inodes IUsed   IFree IUse%  1B-blocks       Used      
Avail Use% File Mounted on
+/dev/loop1     xfs  6291456  1616 6289840    1% 6431965184 4377755648 
2054209536  69% .
/media/sdb5/berny/git/coreutils/build-xfs
 + stat big
   File: 'big'
-  Size: 2147483648     Blocks: 4194312    IO Block: 4096   regular file
-Device: 702h/1794d     Inode: 1011        Links: 1
+  Size: 2147483648     Blocks: 8364928    IO Block: 4096   regular file
+Device: 701h/1793d     Inode: 25174390    Links: 1
 Access: (0644/-rw-r--r--)  Uid: ( 1000/   berny)   Gid: (  100/   users)
-Access: 2014-09-16 12:27:17.948898626 +0200
-Modify: 2014-09-16 12:27:21.650945129 +0200
-Change: 2014-09-16 12:27:21.650945129 +0200
+Access: 2014-09-16 12:27:27.258015564 +0200
+Modify: 2014-09-16 12:27:30.225052837 +0200
+Change: 2014-09-16 12:27:30.225052837 +0200
  Birth: -
 + ls -l big
 -rw-r--r-- 1 berny users 2147483648 Sep 16 12:27 big
 + du --block-size=1 big
-2147487744     big
+4282843136     big
 + du --block-size=1 --apparent-size big
 2147483648     big
 + du -k big
@@ -525,15 +525,21 @@
 + case $? in
 + compare_ exp out
 + diff -u exp out
-+ Exit 0
+--- exp        2014-09-16 12:27:33.461093490 +0200
++++ out        2014-09-16 12:27:33.460093477 +0200
+@@ -1 +1 @@
+-~2M
++4182464       big
++ fail=1
++ Exit 1
 + set +e
-+ exit 0
-+ exit 0
++ exit 1
++ exit 1
 + remove_tmp_
-+ __st=0
++ __st=1
 + cleanup_
 + :

I repeated the test with having XFS on a bare partition to rule out the
loopback driver, and the result was the same.

Finally, here's the strace-output on EXT4:

newfstatat(AT_FDCWD, "big", {st_dev=makedev(7, 2), st_ino=1011, 
st_mode=S_IFREG|0644, st_nlink=1, st_uid=1000,
st_gid=100, st_blksize=4096, st_blocks=4194312, st_size=2147483648, 
st_atime=2014/09/16-12:45:05,
st_mtime=2014/09/16-12:45:09, st_ctime=2014/09/16-12:45:09}, 
AT_SYMLINK_NOFOLLOW) = 0
fstat(1, {st_dev=makedev(7, 2), st_ino=1012, st_mode=S_IFREG|0644, st_nlink=1, 
st_uid=1000, st_gid=100, st_blksize=4096,
st_blocks=0, st_size=0, st_atime=2014/09/16-12:45:09, 
st_mtime=2014/09/16-12:45:09, st_ctime=2014/09/16-12:45:09}) = 0

versus that on XFS:

newfstatat(AT_FDCWD, "big", {st_dev=makedev(7, 1), st_ino=1091, 
st_mode=S_IFREG|0644, st_nlink=1, st_uid=1000,
st_gid=100, st_blksize=4096, st_blocks=8364928, st_size=2147483648, 
st_atime=2014/09/16-12:45:17,
st_mtime=2014/09/16-12:45:20, st_ctime=2014/09/16-12:45:20}, 
AT_SYMLINK_NOFOLLOW) = 0
fstat(1, {st_dev=makedev(7, 1), st_ino=1092, st_mode=S_IFREG|0644, st_nlink=1, 
st_uid=1000, st_gid=100, st_blksize=4096,
st_blocks=0, st_size=0, st_atime=2014/09/16-12:45:20, 
st_mtime=2014/09/16-12:45:20, st_ctime=2014/09/16-12:45:20}) = 0


So XFS really allocates 4G for a 2G file. ;-/
I consider this a bug in XFS - maybe on {open,}SUSE only? - but the test
shouldn't fail, because it should only prove that du(1) doesn't overflow
on the 2G limit.

Have a nice day,
Berny