[Qemu-devel] Re: virtio-blk XFS corruption

[Peter Lieven]

Am 25.09.2010 um 16:44 schrieb Stefan Hajnoczi:

> On Sat, Sep 25, 2010 at 2:43 PM, Peter Lieven <address@hidden> wrote:
>> we experience filesystem corruption using virtio-blk on some guest systems 
>> togehter with XFS. We still use qemu-kvm 0.12.4.
> [...]
>> It seems that 64-bit Ubuntu LTS 10.04.1 is affected as well as an older 
>> openSuse 11.1 system with kernel 2.6.27.45-0.1-pae. Suprisingly we have
>> an openSuse 11.1 with 2.6.27.45-0.1-default working absolutely stable for 
>> months.
> 
> Affected guests: 64-bit Ubuntu LTS 10.04.1, openSuse 11.1 2.6.27.45-0.1-pae
> Unaffected guests: openSuse 11.1 2.6.27.45-0.1-default
> qemu-kvm version: 0.12.4
> qemu-kvm command-line: ?
/usr/bin/qemu-kvm-0.12.4  -net tap,vlan=140,script=no,downscript=no,ifname=tap6 
-net nic,vlan=140,model=e1000,macaddr=52:54:00:fe:00:ee   -drive 
format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-7c5e1ab04-c6f7a1269cc4c99d-ubuntu-newsfeed-system,if=ide,boot=on,cache=none,aio=native
  -drive 
format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-a1e4ce107-9c6000e78ec4c988-ubuntu-newsfeed-data,if=scsi,boot=off,cache=none,aio=native
  -drive 
format=host_device,file=/dev/mapper/iqn.2001-05.com.equallogic:0-8a0906-fb34ce107-9ff000e78ef4c98a-ubuntu-newsfeed-data2,if=virtio,boot=off,cache=none,aio=native
  -m 4096 -smp 4 -monitor tcp:0:4012,server,nowait -vnc :12 -name 
'ubuntu-newsfeed'  -boot order=dc,menu=off  -k de  -pidfile 
/var/run/qemu/vm-244.pid  -mem-path /hugepages -mem-prealloc  -cpu 
qemu64,model_id='Intel(R) Xeon(R) CPU           L5640  @ 2.27GHz',-tsc,-nx  
-rtc base=utc -no-hpet -no-kvm-clock 

> Disk image format: ?

raw host_device (iscsi san via open-iscsi and multipathd)

> Steps to reproduce: ?

we encounter this problem only when running a diablo newsserver with heavy i/o. 
however, i will try to construct a test scenario that can
be reproduced.

might this bug related to the barrier fix for virtio_blk that has been 
introduced in 0.12.5. ?

> 
> Can you please provide information on the unknown items above?
> 
> Does this happen with IDE (-drive file=myimage.img,if=ide) or only
> with virtio-blk?

if i take the same vm and change the emulation to scsi the system works without 
any problems.

> 
>> The only thing I have seen in the syslog of the 64-bit Ubuntu LTS 10.04.1 is 
>> the following:
>> 
>> [19001.346897] Filesystem "vda1": XFS internal error xfs_trans_cancel at 
>> line 1162 of file /build/buildd/linux-2.6.32/fs/xfs/xfs_trans.c.  Caller 
>> 0xffffffffa013091d
>> [19001.346897]
>> [19002.174492] Pid: 1210, comm: diablo Not tainted 2.6.32-24-server 
>> #43-Ubuntu
>> [19002.174492] Call Trace:
>> [19002.174492]  [<ffffffffa010f403>] xfs_error_report+0x43/0x50 [xfs]
>> [19002.174492]  [<ffffffffa013091d>] ? xfs_create+0x1dd/0x5f0 [xfs]
>> [19002.174492]  [<ffffffffa012bb35>] xfs_trans_cancel+0xf5/0x120 [xfs]
>> [19002.174492]  [<ffffffffa013091d>] xfs_create+0x1dd/0x5f0 [xfs]
> 
> XFS has given up because of an error in xfs_create() but I don't think
> this output gives enough information to determine what the error was.

ok, thank you. here is the log output from the opensuse system:

Sep 20 15:44:04 newsfeed2-neu kernel: ------------[ cut here ]------------
Sep 20 15:44:04 newsfeed2-neu kernel: WARNING: at fs/inode.c:602 
unlock_new_inode+0x29/0x49()
Sep 20 15:44:04 newsfeed2-neu kernel: Modules linked in: af_packet nls_utf8 
joydev st ide_disk ide_cd_mod binfmt_misc ipv6 fuse xfs loop dm_mod virtio_blk 
i2c_piix4 virtio_pci sr_mod ppdev e1000 rtc_cmos pcspkr i2c_core virtio_ring 
cdrom button rtc_core virtio parport_pc rtc_lib parport floppy sg sd_mod 
crc_t10dif edd ext3 mbcache jbd fan ide_pci_generic piix ide_core ata_generic 
ata_piix libata scsi_mod dock thermal processor thermal_sys hwmon [last 
unloaded: speedstep_lib]
Sep 20 15:44:04 newsfeed2-neu kernel: Supported: No, Unsupported modules are 
loaded
Sep 20 15:44:04 newsfeed2-neu kernel: Pid: 14732, comm: diablo Tainted: G       
 W 2.6.27.45-0.1-pae #1
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c010622c>] dump_trace+0x6b/0x249
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0106d03>] show_trace+0x20/0x39
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0349303>] dump_stack+0x71/0x76
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0129d50>] warn_on_slowpath+0x4d/0x70
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c01a324d>] unlock_new_inode+0x29/0x49
Sep 20 15:44:04 newsfeed2-neu kernel:  [<f8f75148>] xfs_ialloc+0x516/0x52c [xfs]
Sep 20 15:44:04 newsfeed2-neu kernel:  [<f8f88519>] xfs_dir_ialloc+0x7f/0x27a 
[xfs]
Sep 20 15:44:04 newsfeed2-neu kernel:  [<f8f8b501>] xfs_create+0x2be/0x549 [xfs]
Sep 20 15:44:04 newsfeed2-neu kernel:  [<f8f9545c>] xfs_vn_mknod+0x138/0x205 
[xfs]
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0199c9c>] vfs_create+0x12e/0x1a2
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c019c102>] do_filp_open+0x1b8/0x6ee
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0190c17>] do_sys_open+0x46/0xcc
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0190ceb>] sys_open+0x23/0x28
Sep 20 15:44:04 newsfeed2-neu kernel:  [<c0104ad2>] syscall_call+0x7/0xb
Sep 20 15:44:04 newsfeed2-neu kernel:  [<ffffe422>] 0xffffe422
Sep 20 15:44:04 newsfeed2-neu kernel:  =======================
Sep 20 15:44:04 newsfeed2-neu kernel: ---[ end trace b8b179fad244f381 ]---


> 
> Stefan

thanks,
peter