Re: [Qemu-devel] qemu 2.0, deadlock in block-commit

[Marcin Gibuła]

On 23.05.2014 10:19, Paolo Bonzini wrote:
> Il 22/05/2014 23:05, Marcin Gibuła ha scritto:
> > Some more info.
> > VM was doing lot of write IO during this test.
>
> QEMU is waiting for librados to complete I/O.  Can you reproduce it with
> a different driver?

Hi,

I've reproduced it without RBD. Backtrace below:

(gdb) thread apply all backtrace

Thread 4 (Thread 0x7f9c8cccd700 (LWP 2017)):
#0  0x00007f9c907717a4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9c9076d19c in _L_lock_518 () from /lib64/libpthread.so.0
#2  0x00007f9c9076cfeb in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f9c947addf9 in qemu_mutex_lock 
(address@hidden <qemu_global_mutex>) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/util/qemu-thread-posix.c:76
#4  0x00007f9c946b3a10 in qemu_mutex_lock_iothread () at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/cpus.c:1043
#5  0x00007f9c9470cf3d in kvm_cpu_exec (address@hidden) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/kvm-all.c:1683
#6  0x00007f9c946b271c in qemu_kvm_cpu_thread_fn (arg=0x7f9c968bf290) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/cpus.c:873
#7  0x00007f9c9076af3a in start_thread () from /lib64/libpthread.so.0
#8  0x00007f9c904a4dad in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f9c87fff700 (LWP 2018)):
#0  0x00007f9c9049c897 in ioctl () from /lib64/libc.so.6
#1  0x00007f9c9470cdf9 in kvm_vcpu_ioctl (address@hidden, 
address@hidden) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/kvm-all.c:1796
#2  0x00007f9c9470cf35 in kvm_cpu_exec (address@hidden) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/kvm-all.c:1681
#3  0x00007f9c946b271c in qemu_kvm_cpu_thread_fn (arg=0x7f9c968fa300) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/cpus.c:873
#4  0x00007f9c9076af3a in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9c904a4dad in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f9c869ff700 (LWP 2020)):
#0  0x00007f9c9076ed0c in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00007f9c947ae019 in qemu_cond_wait 
(address@hidden, address@hidden)
    at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/util/qemu-thread-posix.c:135
#2  0x00007f9c946a270b in vnc_worker_thread_loop 
(address@hidden) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/ui/vnc-jobs.c:222
#3  0x00007f9c946a2ae0 in vnc_worker_thread (arg=0x7f9c9695a250) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/ui/vnc-jobs.c:323
#4  0x00007f9c9076af3a in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9c904a4dad in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f9c94448900 (LWP 2013)):
#0  0x00007f9c9049b286 in ppoll () from /lib64/libc.so.6
#1  0x00007f9c9466ed9b in ppoll (__ss=0x0, __timeout=0x0, 
__nfds=<optimized out>, __fds=<optimized out>) at 
/usr/include/bits/poll2.h:77
#2  qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, 
timeout=<optimized out>) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/qemu-timer.c:311
#3  0x00007f9c945027e0 in aio_poll (ctx=0x7f9c95d5bc00, 
address@hidden) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/aio-posix.c:221
#4  0x00007f9c94510c0a in bdrv_drain_all () at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/block.c:1805
#5  0x00007f9c9451787e in bdrv_close (address@hidden) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/block.c:1695
#6  0x00007f9c945175fa in bdrv_delete (bs=0x7f9c969b7d90) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/block.c:1978
#7  bdrv_unref (bs=0x7f9c969b7d90) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/block.c:5198
#8  0x00007f9c94517812 in bdrv_drop_intermediate 
(address@hidden, address@hidden, 
address@hidden)
    at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/block.c:2567
#9  0x00007f9c9451c963 in commit_run (opaque=0x7f9c96a1e280) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/block/commit.c:144
#10 0x00007f9c9455bdca in coroutine_trampoline (i0=<optimized out>, 
i1=<optimized out>) at 
/var/tmp/portage/app-emulation/qemu-2.0.0_rc2/work/qemu-2.0.0-rc2/coroutine-ucontext.c:118
#11 0x00007f9c904009f0 in ?? () from /lib64/libc.so.6
#12 0x00007fffe4bcfee0 in ?? ()
#13 0x0000000000000000 in ?? ()

I still have this process running (hanging ;) ) if you need any more 
info. I also have no problems with reproducing it.

--
mg