[Qemu-devel] Debugging io deadlock

[BALATON Zoltan]

Hello,

I'm seeing a possible deadlock that I don't know how to debug. Any hint on 
how to find the cause or what should be checked further to identify the 
reason why this is happening and how to fix it is greatly appreciated.

Here are the state of threads:

(gdb) info thr
  Id   Target Id         Frame
* 4    Thread 0x7fffba76c700 (LWP 3445) "qemu-system-ppc" 0x0000555555cbec04 in 
worker_thread (opaque=0x7fffe40c9000) at util/thread-pool.c:92
  3    Thread 0x7fffe8829700 (LWP 3443) "qemu-system-ppc" 0x00007ffff78d267f in 
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2    Thread 0x7ffff111b700 (LWP 3442) "qemu-system-ppc" 0x00007ffff42cad29 in 
syscall () from /lib64/libc.so.6
  1    Thread 0x7ffff7fc7b00 (LWP 3441) "qemu-system-ppc" 0x00007ffff42c4e31 in 
ppoll () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff78d4830 in sem_timedwait () from /lib64/libpthread.so.0
#1  0x0000555555cc572e in qemu_sem_timedwait (sem=0x7fffe40c9078, ms=10000) at 
util/qemu-thread-posix.c:289
#2  0x0000555555cbec04 in worker_thread (opaque=0x7fffe40c9000) at 
util/thread-pool.c:92
#3  0x00007ffff78cd5bd in start_thread () from /lib64/libpthread.so.0
#4  0x00007ffff42d062d in clone () from /lib64/libc.so.6
(gdb) thr 3
[Switching to thread 3 (Thread 0x7fffe8829700 (LWP 3443))]
#0  0x00007ffff78d267f in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff78d267f in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x0000555555cc5458 in qemu_cond_wait (cond=0x555556b47b90, mutex=0x5555565b5220 
<qemu_global_mutex>) at util/qemu-thread-posix.c:161
#2  0x00005555557e6690 in qemu_tcg_wait_io_event (cpu=0x7ffff7e20010) at 
cpus.c:1084
#3  0x00005555557e6f00 in qemu_tcg_rr_cpu_thread_fn (arg=0x7ffff7e20010) at 
cpus.c:1396
#4  0x00007ffff78cd5bd in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff42d062d in clone () from /lib64/libc.so.6
(gdb) thr 2
[Switching to thread 2 (Thread 0x7ffff111b700 (LWP 3442))]
#0  0x00007ffff42cad29 in syscall () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff42cad29 in syscall () from /lib64/libc.so.6
#1  0x0000555555cc58a7 in qemu_futex_wait (f=0x555556a01134 
<rcu_call_ready_event>, val=4294967295) at include/qemu/futex.h:29
#2  0x0000555555cc5a74 in qemu_event_wait (ev=0x555556a01134 
<rcu_call_ready_event>) at util/qemu-thread-posix.c:442
#3  0x0000555555cdd92c in call_rcu_thread (opaque=0x0) at util/rcu.c:249
#4  0x00007ffff78cd5bd in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff42d062d in clone () from /lib64/libc.so.6
(gdb) thr 1
[Switching to thread 1 (Thread 0x7ffff7fc7b00 (LWP 3441))]
#0  0x00007ffff42c4e31 in ppoll () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff42c4e31 in ppoll () from /lib64/libc.so.6
#1  0x0000555555cbfe86 in qemu_poll_ns (fds=0x555557c17620, nfds=5, 
timeout=29806320) at util/qemu-timer.c:334
#2  0x0000555555cc0eab in os_host_main_loop_wait (timeout=29806320) at 
util/main-loop.c:255
#3  0x0000555555cc0f7d in main_loop_wait (nonblocking=0) at util/main-loop.c:515
#4  0x000055555599e2b3 in main_loop () at vl.c:1995
#5  0x00005555559a6353 in main (argc=21, argv=0x7fffffffdef8, 
envp=0x7fffffffdfa8) at vl.c:4911

Then if I wait a little, thread 4 exits due to sem_timedwait returning -1 
with errno=ETIMEDOUT leaving other threads waiting for something to happen 
but this is apparently a deadlock as it will be stuck here (thread 1-3 are 
still as above). Any idea why this could happen and how to debug it 
furhter?

Thank you,
BALATON Zoltan