Re: [Qemu-devel] the whole virtual machine hangs when IO does not come back!

[Gonglei (Arei)]

Hi,

Cc'ing Kevin, Stefan and Paolo for more attentions.


Best regards,
-Gonglei


> -----Original Message-----
> Subject: [Qemu-devel] the whole virtual machine hangs when IO does not come
> back!
> 
> Hi,
> 
> I tested the reliability of qemu in the IPSAN environment as follows:
> (1) create one VM on a X86 server which is connected to an IPSAN, and
> the VM has only one system volume which is on the IPSAN;
> (2) disconnect the network between the server and the IPSAN. On the
> server, I have a "multipath" software which can hold the IO for a long
> time (configurable) when the network is disconnected;
> (3) about 30 seconds later, the whole VM hangs there, nothing can be
> done to the VM!
> 
> Then, I used "gstack" tool to collect the stacks of all qemu threads, it
> looked like:
> 
> Thread 8 (Thread 0x7fd840bb5700 (LWP 6671)):
> #0  0x00007fd84253a4f6 in poll () from /lib64/libc.so.6
> #1  0x00007fd84410ceff in aio_poll ()
> #2  0x00007fd84429bb05 in qemu_aio_wait ()
> #3  0x00007fd844120f51 in bdrv_drain_all ()
> #4  0x00007fd8441f1a4a in bmdma_cmd_writeb ()
> #5  0x00007fd8441f216e in bmdma_write ()
> #6  0x00007fd8443a93cf in memory_region_write_accessor ()
> #7  0x00007fd8443a94a6 in access_with_adjusted_size ()
> #8  0x00007fd8443a9901 in memory_region_iorange_write ()
> #9  0x00007fd8443a19bd in ioport_writeb_thunk ()
> #10 0x00007fd8443a13a8 in ioport_write ()
> #11 0x00007fd8443a1f55 in cpu_outb ()
> #12 0x00007fd8443a5b12 in kvm_handle_io ()
> #13 0x00007fd8443a64a9 in kvm_cpu_exec ()
> #14 0x00007fd844330962 in qemu_kvm_cpu_thread_fn ()
> #15 0x00007fd8427e77b6 in start_thread () from /lib64/libpthread.so.0
> #16 0x00007fd8425439cd in clone () from /lib64/libc.so.6
> #17 0x0000000000000000 in ?? ()
> 
> Thread 7 (Thread 0x7fd8403b4700 (LWP 6672)):
> #0  0x00007fd8427ee294 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x00007fd8427e9619 in _L_lock_1008 () from /lib64/libpthread.so.0
> #2  0x00007fd8427e942e in pthread_mutex_lock () from
> /lib64/libpthread.so.0
> #3  0x00007fd8444526bd in qemu_mutex_lock ()
> #4  0x00007fd844330f47 in qemu_mutex_lock_iothread ()
> #5  0x00007fd8443a63b9 in kvm_cpu_exec ()
> #6  0x00007fd844330962 in qemu_kvm_cpu_thread_fn ()
> #7  0x00007fd8427e77b6 in start_thread () from /lib64/libpthread.so.0
> #8  0x00007fd8425439cd in clone () from /lib64/libc.so.6
> #9  0x0000000000000000 in ?? ()
> 
> Thread 6 (Thread 0x7fd83fbb3700 (LWP 6673)):
> #0  0x00007fd8427ee294 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x00007fd8427e9619 in _L_lock_1008 () from /lib64/libpthread.so.0
> #2  0x00007fd8427e942e in pthread_mutex_lock () from
> /lib64/libpthread.so.0
> #3  0x00007fd8444526bd in qemu_mutex_lock ()
> #4  0x00007fd844330f47 in qemu_mutex_lock_iothread ()
> #5  0x00007fd8443a63b9 in kvm_cpu_exec ()
> #6  0x00007fd844330962 in qemu_kvm_cpu_thread_fn ()
> #7  0x00007fd8427e77b6 in start_thread () from /lib64/libpthread.so.0
> #8  0x00007fd8425439cd in clone () from /lib64/libc.so.6
> #9  0x0000000000000000 in ?? ()
> 
> Thread 5 (Thread 0x7fd83f3b2700 (LWP 6674)):
> #0  0x00007fd8427ee294 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x00007fd8427e9619 in _L_lock_1008 () from /lib64/libpthread.so.0
> #2  0x00007fd8427e942e in pthread_mutex_lock () from
> /lib64/libpthread.so.0
> #3  0x00007fd8444526bd in qemu_mutex_lock ()
> #4  0x00007fd844330f47 in qemu_mutex_lock_iothread ()
> #5  0x00007fd8443a63b9 in kvm_cpu_exec ()
> #6  0x00007fd844330962 in qemu_kvm_cpu_thread_fn ()
> #7  0x00007fd8427e77b6 in start_thread () from /lib64/libpthread.so.0
> #8  0x00007fd8425439cd in clone () from /lib64/libc.so.6
> #9  0x0000000000000000 in ?? ()
> 
> Thread 4 (Thread 0x7fd83ebb1700 (LWP 6675)):
> #0  0x00007fd8427ee294 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x00007fd8427e9619 in _L_lock_1008 () from /lib64/libpthread.so.0
> #2  0x00007fd8427e942e in pthread_mutex_lock () from
> /lib64/libpthread.so.0
> #3  0x00007fd8444526bd in qemu_mutex_lock ()
> #4  0x00007fd844330f47 in qemu_mutex_lock_iothread ()
> #5  0x00007fd8443a63b9 in kvm_cpu_exec ()
> #6  0x00007fd844330962 in qemu_kvm_cpu_thread_fn ()
> #7  0x00007fd8427e77b6 in start_thread () from /lib64/libpthread.so.0
> #8  0x00007fd8425439cd in clone () from /lib64/libc.so.6
> #9  0x0000000000000000 in ?? ()
> 
> Thread 3 (Thread 0x7fd83e3b0700 (LWP 6676)):
> #0  0x00007fd8427ee294 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x00007fd8427e9619 in _L_lock_1008 () from /lib64/libpthread.so.0
> #2  0x00007fd8427e942e in pthread_mutex_lock () from
> /lib64/libpthread.so.0
> #3  0x00007fd8444526bd in qemu_mutex_lock ()
> #4  0x00007fd844330f47 in qemu_mutex_lock_iothread ()
> #5  0x00007fd8443a63b9 in kvm_cpu_exec ()
> #6  0x00007fd844330962 in qemu_kvm_cpu_thread_fn ()
> #7  0x00007fd8427e77b6 in start_thread () from /lib64/libpthread.so.0
> #8  0x00007fd8425439cd in clone () from /lib64/libc.so.6
> #9  0x0000000000000000 in ?? ()
> 
> Thread 2 (Thread 0x7fd23b7ff700 (LWP 6679)):
> #0  0x00007fd8427eb61c in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00007fd8444528f0 in qemu_cond_wait ()
> #2  0x00007fd844312d9d in vnc_worker_thread_loop ()
> #3  0x00007fd844313315 in vnc_worker_thread ()
> #4  0x00007fd8427e77b6 in start_thread () from /lib64/libpthread.so.0
> #5  0x00007fd8425439cd in clone () from /lib64/libc.so.6
> #6  0x0000000000000000 in ?? ()
> 
> Thread 1 (Thread 0x7fd844068840 (LWP 6662)):
> #0  0x00007fd8427ee294 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x00007fd8427e9619 in _L_lock_1008 () from /lib64/libpthread.so.0
> #2  0x00007fd8427e942e in pthread_mutex_lock () from
> /lib64/libpthread.so.0
> #3  0x00007fd8444526bd in qemu_mutex_lock ()
> #4  0x00007fd844330f47 in qemu_mutex_lock_iothread ()
> #5  0x00007fd84429b991 in os_host_main_loop_wait ()
> #6  0x00007fd84429ba50 in main_loop_wait ()
> #7  0x00007fd844322793 in main_loop ()
> #8  0x00007fd844329a9f in main ()
> 
> I think the VM hangs there because the  VCPU thread holds the global
> qemu metux lock and waits for IO to come back. However,  in my test, the
> IO doesn't come back (because of the multipath software). Therefore, the
> VCPU thread never releases the global lock, and other threads can never
> get the lock. Is there any idea to solve the whole vm hanging problem?
> I also did the same test on the VMware platform, the IO hangs but the VM
> is still working. Thanks!
>