Re: [Qemu-devel] intermittent make check failure: "tcg_handle_interrupt: assertion failed: (qemu_mutex_iothread_locked())"

[Thomas Huth]

On 01.03.2017 16:19, Alex Bennée wrote:
> 
> Mark Cave-Ayland <address@hidden> writes:
> 
>> On 01/03/17 12:41, Alex Bennée wrote:
>>
>>>> Here's the backtrace from my test case of qemu-system-sparc running on
>>>> an x86_64 host that I posted yesterday:
>>>>
>>>>
>>>> $ gdb --args ./qemu-system-sparc -cdrom
>>>> /home/build/src/qemu/image/sparc32/debian-40r4a-sparc-netinst.iso -boot d
>>>> GNU gdb (GDB) 7.4.1-debian
>>>> Copyright (C) 2012 Free Software Foundation, Inc.
>>>> License GPLv3+: GNU GPL version 3 or later
>>>> <http://gnu.org/licenses/gpl.html>
>>>> This is free software: you are free to change and redistribute it.
>>>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>>>> and "show warranty" for details.
>>>> This GDB was configured as "x86_64-linux-gnu".
>>>> For bug reporting instructions, please see:
>>>> <http://www.gnu.org/software/gdb/bugs/>...
>>>> Reading symbols from /home/build/rel-qemu-git/bin/qemu-system-sparc...done.
>>>> (gdb) r
>>>> Starting program: /home/build/rel-qemu-git/bin/qemu-system-sparc -cdrom
>>>> /home/build/src/qemu/image/sparc32/debian-40r4a-sparc-netinst.iso -boot d
>>>> warning: no loadable sections found in added symbol-file system-supplied
>>>> DSO at 0x7ffff7ffa000
>>>> [Thread debugging using libthread_db enabled]
>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>> [New Thread 0x7fffe9eee700 (LWP 18040)]
>>>> [New Thread 0x7fffe66d6700 (LWP 18041)]
>>>> [New Thread 0x7fffe5ed5700 (LWP 18042)]
>>>> qemu-system-sparc: -cdrom
>>>> /home/build/src/qemu/image/sparc32/debian-40r4a-sparc-netinst.iso:
>>>> warning: bus=0,unit=2 is deprecated with this machine type
>>>> [Thread 0x7fffe66d6700 (LWP 18041) exited]
>>>> [New Thread 0x7fffe66d6700 (LWP 18043)]
>>>> **
>>>> ERROR:/home/build/src/qemu/git/qemu/translate-common.c:34:tcg_handle_interrupt:
>>>> assertion failed: (qemu_mutex_iothread_locked())
>>>>
>>>> Program received signal SIGABRT, Aborted.
>>>> [Switching to Thread 0x7fffe5ed5700 (LWP 18042)]
>>>> 0x00007ffff2939125 in *__GI_raise (sig=<optimized out>) at
>>>> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
>>>>
>>>>
>>>>
>>>> 64      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
>>>> (gdb) thread apply all bt
>>>>
>>>> Thread 5 (Thread 0x7fffe66d6700 (LWP 18043)):
>>>> #0  sem_timedwait () at
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:106
>>>> #1  0x000055555591435c in qemu_sem_timedwait (sem=0x5555561ad408,
>>>> ms=10000) at util/qemu-thread-posix.c:255
>>>> #2  0x000055555590cb51 in worker_thread (opaque=0x5555561ad3a0) at
>>>> util/thread-pool.c:92
>>>> #3  0x00007ffff2c9ab50 in start_thread (arg=<optimized out>) at
>>>> pthread_create.c:304
>>>> #4  0x00007ffff29e4fbd in clone () at
>>>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
>>>> #5  0x0000000000000000 in ?? ()
>>>>
>>>> Thread 4 (Thread 0x7fffe5ed5700 (LWP 18042)):
>>>> #0  0x00007ffff2939125 in *__GI_raise (sig=<optimized out>) at
>>>> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
>>>> #1  0x00007ffff293c3a0 in *__GI_abort () at abort.c:92
>>>> #2  0x00007ffff407e477 in g_assertion_message () from
>>>> /lib/x86_64-linux-gnu/libglib-2.0.so.0
>>>> #3  0x00007ffff407e994 in g_assertion_message_expr () from
>>>> /lib/x86_64-linux-gnu/libglib-2.0.so.0
>>>> #4  0x000055555561b54f in tcg_handle_interrupt (cpu=0x5555561b3af0,
>>>> mask=2) at /home/build/src/qemu/git/qemu/translate-common.c:34
>>>> #5  0x00005555556a8e7d in cpu_interrupt (cpu=0x5555561b3af0, mask=2) at
>>>> /home/build/src/qemu/git/qemu/include/qom/cpu.h:801
>>>> #6  0x00005555556a95ec in cpu_check_irqs (env=0x5555561bbd80) at
>>>> /home/build/src/qemu/git/qemu/hw/sparc/sun4m.c:157
>>>
>>> Am I correct assuming all the helper functions themselves are concerned
>>> with the vCPU environment they are running in? If so it looking like
>>> cpu_check_irqs() would be the place to wrap the BQL lock in.
>>>
>>> I'll make the change and test against your test case.
>>
>> For the window helpers, definitely yes - each SPARC vCPU contains a
>> large bank of registers of which the current window is a pointer as to
>> which register set is currently active. The current window itself is
>> controlled by a special register called CWP (current window pointer)
>> which is part of the PSR register in 32-bit CPUs or a standalone
>> register for 64-bit CPUs.
> 
> Actually looking at where cpu_check_irq is I've pushed the BQL wraps
> higher up:
> 
>   
> https://github.com/stsquad/qemu/commit/f52e01bf4d3415ce4fad54cbb491f6c30a8e1903
> 
> The reasoning is HW emulation code can expect to be run under the BQL.
> Anything triggered by MMIO for example will automatically have the lock.
> The problem is helpers which trigger hardware actions (ARM has ARM_CP_IO
> for this). We talk about this in the design document:
> 
>   
> http://git.qemu-project.org/?p=qemu.git;a=blob;f=docs/multi-thread-tcg.txt;h=a99b4564c6258576acfa141f51f4b80131969519;hb=HEAD#l201
> 
>> I ran out of time yesterday to run through all my SPARC64 tests, however
>> if this is the case then the issue will definitely appear in
>> qemu-system-sparc64 which seems to be supported by people seeing
>> intermittent failures in prom-env-test.
> 
> So I made the change to both sparc and sparc64 in the above commit.
> 
> I haven't managed to run into the intermittent trigger yet. How do I run
> the test directly?
> 
>   make tests/prom-env-test
> 
> Just makes the test case and
> 
>   ./tests/prom-env-test
> **
> ERROR:tests/libqtest.c:589:qtest_get_arch: assertion failed: (qemu != NULL)
> Aborted (core dumped)

QTEST_QEMU_BINARY=sparc64-softmmu/qemu-system-sparc64 \
 tests/prom-env-test

... that should do the job here.
(or use "make check-qtest-sparc64" to run all sparc64 tests)

 Thomas