Re: [Qemu-devel] Crashing in tcp_close

[Brian Candler]

On 06/11/2016 18:04, Samuel Thibault wrote:
> Brian, could you run it with
>
> export MALLOC_CHECK_=2
>
> and also this could be useful:
>
> export MALLOC_PERTURB_=1234
>
> Also, to rule out the double-free scenario, and try to catch a buffer
> overflow coming from the socket structure itself, I have attached a
> patch which adds some debugging.

Thanks. I've added the patch, and re-run the stress test.

(Aside: since last post I've replaced the Mac Mini with Intel NUCi6KYK 
so it's not *exactly* the same environment, although both machines are 
quad-core i7)

Unfortunately it doesn't crash every time. Here are the first two 
crashes I've managed to obtain, and they don't seem to by anything to do 
with tcp_close, but I have pasted them below. I have kept the coredumps 
if there's anything more useful I can extract from them.

The full command line is something like this (taken from a later run):

/usr/local/bin/qemu-system-x86_64 -m 4G -machine type=pc,accel=kvm 
-netdev user,id=user.0,hostfwd=tcp::2305-:22 -device 
virtio-scsi-pci,id=scsi0 -device scsi-hd,bus=scsi0.0,drive=drive0 
-device virtio-net,netdev=user.0 -name vtp-nmm-201611070837.qcow2 -drive 
if=none,file=output-qemu-vtp-nmm/vtp-nmm-201611070837.qcow2,id=drive0,cache=writeback,discard=unmap,format=qcow2 
-boot c -vnc [::]:24

The following crashes occurred when running with a single vcpu. Normally 
I have been running with -smp 8,sockets=1,cores=4,threads=2 as it seems 
to crash less with those settings; however I'm trying it again like that 
in a loop to see if I can get a crash.

Regards,

Brian.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/bin/qemu-system-x86_64 -m 4G -name 
vtp-nmm-201611062024.qcow2 -machi'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f366c4ce428 in __GI_raise (address@hidden) at 
../sysdeps/unix/sysv/linux/raise.c:54
54      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f366deeea80 (LWP 9030))]
(gdb) bt
#0  0x00007f366c4ce428 in __GI_raise (address@hidden) at 
../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007f366c4d002a in __GI_abort () at abort.c:89
#2  0x00007f366c4c6bd7 in __assert_fail_base (fmt=<optimised out>,
    address@hidden "mr != NULL",
    address@hidden "/home/nsrc/qemu-2.7.0/exec.c", 
address@hidden,
    address@hidden <__PRETTY_FUNCTION__.42881> 
"address_space_unmap")
    at assert.c:92
#3  0x00007f366c4c6c82 in __GI___assert_fail (
    address@hidden "mr != NULL",
    address@hidden "/home/nsrc/qemu-2.7.0/exec.c", 
address@hidden,
    address@hidden <__PRETTY_FUNCTION__.42881> 
"address_space_unmap")
    at assert.c:101
#4  0x000056090749dffe in address_space_unmap (as=<optimised out>, 
buffer=<optimised out>,
    len=<optimised out>, is_write=1, access_len=8192) at 
/home/nsrc/qemu-2.7.0/exec.c:2967
#5  0x00005609075af586 in dma_memory_unmap (access_len=<optimised out>, 
dir=<optimised out>,
    len=<optimised out>, buffer=<optimised out>, as=<optimised out>)
    at /home/nsrc/qemu-2.7.0/include/sysemu/dma.h:144
#6  dma_blk_unmap (address@hidden) at 
/home/nsrc/qemu-2.7.0/dma-helpers.c:102
#7  0x00005609075af766 in dma_complete (ret=0, dbs=0x560909ceca90)
    at /home/nsrc/qemu-2.7.0/dma-helpers.c:113
#8  dma_blk_cb (opaque=0x560909ceca90, ret=0) at 
/home/nsrc/qemu-2.7.0/dma-helpers.c:137
#9  0x000056090775d25a in blk_aio_complete (acb=0x56090909aba0)
    at /home/nsrc/qemu-2.7.0/block/block-backend.c:923
#10 0x00005609077ccaea in coroutine_trampoline (i0=<optimised out>, 
i1=<optimised out>)
    at /home/nsrc/qemu-2.7.0/util/coroutine-ucontext.c:78
#11 0x00007f366c4e35d0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#12 0x00007fffd8d31f20 in ?? ()
#13 0x2d2d2d2d2d2d2d2d in ?? ()
#14 0x00000000000000d0 in ?? ()
#15 0x0000000000000000 in ?? ()
(gdb)


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/bin/qemu-system-x86_64 -boot c -vnc 
[::]:78 -name vtp-nmm-2016110621'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007fc226c98428 in __GI_raise (address@hidden) at 
../sysdeps/unix/sysv/linux/raise.c:54
54      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7fc2286b8a80 (LWP 10267))]
(gdb) bt
#0  0x00007fc226c98428 in __GI_raise (address@hidden) at 
../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007fc226c9a02a in __GI_abort () at abort.c:89
#2  0x00007fc226c90bd7 in __assert_fail_base (fmt=<optimised out>,
    address@hidden "mr != NULL",
    address@hidden "/home/nsrc/qemu-2.7.0/exec.c", 
address@hidden,
    address@hidden <__PRETTY_FUNCTION__.42881> 
"address_space_unmap")
    at assert.c:92
#3  0x00007fc226c90c82 in __GI___assert_fail (
    address@hidden "mr != NULL",
    address@hidden "/home/nsrc/qemu-2.7.0/exec.c", 
address@hidden,
    address@hidden <__PRETTY_FUNCTION__.42881> 
"address_space_unmap")
    at assert.c:101
#4  0x000056186717fffe in address_space_unmap (as=<optimised out>, 
buffer=<optimised out>,
    len=<optimised out>, is_write=1, access_len=4096) at 
/home/nsrc/qemu-2.7.0/exec.c:2967
#5  0x0000561867202beb in virtqueue_unmap_sg 
(address@hidden, address@hidden,
    vq=0x5618695a8500) at /home/nsrc/qemu-2.7.0/hw/virtio/virtio.c:254
#6  0x0000561867203422 in virtqueue_fill (address@hidden,
    address@hidden, len=32876, address@hidden)
    at /home/nsrc/qemu-2.7.0/hw/virtio/virtio.c:282
#7  0x00005618672035db in virtqueue_push (address@hidden,
    address@hidden, len=<optimised out>)
    at /home/nsrc/qemu-2.7.0/hw/virtio/virtio.c:308
#8  0x00005618671f0885 in virtio_scsi_complete_req (req=0x5618694fc610)
    at /home/nsrc/qemu-2.7.0/hw/scsi/virtio-scsi.c:70
#9  0x00005618671f09e6 in virtio_scsi_complete_cmd_req (req=0x5618694fc610)
    at /home/nsrc/qemu-2.7.0/hw/scsi/virtio-scsi.c:443
#10 virtio_scsi_command_complete (r=<optimised out>, status=0, resid=0)
    at /home/nsrc/qemu-2.7.0/hw/scsi/virtio-scsi.c:470
#11 0x0000561867365c98 in scsi_req_complete (req=0x561868a72e40, 
status=<optimised out>)
    at /home/nsrc/qemu-2.7.0/hw/scsi/scsi-bus.c:1775
#12 0x0000561867360210 in scsi_dma_complete_noio (r=0x561868a72e40, 
ret=<optimised out>)
    at /home/nsrc/qemu-2.7.0/hw/scsi/scsi-disk.c:278
---Type <return> to continue, or q <return> to quit---
#13 0x0000561867291779 in dma_complete (ret=0, dbs=0x561868371050)
    at /home/nsrc/qemu-2.7.0/dma-helpers.c:115
#14 dma_blk_cb (opaque=0x561868371050, ret=0) at 
/home/nsrc/qemu-2.7.0/dma-helpers.c:137
#15 0x000056186743f25a in blk_aio_complete (acb=0x561869661b90)
    at /home/nsrc/qemu-2.7.0/block/block-backend.c:923
#16 0x00005618674aeaea in coroutine_trampoline (i0=<optimised out>, 
i1=<optimised out>)
    at /home/nsrc/qemu-2.7.0/util/coroutine-ucontext.c:78
#17 0x00007fc226cad5d0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#18 0x00007ffd10673d70 in ?? ()
#19 0x2d2d2d2d2d2d2d2d in ?? ()
#20 0x00000000000000d4 in ?? ()
#21 0x0000000000000000 in ?? ()
(gdb)