Re: [Qemu-devel] Crashing in tcp_close

[Stefan Hajnoczi]

On Mon, Nov 07, 2016 at 08:42:17AM +0000, Brian Candler wrote:
> On 06/11/2016 18:04, Samuel Thibault wrote:
> > Brian, could you run it with
> > 
> > export MALLOC_CHECK_=2
> > 
> > and also this could be useful:
> > 
> > export MALLOC_PERTURB_=1234
> > 
> > Also, to rule out the double-free scenario, and try to catch a buffer
> > overflow coming from the socket structure itself, I have attached a
> > patch which adds some debugging.
> 
> Thanks. I've added the patch, and re-run the stress test.
> 
> (Aside: since last post I've replaced the Mac Mini with Intel NUCi6KYK so
> it's not *exactly* the same environment, although both machines are
> quad-core i7)
> 
> Unfortunately it doesn't crash every time. Here are the first two crashes
> I've managed to obtain, and they don't seem to by anything to do with
> tcp_close, but I have pasted them below. I have kept the coredumps if
> there's anything more useful I can extract from them.
> 
> The full command line is something like this (taken from a later run):
> 
> /usr/local/bin/qemu-system-x86_64 -m 4G -machine type=pc,accel=kvm -netdev
> user,id=user.0,hostfwd=tcp::2305-:22 -device virtio-scsi-pci,id=scsi0
> -device scsi-hd,bus=scsi0.0,drive=drive0 -device virtio-net,netdev=user.0
> -name vtp-nmm-201611070837.qcow2 -drive 
> if=none,file=output-qemu-vtp-nmm/vtp-nmm-201611070837.qcow2,id=drive0,cache=writeback,discard=unmap,format=qcow2
> -boot c -vnc [::]:24
> 
> The following crashes occurred when running with a single vcpu. Normally I
> have been running with -smp 8,sockets=1,cores=4,threads=2 as it seems to
> crash less with those settings; however I'm trying it again like that in a
> loop to see if I can get a crash.

Let's try to isolate the cause of this crash:

Are you able to switch -netdev user to -netdev tap so we can rule out
the slirp user network stack as the source of memory corruption?

Alternatively could you re-run with virtio-blk instead of virtio-scsi to
see if that eliminates crashes?

The core dumps are likely to contain more clues.  If you are comfortable
with gdb and debugging C code you could dump the memory surround where
the junk value (mr) was loaded from.  Perhaps there is a hint about who
zeroed the memory.  In the first core dump you could start with:

 (gdb) up 6  # go to the dma_blk_unmap() stack frame
 (gdb) p *(DMAAIOCB *)0x560909ceca90
 (gdb) p *((DMAAIOCB *)0x560909ceca90).sg

> Regards,
> 
> Brian.
> 
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `/usr/local/bin/qemu-system-x86_64 -m 4G -name
> vtp-nmm-201611062024.qcow2 -machi'.
> Program terminated with signal SIGABRT, Aborted.
> #0  0x00007f366c4ce428 in __GI_raise (address@hidden) at
> ../sysdeps/unix/sysv/linux/raise.c:54
> 54      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> [Current thread is 1 (Thread 0x7f366deeea80 (LWP 9030))]
> (gdb) bt
> #0  0x00007f366c4ce428 in __GI_raise (address@hidden) at
> ../sysdeps/unix/sysv/linux/raise.c:54
> #1  0x00007f366c4d002a in __GI_abort () at abort.c:89
> #2  0x00007f366c4c6bd7 in __assert_fail_base (fmt=<optimised out>,
>     address@hidden "mr != NULL",
>     address@hidden "/home/nsrc/qemu-2.7.0/exec.c",
> address@hidden,
>     address@hidden <__PRETTY_FUNCTION__.42881>
> "address_space_unmap")
>     at assert.c:92
> #3  0x00007f366c4c6c82 in __GI___assert_fail (
>     address@hidden "mr != NULL",
>     address@hidden "/home/nsrc/qemu-2.7.0/exec.c",
> address@hidden,
>     address@hidden <__PRETTY_FUNCTION__.42881>
> "address_space_unmap")
>     at assert.c:101
> #4  0x000056090749dffe in address_space_unmap (as=<optimised out>,
> buffer=<optimised out>,
>     len=<optimised out>, is_write=1, access_len=8192) at
> /home/nsrc/qemu-2.7.0/exec.c:2967
> #5  0x00005609075af586 in dma_memory_unmap (access_len=<optimised out>,
> dir=<optimised out>,
>     len=<optimised out>, buffer=<optimised out>, as=<optimised out>)
>     at /home/nsrc/qemu-2.7.0/include/sysemu/dma.h:144
> #6  dma_blk_unmap (address@hidden) at
> /home/nsrc/qemu-2.7.0/dma-helpers.c:102
> #7  0x00005609075af766 in dma_complete (ret=0, dbs=0x560909ceca90)
>     at /home/nsrc/qemu-2.7.0/dma-helpers.c:113
> #8  dma_blk_cb (opaque=0x560909ceca90, ret=0) at
> /home/nsrc/qemu-2.7.0/dma-helpers.c:137
> #9  0x000056090775d25a in blk_aio_complete (acb=0x56090909aba0)
>     at /home/nsrc/qemu-2.7.0/block/block-backend.c:923
> #10 0x00005609077ccaea in coroutine_trampoline (i0=<optimised out>,
> i1=<optimised out>)
>     at /home/nsrc/qemu-2.7.0/util/coroutine-ucontext.c:78
> #11 0x00007f366c4e35d0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #12 0x00007fffd8d31f20 in ?? ()
> #13 0x2d2d2d2d2d2d2d2d in ?? ()
> #14 0x00000000000000d0 in ?? ()
> #15 0x0000000000000000 in ?? ()
> (gdb)
> 
> 
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `/usr/local/bin/qemu-system-x86_64 -boot c -vnc
> [::]:78 -name vtp-nmm-2016110621'.
> Program terminated with signal SIGABRT, Aborted.
> #0  0x00007fc226c98428 in __GI_raise (address@hidden) at
> ../sysdeps/unix/sysv/linux/raise.c:54
> 54      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> [Current thread is 1 (Thread 0x7fc2286b8a80 (LWP 10267))]
> (gdb) bt
> #0  0x00007fc226c98428 in __GI_raise (address@hidden) at
> ../sysdeps/unix/sysv/linux/raise.c:54
> #1  0x00007fc226c9a02a in __GI_abort () at abort.c:89
> #2  0x00007fc226c90bd7 in __assert_fail_base (fmt=<optimised out>,
>     address@hidden "mr != NULL",
>     address@hidden "/home/nsrc/qemu-2.7.0/exec.c",
> address@hidden,
>     address@hidden <__PRETTY_FUNCTION__.42881>
> "address_space_unmap")
>     at assert.c:92
> #3  0x00007fc226c90c82 in __GI___assert_fail (
>     address@hidden "mr != NULL",
>     address@hidden "/home/nsrc/qemu-2.7.0/exec.c",
> address@hidden,
>     address@hidden <__PRETTY_FUNCTION__.42881>
> "address_space_unmap")
>     at assert.c:101
> #4  0x000056186717fffe in address_space_unmap (as=<optimised out>,
> buffer=<optimised out>,
>     len=<optimised out>, is_write=1, access_len=4096) at
> /home/nsrc/qemu-2.7.0/exec.c:2967
> #5  0x0000561867202beb in virtqueue_unmap_sg
> (address@hidden, address@hidden,
>     vq=0x5618695a8500) at /home/nsrc/qemu-2.7.0/hw/virtio/virtio.c:254
> #6  0x0000561867203422 in virtqueue_fill (address@hidden,
>     address@hidden, len=32876, address@hidden)
>     at /home/nsrc/qemu-2.7.0/hw/virtio/virtio.c:282
> #7  0x00005618672035db in virtqueue_push (address@hidden,
>     address@hidden, len=<optimised out>)
>     at /home/nsrc/qemu-2.7.0/hw/virtio/virtio.c:308
> #8  0x00005618671f0885 in virtio_scsi_complete_req (req=0x5618694fc610)
>     at /home/nsrc/qemu-2.7.0/hw/scsi/virtio-scsi.c:70
> #9  0x00005618671f09e6 in virtio_scsi_complete_cmd_req (req=0x5618694fc610)
>     at /home/nsrc/qemu-2.7.0/hw/scsi/virtio-scsi.c:443
> #10 virtio_scsi_command_complete (r=<optimised out>, status=0, resid=0)
>     at /home/nsrc/qemu-2.7.0/hw/scsi/virtio-scsi.c:470
> #11 0x0000561867365c98 in scsi_req_complete (req=0x561868a72e40,
> status=<optimised out>)
>     at /home/nsrc/qemu-2.7.0/hw/scsi/scsi-bus.c:1775
> #12 0x0000561867360210 in scsi_dma_complete_noio (r=0x561868a72e40,
> ret=<optimised out>)
>     at /home/nsrc/qemu-2.7.0/hw/scsi/scsi-disk.c:278
> ---Type <return> to continue, or q <return> to quit---
> #13 0x0000561867291779 in dma_complete (ret=0, dbs=0x561868371050)
>     at /home/nsrc/qemu-2.7.0/dma-helpers.c:115
> #14 dma_blk_cb (opaque=0x561868371050, ret=0) at
> /home/nsrc/qemu-2.7.0/dma-helpers.c:137
> #15 0x000056186743f25a in blk_aio_complete (acb=0x561869661b90)
>     at /home/nsrc/qemu-2.7.0/block/block-backend.c:923
> #16 0x00005618674aeaea in coroutine_trampoline (i0=<optimised out>,
> i1=<optimised out>)
>     at /home/nsrc/qemu-2.7.0/util/coroutine-ucontext.c:78
> #17 0x00007fc226cad5d0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #18 0x00007ffd10673d70 in ?? ()
> #19 0x2d2d2d2d2d2d2d2d in ?? ()
> #20 0x00000000000000d4 in ?? ()
> #21 0x0000000000000000 in ?? ()
> (gdb)
> 
>

signature.asc
Description: PGP signature