[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [BUG] QEMU crashes with dpdk virtio pmd
From: |
Gonglei (Arei) |
Subject: |
Re: [Qemu-devel] [BUG] QEMU crashes with dpdk virtio pmd |
Date: |
Thu, 27 Apr 2017 07:01:11 +0000 |
CCing Paolo and Stefan, since it has a relationship with bh in Qemu.
> -----Original Message-----
> From: Jason Wang [mailto:address@hidden
>
>
> On 2017年04月25日 19:37, wangyunjian wrote:
> > The q->tx_bh will free in virtio_net_del_queue() function, when remove
> > virtio
> queues
> > if the guest doesn't support multiqueue. But it might be still referenced by
> others (eg . virtio_net_set_status()),
> > which need so set NULL.
> >
> > diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> > index 7d091c9..98bd683 100644
> > --- a/hw/net/virtio-net.c
> > +++ b/hw/net/virtio-net.c
> > @@ -1522,9 +1522,12 @@ static void virtio_net_del_queue(VirtIONet *n,
> int index)
> > if (q->tx_timer) {
> > timer_del(q->tx_timer);
> > timer_free(q->tx_timer);
> > + q->tx_timer = NULL;
> > } else {
> > qemu_bh_delete(q->tx_bh);
> > + q->tx_bh = NULL;
> > }
> > + q->tx_waiting = 0;
> > virtio_del_queue(vdev, index * 2 + 1);
> > }
>
> Thanks a lot for the fix.
>
> Two questions:
>
> - If virtio_net_set_status() is the only function that may access tx_bh,
> it looks like setting tx_waiting to zero is sufficient?
Currently yes, but we don't assure that it works for all scenarios, so
we set the tx_bh and tx_timer to NULL to avoid to possibly access wild pointer,
which is the common method for usage of bh in Qemu.
I have another question about the root cause of this issure.
This below trace is the path of setting tx_waiting to one in
virtio_net_handle_tx_bh() :
Breakpoint 1, virtio_net_handle_tx_bh (vdev=0x0, vq=0x7f335ad13900) at
/data/wyj/git/qemu/hw/net/virtio-net.c:1398
1398 {
(gdb) bt
#0 virtio_net_handle_tx_bh (vdev=0x0, vq=0x7f335ad13900) at
/data/wyj/git/qemu/hw/net/virtio-net.c:1398
#1 0x00007f3357bddf9c in virtio_bus_set_host_notifier (bus=<optimized out>,
address@hidden, address@hidden) at hw/virtio/virtio-bus.c:297
#2 0x00007f3357a0055d in vhost_dev_disable_notifiers (address@hidden,
address@hidden) at /data/wyj/git/qemu/hw/virtio/vhost.c:1422
#3 0x00007f33579e3373 in vhost_net_stop_one (net=0x7f335ad84dc0,
dev=0x7f335c6f5f90) at /data/wyj/git/qemu/hw/net/vhost_net.c:289
#4 0x00007f33579e385b in vhost_net_stop (address@hidden, ncs=<optimized out>,
address@hidden) at /data/wyj/git/qemu/hw/net/vhost_net.c:367
#5 0x00007f33579e15de in virtio_net_vhost_status (status=<optimized out>,
n=0x7f335c6f5f90) at /data/wyj/git/qemu/hw/net/virtio-net.c:176
#6 virtio_net_set_status (vdev=0x7f335c6f5f90, status=0 '\000') at
/data/wyj/git/qemu/hw/net/virtio-net.c:250
#7 0x00007f33579f8dc6 in virtio_set_status (address@hidden, address@hidden
'\000') at /data/wyj/git/qemu/hw/virtio/virtio.c:1146
#8 0x00007f3357bdd3cc in virtio_ioport_write (val=0, addr=18,
opaque=0x7f335c6eda80) at hw/virtio/virtio-pci.c:387
#9 virtio_pci_config_write (opaque=0x7f335c6eda80, addr=18, val=0,
size=<optimized out>) at hw/virtio/virtio-pci.c:511
#10 0x00007f33579b2155 in memory_region_write_accessor (mr=0x7f335c6ee470,
addr=18, value=<optimized out>, size=1, shift=<optimized out>, mask=<optimized
out>, attrs=...) at /data/wyj/git/qemu/memory.c:526
#11 0x00007f33579af2e9 in access_with_adjusted_size (address@hidden,
address@hidden, address@hidden, access_size_min=<optimized out>,
access_size_max=<optimized out>, address@hidden
0x7f33579b20f0 <memory_region_write_accessor>, address@hidden,
address@hidden) at /data/wyj/git/qemu/memory.c:592
#12 0x00007f33579b2e15 in memory_region_dispatch_write (address@hidden,
address@hidden, data=0, address@hidden, address@hidden) at
/data/wyj/git/qemu/memory.c:1319
#13 0x00007f335796cd93 in address_space_write_continue (mr=0x7f335c6ee470, l=1,
addr1=18, len=1, buf=0x7f335773d000 "", attrs=..., addr=49170,
as=0x7f3358317060 <address_space_io>) at /data/wyj/git/qemu/exec.c:2834
#14 address_space_write (as=<optimized out>, addr=<optimized out>, attrs=...,
buf=<optimized out>, len=<optimized out>) at /data/wyj/git/qemu/exec.c:2879
#15 0x00007f335796d3ad in address_space_rw (as=<optimized out>, address@hidden,
attrs=..., address@hidden, buf=<optimized out>, address@hidden, address@hidden)
at /data/wyj/git/qemu/exec.c:2981
#16 0x00007f33579ae226 in kvm_handle_io (count=1, size=1, direction=<optimized
out>, data=<optimized out>, attrs=..., port=49170) at
/data/wyj/git/qemu/kvm-all.c:1803
#17 kvm_cpu_exec (address@hidden) at /data/wyj/git/qemu/kvm-all.c:2032
#18 0x00007f335799b632 in qemu_kvm_cpu_thread_fn (arg=0x7f335ae82070) at
/data/wyj/git/qemu/cpus.c:1118
#19 0x00007f3352983dc5 in start_thread () from /usr/lib64/libpthread.so.0
#20 0x00007f335113571d in clone () from /usr/lib64/libc.so.6
It calls qemu_bh_schedule(q->tx_bh) at the bottom of virtio_net_handle_tx_bh(),
I don't know why virtio_net_tx_bh() doesn't be invoked, so that the
q->tx_waiting is not zero.
[ps: we added logs in virtio_net_tx_bh() to verify that]
Some other information:
It won't crash if we don't use vhost-net.
Thanks,
-Gonglei
> - Can you post a formal patch for this?
>
> Thanks
>
> > From: wangyunjian
> > Sent: Monday, April 24, 2017 6:10 PM
> > To: address@hidden; Michael S. Tsirkin <address@hidden>; 'Jason
> Wang' <address@hidden>
> > Cc: wangyunjian <address@hidden>; caihe <address@hidden>
> > Subject: [Qemu-devel][BUG] QEMU crashes with dpdk virtio pmd
> >
> > Qemu crashes, with pre-condition:
> > vm xml config with multiqueue, and the vm's driver virtio-net support
> multi-queue
> >
> > reproduce steps:
> > i. start dpdk testpmd in VM with the virtio nic
> > ii. stop testpmd
> > iii. reboot the VM
> >
> > This commit "f9d6dbf0 remove virtio queues if the guest doesn't support
> multiqueue" is introduced.
> >
> > Qemu version: QEMU emulator version 2.9.50 (v2.9.0-137-g32c7e0a)
> > VM DPDK version: DPDK-1.6.1
> >
> > Call Trace:
> > #0 0x00007f60881fe5d7 in raise () from /usr/lib64/libc.so.6
> > #1 0x00007f60881ffcc8 in abort () from /usr/lib64/libc.so.6
> > #2 0x00007f608823e2f7 in __libc_message () from /usr/lib64/libc.so.6
> > #3 0x00007f60882456d3 in _int_free () from /usr/lib64/libc.so.6
> > #4 0x00007f608900158f in g_free () from /usr/lib64/libglib-2.0.so.0
> > #5 0x00007f6088fea32c in iter_remove_or_steal () from
> /usr/lib64/libglib-2.0.so.0
> > #6 0x00007f608edc0986 in object_property_del_all (obj=0x7f6091e74800)
> at qom/object.c:410
> > #7 object_finalize (data=0x7f6091e74800) at qom/object.c:467
> > #8 object_unref (address@hidden) at qom/object.c:903
> > #9 0x00007f608eaf1fd3 in phys_section_destroy (mr=0x7f6091e74800) at
> git/qemu/exec.c:1154
> > #10 phys_sections_free (map=0x7f6090b72bb0) at git/qemu/exec.c:1163
> > #11 address_space_dispatch_free (d=0x7f6090b72b90) at
> git/qemu/exec.c:2514
> > #12 0x00007f608ee91ace in call_rcu_thread (opaque=<optimized out>) at
> util/rcu.c:272
> > #13 0x00007f6089b0ddc5 in start_thread () from /usr/lib64/libpthread.so.0
> > #14 0x00007f60882bf71d in clone () from /usr/lib64/libc.so.6
> >
> > Call Trace:
> > #0 0x00007fdccaeb9790 in ?? ()
> > #1 0x00007fdcd82d09fc in object_property_del_all (obj=0x7fdcdb8acf60) at
> qom/object.c:405
> > #2 object_finalize (data=0x7fdcdb8acf60) at qom/object.c:467
> > #3 object_unref (address@hidden) at qom/object.c:903
> > #4 0x00007fdcd8001fd3 in phys_section_destroy (mr=0x7fdcdb8acf60) at
> git/qemu/exec.c:1154
> > #5 phys_sections_free (map=0x7fdcdc86aa00) at git/qemu/exec.c:1163
> > #6 address_space_dispatch_free (d=0x7fdcdc86a9e0) at
> git/qemu/exec.c:2514
> > #7 0x00007fdcd83a1ace in call_rcu_thread (opaque=<optimized out>) at
> util/rcu.c:272
> > #8 0x00007fdcd301ddc5 in start_thread () from /usr/lib64/libpthread.so.0
> > #9 0x00007fdcd17cf71d in clone () from /usr/lib64/libc.so.6
> >
> >