Re: [Qemu-devel] [Problem] qemu crash when vhost_net_start

[Xulei (Stone)]

> 
> Hi
> 
> ----- Original Message -----
> > Hi, all
> > Recently, I have a VM with a vhost-user network card created by qemu 2.6.0.
> > Once, I restart OpenVSwitch service
> > and start this VM in the same time. I found qemu may probably crash
> > with following stack:
> >
> > (gdb) bt
> > #0  0x00007f0f9179a5d7 in raise () from /usr/lib64/libc.so.6
> > #1  0x00007f0f9179bcc8 in abort () from /usr/lib64/libc.so.6
> > #2  0x000000000045a202 in kvm_io_ioeventfd_add ()
> > #3  0x000000000045cffc in address_space_add_del_ioeventfds ()
> > #4  0x000000000045fa0e in address_space_update_ioeventfds ()
> > #5  0x0000000000460f40 in memory_region_transaction_commit ()
> > #6  0x0000000000461ce5 in memory_region_add_eventfd ()
> > #7  0x000000000066a1e5 in virtio_pci_set_host_notifier_internal ()
> > #8  0x00000000004ae08a in vhost_dev_enable_notifiers ()
> > #9  0x0000000000492743 in vhost_net_start_one ()
> > #10 0x00000000004930bf in vhost_net_start ()
> > #11 0x000000000048efd4 in virtio_net_vhost_status ()
> > #12 0x000000000048f16a in virtio_net_set_status ()
> > #13 0x0000000000686bcd in qmp_set_link ()
> > #14 0x000000000068dcc3 in net_vhost_user_event ()
> > #15 0x000000000051f043 in tcp_chr_new_client ()
> > #16 0x000000000051f10f in qemu_chr_socket_connected ()
> > #17 0x000000000073cb10 in qio_task_complete ()
> > #18 0x000000000073cb7b in gio_task_thread_result ()
> > #19 0x00007f0f929fb99a in g_main_context_dispatch () from
> > /usr/lib64/libglib-2.0.so.0
> > #20 0x00000000006d2275 in os_host_main_loop_wait ()
> > #21 0x00000000006d2393 in main_loop_wait ()
> > #22 0x000000000052a0f2 in main_loop ()
> > #23 0x000000000041bcd3 in main ()
> >
> > This seems a bug triggering when backend starts vhost_net and
> > meanwhile the frontend rmmod/modprobe virtio-net.
> > Is this a known issue or any patch can fix this?
> >
> 
> 
> Thanks for the report.
> 
> Could you provide step-by-step instructions on how to reproduce?
> 
> (if you could bisect qemu.git that would be also helpful !)
> 
> thanks

Thanks for reply.

Your patch "vhost-user: check vhost_user_{read,write}() return value" or
Gonglei's "vhost-user: fix unreasonable return value when vhost-user read 
failed"
seems inspired me. 
Qemu 2.6 has not merged your patch, so vhost_user_init() will get a 
random feature value when vhost_user_{write,read}() failed. I think the
crash has certain relations with this. Because I have tried following 
modification
which can let this problem be inevitable (start a vm, and restart openvswitch):

diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
index 1580929..3628382 100644
--- a/hw/virtio/vhost-user.c
+++ b/hw/virtio/vhost-user.c
@@ -469,6 +469,12 @@ static int vhost_user_get_u64(struct vhost_dev *dev, int 
request, uint64_t *u64)
         return 0;
     }

+       if (request == VHOST_USER_GET_FEATURES) {
+            vhost_user_features_init(u64);
+            return 0;
+    }
+
     if (vhost_user_write(dev, &msg, NULL, 0) < 0) {

So, I guess the crash problem has relations with vhost user feature. Then
I tried a lot to find which features and finally I found following patch can 
fix crash problem:

diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
index 1580929..e861e8a 100644
--- a/hw/virtio/vhost-user.c
+++ b/hw/virtio/vhost-user.c
@@ -455,7 +455,8 @@ static int vhost_user_set_protocol_features(struct 
vhost_dev *dev,
 static void vhost_user_features_init(void *arg)
 {
     #define VIRTIO_NET_F_MRG_RXBUF  15      /* Host can merge receive buffers. 
*/
-    *((__u64 *) arg) = ((1ULL << VIRTIO_NET_F_MRG_RXBUF) | (1ULL << 
VHOST_F_LOG_ALL));
+    *((__u64 *) arg) = ((1ULL << VIRTIO_NET_F_MRG_RXBUF) | (1ULL << 
VHOST_F_LOG_ALL)
+                                       |(1ULL << 
VHOST_USER_F_PROTOCOL_FEATURES));
 }

 static int vhost_user_get_u64(struct vhost_dev *dev, int request, uint64_t 
*u64)
@@ -469,6 +470,7 @@ static int vhost_user_get_u64(struct vhost_dev *dev, int 
request, uint64_t *u64)
     if (vhost_user_write(dev, &msg, NULL, 0) < 0) {
+        if (request == VHOST_USER_GET_FEATURES) {
+            vhost_user_features_init(u64);
+        }
        return 0;
    }

Merely, I could not figure out why VHOST_USER_F_PROTOCOL_FEATURES feature could 
led to
crash. Hoping above information can help you to tell me the reason.