Re: [Qemu-devel] [PATCH 0/4] Tweaks around virtio-blk start/stop

[Christian Borntraeger]

On 03/17/2016 04:02 PM, Paolo Bonzini wrote:
> 
> 
> On 17/03/2016 13:39, Christian Borntraeger wrote:
>> As an interesting side note, I updated my system from F20 to F23 some days 
>> ago
>> (after the initial report). While To Bo is still on a F20 system. I was not 
>> able
>> to reproduce the original crash on f23. but going back to F20 made this
>> problem re-appear.
>>  
>>   Stack trace of thread 26429:
>>                 #0  0x00000000802008aa tracked_request_begin 
>> (qemu-system-s390x)
>>                 #1  0x0000000080203f3c bdrv_co_do_preadv (qemu-system-s390x)
>>                 #2  0x000000008020567c bdrv_co_do_readv (qemu-system-s390x)
>>                 #3  0x000000008025d0f4 coroutine_trampoline 
>> (qemu-system-s390x)
>>                 #4  0x000003ff943d150a __makecontext_ret (libc.so.6)
>>
>> this is with patch 2-4 plus the removal of virtio_queue_host_notifier_read.
>>
>> Without removing virtio_queue_host_notifier_read, I get the same mutex 
>> lockup (as expected).
>>
>> Maybe we have two independent issues here and this is some old bug in glibc 
>> or
>> whatever?
> 
> I'm happy to try and reproduce on x86 if you give me some instruction
> (RHEL7 should be close enough to Fedora 20).

Tu Bo has a standard guest that he starting multiple times. I can trigger some
issues by starting 20 guests that only have a kernel (with virtio-blk and bus
driver compiled in) and a busybox ramdisk that simply calls reboot. Sooner or
later a qemu crashes.
This guest has several virtio devices attached as well (partition detection will
do minimal reads)

ala 

  <qemu:commandline>
    <qemu:arg value='-drive'/>
    <qemu:arg value='driver=null-co,id=null1,if=none,size=100G'/>
    <qemu:arg value='-device'/>
    <qemu:arg 
value='virtio-blk-ccw,scsi=off,drive=null1,serial=null1,iothread=iothread1'/>
    <qemu:arg value='-drive'/>
    <qemu:arg value='driver=null-co,id=null2,if=none,size=100G'/>
    <qemu:arg value='-device'/>
    <qemu:arg 
value='virtio-blk-ccw,scsi=off,drive=null2,serial=null2,iothread=iothread1'/>
    <qemu:arg value='-drive'/>
    <qemu:arg value='driver=null-co,id=null3,if=none,size=100G'/>
    <qemu:arg value='-device'/>
    <qemu:arg 
value='virtio-blk-ccw,scsi=off,drive=null3,serial=null3,iothread=iothread1'/>
    <qemu:arg value='-drive'/>
    <qemu:arg value='driver=null-co,id=null4,if=none,size=100G'/>
    <qemu:arg value='-device'/>
    <qemu:arg 
value='virtio-blk-ccw,scsi=off,drive=null4,serial=null4,iothread=iothread1'/>

    <qemu:arg value='-drive'/>
    <qemu:arg value='driver=null-co,id=null5,if=none,size=100G'/>
    <qemu:arg value='-device'/>
    <qemu:arg 
value='virtio-blk-ccw,scsi=off,drive=null5,serial=null5,iothread=iothread1'/>
    <qemu:arg value='-drive'/>
    <qemu:arg value='driver=null-co,id=null6,if=none,size=100G'/>
    <qemu:arg value='-device'/>
    <qemu:arg 
value='virtio-blk-ccw,scsi=off,drive=null6,serial=null6,iothread=iothread1'/>
    <qemu:arg value='-drive'/>
    <qemu:arg value='driver=null-co,id=null7,if=none,size=100G'/>
    <qemu:arg value='-device'/>
    <qemu:arg 
value='virtio-blk-ccw,scsi=off,drive=null7,serial=null7,iothread=iothread1'/>
    <qemu:arg value='-drive'/>
    <qemu:arg value='driver=null-co,id=null8,if=none,size=100G'/>
    <qemu:arg value='-device'/>
    <qemu:arg 
value='virtio-blk-ccw,scsi=off,drive=null8,serial=null8,iothread=iothread1'/>


  </qemu:commandline>





> 
> Can you add an assert in virtio_blk_handle_output to catch reentrancy, like

Will do.


> 
> diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
> index a7ec572..96ea896 100644
> --- a/hw/block/virtio-blk.c
> +++ b/hw/block/virtio-blk.c
> @@ -591,6 +591,8 @@ static void virtio_blk_handle_output(VirtIODevice
> *vdev, VirtQueue *vq)
>          return;
>      }
> 
> +    int x = atomic_fetch_inc(&s->test);
> +    assert(x == 0);
>      blk_io_plug(s->blk);
> 
>      while ((req = virtio_blk_get_request(s))) {
> @@ -602,6 +604,7 @@ static void virtio_blk_handle_output(VirtIODevice
> *vdev, VirtQueue *vq)
>      }
> 
>      blk_io_unplug(s->blk);
> +    atomic_dec(&s->test);
>  }
> 
>  static void virtio_blk_dma_restart_bh(void *opaque)
> diff --git a/include/hw/virtio/virtio-blk.h b/include/hw/virtio/virtio-blk.h
> index ae84d92..6472503 100644
> --- a/include/hw/virtio/virtio-blk.h
> +++ b/include/hw/virtio/virtio-blk.h
> @@ -48,6 +48,7 @@ typedef struct VirtIOBlock {
>      BlockBackend *blk;
>      VirtQueue *vq;
>      void *rq;
> +    int test;
>      QEMUBH *bh;
>      VirtIOBlkConf conf;
>      unsigned short sector_mask;
> 
> ?
> 
> Paolo
>