Re: [Qemu-devel] vhost-scsi port to v1.1.0 + MSI-X performance regression

[Nicholas A. Bellinger]

On Tue, 2012-07-24 at 13:20 -0700, Nicholas A. Bellinger wrote:
> On Tue, 2012-07-24 at 09:57 +0200, Jan Kiszka wrote:
> > On 2012-07-24 09:42, Nicholas A. Bellinger wrote:
> > > Hi Anthony, Stefan & QEMU folks,
> > > 
> 
> <SNIP>
> 
> > > However, thus far I've not been able to get virtio-scsi <-> tcm_vhost
> > > I/O to actually work against the latest qemu.git/master..  
> > > 
> > > So while doing a (manual) bisection w/ this series to track down the
> > > issue with qemu/master, I managed to run across something else..  With
> > > the vhost-scsi series applied, everything is working as expected up
> > > until the following commit:
> > > 
> > > commit 1523ed9e1d46b0b54540049d491475ccac7e6421
> > > Author: Jan Kiszka <address@hidden>
> > > Date:   Thu May 17 10:32:39 2012 -0300
> > > 
> > >     virtio/vhost: Add support for KVM in-kernel MSI injection
> > > 
> > > 
> > > This commit ends up triggering the following assert immediately after
> > > starting qemu with virtio-scsi <-> tcm_vhost:
> > > 
> > > qemu-system-x86_64: /usr/src/qemu.git/hw/msix.c:515:
> > >        msix_unset_vector_notifiers: Assertion 
> > > `dev->msix_vector_use_notifier &&
> > >                                     dev->msix_vector_release_notifier' 
> > > failed.
> > > 
> > > OK, so adding the following hack allows me to boot:

<SNIP>

> > Also, is there a git tree and a way to reproduce this without special
> > hardware needs?
> > 
> 
> I'll push this series + branches to demonstrate the issue into an public
> tree this afternoon.
> 

OK, the updated vhost-scsi QEMU working tree is now available here:

http://git.kernel.org/?p=virt/kvm/nab/qemu-kvm.git;a=summary

Please ignore the qemu-kvm.git name for the moment, as it really is
based on upstream qemu.  ;)

The branch breakdown looks like:

  master                401a663 remove unused QemuOpts parameter from net init 
functions
* vhost-scsi            bdd00bd msix: Add msix_nr_vectors_allocated
  vhost-scsi-merge      e4fdb8c [ahead 9] vhost-scsi: add -vhost-scsi host 
device
  vhost-scsi-workaround 32ed0db msix: Work-around for vhost-scsi with KVM 
in-kernel MSI injection

- 'vhost-scsi' contains vhost-scsi patches + pre commit 1523ed9e1d logic
   (Running stable at ~67.5K IOPs, see below)
- 'vhost-scsi-merge' contains a vhost-scsi series against yesterday's qemu.git 
HEAD
   (not working atm, but will be sending out for RFC shortly)
- 'vhost-scsi-workaround' contains the msix.c workaround to get commit 
1523ed9e1d to boot
   (reproducible performance regression, down to 6K IOPs)

> Also, the particular backend is a Fusion-IO raw block flash device, but
> I'm pretty sure that using a TCM RAMDISK into tcm_vhost would exhibit
> the same type of behavior.  (Will double check on that shortly..)
> 

Interestingly enough, after running some test with the vhost-scsi branch
above with the last good working commit (bdd00bd), the same fio
benchmark inline below is now reaching ~67.5K IOPs using v3.5-rc2 guest
+ host..

rw=randrw
rwmixwrite=25
rwmixread=75
size=32768m
ioengine=libaio
direct=1
iodepth=64
blocksize=4k
numjobs=2
filename=/dev/sdb


This ends up being within a few percentage of what the bare-metal HW is
capable of.. (~70K IOPs per LUN for an ioDrive Duo, Jen's CC'ed).  Note
the older QEMU vhost-scsi tree based on pre 1.1.0 code was doing ~60K
IOPs for the same test..

So this looks promising pre commit 1523ed9e1d vs. what we've been using
with vhost-scsi so far..  ;)

--nab