Re: [Qemu-devel] [regression] dataplane: throughout -40% by commit 580b6b2aa2

[Stefan Hajnoczi]

On Tue, Jul 01, 2014 at 06:49:30PM +0200, Paolo Bonzini wrote:
> Il 01/07/2014 16:49, Ming Lei ha scritto:
> >Let me provide some data when running randread(bs 4k, libaio)
> >from VM for 10sec:
> >
> >1), qemu.git/master
> >- write(): 731K
> >- rt_sigprocmask(): 417K
> >- read(): 21K
> >- ppoll(): 10K
> >- io_submit(): 5K
> >- io_getevents(): 4K
> >
> >2), qemu 2.0
> >- write(): 9K
> >- read(): 28K
> >- ppoll(): 16K
> >- io_submit(): 12K
> >- io_getevents(): 10K
> >
> >>> The sigprocmask can probably be optimized away since the thread's
> >>> signal mask remains unchanged most of the time.
> >>>
> >>> I'm not sure what is causing the write().
> >I am investigating it...
> 
> I would guess sigprocmask is getcontext (from qemu_coroutine_new) and write
> is aio_notify (from qemu_bh_schedule).

Aha!  We shouldn't be executing qemu_coroutine_new() very often since we
try to keep a freelist of coroutines.

I think a tweak to the freelist could make the rt_sigprocmask() calls go
away since we should be reusing coroutines instead of allocating/freeing
them all the time.

> Both can be eliminated by introducing a fast path in bdrv_aio_{read,write}v,
> that bypasses coroutines in the common case of no I/O throttling, no
> copy-on-write, etc.

I tried that in 2012 and couldn't measure an improvement above the noise
threshold, although it was without dataplane.

BTW, we cannot eliminate the BH because the block layer guarantees that
callbacks are not invoked with reentrancy.  They are always invoked
directly from the event loop through a BH.  This simplifies callers
since they don't need to worry about callbacks happening while they are
still in bdrv_aio_readv(), for example.

Removing this guarantee (by making callers safe first) is orthogonal to
coroutines.  But it's hard to do since it requires auditing a lot of
code.

Another idea is to skip aio_notify() when we're sure the event loop
isn't blocked in g_poll().  Doing this is a thread-safe and lockless way
might be tricky though.

So to recap, three issues are being discussed here:

1. rt_sigprocmask due to getcontext() in qemu_coroutine_new().  We
shouldn't be invoking qemu_coroutine_new() often.  The freelist is
probably too small.

2. Coroutines might be slower than the non-coroutine aio codepath.  I
don't think this is the case, they are very cheap and I was never able
to measure a real difference.

3. The block layer requires a BH with aio_notify() for
bdrv_aio_readv()/bdrv_aio_writev()/bdrv_aio_flush() callbacks regardless
of coroutines or not.  Eliminating the BH or skipping aio_notify() will
take some work but could speed up QEMU as a whole.

Stefan

pgpf5RosZHPut.pgp
Description: PGP signature