Re: [Qemu-devel] Docs for and debugging of Asynchronous I/O

[Ot ten Thije]

On Tue, Jul 20, 2010 at 9:57 PM, Stefan Hajnoczi <address@hidden> wrote:
>
> On Tue, Jul 20, 2010 at 7:34 PM, Ot ten Thije <address@hidden> wrote:
> > Hello,
> > I am working on fixing the savevm/loadvm functionality in the Android
> > emulator, and the two issues I've encountered so far both appear to stem
> > from the asynchronous I/O (AIO) code. In both cases, the emulator busy-waits
> > indefinitely for an operation that never signals completion.
> > Unfortunately I am not really familiar with AIO, so I was hoping one of the
> > emulator devs could point me some resources (design docs, general
> > introduction, etc.). I've done some searching myself and found some docs for
> > the Linux kernel AIO implementation
> > (http://lse.sourceforge.net/io/aio.html), but I'm not sure to what extent it
> > applies to the QEMU code.
>
> qemu.git has support for Linux AIO, which you linked to.  The code for
> that is in linux-aio.c.  But by default it uses a thread pool to
> perform the normal blocking syscalls out-of-line with respect to the
> VM, see posix-aio-compat.c.
>
> Both of these aio backends are driven from the QEMU block layer, which
> provides a block device interface that several drivers implement
> (qcow2, raw, ...).  bdrv_aio_readv() and bdrv_aio_writev() are the
> main entry points.
>
> I wonder if you can track down the sequence of block interface calls
> that causes the hang.  Then it should be possible to construct a
> simple test case and even try it again qemu.git/master.
>

I've been trying to track this down, and it seems that the program
blocks/busy-waits the first time data is actually being flushed to
file. Unfortunately I haven't been able to identify the exact point at
which the background thread is started that takes care of the actual
I/O.

I would like to put a breakpoint on that code to see what exactly is
going on there. Could you tell me which function does this in mainline
QEMU? I've seen some suggestive names (qcow_aio_setup and
qemu_iovec_init_external), but their implementations do not hint in
any way at initiating a background process/thread. Or am I
misunderstanding the nature of AIO calls entirely?


> > Tips for debugging AIO would also be greatly appreciated. I can trace the
> > execution until I am within the (emulated) device driver (i.e.
> > block/qcow2.c:qcow_aio_writev()), but haven't been able to pinpoint the
> > exact location where the actual async call is made. This makes it difficult
> > to identify the code that should signal completion back to the main process
> > (and apparently fails to do so). I know this code is called though, because
> > some asynchronous calls *do* signal completion.
> > I realize that the Android emulator is a rather heavy fork of QEMU, so
> > giving specific advice will probably be difficult. However, the overall
> > approach is still the same, so I hope you can help me get a better
> > understanding of that.
>
> Have you checked QEMU's git log to see if there are fixes in qemu.git
> that could fix this for the Android emulator?

I've gone through the logs as you suggested (grepping for "aio" and
"qcow"). Unfortunately the number of changes in qcow/aio related code
since the last merge from mainline to Android is quite large, making
it very difficult to pinpoint any specific change as a breaking point.
A great deal of the API has changed as well, making it difficult to
replace the current implementations with those from mainline.

Ot

>
> Stefan
>
> >
> > Ot ten Thije