Re: [Qemu-devel] Performance about x-data-plane

[Weiwei Jia]

On Mon, Jan 16, 2017 at 10:00 AM, Karl Rister <address@hidden> wrote:
> On 01/16/2017 07:15 AM, Stefan Hajnoczi wrote:
>> On Tue, Jan 03, 2017 at 12:02:14PM -0500, Weiwei Jia wrote:
>>>> The expensive part is the virtqueue kick.  Recently we tried polling the
>>>> virtqueue instead of waiting for the ioeventfd file descriptor and got
>>>> double-digit performance improvements:
>>>> https://lists.gnu.org/archive/html/qemu-devel/2016-12/msg00148.html
>>>>
>>>> If you want to understand the performance of your benchmark you'll have
>>>> compare host/guest disk stats (e.g. request lifetime, disk utilization,
>>>> queue depth, average request size) to check that the bare metal and
>>>> guest workloads are really sending comparable I/O patterns to the
>>>> physical disk.
>>>>
>>>> Then you using Linux and/or QEMU tracing to analyze the request latency
>>>> by looking at interesting points in the request lifecycle like virtqueue
>>>> kick, host Linux AIO io_submit(2), etc.
>>>>
>>>
>>> Thank you. I will look into "polling the virtqueue" as you said above.
>>> Currently, I just use blktrace to see disk stats and add logs in the
>>> I/O workload to see the time latency for each request. What kind of
>>> tools are you using to analyze request lifecycle like virtqueue kick,
>>> host Linux AIO iosubmit, etc.
>>>
>>> Do you trace the lifecycle like this
>>> (http://www.linux-kvm.org/page/Virtio/Block/Latency#Performance_data)
>>> but it seems to be out of date. Does it
>>> (http://repo.or.cz/qemu-kvm/stefanha.git/shortlog/refs/heads/tracing-dev-0.12.4)
>>> still work on QEMU 2.4.1?
>>
>> The details are out of date but the general approach to tracing the I/O
>> request lifecycle still apply.
>>
>> There are multiple tracing tools that can do what you need.  I've CCed
>> Karl Rister who did the latest virtio-blk dataplane tracing.
>
> I roughly followed this guide by Luiz Capitulino:
>
> https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html
>
> I tweaked his trace-host-and-guest script to avoid doing IO while
> tracing is enabled, my version is available here:
>
> http://people.redhat.com/~krister/tracing/trace-host-and-guest
>
> I built QEMU with --enable-trace-backends=ftrace and then turned on the
> QEMU trace events I was interested in with this bit of bash:
>
> for event in $(/usr/libexec/qemu-kvm -trace help 2>&1|grep virtio|grep
> -v "gpu\|console\|serial\|rng\|balloon\|ccw"); do virsh
> qemu-monitor-command master --hmp trace-event ${event} on; done
>
> At this point, the QEMU trace events are automatically inserted into the
> ftrace buffers and the methodology outlined by Luiz gets the guest
> kernel, host kernel, and QEMU events properly interleaved.


Thank you. I will try it.


Best,
Weiwei Jia