Re: [Qemu-block] [PATCH v5 0/2] block: enforce minimal 4096 alignment in qemu_blockalign

[Stefan Hajnoczi]

On Mon, May 11, 2015 at 07:47:41PM +0300, Denis V. Lunev wrote:
> On 11/05/15 19:07, Denis V. Lunev wrote:
> >On 11/05/15 18:08, Stefan Hajnoczi wrote:
> >>On Mon, May 04, 2015 at 04:42:22PM +0300, Denis V. Lunev wrote:
> >>>The difference is quite reliable and the same 5%.
> >>>   qemu-io -n -c 'write -P 0xaa 0 1G' 1.img
> >>>for image in qcow2 format is 1% faster.
> >>I looked a little at the qemu-io invocation but am not clear why there
> >>would be a measurable performance difference.  Can you explain?
> >>
> >>What about real qemu-img or QEMU use cases?
> >>
> >>I'm okay with the patches themselves, but I don't really understand why
> >>this code change is justified.
> >>
> >>Stefan
> >There is a problem in the Linux kernel when the buffer
> >is not aligned to the page size. Actually the strict requirement
> >is the alignment to the 512 (one physical sector).
> >
> >This comes into the account in qemu-img and qemu-io
> >when buffers are allocated inside the application. QEMU
> >is free of this problem as the guest sends buffers
> >aligned to page already.
> >
> >You can see below results of qemu-img, they are exactly
> >the same as for qemu-io.
> >
> >qemu-img create -f qcow2 1.img 64G
> >qemu-io -n -c 'write -P 0xaa 0 1G' 1.img
> >time for i in `seq 1 30` ; do /home/den/src/qemu/qemu-img convert 1.img -t
> >none -O raw 2.img ; rm -rf 2.img ; done
> >
> >==== without patches ====:
> >real    2m6.287s
> >user    0m1.322s
> >sys    0m8.819s
> >
> >real    2m7.483s
> >user    0m1.614s
> >sys    0m9.096s
> >
> >==== with patches ====:
> >real    1m59.715s
> >user    0m1.453s
> >sys    0m9.365s
> >
> >real    1m58.739s
> >user    0m1.419s
> >sys    0m8.530s
> >
> >I could not exactly say where the difference comes, but
> >the problem comes from the fact that real IO operation
> >over the block device should be
> >  a) page aligned for the buffer
> >  b) page aligned for the offset
> >This is how buffer cache is working in the kernel. And
> >with non-aligned buffer in userspace the kernel should collect
> >kernel page for IO from 2 userspaces pages instead of one.
> >Something is not optimal here I presume. I can assume
> >that the user page could be sent immediately to the
> >controller is buffer is aligned and no additional memory
> >allocation is needed. Though I don't know exactly.
> >
> >Regards,
> >    Den
> 
> Here are results of blktrace on my host. Logs are collected using
>   sudo blktrace -d /dev/md0 -o - | blkparse -i -
> 
> Test command:
> /home/den/src/qemu/qemu-img convert 1.img -t none -O raw 2.img
> 
> In general, not patched qemu-img IO pattern looks like this:
>   9,0   11        1     0.000000000 11151  Q  WS 312737792 + 1023 [qemu-img]
>   9,0   11        2     0.000007938 11151  Q  WS 312738815 + 8 [qemu-img]
>   9,0   11        3     0.000030735 11151  Q  WS 312738823 + 1016 [qemu-img]
>   9,0   11        4     0.000032482 11151  Q  WS 312739839 + 8 [qemu-img]
>   9,0   11        5     0.000041379 11151  Q  WS 312739847 + 1016 [qemu-img]
>   9,0   11        6     0.000042818 11151  Q  WS 312740863 + 8 [qemu-img]
>   9,0   11        7     0.000051236 11151  Q  WS 312740871 + 1017 [qemu-img]
>   9,0    5        1     0.169071519 11151  Q  WS 312741888 + 1023 [qemu-img]
>   9,0    5        2     0.169075331 11151  Q  WS 312742911 + 8 [qemu-img]
>   9,0    5        3     0.169085244 11151  Q  WS 312742919 + 1016 [qemu-img]
>   9,0    5        4     0.169086786 11151  Q  WS 312743935 + 8 [qemu-img]
>   9,0    5        5     0.169095740 11151  Q  WS 312743943 + 1016 [qemu-img]
> 
> and patched one:
>   9,0    6        1     0.000000000 12422  Q  WS 314834944 + 1024 [qemu-img]
>   9,0    6        2     0.000038527 12422  Q  WS 314835968 + 1024 [qemu-img]
>   9,0    6        3     0.000072849 12422  Q  WS 314836992 + 1024 [qemu-img]
>   9,0    6        4     0.000106276 12422  Q  WS 314838016 + 1024 [qemu-img]
>   9,0    2        1     0.171038202 12422  Q  WS 314839040 + 1024 [qemu-img]
>   9,0    2        2     0.171073156 12422  Q  WS 314840064 + 1024 [qemu-img]
> 
> Thus the load to the disk is MUCH higher without the patch!
> 
> Total amount of lines (IO requests sent to disks) are the following:
> 
> hades ~ $ wc -l *.blk
>   3622 non-patched.blk
>   2086 patched.blk
>   5708 total
> hades ~ $
> 
> and this from my point of view explains everything! With aligned buffers the
> amount of IO requests is almost 2 times less.

The blktrace shows 512 KB I/Os.  I think qemu-img convert uses 2 MB
buffers by default.  What syscalls is qemu-img making?

I'm curious whether the kernel could be splitting up requests more
efficiently.  This would benefit all applications and not just qemu-img.

Stefan

pgp2KBiragNw6.pgp
Description: PGP signature