Re: [Qemu-devel] ?????? ?????? ?????? ?????? q emu vm big network latency when met heavy io

[Josh Durgin]

On 01/13/2014 10:39 PM, ?????? wrote:
> Hi Josh
>
> Thanks for your reply.
>
> librbd version : 0.67.5-1
>
> /etc/ceph/ceph.conf config file:
> the ip and hostname was hiddened.
> [global]
>          ;open auth.
>          auth cluster required = cephx
>          auth service required = cephx
>          auth client required = cephx
>          ;global pid & log setting.
>          admin socket = /home/ceph/var/run/$cluster-$name.asok
>
> [mon]
>          keyring = /home/ceph/var/lib/$type/$cluster-$id/keyring
>          mon data = /home/ceph/var/lib/$type/$cluster-$id
>          mon cluster log file = /home/ceph/log/$cluster.log
> [mon.a]
>          host = cld-xx
>          mon addr = x.x.x.x:6789
>          user = ceph
> [mon.b]
>          host = cld-xx
>          mon addr = x.x.x.x:6789
>          user = ceph
> [mon.c]
>          host = cld-xx
>          mon addr = x.x.x.x:6789
>          user = ceph
> [mon.d]
>          host = cld-xx
>          mon addr = x.x.x.x:6789
>          user = ceph
> [mon.e]
>          host = cld-xx
>          mon addr = x.x.x.x:6789
>          user = ceph
>
> [osd]
>          keyring = /home/ceph/var/lib/$type/$cluster-$id/keyring
>          osd data = /home/ceph/var/lib/$type/$cluster-$id
>          osd journal = /home/ceph/var/lib/$type/$cluster-$id/journal
>          osd journal size = 1000
>          osd mkfs type = xfs
>          osd mount options xfs = rw,noatime,inode64
> [osd.0]
>          host = cld-xx
>          addr = x.x.x.x
>          user = ceph
>          devs = /dev/sdb1
> [osd.1]
>          host = cld-xx
>          addr = x.x.x.x
>          user = ceph
>          devs = /dev/sda1
> [osd.2]
>          host = cld-xx
>          addr = x.x.x.x
>          user = ceph
>          devs = /dev/sdb1
> [osd.3]
>          host = cld-xx
>          addr = x.x.x.x
>          user = ceph
>          devs = /dev/sda1
> [osd.4]
>          host = cld-xx
>          addr = x.x.x.x
>          user = ceph
>          devs = /dev/sdb1
> [osd.5]
>          host = cld-xx
>          addr = x.x.x.x
>          user = ceph
>          devs = /dev/sda1
>
> [client.libvirt]
>          rbd cache = true

Turning on rbd caching in ceph.conf, while telling QEMU it is not
writeback in libvirt's xml like:

    <driver name='qemu' type='raw' cache='none'/>

is not safe since QEMU won't propagate flush requests from the guest
to librbd's cache. Since QEMU 1.3 setting the QEMU cache mode will
also set the librbd cache mode appropriately, so you don't need to
have the setting in your ceph.conf file at all.

Can you verify that your QEMU binary was built against librbd v0.60 or
later?

It would also help to see whether you can reproduce the problem when
QEMU is sending flushes (cache=writeback), and with the cache
disabled (cache=none, no 'rbd cache' setting in ceph.conf).

The next step would be gathering a log from librbd (debug rbd = 20,
debug objectcacher=20, debug objecter=20, debug ms = 1) when this
problem is occurring, and correlating it with a timestamped log of
the network latency.

Thanks,
Josh

>          auth support = cephx none
>          mon host =
> x.x.x.x:6789;x.x.x.x:6789;x.x.x.x:6789;x.x.x.x:6789;x.x.x.x:6789
>
> Regards
>
> Alan ye
>
> ------------------
> ??????
> Alan Ye
>
>
>
> ------------------ ???????? ------------------
> *??????:* "Josh Durgin";<address@hidden>;
> *????????:* 2014??1??14??(??????) ????2:24
> *??????:* "Stefan Hajnoczi"<address@hidden>; "??????"
> <address@hidden>;
> *????:* "qemu-devel"<address@hidden>;
> *????:* Re: ?????? ?????? ?????? [Qemu-devel] qemu vm big network
> latency when met heavy io
>
> On 01/12/2014 06:39 PM, Stefan Hajnoczi wrote:
>  > On Fri, Jan 10, 2014 at 11:50 AM, ?????? <address@hidden> wrote:
>  >
>  > Please use Reply-all to keep the CC list in tact.  That way the
>  > conversation stays on the mailing list and others can participate.
>  >
>  >>> Is the sum of guests' RAM less than the total physical RAM on the host
>  >> The host run 3 vms, each vm use one vcpu core and 1G ram.
>  >> # free -m
>  >>
>  >>               total       used       free     shared    buffers
> cached
>  >>
>  >> Mem:         32242       4808      27434          0        278
> 2058
>  >>
>  >> -/+ buffers/cache:       2471      29771
>  >>
>  >> Swap:         4095          0       4095
>  >>
>  >>
>  >> The host has 8 cores.
>  >> # cat /proc/cpuinfo | grep processor
>  >> processor : 0
>  >> processor : 1
>  >> processor : 2
>  >> processor : 3
>  >> processor : 4
>  >> processor : 5
>  >> processor : 6
>  >> processor : 7
>  >>
>  >> so, both of two question's answer is 'yes'.
>  >> When I runing random write test, the host use 0 swap.
>  >
>  > Great.  That means the host is not overcommitted.
>  >
>  > It's likely that the problem is a bug in QEMU's rbd driver or librados.
>  >
>  > Josh: Perhaps something you're interested in looking into?
>
> Yes, thanks for bringing it to my attention. It does sound like a bug in
> QEMU's rbd driver or ceph's userspace libraries.
>
> Could you share what version of librbd you're using, and your
> /etc/ceph/ceph.conf?
>
> Thanks,
> Josh