Re: [Qemu-devel] [RFC] Split migration bitmaps by ramblock

[Dr. David Alan Gilbert]

* Juan Quintela (address@hidden) wrote:
> Hi
> 
> This series split the migration and unsent bitmaps by ramblock.  This
> makes it easier to synchronize in small bits.  This is on top of the
> RAMState and not-hotplug series.

So I think generally this is a good idea; my main reason is I'd like
to see per-NUMA node syncing, preferably tied with multi-fd so that
each of the NUMA nodes syncs and stuffs data over it's own NIC.
Although we would still have to be careful about cases where one
node is hammering it's RAM and the others are idleing.

> Why?
> 
> reason 1:
> 
> People have complained that by the time that we detect that a page is
> sent, it has already been marked dirty "again" inside kvm, so we are
> going to send it again.  On top of this patch, my idea is, for words
> of the bitmap that have any bit set, just synchonize the bitmap before
> sending the pages.  I have not looking into performance numbers yet,
> jsut asking for comments about how it is done.
> 
> reason 2:
> 
> In case where the host page is a multiple of the the TARGET_PAGE_SIZE,
> we do a lot of work when we are synchronizing the bitmaps to pass it
> to target page size.  The idea is to change the bitmaps on that
> RAMBlocks to mean host page size and not TARGET_PAGE_SIZE.
> 
> Note that there are two reason for this, ARM and PPC do things like
> guests with 4kb pages on hosts with 16/64kb hosts, and then we have
> HugePages.  Note all the workarounds that postcopy has to do because
> to work in HugePages size.

There are some fun problems with changing the bitmap page size;
off the top of my head, the ones I can remember include:
    a) I'm sure I've seen rare cases where a target page is marked as
       dirty inside a hostpage; I'm guessing that was qemu's doing, but
       there are more subtle cases, e.g. running a 4kb guest on a 64kb host;
       it's legal - and 4kb power guests used to exist;  I think in those
       cases you see KVM only marking one target page as dirty.

    b) Are we required to support migration across hosts of different pagesize;
       and if we do that what size should a bit represent?
       People asked about it during postcopy but I think it's restricted to
       matching sizes.  I don't think precopy has any requirement for matching
       host pagesize at the moment.  64bit ARM does 4k, 64k and I think 16k was
       added later.

    c) Hugepages have similar issues; precopy doesn't currently have any
       requirement for the hugepage selection on the two hosts to match,
       but it does on postcopy.  Also you don't want to have a single dirty
       bit for a 1GB host hugepage if you can handle detecting changes at
       a finer grain level.

Dave
> Please, comment?
> 
> Later, Juan.
> 
> Juan Quintela (1):
>   ram: Split dirty bitmap by RAMBlock
> 
>  include/exec/ram_addr.h |  13 +++-
>  migration/ram.c         | 201 
> ++++++++++++++++++------------------------------
>  2 files changed, 85 insertions(+), 129 deletions(-)
> 
> -- 
> 2.9.3
> 
--
Dr. David Alan Gilbert / address@hidden / Manchester, UK