Re: [Qemu-devel] memory allocation of migration changed?

[Stefan Hajnoczi]

On Fri, Feb 14, 2014 at 07:16:17PM +0100, Stefan Priebe wrote:
> 
> Am 14.02.2014 16:03, schrieb Stefan Hajnoczi:
> >On Tue, Feb 11, 2014 at 07:30:54PM +0100, Stefan Priebe wrote:
> >>Am 11.02.2014 16:44, schrieb Stefan Hajnoczi:
> >>>On Tue, Feb 11, 2014 at 3:54 PM, Stefan Priebe - Profihost AG
> >>><address@hidden> wrote:
> >>>>in the past (Qemu 1.5) a migration failed if there was not enogh memory
> >>>>on the target host available directly at the beginning.
> >>>>
> >>>>Now with Qemu 1.7 i've seen succeeded migrations but the kernel OOM
> >>>>memory killer killing qemu processes. So the migration seems to takes
> >>>>place without having anough memory on the target machine?
> >>>
> >>>How much memory is the guest configured with?  How much memory does
> >>>the host have?
> >>
> >>Guest: 48GB
> >>Host: 192GB
> >>
> >>>I wonder if there are zero pages that can be migrated almost "for
> >>>free" and the destination host doesn't touch.  When they are touched
> >>>for the first time after migration handover, they need to be allocated
> >>>on the destination host.  This can lead to OOM if you overcommitted
> >>>memory.
> >>
> >>In the past the migration failed immediatly with exit code 255.
> >>
> >>>Can you reproduce the OOM reliably?  It should be possible to debug it
> >>>and figure out whether it's just bad luck or a true regression.
> >>
> >>So there is no known patch changing this behaviour?
> >>
> >>What is about those?
> >>fc1c4a5d32e15a4c40c47945da85ef9c1e0c1b54
> >>211ea74022f51164a7729030b28eec90b6c99a08
> >>f1c72795af573b24a7da5eb52375c9aba8a37972
> >
> >Yes, that's what I was referring to when I mentioned zero pages.
> >
> >The problem might just be that the destination host didn't have enough
> >free memory.  Migration succeeded due to memory overcommit on the host,
> >but quickly ran out of memory after handover.  The quick answer there is
> >to reconsider your overcommitting memory and also checking memory
> >availability before live migrating.
> 
> Yes that makes sense in the past it was just working due to a
> failing qemu process.
> 
> What is the right way to check for enough free memory and memory
> usage of a specific vm?

I would approach it in terms of guest RAM allocation plus QEMU overhead:

  host_ram >= num_guests * guest_ram_size + num_guests * qemu_overhead

The qemu_overhead is the question mark.  It depends on the QEMU features
the guests have enabled and number of devices.

QEMU also does not have a strict policy on predictable memory
consumption, which makes it hard to give a formula for it.

You could get an idea by running memtest86 or something that touches all
memory inside the guest and then looking at the RSS memory statistic for
the QEMU process:

  qemu_overhead = rss - guest_ram_size

Even that is just an estimation since RSS only tells you how much host
physical memory is *currently* allocated.  There could be some codepath
that hasn't been executed yet that will require much more memory.

Whatever you do, don't look at the virtual size (VSZ) because that is
very misleading (including mmaps of pages that may never be touched).

Stefan