Re: [Qemu-devel] [BUG] Balloon malfunctions with memory hotplug

[Luiz Capitulino]

On Fri, 27 Feb 2015 08:27:00 +0100
Markus Armbruster <address@hidden> wrote:

> Luiz Capitulino <address@hidden> writes:
> 
> > Hello,
> >
> > Reproducer:
> >
> > 1. Start QEMU with balloon and memory hotplug support:
> >
> > # qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio
> >
> > 2. Check balloon size:
> >
> > (qemu) info balloon
> > balloon: actual=1024
> > (qemu) 
> >
> > 3. Hotplug some memory:
> >
> > (qemu) object_add memory-backend-ram,id=mem1,size=1G
> > (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
> >
> > 4. This is step is _not_ needed to reproduce the problem,
> >    but you may need to online memory manually on Linux so
> >    that it becomes available in the guest
> >
> > 5. Check balloon size again:
> >
> > (qemu) info balloon
> > balloon: actual=1024
> > (qemu) 
> >
> > BUG: The guest now has 2GB of memory, but the balloon thinks
> >      the guest has 1GB
> 
> Impact other than "info balloon"?

You can only balloon what's reported by "info balloon". If the
guest was booted with 1GB but you hot added another 6GB, then
you'll only be able to balloon 1GB.

> > One may think that the problem is that the balloon driver is
> > ignoring hotplugged memory. This is not what's happening. If
> > you do balloon your guest, there's nothing stopping the
> > balloon driver in the guest from ballooning hotplugged memory.
> >
> > The problem is that the balloon device in QEMU needs to know
> > the current amount of memory available to the guest.
> >
> > Before memory hotplug this information was easy to obtain: the
> > current amount of memory available to the guest is the memory the
> > guest was booted with. This value is stored in the ram_size global
> > variable in QEMU and this is what the balloon device emulation
> > code uses today. However, when memory is hotplugged ram_size is
> > _not_ updated and the balloon device breaks.
> >
> > I see two possible solutions for this problem:
> >
> > 1. In addition to reading ram_size, the balloon device in QEMU
> >    could scan pc-dimm devices to account for hotplugged memory.
> >
> >    This solution was already implemented by zhanghailiang:
> >
> >     http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
> >
> >    It works, except that on Linux memory hotplug is a two-step
> >    procedure: first memory is inserted then it has to be onlined
> >    from user-space. So, if memory is inserted but not onlined
> >    this solution gives the opposite problem: the balloon device
> >    will report a larger memory amount than the guest actually has.
> >
> >    Can we live with that? I guess not, but I'm open for discussion.
> >
> >    If QEMU could be notified when Linux makes memory online, then
> >    the problem would be gone. But I guess this can't be done.
> >
> > 2. Modify the balloon driver in the guest to inform the balloon
> >    device on the host about the current memory available to the
> >    guest. This way, whenever the balloon device in QEMU needs
> >    to know the current amount of memory in the guest, it asks
> >    the guest. This drops any usage of ram_size in the balloon
> >    device
> 
> What happens when the guest lies?

There are two kinds of guests that would lie: a broken guest or
a malicious guest. For a malicious guest, the worst case I can think
of is that we get the same problem we have today. Not a big deal for
the host, I guess. A broken guest has to be fixed :)

However, I'm getting to the conclusion that this solution will
complicate things even more and may add a bunch of new problems.

I'm leaning towards applying zhanghailiang's series for now. This
series fixes the best case: memory is inserted and the guests uses
all of it right away.

The worst case is: memory is inserted and the guest doesn't use it.
In this case QEMU will allow you to balloon more memory than the
guest is using, which can crash the guest. For example, the guest
is booted with 1GB you hot add 6GB but the guest doesn't use it.
info balloon will report 6GB and will allow you to balloon the guest
down to 2GB, which will crash the guest.

In theory I think this case has always been broken, but in practice
it's very hard (almost impossible?) to reproduce in a Linux 64-bit
guest as you'd have to be able to start the guest with more memory
than it can recognize.

> >    I'm not completely sure this is feasible though. For example,
> >    what happens if the guest reports a memory amount to QEMU and
> >    right after this more memory is plugged?
> >
> >    Besides, this solution is more complex than solution 1 and
> >    won't address older guests.
> >
> > Another important detail is that, I *suspect* that a very similar
> > bug already exists with 32-bit guests even without memory
> > hotplug: what happens if you assign 6GB to a 32-bit without PAE
> > support? I think the same problem we're seeing with memory
> > hotplug will happen and solution 1 won't fix this, although
> > no one seems to care about 32-bit guests...
> 
> Fun...
>