Re: [Qemu-devel] [RFC] postcopy livemigration proposal

[Anthony Liguori]

On 08/08/2011 10:11 AM, Dor Laor wrote:
> On 08/08/2011 03:32 PM, Anthony Liguori wrote:
> > On 08/08/2011 04:20 AM, Dor Laor wrote:
> > > That's terrific (nice video also)!
> > > Orit and myself had the exact same idea too (now we can't patent it..).
> > >
> > > Advantages:
> > > - No down time due to memory copying.
> >
> > But non-deterministic down time due to network latency while trying to
> > satisfy a page fault.
>
> True but it is possible to limit it with some dedicated network or
> bandwidth reservation.

Yup.  Any technique that uses RDMA (which is basically what this is) 
requires dedicated network resources.

> > > - Efficient, reduce needed traffic no need to re-send pages.
> >
> > It's not quite that simple. Post-copy needs to introduce a protocol
> > capable of requesting pages.
>
> Just another subsection.. (kidding), still it shouldn't be too
> complicated, just an offset+pagesize and return page_content/error

What I meant by this is that there is potentially a lot of round trip 
overhead.  Pre-copy migration works well with reasonable high latency 
network connections because the downtime is capped only by the maximum 
latency sending from one point to another.

But with something like this, the total downtime is 
2*max_latency*nb_pagefaults.  That's potentially pretty high.

So it may be desirable to try to reduce nb_pagefaults by prefaulting in 
pages, etc.  Suffice to say, this ends up getting complicated and may 
end up burning network traffic too.

> > This is really just a limitation of our implementation. In theory,
> > pre-copy allows you to exert fine grain resource control over the guest
> > which you can use to encourage convergence.
>
> But a very large guest w/ large working set that changes more frequent
> than the network bandwidth might always need huge down time with the
> current system.

In theory, you can do things like reduce the guests' priority to reduce 
the amount of work it can do in order to encourage convergence.

> > One thing I think we need to do is put together a live migration
> > roadmap. We've got a lot of invasive efforts underway with live
> > migration and I fear that without some planning and serialization, some
> > of this useful work with get lost.
>
> Some of them are parallel. I think all the readers here agree that post
> copy migration should be an option while we need to maintain the current
> one.

I actually think they need to be done mostly in sequence while cleaning 
up some of the current infrastructure.  I don't think we really should 
make any major changes (beyond maybe the separate thread) until we 
eliminate QEMUFile.

There's so much overhead involved in using QEMUFile today, I think it's 
hard to talk about performance data when we've got a major bottleneck 
sitting in the middle.

Regards,

Anthony Liguori