Re: [Qemu-devel] [RFC] postcopy livemigration proposal

[Andrea Arcangeli]

Hello everyone,

so basically this is a tradeoff between not having a long latency for
the migration to succeed and reducing the total network traffic (and
CPU load) in the migration source and destination and reducing the
memory footprint a bit, by adding an initial latency to the memory
accesses on the destination of the migration (i.e. causing a more
significant and noticeable slowdown to the guest).

It's more or less like if when the guest starts on the destination
node, it will find all its memory swapped out to a network swap
device, so it needs to do I/O for the first access (side note: and
hopefully it won't run out of memory while the memory is copied to the
destination node or the guest will crash).

On Thu, Aug 11, 2011 at 11:19:19AM +0900, Isaku Yamahata wrote:
> On Wed, Aug 10, 2011 at 04:55:32PM +0300, Avi Kivity wrote:
> > I'm not 100% sure, but I think that thp and ksm need the vma to be  
> > anonymous, not just the page.
> 
> Yes, they seems to check if not only the page is anonymous, but also the vma.
> I'd like to hear from Andrea before digging into the code deeply.

The vma doesn't need to be anonymous for THP, an mmap on /dev/zero
MAP_PRIVATE also is backed by THP. But it must be close to anonymous
and not have special VM_IO/PFNMAP flags or khugepaged/ksm will not
scan it. ->vm_file itself isn't checked by THP/KSM (sure for THP
because of the /dev/zero example which I explicitly fixed as it wasn't
fully handled initially). NOTE a chardevice won't work on RHEL6
because I didn't allow /dev/zero to use it there (it wasn't an
important enough feature and it was more risky) but upstream it should
work already.

A chardevice doing this may work, even if it would be simpler/cleaner
if this was still an anonymous vma. A chardevice could act similar to
/dev/zero MAP_PRIVATE. In theory KSM should work on /dev/zero too, you
can test that if you want. But a chardevice will require dealing with
permissions when we don't actually need special permissions for this.

Another problem is you can't migrate the stuff using hugepages or it'd
multiply the latency 512 times (with 2M contiguous access it won't
make a difference but if the guest is accessing memory randomly it
would make a difference). So you will have to relay on khugepaged to
collapse the hugepages later. That should work but initially the guest
will run slower even when the migration is already fully completed.

> If it is possible to convert the vma into anonymous, swap device or
> backed by device/file wouldn't matter in respect to ksm and thp.
> Acquiring mmap_sem suffices?

A swap device would require root permissions and we don't want qemu to
mangle over the swapdevices automatically. It'd be bad to add new
admin requirements, few people would use it. Ideally the migration API
should remain the same and it should be an internal tweak in qemu to
select which migration mode to use beforehand.

Even if it was a swap device it'd still require special operations to
setup swap entries in the process pagetables before the pages
exists. A swap device may give more complication than it solves.

If it was only KVM accessing the guest physical memory we could just
handle it in KVM, and call get_user_pages_fast, if that fails and it's
the first ever invocation we just talk with QEMU to get the page and
establish it by hand. But qemu can also write to memory and if it's a
partial write and the guest reads the not-written yet part with
get_user_pages_fast+spte establishment, it'll go wrong. Maybe qemu is
already doing all checks on the pages it's going to write and we could
hook there too from qemu side.

Another more generic (not KVM centric) that will not require a special
chardev or a special daemon way could be a new:

        sys_set_muserload(unsigned long start, unsigned long len, int signal)
        sys_muserload(void *from, void *to)

When sys_set_muserload is called the region start,start+len gets
covered by muserload swap entries that trigger special page faults.

When anything touches the memory with an muserload swap entry still
set, the thread gets a signal with force_sig_info_fault(si_signo =
signal), and the signal handler will get the faulting address in
info.si_addr. The signal handler is then responsible to call
sys_userload after talking the thread doing the TCP send/recv()
talking with the qemu source. The recv(mmap(4096), 4096) should
generate a page in the destination node in some random mapping
(aligned).

Then the muserload(tcp_received_page_address,
guest_faulting_physical_address_from_info_si_addr) does get_user_pages
on tcp_received_page_address, takes it away from the
tcp_received_page_address (clears the pte at that address), adjusts
page->index for the new vma, and maps the page zercopy atomically into
the "guest_faulting_physical_address_from_info_si_addr" new address,
if and only if the pagetable at address is still of muserload
type. Then the signal munmap(tcp_received_page_address, 4096) to
truncate/free the vma (the page is already moved and it's already
free) and the signal returns and this time the guest access the page.

Another second thread in the background calls
muserload(tcp_received_page_address, guest_physical_address) maxing
out the network bandwidth using a second TCP socket for the streaming
transfer ignoring if muserload fails (if the async page fault arrives
first and already loaded the information there).

If a double muserload fault happens before the first pending I guess
it'll just hang and kill -9 will solve it (or we could detect a double
fault and sigsegv).

qemu better not touch guest physical ram marked muserload by
set_muserload() from signal context.

The signal handler may communicate with another thread if there's only
one TCP socket for the "sync muserload fault" transfers for each qemu
instance. Alternatively every vcpu thread plus the iothread could talk
with the source with a different TCP socket (also to reduce the
latency) and if multiple vcpu faults on the same address it'll just
loop a bit for the vcpus that didn't ask the page first to the source
qemu. For example if the source will respond "already freed" then
loader returns without calling muserload() and loop continues until
the vcpu that recv()d the page from the source finally calls
muserload().

Too much already for not having though enough about it yet, these are
the first ideas that comes to mind.

Not sure if the guaranteed slowdown on the destination node could be
preferable as a "default" migration mode, but this certainly sounds a
more reliable (i.e. maybe better for enterprise) way of doing
migration because there are no black magic numbers involved to decide
when to stop the source node and transfer all remaining dirty pages,
magic numbers aren't good for enterprise because of the potentially
enormous VM sizes or very heavy workloads. OTOH for desktop virt:
small VM and usually idle and with tiny working set the current
precopy method probably is less visible to the user, but it's less
reliable too. So maybe this is a better default because of the less
black magic and more reliability even if it would most certainly
perform worse for the small use case.

Especially on 100mbit networks it may be pretty bad, the equivalent of
a 10MB/sec swap device will be quite a pretty bad initial slowdown. At
least it won't be slowed down by seeking but swapping in at 10M/sec
like on a real old HD is going to be still very bad. It'll run almost
as slow as a low-end laptop that suspended-to-disk by swapping out
half of the RAM. So pretty noticeable. With 1gigabit and up it'll get
better.

Thanks,
Andrea