Re: [Qemu-devel] Live migrate, inconsistent machine types - new machine type to fix?

[Alex Bligh]

Paolo,

On 19 Jul 2014, at 08:30, Paolo Bonzini <address@hidden> wrote:

> Il 19/07/2014 09:10, Alex Bligh ha scritto:
>>>> Looks like your source and destination machines have different
>>>> iPXE ROMs.  This could be a packaging problem in your distro.
>> 
>> This is Ubuntu 12.04 to Ubuntu 14.04. I think the 14.04 one is
>> correct, and don't want to run with a non-standard qemu there.
> 
> No, the old one is _always_ the correct one.
> 
> The pxe-* ROMs must be 128k, the efi-* ROMs must be 256k.

On 12.04
# ls -la /usr/share/qemu/pxe-virtio.rom
-rw-r--r-- 1 root root 63488 Jun 12 23:23 /usr/share/qemu/pxe-virtio.rom

On 14.04 (after resolving a pile of symlinks)
# ls -la /usr/lib/ipxe/qemu/pxe-virtio.rom
-rw-r--r-- 1 root root 83968 Jan  6  2014 /usr/lib/ipxe/qemu/pxe-virtio.rom

I'm guessing the ROM size gets rounded up to the next power of 2, hence
the ROM size on 12.04 is 64k, and on 14.04 is 128k, right?

So it would appear that if the length is actually coming from the
length of the file on disk, that on 12.04 the pxe-ROM is not 128k
but 64k. So if 'pxe-* ROMs must be 128k' is correct, I'm not sure
that's incompatible with  'the old one is _always_ the correct
one'.

Whilst (per below) your workaround gets past the issue for this ROM,
I'm confused about the numbers in the error message, as 64k and
128k do not equal 10,000 or 20,000:
 Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 10000 in != 20000


>  If distros do
> not do that, they have a bug (and as you noticed it's not one that they
> can easily work around, since now you have different releases with
> different ROM sizes for the same machine types).

Thanks. So I am discovering.

>> Can I use a different ROM (e.g. the old one) with a command line
>> option just for the migrated machines? Obviously when they reboot,
>> they'l get a hardware change (which is fine).
> 
> Yes, "-global foo.romfile" on the destination QEMU.  You can just pass
> an empty 128k file to the destination since ROM contents are migrated
> properly, and hardware will only change when QEMU restarts.

Thanks.

So, copying the old ROM file across (out of an abundance of caution)
plus the magic incantation:

-machine pc-1.0 -global cirrus-vga.vgamem_mb=10 -global 
virtio-net-pci.romfile=pxe-virtio.rom.12.04

would appear to be sufficient to permit the migration to get past the
initial stages and commence the block migration.

However, after the block migration is finished, I now see:

Receiving block device images
Completed 100 %
Unknown savevm section type 255
load of migration failed

This would appear to be similar to the issue you reported here
  http://comments.gmane.org/gmane.comp.emulators.qemu/49702
(though given the dates that particular issue should be fixed
in both versions), which presumably means there is some binary
format incompatibility between sender and receiver.

My guess is this might be the bogus inclusion of PITCommonState
per the last hunk of
 
http://pkgs.fedoraproject.org/cgit/qemu.git/tree/0001-Fix-migration-from-qemu-kvm.patch?h=f20

Is there a technique to debug this sort of thing?

> Migration is extremely complex because you have to strike the right
> balance between compatibility and maintainability.  Whether it works, it
> depends on a lot of factors that vary per distro, and the only thing
> that can keep it working is a lot of testing.

Indeed, which is what I now appear to be spending my weekend doing ...

> I don't know of any distro that actively tests migration except RHEL,
> SLES.  I'm not sure about Debian.  Since there is no free-beer SLES,
> you'd better switch to CentOS 7 than keeping Ubuntu 14.04.

Switching is not an option here; we have way too much other stuff dependent
on Ubuntu.

Ubuntu is about to become tested (and a patch or workaround generated
at least for migrations from 12.04 - the previous LTS version);
whether or not it is upstreamed won't be up to me though.

To be fair, I think the issue is that they are supporting migration from
stock qemu.git pc-1.0 (which is what I think they shipped with the
subsequent non-LTS releases, which means they can't (easily) also
support migration from qemu-kvm pc-1.0. That's the bit I want to fix.

Fortunately we have a relatively limited number of qemu VM configs out
there in the wild.

-- 
Alex Bligh