[Qemu-devel] AMD Opteron 6276 to Intel Xeon E5645 live migration failure

[Nick Thomas]

Hi there,

We recently acquired some of the latter CPUs, and are attempting to add
them to our existing cluster of AMD 6200/6300 KVM hosts. My life is made
much easier if live migration between all hosts in the cluster works
flawlessly, but I can reliably trigger a migration failure when moving a
guest from the 6276 to the E5645 hosts.

The failure looks like:

KVM: injection failed, MSI lost (Operation not permitted)
KVM: entry failed, hardware error 0x80000021

If you're running a guest on an Intel machine without unrestricted mode
support, the failure can be most likely due to the guest entering an
invalid state for Intel VT. For example, the guest maybe running in big
real mode which is not supported on less recent Intel processors.

EAX=00000000 EBX=81600000 ECX=00000000 EDX=00000000
ESI=00000003 EDI=81600000 EBP=8168f060 ESP=81601f10
EIP=8102b36c EFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00

The guest itself is running Linux in ordinary 64-bit mode at the point
where it's migrated.

I also get this in dmesg:

kvm: VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL does not work properly. Using
workaround

I've tried kernels 3.2.54 and 3.10.26 on the source host, and 3.2.54,
3.10.26 and 3.12.9 on the destination; and qemu versions 1.5.3 and
1.7.0. Same error in all cases. I've also tried twiddling the
emulate_invalid_guest_state flag, to no effect.

I can reproduce it with a guest as simple as:

/opt/qemu-1.5.3/bin/qemu-system-x86_64 -enable-kvm -nographic -monitor
stdio -kernel /boot/vmlinuz-3.10.26-bigv-20 -initrd
/boot/initrd.img-3.10.26-bigv-20 -net none

on the source, then migrating that to the destination.

I can reliably boot and run the guest on the E5645 host, and migrate
that guest to the AMD host. Migrating the same guest back again fails
with the same error.

Migrating from a later Intel machine (i7-4600U) to the earlier Intel
seems to work fine.

I'd love to get this working, but I'm a little ignorant on where to
begin, or even if it's possible at all. Are these CPUs just too old, or
is a fixup missing in qemu (or kvm)?

/Nick