Re: [Qemu-devel] troubleshooting live migration

[Marcus Sorensen]

I tried -no-hpet, was still able to replicate the 'lapic' issue. I
find it interesting that I can only trigger it if the vm has been
running awhile.

On Tue, Jan 14, 2014 at 11:08 PM, Marcus Sorensen <address@hidden> wrote:
> Ok, more information. The console spews 'lapic increasing min_delta_ns
> to ################' when this happens.
>
> On Tue, Jan 14, 2014 at 8:31 AM, Marcus Sorensen <address@hidden> wrote:
>> Does anyone have tips on troubleshooting live migration? I've got
>> several E5-2650 servers running in test environment, kernel 3.10.26
>> and qemu 1.7.0. If I start a VM guest (say ubuntu, debian, or centos),
>> I can migrate it around  from host to host to host just fine, but if I
>> wait awhile (say 1 hour), I try to migrate and it succeeds but the
>> guest is hosed. No longer pings, cpu is thrashing. I've tried to
>> strace it and don't see anything that other working hosts aren't
>> doing, and I've tried gdb but I'm not entirely sure what I'm doing. I
>> tried downgrading to qemu 1.6.1. I've found dozens of reports of such
>> behavior, but they're all due to other things (migrating between
>> different host CPUs, someone thinking it's virtio or memballoon only
>> to later find a fix like changing machine type, etc). I'm at a loss.
>> This seems to work just fine with stock CentOS builds.
>>
>> I'd be happy to try to capture a core if someone is willing to look at it.
>>
>> Here's an example xml:
>>
>> <domain type='kvm'>
>>   <name>VM12</name>
>>   <uuid>dd25acfc-e24d-4de6-814c-72ac465bc208</uuid>
>>   <description></description>
>>   <memory unit='KiB'>4194304</memory>
>>   <currentMemory unit='KiB'>4194304</currentMemory>
>>   <vcpu placement='static'>2</vcpu>
>>   <cputune>
>>     <shares>2000</shares>
>>   </cputune>
>>   <resource>
>>     <partition>/machine</partition>
>>   </resource>
>>   <os>
>>     <type arch='x86_64' machine='pc-i440fx-1.7'>hvm</type>
>>     <boot dev='cdrom'/>
>>     <boot dev='hd'/>
>>   </os>
>>   <features>
>>     <acpi/>
>>     <apic/>
>>     <pae/>
>>   </features>
>>   <cpu>
>>   </cpu>
>>   <clock offset='utc'>
>>     <timer name='kvmclock' tickpolicy='catchup'/>
>>   </clock>
>>   <on_poweroff>destroy</on_poweroff>
>>   <on_reboot>restart</on_reboot>
>>   <on_crash>destroy</on_crash>
>>   <devices>
>>     <emulator>/usr/bin/qemu-kvm</emulator>
>>     <disk type='block' device='disk'>
>>       <driver name='qemu' type='raw' cache='none'/>
>>       <source dev='/dev/sdc'/>
>>       <target dev='vda' bus='virtio'/>
>>       <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
>> function='0x0'/>
>>     </disk>
>>     <disk type='file' device='cdrom'>
>>       <driver name='qemu' type='raw' cache='none'/>
>>       <target dev='hdc' bus='ide'/>
>>       <readonly/>
>>       <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>>     </disk>
>>     <controller type='ide' index='0'>
>>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
>> function='0x1'/>
>>     </controller>
>>     <controller type='virtio-serial' index='0'>
>>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
>> function='0x0'/>
>>     </controller>
>>     <controller type='usb' index='0'>
>>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
>> function='0x2'/>
>>     </controller>
>>     <controller type='pci' index='0' model='pci-root'/>
>>     <interface type='bridge'>
>>       <mac address='02:00:09:66:00:18'/>
>>       <source bridge='br1000192'/>
>>       <model type='virtio'/>
>>       <bandwidth>
>>         <inbound average='128000' peak='128000'/>
>>         <outbound average='128000' peak='128000'/>
>>       </bandwidth>
>>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>> function='0x0'/>
>>     </interface>
>>     <serial type='pty'>
>>       <target port='0'/>
>>     </serial>
>>     <console type='pty'>
>>       <target type='serial' port='0'/>
>>     </console>
>>     <channel type='unix'>
>>       <source mode='bind' path='/var/lib/libvirt/qemu/VM12.agent'/>
>>       <target type='virtio' name='VM12.vport'/>
>>       <address type='virtio-serial' controller='0' bus='0' port='1'/>
>>     </channel>
>>     <input type='tablet' bus='usb'/>
>>     <input type='mouse' bus='ps2'/>
>>     <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
>>       <listen type='address' address='0.0.0.0'/>
>>     </graphics>
>>     <video>
>>       <model type='cirrus' vram='9216' heads='1'/>
>>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
>> function='0x0'/>
>>     </video>
>>     <memballoon model='virtio'>
>>       <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>> function='0x0'/>
>>     </memballoon>
>>   </devices>
>>   <seclabel type='none'/>
>> </domain>