Re: [Qemu-devel] troubleshooting live migration

[Marcus Sorensen]

Ok, more information. The console spews 'lapic increasing min_delta_ns
to ################' when this happens.

On Tue, Jan 14, 2014 at 8:31 AM, Marcus Sorensen <address@hidden> wrote:
> Does anyone have tips on troubleshooting live migration? I've got
> several E5-2650 servers running in test environment, kernel 3.10.26
> and qemu 1.7.0. If I start a VM guest (say ubuntu, debian, or centos),
> I can migrate it around  from host to host to host just fine, but if I
> wait awhile (say 1 hour), I try to migrate and it succeeds but the
> guest is hosed. No longer pings, cpu is thrashing. I've tried to
> strace it and don't see anything that other working hosts aren't
> doing, and I've tried gdb but I'm not entirely sure what I'm doing. I
> tried downgrading to qemu 1.6.1. I've found dozens of reports of such
> behavior, but they're all due to other things (migrating between
> different host CPUs, someone thinking it's virtio or memballoon only
> to later find a fix like changing machine type, etc). I'm at a loss.
> This seems to work just fine with stock CentOS builds.
>
> I'd be happy to try to capture a core if someone is willing to look at it.
>
> Here's an example xml:
>
> <domain type='kvm'>
>   <name>VM12</name>
>   <uuid>dd25acfc-e24d-4de6-814c-72ac465bc208</uuid>
>   <description></description>
>   <memory unit='KiB'>4194304</memory>
>   <currentMemory unit='KiB'>4194304</currentMemory>
>   <vcpu placement='static'>2</vcpu>
>   <cputune>
>     <shares>2000</shares>
>   </cputune>
>   <resource>
>     <partition>/machine</partition>
>   </resource>
>   <os>
>     <type arch='x86_64' machine='pc-i440fx-1.7'>hvm</type>
>     <boot dev='cdrom'/>
>     <boot dev='hd'/>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>     <pae/>
>   </features>
>   <cpu>
>   </cpu>
>   <clock offset='utc'>
>     <timer name='kvmclock' tickpolicy='catchup'/>
>   </clock>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>destroy</on_crash>
>   <devices>
>     <emulator>/usr/bin/qemu-kvm</emulator>
>     <disk type='block' device='disk'>
>       <driver name='qemu' type='raw' cache='none'/>
>       <source dev='/dev/sdc'/>
>       <target dev='vda' bus='virtio'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x0'/>
>     </disk>
>     <disk type='file' device='cdrom'>
>       <driver name='qemu' type='raw' cache='none'/>
>       <target dev='hdc' bus='ide'/>
>       <readonly/>
>       <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>     </disk>
>     <controller type='ide' index='0'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x1'/>
>     </controller>
>     <controller type='virtio-serial' index='0'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> function='0x0'/>
>     </controller>
>     <controller type='usb' index='0'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x2'/>
>     </controller>
>     <controller type='pci' index='0' model='pci-root'/>
>     <interface type='bridge'>
>       <mac address='02:00:09:66:00:18'/>
>       <source bridge='br1000192'/>
>       <model type='virtio'/>
>       <bandwidth>
>         <inbound average='128000' peak='128000'/>
>         <outbound average='128000' peak='128000'/>
>       </bandwidth>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>     </interface>
>     <serial type='pty'>
>       <target port='0'/>
>     </serial>
>     <console type='pty'>
>       <target type='serial' port='0'/>
>     </console>
>     <channel type='unix'>
>       <source mode='bind' path='/var/lib/libvirt/qemu/VM12.agent'/>
>       <target type='virtio' name='VM12.vport'/>
>       <address type='virtio-serial' controller='0' bus='0' port='1'/>
>     </channel>
>     <input type='tablet' bus='usb'/>
>     <input type='mouse' bus='ps2'/>
>     <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
>       <listen type='address' address='0.0.0.0'/>
>     </graphics>
>     <video>
>       <model type='cirrus' vram='9216' heads='1'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>
>     </video>
>     <memballoon model='virtio'>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0'/>
>     </memballoon>
>   </devices>
>   <seclabel type='none'/>
> </domain>