[Qemu-devel] [Bug 1192499] Re: virsh migration copy-storage-all fails with "Unable to read from monitor: Connection reset by peer"

[Bug Watch Updater]

Launchpad has imported 15 comments from the remote bug at
https://bugzilla.redhat.com/show_bug.cgi?id=979411.

If you reply to an imported comment from within Launchpad, your comment
will be sent to the remote bug automatically. Read more about
Launchpad's inter-bugtracker facilities at
https://help.launchpad.net/InterBugTracking.

------------------------------------------------------------------------
On 2013-06-28T13:15:38+00:00 chandrashekar wrote:

Created attachment 766573
Source-migrate-logs

Description of problem:

virsh migration copy-storage-all fails with "Unable to read from
monitor: Connection reset by peer" and shut downs the guest on the
source host.

Kernel Version: 3.10.0-rc5+

Libvirt Version: 1.0.6

Qemu Version: 1.5.50

Steps to Reproduce:

1. Created the qemu-img create -f qcow2 vm.qcow2 11G on the destination host 
which is same as the source.
2. Started the guest on the source
3. Started the vncdisplay to monitor the guest
4. Initiated the migration "virsh migrate --live VM1 qemu+ssh://host-ip/system 
tcp://host-ip --verbose --copy-storage-all"
5. It started the copying the storage from souce to destination (conitinously 
monitored it was growing)
6. Guest on the destination was paused and was running on the source
7. At some point the VM on the source got shutdown and migration failed with 
"Unable to read from monitor: Connection reset by peer"

Attached the libvirt debug logs.

The debug logs shows :

2013-06-19 08:43:12.253+0000: 4026: debug : virEventPollInterruptLocked:716 : 
Interrupting
2013-06-19 08:43:12.253+0000: 4026: debug : virEventPollAddTimeout:248 : 
EVENT_POLL_ADD_TIMEOUT: timer=1 frequency=0 cb=0x7fe930baa960 opaque=(nil) 
ff=(nil)

Note: The virsh live migration works fine with nfs storage from source to 
destination and vice versa.
With libvirt 1.0.5 and qemu 1.5 also we were facing the same issue, but with 
that even "Live migration with nfs also was not working".

Guest XML:
----------------

<domain type='kvm'>
  <name>VM1</name>
  <uuid>47feb0e1-0c23-9be9-da12-2ead34864de2</uuid>
  <memory unit='KiB'>4096000</memory>
  <currentMemory unit='KiB'>2048000</currentMemory>
  <vcpu placement='auto'>1</vcpu>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-1.5'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/home/images/VM1.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x2'/>
    </controller>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x1'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='network'>
      <mac address='52:54:00:9d:cf:bb'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' 
function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='selinux'/>
</domain>

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/2

------------------------------------------------------------------------
On 2013-06-28T13:27:20+00:00 Michal wrote:

Can you provide the destination debug logs? Esp. content of
/var/log/libvirt/qemu/VM1.log as there's supposed to be the reason why
qemu died.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/3

------------------------------------------------------------------------
On 2013-06-28T13:59:18+00:00 chandrashekar wrote:

Created attachment 766578
Source Logs

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/4

------------------------------------------------------------------------
On 2013-06-28T13:59:51+00:00 chandrashekar wrote:

Created attachment 766579
Destination Logs

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/5

------------------------------------------------------------------------
On 2013-06-28T14:20:28+00:00 Michal wrote:

>From the VM1_dest.log:

...
Completed 97 %
Completed 98 %
Completed 99 %
qemu: warning: error while loading state section id 1
load of migration failed

So the qemu is unable to migrate itself. Therefore I think this is actually a 
qemu bug.
On the other hand, I wonder why the guest on the source is shut down. There's 
no sign of that in the logs.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/6

------------------------------------------------------------------------
On 2013-07-01T10:02:08+00:00 chandrashekar wrote:

When the migration fails the guest gets shutdown on the source.

address@hidden images]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 5     VM1                     running

address@hidden images]# virsh migrate --live rhel64-64 qemu+ssh:/IP/system 
tcp://IP --verbose --copy-storage-all
 
Migration: [ 93 %]error: Unable to read from monitor: Connection reset by peer


At the destination:

address@hidden images]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     VM1                      shut off

address@hidden virsh list --all
 Id    Name                           State
----------------------------------------------------
 16    VM1                      paused

address@hidden virsh list --all
 Id    Name                           State
----------------------------------------------------

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/8

------------------------------------------------------------------------
On 2013-07-01T10:02:48+00:00 chandrashekar wrote:

Attached the SOS report of the Source and Destination machines.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/9

------------------------------------------------------------------------
On 2013-07-01T10:05:36+00:00 chandrashekar wrote:

Created attachment 767310
source sos

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/10

------------------------------------------------------------------------
On 2013-07-01T10:14:21+00:00 chandrashekar wrote:

Created attachment 767311
Destination sos

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/12

------------------------------------------------------------------------
On 2013-07-01T10:29:44+00:00 Michal wrote:

Unfortunately, the libvirtd.log is missing. I've written some guide as
well:

http://wiki.libvirt.org/page/DebugLogs

Please set the correct debug logs, reproduce and attach the new reports.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/15

------------------------------------------------------------------------
On 2013-07-01T11:34:34+00:00 chandrashekar wrote:

Attached libvirtd logs of both Source and Destination

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/16

------------------------------------------------------------------------
On 2013-07-01T11:36:13+00:00 chandrashekar wrote:

Created attachment 767340
source libvirtd log

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/17

------------------------------------------------------------------------
On 2013-07-01T11:38:53+00:00 chandrashekar wrote:

Created attachment 767342
Destination libvird logs

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/18

------------------------------------------------------------------------
On 2013-07-01T12:19:24+00:00 Michal wrote:

>From the *source* libvirtd log:

2013-07-01 11:30:29.582+0000: 3164: debug : virObjectRef:297 : OBJECT_REF: 
obj=0x7fe97000cb00
2013-07-01 11:30:29.582+0000: 3164: error : qemuMonitorIORead:511 : Unable to 
read from monitor: Connection reset by peer
2013-07-01 11:30:29.582+0000: 3164: debug : qemuMonitorIO:644 : Error on 
monitor Unable to read from monitor: Connection reset by peer
2013-07-01 11:30:29.582+0000: 3164: debug : virObjectUnref:260 : OBJECT_UNREF: 
obj=0x7fe97000cb00
2013-07-01 11:30:29.582+0000: 3165: debug : qemuMonitorSend:905 : Send command 
resulted in error Unable to read from monitor: Connection reset by peer
2013-07-01 11:30:29.582+0000: 3164: debug : qemuMonitorIO:678 : Triggering 
error callback
2013-07-01 11:30:29.582+0000: 3164: debug : qemuProcessHandleMonitorError:351 : 
Received error on 0x7fe9881337f0 'rhel64-64'

This means, qemu died unexpectedly. The qemu error message should be in
/var/log/libvirt/qemu/rhel64-64.log on the source.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/19

------------------------------------------------------------------------
On 2014-12-09T16:04:47+00:00 Michal wrote:

Since this is a qemu bug (probably fixed already) I'm closing this one.
If you disagree please reopen.

Reply at:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1192499/comments/32


** Changed in: libvirt (Fedora)
       Status: Unknown => Won't Fix

** Changed in: libvirt (Fedora)
   Importance: Unknown => High

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1192499

Title:
  virsh migration copy-storage-all  fails with "Unable to read from
  monitor: Connection reset by peer"

Status in QEMU:
  Won't Fix
Status in libvirt package in Ubuntu:
  Invalid
Status in libvirt package in Fedora:
  Won't Fix

Bug description:
  virsh migration copy-storage-all  fails with "Unable to read from
  monitor: Connection reset by peer" and shut downs the guest on the
  source host.

  Kernel Version:  3.10.0-rc5+

  Libvirt Version: 1.0.6

  Qemu Version: 1.5.50

  Steps to reproduce the issue:
  ----------------------------------------
  1. Created the qemu-img create -f qcow2 vm.qcow2 11G on the destination host 
which is same as the source.
  2. Started the guest on the source
  3. Started the vncdisplay to monitor the guest
  4. Initiated the migration "virsh migrate --live VM1 
qemu+ssh://host-ip/system tcp://host-ip --verbose --copy-storage-all"
  5. It started the copying the storage from souce to destination (conitinously 
monitored it was growing)
  6. Guest on the destination was paused and was running on the source
  7. At some point the VM on the source got shutdown and migration failed with 
"Unable to read from monitor: Connection reset by peer"

  Attached the libvirt debug logs.

  The debug logs shows :

  2013-06-19 08:43:12.253+0000: 4026: debug : virEventPollInterruptLocked:716 : 
Interrupting
  2013-06-19 08:43:12.253+0000: 4026: debug : virEventPollAddTimeout:248 : 
EVENT_POLL_ADD_TIMEOUT: timer=1 frequency=0 cb=0x7fe930baa960 opaque=(nil) 
ff=(nil)

  Note: The virsh live migration works fine with nfs storage from source to 
destination and vice versa.
  With libvirt 1.0.5 and qemu 1.5 also we were facing the same issue, but with 
that even "Live migration with nfs also was not working".

  Guest XML:
  ----------------

  <domain type='kvm'>
    <name>VM1</name>
    <uuid>47feb0e1-0c23-9be9-da12-2ead34864de2</uuid>
    <memory unit='KiB'>4096000</memory>
    <currentMemory unit='KiB'>2048000</currentMemory>
    <vcpu placement='auto'>1</vcpu>
    <numatune>
      <memory mode='strict' nodeset='0'/>
    </numatune>
    <os>
      <type arch='x86_64' machine='pc-i440fx-1.5'>hvm</type>
      <boot dev='hd'/>
    </os>
    <features>
      <acpi/>
      <apic/>
      <pae/>
    </features>
    <clock offset='utc'/>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>restart</on_crash>
    <devices>
      <emulator>/usr/local/bin/qemu-system-x86_64</emulator>
      <disk type='file' device='disk'>
        <driver name='qemu' type='qcow2' cache='none'/>
        <source file='/home/images/VM1.qcow2'/>
        <target dev='hda' bus='ide'/>
        <address type='drive' controller='0' bus='0' target='0' unit='0'/>
      </disk>
      <disk type='block' device='cdrom'>
        <driver name='qemu' type='raw'/>
        <target dev='hdc' bus='ide'/>
        <readonly/>
        <address type='drive' controller='0' bus='1' target='0' unit='0'/>
      </disk>
      <controller type='usb' index='0'>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x2'/>
      </controller>
      <controller type='ide' index='0'>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x1'/>
      </controller>
      <controller type='pci' index='0' model='pci-root'/>
      <interface type='network'>
        <mac address='52:54:00:9d:cf:bb'/>
        <source network='default'/>
        <model type='rtl8139'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x0'/>
      </interface>
      <serial type='pty'>
        <target port='0'/>
      </serial>
      <console type='pty'>
        <target type='serial' port='0'/>
      </console>
      <input type='mouse' bus='ps2'/>
      <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'>
        <listen type='address' address='127.0.0.1'/>
      </graphics>
      <video>
        <model type='cirrus' vram='9216' heads='1'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x0'/>
      </video>
      <memballoon model='virtio'>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x05' 
function='0x0'/>
      </memballoon>
    </devices>
    <seclabel type='none' model='selinux'/>
  </domain>

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1192499/+subscriptions