Re: [Qemu-devel] [RFC PATCH V2 00/10] Qemu: Add live migration support for SRIOV NIC

[Wei Yang]

Hi, Tianyu,

I am testing your V2 patch set in our environment, while facing two issues
now. Have a workaround for the first one and hope you could share some light
on the second one :-)

1. Mismatch for ram_block (Have a workaround)
-------------------------------------------------------------------------------
Below is the error message on the destination:

    qemu-system-x86_64: Length mismatch: : 0x3000 in != 0x4000: Invalid argument
    qemu-system-x86_64: error while loading state for instance 0x0 of device 
'ram'
    qemu-system-x86_64: load of migration failed: Invalid argument

With the following command line on source and destination respectively:

    git/qemu/x86_64-softmmu/qemu-system-x86_64 --enable-kvm -m 4096 -smp 4 
--nographic -drive file=/root/nfs/rhel.img,format=raw,cache=none -device 
vfio-sriov,host=0000:03:10.0
    git/qemu/x86_64-softmmu/qemu-system-x86_64 --enable-kvm -m 4096 -smp 4 
--nographic -drive file=/root/nfs/rhel.img,format=raw,cache=none -device 
vfio-sriov,host=0000:03:10.0 --incoming tcp:0:4444

By some debugging, the reason for this error is the ram_block->idstr of
pass-through MMIO region is not set.

My workaround is to add vmstate_register_ram() in vfio_mmap_region() after
memory_region_init_ram_ptr() returns.

I think this is not a good solution, since the ram_block->idstr is coded
with the VF's BDF. So I guess this will not work when the VF has different BDF
from source to destination respectively.

Maybe my test step is not correct?

2. Failed to migrate the MAC address
-------------------------------------------------------------------------------

By adding some code in VF's driver in destination guest, I found the MAC
information has been migrated to destination in adapter->hw.mac. While this is
"reset" by VF's driver, when ixgbevf_migration_task is invoked at the end of
the migration process.

Below is what I have printed:

The ifconfig output from destination:

    eth8      Link encap:Ethernet  HWaddr 52:54:00:81:39:F2  
              inet addr:9.31.210.106  Bcast:9.31.255.255  Mask:255.255.0.0
              inet6 addr: fe80::5054:ff:fe81:39f2/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:66 errors:0 dropped:0 overruns:0 frame:0
              TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:21840 (21.3 KiB)  TX bytes:920 (920.0 b)

The log message I printed in destination's VF driver:
    
    ixgbevf: migration end -- 
    ixgbevf: original mac:52:54:00:81:39:f2
    ixgbevf: after reset mac:52:54:00:92:04:a3
    ixgbevf: migration end  ==
    
I didn't take a close look in the "reset" function, while seems it retrieves
the mac from VF hardware. Hmm... is there some possible way to have the same
mac on both source and destination?

At last, I appreciated all your work and help, learned much from your side.

On Tue, Nov 24, 2015 at 09:35:17PM +0800, Lan Tianyu wrote:
>This patchset is to propose a solution of adding live migration
>support for SRIOV NIC.
>
>During migration, Qemu needs to let VF driver in the VM to know
>migration start and end. Qemu adds faked PCI migration capability
>to help to sync status between two sides during migration.
>
>Qemu triggers VF's mailbox irq via sending MSIX msg when migration
>status is changed. VF driver tells Qemu its mailbox vector index
>via the new PCI capability. In some cases(NIC is suspended or closed),
>VF mailbox irq is freed and VF driver can disable irq injecting via
>new capability.   
>
>VF driver will put down nic before migration and put up again on
>the target machine.
>
>Lan Tianyu (10):
>  Qemu/VFIO: Create head file pci.h to share data struct.
>  Qemu/VFIO: Add new VFIO_GET_PCI_CAP_INFO ioctl cmd definition
>  Qemu/VFIO: Rework vfio_std_cap_max_size() function
>  Qemu/VFIO: Add vfio_find_free_cfg_reg() to find free PCI config space
>    regs
>  Qemu/VFIO: Expose PCI config space read/write and msix functions
>  Qemu/PCI: Add macros for faked PCI migration capability
>  Qemu: Add post_load_state() to run after restoring CPU state
>  Qemu: Add save_before_stop callback to run just before stopping VCPU
>    during migration
>  Qemu/VFIO: Add SRIOV VF migration support
>  Qemu/VFIO: Misc change for enable migration with VFIO
>
> hw/vfio/Makefile.objs       |   2 +-
> hw/vfio/pci.c               | 196 +++++++++-----------------------------------
> hw/vfio/pci.h               | 168 +++++++++++++++++++++++++++++++++++++
> hw/vfio/sriov.c             | 178 ++++++++++++++++++++++++++++++++++++++++
> include/hw/pci/pci_regs.h   |  19 +++++
> include/migration/vmstate.h |   5 ++
> include/sysemu/sysemu.h     |   1 +
> linux-headers/linux/vfio.h  |  16 ++++
> migration/migration.c       |   3 +-
> migration/savevm.c          |  28 +++++++
> 10 files changed, 459 insertions(+), 157 deletions(-)
> create mode 100644 hw/vfio/pci.h
> create mode 100644 hw/vfio/sriov.c
>
>-- 
>1.9.3
>
>

-- 
Richard Yang\nHelp you, Help me