[Qemu-devel] Possible bug in pci slot management, migration abort after hotplug/unplug pci devices

[Gonglei (Arei)]

Hi,

I find a critical problem about pci devices migration 
when we configure a pci device not assign the devfn. Qemu will 
auto assign the devfn in function do_pci_register_device.

At the migration src end, When we hotplug two pci devices, such as virtio-net. 
Qemu will assign devfn 4 (assumed the previous 3 slots and the fifth have been 
assigned)
for the first nic, and assign devfn 6 for the second nic.
After that, I unplug the first nic. 

At the migrate dst end, I start the vm adding a virtio-net nic. 
Try to execute migration, qemu report "Unknown ramblock 
"0000:00:06.0/virtio-net-pci.rom", cannot accept migration.
qemu: warning: error while loading state for instance 0x0 of device 'ram'
load of migration failed "

How reproducible:
100%

Steps to Reproduce:
1. at src end, boot guset with:
/usr/bin/qemu-kvm -enable-kvm -m 512 -smp 2 -name suse11sp3 -boot c -drive 
file=/home/sles11sp3.img,if=none,id=drive-virtio-disk0,format=raw,\
cache=none,aio=native -device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
 -vnc 0.0.0.0:10 \
-monitor stdio -netdev type=tap,id=net0 -device 
virtio-net,id=nic0,netdev=net0,mac=52:54:00:12:34:57

(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc060 [0xc06f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1013:00b8
      BAR0: 32 bit prefetchable memory at 0xfc000000 [0xfdffffff].
      BAR1: 32 bit memory at 0xfebd0000 [0xfebd0fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 11.
      BAR0: I/O at 0xc040 [0xc05f].
      BAR1: 32 bit memory at 0xfebd1000 [0xfebd1fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "nic0"
  Bus  0, device   5, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 10.
      BAR0: I/O at 0xc000 [0xc03f].
      BAR1: 32 bit memory at 0xfebd2000 [0xfebd2fff].
      id "virtio-disk0"
(qemu) info network
nic0: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:12:34:57
 \ net0: 
index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown

2.hotplug two nics:
(qemu) netdev_add user,id=net1
(qemu) device_add virtio-net-pci,netdev=net1,id=nic1,mac=52:54:00:12:34:58
(qemu) netdev_add user,id=net2
(qemu) device_add virtio-net-pci,netdev=net2,id=nic2,mac=52:54:00:12:34:59
(qemu) info network
nic0: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:12:34:57
 \ net0: 
index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
nic1: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:12:34:58
 \ net1: index=0,type=user,net=10.0.2.0,restrict=off
nic2: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:12:34:59
 \ net2: index=0,type=user,net=10.0.2.0,restrict=off

3. unplug the first plugged nic:
(qemu) device_del nic1
(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc060 [0xc06f].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1013:00b8
      BAR0: 32 bit prefetchable memory at 0xfc000000 [0xfdffffff].
      BAR1: 32 bit memory at 0xfebd0000 [0xfebd0fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 11.
      BAR0: I/O at 0xc040 [0xc05f].
      BAR1: 32 bit memory at 0xfebd1000 [0xfebd1fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "nic0"
  Bus  0, device   5, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 10.
      BAR0: I/O at 0xc000 [0xc03f].
      BAR1: 32 bit memory at 0xfebd2000 [0xfebd2fff].
      id "virtio-disk0"
  Bus  0, device   6, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1020 [0x103f].
      BAR1: 32 bit memory at 0x80041000 [0x80041fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "nic2"

4. at the dest end, boot guset with:
/usr/bin/qemu-kvm -enable-kvm -m 512 -smp 2 -name suse11sp3 -boot c -drive 
file=/home/sles11sp3.img,if=none,id=drive-virtio-disk0, \
format=raw,cache=none,aio=native -device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,\
bootindex=1 -vnc 0.0.0.0:10 -monitor stdio -netdev type=tap,id=net0 -device 
virtio-net,id=nic0,netdev=net0,mac=52:54:00:12:34:57 \
-netdev type=tap,id=net2 -device 
virtio-net,id=nic2,netdev=net2,mac=52:54:00:12:34:59 -incoming tcp:0:8888
(qemu) info pci
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xffffffffffffffff [0x000e].
      id ""
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 0.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1013:00b8
      BAR0: 32 bit prefetchable memory at 0xffffffffffffffff [0x01fffffe].
      BAR1: 32 bit memory at 0xffffffffffffffff [0x00000ffe].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   3, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0xffffffffffffffff [0x003e].
      BAR1: 32 bit memory at 0xffffffffffffffff [0x00000ffe].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "nic0"
  Bus  0, device   4, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0xffffffffffffffff [0x003e].
      BAR1: 32 bit memory at 0xffffffffffffffff [0x00000ffe].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "nic2"
  Bus  0, device   5, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 0.
      BAR0: I/O at 0xffffffffffffffff [0x003e].
      BAR1: 32 bit memory at 0xffffffffffffffff [0x00000ffe].
      id "virtio-disk0"
(qemu) info network
nic0: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:12:34:57
 \ net0: 
index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
nic2: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:12:34:59
 \ net2: 
index=0,type=tap,ifname=tap1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown

5. migrate at the src end:
(qemu) migrate -d tcp:9.61.1.16:8888

Actual results:

At the dest end, qemu report "Unknown ramblock 
"0000:00:06.0/virtio-net-pci.rom", cannot accept migration
qemu: warning: error while loading state for instance 0x0 of device 'ram'
load of migration failed" and abort.


Best regards,
-Gonglei