Re: [Qemu-devel] [Bug 1091766] Re: Physical host crash with Mellanox IB PCI passthrough

[Vlastimil Holer]

Both patches against qemu 1.3 *works*, first with traditional PCI
pass., second with VFIO. Mellanox IB card in guest works fine again.
Great early Christmas present, thank you!

Just FYI: between both ways I can see little difference on host system
regarding the numbers of IRQs for MSI/MSI-X:

* VFIO:
| vfio_ecap_init: 0000:20:00.0 hiding ecap address@hidden
| vfio-pci 0000:20:00.0: irq 150 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 150 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 151 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 150 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 151 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 152 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 150 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 151 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 152 for MSI/MSI-X
| vfio-pci 0000:20:00.0: irq 153 for MSI/MSI-X

* old way:
| assign device 0:20:0.0
| pci-stub 0000:20:00.0: restoring config space at offset 0xf (was
0x100, writing 0x10a)
| pci-stub 0000:20:00.0: restoring config space at offset 0x6 (was
0xc, writing 0xdf00000c)
| pci-stub 0000:20:00.0: restoring config space at offset 0x4 (was
0x4, writing 0x91b00004)
| pci-stub 0000:20:00.0: restoring config space at offset 0x3 (was
0x0, writing 0x10)
| pci-stub 0000:20:00.0: restoring config space at offset 0x1 (was
0x100000, writing 0x100042)
| pci-stub 0000:20:00.0: irq 134 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 135 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 136 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 137 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 138 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 139 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 140 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 141 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 142 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 143 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 144 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 145 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 146 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 147 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 148 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 149 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 150 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 151 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 152 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 153 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 154 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 155 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 156 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 157 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 158 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 159 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 160 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 161 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 162 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 163 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 134 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 135 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 136 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 137 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 138 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 139 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 140 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 141 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 142 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 143 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 144 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 145 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 146 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 147 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 148 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 149 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 150 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 151 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 152 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 153 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 154 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 155 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 156 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 157 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 158 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 159 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 160 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 161 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 162 for MSI/MSI-X
| pci-stub 0000:20:00.0: irq 163 for MSI/MSI-X

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1091766

Title:
  Physical host crash with Mellanox IB PCI passthrough

Status in QEMU:
  New

Bug description:
  (from
  http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/100736)

  We have been using PCI passthrough with the Mellanox IB interface
  (MT27500 Family [ConnectX-3]) on Debian 6.0.6, kernel 3.2.23 and
  qemu-kvm-1.0 (both from backports). It worked fine until latest
  update in backports to qemu-kvm-1.1.2. With newer qemu-kvm versions
  IB device probe in guest fails leaving firmware to kill whole physical 
machine.

  I have then compiled qemu-kvm from source, 1.0.1 was OK, 1.1.2 fails and
  even 1.2.0 fails as well. Our setup is based on IBM System X iDataPlex
  dx360 M4 Server.

  Note: Now I have also tested latest qemu-1.3.0 with linux 3.7.1 and
  new VFIO mechanism and behaves the same way.

  On guest the mlx4_core fails to probe device:
  | mlx4_core 0000:00:08.0: irq 74 for MSI/MSI-X
  | mlx4_core 0000:00:08.0: irq 75 for MSI/MSI-X
  | mlx4_core 0000:00:08.0: irq 76 for MSI/MSI-X
  | mlx4_core 0000:00:08.0: irq 77 for MSI/MSI-X
  | mlx4_core 0000:00:08.0: NOP command failed to generate MSI-X interrupt IRQ 
51).
  | mlx4_core 0000:00:08.0: Trying again without MSI-X.
  | mlx4_core 0000:00:08.0: NOP command failed to generate interrupt (IRQ 51), 
aborting.
  | mlx4_core 0000:00:08.0: BIOS or ACPI interrupt routing problem?
  | mlx4_core 0000:00:08.0: PCI INT A disabled
  | mlx4_core: probe of 0000:00:08.0 failed with error -16

  Which immediately results in reset of the whole physical machine:
  | Uhhuh. NMI received for unknown reason 3d on CPU 0.
  | Do you have a strange power saving mode enabled?
  | Dazed and confused, but trying to continue

  Followed by events in hardware management module:
  | A software NMI has occurred on system "SN# xxxxxxx"
  | Fault in slot "All PCI Err" on system "SN# xxxxxxx"
  | Fault in slot "PCI 1" on system "SN# xxxxxxx"
  | A Uncorrectable Bus Error has occurred on system "SN# xxxxxxx"
  | "Host Power" has been Power Cycled
  | System "SN# xxxxxxx" has recovered from an NMI

  Kernel logs for both host/guest machines and different qemu-kvm
  versions are attached. PCI passthrough for e.g. Intel e1000 works
  fine with all tested qemu-kvm versions.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1091766/+subscriptions