[Qemu-discuss] PCI passthrough error with IGB multiport ethernet

[Jan Schermer]

Hi,
I’m trying to passthrough two dual-port cards to a guest.
It seems to work fine until I actually the cards, then I get:

[  306.667890] pcieport 0000:00:03.0: AER: Multiple Uncorrected (Fatal) error 
received: id=0000
[  306.667904] vfio-pci 0000:03:00.0: PCIe Bus Error: severity=Uncorrected 
(Fatal), type=Unaccessible, id=0300(Unregistered Agent ID)
[  306.667998] vfio-pci 0000:03:00.1: PCIe Bus Error: severity=Uncorrected 
(Fatal), type=Unaccessible, id=0301(Unregistered Agent ID)
[  306.668102] vfio-pci 0000:03:00.0: broadcast error_detected message
[  307.710345] pcieport 0000:00:03.0: Root Port link has been reset
[  307.710352] vfio-pci 0000:03:00.0: broadcast mmio_enabled message
[  307.710355] vfio-pci 0000:03:00.0: broadcast resume message
[  307.710357] vfio-pci 0000:03:00.0: AER: Device recovery successful
[  307.710360] vfio-pci 0000:03:00.1: broadcast error_detected message
[  308.734333] pcieport 0000:00:03.0: Root Port link has been reset
[  308.734338] vfio-pci 0000:03:00.1: broadcast mmio_enabled message
[  308.734340] vfio-pci 0000:03:00.1: broadcast resume message
[  308.734342] vfio-pci 0000:03:00.1: AER: Device recovery successful

I swapped one card already in case it was faulty, now I got the same error on 
the second one (above).

This is the configuration:

   <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' 
multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' 
function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' 
multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' 
function='0x1'/>
    </hostdev>

Am I passing those devices correctly or can I somehow pass the “whole” device? 

When I kill the guest the host sometimes recovers and sees both cards (all 4 
ports), sometimes it doesn’t.

I found this 
http://events.linuxfoundation.org/sites/events/files/slides/AER%20functionality%20of%20pass-through%20PCI-e%20device%20in%20Qemu.pdf
 
<http://events.linuxfoundation.org/sites/events/files/slides/AER%20functionality%20of%20pass-through%20PCI-e%20device%20in%20Qemu.pdf>
 which didn’t exactly make me happy, but I’m already using a somewhat similiar 
setup on a different machine where it just works. (there I’m passing one 
function to VM1 and another to VM2, which I’m not sure now should even work? 
:)).

I tried options vfio-pci disable_idle_d3=1 nointxmask=1 but that didn’t help.

Any ideas how to get it working? Is this a HW issue? A platform problem? 

Hypervisor is Ubuntu 17.10, guest is Ubuntu 17.10 as well (kernel 4.13.0, qemu 
2.10)

Thanks

Jan