Re: [Qemu-discuss] How to resolve "Failed to mmap" error?

[Alex Williamson]

On Fri, 14 Oct 2016 10:56:44 -0600
Alex Williamson <address@hidden> wrote:

> On Fri, 14 Oct 2016 10:47:42 -0600
> Alex Williamson <address@hidden> wrote:
> 
> > On Mon, 10 Oct 2016 20:05:02 -0700
> > A223 A223 <address@hidden> wrote:
> >   
> > > I just tried testing the patch.
> > > 
> > > Initially, it seemed to work fine (log messages below).
> > > However, I am getting hard lockup of the host machine shortly after
> > > the vfio_region_setup related log lines print out to the screen. I
> > > would say that rougly 3 of the 9 VM startups resulted in a hard lock
> > > like this. I wasn't getting these hard locks before applying the patch
> > > that I can remember.
> > > 
> > > I removed the patch and tried to replicate the hard lock and haven't
> > > been able to, though admittedly I was only able to try a few times.
> > > Unfortunately, repeatedly trying to boot the VM to test things starts
> > > to become time consuming, because once the VM has been through a
> > > single startup-shutdown cycle, qemu refuses to start, printing these
> > > errors:
> > > qemu-system-x86_64: -device
> > > vfio-pci,host=05:00.0,bus=root.1,addr=00.5: vfio: Error: Failed to
> > > setup INTx fd: Device or resource busy
> > > qemu-system-x86_64: -device
> > > vfio-pci,host=05:00.0,bus=root.1,addr=00.5: Device initialization
> > > failed
> > > 
> > > In the host's kernel log, there are a ton of these
> > > [  774.069113] vfio_ecap_init: 0000:05:00.0 hiding ecap address@hidden
> > > Followed by these kernel log errors:
> > > [  774.070330] genirq: Flags mismatch irq 16. 00000000
> > > (vfio-intx(0000:05:00.0)) vs. 00000080 (ehci_hcd:usb1)
> > > [  774.085595] vfio-pci 0000:05:00.0: Refused to change power state,
> > > currently in D3
> > > [  774.797671] vfio-pci 0000:05:00.0: timed out waiting for pending
> > > transaction; performing function level reset anyway
> > > [  775.945685] vfio-pci 0000:05:00.0: Failed to return from FLR
> > > 
> > > This is not a problem with your patch, but it does complicate my
> > > ability to test the patch repeatedly, since a host restart is needed
> > > between every try. If you have any idea what could be going there,
> > > please do let me know.    
> > 
> > Does this kernel patch help?
> > 
> > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > index 5bb5609..0e48631 100644
> > --- a/drivers/pci/quirks.c
> > +++ b/drivers/pci/quirks.c
> > @@ -3198,6 +3198,8 @@ static void quirk_no_bus_reset(struct pci_dev *dev)
> >  DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0030, 
> > quirk_no_bus_reset);
> >  DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0032, 
> > quirk_no_bus_reset);
> >  DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x003c, 
> > quirk_no_bus_reset);
> > +/* Fresco Logic FL1100 USB 3.0 Host Controller */
> > +DECLARE_PCI_FIXUP_HEADER(0x1b73, 0x1100, quirk_no_bus_reset);  
> 
> Please also verify with 'lspci -nns 5:' before trying this that
> 1b73:1100 matches your device.

I picked up a card based on the same Fresco FL1100 (rev 10) chip, I
don't think it needs the no-bus-reset quirk above, resets seem to work
fine for me.  In fact the card works as expected with the previous
patch I sent, no warning about mmaps.  There's no performance loss due
to the mapping of the MSI-X table BAR with or without that patch, that
BAR seems to only be used for MSI-X and nothing changes once setup, at
least with modern versions of Linux as the guest.  Tracing shows
absolutely no traps through QEMU once it's setup.  The device does run
at a high interrupt rate, which I expect is the primary source of
performance loss when running in a VM.  Hardware solutions like APICv
and Posted Interrupts should close that gap.

What you're seeing above still suggests to me that the device is not
returning from a bus reset, all reads from the device are returning
-1.  Since I'm unable to reproduce with my card, it would seem to imply
that the problem is not an intrinsic issue with the controller chip
itself.  Are you overclocking in any way?  Thanks,

Alex