XIVE VFIO kernel resample failure in INTx mode under heavy load

[Timothy Pearson]

All,

I've been struggling for some time with what is looking like a potential bug in 
QEMU/KVM on the POWER9 platform.  It appears that in XIVE mode, when the 
in-kernel IRQ chip is enabled, an external device that rapidly asserts IRQs via 
the legacy INTx level mechanism will only receive one interrupt in the KVM 
guest.

Changing any one of those items appears to avoid the glitch, e.g. XICS mode 
with the in-kernel IRQ chip works (all interrupts are passed through), and XIVE 
mode with the in-kernel IRQ chip disabled also works.  We are also not seeing 
any problems in XIVE mode with the in-kernel chip from MSI/MSI-X devices.

The device in question is a real time card that needs to raise an interrupt 
every 1ms.  It works perfectly on the host, but fails in the guest -- with the 
in-kernel IRQ chip and XIVE enabled, it receives exactly one interrupt, at 
which point the host continues to see INTx+ but the guest sees INTX-, and the 
IRQ handler in the guest kernel is never reentered.

We have also seen some very rare glitches where, over a long period of time, we 
can enter a similar deadlock in XICS mode.  Disabling the in-kernel IRQ chip in 
XIVE mode will also lead to the lockup with this device, since the userspace 
IRQ emulation cannot keep up with the rapid interrupt firing (measurements show 
around 100ms required for processing each interrupt in the user mode).

My understanding is the resample mechanism does some clever tricks with level 
IRQs, but that QEMU needs to check if the IRQ is still asserted by the device 
on guest EOI.  Since a failure here would explain these symptoms I'm wondering 
if there is a bug in either QEMU or KVM for POWER / pSeries (SPAPr) where the 
IRQ is not resampled and therefore not re-fired in the guest?

Unfortunately I lack the resources at the moment to dig through the QEMU 
codebase and try to find the bug.  Any IBMers here that might be able to help 
out?  I can provide access to a test setup if desired.

Thanks!