Re: [PATCH 0/5] vfio/pci: Fix up breakage against split irqchip and INTx

[Auger Eric]

Hi Peter,

On 2/26/20 11:50 PM, Peter Xu wrote:
> VFIO INTx is not working with split irqchip.  On new kernels KVM_IRQFD
> will directly fail with resamplefd attached so QEMU will automatically
> fallback to the INTx slow path.  However on old kernels it's still
> broken.
I think you should quote the commit that changes the behavior:
654f1f13ea56  kvm: Check irqchip mode before assign irqfd?
so that one can understand what it fixed.

So on kernels that do not feature that commit, the ioctl succeeds and we
pretend the resamplefd was properly attached whereas in practice it was
never triggered.

Thanks

Eric

> 
> Only until recently I noticed that this could also break PXE boot for
> assigned NICs [1].  My wild guess is that the PXE ROM will be mostly
> using INTx as well, which means we can't bypass that even if we
> enables MSI for the guest kernel.
> 
> This series tries to first fix this issue function-wise, then speed up
> for the INTx again with resamplefd (mostly following the ideas
> proposed by Paolo one year ago [2]).  My TCP_RR test shows that:

> 
>   - Before this series: this is broken, no number to show
> 
>   - After patch 1 (enable slow path): get 63% perf comparing to full
>     kernel irqchip
> 
>   - After whole series (enable fast path partly, irq injection will be
>     the same as fast path, however userspace needs to intercept for
>     EOI broadcast to resamplefd, though should still be faster than
>     the MMIO trick for intx eoi): get 93% perf comparing to full
>     kernel irqchip, which is a 46% performance boost
> 
> I think we can consider to apply patch 1 even sooner than the rest of
> the series to unbreak intx+split first.
> 
> The whole test matrix for reference:
> 
>   
> |----------+---------+-------------------+--------------+--------------------|
>   | IRQ type | irqchip | TCP_STREAM (Gbps) | TCP_RR (pps) | note              
>  |
>   
> |----------+---------+-------------------+--------------+--------------------|
>   | msi      | on      |              9.39 |        17567 |                   
>  |
>   | nomsi    | on      |              9.29 |        14056 |                   
>  |
>   | msi      | split   |              9.36 |        17330 |                   
>  |
>   | nomsi    | split   |                 / |            / | currently broken  
>  |
>   | nomsi    | split   |              8.98 |         8977 | after patch 1     
>  |
>   | nomsi    | split   |              9.21 |        13142 | after whole 
> series |
>   
> |----------+---------+-------------------+--------------+--------------------|
> 
> Any review comment is welcomed.  Thanks,
> 
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1786404
> [2] https://patchwork.kernel.org/patch/10738541/#22609933
> 
> Peter Xu (5):
>   vfio/pci: Disable INTx fast path if using split irqchip
>   vfio/pci: Use kvm_irqchip_add_irqfd_notifier_gsi() for irqfds
>   KVM: Pass EventNotifier into kvm_irqchip_assign_irqfd
>   KVM: Kick resamplefd for split kernel irqchip
>   Revert "vfio/pci: Disable INTx fast path if using split irqchip"
> 
>  accel/kvm/kvm-all.c    | 83 +++++++++++++++++++++++++++++++++++++++---
>  accel/kvm/trace-events |  1 +
>  hw/intc/ioapic.c       | 11 +++++-
>  hw/vfio/pci.c          | 37 ++++++++-----------
>  include/sysemu/kvm.h   |  4 ++
>  5 files changed, 106 insertions(+), 30 deletions(-)
>