Re: [Qemu-devel] [question] PIC and APIC

[Paolo Bonzini]

Il 05/09/2014 17:51, Richard Bilson ha scritto:
> Greetings folks,
> 
> I've been tracking down certain obscure failures that we see running our
> kernel on qemu x86 smp. Based on a few days diving into qemu internals it
> seems that the problems relate to mixing PIC and APIC-generated
> interrupts. I'd appreciate some help in understanding if this is a
> limitation of qemu, or if we are misconfiguring the system in some way.
> 
> Scenario:
> - All hardware interrupts come via the PIC, and are delivered by the cpu 0
> LAPIC in ExtINT mode
> - IPIs are delivered by the LAPIC in fixed mode
> 
> Failure mode #1:
> - IPI sent to cpu 0 (bit set in APIC irr)
> - IPI accepted by cpu 0 (bit cleared in irr, set in isr)
> - IPI sent to cpu 0 (bit set in both irr and isr)
> - PIC interrupt sent to cpu 0
> 
> The PIC interrupt causes CPU_INTERRUPT_HARD to be set, but
> apic_irq_pending observes that the highest pending APIC interrupt priority
> (the IPI) is the same as the processor priority (since the IPI is still
> being handled), so apic_get_interrupt returns a spurious interrupt rather
> than the pending PIC interrupt. The result is an endless sequence of
> spurious interrupts, since nothing will clear CPU_INTERRUPT_HARD.

ExtINT interrupts should have ignored the processor priority, right?

> Failure mode #2:
> - cpu 0 masks a particular PIC interrupt
> - IPI sent to cpu 0 (CPU_INTERRUPT_HARD is set)
> - before the IPI is accepted, the masked interrupt line is asserted by the
> device
> 
> Since the interrupt is masked, apic_deliver_pic_intr will clear
> CPU_INTERRUPT_HARD. The IPI will still be set in the APIC irr, but since
> CPU_INTERRUPT_HARD is not set the cpu will not notice. Depending on the
> scenario this can cause a system hang, i.e. if cpu 0 is expected to unmask
> the interrupt.
> 
> The commonality here seems to be that the APIC code is attempting to use
> the CPU_INTERRUPT_HARD state for both PIC and APIC interrupts, and one of
> these purposes blocks the other. It seems that we should be able to solve
> the problem on our side by using a pure-APIC interrupt routing (and indeed
> we haven't observed these problems when so configured), but we've never
> noticed these kinds of problems on real hardware using legacy PIC mode.
> 
> Any help in confirming or refuting the above analysis is appreciated.

It sounds plausible.  Can you confirm that you are using TCG
(emulation)?  Does it work with KVM?  What about KVM with -machine
kernel_irqchip=off?

It would be great if you could reproduce the issue using the harness at
http://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git (it works just
as well without KVM).

Paolo