[Qemu-devel] [question] PIC and APIC

[Richard Bilson]

Greetings folks,

I've been tracking down certain obscure failures that we see running our
kernel on qemu x86 smp. Based on a few days diving into qemu internals it
seems that the problems relate to mixing PIC and APIC-generated
interrupts. I'd appreciate some help in understanding if this is a
limitation of qemu, or if we are misconfiguring the system in some way.

Scenario:
- All hardware interrupts come via the PIC, and are delivered by the cpu 0
LAPIC in ExtINT mode
- IPIs are delivered by the LAPIC in fixed mode

Failure mode #1:
- IPI sent to cpu 0 (bit set in APIC irr)
- IPI accepted by cpu 0 (bit cleared in irr, set in isr)
- IPI sent to cpu 0 (bit set in both irr and isr)
- PIC interrupt sent to cpu 0

The PIC interrupt causes CPU_INTERRUPT_HARD to be set, but
apic_irq_pending observes that the highest pending APIC interrupt priority
(the IPI) is the same as the processor priority (since the IPI is still
being handled), so apic_get_interrupt returns a spurious interrupt rather
than the pending PIC interrupt. The result is an endless sequence of
spurious interrupts, since nothing will clear CPU_INTERRUPT_HARD.

Failure mode #2:
- cpu 0 masks a particular PIC interrupt
- IPI sent to cpu 0 (CPU_INTERRUPT_HARD is set)
- before the IPI is accepted, the masked interrupt line is asserted by the
device

Since the interrupt is masked, apic_deliver_pic_intr will clear
CPU_INTERRUPT_HARD. The IPI will still be set in the APIC irr, but since
CPU_INTERRUPT_HARD is not set the cpu will not notice. Depending on the
scenario this can cause a system hang, i.e. if cpu 0 is expected to unmask
the interrupt.

The commonality here seems to be that the APIC code is attempting to use
the CPU_INTERRUPT_HARD state for both PIC and APIC interrupts, and one of
these purposes blocks the other. It seems that we should be able to solve
the problem on our side by using a pure-APIC interrupt routing (and indeed
we haven't observed these problems when so configured), but we've never
noticed these kinds of problems on real hardware using legacy PIC mode.

Any help in confirming or refuting the above analysis is appreciated.

Thanks,
Richard Bilson
QNX Software Systems Ltd.