Re: [Qemu-devel] [question] PIC and APIC

[Richard Bilson]

On 2014-09-05, 13:03, "Paolo Bonzini" <address@hidden> wrote:
>Il 05/09/2014 17:51, Richard Bilson ha scritto:
>> Greetings folks,
>> 
>> I've been tracking down certain obscure failures that we see running our
>> kernel on qemu x86 smp. Based on a few days diving into qemu internals
>>it
>> seems that the problems relate to mixing PIC and APIC-generated
>> interrupts. I'd appreciate some help in understanding if this is a
>> limitation of qemu, or if we are misconfiguring the system in some way.
>> 
>> Scenario:
>> - All hardware interrupts come via the PIC, and are delivered by the
>>cpu 0
>> LAPIC in ExtINT mode
>> - IPIs are delivered by the LAPIC in fixed mode
>> 
>> Failure mode #1:
>> - IPI sent to cpu 0 (bit set in APIC irr)
>> - IPI accepted by cpu 0 (bit cleared in irr, set in isr)
>> - IPI sent to cpu 0 (bit set in both irr and isr)
>> - PIC interrupt sent to cpu 0
>> 
>> The PIC interrupt causes CPU_INTERRUPT_HARD to be set, but
>> apic_irq_pending observes that the highest pending APIC interrupt
>>priority
>> (the IPI) is the same as the processor priority (since the IPI is still
>> being handled), so apic_get_interrupt returns a spurious interrupt
>>rather
>> than the pending PIC interrupt. The result is an endless sequence of
>> spurious interrupts, since nothing will clear CPU_INTERRUPT_HARD.
>
>ExtINT interrupts should have ignored the processor priority, right?

That's my understanding, yes.

>> Failure mode #2:
>> - cpu 0 masks a particular PIC interrupt
>> - IPI sent to cpu 0 (CPU_INTERRUPT_HARD is set)
>> - before the IPI is accepted, the masked interrupt line is asserted by
>>the
>> device
>> 
>> Since the interrupt is masked, apic_deliver_pic_intr will clear
>> CPU_INTERRUPT_HARD. The IPI will still be set in the APIC irr, but since
>> CPU_INTERRUPT_HARD is not set the cpu will not notice. Depending on the
>> scenario this can cause a system hang, i.e. if cpu 0 is expected to
>>unmask
>> the interrupt.
>> 
>> The commonality here seems to be that the APIC code is attempting to use
>> the CPU_INTERRUPT_HARD state for both PIC and APIC interrupts, and one
>>of
>> these purposes blocks the other. It seems that we should be able to
>>solve
>> the problem on our side by using a pure-APIC interrupt routing (and
>>indeed
>> we haven't observed these problems when so configured), but we've never
>> noticed these kinds of problems on real hardware using legacy PIC mode.
>> 
>> Any help in confirming or refuting the above analysis is appreciated.
>
>It sounds plausible.  Can you confirm that you are using TCG
>(emulation)?  Does it work with KVM?  What about KVM with -machine
>kernel_irqchip=off?

I am definitely using TCG. I will try the KVM variants asap.

>It would be great if you could reproduce the issue using the harness at
>http://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git (it works just
>as well without KVM).

I'll look into this as well.