Re: [Qemu-devel] [Xen-devel] Xen hypervisor inside KVM guest with x2apic CPU feature fails to boot

[Andrew Cooper]

On 02/06/14 13:45, Daniel P. Berrange wrote:
> I'm running
>
>  kernel-3.14.4-200.fc20.x86_64
>  qemu-1.6.2-5.fc20.x86_64
>  xen-4.4.0-4.fc21
>
> In process of trying to get a Xen hypervisor running inside a KVM guest I
> found that there's a problem with x2apic. NB I do *not* use nested-VMX
> here, just trying to get plain Xen paravirt working before trying todo
> nested HVM.
>
> Any time I enable the 'x2apic' CPU flag for the KVM guest, the Xen hypervisor
> running inside the guest will fail to boot:
>
> The QEMU/KVM -cpu arg is
>
>   -cpu 
> core2duo,+erms,+smep,+fsgsbase,+lahf_lm,+rdtscp,+rdrand,+f16c,+avx,+osxsave,+xsave,+aes,+tsc-deadline,+popcnt,+x2apic,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds
>  
>
> The Xen logs indicate it isn't liking the x2apic feature and is disabling
> it, but then it obviously fails to setup the non-x2apic codepath it is
> following - even though the non-x2apic codepath works fine if you don't
> have +x2apic set for the KVM guest.
>
> (XEN) Not enabling x2APIC: depends on iommu_supports_eim.
> (XEN) XSM Framework v1.0.0 initialized
> (XEN) Flask:  Initializing.
> (XEN) AVC INITIALIZED
> (XEN) Flask:  Starting in permissive mode.
> (XEN) Using scheduler: SMP Credit Scheduler (credit)
> (XEN) Detected 2693.939 MHz processor.
> (XEN) Initing memory sharing.
> (XEN) traps.c:3071: GPF (0000): ffff82d0801b83c7 -> ffff82d08023386b
> (XEN) mce_intel.c:717: MCA Capability: BCAST 1 SER 1 CMCI 0 firstbank 1 
> extended MCE MSR 0
> (XEN) Intel machine check reporting enabled
> (XEN) I/O virtualisation disabled
> (XEN) Getting VERSION: 1050014
> (XEN) Getting VERSION: 1050014
> (XEN) Enabled directed EOI with ioapic_ack_old on!
> (XEN) Getting ID: 0
> (XEN) Getting LVT0: 8700
> (XEN) Getting LVT1: 8400
> (XEN) Suppress EOI broadcast on CPU#0
> (XEN) enabled ExtINT on CPU#0
> (XEN) ENABLING IO-APIC IRQs
> (XEN)  -> Using old ACK method
> (XEN) init IO_APIC IRQs
> (XEN)  IO-APIC (apicid-pin) 0-0, 0-16, 0-17, 0-18, 0-19, 0-20, 0-21, 0-22, 
> 0-23 not connected.
> (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
> (XEN) ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> (XEN) ...trying to set up timer (IRQ0) through the 8259A ...  failed.
> (XEN) ...trying to set up timer as Virtual Wire IRQ... failed.
> (XEN) ...trying to set up timer as ExtINT IRQ... failed :(.
> (XEN) 
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) IO-APIC + timer doesn't work!  Boot with apic_verbosity=debug and send 
> a report.  Then try booting with the 'noapic' option
> (XEN) ****************************************
>
> Will attach the full non-trimmed Xen log to this mail, along with a log
> showing successful boot when 'x2apic' isn't given to KVM.
>
> I'm unclear if this is a Xen bug or KVM bug or QEMU bug, or a combination
> of them....
>
> Regards,
> Daniel

So have just debugged a kexec issue which ends up looking like this
(except it was the kdump kernel failing to find its timer interrupts).

This set of errors at the end means that Xen couldn't configure any mode
where it could end up getting legacy timer interrupts.

I am going to go out on a limb and say it is unlikely for the Xen side
of things to be broken.  We have machines in XenServers test lab which
support x2apic but have it disabled in this manor because of a lack of
EIM (or due to errata).

I would look into what KVM/qemu does differently as a result of the
"+x2apic" flag, and whether it is simply the reported cpuid feature, or
whether it affects the legacy ioapic emulation.

~Andrew