[Qemu-devel] odd calls to apic_set_irq on x86_64 startup

[Max Filippov]

Hi.

During qemu-system-x86_64 startup there are odd calls to apic_set_irq
with vector_num=0.
I observe the following call sequence: PIT reset

Breakpoint 1, apic_set_irq (s=0x89e3e18, vector_num=0, trigger_mode=0)
at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/apic.c:424
424     {
(gdb) bt
#0  apic_set_irq (s=0x89e3e18, vector_num=0, trigger_mode=0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/apic.c:424
#1  0x081cd717 in apic_local_deliver (s=0x89e3e18, vector=3) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/apic.c:182
#2  0x081cd76f in apic_deliver_pic_intr (d=0x89e3e18, level=1) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/apic.c:191
#3  0x08285846 in pic_irq_request (opaque=0x0, irq=0, level=1) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/pc.c:177
#4  0x080b4b06 in qemu_set_irq (irq=0x89e9338, level=1) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/irq.c:38
#5  0x080ea926 in qemu_irq_raise (irq=0x89e9338) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/irq.h:12
#6  0x080eaafe in pic_update_irq (s=0x8a31a90) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/i8259.c:136
#7  0x080eac3a in pic_set_irq (opaque=0x8a31a90, irq=0, level=1) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/i8259.c:189
#8  0x080b4b06 in qemu_set_irq (irq=0x89e9390, level=1) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/irq.c:38
#9  0x0828551d in gsi_handler (opaque=0x89ea9d0, n=0, level=1) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/pc.c:97
#10 0x080b4b06 in qemu_set_irq (irq=0x89ee9a0, level=1) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/irq.c:38
#11 0x080ea3f8 in pit_irq_timer_update (s=0x8a34fc4, current_time=0)
at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/i8254.c:385
#12 0x080e9f27 in pit_load_count (s=0x8a34fc4, val=65536) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/i8254.c:246
#13 0x080ea6df in pit_reset (dev=0x8a34ee0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/i8254.c:484
#14 0x08164f94 in qdev_reset_one (dev=0x8a34ee0, opaque=0x0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:334
#15 0x081657d1 in qdev_walk_children (dev=0x8a34ee0, devfn=0x8164f59
<qdev_reset_one>, busfn=0x8164fe3 <qbus_reset_one>, opaque=0x0)
    at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:530
#16 0x0816575b in qbus_walk_children (bus=0x8a343e0, devfn=0x8164f59
<qdev_reset_one>, busfn=0x8164fe3 <qbus_reset_one>, opaque=0x0)
    at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:514
#17 0x0816580a in qdev_walk_children (dev=0x8a33c80, devfn=0x8164f59
<qdev_reset_one>, busfn=0x8164fe3 <qbus_reset_one>, opaque=0x0)
    at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:537
#18 0x08165072 in qdev_reset_all (dev=0x8a33c80) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:358
#19 0x0810c4b9 in pci_device_reset (dev=0x8a33c80) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/pci.c:165
#20 0x0810c6ec in pci_bus_reset (bus=0x89ef8c0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/pci.c:209
#21 0x0810c735 in pcibus_reset (qbus=0x89ef8c0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/pci.c:216
#22 0x0816501e in qbus_reset_one (bus=0x89ef8c0, opaque=0x0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:351
#23 0x08165722 in qbus_walk_children (bus=0x89ef8c0, devfn=0x8164f59
<qdev_reset_one>, busfn=0x8164fe3 <qbus_reset_one>, opaque=0x0)
    at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:507
#24 0x0816580a in qdev_walk_children (dev=0x89eeac8, devfn=0x8164f59
<qdev_reset_one>, busfn=0x8164fe3 <qbus_reset_one>, opaque=0x0)
    at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:537
#25 0x0816575b in qbus_walk_children (bus=0x89e3dd8, devfn=0x8164f59
<qdev_reset_one>, busfn=0x8164fe3 <qbus_reset_one>, opaque=0x0)
    at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:514
#26 0x081650c5 in qbus_reset_all_fn (opaque=0x89e3dd8) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:364
#27 0x0813c88a in qemu_system_reset (report=false) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/vl.c:1381
#28 0x08141bfa in main (argc=6, argv=0xffffd814, envp=0xffffd830) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/vl.c:3462


then PIC reset

Breakpoint 2, apic_set_irq (s=0x89e3e18, vector_num=0, trigger_mode=0)
at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/apic.c:424
424     {
(gdb) bt
#0  apic_set_irq (s=0x89e3e18, vector_num=0, trigger_mode=0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/apic.c:424
#1  0x081cd717 in apic_local_deliver (s=0x89e3e18, vector=3) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/apic.c:182
#2  0x081cd76f in apic_deliver_pic_intr (d=0x89e3e18, level=1) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/apic.c:191
#3  0x08285846 in pic_irq_request (opaque=0x0, irq=0, level=1) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/pc.c:177
#4  0x080b4b06 in qemu_set_irq (irq=0x89e9338, level=1) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/irq.c:38
#5  0x080ea926 in qemu_irq_raise (irq=0x89e9338) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/irq.h:12
#6  0x080eaafe in pic_update_irq (s=0x8a31a90) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/i8259.c:136
#7  0x080eac3a in pic_set_irq (opaque=0x8a31a90, irq=2, level=0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/i8259.c:189
#8  0x080b4b06 in qemu_set_irq (irq=0x89e93a8, level=0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/irq.c:38
#9  0x080ea963 in qemu_irq_lower (irq=0x89e93a8) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/irq.h:17
#10 0x080eab0e in pic_update_irq (s=0x8a348a0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/i8259.c:138
#11 0x080eae74 in pic_init_reset (s=0x8a348a0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/i8259.c:267
#12 0x080eaeb5 in pic_reset (dev=0x8a348a0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/i8259.c:274
#13 0x08164f94 in qdev_reset_one (dev=0x8a348a0, opaque=0x0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:334
#14 0x081657d1 in qdev_walk_children (dev=0x8a348a0, devfn=0x8164f59
<qdev_reset_one>, busfn=0x8164fe3 <qbus_reset_one>, opaque=0x0)
    at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:530
#15 0x0816575b in qbus_walk_children (bus=0x8a343e0, devfn=0x8164f59
<qdev_reset_one>, busfn=0x8164fe3 <qbus_reset_one>, opaque=0x0)
    at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:514
#16 0x0816580a in qdev_walk_children (dev=0x8a33c80, devfn=0x8164f59
<qdev_reset_one>, busfn=0x8164fe3 <qbus_reset_one>, opaque=0x0)
    at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:537
#17 0x08165072 in qdev_reset_all (dev=0x8a33c80) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:358
#18 0x0810c4b9 in pci_device_reset (dev=0x8a33c80) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/pci.c:165
#19 0x0810c6ec in pci_bus_reset (bus=0x89ef8c0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/pci.c:209
#20 0x0810c735 in pcibus_reset (qbus=0x89ef8c0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/pci.c:216
#21 0x0816501e in qbus_reset_one (bus=0x89ef8c0, opaque=0x0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:351
#22 0x08165722 in qbus_walk_children (bus=0x89ef8c0, devfn=0x8164f59
<qdev_reset_one>, busfn=0x8164fe3 <qbus_reset_one>, opaque=0x0)
    at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:507
#23 0x0816580a in qdev_walk_children (dev=0x89eeac8, devfn=0x8164f59
<qdev_reset_one>, busfn=0x8164fe3 <qbus_reset_one>, opaque=0x0)
    at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:537
#24 0x0816575b in qbus_walk_children (bus=0x89e3dd8, devfn=0x8164f59
<qdev_reset_one>, busfn=0x8164fe3 <qbus_reset_one>, opaque=0x0)
    at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:514
#25 0x081650c5 in qbus_reset_all_fn (opaque=0x89e3dd8) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:364
#26 0x0813c88a in qemu_system_reset (report=false) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/vl.c:1381
#27 0x08141bfa in main (argc=6, argv=0xffffd814, envp=0xffffd830) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/vl.c:3462


and then APIC reset

Breakpoint 3, apic_reset (d=0x89e3e18) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/apic.c:963
963     {
(gdb) bt
#0  apic_reset (d=0x89e3e18) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/apic.c:963
#1  0x08164f94 in qdev_reset_one (dev=0x89e3e18, opaque=0x0) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:334
#2  0x081657d1 in qdev_walk_children (dev=0x89e3e18, devfn=0x8164f59
<qdev_reset_one>, busfn=0x8164fe3 <qbus_reset_one>, opaque=0x0)
    at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:530
#3  0x0816575b in qbus_walk_children (bus=0x89e3dd8, devfn=0x8164f59
<qdev_reset_one>, busfn=0x8164fe3 <qbus_reset_one>, opaque=0x0)
    at /home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:514
#4  0x081650c5 in qbus_reset_all_fn (opaque=0x89e3dd8) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/hw/qdev.c:364
#5  0x0813c88a in qemu_system_reset (report=false) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/vl.c:1381
#6  0x08141bfa in main (argc=6, argv=0xffffd814, envp=0xffffd830) at
/home/jcmvbkbc/ws/xtensa/qemu-xtensa/vl.c:3462


I see that it happens in one loop. However the order is important
here: if APIC is not reset last then spurious interrupt is delivered.
The question is the following: what guarantees that the APIC will be reset last?

With current qemu wrong reset order has the following side effect for me:

[    0.225110] Performance Events: AMD PMU driver.
[    0.228504] ... version:                0
[    0.228950] ... bit width:              48
[    0.229376] ... generic registers:      4
[    0.230000] ... value mask:             0000ffffffffffff
[    0.230141] ... max period:             00007fffffffffff
[    0.230655] ... fixed-purpose events:   0
[    0.231091] ... event mask:             000000000000000f
[    0.240822] SMP alternatives: switching to UP code
[    0.580000] Freeing SMP alternatives: 16k freed
[    0.588768] ACPI: Core revision 20100702
[    0.799949] Setting APIC routing to flat
[    1.812234] ------------[ cut here ]------------
[    1.816885] WARNING: at arch/x86/kernel/apic/apic.c:1238
setup_local_APIC+0x18f/0x285()
[    1.817855] Hardware name: Bochs
[    1.818452] Modules linked in:
[    1.821957] Pid: 1, comm: swapper Not tainted 2.6.36.2-cosim.4 #3
[    1.822811] Call Trace:
[    1.825945]  [<ffffffff81037da7>] warn_slowpath_common+0x80/0x98
[    1.826835]  [<ffffffff81037dd4>] warn_slowpath_null+0x15/0x17
[    1.827463]  [<ffffffff812d6ce3>] setup_local_APIC+0x18f/0x285
[    1.828125]  [<ffffffff8166b84f>] native_smp_prepare_cpus+0x2b1/0x388
[    1.828808]  [<ffffffff816615c2>] kernel_init+0x79/0x1e5
[    1.829387]  [<ffffffff81003734>] kernel_thread_helper+0x4/0x10
[    1.830000]  [<ffffffff81661549>] ? kernel_init+0x0/0x1e5
[    1.830000]  [<ffffffff81003730>] ? kernel_thread_helper+0x0/0x10
[    1.838868] ---[ end trace 4eaa2a86a8e2da22 ]---

And there's a number of bug reports with the similar symptoms, e.g.
http://comments.gmane.org/gmane.comp.emulators.kvm.devel/57940

-- 
Thanks.
-- Max