Re: [Qemu-devel] [PATCH] pci: add standard bridge device

[Wen Congyang]

At 09/08/2011 02:15 PM, Wen Congyang Write:
> At 09/07/2011 07:52 PM, Michael S. Tsirkin Write:
>> On Wed, Sep 07, 2011 at 12:39:09PM +0800, Wen Congyang wrote:
>>> At 09/06/2011 03:45 PM, Avi Kivity Write:
>>>> On 09/06/2011 06:06 AM, Wen Congyang wrote:
>>>>>>  Use the uio driver -
>>>>>>  http://docs.blackfin.uclinux.org/kernel/generated/uio-howto/.  You
>>>>> just
>>>>>>  mmap() the BAR from userspace and play with it.
>>>>>
>>>>> When I try to bind ivshmem to uio_pci_generic, I get the following
>>>>> messages:
>>>>> uio_pci_generic 0000:01:01.0: No IRQ assigned to device: no support
>>>>> for interrupts?
>>>>>
>>>>
>>>> No idea what this means.
>>>
>>> PCI 3.0 6.2.4
>>> For x86 based PCs, the values in this register correspond to IRQ numbers 
>>> (0-15) of the standard dual
>>> 8259 configuration. The value 255 is defined as meaning "unknown" or "no 
>>> connection" to the interrupt
>>> controller. Values between 15 and 254 are reserved.
>>>
>>> The register is interrupt line.
>>>
>>> I read the config of this device, the interrupt line is 0. It means that it 
>>> uses the IRQ0.
>>>
>>> The following is the uio_pci_generic's code:
>>> static int __devinit probe(struct pci_dev *pdev,
>>>                        const struct pci_device_id *id)
>>> {
>>>     struct uio_pci_generic_dev *gdev;
>>>     int err;
>>>
>>>     err = pci_enable_device(pdev);
>>>     if (err) {
>>>             dev_err(&pdev->dev, "%s: pci_enable_device failed: %d\n",
>>>                     __func__, err);
>>>             return err;
>>>     }
>>>
>>>     if (!pdev->irq) {
>>>             dev_warn(&pdev->dev, "No IRQ assigned to device: "
>>>                      "no support for interrupts?\n");
>>>             pci_disable_device(pdev);
>>>             return -ENODEV;
>>>     }
>>> ...
>>> }
>>>
>>> This function will be called when we write 'domain:bus:slot.function' to 
>>> /sys/bus/pci/drivers/uio_pci_generic/bind.
>>> pdev->irq is 0, it means the device uses IRQ0. But we refuse it. I do not 
>>> why.
>>>
>>> To Michael S. Tsirkin
>>> This code is writen by you. Do you know why you check whether pdev->irq is 
>>> 0?
>>>
>>> Thanks
>>> Wen Congyang
>>>
>>>>
>>
>> Well I see this in linux:
>>
>> /*
>>  * Read interrupt line and base address registers.
>>  * The architecture-dependent code can tweak these, of course.
>>  */
>> static void pci_read_irq(struct pci_dev *dev)
>> {
>>         unsigned char irq;
>>
>>         pci_read_config_byte(dev, PCI_INTERRUPT_PIN, &irq);
>>         dev->pin = irq;
>>         if (irq)
>>                 pci_read_config_byte(dev, PCI_INTERRUPT_LINE, &irq);
>>         dev->irq = irq;
>> }
>>
>> Thus a device without an interrupt pin will get irq set to 0,
>> and this seems the right way to detect such devices.
>> I don't think PCI devices really use IRQ0 in practice,
>> its probably used for PC things. More likely the system is
>> misconfigured.  Try lspci -vv to see what went wrong.
> 
> Yes, the PCI device shoulde not use IRQ0. I debug qemu's code, and find the
> PCI_INTERRUPT_LINE register is not set by qemu:
> =============
> Hardware watchpoint 6: ((uint8_t *) 0x164e410)[0x3c]
> 
> Old value = 0 '\000'
> New value = 10 '\n'
> pci_default_write_config (d=0x1653ed0, addr=60, val=10, l=1) at 
> /home/wency/source/qemu/hw/pci.c:1115
> 1115          d->config[addr + i] &= ~(val & w1cmask); /* W1C: Write 1 to 
> Clear */
> Missing separate debuginfos, use: debuginfo-install 
> cyrus-sasl-gssapi-2.1.23-8.el6.x86_64 cyrus-sasl-md5-2.1.23-8.el6.x86_64 
> cyrus-sasl-plain-2.1.23-8.el6.x86_64 db4-4.7.25-16.el6.x86_64
> (gdb) bt
> #0  pci_default_write_config (d=0x1653ed0, addr=60, val=10, l=1) at 
> /home/wency/source/qemu/hw/pci.c:1115
> #1  0x00000000004d5827 in pci_host_config_write_common (pci_dev=0x1653ed0, 
> addr=60, limit=256, val=10, len=1) at /home/wency/source/qemu/hw/pci_host.c:54
> #2  0x00000000004d5939 in pci_data_write (s=0x15f95a0, addr=2147502140, 
> val=10, len=1) at /home/wency/source/qemu/hw/pci_host.c:75
> #3  0x00000000004d5b19 in pci_host_data_write (handler=0x15f9570, addr=3324, 
> val=10, len=1) at /home/wency/source/qemu/hw/pci_host.c:125
> #4  0x000000000063ee06 in ioport_simple_writeb (opaque=0x15f9570, addr=3324, 
> value=10) at /home/wency/source/qemu/rwhandler.c:48
> #5  0x0000000000470db9 in ioport_write (index=0, address=3324, data=10) at 
> ioport.c:81
> #6  0x00000000004717bc in cpu_outb (addr=3324, val=10 '\n') at ioport.c:273
> #7  0x00000000005ef25d in kvm_handle_io (port=3324, data=0x7ffff7ff8000, 
> direction=1, size=1, count=1) at /home/wency/source/qemu/kvm-all.c:834
> #8  0x00000000005ef7e6 in kvm_cpu_exec (env=0x13da0d0) at 
> /home/wency/source/qemu/kvm-all.c:976
> #9  0x00000000005c1a7b in qemu_kvm_cpu_thread_fn (arg=0x13da0d0) at 
> /home/wency/source/qemu/cpus.c:661
> #10 0x00000032864077e1 in start_thread () from /lib64/libpthread.so.0
> #11 0x00000032858e68ed in clone () from /lib64/libc.so.6
> =============
> 
> If I put ivshmem on bus 0, the PCI_INTERRUPT_LINE register can be set. So I 
> guess this register is set by bios.
> I use the newest seabios, and PCI_INTERRUPT_LINE register is not set if the 
> deivce is not on bus0.

Here is the seabios's code:
==========
static void pci_bios_init_device(struct pci_device *pci)
{
    u16 bdf = pci->bdf;
    int pin, pic_irq;

    dprintf(1, "PCI: bus=%d devfn=0x%02x: vendor_id=0x%04x device_id=0x%04x\n"
            , pci_bdf_to_bus(bdf), pci_bdf_to_devfn(bdf)
            , pci->vendor, pci->device);
    pci_init_device(pci_class_tbl, pci, NULL);

    /* enable memory mappings */
    pci_config_maskw(bdf, PCI_COMMAND, 0, PCI_COMMAND_IO | PCI_COMMAND_MEMORY);

    /* map the interrupt */
    pin = pci_config_readb(bdf, PCI_INTERRUPT_PIN);
    if (pin != 0) {
        pin = pci_slot_get_pirq(bdf, pin - 1);
        pic_irq = pci_irqs[pin];
        pci_config_writeb(bdf, PCI_INTERRUPT_LINE, pic_irq);
    }

    pci_init_device(pci_device_tbl, pci, NULL);
}

static void pci_bios_init_device_in_bus(int bus)
{
    struct pci_device *pci;
    foreachpci(pci) {
        u8 pci_bus = pci_bdf_to_bus(pci->bdf);
        if (pci_bus < bus)
            continue;
        if (pci_bus > bus)
            break;
        pci_bios_init_device(pci);
    }
}

...
void
pci_setup(void)
{
...
    pci_bios_map_device_in_bus(0 /* host bus */);

    pci_bios_init_device_in_bus(0 /* host bus */);
...
}
==========

The PCI_INTERRUPT_LINE register is set in the function
pci_setup() calls pci_bios_init_device_in_bus() calls pci_bios_init_device().
According to the code, it only inits the PCI device on bus 0.

I modify the code like this, and the PCI_INTERRUPT_LINE register is set, and I 
can bind
it to uio_pci_generic:
diff --git a/src/pciinit.c b/src/pciinit.c
index 597c8ea..c2e97d7 100644
--- a/src/pciinit.c
+++ b/src/pciinit.c
@@ -575,6 +575,8 @@ static int pci_bios_init_root_regions(u32 start, u32 end)
 void
 pci_setup(void)
 {
+    int bus;
+
     if (CONFIG_COREBOOT || usingXen()) {
         // PCI setup already done by coreboot or Xen - just do probe.
         pci_probe();
@@ -603,9 +605,11 @@ pci_setup(void)
     dprintf(1, "=== PCI new allocation pass #2 ===\n");
     dprintf(1, "PCI: init bases bus 0 (primary)\n");
     pci_bios_init_bus_bases(&busses[0]);
-    pci_bios_map_device_in_bus(0 /* host bus */);
+    for (bus = 0; bus <= MaxPCIBus; bus++) {
+        pci_bios_map_device_in_bus(bus /* host bus */);
 
-    pci_bios_init_device_in_bus(0 /* host bus */);
+        pci_bios_init_device_in_bus(bus /* host bus */);
+    }
 
     struct pci_device *pci;
     foreachpci(pci) {


> 
> # lspci -vv
> 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
>       Subsystem: Red Hat, Inc Qemu virtual machine
>       Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
> <MAbort- >SERR- <PERR- INTx-
> 
> 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
>       Subsystem: Red Hat, Inc Qemu virtual machine
>       Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> 
> 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 
> (prog-if 80 [Master])
>       Subsystem: Red Hat, Inc Qemu virtual machine
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable) 
> [size=8]
>       Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable) 
> [size=1]
>       Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable) 
> [size=8]
>       Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable) 
> [size=1]
>       Region 4: I/O ports at d100 [size=16]
>       Kernel driver in use: ata_piix
>       Kernel modules: ata_piix
> 
> 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
>       Subsystem: Red Hat, Inc Qemu virtual machine
>       Physical Slot: 1
>       Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>       Interrupt: pin A routed to IRQ 9
>       Kernel driver in use: piix4_smbus
>       Kernel modules: i2c-piix4
> 
> 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 (prog-if 00 [VGA 
> controller])
>       Subsystem: Red Hat, Inc Device 1100
>       Physical Slot: 2
>       Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
> <MAbort- >SERR- <PERR- INTx-
>       Region 0: Memory at fc000000 (32-bit, prefetchable) [size=32M]
>       Region 1: Memory at f8020000 (32-bit, non-prefetchable) [size=4K]
>       Expansion ROM at f8000000 [disabled] [size=64K]
>       Kernel modules: cirrusfb
> 
> 00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. 
> RTL-8139/8139C/8139C+ (rev 20)
>       Subsystem: Red Hat, Inc Device 1100
>       Physical Slot: 3
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0, Cache Line Size: 64 bytes
>       Interrupt: pin A routed to IRQ 11
>       Region 0: I/O ports at d000 [size=256]
>       Region 1: Memory at f8021000 (32-bit, non-prefetchable) [size=256]
>       Expansion ROM at f8010000 [disabled] [size=64K]
>       Kernel driver in use: 8139cp
>       Kernel modules: 8139too, 8139cp
> 
> 00:08.0 PCI bridge: Red Hat, Inc. Device 0001 (prog-if 00 [Normal decode])
>       Physical Slot: 8
>       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- 
> <MAbort- >SERR- <PERR- INTx-
>       Latency: 0
>       Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>       I/O behind bridge: 0000c000-0000cfff
>       Memory behind bridge: f0000000-f7ffffff
>       Prefetchable memory behind bridge: fe000000-fe0fffff
>       Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- 
> <MAbort- <SERR- <PERR-
>       BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> 
> 01:01.0 RAM memory: Red Hat, Inc Device 1110
>       Subsystem: Red Hat, Inc Device 1100
>       Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
> <MAbort- >SERR- <PERR- INTx-
>       Interrupt: pin A routed to IRQ 0
>       Region 0: Memory at f4000000 (32-bit, non-prefetchable) [disabled] 
> [size=256]
>       Region 2: Memory at f0000000 (32-bit, non-prefetchable) [disabled] 
> [size=32M]
>       Kernel modules: virtio_pci
> 
> 01:01.1 RAM memory: Red Hat, Inc Device 1110
>       Subsystem: Red Hat, Inc Device 1100
>       Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>       Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
> <MAbort- >SERR- <PERR- INTx-
>       Interrupt: pin A routed to IRQ 0
>       Region 0: Memory at f4001000 (32-bit, non-prefetchable) [disabled] 
> [size=256]
>       Region 2: Memory at f2000000 (32-bit, non-prefetchable) [disabled] 
> [size=32M]
>       Kernel modules: virtio_pci
> 
> 
>>
> 
> 
>