Re: [Qemu-devel] Template for developing a Qemu device with?PCIe?and MSI-X

[Isaku Yamahata]

On Thu, Aug 26, 2010 at 01:17:38PM -0500, Adnan Khaleel wrote:
>     You also want to catch up pci api clean up.
>     pci_{set, get}_{byte, word, long, quad}(),
>     pci_config_set_vendor() ...
> 
> Are you referring to the setting up of the config registers where we pass on
> the vendor id and device id etc? Would you elaborate a little more.

Yes. Now there are helper functions to address pci endian.
So there is no need to ugly bit operation in  pcie_msix_initfn().


> Also, I've got a bunch of questions but let me state my assumptions first so
> that you have a better idea of what I'm referring to.
> - The device template is a pcie endpoint
> - I want to be able to setup 64bit bar addresses, with large apertures

Use PCI_BASE_ADDRESS_MEM_TYPE_64.


> - For now, I'd like to be able to do basic MMIO and regular IO reads and
> writes.
> 
> 
> PCIe questions.
> 1. What does the topology of the bridge with respect to the root look like? Is
> it
> 
> Root <---> PCIe Bridge

lspci -t would help.
Roughly the bus topology looks like

  root port - upstream port --- downstream port -> end node device
            |                |
            ...             ...


> 2. If so, where is the slot where I can insert the PCIe device? Is it off the
> Bridge or would it be better for it to be off the root?
> 
> Root <---> PCIe Bridge <---> PCIe/MSI-X device
> 
> Or
> 
> Root <---> PCIe Bridge
>      <---> PCIe/MSI-X Device

On downstream port switch. See above.


> And hence my confusion about how to do the following:
> static void pcie_msix_register(void)
> {  
>     pci_bridge_qdev_register(&pcie_msix_info);  // Is this what I should be
> doing?
>         OR
>     pci_qdev_register(&pcie_msix_info);         // Or this
> }

pci_qdev_register()


> 3. I wasn't sure how to register the device how to do the initializing. Please
> see the following section of code:
> 
> void pcie_msix_init(PCIBus *bus)
> {
>     // Is this how we should be doing this?
>     pci_create_simple(bus, -1, "pcie_msix_device");
>         OR
>         pci_bridge_create(...);
> }
> Or if should I use pci_bridge_create(...) in place of the pci_create_simple
> (...)

pci_create_simple()


> Also, this confusion led me to being unsure what the following device struct
> should look like
> 
> typedef struct PCIE_MSIX_DEVState_St {
>     PCIDevice dev;
>     int mmio_index;
> } PCIE_MSIX_DEVState;
> 
> For the simple device function that I've described above, what is the purpose
> of this struct? What other data should be captured?
> 
> Which include the initializing of the following static structs. Btw, can you
> tell me what VMStateDescrption is used for by Qemu? Also, what should the
> "fields" member contain? I couldn't quite make out.
> 
> static const VMStateDescription vmstate_pcie_msix = {
>     .name = "pcie-msix-device",
>     .version_id = 1,
>     .minimum_version_id = 1,
>     .minimum_version_id_old = 1,
>     .fields = (VMStateField[]) {
>         VMSTATE_PCIE_DEVICE(dev, PCIE_MSIX_DEVState),
>         VMSTATE_STRUCT(dev.aer_log, PCIE_MSIX_DEVState, 0,
> vmstate_pcie_aer_log, struct pcie_aer_log),
>         VMSTATE_END_OF_LIST()
>     }
> };
>
> 4. What is the qdev.props field used for?
> static PCIDeviceInfo pcie_msix_info = {
>     .qdev.name = PCIE_MSIX_DEVICE,
>     .qdev.desc = "PCIE MSIX device template",
>     .qdev.size = sizeof(PCIE_MSIX_DEVState),
>     .qdev.reset = pcie_msix_reset,
>     .qdev.vmsd = &vmstate_pcie_msix,
>     .is_express = 1,
>     .config_write = pcie_msix_write_config,
>     .init = pcie_msix_initfn,
>     .exit = pcie_msix_exitfn,
>     .qdev.props = (Property[]) {       
>         DEFINE_PROP_END_OF_LIST(),
>     }
> };

Perhaps at first it would be better to get a idea of what qdev device model is.
The following slide would be a good start point.
http://www.linux-kvm.org/wiki/images/f/fe/2010-forum-armbru-qdev.pdf


> 5. Device instantiation
> I init the device in pc_q35_bridge_init() in pc_q35.c
> 
> pcie_msix_init(root_port_bus);
> 
> I know I'm doing this incorrectly since I'm not specifying several things.
> Again, is this the correct place to init the device?

Yes if you really wanted the end node device to be on root port.
Otherwise downstream port is the place on which the end node device is.


> MSI/MSIX questions
> 1. How is an interrupt notification passed on to Qemu? In the regular case I'd
> use qemu_set_irq(..) to do so but what is the correct way of doing it in the
> MSIX paradigm? For example in the case of a DMA transfer.

Use msix_notify()/msi_notify().
virtio_pci_notify() in virtio-pci.c is an example.
pcie_notify() in pcie.c is also another example.
-- 
yamahata