Re: [Qemu-devel] Template for developing a Qe mu device with PCIe?and MSI-X

[Adnan Khaleel]

Hi there. I should have sent a lot of these with my note yesterday but I was in a hurry to get  the files to you first. 
See my comments below and thanks again.

AK
> pcie_msix_write_config() should call pci_default_write_config()
> unless you did it so intentionally.
I've made this change. Thanks for the pointer.
> You also want to catch up pci api clean up.
> pci_{set, get}_{byte, word, long, quad}(),
> pci_config_set_vendor() ...
Are you referring to the setting up of the config registers where we pass on the vendor id and device id etc? Would you elaborate a little more.

Also, I've got a bunch of questions but let me state my assumptions first so that you have a better idea of what I'm referring to.
- The device template is a pcie endpoint
- I want to be able to setup 64bit bar addresses, with large apertures
- For now, I'd like to be able to do basic MMIO and regular IO reads and writes. 


PCIe questions.
1. What does the topology of the bridge with respect to the root look like? Is it

Root <---> PCIe Bridge

2. If so, where is the slot where I can insert the PCIe device? Is it off the Bridge or would it be better for it to be off the root?

Root <---> PCIe Bridge <---> PCIe/MSI-X device

Or

Root <---> PCIe Bridge
     <---> PCIe/MSI-X Device

And hence my confusion about how to do the following:
static void pcie_msix_register(void)
{   
    pci_bridge_qdev_register(&pcie_msix_info);  // Is this what I should be doing?
        OR
    pci_qdev_register(&pcie_msix_info);         // Or this
}

3. I wasn't sure how to register the device how to do the initializing. Please see the following section of code:

void pcie_msix_init(PCIBus *bus)
{
    // Is this how we should be doing this?
    pci_create_simple(bus, -1, "pcie_msix_device");
        OR
         pci_bridge_create(...);
}

Or if should I use pci_bridge_create(...) in place of the pci_create_simple(...)

Also, this confusion led me to being unsure what the following device struct should look like

typedef struct PCIE_MSIX_DEVState_St {
    PCIDevice dev;
    int mmio_index;
} PCIE_MSIX_DEVState;

For the simple device function that I've described above, what is the purpose of this struct? What other data should be captured?

Which include the initializing of the following static structs. Btw, can you tell me what VMStateDescrption is used for by Qemu? Also, what should the "fields" member contain? I couldn't quite make out.

static const VMStateDescription vmstate_pcie_msix = {
    .name = "pcie-msix-device",
    .version_id = 1,
    .minimum_version_id = 1,
    .minimum_version_id_old = 1,
    .fields = (VMStateField[]) {
        VMSTATE_PCIE_DEVICE(dev, PCIE_MSIX_DEVState),
        VMSTATE_STRUCT(dev.aer_log, PCIE_MSIX_DEVState, 0, vmstate_pcie_aer_log, struct pcie_aer_log),
        VMSTATE_END_OF_LIST()
    }
};

4. What is the qdev.props field used for?
static PCIDeviceInfo pcie_msix_info = {
    .qdev.name = PCIE_MSIX_DEVICE,
    .qdev.desc = "PCIE MSIX device template",
    .qdev.size = sizeof(PCIE_MSIX_DEVState),
    .qdev.reset = pcie_msix_reset,
    .qdev.vmsd = &vmstate_pcie_msix,
    .is_express = 1,
    .config_write = pcie_msix_write_config,
    .init = pcie_msix_initfn,
    .exit = pcie_msix_exitfn,
    .qdev.props = (Property[]) {        
        DEFINE_PROP_END_OF_LIST(),
    }
};


5. Device instantiation
I init the device in pc_q35_bridge_init() in pc_q35.c

pcie_msix_init(root_port_bus);

I know I'm doing this incorrectly since I'm not specifying several things. Again, is this the correct place to init the device?


MSI/MSIX questions
1. How is an interrupt notification passed on to Qemu? In the regular case I'd use qemu_set_irq(..) to do so but what is the correct way of doing it in the MSIX paradigm? For example in the case of a DMA transfer.