Re: [Qemu-devel] Could not add PCI device with big memory to aarch64 VMs

[Laszlo Ersek]

On 11/04/15 23:22, liang yan wrote:
> Hello, Laszlo,
> 
> (1)I am trying to add ivshmem device(PCI device with big memory) to my
> aarch64 vm.
> So far, I could find device information from vm. But it seems vm did not
> create
> correct resource file for this device. Do you have any idea that this
> happens?
> 
> I used the upstream EDK2 to build my UEFI firmware.
> 
> There are three BARs for this device, and memory map is assigned too,
> but only one
> resource file is created.
> 
> My qemu supports ACPI 5.1 and the command line is :
> 
>   -device ivshmem,size=256M,chardev=ivshmem,msi=on,ioeventfd=on \
>   -chardev socket,path=/tmp/ivshmem_socket,id=ivshmem \
> 
> The lspci information:
> 
> 00:00.0 Host bridge: Red Hat, Inc. Device 0008
>     Subsystem: Red Hat, Inc Device 1100
>     Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>     Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> 
> 00:01.0 RAM memory: Red Hat, Inc Inter-VM shared memory
>     Subsystem: Red Hat, Inc QEMU Virtual Machine
>     Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>     Interrupt: pin A routed to IRQ 255
>     Region 0: Memory at 20001000 (32-bit, non-prefetchable) [disabled]
> [size=256]
>     Region 1: Memory at 20000000 (32-bit, non-prefetchable) [disabled]
> [size=4K]
>     Region 2: Memory at 10000000 (64-bit, prefetchable) [disabled]
> [size=256M]
>     Capabilities: [40] MSI-X: Enable- Count=1 Masked-
>         Vector table: BAR=1 offset=00000000
>         PBA: BAR=1 offset=00000800
> 
> Boot information:
> 
> [    2.380924] pci 0000:00:01.0: BAR 2: assigned [mem
> 0x10000000-0x1fffffff 64bit pref]
> [    2.382836] pci 0000:00:01.0: BAR 1: assigned [mem
> 0x20000000-0x20000fff]
> [    2.383557] pci 0000:00:01.0: BAR 0: assigned [mem
> 0x20001000-0x200010ff]
> 
> 
> Files under /sys/devices/pci0000:00/0000:00:01.0
> 
> broken_parity_status      devspec       local_cpus  resource
> class              dma_mask_bits    modalias    subsystem
> config              driver_override  msi_bus     subsystem_device
> consistent_dma_mask_bits  enable       power       subsystem_vendor
> d3cold_allowed          irq           remove      uevent
> device              local_cpulist    rescan      vendor
> 
> Information for resource:
> 
> 0x0000000020001000 0x00000000200010ff 0x0000000000040200
> 0x0000000020000000 0x0000000020000fff 0x0000000000040200
> 0x0000000010000000 0x000000001fffffff 0x000000000014220c
> 0x0000000000000000 0x0000000000000000 0x0000000000000000
> 0x0000000000000000 0x0000000000000000 0x0000000000000000
> 0x0000000000000000 0x0000000000000000 0x0000000000000000
> 0x0000000000000000 0x0000000000000000 0x0000000000000000
> 
> 
> 
> 
> (2)It also has a problem that once I use a memory bigger than 256M for
> ivshmem, it could not get through UEFI,
> the error message is
> 
> PciBus: Discovered PCI @ [00|01|00]
>    BAR[0]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x100; Offset =
> 0x10
>    BAR[1]: Type =  Mem32; Alignment = 0xFFF;    Length = 0x1000; Offset
> = 0x14
>    BAR[2]: Type = PMem64; Alignment = 0x3FFFFFFF;    Length =
> 0x40000000;    Offset = 0x18
> 
> PciBus: HostBridge->SubmitResources() - Success
> ASSERT
> /home/liang/studio/edk2/ArmVirtPkg/PciHostBridgeDxe/PciHostBridge.c(449): 
> ((BOOLEAN)(0==1))
> 
> 
> I am wandering if there are memory limitation for pcie devices under
> Qemu environment?
> 
> 
> Just thank you in advance and any information would be appreciated.

(CC'ing Ard.)

"Apparently", the firmware-side counterpart of QEMU commit 5125f9cd2532
has never been contributed to edk2.

Therefore the the ProcessPciHost() function in
"ArmVirtPkg/VirtFdtDxe/VirtFdtDxe.c" ignores the
DTB_PCI_HOST_RANGE_MMIO64 type range from the DTB. (Thus only
DTB_PCI_HOST_RANGE_MMIO32 is recognized as PCI MMIO aperture.)

However, even if said driver was extended to parse the new 64-bit
aperture into PCDs (which wouldn't be hard), the
ArmVirtPkg/PciHostBridgeDxe driver would still have to be taught to look
at that aperture (from the PCDs) and to serve MMIO BAR allocation
requests from it. That could be hard.

Please check edk2 commits e48f1f15b0e2^..e5ceb6c9d390, approximately,
for the background on the current code. See also chapter 13 "Protocols -
PCI Bus Support" in the UEFI spec.

Patches welcome. :)

(A separate note on ACPI vs. DT: the firmware forwards *both* from QEMU
to the runtime guest OS. However, the firmware parses only the DT for
its own purposes.)

Thanks
Laszlo