Re: [Qemu-devel] [PULL 14/28] exec: make address spaces 64-bit wide

[Alex Williamson]

On Mon, 2014-01-20 at 11:20 -0500, Mike Day wrote:
> Do you know which device is writing to the BAR below? From the trace
> it appears it should be restoring the memory address to the BAR after
> writing all 1s to the BAR and reading back the contents. (the protocol
> for finding the length of the bar memory.)

The guest itself is writing the the BARs.  This is a standard sizing
operation by the guest.

> On Thu, Jan 9, 2014 at 12:24 PM, Alex Williamson
> <address@hidden> wrote:
> > On Wed, 2013-12-11 at 20:30 +0200, Michael S. Tsirkin wrote:
> >> From: Paolo Bonzini <address@hidden>
> > vfio: vfio_pci_read_config(0000:01:10.0, @0x10, len=0x4) febe0004
> > (save lower 32bits of BAR)
> > vfio: vfio_pci_write_config(0000:01:10.0, @0x10, 0xffffffff, len=0x4)
> > (write mask to BAR)
> 
> Here the device should restore the memory address (original contents)
> to the BAR.

Sorry if it's not clear, the trace here is what the vfio-pci driver
sees.  We're just observing the sizing operation of the guest, therefore
we see:

1) orig = read()
2) write(0xffffffff)
3) size_mask = read()
4) write(orig)

We're only at step 2)

> > vfio: region_del febe0000 - febe3fff
> > (memory region gets unmapped)
> > vfio: vfio_pci_read_config(0000:01:10.0, @0x10, len=0x4) ffffc004
> > (read size mask)

step 3)

> > vfio: vfio_pci_write_config(0000:01:10.0, @0x10, 0xfebe0004, len=0x4)
> > (restore BAR)

step 4)

> > vfio: region_add febe0000 - febe3fff [0x7fcf3654d000]
> > (memory region re-mapped)
> > vfio: vfio_pci_read_config(0000:01:10.0, @0x14, len=0x4) 0
> > (save upper 32bits of BAR)
> > vfio: vfio_pci_write_config(0000:01:10.0, @0x14, 0xffffffff, len=0x4)
> > (write mask to BAR)
> 
> and here ...

This is the same as above to the next BAR, which is the upper 32bits of
the 64bit BAR.

> > vfio: region_del febe0000 - febe3fff
> > (memory region gets unmapped)
> > vfio: region_add fffffffffebe0000 - fffffffffebe3fff [0x7fcf3654d000]
> > (memory region gets re-mapped with new address)
> > qemu-system-x86_64: vfio_dma_map(0x7fcf38861710, 0xfffffffffebe0000, 
> > 0x4000, 0x7fcf3654d000) = -14 (Bad address)
> > (iommu barfs because it can only handle 48bit physical addresses)
> 
> I looked around some but I couldn't find an obvious culprit. Could it
> be that the BAR is getting unmapped automatically due to
> x-intx-mmap-timeout-ms before the device has a chance to finish
> restoring the correct value to the BAR?

No, this is simply the guest sizing the BAR, this is not an internally
generated operation.  The INTx emulation isn't used here as KVM
acceleration is enabled.  That also only toggles the enable setting on
the mmap'd MemoryRegion, it doesn't change the address it's mapped to.
Thanks,

Alex