Re: [Qemu-devel] QEMU and vIOMMU support for emulated VF passthrough to nested (L2) VM

[Elijah Shakkour]

> -----Original Message-----
> From: Peter Xu <address@hidden>
> Sent: Wednesday, April 3, 2019 5:40 AM
> To: Elijah Shakkour <address@hidden>
> Cc: Knut Omang <address@hidden>; Michael S. Tsirkin
> <address@hidden>; Alex Williamson <address@hidden>;
> Marcel Apfelbaum <address@hidden>; Stefan Hajnoczi
> <address@hidden>; address@hidden
> Subject: Re: QEMU and vIOMMU support for emulated VF passthrough to
> nested (L2) VM
> 
> On Tue, Apr 02, 2019 at 03:41:10PM +0000, Elijah Shakkour wrote:
> >
> >
> > > -----Original Message-----
> > > From: Knut Omang <address@hidden>
> > > Sent: Monday, April 1, 2019 5:24 PM
> > > To: Elijah Shakkour <address@hidden>; Peter Xu
> > > <address@hidden>
> > > Cc: Michael S. Tsirkin <address@hidden>; Alex Williamson
> > > <address@hidden>; Marcel Apfelbaum
> > > <address@hidden>; Stefan Hajnoczi
> <address@hidden>;
> > > address@hidden
> > > Subject: Re: QEMU and vIOMMU support for emulated VF passthrough
> to
> > > nested (L2) VM
> > >
> > > On Mon, 2019-04-01 at 14:01 +0000, Elijah Shakkour wrote:
> > > >
> > > > > -----Original Message-----
> > > > > From: Peter Xu <address@hidden>
> > > > > Sent: Monday, April 1, 2019 1:25 PM
> > > > > To: Elijah Shakkour <address@hidden>
> > > > > Cc: Knut Omang <address@hidden>; Michael S. Tsirkin
> > > > > <address@hidden>; Alex Williamson <address@hidden>;
> > > > > Marcel Apfelbaum <address@hidden>; Stefan Hajnoczi
> > > > > <address@hidden>; address@hidden
> > > > > Subject: Re: QEMU and vIOMMU support for emulated VF
> passthrough
> > > to
> > > > > nested (L2) VM
> > > > >
> > > > > On Mon, Apr 01, 2019 at 09:12:38AM +0000, Elijah Shakkour wrote:
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Peter Xu <address@hidden>
> > > > > > > Sent: Monday, April 1, 2019 5:47 AM
> > > > > > > To: Elijah Shakkour <address@hidden>
> > > > > > > Cc: Knut Omang <address@hidden>; Michael S. Tsirkin
> > > > > > > <address@hidden>; Alex Williamson
> > > > > > > <address@hidden>; Marcel Apfelbaum
> > > > > > > <address@hidden>; Stefan Hajnoczi
> > > > > > > <address@hidden>; address@hidden
> > > > > > > Subject: Re: QEMU and vIOMMU support for emulated VF
> > > passthrough
> > > > > to
> > > > > > > nested (L2) VM
> > > > > > >
> > > > > > > On Sun, Mar 31, 2019 at 11:15:00AM +0000, Elijah Shakkour wrote:
> > > > > > >
> > > > > > > [...]
> > > > > > >
> > > > > > > > I didn't have DMA nor MMIO read/write working with my old
> > > > > > > > command
> > > > > > > line.
> > > > > > > > But, when I removed all CPU flags and only provided "-cpu
> > > > > > > > host", I see that
> > > > > > > MMIO works.
> > > > > > > > Still, DMA read/write from emulated device doesn't work for VF.
> > > > > > > > For
> > > > > > > example:
> > > > > > > > Driver provides me a buffer pointer through MMIO write,
> > > > > > > > this address
> > > > > > > (pointer) is GPA of L2, and when I try to call
> > > > > > > pci_dma_read() with this address I get:
> > > > > > > > "
> > > > > > > > Unassigned mem read 0000000000000000 "
> > > > > > >
> > > > > > > I don't know where this error log was dumped but if it's
> > > > > > > during DMA then I agree it can probably be related to vIOMMU.
> > > > > > >
> > > > > >
> > > > > > This log is dumped from:
> > > > > > memory.c: unassigned_mem_read()
> > > > > >
> > > > > > > > As I said, my problem now is in translation of L2 GPA
> > > > > > > > provided by driver,
> > > > > > > when I call DMA read/write for this address from VF.
> > > > > > > > Any insights?
> > > > > > >
> > > > > > > I just noticed that you were using QEMU 2.12 [1].  If that's
> > > > > > > the case, please rebase to the latest QEMU, at least >=3.0
> > > > > > > because there's major refactor of the shadow logic during
> > > > > > > 3.0 devel cycle
> > > AFAICT.
> > > > > > >
> > > > > >
> > > > > > Rebased to QEMU 3.1
> > > > > > Now I see the address I'm trying to read from in log but still same
> error:
> > > > > > "
> > > > > > Unassigned mem read 00000000f0481000 "
> > > > > > What do you suggest?
> > > > >
> > > > > Would you please answer the questions that Knut asked?  Is it
> > > > > working for L1 guest?  How about PF?
> > > >
> > > > Both VF and PF are working for L1 guest.
> > > > I don't know how to passthrough PF to nested VM in hyper-v.
> > >
> > > On Linux passing through VFs and PFs are the same.
> > > Maybe you can try passthrough with all Linux first? (first PF then VF) ?
> > >
> > > > I don't invoke VF manually in hyper-v and pass it through to
> > > > nested VM. I use hyper-v manager to configure and provide a VF for
> > > > nested VM (I can see the VF only in the nested VM).
> > > >
> > > > Did someone try to run emulated device in linux RH as nested L2
> > > > where
> > > > L1 is windows hyper-v? Does DMA read/write work for this emulated
> > > device in this case?
> > >
> > > I have never tried that, I have only used Linux as L2, Windows might
> > > be pickier about what it expects, so starting with Linux to rule
> > > that out is probably a good idea.
> >
> > Will move to this solution after I/we give up 😊
> >
> > >
> > > > >
> > > > > You can also try to enable VT-d device log by appending:
> > > > >
> > > > >   -trace enable="vtd_*"
> > > > >
> > > > > In case it dumps anything useful for you.
> >
> > Here is the relevant dump (dev 01:00.01 is my VF):
> > "
> > vtd_inv_desc_cc_device context invalidate device 01:00.01
> > vtd_ce_not_present Context entry bus 1 devfn 1 not present
> > vtd_switch_address_space Device 01:00.1 switching address space (iommu
> > enabled=1) vtd_ce_not_present Context entry bus 1 devfn 1 not present
> > vtd_err Detected invalid context entry when trying to sync shadow page
> > table
> 
> These lines mean that the guest sent a device invalidation to your VF but the
> IOMMU found that the device context entry is missing.
> 
> > vtd_iotlb_cc_update IOTLB context update bus 0x1 devfn 0x1 high 0x102
> > low 0x2d007003 gen 0 -> gen 2 vtd_err_dmar_slpte_resv_error iova
> > 0xf08e7000 level 2 slpte 0x2a54008f7
> 
> This line should not exist in latest QEMU.  Are you sure you're using the 
> latest
> QEMU?

I moved now to QEMU 4.0 RC2.
This is the what I get now:
vtd_iotlb_cc_update IOTLB context update bus 0x1 devfn 0x1 high 0x102 low 
0x2f007003 gen 0 -> gen 1
qemu-system-x86_64: vtd_iova_to_slpte: detected splte reserve non-zero 
iova=0xf0d29000, level=0x2slpte=0x29f6008f7)
vtd_fault_disabled Fault processing disabled for context entry
qemu-system-x86_64: vtd_iommu_translate: detected translation failure 
(dev=01:00:01, iova=0xf0d29000)
Unassigned mem read 00000000f0d29000

I'm not familiar with vIOMMU registers, but I noticed that I must report snoop 
control support to Hyper-V (i.e. bit 7 in extended capability register of 
vIOMMU) in-order to satisfy IOMMU support for SRIOV.
vIOMMU.ecap before    0xf00f5e
vIOMMU.ecap after       0xf00fde
But I see that vIOMMU doesn't really support snoop control.
Could this be the problem that fails IOVA range check in this function 
vtd_iova_range_check()?
If so, is there a way to work-around this issue?

> 
> I agree with Knut that I would suggest you to use use Linux in both host and
> guests first to triage the issue.
> 
> Regards,
> 
> --
> Peter Xu