Re: [Qemu-devel] QEMU and vIOMMU support for emulated VF passthrough to nested (L2) VM

[Elijah Shakkour]

> -----Original Message-----
> From: Elijah Shakkour
> Sent: Thursday, April 4, 2019 12:57 AM
> To: 'Peter Xu' <address@hidden>
> Cc: Knut Omang <address@hidden>; Michael S. Tsirkin
> <address@hidden>; Alex Williamson <address@hidden>;
> Marcel Apfelbaum <address@hidden>; Stefan Hajnoczi
> <address@hidden>; address@hidden
> Subject: RE: QEMU and vIOMMU support for emulated VF passthrough to
> nested (L2) VM
> 
> 
> 
> > -----Original Message-----
> > From: Peter Xu <address@hidden>
> > Sent: Wednesday, April 3, 2019 5:40 AM
> > To: Elijah Shakkour <address@hidden>
> > Cc: Knut Omang <address@hidden>; Michael S. Tsirkin
> > <address@hidden>; Alex Williamson <address@hidden>;
> Marcel
> > Apfelbaum <address@hidden>; Stefan Hajnoczi
> > <address@hidden>; address@hidden
> > Subject: Re: QEMU and vIOMMU support for emulated VF passthrough to
> > nested (L2) VM
> >
> > On Tue, Apr 02, 2019 at 03:41:10PM +0000, Elijah Shakkour wrote:
> > >
> > >
> > > > -----Original Message-----
> > > > From: Knut Omang <address@hidden>
> > > > Sent: Monday, April 1, 2019 5:24 PM
> > > > To: Elijah Shakkour <address@hidden>; Peter Xu
> > > > <address@hidden>
> > > > Cc: Michael S. Tsirkin <address@hidden>; Alex Williamson
> > > > <address@hidden>; Marcel Apfelbaum
> > > > <address@hidden>; Stefan Hajnoczi
> > <address@hidden>;
> > > > address@hidden
> > > > Subject: Re: QEMU and vIOMMU support for emulated VF passthrough
> > to
> > > > nested (L2) VM
> > > >
> > > > On Mon, 2019-04-01 at 14:01 +0000, Elijah Shakkour wrote:
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Peter Xu <address@hidden>
> > > > > > Sent: Monday, April 1, 2019 1:25 PM
> > > > > > To: Elijah Shakkour <address@hidden>
> > > > > > Cc: Knut Omang <address@hidden>; Michael S. Tsirkin
> > > > > > <address@hidden>; Alex Williamson
> > > > > > <address@hidden>; Marcel Apfelbaum
> > > > > > <address@hidden>; Stefan Hajnoczi
> > > > > > <address@hidden>; address@hidden
> > > > > > Subject: Re: QEMU and vIOMMU support for emulated VF
> > passthrough
> > > > to
> > > > > > nested (L2) VM
> > > > > >
> > > > > > On Mon, Apr 01, 2019 at 09:12:38AM +0000, Elijah Shakkour wrote:
> > > > > > >
> > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Peter Xu <address@hidden>
> > > > > > > > Sent: Monday, April 1, 2019 5:47 AM
> > > > > > > > To: Elijah Shakkour <address@hidden>
> > > > > > > > Cc: Knut Omang <address@hidden>; Michael S. Tsirkin
> > > > > > > > <address@hidden>; Alex Williamson
> > > > > > > > <address@hidden>; Marcel Apfelbaum
> > > > > > > > <address@hidden>; Stefan Hajnoczi
> > > > > > > > <address@hidden>; address@hidden
> > > > > > > > Subject: Re: QEMU and vIOMMU support for emulated VF
> > > > passthrough
> > > > > > to
> > > > > > > > nested (L2) VM
> > > > > > > >
> > > > > > > > On Sun, Mar 31, 2019 at 11:15:00AM +0000, Elijah Shakkour
> wrote:
> > > > > > > >
> > > > > > > > [...]
> > > > > > > >
> > > > > > > > > I didn't have DMA nor MMIO read/write working with my
> > > > > > > > > old command
> > > > > > > > line.
> > > > > > > > > But, when I removed all CPU flags and only provided
> > > > > > > > > "-cpu host", I see that
> > > > > > > > MMIO works.
> > > > > > > > > Still, DMA read/write from emulated device doesn't work for
> VF.
> > > > > > > > > For
> > > > > > > > example:
> > > > > > > > > Driver provides me a buffer pointer through MMIO write,
> > > > > > > > > this address
> > > > > > > > (pointer) is GPA of L2, and when I try to call
> > > > > > > > pci_dma_read() with this address I get:
> > > > > > > > > "
> > > > > > > > > Unassigned mem read 0000000000000000 "
> > > > > > > >
> > > > > > > > I don't know where this error log was dumped but if it's
> > > > > > > > during DMA then I agree it can probably be related to vIOMMU.
> > > > > > > >
> > > > > > >
> > > > > > > This log is dumped from:
> > > > > > > memory.c: unassigned_mem_read()
> > > > > > >
> > > > > > > > > As I said, my problem now is in translation of L2 GPA
> > > > > > > > > provided by driver,
> > > > > > > > when I call DMA read/write for this address from VF.
> > > > > > > > > Any insights?
> > > > > > > >
> > > > > > > > I just noticed that you were using QEMU 2.12 [1].  If
> > > > > > > > that's the case, please rebase to the latest QEMU, at
> > > > > > > > least >=3.0 because there's major refactor of the shadow
> > > > > > > > logic during
> > > > > > > > 3.0 devel cycle
> > > > AFAICT.
> > > > > > > >
> > > > > > >
> > > > > > > Rebased to QEMU 3.1
> > > > > > > Now I see the address I'm trying to read from in log but
> > > > > > > still same
> > error:
> > > > > > > "
> > > > > > > Unassigned mem read 00000000f0481000 "
> > > > > > > What do you suggest?
> > > > > >
> > > > > > Would you please answer the questions that Knut asked?  Is it
> > > > > > working for L1 guest?  How about PF?
> > > > >
> > > > > Both VF and PF are working for L1 guest.
> > > > > I don't know how to passthrough PF to nested VM in hyper-v.
> > > >
> > > > On Linux passing through VFs and PFs are the same.
> > > > Maybe you can try passthrough with all Linux first? (first PF then VF) ?
> > > >
> > > > > I don't invoke VF manually in hyper-v and pass it through to
> > > > > nested VM. I use hyper-v manager to configure and provide a VF
> > > > > for nested VM (I can see the VF only in the nested VM).
> > > > >
> > > > > Did someone try to run emulated device in linux RH as nested L2
> > > > > where
> > > > > L1 is windows hyper-v? Does DMA read/write work for this
> > > > > emulated
> > > > device in this case?
> > > >
> > > > I have never tried that, I have only used Linux as L2, Windows
> > > > might be pickier about what it expects, so starting with Linux to
> > > > rule that out is probably a good idea.
> > >
> > > Will move to this solution after I/we give up 😊
> > >
> > > >
> > > > > >
> > > > > > You can also try to enable VT-d device log by appending:
> > > > > >
> > > > > >   -trace enable="vtd_*"
> > > > > >
> > > > > > In case it dumps anything useful for you.
> > >
> > > Here is the relevant dump (dev 01:00.01 is my VF):
> > > "
> > > vtd_inv_desc_cc_device context invalidate device 01:00.01
> > > vtd_ce_not_present Context entry bus 1 devfn 1 not present
> > > vtd_switch_address_space Device 01:00.1 switching address space
> > > (iommu
> > > enabled=1) vtd_ce_not_present Context entry bus 1 devfn 1 not
> > > present vtd_err Detected invalid context entry when trying to sync
> > > shadow page table
> >
> > These lines mean that the guest sent a device invalidation to your VF
> > but the IOMMU found that the device context entry is missing.
> >
> > > vtd_iotlb_cc_update IOTLB context update bus 0x1 devfn 0x1 high
> > > 0x102 low 0x2d007003 gen 0 -> gen 2 vtd_err_dmar_slpte_resv_error
> > > iova
> > > 0xf08e7000 level 2 slpte 0x2a54008f7
> >
> > This line should not exist in latest QEMU.  Are you sure you're using
> > the latest QEMU?
> 
> I moved now to QEMU 4.0 RC2.
> This is the what I get now:
> vtd_iotlb_cc_update IOTLB context update bus 0x1 devfn 0x1 high 0x102 low
> 0x2f007003 gen 0 -> gen 1
> qemu-system-x86_64: vtd_iova_to_slpte: detected splte reserve non-zero
> iova=0xf0d29000, level=0x2slpte=0x29f6008f7) vtd_fault_disabled Fault
> processing disabled for context entry
> qemu-system-x86_64: vtd_iommu_translate: detected translation failure
> (dev=01:00:01, iova=0xf0d29000) Unassigned mem read 00000000f0d29000
> 
> I'm not familiar with vIOMMU registers, but I noticed that I must report
> snoop control support to Hyper-V (i.e. bit 7 in extended capability register 
> of
> vIOMMU) in-order to satisfy IOMMU support for SRIOV.
> vIOMMU.ecap before    0xf00f5e
> vIOMMU.ecap after       0xf00fde
> But I see that vIOMMU doesn't really support snoop control.
> Could this be the problem that fails IOVA range check in this function
> vtd_iova_range_check()?

Sorry, I meant the SLPTE reserved non-zero check failure in 
vtd_slpte_nonzero_rsvd()
And NOT IOVA range check failure (since range check didn't fail)

> If so, is there a way to work-around this issue?
> 
> >
> > I agree with Knut that I would suggest you to use use Linux in both
> > host and guests first to triage the issue.
> >
> > Regards,
> >
> > --
> > Peter Xu