Re: [Qemu-devel] [RFC PATCH v2 3/3] VFIO: Type1 IOMMU mapping support for vGPU

[Alex Williamson]

On Fri, 11 Mar 2016 04:46:23 +0000
"Tian, Kevin" <address@hidden> wrote:

> > From: Neo Jia [mailto:address@hidden
> > Sent: Friday, March 11, 2016 12:20 PM
> > 
> > On Thu, Mar 10, 2016 at 11:10:10AM +0800, Jike Song wrote:  
> > >  
> > > >> Is it supposed to be the caller who should set
> > > >> up IOMMU by DMA api such as dma_map_page(), after calling
> > > >> vgpu_dma_do_translate()?
> > > >>  
> > > >
> > > > Don't think you need to call dma_map_page here. Once you have the pfn 
> > > > available
> > > > to your GPU kernel driver, you can just go ahead to setup the mapping 
> > > > as you
> > > > normally do such as calling pci_map_sg and its friends.
> > > >  
> > >
> > > Technically it's definitely OK to call DMA API from the caller rather 
> > > than here,
> > > however personally I think it is a bit counter-intuitive: IOMMU page 
> > > tables
> > > should be constructed within the VFIO IOMMU driver.
> > >  
> > 
> > Hi Jike,
> > 
> > For vGPU, what we have is just a virtual device and a fake IOMMU group, 
> > therefore
> > the actual interaction with the real GPU should be managed by the GPU 
> > vendor driver.
> >   
> 
> Hi, Neo,
> 
> Seems we have a different thought on this. Regardless of whether it's a 
> virtual/physical 
> device, imo, VFIO should manage IOMMU configuration. The only difference is:
> 
> - for physical device, VFIO directly invokes IOMMU API to set IOMMU entry 
> (GPA->HPA);
> - for virtual device, VFIO invokes kernel DMA APIs which indirectly lead to 
> IOMMU entry 
> set if CONFIG_IOMMU is enabled in kernel (GPA->IOVA);
> 
> This would provide an unified way to manage the translation in VFIO, and then 
> vendor
> specific driver only needs to query and use returned IOVA corresponding to a 
> GPA. 
> 
> Doing so has another benefit, to make underlying vGPU driver VMM agnostic. 
> For KVM,
> yes we can use pci_map_sg. However for Xen it's different (today Dom0 doesn't 
> see
> IOMMU. In the future there'll be a PVIOMMU implementation) so different code 
> path is 
> required. It's better to abstract such specific knowledge out of vGPU driver, 
> which just
> uses whatever dma_addr returned by other agent (VFIO here, or another Xen 
> specific
> agent) in a centralized way.
> 
> Alex, what's your opinion on this?

The sticky point is how vfio, which is only handling the vGPU, has a
reference to the physical GPU on which to call DMA API operations.  If
that reference is provided by the vendor vGPU driver, for example
vgpu_dma_do_translate_for_pci(gpa, pci_dev), I don't see any reason to
be opposed to such an API.  I would not condone vfio deriving or owning
a reference to the physical device on its own though, that's in the
realm of the vendor vGPU driver.  It does seem a bit cleaner and should
reduce duplicate code if the vfio vGPU iommu interface could handle the
iommu mapping for the vendor vgpu driver when necessary.  Thanks,

Alex