Re: [Qemu-devel] vfio for platform devices - 9/5/2012 - minutes

[Bhushan Bharat-R65777]

> -----Original Message-----
> From: Alex Williamson [mailto:address@hidden
> Sent: Wednesday, September 11, 2013 10:45 PM
> To: Yoder Stuart-B08248
> Cc: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777; 'Peter
> Maydell'; 'Santosh Shukla'; 'Alexander Graf'; 'Antonios Motakis'; 'Christoffer
> Dall'; 'address@hidden'; address@hidden; kvm-
> address@hidden; address@hidden
> Subject: Re: vfio for platform devices - 9/5/2012 - minutes
> 
> On Wed, 2013-09-11 at 16:42 +0000, Yoder Stuart-B08248 wrote:
> >
> > > -----Original Message-----
> > > From: Yoder Stuart-B08248
> > > Sent: Thursday, September 05, 2013 12:51 PM
> > > To: Wood Scott-B07421; Sethi Varun-B16395; Bhushan Bharat-R65777;
> > > 'Peter Maydell'; 'Santosh Shukla'; 'Alex Williamson'; 'Alexander
> > > Graf'; 'Antonios Motakis'; 'Christoffer Dall'; 'address@hidden'
> > > Cc: address@hidden; 'address@hidden'; 'qemu-
> > > address@hidden'
> > > Subject: vfio for platform devices - 9/5/2012 - minutes
> > >
> > > We had a call with those interested and/or working on vfio for
> > > platform devices.
> > >
> > > Participants: Scott Wood, Varun Sethi, Bharat Bhushan, Peter Maydell,
> > >               Santosh Shukla, Alex Williamson, Alexander Graf,
> > >               Antonios Motakis, Christoffer Dall, Kim Phillips,
> > >               Stuart Yoder
> > >
> > > Several aspects to vfio for platform devices:
> > >
> > > 1. IOMMU groups
> > >
> > >  -iommu driver needs to register a bus notifier for the platform bus
> > >   and create groups for relevant platform devices  -Antonios is
> > > looking at this for several ARM IOMMUs  -PAMU (Freescale) driver
> > > already does this
> > >
> > > 2. unbinding device from host
> > >
> > >  PCI:
> > >    echo 0000:06:0d.0 >
> > > /sys/bus/pci/devices/0000:06:0d.0/driver/unbind
> > >  Platform:
> > >    echo ffe101300.dma >
> > > /sys/bus/platform/devices/ffe101300.dma/driver/unbind
> > >
> > >  -don't believe there are issues or work to do here
> > >
> > > 3. binding device to vfio-platform driver
> > >
> > >  PCI:
> > >    echo 1102 0002 > /sys/bus/pci/drivers/vfio-pci/new_id
> > >
> > >  -this is probably the least understood issue-- platform drivers
> > >   register themselves with the bus for a specific "name"
> > >   string.  That is matched with device tree "compatible" strings
> > >   later to bind a device to a driver  -we want is to have the
> > > vfio-platform driver to dynamically bind
> > >   to a variety of platform devices previously unknown to
> > >   vfio-platform
> > >  -ideally unbinding and binding could be an atomic operation  -Alex
> > > W pointed out that x86 could leverage this work so
> > >   keep that in mind in what we design  -Kim Phillips (Linaro) will
> > > start working on this
> >
> > One thing we didn't discuss needs to be considered (probably by Kim
> > who is looking at the 'binding device' issue) is around returning a
> > passthru device back to the host.
> >
> > After a platform device has been bound to vfio and is in use by user
> > space or a virtual machine, we also need to be able to unwind all that
> > and return the device back to the host in a sane state.
> >
> > What happens when user space exits and the vfio file descriptors are
> > closed?
> 
> For reference, expectations of how vfio-pci handles these situations:
> 
> For vfio-pci, when the reference count on the device fd drops to zero we call 
> a
> device disable function that includes disabling the bus master bit in config
> space stop ongoing DMA.

There is no bus mastering for platform devices and device disabling is not a 
standard like PCI. Do you think that we might need to do some device specific 
handling. For example for DMA controller, we need to atleast disable the DMA 
controller and mask its interrupts (may be ensure that there is no 
pending/running dma transaction, release irqs etc). As we are not yet having 
any linkage to respective kernel driver then I am not sure how we will do that 
specific handling?

> 
> > What if the device is still active and doing bus
> > mastering?   (e.g. a VM crashed causing a QEMU
> > exit)
> 
> If the VM crashes the vfio fds get released resulting in the above opportunity
> for the vfio device driver to quiesce the device.

I think the quiesing of devices with we device specific, so the generic 
vfio=platform driver may not be able to handle that, right?

> 
> > How can the vfio-platform layer in the host kernel get a specific
> > device in a sane state?
> 
> It's not easy on pci either.  We save config space prior to exposing the 
> device
> and restore config space later, but it's not complete.  We mostly rely on 
> device
> (function) resets, to put things in a sane state, but those aren't always
> supported.

All platform devices also may not have reset capability (and if any then not 
generic way for all devices).

>  I just introduced patches for v3.12 that enable a PCI bus reset
> interface, but it's mostly useful for userspace, since on PCI it's often the
> case that a bus contains multiple devices which don't necessarily align to 
> iommu
> group boundaries.
> 
> > When a plaform device is 'unbound' from vfio, what specifically
> > happens to the device?
> 
> The driver remove function is called, which needs to call 
> vfio_del_group_dev(),
> which will block if the device is in use.  If it's unused, the release 
> function
> should have already been called and the device should be in a quiesced state.

What does it mean by quiesced state here? Does this mean:
 - Device does not take more work from software; This will done if device is 
not attached to any driver.
 - Device does not create work for software (no interrupt etc). This can 
achieved if interrupts masked at device/PIC and device in disabled.

Thanks
-Bharat

> On x86/pci, there is no auto probe of devices when unbound, so the device sits
> without a driver until manually bound or until something causes an auto probe.
> Thanks,
> 
> Alex
> 
> > Platform devices don't have generic mechanisms like on PCI to disable
> > bus mastering or even disable or reset a device.
> >
> > Haven't thought through all this yet, but just raising some issues I
> > see.
> >
> > Regards,
> > Stuart
> >
> >
> >
> >
> 
> 
>