Re: [Qemu-devel] [RFC PATCH] vfio: VFIO Driver core framework

[Scott Wood]

On 12/02/2011 08:40 AM, Stuart Yoder wrote:
> On Thu, Dec 1, 2011 at 3:25 PM, Alex Williamson
> <address@hidden> wrote:
>> On Thu, 2011-12-01 at 14:58 -0600, Stuart Yoder wrote:
>>> One other mechanism we need as well is the ability to
>>> enable/disable a domain.
>>>
>>> For example-- suppose a device is assigned to a VM, the
>>> device is in use when the VM is abruptly terminated.  The
>>> VM terminate would shut off DMA at the IOMMU, but now
>>> the device is in an indeterminate state.   Some devices
>>> have no simple reset bit and getting the device back into
>>> a sane state could be complicated-- something the hypervisor
>>> doesn't want to do.
>>>
>>> So now KVM restarts the VM, vfio init happens for the device
>>> and  the IOMMU for that device is re-configured,
>>> etc, but we really can't re-enable DMA until the guest OS tells us
>>> (via an hcall) that it is ready.   The guest needs to get the
>>> assigned device in a sane state before DMA is enabled.
>>
>> Giant red flag.  We need to paravirtualize the guest?  Not on x86.
> 
> It's the reality we have to deal with, but doing this would obviously
> only apply to platforms that need it.

By "x86" I assume you mean "PCI" and thus a bus-master enable flag that
you rely on the guest not setting until the device has been reset or
otherwise quiesced from any previous activity, in the absence of
function-level reset.

We don't have such a thing on our non-PCI devices.

>> Some
>> devices are better for assignment than others.  PCI devices are moving
>> towards supporting standard reset mechanisms.
>>
>>> Does this warrant a new domain enable/disable API, or should
>>> we make this part of the setup API we are discussing
>>> here?
>>
>> What's wrong with simply not adding any DMA mapping entries until you
>> think your guest is ready?  Isn't that effectively the same thing?
>> Unmap ~= disable.  If the IOMMU API had a mechanism to toggle the iommu
>> domain on and off, I wouldn't be opposed to adding an ioctl to do it,
>> but it really seems like just a shortcut vs map/unmap.  Thanks,
> 
> Yes, we could do something like that I guess.

It would mean that we don't see any errors relating to impossible map
requests until after the guest is running and decides to enable DMA.
Depending on how PAMU table allocation is handled, it could introduce a
risk of failing even later when a guest reboots and we need to
temporarily disable DMA (e.g. if another vfio user consumes the same
table space for another group in the meantime).

It would add latency to failovers -- some customers have somewhat tight
requirements there.

-Scott