Re: [Qemu-devel] rfc: vhost user enhancements for vm2vm communication

[Jan Kiszka]

On 2015-09-03 10:08, Michael S. Tsirkin wrote:
> On Tue, Sep 01, 2015 at 06:28:28PM +0200, Jan Kiszka wrote:
>> On 2015-09-01 18:02, Michael S. Tsirkin wrote:
>>> On Tue, Sep 01, 2015 at 05:34:37PM +0200, Jan Kiszka wrote:
>>>> On 2015-09-01 16:34, Michael S. Tsirkin wrote:
>>>>> On Tue, Sep 01, 2015 at 04:09:44PM +0200, Jan Kiszka wrote:
>>>>>> On 2015-09-01 11:24, Michael S. Tsirkin wrote:
>>>>>>> On Tue, Sep 01, 2015 at 11:11:52AM +0200, Jan Kiszka wrote:
>>>>>>>> On 2015-09-01 10:01, Michael S. Tsirkin wrote:
>>>>>>>>> On Tue, Sep 01, 2015 at 09:35:21AM +0200, Jan Kiszka wrote:
>>>>>>>>>> Leaving all the implementation and interface details aside, this
>>>>>>>>>> discussion is first of all about two fundamentally different 
>>>>>>>>>> approaches:
>>>>>>>>>> static shared memory windows vs. dynamically remapped shared windows 
>>>>>>>>>> (a
>>>>>>>>>> third one would be copying in the hypervisor, but I suppose we all 
>>>>>>>>>> agree
>>>>>>>>>> that the whole exercise is about avoiding that). Which way do we 
>>>>>>>>>> want or
>>>>>>>>>> have to go?
>>>>>>>>>>
>>>>>>>>>> Jan
>>>>>>>>>
>>>>>>>>> Dynamic is a superset of static: you can always make it static if you
>>>>>>>>> wish. Static has the advantage of simplicity, but that's lost once you
>>>>>>>>> realize you need to invent interfaces to make it work.  Since we can 
>>>>>>>>> use
>>>>>>>>> existing IOMMU interfaces for the dynamic one, what's the 
>>>>>>>>> disadvantage?
>>>>>>>>
>>>>>>>> Complexity. Having to emulate even more of an IOMMU in the hypervisor
>>>>>>>> (we already have to do a bit for VT-d IR in Jailhouse) and doing this
>>>>>>>> per platform (AMD IOMMU, ARM SMMU, ...) is out of scope for us. In that
>>>>>>>> sense, generic grant tables would be more appealing.
>>>>>>>
>>>>>>> That's not how we do things for KVM, PV features need to be
>>>>>>> modular and interchangeable with emulation.
>>>>>>
>>>>>> I know, and we may have to make some compromise for Jailhouse if that
>>>>>> brings us valuable standardization and broad guest support. But we will
>>>>>> surely not support an arbitrary amount of IOMMU models for that reason.
>>>>>>
>>>>>>>
>>>>>>> If you just want something that's cross-platform and easy to
>>>>>>> implement, just build a PV IOMMU. Maybe use virtio for this.
>>>>>>
>>>>>> That is likely required to keep the complexity manageable and to allow
>>>>>> static preconfiguration.
>>>>>
>>>>> Real IOMMU allow static configuration just fine. This is exactly
>>>>> what VFIO uses.
>>>>
>>>> Please specify more precisely which feature in which IOMMU you are
>>>> referring to. Also, given that you refer to VFIO, I suspect we have
>>>> different thing in mind. I'm talking about an IOMMU device model, like
>>>> the one we have in QEMU now for VT-d. That one is not at all
>>>> preconfigured by the host for VFIO.
>>>
>>> I really just mean that VFIO creates a mostly static IOMMU configuration.
>>>
>>> It's configured by the guest, not the host.
>>
>> OK, that resolves my confusion.
>>
>>>
>>> I don't see host control over configuration as being particularly important.
>>
>> We do, see below.
>>
>>>
>>>
>>>>>
>>>>>> Well, we could declare our virtio-shmem device to be an IOMMU device
>>>>>> that controls access of a remote VM to RAM of the one that owns the
>>>>>> device. In the static case, this access may at most be enabled/disabled
>>>>>> but not moved around. The static regions would have to be discoverable
>>>>>> for the VM (register read-back), and the guest's firmware will likely
>>>>>> have to declare those ranges reserved to the guest OS.
>>>>>> In the dynamic case, the guest would be able to create an alternative
>>>>>> mapping.
>>>>>
>>>>>
>>>>> I don't think we want a special device just to support the
>>>>> static case. It might be a bit less code to write, but
>>>>> eventually it should be up to the guest.
>>>>> Fundamentally, it's policy that host has no business
>>>>> dictating.
>>>>
>>>> "A bit less" is to be validated, and I doubt its just "a bit". But if
>>>> KVM and its guests will also support some PV-IOMMU that we can reuse for
>>>> our scenarios, than that is fine. KVM would not have to mandate support
>>>> for it while we would, that's all.
>>>
>>> Someone will have to do this work.
>>>
>>>>>
>>>>>> We would probably have to define a generic page table structure
>>>>>> for that. Or do you rather have some MPU-like control structure in mind,
>>>>>> more similar to the memory region descriptions vhost is already using?
>>>>>
>>>>> I don't care much. Page tables use less memory if a lot of memory needs
>>>>> to be covered. OTOH if you want to use virtio (e.g. to allow command
>>>>> batching) that likely means commands to manipulate the IOMMU, and
>>>>> maintaining it all on the host. You decide.
>>>>
>>>> I don't care very much about the dynamic case as we won't support it
>>>> anyway. However, if the configuration concept used for it is applicable
>>>> to static mode as well, then we could reuse it. But preconfiguration
>>>> will required register-based region description, I suspect.
>>>
>>> I don't know what you mean by preconfiguration exactly.
>>>
>>> Do you want the host to configure the IOMMU? Why not let the
>>> guest do this?
>>
>> We simply freeze GPA-to-HPA mappings during runtime. Avoids having to
>> validate and synchronize guest-triggered changes.
> 
> Fine, but this assumes guest does very specific things, right?
> E.g. should guest reconfigure device's BAR, you would have
> to change GPA to HPA mappings?
> 

Yes, that's why we only support size exploration, not reallocation.

> 
>>>>>
>>>>>> Also not yet clear to me are how the vhost-pci device and the
>>>>>> translations it will have to do should look like for VM2.
>>>>>
>>>>> I think we can use vhost-pci BAR + VM1 bus address as the
>>>>> VM2 physical address. In other words, all memory exposed to
>>>>> virtio-pci by VM1 through it's IOMMU is mapped into BAR of
>>>>> vhost-pci.
>>>>>
>>>>> Bus addresses can be validated to make sure they fit
>>>>> in the BAR.
>>>>
>>>> Sounds simple but may become challenging for VMs that have many of such
>>>> devices (in order to connect to many possibly large VMs).
>>>
>>> You don't need to be able to map all guest memory if you know
>>> guest won't try to allow device access to all of it.
>>> It's a question of how good is the bus address allocator.
>>
>> But those BARs need to allocate a guest-physical address range as large
>> as the other guest's RAM is, possibly even larger if that RAM is not
>> contiguous, and you can't put other resources into potential holes
>> because VM2 does not know where those holes will be.
> 
> No - only the RAM that you want addressable by VM2.

That's in the hand of VM1, not VM2 or the hypervisor, in case of
reconfigurable mapping. It's indeed a non-issue in our static case.

> 
> IOW if you wish, you actually can create a shared memory device,
> make it accessible to the IOMMU and place some or all
> data there.
> 

Actually, that could also be something more sophisticated, including
virtio-net, IF that device will be able to express its DMA window
restrictions (a bit like 32-bit PCI devices being restricted to <4G
addresses or ISA devices <1M).

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux