Re: [Qemu-devel] intel-iommu and vhost: Do we need 'device-iotlb' and 'ats'?

[Jintack Lim]

On Fri, Feb 23, 2018 at 2:09 AM, Peter Xu <address@hidden> wrote:
> On Fri, Feb 23, 2018 at 06:34:04AM +0000, Jintack Lim wrote:
>> On Fri, Feb 23, 2018 at 1:10 AM Peter Xu <address@hidden> wrote:
>>
>> > On Fri, Feb 23, 2018 at 12:32:13AM -0500, Jintack Lim wrote:
>> > > Hi Peter,
>> > >
>> > > Hope you had great holidays!
>> > >
>> > > On Thu, Feb 22, 2018 at 10:55 PM, Peter Xu <address@hidden> wrote:
>> > > > On Tue, Feb 20, 2018 at 11:03:46PM -0500, Jintack Lim wrote:
>> > > >> Hi,
>> > > >>
>> > > >> I'm using vhost with the virtual intel-iommu, and this page[1] shows
>> > > >> the QEMU command line example.
>> > > >>
>> > > >> qemu-system-x86_64 -M q35,accel=kvm,kernel-irqchip=split -m 2G \
>> > > >>                    -device intel-iommu,intremap=on,device-iotlb=on \
>> > > >>                    -device ioh3420,id=pcie.1,chassis=1 \
>> > > >>                    -device
>> > > >>
>> > virtio-net-pci,bus=pcie.1,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on
>> > > >> \
>> > > >>                    -netdev tap,id=net0,vhostforce \
>> > > >>                    $IMAGE_PATH
>> > > >>
>> > > >> I wonder what's the impact of using device-iotlb and ats options as
>> > > >> they are described necessary.
>> > > >>
>> > > >> In my understanding, vhost in the kernel only looks at
>> > > >> VIRTIO_F_IOMMU_PLATFORM, and when it is set, vhost uses a
>> > > >> device-iotlb. In addition, vhost and QEMU communicate using vhost_msg
>> > > >> basically to cache mappings correctly in the vhost, so I wonder what's
>> > > >> the role of ats in this case.
>> > > >
>> > > > The "ats" as virtio device parameter will add ATS capability to the
>> > > > PCI device.
>> > > >
>> > > > The "device-iotlb" as intel-iommu parameter will enable ATS in the
>> > > > IOMMU device (and also report that in ACPI field).
>> > > >
>> > > > If both parameters are provided IIUC it means guest will know virtio
>> > > > device has device-iotlb and it'll treat the device specially (e.g.,
>> > > > guest will need to send device-iotlb invalidations).
>> > >
>> > > Oh, I see. I was focusing on how QEMU and vhost work in the host, but
>> > > I think I missed the guest part! Thanks. I see that the Intel IOMMU
>> > > driver has has_iotlb_device flag for that purpose.
>> > >
>> > > >
>> > > > We'd better keep these parameters when running virtio devices with
>> > > > vIOMMU.  For the rest of vhost/arm specific questions, I'll leave to
>> > > > others.
>> > >
>> > > It seems like SMMU is not checking ATS capability - at least
>> > > ats_enabled flag - but I may miss something here as well :)
>> > >
>> > > >
>> > > > PS: Though IIUC the whole ATS thing may not really be necessary for
>> > > > current VT-d emulation, since even with ATS vhost is registering UNMAP
>> > > > IOMMU notifiers (see vhost_iommu_region_add()), and IIUC that means
>> > > > vhost will receive IOTLB invalidations even without ATS support, and
>> > > > it _might_ still work.
>> > >
>> > > Right. That's what I thought.
>> > >
>> > > Come to think of it, I'm not sure why we need to flush mappings in
>> > > IOMMU and devices separately in the first place... Any thoughts?
>> >
>> > I don't know ATS much, neither.
>> >
>> > You can have a look at chap 4 of vt-d spec:
>> >
>> >         One approach to scaling IOTLBs is to enable I/O devices to
>> >         participate in the DMA remapping with IOTLBs implemented at
>> >         the devices. The Device-IOTLBs alleviate pressure for IOTLB
>> >         resources in the core logic, and provide opportunities for
>> >         devices to improve performance by pre-fetching address
>> >         translations before issuing DMA requests. This may be useful
>> >         for devices with strict DMA latency requirements (such as
>> >         isochronous devices), and for devices that have large DMA
>> >         working set or multiple active DMA streams.
>> >
>> > So I think it's for performance's sake. For example, the DMA operation
>> > won't need to be translated at all if it's pre-translated, so it can
>> > have less latency.  And also, that'll offload some of the translation
>> > process so that workload can be more distributed.
>> >
>> > When with that (caches located both on IOMMU's and device's side), we
>> > need to invalidate all the cache when needed.
>> >
>>
>> Right. I think my question was not clear. My question was that why don’t
>> IOMMU invalidate device-iotlb along with its mappings in one go. Then IOMMU
>> device driver doesn’t need to flush device-iotlb explicitly. Maybe the
>> reason is that ATS and IOMMU are not always coupled.. but I guess it’s time
>> for me to get some more background :)
>
> Ah, I see your point.
>
> I don't know the answer.  My wild guess is that IOMMU is just trying
> to be simple and only provide most basic functionalities, leaving
> complex stuff to CPU.  For example, if IOMMU takes over the ownership
> to deliever device-iotlb invalidations when receiving iotlb
> invalidations, it possibly needs to traverse the device tree sometimes
> (e.g., for domain invalidations) to know what device is under what
> domain, which is really compliated.  While it'll be simpler for CPU to
> do this since it's very possible that the OS keeps a list of devices
> for a domain already.
>
> IMHO that follows the *nix philosophy too - Do One Thing And Do It
> Well.  Though again, it's wild guess and I may be wrong. :)

Cool. Makes sense to me!

>
> CCing Alex, in case he has quick answers.
>
>>
>>
>> > >
>> > > Your reply was really helpful to me. I appreciate it.
>> >
>> > My pleasure.  Thanks,
>> >
>> > >
>> > > Thanks,
>> > > Jintack
>> > >
>> > > > But there can be other differences, like
>> > > > performance, etc.
>> > > >
>> > > >>
>> > > >> A related question is that if we use SMMU emulation[2] on ARM without
>> > > >> those options, does vhost cache mappings as if it has a device-iotlb?
>> > > >> (I guess this is the case.)
>> > > >>
>> > > >> I'm pretty new to QEMU code, so I might be missing something. Can
>> > > >> somebody shed some light on it?
>> > > >>
>> > > >> [1] https://wiki.qemu.org/Features/VT-d
>> > > >> [2]
>> > http://lists.nongnu.org/archive/html/qemu-devel/2018-02/msg04736.html
>> > > >>
>> > > >> Thanks,
>> > > >> Jintack
>> > > >>
>> > > >
>> > > > --
>> > > > Peter Xu
>> > > >
>> > >
>> >
>> > --
>> > Peter Xu
>> >
>> >
>
> --
> Peter Xu
>