Re: [Qemu-arm] [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device

[Bharat Bhushan]

Hi Eric,

> -----Original Message-----
> From: Auger Eric [mailto:address@hidden
> Sent: Monday, June 26, 2017 1:25 PM
> To: Bharat Bhushan <address@hidden>;
> address@hidden; address@hidden;
> address@hidden; address@hidden; address@hidden;
> address@hidden; address@hidden
> Cc: address@hidden; address@hidden; address@hidden;
> address@hidden; address@hidden; address@hidden;
> address@hidden; address@hidden
> Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device
> 
> Hi Bharat,
> 
> On 19/06/2017 09:54, Bharat Bhushan wrote:
> > Hi Eric,
> >
> > I started added replay in virtio-iommu and came across how MSI interrupts
> with work with VFIO.
> > I understand that on intel this works differently but vsmmu will have same
> requirement.
> > kvm-msi-irq-route are added using the msi-address to be translated by
> viommu and not the final translated address.
> > While currently the irqfd framework does not know about emulated
> iommus (virtio-iommu, vsmmuv3/vintel-iommu).
> > So in my view we have following options:
> > - Programming with translated address when setting up
> > kvm-msi-irq-route
> > - Route the interrupts via QEMU, which is bad from performance
> > - vhost-virtio-iommu may solve the problem in long term
> 
> Sorry for the delay. With regard to the vsmmuv3/vfio integration I think we
> need to use the guest physical address otherwise the MSI address will not be
> recognized as an MSI doorbell.
> 
> Also the fact on ARM we map the MSI doorbell causes an assert in
> vfio_get_vaddr() as the vITS doorbell is not a RAM region. We will need to
> handle this specifically.

Also when setup msi-route kvm_irqchip_add_msi_route() we needed to provide the 
translated address.
According to my understanding this is required because kernel does no go 
through viommu translation when generating interrupt, no? 

Thanks
-Bharat

> 
> Besides I have not looked specifically at the virtio-iommu/vfio integration
> yet.
> 
> Thanks
> 
> Eric
> >
> > Is there any other better option I am missing?
> >
> > Thanks
> > -Bharat
> >
> >> -----Original Message-----
> >> From: Auger Eric [mailto:address@hidden
> >> Sent: Friday, June 09, 2017 5:24 PM
> >> To: Bharat Bhushan <address@hidden>;
> >> address@hidden; address@hidden;
> >> address@hidden; address@hidden; address@hidden;
> >> address@hidden; address@hidden
> >> Cc: address@hidden; address@hidden; address@hidden;
> >> address@hidden; address@hidden; address@hidden;
> >> address@hidden; address@hidden
> >> Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device
> >>
> >> Hi Bharat,
> >>
> >> On 09/06/2017 13:30, Bharat Bhushan wrote:
> >>> Hi Eric,
> >>>
> >>>> -----Original Message-----
> >>>> From: Auger Eric [mailto:address@hidden
> >>>> Sent: Friday, June 09, 2017 12:14 PM
> >>>> To: Bharat Bhushan <address@hidden>;
> >>>> address@hidden; address@hidden;
> >>>> address@hidden; address@hidden; qemu-
> address@hidden;
> >>>> address@hidden; address@hidden
> >>>> Cc: address@hidden; address@hidden;
> >> address@hidden;
> >>>> address@hidden; address@hidden;
> >>>> address@hidden; address@hidden; address@hidden
> >>>> Subject: Re: [RFC v2 0/8] VIRTIO-IOMMU device
> >>>>
> >>>> Hi Bharat,
> >>>>
> >>>> On 09/06/2017 08:16, Bharat Bhushan wrote:
> >>>>> Hi Eric,
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Eric Auger [mailto:address@hidden
> >>>>>> Sent: Wednesday, June 07, 2017 9:31 PM
> >>>>>> To: address@hidden; address@hidden;
> >>>>>> address@hidden; address@hidden;
> >>>> address@hidden;
> >>>>>> address@hidden; address@hidden; jean-
> >>>>>> address@hidden
> >>>>>> Cc: address@hidden; address@hidden;
> >>>> address@hidden;
> >>>>>> address@hidden; address@hidden;
> >>>>>> address@hidden; address@hidden; address@hidden; Bharat
> >>>> Bhushan
> >>>>>> <address@hidden>
> >>>>>> Subject: [RFC v2 0/8] VIRTIO-IOMMU device
> >>>>>>
> >>>>>> This series implements the virtio-iommu device. This is a proof
> >>>>>> of concept based on the virtio-iommu specification written by
> >>>>>> Jean-Philippe
> >>>> Brucker [1].
> >>>>>> This was tested with a guest using the virtio-iommu driver [2]
> >>>>>> and exposed with a virtio-net-pci using dma ops.
> >>>>>>
> >>>>>> The device gets instantiated using the "-device virtio-iommu-device"
> >>>>>> option. It currently works with ARM virt machine only as the
> >>>>>> machine must handle the dt binding between the virtio-mmio
> "iommu"
> >>>>>> node and the PCI host bridge node. ACPI booting is not yet
> supported.
> >>>>>>
> >>>>>> This should allow to start some benchmarking activities against
> >>>>>> pure emulated IOMMU (especially ARM SMMU).
> >>>>>
> >>>>> I am testing this on ARM64 and see below continuous error prints:
> >>>>>
> >>>>>         virtio_iommu_translate sid=8 is not known!!
> >>>>>         virtio_iommu_translate sid=8 is not known!!
> >>>>>         virtio_iommu_translate sid=8 is not known!!
> >>>>>         virtio_iommu_translate sid=8 is not known!!
> >>>>>         virtio_iommu_translate sid=8 is not known!!
> >>>>>         virtio_iommu_translate sid=8 is not known!!
> >>>>>         virtio_iommu_translate sid=8 is not known!!
> >>>>>         virtio_iommu_translate sid=8 is not known!!
> >>>>>         virtio_iommu_translate sid=8 is not known!!
> >>>>>         virtio_iommu_translate sid=8 is not known!!
> >>>>>
> >>>>>
> >>>>> Also in guest I do not see device-tree node with virtio-iommu.
> >>>> do you mean the virtio-mmio with #iommu-cells property?
> >>>>
> >>>> This one is created statically by virt machine. I would be
> >>>> surprised if it were not there. Are you using the virt = virt2.10 
> >>>> machine.
> >>>> Machines before do not support its instantiation.
> >>>>
> >>>> Please can you add a printf in hw/arm/virt.c create_virtio_mmio()
> >>>> at the moment when this node is created. Also you can add a printf
> >>>> in
> >>>> bind_virtio_iommu_device() to make sure the binding with the PCI
> >>>> host bridge is added on machine init done.
> >>>>
> >>>> Also worth to check, CONFIG_VIRTIO_IOMMU=y on guest side.
> >>>
> >>> It works on my side.
> >> Great.
> >>
> >>  The driver config was disabled and also I was using guest kernel
> >> which was not have deferred-probing.
> >> Yes I did not mention in my cover letter the guest I have been using
> >> is based on Jean-Philippe's branch, featuring deferred IOMMU probing.
> >> I I have not tried yet with an upstream guest.
> >>  Now after fixing it works on my side
> >>> I placed some prints to see dma-map are mapping regions in
> >>> virtio-iommu,
> >> it uses emulated iommu.
> >>>
> >>> I will continue to add VFIO support now on this and more testing !!
> >>
> >> OK. I will do the VFIO integration first on the vsmmuv3 device as I
> >> already prepared the VFIO replay and hopefully we will sync ;-)
> >>
> >> Thanks
> >>
> >> Eric
> >>>
> >>> Thanks
> >>> -Bharat
> >>>
> >>>>
> >>>> Thanks
> >>>>
> >>>> Eric
> >>>>
> >>>>> I am using qemu-tree you mentioned below and iommu-driver
> patches
> >>>> published by Jean-P.
> >>>>> Qemu command line have additional ""-device virtio-iommu-device".
> >>>>> What
> >>>> I am missing ?
> >>>>
> >>>>
> >>>>>
> >>>>> Thanks
> >>>>> -Bharat
> >>>>>
> >>>>>>
> >>>>>> Best Regards
> >>>>>>
> >>>>>> Eric
> >>>>>>
> >>>>>> This series can be found at:
> >>>>>> https://github.com/eauger/qemu/tree/virtio-iommu-rfcv2
> >>>>>>
> >>>>>> References:
> >>>>>> [1] [RFC 0/3] virtio-iommu: a paravirtualized IOMMU, [2] [RFC
> >>>>>> PATCH linux]
> >>>>>> iommu: Add virtio-iommu driver [3] [RFC PATCH kvmtool 00/15] Add
> >>>>>> virtio- iommu
> >>>>>>
> >>>>>> History:
> >>>>>> v1 -> v2:
> >>>>>> - fix redifinition of viommu_as typedef
> >>>>>>
> >>>>>> Eric Auger (8):
> >>>>>>   update-linux-headers: import virtio_iommu.h
> >>>>>>   linux-headers: Update for virtio-iommu
> >>>>>>   virtio_iommu: add skeleton
> >>>>>>   virtio-iommu: Decode the command payload
> >>>>>>   virtio_iommu: Add the iommu regions
> >>>>>>   virtio-iommu: Implement the translation and commands
> >>>>>>   hw/arm/virt: Add 2.10 machine type
> >>>>>>   hw/arm/virt: Add virtio-iommu the virt board
> >>>>>>
> >>>>>>  hw/arm/virt.c                                 | 116 ++++-
> >>>>>>  hw/virtio/Makefile.objs                       |   1 +
> >>>>>>  hw/virtio/trace-events                        |  14 +
> >>>>>>  hw/virtio/virtio-iommu.c                      | 623
> >>>> ++++++++++++++++++++++++++
> >>>>>>  include/hw/arm/virt.h                         |   5 +
> >>>>>>  include/hw/virtio/virtio-iommu.h              |  60 +++
> >>>>>>  include/standard-headers/linux/virtio_ids.h   |   1 +
> >>>>>>  include/standard-headers/linux/virtio_iommu.h | 142 ++++++
> >>>>>>  linux-headers/linux/virtio_iommu.h            |   1 +
> >>>>>>  scripts/update-linux-headers.sh               |   3 +
> >>>>>>  10 files changed, 957 insertions(+), 9 deletions(-)  create mode
> >>>>>> 100644 hw/virtio/virtio-iommu.c  create mode 100644
> >>>>>> include/hw/virtio/virtio- iommu.h  create mode 100644
> >>>>>> include/standard- headers/linux/virtio_iommu.h  create mode
> >>>>>> 100644 linux-headers/linux/virtio_iommu.h
> >>>>>>
> >>>>>> --
> >>>>>> 2.5.5
> >>>>>
> >>>
> >