Re: [Qemu-arm] [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device

[Bharat Bhushan]

Hi Eric,

> -----Original Message-----
> From: Auger Eric [mailto:address@hidden
> Sent: Wednesday, July 05, 2017 2:14 PM
> To: Bharat Bhushan <address@hidden>;
> address@hidden; address@hidden;
> address@hidden; address@hidden; address@hidden;
> address@hidden; address@hidden
> Cc: address@hidden; address@hidden; address@hidden;
> address@hidden; address@hidden; address@hidden;
> address@hidden; address@hidden
> Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device
> 
> Hi Bharat,
> 
> On 05/07/2017 10:23, Bharat Bhushan wrote:
> > Hi Eric,
> >
> >> -----Original Message-----
> >> From: Auger Eric [mailto:address@hidden
> >> Sent: Monday, June 26, 2017 1:25 PM
> >> To: Bharat Bhushan <address@hidden>;
> >> address@hidden; address@hidden;
> >> address@hidden; address@hidden; address@hidden;
> >> address@hidden; address@hidden
> >> Cc: address@hidden; address@hidden; address@hidden;
> >> address@hidden; address@hidden; address@hidden;
> >> address@hidden; address@hidden
> >> Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device
> >>
> >> Hi Bharat,
> >>
> >> On 19/06/2017 09:54, Bharat Bhushan wrote:
> >>> Hi Eric,
> >>>
> >>> I started added replay in virtio-iommu and came across how MSI
> >>> interrupts
> >> with work with VFIO.
> >>> I understand that on intel this works differently but vsmmu will
> >>> have same
> >> requirement.
> >>> kvm-msi-irq-route are added using the msi-address to be translated
> >>> by
> >> viommu and not the final translated address.
> >>> While currently the irqfd framework does not know about emulated
> >> iommus (virtio-iommu, vsmmuv3/vintel-iommu).
> >>> So in my view we have following options:
> >>> - Programming with translated address when setting up
> >>> kvm-msi-irq-route
> >>> - Route the interrupts via QEMU, which is bad from performance
> >>> - vhost-virtio-iommu may solve the problem in long term
> >>
> >> Sorry for the delay. With regard to the vsmmuv3/vfio integration I
> >> think we need to use the guest physical address otherwise the MSI
> >> address will not be recognized as an MSI doorbell.
> >>
> >> Also the fact on ARM we map the MSI doorbell causes an assert in
> >> vfio_get_vaddr() as the vITS doorbell is not a RAM region. We will
> >> need to handle this specifically.
> >
> > Also when setup msi-route kvm_irqchip_add_msi_route() we needed to
> provide the translated address.
> > According to my understanding this is required because kernel does no go
> through viommu translation when generating interrupt, no?
> 
> yes this is needed when KVM MSI routes are set up, ie. along with GICV3 ITS.
> With GICv2M, qemu direct gsi mapping is used and this is not needed.
> 
> So I do not understand your previous sentence saying "MSI interrupts works
> without any change".

I have almost completed vfio integration with virtio-iommu and now testing the 
changes by assigning e1000 device to VM. For this I have changed virtio-iommu 
driver to use IOMMU_RESV_MSI rather than sw-msi and this does not need changed 
in vfio_get_addr()  and kvm_irqchip_add_msi_route()

Thanks
-Bharat

> 
> Thanks
> 
> Eric
> 
> >
> > Thanks
> > -Bharat
> >
> >>
> >> Besides I have not looked specifically at the virtio-iommu/vfio
> >> integration yet.
> >>
> >> Thanks
> >>
> >> Eric
> >>>
> >>> Is there any other better option I am missing?
> >>>
> >>> Thanks
> >>> -Bharat
> >>>
> >>>> -----Original Message-----
> >>>> From: Auger Eric [mailto:address@hidden
> >>>> Sent: Friday, June 09, 2017 5:24 PM
> >>>> To: Bharat Bhushan <address@hidden>;
> >>>> address@hidden; address@hidden;
> >>>> address@hidden; address@hidden; qemu-
> address@hidden;
> >>>> address@hidden; address@hidden
> >>>> Cc: address@hidden; address@hidden; address@hidden;
> >>>> address@hidden; address@hidden; address@hidden;
> >>>> address@hidden; address@hidden
> >>>> Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device
> >>>>
> >>>> Hi Bharat,
> >>>>
> >>>> On 09/06/2017 13:30, Bharat Bhushan wrote:
> >>>>> Hi Eric,
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Auger Eric [mailto:address@hidden
> >>>>>> Sent: Friday, June 09, 2017 12:14 PM
> >>>>>> To: Bharat Bhushan <address@hidden>;
> >>>>>> address@hidden; address@hidden;
> >>>>>> address@hidden; address@hidden; qemu-
> >> address@hidden;
> >>>>>> address@hidden; address@hidden
> >>>>>> Cc: address@hidden; address@hidden;
> >>>> address@hidden;
> >>>>>> address@hidden; address@hidden;
> >>>>>> address@hidden; address@hidden; address@hidden
> >>>>>> Subject: Re: [RFC v2 0/8] VIRTIO-IOMMU device
> >>>>>>
> >>>>>> Hi Bharat,
> >>>>>>
> >>>>>> On 09/06/2017 08:16, Bharat Bhushan wrote:
> >>>>>>> Hi Eric,
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Eric Auger [mailto:address@hidden
> >>>>>>>> Sent: Wednesday, June 07, 2017 9:31 PM
> >>>>>>>> To: address@hidden; address@hidden;
> >>>>>>>> address@hidden; address@hidden;
> >>>>>> address@hidden;
> >>>>>>>> address@hidden; address@hidden; jean-
> >>>>>>>> address@hidden
> >>>>>>>> Cc: address@hidden; address@hidden;
> >>>>>> address@hidden;
> >>>>>>>> address@hidden; address@hidden;
> >>>>>>>> address@hidden; address@hidden; address@hidden;
> Bharat
> >>>>>> Bhushan
> >>>>>>>> <address@hidden>
> >>>>>>>> Subject: [RFC v2 0/8] VIRTIO-IOMMU device
> >>>>>>>>
> >>>>>>>> This series implements the virtio-iommu device. This is a proof
> >>>>>>>> of concept based on the virtio-iommu specification written by
> >>>>>>>> Jean-Philippe
> >>>>>> Brucker [1].
> >>>>>>>> This was tested with a guest using the virtio-iommu driver [2]
> >>>>>>>> and exposed with a virtio-net-pci using dma ops.
> >>>>>>>>
> >>>>>>>> The device gets instantiated using the "-device virtio-iommu-
> device"
> >>>>>>>> option. It currently works with ARM virt machine only as the
> >>>>>>>> machine must handle the dt binding between the virtio-mmio
> >> "iommu"
> >>>>>>>> node and the PCI host bridge node. ACPI booting is not yet
> >> supported.
> >>>>>>>>
> >>>>>>>> This should allow to start some benchmarking activities against
> >>>>>>>> pure emulated IOMMU (especially ARM SMMU).
> >>>>>>>
> >>>>>>> I am testing this on ARM64 and see below continuous error prints:
> >>>>>>>
> >>>>>>>       virtio_iommu_translate sid=8 is not known!!
> >>>>>>>       virtio_iommu_translate sid=8 is not known!!
> >>>>>>>       virtio_iommu_translate sid=8 is not known!!
> >>>>>>>       virtio_iommu_translate sid=8 is not known!!
> >>>>>>>       virtio_iommu_translate sid=8 is not known!!
> >>>>>>>       virtio_iommu_translate sid=8 is not known!!
> >>>>>>>       virtio_iommu_translate sid=8 is not known!!
> >>>>>>>       virtio_iommu_translate sid=8 is not known!!
> >>>>>>>       virtio_iommu_translate sid=8 is not known!!
> >>>>>>>       virtio_iommu_translate sid=8 is not known!!
> >>>>>>>
> >>>>>>>
> >>>>>>> Also in guest I do not see device-tree node with virtio-iommu.
> >>>>>> do you mean the virtio-mmio with #iommu-cells property?
> >>>>>>
> >>>>>> This one is created statically by virt machine. I would be
> >>>>>> surprised if it were not there. Are you using the virt = virt2.10
> machine.
> >>>>>> Machines before do not support its instantiation.
> >>>>>>
> >>>>>> Please can you add a printf in hw/arm/virt.c create_virtio_mmio()
> >>>>>> at the moment when this node is created. Also you can add a
> >>>>>> printf in
> >>>>>> bind_virtio_iommu_device() to make sure the binding with the PCI
> >>>>>> host bridge is added on machine init done.
> >>>>>>
> >>>>>> Also worth to check, CONFIG_VIRTIO_IOMMU=y on guest side.
> >>>>>
> >>>>> It works on my side.
> >>>> Great.
> >>>>
> >>>>  The driver config was disabled and also I was using guest kernel
> >>>> which was not have deferred-probing.
> >>>> Yes I did not mention in my cover letter the guest I have been
> >>>> using is based on Jean-Philippe's branch, featuring deferred IOMMU
> probing.
> >>>> I I have not tried yet with an upstream guest.
> >>>>  Now after fixing it works on my side
> >>>>> I placed some prints to see dma-map are mapping regions in
> >>>>> virtio-iommu,
> >>>> it uses emulated iommu.
> >>>>>
> >>>>> I will continue to add VFIO support now on this and more testing !!
> >>>>
> >>>> OK. I will do the VFIO integration first on the vsmmuv3 device as I
> >>>> already prepared the VFIO replay and hopefully we will sync ;-)
> >>>>
> >>>> Thanks
> >>>>
> >>>> Eric
> >>>>>
> >>>>> Thanks
> >>>>> -Bharat
> >>>>>
> >>>>>>
> >>>>>> Thanks
> >>>>>>
> >>>>>> Eric
> >>>>>>
> >>>>>>> I am using qemu-tree you mentioned below and iommu-driver
> >> patches
> >>>>>> published by Jean-P.
> >>>>>>> Qemu command line have additional ""-device virtio-iommu-
> device".
> >>>>>>> What
> >>>>>> I am missing ?
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>> -Bharat
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Best Regards
> >>>>>>>>
> >>>>>>>> Eric
> >>>>>>>>
> >>>>>>>> This series can be found at:
> >>>>>>>> https://github.com/eauger/qemu/tree/virtio-iommu-rfcv2
> >>>>>>>>
> >>>>>>>> References:
> >>>>>>>> [1] [RFC 0/3] virtio-iommu: a paravirtualized IOMMU, [2] [RFC
> >>>>>>>> PATCH linux]
> >>>>>>>> iommu: Add virtio-iommu driver [3] [RFC PATCH kvmtool 00/15]
> >>>>>>>> Add
> >>>>>>>> virtio- iommu
> >>>>>>>>
> >>>>>>>> History:
> >>>>>>>> v1 -> v2:
> >>>>>>>> - fix redifinition of viommu_as typedef
> >>>>>>>>
> >>>>>>>> Eric Auger (8):
> >>>>>>>>   update-linux-headers: import virtio_iommu.h
> >>>>>>>>   linux-headers: Update for virtio-iommu
> >>>>>>>>   virtio_iommu: add skeleton
> >>>>>>>>   virtio-iommu: Decode the command payload
> >>>>>>>>   virtio_iommu: Add the iommu regions
> >>>>>>>>   virtio-iommu: Implement the translation and commands
> >>>>>>>>   hw/arm/virt: Add 2.10 machine type
> >>>>>>>>   hw/arm/virt: Add virtio-iommu the virt board
> >>>>>>>>
> >>>>>>>>  hw/arm/virt.c                                 | 116 ++++-
> >>>>>>>>  hw/virtio/Makefile.objs                       |   1 +
> >>>>>>>>  hw/virtio/trace-events                        |  14 +
> >>>>>>>>  hw/virtio/virtio-iommu.c                      | 623
> >>>>>> ++++++++++++++++++++++++++
> >>>>>>>>  include/hw/arm/virt.h                         |   5 +
> >>>>>>>>  include/hw/virtio/virtio-iommu.h              |  60 +++
> >>>>>>>>  include/standard-headers/linux/virtio_ids.h   |   1 +
> >>>>>>>>  include/standard-headers/linux/virtio_iommu.h | 142 ++++++
> >>>>>>>>  linux-headers/linux/virtio_iommu.h            |   1 +
> >>>>>>>>  scripts/update-linux-headers.sh               |   3 +
> >>>>>>>>  10 files changed, 957 insertions(+), 9 deletions(-)  create
> >>>>>>>> mode
> >>>>>>>> 100644 hw/virtio/virtio-iommu.c  create mode 100644
> >>>>>>>> include/hw/virtio/virtio- iommu.h  create mode 100644
> >>>>>>>> include/standard- headers/linux/virtio_iommu.h  create mode
> >>>>>>>> 100644 linux-headers/linux/virtio_iommu.h
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> 2.5.5
> >>>>>>>
> >>>>>
> >>>