Re: [Qemu-arm] [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device

[Auger Eric]

Hi Bharat,

On 05/07/2017 10:23, Bharat Bhushan wrote:
> Hi Eric,
> 
>> -----Original Message-----
>> From: Auger Eric [mailto:address@hidden
>> Sent: Monday, June 26, 2017 1:25 PM
>> To: Bharat Bhushan <address@hidden>;
>> address@hidden; address@hidden;
>> address@hidden; address@hidden; address@hidden;
>> address@hidden; address@hidden
>> Cc: address@hidden; address@hidden; address@hidden;
>> address@hidden; address@hidden; address@hidden;
>> address@hidden; address@hidden
>> Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device
>>
>> Hi Bharat,
>>
>> On 19/06/2017 09:54, Bharat Bhushan wrote:
>>> Hi Eric,
>>>
>>> I started added replay in virtio-iommu and came across how MSI interrupts
>> with work with VFIO.
>>> I understand that on intel this works differently but vsmmu will have same
>> requirement.
>>> kvm-msi-irq-route are added using the msi-address to be translated by
>> viommu and not the final translated address.
>>> While currently the irqfd framework does not know about emulated
>> iommus (virtio-iommu, vsmmuv3/vintel-iommu).
>>> So in my view we have following options:
>>> - Programming with translated address when setting up
>>> kvm-msi-irq-route
>>> - Route the interrupts via QEMU, which is bad from performance
>>> - vhost-virtio-iommu may solve the problem in long term
>>
>> Sorry for the delay. With regard to the vsmmuv3/vfio integration I think we
>> need to use the guest physical address otherwise the MSI address will not be
>> recognized as an MSI doorbell.
>>
>> Also the fact on ARM we map the MSI doorbell causes an assert in
>> vfio_get_vaddr() as the vITS doorbell is not a RAM region. We will need to
>> handle this specifically.
> 
> Also when setup msi-route kvm_irqchip_add_msi_route() we needed to provide 
> the translated address.
> According to my understanding this is required because kernel does no go 
> through viommu translation when generating interrupt, no? 

yes this is needed when KVM MSI routes are set up, ie. along with GICV3
ITS. With GICv2M, qemu direct gsi mapping is used and this is not needed.

So I do not understand your previous sentence saying "MSI interrupts
works without any change".

Thanks

Eric

> 
> Thanks
> -Bharat
> 
>>
>> Besides I have not looked specifically at the virtio-iommu/vfio integration
>> yet.
>>
>> Thanks
>>
>> Eric
>>>
>>> Is there any other better option I am missing?
>>>
>>> Thanks
>>> -Bharat
>>>
>>>> -----Original Message-----
>>>> From: Auger Eric [mailto:address@hidden
>>>> Sent: Friday, June 09, 2017 5:24 PM
>>>> To: Bharat Bhushan <address@hidden>;
>>>> address@hidden; address@hidden;
>>>> address@hidden; address@hidden; address@hidden;
>>>> address@hidden; address@hidden
>>>> Cc: address@hidden; address@hidden; address@hidden;
>>>> address@hidden; address@hidden; address@hidden;
>>>> address@hidden; address@hidden
>>>> Subject: Re: [Qemu-devel] [RFC v2 0/8] VIRTIO-IOMMU device
>>>>
>>>> Hi Bharat,
>>>>
>>>> On 09/06/2017 13:30, Bharat Bhushan wrote:
>>>>> Hi Eric,
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Auger Eric [mailto:address@hidden
>>>>>> Sent: Friday, June 09, 2017 12:14 PM
>>>>>> To: Bharat Bhushan <address@hidden>;
>>>>>> address@hidden; address@hidden;
>>>>>> address@hidden; address@hidden; qemu-
>> address@hidden;
>>>>>> address@hidden; address@hidden
>>>>>> Cc: address@hidden; address@hidden;
>>>> address@hidden;
>>>>>> address@hidden; address@hidden;
>>>>>> address@hidden; address@hidden; address@hidden
>>>>>> Subject: Re: [RFC v2 0/8] VIRTIO-IOMMU device
>>>>>>
>>>>>> Hi Bharat,
>>>>>>
>>>>>> On 09/06/2017 08:16, Bharat Bhushan wrote:
>>>>>>> Hi Eric,
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Eric Auger [mailto:address@hidden
>>>>>>>> Sent: Wednesday, June 07, 2017 9:31 PM
>>>>>>>> To: address@hidden; address@hidden;
>>>>>>>> address@hidden; address@hidden;
>>>>>> address@hidden;
>>>>>>>> address@hidden; address@hidden; jean-
>>>>>>>> address@hidden
>>>>>>>> Cc: address@hidden; address@hidden;
>>>>>> address@hidden;
>>>>>>>> address@hidden; address@hidden;
>>>>>>>> address@hidden; address@hidden; address@hidden; Bharat
>>>>>> Bhushan
>>>>>>>> <address@hidden>
>>>>>>>> Subject: [RFC v2 0/8] VIRTIO-IOMMU device
>>>>>>>>
>>>>>>>> This series implements the virtio-iommu device. This is a proof
>>>>>>>> of concept based on the virtio-iommu specification written by
>>>>>>>> Jean-Philippe
>>>>>> Brucker [1].
>>>>>>>> This was tested with a guest using the virtio-iommu driver [2]
>>>>>>>> and exposed with a virtio-net-pci using dma ops.
>>>>>>>>
>>>>>>>> The device gets instantiated using the "-device virtio-iommu-device"
>>>>>>>> option. It currently works with ARM virt machine only as the
>>>>>>>> machine must handle the dt binding between the virtio-mmio
>> "iommu"
>>>>>>>> node and the PCI host bridge node. ACPI booting is not yet
>> supported.
>>>>>>>>
>>>>>>>> This should allow to start some benchmarking activities against
>>>>>>>> pure emulated IOMMU (especially ARM SMMU).
>>>>>>>
>>>>>>> I am testing this on ARM64 and see below continuous error prints:
>>>>>>>
>>>>>>>         virtio_iommu_translate sid=8 is not known!!
>>>>>>>         virtio_iommu_translate sid=8 is not known!!
>>>>>>>         virtio_iommu_translate sid=8 is not known!!
>>>>>>>         virtio_iommu_translate sid=8 is not known!!
>>>>>>>         virtio_iommu_translate sid=8 is not known!!
>>>>>>>         virtio_iommu_translate sid=8 is not known!!
>>>>>>>         virtio_iommu_translate sid=8 is not known!!
>>>>>>>         virtio_iommu_translate sid=8 is not known!!
>>>>>>>         virtio_iommu_translate sid=8 is not known!!
>>>>>>>         virtio_iommu_translate sid=8 is not known!!
>>>>>>>
>>>>>>>
>>>>>>> Also in guest I do not see device-tree node with virtio-iommu.
>>>>>> do you mean the virtio-mmio with #iommu-cells property?
>>>>>>
>>>>>> This one is created statically by virt machine. I would be
>>>>>> surprised if it were not there. Are you using the virt = virt2.10 
>>>>>> machine.
>>>>>> Machines before do not support its instantiation.
>>>>>>
>>>>>> Please can you add a printf in hw/arm/virt.c create_virtio_mmio()
>>>>>> at the moment when this node is created. Also you can add a printf
>>>>>> in
>>>>>> bind_virtio_iommu_device() to make sure the binding with the PCI
>>>>>> host bridge is added on machine init done.
>>>>>>
>>>>>> Also worth to check, CONFIG_VIRTIO_IOMMU=y on guest side.
>>>>>
>>>>> It works on my side.
>>>> Great.
>>>>
>>>>  The driver config was disabled and also I was using guest kernel
>>>> which was not have deferred-probing.
>>>> Yes I did not mention in my cover letter the guest I have been using
>>>> is based on Jean-Philippe's branch, featuring deferred IOMMU probing.
>>>> I I have not tried yet with an upstream guest.
>>>>  Now after fixing it works on my side
>>>>> I placed some prints to see dma-map are mapping regions in
>>>>> virtio-iommu,
>>>> it uses emulated iommu.
>>>>>
>>>>> I will continue to add VFIO support now on this and more testing !!
>>>>
>>>> OK. I will do the VFIO integration first on the vsmmuv3 device as I
>>>> already prepared the VFIO replay and hopefully we will sync ;-)
>>>>
>>>> Thanks
>>>>
>>>> Eric
>>>>>
>>>>> Thanks
>>>>> -Bharat
>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Eric
>>>>>>
>>>>>>> I am using qemu-tree you mentioned below and iommu-driver
>> patches
>>>>>> published by Jean-P.
>>>>>>> Qemu command line have additional ""-device virtio-iommu-device".
>>>>>>> What
>>>>>> I am missing ?
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>> -Bharat
>>>>>>>
>>>>>>>>
>>>>>>>> Best Regards
>>>>>>>>
>>>>>>>> Eric
>>>>>>>>
>>>>>>>> This series can be found at:
>>>>>>>> https://github.com/eauger/qemu/tree/virtio-iommu-rfcv2
>>>>>>>>
>>>>>>>> References:
>>>>>>>> [1] [RFC 0/3] virtio-iommu: a paravirtualized IOMMU, [2] [RFC
>>>>>>>> PATCH linux]
>>>>>>>> iommu: Add virtio-iommu driver [3] [RFC PATCH kvmtool 00/15] Add
>>>>>>>> virtio- iommu
>>>>>>>>
>>>>>>>> History:
>>>>>>>> v1 -> v2:
>>>>>>>> - fix redifinition of viommu_as typedef
>>>>>>>>
>>>>>>>> Eric Auger (8):
>>>>>>>>   update-linux-headers: import virtio_iommu.h
>>>>>>>>   linux-headers: Update for virtio-iommu
>>>>>>>>   virtio_iommu: add skeleton
>>>>>>>>   virtio-iommu: Decode the command payload
>>>>>>>>   virtio_iommu: Add the iommu regions
>>>>>>>>   virtio-iommu: Implement the translation and commands
>>>>>>>>   hw/arm/virt: Add 2.10 machine type
>>>>>>>>   hw/arm/virt: Add virtio-iommu the virt board
>>>>>>>>
>>>>>>>>  hw/arm/virt.c                                 | 116 ++++-
>>>>>>>>  hw/virtio/Makefile.objs                       |   1 +
>>>>>>>>  hw/virtio/trace-events                        |  14 +
>>>>>>>>  hw/virtio/virtio-iommu.c                      | 623
>>>>>> ++++++++++++++++++++++++++
>>>>>>>>  include/hw/arm/virt.h                         |   5 +
>>>>>>>>  include/hw/virtio/virtio-iommu.h              |  60 +++
>>>>>>>>  include/standard-headers/linux/virtio_ids.h   |   1 +
>>>>>>>>  include/standard-headers/linux/virtio_iommu.h | 142 ++++++
>>>>>>>>  linux-headers/linux/virtio_iommu.h            |   1 +
>>>>>>>>  scripts/update-linux-headers.sh               |   3 +
>>>>>>>>  10 files changed, 957 insertions(+), 9 deletions(-)  create mode
>>>>>>>> 100644 hw/virtio/virtio-iommu.c  create mode 100644
>>>>>>>> include/hw/virtio/virtio- iommu.h  create mode 100644
>>>>>>>> include/standard- headers/linux/virtio_iommu.h  create mode
>>>>>>>> 100644 linux-headers/linux/virtio_iommu.h
>>>>>>>>
>>>>>>>> --
>>>>>>>> 2.5.5
>>>>>>>
>>>>>
>>>