Re: [Qemu-devel] iommu emulation

[Jintack Lim]

On Thu, Mar 2, 2017 at 5:20 PM, Bandan Das <address@hidden> wrote:
> Jintack Lim <address@hidden> writes:
>
>> [cc Bandan]
>>
>> On Tue, Feb 21, 2017 at 5:33 AM, Jintack Lim <address@hidden>
>> wrote:
>>
>>>
>>>
>>> On Wed, Feb 15, 2017 at 9:47 PM, Alex Williamson <
>>> address@hidden> wrote:
> ...
>>>
>>
>> I've tried another network device on a different machine. It has "Intel
>> Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection" ethernet
>> controller. I got the same problem of getting the network device
>> initialization failure in L2. I think I'm missing something since I heard
>> from Bandan that he had no problem to assign a device to L2 with ixgbe.
>>
>> This is the error message from dmesg in L2.
>>
>> [    3.692871] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver -
>> version 4.2.1-k
>> [    3.697716] ixgbe: Copyright (c) 1999-2015 Intel Corporation.
>> [    3.964875] ixgbe 0000:00:02.0: HW Init failed: -12
>> [    3.972362] ixgbe: probe of 0000:00:02.0 failed with error -12
>>
>> I checked that L2 indeed had that device.
>> address@hidden:~# lspci
>> 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM
>> Controller
>> 00:01.0 VGA compatible controller: Device 1234:1111 (rev 02)
>> 00:02.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+
>> Network Connection (rev 01)
>
> Jintack, any progress with this ?

Not much, unfortunately.

>
> I am testing on a X540-AT2 and I see a different behavior. It appears
> config succeeds but the driver keeps resetting the device due to a Tx
> hang:

Thanks for your effort!

>
> [ 568.612391 ] ixgbe 0000:00:03.0 enp0s3: tx hang 38 detected on queue 0,
> resetting adapter
> [ 568.612393 ]  ixgbe 0000:00:03.0 enp0s3: initiating reset due to tx
> timeout
> [ 568.612397 ]  ixgbe 0000:00:03.0 enp0s3: Reset adapter
>
> This may be device specific but I think the actual behavior you see is
> also dependent on the ixgbe driver in the guest. Are you on a recent
> kernel ? Also, can you point me to the hack (by Peter) that you have
> mentioned above ?

I was using 4.6.0-rc on the machine with Mellanox device, and
4.10.0-rc on the machine with Intel device. L0, L1 and L2 had the same
version.

This is the initial hack from Peter,
------8<-------
diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 332f41d..bacd302 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -1925,11 +1925,6 @@ static void vfio_add_ext_cap(VFIOPCIDevice *vdev)

     }

-    /* Cleanup chain head ID if necessary */
-    if (pci_get_word(pdev->config + PCI_CONFIG_SPACE_SIZE) == 0xFFFF) {
-        pci_set_word(pdev->config + PCI_CONFIG_SPACE_SIZE, 0);
-    }
-
     g_free(config);
     return;
 }
------>8-------

and I believe this is the commit merged into QEMU repo.

commit d0d1cd70d10639273e2a23870e7e7d80b2bc4e21
Author: Alex Williamson <address@hidden>
Date:   Wed Feb 22 13:19:58 2017 -0700

    vfio/pci: Improve extended capability comments, skip masked caps


Thanks,
Jintack

>
> Thanks,
> Bandan
>
>> I'm describing steps I took, so if you notice something wrong, PLEASE let
>> me know.
>>
>> 1. [L0] Check the device with lspci. Result is [1]
>> 2. [L0] Unbind from the original driver and bind to vfio-pci driver
>> following [2][3]
>> 3. [L0] Start L1 with this script. [4]
>> 4. [L1] L1 is able to use the network device.
>> 5. [L1] Unbind from the original driver and bind to vfio-pci driver same as
>> the step 2.
>> 6. [L1] Start L2 with this script. [5]
>> 7. [L2] Got the init failure error message above.
>>
>> [1] https://paste.ubuntu.com/24055745/
>> [2] http://www.linux-kvm.org/page/10G_NIC_performance:_VFIO_vs_virtio
>> [3] http://www.linux-kvm.org/images/b/b4/2012-forum-VFIO.pdf
>> [4] https://paste.ubuntu.com/24055715/
>> [5] https://paste.ubuntu.com/24055720/
>>
>> Thanks,
>> Jintack
>>
>>
>>>
>>>
>>>> Alex
>>>>
>>>>
>>>
>