Re: [Qemu-devel] vfio in the guest: no available reset mechanism

[Le Tan]

2014-08-01 23:25 GMT+08:00 Alex Williamson <address@hidden>:
> On Fri, 2014-08-01 at 09:35 +0800, Le Tan wrote:
>> Hi Alex,
>>
>> 2014-07-30 22:46 GMT+08:00 Alex Williamson <address@hidden>:
>> > On Wed, 2014-07-30 at 22:16 +0800, Le Tan wrote:
>> >> Hi Michael,
>> >>
>> >> 2014-07-30 21:16 GMT+08:00 Michael S. Tsirkin <address@hidden>:
>> >> > On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote:
>> >> >> Hi,
>> >> >> I am testing vfio in L1 with my VT-d emulation project. I assigned one
>> >> >> of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU
>> >> >> in L1, it complains that:
>> >> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
>> >> >> available reset mechanism.
>> >> >> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
>> >> >> available reset mechanism.
>> >> >>
>> >> >> Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I
>> >> >> look into this and found that:
>> >> >> val = ahci_ctrl_readl(ctrl, HOST_CTL);
>> >> >> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN);
>> >> >> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which
>> >> >> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit
>> >> >> should be cleared by default. So maybe L1 didn't reset the device
>> >> >> before assigning it to L2?
>> >> >> Then the BIOS tried to write back to HOST_CTL and it was stuck here. :(
>> >> >>
>> >> >> So can anyone give me some advice? About the state of PCI device or
>> >> >> bus-level reset?
>> >> >>
>> >> >> Here is the detail of the environment and the way I did the vfio.
>> >> >> 1. lspci in L1 said:
>> >> >> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
>> >> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
>> >> >> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC
>> >> >> Interface Controller [8086:2918] (rev 02)
>> >> >> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
>> >> >> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
>> >> >> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus
>> >> >> Controller [8086:2930] (rev 02)
>> >> >> 2. Unbind 00:03.0 and do vfio:
>> >> >> modprobe -r vfio_iommu_type1
>> >> >> modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
>> >> >> modprobe vfio-pci
>> >> >> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind
>> >> >> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id
>> >> >> 3. run L2 with "-device vfio-pci,host=00:03.0"
>> >> >>
>> >> >> Any help is appreciated! Thanks very much!
>> >> >>
>> >> >> Regards,
>> >> >> Le
>> >> >
>> >> > Clearly, bus level reset can't work for the root bus :)
>> >>
>> >> Thanks very much!
>> >> I test the vfio with a second-bus ahci controller and it didn't
>> >> complain about the lack of reset mechanism. :) And the return val of
>> >> HOST_CTL is normal now (the same as emulated ahci controller).
>> >> However, it still paused when the BIOS tried to write to the HOST_CTL.
>> >> Do you have any idea?
>> >> And we should just test vfio and legacy pci-assignment with second bus
>> >> devices, not considering the root-bus devices?
>> >
>> > AHCI seems like a poor choice of device for this work, they typically
>> > don't support any kind of reset and they can be troublesome even for the
>> > L1 assignment.  You really want something with FLR support so that both
>> > the host and L1 guest can potentially reset the device.  That said, you
>> > may still run into bugs with a L1 guest directed FLR.  Thanks,
>> >
>> > Alex
>> >
>>
>> So what device do you think is suitable for the pci-assign test? e1000?
>> I just tested it with sound card ac97 and USB controller. But I don't
>> know how to attach them to a pcie-to-pci bridge, so maybe they weren't
>> reset before being assigned to L2. But it seems that they can work.
>> 1. With the sound card, I assigned it to L2 via both vfio and legacy
>> pci-assign and I could hear the music played in L2 from host's
>> speakers. Of course, the vfio also complained about the lack of reset
>> mechanism.
>
> First off, maybe I'm a little confused, are you trying to assigned
> emulated devices for the L1 guest to the L2 guest or are these assigned
> devices to the L1 guest that you want to re-assign to L2?  AFAIK, we
> don't have any emulated devices that support reset, but it wouldn't be
> that hard to add a PM capability to one that advertises a soft reset on
> D3hot->D0 transition and calls the QEMU driver reset function when that
> occurs.  This would provide the most flexibility.

I am trying to assign emulated devices for the L1 guest to the L2
guest. According to Jan's suggestions, I will first go on to add more
features to the VT-d emulation.
Thanks very much! Learn a lot from you! :)

Le

> The other choice is to create a topology where an individual device can
> be reset via a bus reset, so putting a single device behind a bridge,
> root port, or downstream switch port.
>
> Otherwise, the emulated e1000 is probably a reasonable choice, network
> drivers generally seem to be pretty good about accepting a device in a
> running state since it might have been used as a boot device.
>
> If you're looking at re-assigning an L1 assigned device to an L2 guest,
> I'm not sure how that's supposed to work.  The VT-d emulation would need
> to somehow push L2 physical mappings to the host, but the L1 guest
> doesn't know the true isolation constraints of the host devices and
> whether they can actually be mapped separately from other devices.
> Since we try to use as few IOMMU domains as possible, you'd end up
> modifying translations for all devices assigned to the L1 guest.
>
>> 2. With the USB controller, I used "-usb -usbdevice disk:file" to
>> attach a USB disk to L1. But there were 4 related devices in L1, so I
>> didn't know what should be assigned to L2. the info qtree was like
>> this:
>>  dev: ich9-usb-uhci3, id ""
>>         masterbus = "usb-bus.0"
>>         firstport = 4 (0x4)
>>         bandwidth = 1280 (0x500)
>>         maxframes = 128 (0x80)
>>         addr = 1d.2
>> dev: ich9-usb-uhci2, id ""
>>         masterbus = "usb-bus.0"
>>         firstport = 2 (0x2)
>>         bandwidth = 1280 (0x500)
>>         maxframes = 128 (0x80)
>>         addr = 1d.1
>>  dev: ich9-usb-uhci1, id ""
>>         masterbus = "usb-bus.0"
>>         firstport = 0 (0x0)
>>         bandwidth = 1280 (0x500)
>>         maxframes = 128 (0x80)
>>         addr = 1d.0
>> dev: ich9-usb-ehci1, id ""
>>         maxframes = 128 (0x80)
>>         addr = 1d.7
>>          bus: usb-bus.0
>>           type usb-bus
>>           dev: usb-storage, id ""
>>             drive = ""
>>             logical_block_size = 512 (0x200)
>>             physical_block_size = 512 (0x200)
>>             min_io_size = 0 (0x0)
>>             opt_io_size = 0 (0x0)
>>             bootindex = -1 (0xffffffffffffffff)
>>             discard_granularity = 4294967295 (0xffffffff)
>>             removable = false
>>             port = ""
>>             serial = ""
>>             full-path = true
>>             msos-desc = true
>>             addr 0.1, port 1, speed 480, name QEMU USB MSD, attached
>>             bus: scsi.0
>>               type SCSI
>>               dev: scsi-disk, id ""
>>                 drive = "usb0"
>>                 logical_block_size = 512 (0x200)
>>                 physical_block_size = 512 (0x200)
>>                 min_io_size = 0 (0x0)
>>                 opt_io_size = 0 (0x0)
>> Then I tried to unbind and bind both 1d.7 and 1d.0. Then I assigned
>> 1d.0 to L2. L2 could boot smoothly and I could mount and access the
>> USB disk.
>> So what do you think?
>
> Sounds like a reasonable test of L1 emulated assigned to L2.  Thanks,
>
> Alex
>
>