Re: [Qemu-devel] [v10][PATCH 03/10] piix: create host bridge to passthrough

[Alex Williamson]

On Thu, 11 Feb 2016 02:25:51 +0000
"Kay, Allen M" <address@hidden> wrote:

> > -----Original Message-----
> > From: Alex Williamson [mailto:address@hidden
> > Sent: Tuesday, February 09, 2016 10:01 PM
> > To: Kay, Allen M <address@hidden>
> > Cc: address@hidden; address@hidden;
> > address@hidden; address@hidden;
> > address@hidden; address@hidden; Ruan, Shuai
> > <address@hidden>
> > Subject: Re: [Qemu-devel] [v10][PATCH 03/10] piix: create host bridge to
> > passthrough
> > 
> > I can run that system as either primary or secondary.  On both systems I'm
> > using pci-stub to keep i915 from interfering on the host, on the SNB laptop 
> > I
> > also use video=efifb:off since it's configured for UEFI boot, I'm not sure 
> > if
> > that play any role in why it's not working.
> >   
> 
> Hi Alex,
> 
> My understand if that your IVB system is a desktop running legacy mode BIOS 
> and your SNB is a laptop running EFI mode BIOS, correct?  I'm curious how 
> does your SNB system getting hold of VBT table, which is needed for lighting 
> up local display.
> 
> There are two ways the driver can get VBT table:  1) VGA 0xc0000 memory,  2) 
> OpRegion
> 
> On your IVB system, the guest driver would try to get VBT from 0xc0000 memory 
> if IGD is configured as primary controller in the guest, assuming KVM/QEMU 
> copies VGA 0xc0000 memory from host to guest 0xc0000 area.   If IGD is 
> configured as secondary controller, the driver would get VBT from OpRegion.  
> Given IGD works in both configurations on IVB, this mean guest driver can 
> successfully read VBT from both 0xc0000 area and OpRegion.
> 
> On your SNB system that is running in EFI mode on the host, 0xc0000 memory 
> does not contain VBIOS/VBT.  This means if you configure IGD as primary 
> controller in the guest, the driver cannot get to VBT at 0xc0000 memory even 
> if KVM/QEMU copies 0xc0000 memory from host to guest.  If you configure IGD 
> as secondary controller in the guest, and guest driver should be able to read 
> VBT from the OpRegion.  You might want to trace i915 to see how does the  
> guest driver get to VBT via OpRegion in this configuration.

Hi Allen,

Success!  We don't directly pass 0xc0000 from the host, but we can use
PCI option ROMs loaded via files into QEMU.  For the IVB system I
haven't been bothering with this, even in primary mode I've just been
letting the display initialize later in the guest boot.  I also haven't
been enabling VGA mode access in that case.  With the SNB laptop, if I
extract the ROM from sysfs, modify the device ID and fix the checksum,
the panel lights up, with or without VGA mode access (though of course
I only see a flash of SeaBIOS when VGA mode is enabled).  So it seems
that either the user is going to need to hack their own ROM file or I'm
going to need to make vfio do this automatically and pretend that the
ROM is actually implemented as a BAR on IGD.  The latter seems far more
accessible.  Lacking an actual ROM BAR, we'll also of course only be
able to do that when IGD is primary graphics on the host.

I'm not sure how we can do secondary mode in the VM with this config
though since only the primary graphics gets the coveted 0xc0000
location.  FWIW, to make this work I added 'romfile=igd.rom' to the
vfio-pci device with igd.rom being the modified copy retrieved from
sysfs.  Then I used '-vga none' to disable the QEMU primary VGA device
(-nodefaults is probably an option too).  I then added 'addr=2.0' to
the vfio-pci device to make it be the primary graphics from SeaBIOS'
perspective and added '-device secondary-vga,addr=3.0 -vnc :1', which
predominantly gives me a VNC connection where I can interact with the
VM via mouse and keyboard.

I'll need to do some further investigation to see what we're really
getting with the OpRegion.  Prior to adding the ROM, the Xorg log file
sure seems like it knew the LVDS panel was 1920x1080, but it might be
seeing more modes now that it has a video BIOS.  I'll also try to prune
the registers copied into the virtual host bridge and ISA bridge config
space to the minimum we need.

> Another potential problem to watch out for with laptop vs. desktop has to do 
> FLR timeout.  If you are working with a laptop with LCD panel attached (the 
> usual case), FLR will take much longer than 100ms to finish.  The reason is 
> the devices needs to power down the LCD panel before it can finish FLR. I 
> have seen it takes more than 500ms for FLR to finish.  As a work around, I 
> have increase FLR time out in Linux PCI driver to 1000ms when working with 
> LCD panels.   Given that you have seen evidences that IGD HW is working, this 
> might not be your issue.   I would focus tracing how does i915 get VBT from 
> the OpRegion when IGD is configured as secondary controller in the guest.

Ok, I did see evidence of this.  In my case the VM would always fail to
start on the first try with errors in dmesg indicating that vfio was
reading back -1 from config space of the device.  Presumably it was
only the first attempt in my case since previously I was never getting
the panel turned back on and subsequent resets were faster.  I also
notice that I see DMAR faults on the first reset that seem to be
accesses to the host stolen memory region pointed to by the BDSM.  I
don't see this if I reset the device from sysfs before trying to use it
with the VM, so maybe vfio should try to do a reset on probing IGD,
before it gets placed in an IOMMU domain that blocks access to that
host stolen memory.  Of course the next question is whether we can
easily determine whether an IGD has an LCD panel so we know which
timeout to use.  I suppose we could use a 1000ms timeout for all Intel
VGA class devices, but handling it with a quirk only for panel attached
configs would probably be preferable.  Thanks,

Alex