Re: [Qemu-devel] VFIO VGA passthrough

[Alex Williamson]

On Tue, 2012-11-20 at 19:47 +0800, ching wrote:
> My PC is gentoo x64, kernel 3.6.6, intel sandy bridge i7 2600 + asrock Z77 
> exterme 4 MB
> 
> I download qemu from git and try to pass through secondary VGA card to 
> Windows 7 x64 SP1 VM:
> 
> 
> 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI 
> Cayman PRO [Radeon HD 6950] (prog-if 00 [VGA controller])
>     Subsystem: XFX Pine Group Inc. Device 3125
>     Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
> <MAbort- >SERR- <PERR- INTx-
>     Interrupt: pin A routed to IRQ 11
>     Region 0: Memory at d0000000 (64-bit, prefetchable) [disabled] [size=256M]
>     Region 2: Memory at f4b20000 (64-bit, non-prefetchable) [disabled] 
> [size=128K]
>     Region 4: I/O ports at e000 [disabled] [size=256]
>     Expansion ROM at f4b00000 [disabled] [size=128K]
>     Capabilities: [50] Power Management version 3
>         Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA 
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>         Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
>     Capabilities: [58] Express (v2) Legacy Endp5. grant oint, MSI 00
>         DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 
> unlimited
>             ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>         DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>             RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
>             MaxPayload 128 bytes, MaxReadReq 512 bytes
>         DevSta:    CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
>         LnkCap:    Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 
> <64ns, L1 <1us
>             ClockPM- Surprise- LLActRep- BwNot-
>         LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
>             ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>         LnkSta:    Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- 
> BWMgmt- ABWMgmt-
>         DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF 
> Not Supported
>         DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF 
> Disabled
>         LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
>              Transmit Margin: Normal Operating Range, 
> EnterModifiedCompliance- ComplianceSOS-
>              Compliance De-emphasis: -6dB
>         LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, 
> EqualizationPhase1-
>              EqualizationPhase2-, EqualizationPhase3-, 
> LinkEqualizationRequest-
>     Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
>         Address: 0000000000000000  Data: 0000
>     Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 
> <?>
>     Capabilities: [150 v1] Advanced Error Reporting
>         UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- 
> MalfTLP- ECRC- UnsupReq- ACSViol-
>         UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- 
> MalfTLP- ECRC- UnsupReq- ACSViol-
>         UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ 
> MalfTLP+ ECRC- UnsupReq- ACSViol-
>         CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>         CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>         AERCap:    First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
>     Kernel driver in use: vfio-pci
> 
> 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cayman/Antilles 
> HDMI Audio [Radeon HD 6900 Series]
>     Subsystem: XFX Pine Group Inc. Device aa80
>     Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- 
> Stepping- SERR- FastB2B- DisINTx-
>     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- 
> <MAbort- >SERR- <PERR- INTx-
>     Interrupt: pin B routed to IRQ 105. grant
>     Region 0: Memory at f4b40000 (64-bit, non-prefetchable) [disabled] 
> [size=16K]
>     Capabilities: [50] Power Management version 3
>         Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA 
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>         Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
>     Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
>         DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 
> unlimited
>             ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>         DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>             RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
>             MaxPayload 128 bytes, MaxReadReq 512 bytes
>         DevSta:    CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
>         LnkCap:    Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 
> <64ns, L1 <1us
>             ClockPM- Surprise- LLActRep- BwNot-
>         LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
>             ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>         LnkSta:    Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- 
> BWMgmt- ABWMgmt-
>         DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF 
> Not Supported
>         DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF 
> Disabled
>         LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, 
> EqualizationPhase1-
>              EqualizationPhase2-, EqualizationPhase3-, 
> LinkEqualizationRequest-
>     Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
>         Address: 0000000000000000  Data: 0000
>     Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 
> <?>
>     Capabilities: [150 v1] Advanced Error Reporting
>         UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- 
> MalfTLP- ECRC- UnsupReq- ACSViol-
>         UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- 
> MalfTLP- ECRC- UnsupReq- ACSViol-
>         UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ 
> MalfTLP+ ECRC- UnsupReq- ACSViol-
>         CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>         CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>         AERCap:    First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
>     Kernel driver in use: vfio-pci
> 
> 
> 
> 
> i google related threads and carried out preparation:
> 1. set module parameter "vfio_iommu_type1.allow_unsafe_interrupts"=1
> 2. raise ulimit: ulimit -l unlimited
> 3. compile kernel without swap memory support, ATI KMS, alsa for ATI if any
> 4. compile xorg without ati, vesa driver
> 
> 
> 
> But the guest boots with BSOD and qemu shows the following messages, can 
> anyone helps?

Getting VGA to work in a guest is mostly black magic and luck at this
point.  The warning you're getting isn't preventing anything.  We use
device reset to try to clear the state of the hardware between runs.  If
it's not available we can potentially leak data through the device and
may see issues with reproducibility since we can't return the device to
a known state.

The assigned VGA device should not be used for anything on the host,
which it seems like you've taken care of.  The host shouldn't even have
a blinking cursor on the display.

It's been reported that "-cpu host" helps, but that's possibly very
similar to what you specify below.  When I tested secondary VGA with an
ATI card I used "-vga std" and I made the devices appear as
multifunction in the guest, ex:

-device vfio-pci,host=1:00.0,multifunction=on,addr=4.0 \
-device vfio-pci,host=1:00.1,addr=4.1

I also first booted without the AMD Catalyst driver installed.  The
assigned devices are unused, but the guest still boots.  Then installed
the Catalyst driver, after which rebooting causes the std VGA to go
blank after an initial startup screen and Windows uses the assigned
device exclusively.  Good luck, there's obviously still work to do in
this space.  Thanks,

Alex


> for dev in $(ls /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices); do
>     vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
>     device=$(cat /sys/bus/pci/devices/$dev/device)
>     if [ -e /sys/bus/pci/devices/$dev/driver ]; then
>         echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
>     fi
>     echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
> done
> 
> # qemu-system-x86_64 \
> > -runas zvm \
> > -name Windows -M pc-1.3 -enable-kvm \
> > -cpu 
> > SandyBridge,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
> >  \
> > -m 8192 -smp 2,sockets=2,cores=1,threads=1 \
> > -rtc base=utc,driftfix=slew \
> > -net nic,model=virtio -net user \
> > -drive 
> > file=/custom/vm/image/Temp.raw_image,if=virtio,format=raw,cache=unsafe,aio=native
> >  \
> > -device vfio-pci,host=0000:01:00.0 \
> > -device vfio-pci,host=0000:01:00.1 \
> > -vga cirrus
> qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0: Warning, device 
> 0000:01:00.0 does not support reset
> 
> qemu-system-x86_64: -device vfio-pci,host=0000:01:00.1: Warning, device 
> 0000:01:00.1 does not support reset
> 
>