Re: [Qemu-devel] VFIO VGA test branches

[Alex Williamson]

On Wed, 2013-05-29 at 18:16 +0200, Maik Broemme wrote:
> Hi,
> 
> Maik Broemme <address@hidden> wrote:
> > > > 
> > > > Here is the DEBUG_VFIO output:
> > > > 
> > > > vfio: vfio_initfn(0000:04:00.0) group 14
> > > > vfio: region_add 0 - afffffff [0x7f8698000000]
> > > > vfio: SKIPPING region_add fec00000 - fec00fff
> > > > vfio: SKIPPING region_add fed00000 - fed003ff
> > > > vfio: SKIPPING region_add fee00000 - feefffff
> > > > vfio: region_add fffe0000 - ffffffff [0x7f88aa400000]
> > > > vfio: region_add 100000000 - 24fffffff [0x7f8748000000]
> > > > vfio: Device 0000:04:00.0 flags: 3, regions: 9, irgs: 4
> > > > vfio: Device 0000:04:00.0 region 0:
> > > > vfio:   size: 0x10000000, offset: 0x0, flags: 0x7
> > > > vfio: Device 0000:04:00.0 region 1:
> > > > vfio:   size: 0x0, offset: 0x10000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.0 region 2:
> > > > vfio:   size: 0x40000, offset: 0x20000000000, flags: 0x7
> > > > vfio: Device 0000:04:00.0 region 3:
> > > > vfio:   size: 0x0, offset: 0x30000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.0 region 4:
> > > > vfio:   size: 0x100, offset: 0x40000000000, flags: 0x3
> > > > vfio: Device 0000:04:00.0 region 5:
> > > > vfio:   size: 0x0, offset: 0x50000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.0 ROM:
> > > > vfio:   size: 0x20000, offset: 0x60000000000, flags: 0x1
> > > > vfio: Device 0000:04:00.0 config:
> > > > vfio:   size: 0x1000, offset: 0x70000000000, flags: 0x3
> > > > vfio: vfio_load_rom(0000:04:00.0)
> > > > vfio: Enabled ATI/AMD BAR2 0x4000 quirk for device 0000:04:00.0
> > > > vfio: Enabled ATI/AMD BAR4 window quirk for device 0000:04:00.0
> > > > vfio: Enabled ATI/AMD quirk 0x3c3 BAR4 for device 0000:04:00.0
> > > > vfio: 0000:04:00.0 PCI MSI CAP @0xa0
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > > vfio: vfio_enable_intx(0000:04:00.0)
> > > > vfio: vfio_initfn(0000:04:00.1) group 14
> > > > vfio: Device 0000:04:00.1 flags: 3, regions: 9, irgs: 4
> > > > vfio: Device 0000:04:00.1 region 0:
> > > > vfio:   size: 0x4000, offset: 0x0, flags: 0x7
> > > > vfio: Device 0000:04:00.1 region 1:
> > > > vfio:   size: 0x0, offset: 0x10000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.1 region 2:
> > > > vfio:   size: 0x0, offset: 0x20000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.1 region 3:
> > > > vfio:   size: 0x0, offset: 0x30000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.1 region 4:
> > > > vfio:   size: 0x0, offset: 0x40000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.1 region 5:
> > > > vfio:   size: 0x0, offset: 0x50000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.1 ROM:
> > > > vfio:   size: 0x0, offset: 0x60000000000, flags: 0x0
> > > > vfio: Device 0000:04:00.1 config:
> > > > vfio:   size: 0x1000, offset: 0x70000000000, flags: 0x3
> > > > vfio: 0000:04:00.1 PCI MSI CAP @0xa0
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > > vfio: vfio_enable_intx(0000:04:00.1)
> > > > vfio: region_del 0 - afffffff
> > > > vfio: region_add 0 - bffff [0x7f8698000000]
> > > > vfio: region_add c0000 - dffff [0x7f88aa200000]
> > > > vfio: region_add e0000 - fffff [0x7f88aa400000]
> > > > vfio: region_add 100000 - afffffff [0x7f8698100000]
> > > > vfio: vfio_pci_reset(0000:04:00.0)
> > > > vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> > > > vfio: vfio_disable_intx(0000:04:00.0)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x54, len=0x2) 0
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 3
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x0, len=0x2)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > > vfio: vfio_enable_intx(0000:04:00.0)
> > > > vfio: vfio_pci_reset(0000:04:00.1)
> > > > vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> > > > vfio: vfio_disable_intx(0000:04:00.1)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x54, len=0x2) 0
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 6
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x0, len=0x2)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > > vfio: vfio_enable_intx(0000:04:00.1)
> > > > vfio: region_del 0 - bffff
> > > > vfio: region_del c0000 - dffff
> > > > vfio: region_add 0 - c7fff [0x7f8698000000]
> > > > vfio: region_add c8000 - dffff [0x7f88aa208000]
> > > > vfio: region_del 0 - c7fff
> > > > vfio: region_del c8000 - dffff
> > > > vfio: region_add 0 - cffff [0x7f8698000000]
> > > > vfio: region_add d0000 - dffff [0x7f88aa210000]
> > > > vfio: region_del 0 - cffff
> > > > vfio: region_del d0000 - dffff
> > > > vfio: region_add 0 - d7fff [0x7f8698000000]
> > > > vfio: region_add d8000 - dffff [0x7f88aa218000]
> > > > vfio: region_del 0 - d7fff
> > > > vfio: region_del d8000 - dffff
> > > > vfio: region_add 0 - dffff [0x7f8698000000]
> > > > vfio: region_del 0 - dffff
> > > > vfio: region_del e0000 - fffff
> > > > vfio: region_add 0 - e7fff [0x7f8698000000]
> > > > vfio: region_add e8000 - fffff [0x7f88aa408000]
> > > > vfio: region_del 0 - e7fff
> > > > vfio: region_del e8000 - fffff
> > > > vfio: region_add 0 - effff [0x7f8698000000]
> > > > vfio: region_add f0000 - fffff [0x7f88aa410000]
> > > > vfio: region_del 0 - effff
> > > > vfio: region_del f0000 - fffff
> > > > vfio: region_del 100000 - afffffff
> > > > vfio: region_add 0 - afffffff [0x7f8698000000]
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x4) 68181002
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x8, len=0x4) 3000000
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x4) aab01002
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x8, len=0x4) 4030000
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0xe, len=0x1) 80
> > > > vfio: SKIPPING region_add b0000000 - bfffffff
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) c000000c
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) f000000c
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xc000000c, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) ffffffff
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fde80004
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fffc0004
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfde80004, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) ffffffff
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ce01
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ffffff01
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xce01, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffff800, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) fdefc004
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) ffffc004
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfdefc004, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) ffffffff
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0xffffffff, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0x0, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0xfffff800, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0x0, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xc000, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfea00000, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfea60000, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xe0000000, len=0x4)
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> > > > vfio: SKIPPING region_add feb40000 - feb4002f
> > > > vfio: SKIPPING region_add feb40800 - feb40807
> > > > vfio: SKIPPING region_add feb41000 - feb4101f
> > > > vfio: SKIPPING region_add feb41800 - feb41807
> > > > vfio: vfio_update_irq(0000:04:00.1) IRQ moved 20 -> 10
> > > > vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> > > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > > vfio: vfio_update_irq(0000:04:00.0) IRQ moved 23 -> 11
> > > > vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> > > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > > vfio: SKIPPING region_add feb42000 - feb42fff
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x3c, 0xb, len=0x1)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> > > > vfio: region_add e0000000 - efffffff [0x7f8688000000]
> > > > vfio: region_add fea00000 - fea03fff [0x7f88aa7b8000]
> > > > vfio: SKIPPING region_add fea04000 - fea04fff
> > > > vfio: region_add fea05000 - fea3ffff [0x7f88aa7bd000]
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x3c, 0xa, len=0x1)
> > > > vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 0
> > > > vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x103, len=0x2)
> > > > vfio: region_add fea60000 - fea63fff [0x7f88bc710000]
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> > > > vfio: region_del 0 - afffffff
> > > > vfio: region_add 0 - 9ffff [0x7f8698000000]
> > > > vfio: SKIPPING region_add a0000 - bffff
> > > > vfio: region_add c0000 - afffffff [0x7f86980c0000]
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fea40000
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffffffe, len=0x4)
> > > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> > > 
> > > Here the option ROM was sized
> > > 
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40001, len=0x4)
> > > 
> > > Then enabled
> > > 
> > > > vfio: region_add fea40000 - fea5ffff [0x7f88a9e00000]
> > > 
> > > Adding this memory region
> > > 
> > > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> > > > vfio: region_del fea40000 - fea5ffff
> > > 
> > > Then disabled, removing the memory region.  Presumably between the
> > > enable and disable the contents were read and copied to 0xc0000, which
> > > is where the VGA BIOS is shadowed.
> > > 
> > > > Here is the strace output from this failure:
> > > > 
> > > > 1110  ioctl(14, KVM_RUN, 0)             = 0
> > > > 1110  pread(20,  <unfinished ...>
> > > > 1099  <... poll resumed> )              = 1 ([{fd=0, revents=POLLIN}])
> > > > 1099  futex(0x7ff73ca62fa0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
> > > > 1109  <... futex resumed> )             = -1 ETIMEDOUT (Connection 
> > > > timed out)
> > > > 1109  madvise(0x7ff72fe17000, 8368128, MADV_DONTNEED) = 0
> > > > 1109  _exit(0)                          = ?
> > > > 1109  +++ exited with 0 +++
> > > > 
> > > > From reading the source 'hw/misc/vfio.c' it looks like the following
> > > > in 'vfio_vga_read' never finished:
> > > > 
> > > >     if (pread(vga->fd, &buf, size, offset) != size) {
> > > >         error_report("%s(,0x%"HWADDR_PRIx", %d) failed: %m",
> > > >                      __func__, region->offset + addr, size);
> > > >         return (uint64_t)-1;
> > > >     }
> > > 
> > > I agree.  Every VGA access requires us to lock the VGA resources on the
> > > device, so if we can't get the lock, we stop making progress.  I took a
> > > look at Xorg last night and it seems like it should be taking and
> > > releasing the VGA arbiter lock in a way that would be compatible with
> > > our use.  That's in the xserver, not the actual display hardware driver,
> > > and it wraps access functions in the arbiter support, so should be
> > > transparent to the drivers.  So for nouveau, it seems like it should
> > > work.  For nvidia, we don't really know, it could be locking the device
> > > from the kernel module.
> > > 
> > > You could instrument vga_get, vga_tryget, and vga_put to figure out
> > > what's happening.  It might be enough to look at /dev/vga_arbiter at
> > > each step in the sequence to reproduce (sudo head
> > > --lines=1 /dev/vga_arbiter).  Thanks,
> > > 
> > 
> > I've played a bit more with it and there are some differences in
> > behavior with 'nouveau' and 'nvidia' driver. As soon as I load the
> > binary driver I see the following:
> > 
> > [   18.628676] [drm] Initialized drm 1.1.0 20060810
> > [   18.668038] nvidia: module license 'NVIDIA' taints kernel.
> > [   18.668107] Disabling lock debugging due to kernel taint
> > [   18.676638] vgaarb: device changed decodes: 
> > PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
> > [   18.676722] vgaarb: transferring owner from PCI:0000:01:00.0 to 
> > PCI:0000:04:00.0
> > [   18.677007] [drm] Initialized nvidia-drm 0.0.0 20130102 for 0000:01:00.0 
> > on minor 0
> > [   18.677090] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  319.23  Thu 
> > May 16 19:36:02 PDT 2013
> > 
> > address@hidden:~# head --lines=1 /dev/vga_arbiter
> > count:2,PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none(0:0)
> > address@hidden:~# modprobe nvidia
> > address@hidden:~# head --lines=1 /dev/vga_arbiter
> > count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=io+mem(1:1)
> > address@hidden:~# /usr/local/bin/qemu-system-x86_64 \
> >         -L /usr/local/share/qemu \
> >         -L /usr/local/share/qemu \
> >         -M q35 -enable-kvm -cpu host -smp cores=4,threads=1,sockets=1 \
> >         -m 8192 -rtc base=localtime -k de -nodefaults -vga none \
> >         -drive 
> > file=/home/mbroemme/.kvm/maggie.img,id=drive0,if=none,cache=none,aio=threads
> >  \
> >         -device virtio-blk-pci,drive=drive0,ioeventfd=on \
> >         -device ioh3420,id=pcie0,multifunction=on \
> >         -device 
> > vfio-pci,host=04:00.0,addr=0.0,bus=pcie0,multifunction=on,x-vga=on \
> >         -device vfio-pci,host=04:00.1,addr=0.1,bus=pcie0 -monitor stdio 
> > -nographic
> > address@hidden:~# head --lines=1 /dev/vga_arbiter
> > count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=io+mem(1:1)
> > 
> > It looks like nvidia binary driver locks some resources. :( Whereas
> > with nouveau driver the VGA arbitration starts first if I start the
> > VM with VGA passthrough:
> > 
> > [  178.187706] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003)
> > [  178.209599] vfio_ecap_init: 0000:04:00.0 hiding ecap address@hidden
> > [  178.209631] vfio_ecap_init: 0000:04:00.0 hiding ecap address@hidden
> > [  181.198191] vgaarb: device changed decodes: 
> > PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
> > [  181.198208] vgaarb: transferring owner from PCI:0000:01:00.0 to 
> > PCI:0000:04:00.0
> > 
> > address@hidden:~# head --lines=1 /dev/vga_arbiter
> > count:2,PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none(0:0)
> > address@hidden:~# /usr/local/bin/qemu-system-x86_64 \
> >         -L /usr/local/share/qemu \
> >         -L /usr/local/share/qemu \
> >         -M q35 -enable-kvm -cpu host -smp cores=4,threads=1,sockets=1 \
> >         -m 8192 -rtc base=localtime -k de -nodefaults -vga none \
> >         -drive 
> > file=/home/mbroemme/.kvm/maggie.img,id=drive0,if=none,cache=none,aio=threads
> >  \
> >         -device virtio-blk-pci,drive=drive0,ioeventfd=on \
> >         -device ioh3420,id=pcie0,multifunction=on \
> >         -device 
> > vfio-pci,host=04:00.0,addr=0.0,bus=pcie0,multifunction=on,x-vga=on \
> >         -device vfio-pci,host=04:00.1,addr=0.1,bus=pcie0 -monitor stdio 
> > -nographic
> > address@hidden:~# head --lines=1 /dev/vga_arbiter
> > count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=none(0:0)
> > 
> 
> Alex you pointed me in the right direction, many thanks! I got it
> working with 'nvidia' binary driver now, but I had to patch it. I've
> attached it here just for reference if others want to try the same
> and I don't know if it is the proper way. After some testing (Xorg,
> DRI, VDPAU, switching between X and text-console) it looks like a stable
> workaround for me now until NVIDIA will fix their driver.

Hmm, the code doesn't make much sense to me.  They never do a vga_put,
so they must not realize that the vga_tryget is actually locking vga
arbitration.  It looks like they just want to point VGA routing to the
default device and indicate they don't use VGA I/O.  I don't really see
the point of the vga_tryget though since by declaring that they don't
use VGA I/O they give up all locks.  A simpler fix would be to simply
skip the vga_tryget call.  Or check the return value:

if (!vga_tryget(...))
    vga_put(...)

The VGA arbitration code is lazy, so VGA routing will still point to the
default devices, but it won't be locked there.  Thanks,

Alex