Re: [Qemu-devel] [Bug?] qemu abort when trying to passthrough BCM5719 Gigabit Ethernet

[Alex Williamson]

On Sat, 2014-10-11 at 13:58 +0800, zhanghailiang wrote:
> Hi all,
> 
> When i try to passthrough BCM5719 Gigabit Ethernet to guest using the qemu 
> master branch, it aborted,
> and show kvm_set_phys_mem:error registering slot:Bad Address.
> 
> qemu command:
> #./qemu/qemu/x86_64-softmmu/qemu-system-x86_64 --enable-kvm -smp 4 -m 4096 
> -vnc :99 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3  -drive 
> file=/home/suse11_sp3_64,if=none,id=drive-scsi0-0-0-0,format=raw,cache=none,aio=native
>  -device scsi-hd,bus=scsi0.0,channel=0,scsi-
> id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 -device 
> pci-assign,host=01:00.1,id=mydevice -net none
> 
> info about guest and host:
> host OS: 3.16.5
> *guest OS: Novell SuSE Linux Enterprise Server 11 SP3*
> #cat /proc/cpuinfo
> processor       : 31
> vendor_id       : GenuineIntel
> cpu family      : 6
> model           : 62
> model name      : Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
> stepping        : 4
> microcode       : 0x416
> cpu MHz         : 1926.875
> cache size      : 20480 KB
> physical id     : 1
> siblings        : 16
> core id         : 7
> cpu cores       : 8
> apicid          : 47
> initial apicid  : 47
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 13
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
> pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology 
> nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx
> smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt 
> tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln
> pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
> bogomips        : 4005.35
> clflush size    : 64
> cache_alignment : 64
> address sizes   : 46 bits physical, 48 bits virtual
> power management:
> 
> gdb info:
> (gdb) bt
> #0  0x00007ffff1ce9989 in raise () from /usr/lib64/libc.so.6
> #1  0x00007ffff1ceb098 in abort () from /usr/lib64/libc.so.6
> #2  0x00005555556275cf in kvm_set_phys_mem (section=0x7fffedcea790, add=true) 
> at /home/qemu/qemu/kvm-all.c:711
> #3  0x000055555562980f in address_space_update_topology_pass (address@hidden 
> <address_space_memory>, address@hidden,
>      new_view=<optimized out>, new_view=<optimized out>, 
> old_view=0x7fffe8022a90, old_view=0x7fffe8022a90) at 
> /home/qemu/qemu/memory.c:752
> #4  0x000055555562b910 in address_space_update_topology (as=0x555555d01ca0 
> <address_space_memory>) at /home/qemu/qemu/memory.c:767
> #5  memory_region_transaction_commit () at /home/qemu/qemu/memory.c:808
> #6  0x00005555557a75b4 in pci_update_mappings (d=0x5555562ba9f0) at 
> hw/pci/pci.c:1113
> #7  0x00005555557a7932 in pci_default_write_config (address@hidden, 
> address@hidden, address@hidden, address@hidden)
>      at hw/pci/pci.c:1165
> #8  0x000055555566c17e in assigned_dev_pci_write_config 
> (pci_dev=0x5555562ba9f0, address=20, val=4294967295, len=4)
>      at /home/qemu/qemu/hw/i386/kvm/pci-assign.c:1196
> #9  0x0000555555628fea in access_with_adjusted_size (address@hidden, 
> address@hidden, address@hidden,
>      access_size_min=<optimized out>, access_size_max=<optimized out>, 
> access=0x555555629160 <memory_region_write_accessor>, mr=0x555556231f00)
>      at /home/qemu/qemu/memory.c:480
> #10 0x000055555562dbf7 in memory_region_dispatch_write (size=4, 
> data=18446744073709551615, addr=0, mr=0x555556231f00) at 
> /home/qemu/qemu/memory.c:1122
> #11 io_mem_write (address@hidden, addr=0, val=<optimized out>, size=4) at 
> /home/qemu/qemu/memory.c:1958
> #12 0x00005555555f8963 in address_space_rw (as=0x555555d01d80 
> <address_space_io>, address@hidden, buf=0x7ffff7fec000 "\377\377\377\377",
>      address@hidden, address@hidden) at /home/qemu/qemu/exec.c:2145
> #13 0x0000555555628491 in kvm_handle_io (count=1, size=4, 
> direction=<optimized out>, data=<optimized out>, port=3324) at 
> /home/qemu/qemu/kvm-all.c:1614
> #14 kvm_cpu_exec (address@hidden) at /home/qemu/qemu/kvm-all.c:1771
> #15 0x0000555555617182 in qemu_kvm_cpu_thread_fn (arg=0x55555620e610) at 
> /home/qemu/qemu/cpus.c:953
> #16 0x00007ffff6ba2df3 in start_thread () from /usr/lib64/libpthread.so.0
> #17 0x00007ffff1daa3dd in clone () from /usr/lib64/libc.so.6
> 
> messages log:
> Oct 10 07:43:18 localhost kernel: kvm: zapping shadow pages for mmio 
> generation wraparound
> Oct 10 07:43:27 localhost kernel: kvm [13251]: vcpu0 disabled perfctr wrmsr: 
> 0xc1 data 0xabcd
> Oct 10 07:43:28 localhost kernel: intel_iommu_map: iommu width (48) is not 
> sufficient for the mapped address (fffffffffe001000)
> Oct 10 07:43:28 localhost kernel: kvm_iommu_map_address:iommu failed to map 
> pfn=94880
> 
> After bisected the commits, i found the commit 453c43a4a241a7 leads to this 
> problem.
> 
> commit 57271d63c4d93352406704d540453c43a4a241a7
> Author: Paolo Bonzini <address@hidden>
> Date:   Thu Nov 7 17:14:37 2013 +0100
> 
>      exec: make address spaces 64-bit wide
> 
>      As an alternative to commit 818f86b (exec: limit system memory
>      size, 2013-11-04) let's just make all address spaces 64-bit wide.
>      This eliminates problems with phys_page_find ignoring bits above
>      TARGET_PHYS_ADDR_SPACE_BITS and address_space_translate_internal
>      consequently messing up the computations.
> 
>      In Luiz's reported crash, at startup gdb attempts to read from address
>      0xffffffffffffffe6 to 0xffffffffffffffff inclusive.  The region it gets
>      is the newly introduced master abort region, which is as big as the PCI
>      address space (see pci_bus_init).  Due to a typo that's only 2^63-1,
>      not 2^64.  But we get it anyway because phys_page_find ignores the upper
>      bits of the physical address.  In address_space_translate_internal then
> 
>          diff = int128_sub(section->mr->size, int128_make64(addr));
>          *plen = int128_get64(int128_min(diff, int128_make64(*plen)));
> 
>      diff becomes negative, and int128_get64 booms.
> 
>      The size of the PCI address space region should be fixed anyway
> 
> For it works fine when we use VFIO way, i don't know if it is really a QEMU's 
> bug or KVM's bug.
> Any help will be appreciated.

VFIO had to be updated to ignore certain high address mappings that are
outside of the IOMMU address space.  Legacy KVM would need a similar
change somewhere in the IOMMU mapping path in the KVM kernel module.
Legacy KVM assignment is deprecated though and hopefully the fact that
it has taken this long to see a bug report indicates that users are
switching to VFIO.  Thanks,

Alex