[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] [question] virtio-blk performance degradation happened
|
From: |
Zhang Haoyu |
|
Subject: |
Re: [Qemu-devel] [question] virtio-blk performance degradation happened with virito-serial |
|
Date: |
Fri, 12 Sep 2014 11:21:37 +0800 |
>>> > > If virtio-blk and virtio-serial share an IRQ, the guest operating
>>> > > system has to check each virtqueue for activity. Maybe there is some
>>> > > inefficiency doing that.
>>> > > AFAIK virtio-serial registers 64 virtqueues (on 31 ports + console)
>>> > > even if everything is unused.
>>> >
>>> > That could be the case if MSI is disabled.
>>>
>>> Do the windows virtio drivers enable MSIs, in their inf file?
>>
>>It depends on the version of the drivers, but it is a reasonable guess
>>at what differs between Linux and Windows. Haoyu, can you give us the
>>output of lspci from a Linux guest?
>>
>I made a test with fio on rhel-6.5 guest, the same degradation happened too,
>this degradation can be reproduced on rhel6.5 guest 100%.
>virtio_console module installed:
>64K-write-sequence: 285 MBPS, 4380 IOPS
>virtio_console module uninstalled:
>64K-write-sequence: 370 MBPS, 5670 IOPS
>
I use top -d 1 -H -p <qemu-pid> to monitor the cpu usage, and found that,
virtio_console module installed:
qemu main thread cpu usage: 98%
virtio_console module uninstalled:
qemu main thread cpu usage: 60%
perf top -p <qemu-pid> result,
virtio_console module installed:
PerfTop: 9868 irqs/sec kernel:76.4% exact: 0.0% [4000Hz cycles],
(target_pid: 88381)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
11.80% [kernel] [k] _raw_spin_lock_irqsave
8.42% [kernel] [k] _raw_spin_unlock_irqrestore
7.33% [kernel] [k] fget_light
6.28% [kernel] [k] fput
3.61% [kernel] [k] do_sys_poll
3.30% qemu-system-x86_64 [.] qcow2_check_metadata_overlap
3.10% [kernel] [k] __pollwait
2.15% qemu-system-x86_64 [.] qemu_iohandler_poll
1.44% libglib-2.0.so.0.3200.4 [.] g_array_append_vals
1.36% libc-2.13.so [.] 0x000000000011fc2a
1.31% libpthread-2.13.so [.] pthread_mutex_lock
1.24% libglib-2.0.so.0.3200.4 [.] 0x000000000001f961
1.20% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
0.99% [kernel] [k] eventfd_poll
0.98% [vdso] [.] 0x0000000000000771
0.97% [kernel] [k] remove_wait_queue
0.96% qemu-system-x86_64 [.] qemu_iohandler_fill
0.95% [kernel] [k] add_wait_queue
0.69% [kernel] [k] __srcu_read_lock
0.58% [kernel] [k] poll_freewait
0.57% [kernel] [k] _raw_spin_lock_irq
0.54% [kernel] [k] __srcu_read_unlock
0.47% [kernel] [k] copy_user_enhanced_fast_string
0.46% [kvm_intel] [k] vmx_vcpu_run
0.46% [kvm] [k] vcpu_enter_guest
0.42% [kernel] [k] tcp_poll
0.41% [kernel] [k] system_call_after_swapgs
0.40% libglib-2.0.so.0.3200.4 [.] g_slice_alloc
0.40% [kernel] [k] system_call
0.38% libpthread-2.13.so [.] 0x000000000000e18d
0.38% libglib-2.0.so.0.3200.4 [.] g_slice_free1
0.38% qemu-system-x86_64 [.] address_space_translate_internal
0.38% [kernel] [k] _raw_spin_lock
0.37% qemu-system-x86_64 [.] phys_page_find
0.36% [kernel] [k] get_page_from_freelist
0.35% [kernel] [k] sock_poll
0.34% [kernel] [k] fsnotify
0.31% libglib-2.0.so.0.3200.4 [.] g_main_context_check
0.30% [kernel] [k] do_direct_IO
0.29% libpthread-2.13.so [.] pthread_getspecific
virtio_console module uninstalled:
PerfTop: 9138 irqs/sec kernel:71.7% exact: 0.0% [4000Hz cycles],
(target_pid: 88381)
------------------------------------------------------------------------------------------------------------------------------
5.72% qemu-system-x86_64 [.] qcow2_check_metadata_overlap
4.51% [kernel] [k] fget_light
3.98% [kernel] [k] _raw_spin_lock_irqsave
2.55% [kernel] [k] fput
2.48% libpthread-2.13.so [.] pthread_mutex_lock
2.46% [kernel] [k] _raw_spin_unlock_irqrestore
2.21% libpthread-2.13.so [.] __pthread_mutex_unlock_usercnt
1.71% [vdso] [.] 0x000000000000060c
1.68% libc-2.13.so [.] 0x00000000000e751f
1.64% libglib-2.0.so.0.3200.4 [.] 0x000000000004fca0
1.20% [kernel] [k] __srcu_read_lock
1.14% [kernel] [k] do_sys_poll
0.96% [kernel] [k] _raw_spin_lock_irq
0.95% [kernel] [k] __pollwait
0.91% [kernel] [k] __srcu_read_unlock
0.78% [kernel] [k] tcp_poll
0.74% [kvm] [k] vcpu_enter_guest
0.73% [kvm_intel] [k] vmx_vcpu_run
0.72% [kernel] [k] _raw_spin_lock
0.72% [kernel] [k] system_call_after_swapgs
0.70% [kernel] [k] copy_user_enhanced_fast_string
0.67% libglib-2.0.so.0.3200.4 [.] g_slice_free1
0.66% libpthread-2.13.so [.] 0x000000000000e12d
0.65% [kernel] [k] system_call
0.61% [kernel] [k] do_direct_IO
0.57% qemu-system-x86_64 [.] qemu_iohandler_poll
0.57% [kernel] [k] fsnotify
0.54% libglib-2.0.so.0.3200.4 [.] g_slice_alloc
0.50% [kernel] [k] vfs_write
0.49% libpthread-2.13.so [.] pthread_getspecific
0.48% qemu-system-x86_64 [.] qemu_event_reset
0.47% libglib-2.0.so.0.3200.4 [.] g_main_context_check
0.46% qemu-system-x86_64 [.] address_space_translate_internal
0.46% [kernel] [k] sock_poll
0.46% libpthread-2.13.so [.] __pthread_disable_asynccancel
0.44% [kernel] [k] resched_task
0.43% libpthread-2.13.so [.] __pthread_enable_asynccancel
0.42% qemu-system-x86_64 [.] phys_page_find
0.39% qemu-system-x86_64 [.] object_dynamic_cast_assert
>And, virio-blk's interrupt mode always is MSI, no matter if virtio_console
>module is installed or uninstalled.
>25: 2245933 PCI-MSI-edge virtio1-requests
>
>fio command:
>fio -filename /dev/vda -direct=1 -iodepth=1 -thread -rw=write -ioengine=psync
>-bs=64k -size=30G -numjobs=1 -name=mytest
>
>QEMU comamnd:
>/usr/bin/kvm -id 5497356709352 -chardev
>socket,id=qmp,path=/var/run/qemu-server/5497356709352.qmp,server,nowait -mon
>chardev=qmp,mode=control -vnc :0,websocket,to=200 -enable-kvm -pidfile
>/var/run/qemu-server/5497356709352.pid -daemonize -name io-test-rhel-6.5 -smp
>sockets=1,cores=1 -cpu core2duo -nodefaults -vga cirrus -no-hpet -k en-us
>-boot menu=on,splash-time=8000 -m 4096 -usb -drive
>file=/sf/data/local/zhanghaoyu/rhel-server-6.5-x86_64-dvd.iso,if=none,id=drive-ide0,media=cdrom,aio=native,forecast=disable
> -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200 -drive
>file=/sf/data/local/images/host-1051721dff13/io-test-rhel-6.5.vm/vm-disk-1.qcow2,if=none,id=drive-virtio1,cache=none,aio=native
> -device virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb
>-drive
>file=/sf/data/local/images/host-1051721dff13/io-test-rhel-6.5.vm/vm-disk-2.qcow2,if=none,id=drive-virtio2,cache=none,aio=native
> -device virtio-blk-pci,drive=drive-virtio2,id=virtio2,bus=pc
i
> .0,addr=0xc,bootindex=101 -netdev
> type=tap,id=net0,ifname=164922379979200,script=/sf/etc/kvm/vtp-bridge,vhost=on,vhostforce=on
> -device
> virtio-net-pci,mac=FE:FC:FE:C6:47:F6,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
> -rtc driftfix=slew,clock=rt -global kvm-pit.lost_tick_policy=discard -global
> PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -chardev
> socket,path=/run/virtser/1649223799792.sock,server,nowait,id=channelser
> -device virtio-serial,vectors=4 -device
> virtserialport,chardev=channelser,name=channelser.virtserial0.0
>
>[environment]
>Host:linux-3.10(RHEL7-rc1)
>QEMU: qemu-2.0.1
>Guest: RHEL6.5
>
># lspci -tv
>-[0000:00]-+-00.0 Intel Corporation 440FX - 82441FX PMC [Natoma]
> +-01.0 Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
> +-01.1 Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
> +-01.2 Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II]
> +-01.3 Intel Corporation 82371AB/EB/MB PIIX4 ACPI
> +-02.0 Cirrus Logic GD 5446
> +-03.0 Red Hat, Inc Virtio console
> +-0b.0 Red Hat, Inc Virtio block device
> +-0c.0 Red Hat, Inc Virtio block device
> \-12.0 Red Hat, Inc Virtio network device
>
># lspci -vvv
>00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
> Subsystem: Red Hat, Inc Qemu virtual machine
> Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
>00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
> Subsystem: Red Hat, Inc Qemu virtual machine
> Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>
>00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
>(prog-if 80 [Master])
> Subsystem: Red Hat, Inc Qemu virtual machine
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0
> Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable)
> [size=8]
> Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable)
> Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable)
> [size=8]
> Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable)
> Region 4: I/O ports at c0e0 [size=16]
> Kernel driver in use: ata_piix
> Kernel modules: ata_generic, pata_acpi, ata_piix
>
>00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II]
>(rev 01) (prog-if 00 [UHCI])
> Subsystem: Red Hat, Inc Qemu virtual machine
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0
> Interrupt: pin D routed to IRQ 11
> Region 4: I/O ports at c080 [size=32]
> Kernel driver in use: uhci_hcd
>
>00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
> Subsystem: Red Hat, Inc Qemu virtual machine
> Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Interrupt: pin A routed to IRQ 9
> Kernel driver in use: piix4_smbus
> Kernel modules: i2c-piix4
>
>00:02.0 VGA compatible controller: Cirrus Logic GD 5446 (prog-if 00 [VGA
>controller])
> Subsystem: Red Hat, Inc Device 1100
> Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Region 0: Memory at fc000000 (32-bit, prefetchable) [size=32M]
> Region 1: Memory at febd0000 (32-bit, non-prefetchable) [size=4K]
> Expansion ROM at febc0000 [disabled] [size=64K]
> Kernel modules: cirrusfb
>
>00:03.0 Communication controller: Red Hat, Inc Virtio console
> Subsystem: Red Hat, Inc Device 0003
> Physical Slot: 3
> Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Interrupt: pin A routed to IRQ 10
> Region 0: I/O ports at c0a0 [size=32]
> Region 1: Memory at febd1000 (32-bit, non-prefetchable) [size=4K]
> Capabilities: [40] MSI-X: Enable- Count=4 Masked-
> Vector table: BAR=1 offset=00000000
> PBA: BAR=1 offset=00000800
> Kernel driver in use: virtio-pci
> Kernel modules: virtio_pci
>
>00:0b.0 SCSI storage controller: Red Hat, Inc Virtio block device
> Subsystem: Red Hat, Inc Device 0002
> Physical Slot: 11
> Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Interrupt: pin A routed to IRQ 10
> Region 0: I/O ports at c000 [size=64]
> Region 1: Memory at febd2000 (32-bit, non-prefetchable) [size=4K]
> Capabilities: [40] MSI-X: Enable+ Count=2 Masked-
> Vector table: BAR=1 offset=00000000
> PBA: BAR=1 offset=00000800
> Kernel driver in use: virtio-pci
> Kernel modules: virtio_pci
>
>00:0c.0 SCSI storage controller: Red Hat, Inc Virtio block device
> Subsystem: Red Hat, Inc Device 0002
> Physical Slot: 12
> Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Interrupt: pin A routed to IRQ 11
> Region 0: I/O ports at c040 [size=64]
> Region 1: Memory at febd3000 (32-bit, non-prefetchable) [size=4K]
> Capabilities: [40] MSI-X: Enable+ Count=2 Masked-
> Vector table: BAR=1 offset=00000000
> PBA: BAR=1 offset=00000800
> Kernel driver in use: virtio-pci
> Kernel modules: virtio_pci
>
>00:12.0 Ethernet controller: Red Hat, Inc Virtio network device
> Subsystem: Red Hat, Inc Device 0001
> Physical Slot: 18
> Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR+ FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Interrupt: pin A routed to IRQ 10
> Region 0: I/O ports at c0c0 [size=32]
> Region 1: Memory at febd4000 (32-bit, non-prefetchable) [size=4K]
> Expansion ROM at feb80000 [disabled] [size=256K]
> Capabilities: [40] MSI-X: Enable+ Count=3 Masked-
> Vector table: BAR=1 offset=00000000
> PBA: BAR=1 offset=00000800
> Kernel driver in use: virtio-pci
> Kernel modules: virtio_pci
>
>Thanks,
>Zhang Haoyu
- Re: [Qemu-devel] [question] virtio-blk performance degradationhappened with virito-serial, (continued)
- Re: [Qemu-devel] [question] virtio-blk performance degradationhappened with virito-serial, Zhang Haoyu, 2014/09/01
- Re: [Qemu-devel] [question] virtio-blk performance degradationhappened with virito-serial, Christian Borntraeger, 2014/09/01
- Re: [Qemu-devel] [question] virtio-blk performance degradationhappened with virito-serial, Paolo Bonzini, 2014/09/01
- Re: [Qemu-devel] [question] virtio-blk performance degradationhappened with virito-serial, Christian Borntraeger, 2014/09/01
- Re: [Qemu-devel] [question] virtio-blk performance degradationhappened with virito-serial, Paolo Bonzini, 2014/09/01
- Re: [Qemu-devel] [question] virtio-blk performance degradationhappened with virito-serial, Christian Borntraeger, 2014/09/01
- Re: [Qemu-devel] [question] virtio-blk performance degradationhappened with virito-serial, Christian Borntraeger, 2014/09/01
- Re: [Qemu-devel] [question] virtio-blk performance degradationhappenedwith virito-serial, Zhang Haoyu, 2014/09/04
- Re: [Qemu-devel] [question] virtio-blk performance degradationhappenedwith virito-serial, Zhang Haoyu, 2014/09/07
- Re: [Qemu-devel] [question] virtio-blk performance degradationhappenedwith virito-serial, Amit Shah, 2014/09/11
- Re: [Qemu-devel] [question] virtio-blk performance degradation happened with virito-serial,
Zhang Haoyu <=
- Re: [Qemu-devel] [question] virtio-blk performance degradation happened with virito-serial, Stefan Hajnoczi, 2014/09/12
- Re: [Qemu-devel] [question] virtio-blk performance degradation happened with virito-serial, Max Reitz, 2014/09/13
- Re: [Qemu-devel] [question] virtio-blk performance degradation happened with virito-serial, Zhang Haoyu, 2014/09/16
Re: [Qemu-devel] [question] virtio-blk performance degradationhappened with virito-serial, Amit Shah, 2014/09/02
Re: [Qemu-devel] [question] virtio-blk performancedegradationhappened with virito-serial, Zhang Haoyu, 2014/09/03
Re: [Qemu-devel] [question] virtio-blk performance degradationhappened with virito-serial, Fam Zheng, 2014/09/19
Re: [Qemu-devel] [question] virtio-blk performance degradationhappened with virito-serial, Paolo Bonzini, 2014/09/19