[Qemu-devel] [Bug 1829696] Re: qemu-kvm takes 100% CPU when running redhat/centos 7.6 guest VM OS

[Kashyap Chamarthy]

** Project changed: nova => qemu

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1829696

Title:
  qemu-kvm takes 100% CPU when running redhat/centos 7.6 guest VM OS

Status in QEMU:
  New

Bug description:
  Description
  ===========
  When running redhat or centos 7.6 guest os on vm,
  the cpu usage is very low on vm(100% idle), but on host,
  qemu-kvm reports 100% cpu busy usage.

  After searching some related bugs report,
  I suspect that it is due to the clock settings in vm's domain xml.
  My Openstack cluster uses the default clock settings as follow:
      <clock offset='utc'>
        <timer name='rtc' tickpolicy='catchup'/>
        <timer name='pit' tickpolicy='delay'/>
        <timer name='hpet' present='no'/>
      </clock>
  And in this report, https://bugs.launchpad.net/qemu/+bug/1174654
  it claims that <timer name='rtc' track='guest'/> can solve the 100% cpu usage 
problem when using Windows Image Guest OS,
  but I makes some tests, the solusion dose not work for me.

  
  Steps to reproduce
  ==================
  * create a vm using centos or redhat 7.6 image
  * using sar tool inside vm and host to check the cpu usage, and compare them

  
  Expected result
  ===============
  host's cpu usage report should be same with vm's cpu usage

  
  Actual result
  =============
  vm's cpu usage is 100% idle, host's cpu usage is 100% busy

  
  Environment
  ===========
  1. Exact version of OpenStack you are running.
  # rpm -qa | grep nova
  openstack-nova-compute-13.1.2-1.el7.noarch
  python2-novaclient-3.3.2-1.el7.noarch
  python-nova-13.1.2-1.el7.noarch
  openstack-nova-common-13.1.2-1.el7.noarch

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?
  # libvirtd -V
  libvirtd (libvirt) 3.9.0

  # /usr/libexec/qemu-kvm --version
  QEMU emulator version 2.6.0 (qemu-kvm-ev-2.6.0-28.el7_3.6.1), Copyright (c) 
2003-2008 Fabrice Bellard

  
  Logs & Configs
  ==============
  The VM xml:
  <domain type='kvm' id='29'>
    <name>instance-00005022</name>
    <uuid>7f5a66a5-****-****-****-75dec****bbb</uuid>
    <metadata>
      <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
        <nova:package version="13.1.2-1.el7"/>
        <nova:name>*******</nova:name>
        <nova:creationTime>2019-05-20 03:08:46</nova:creationTime>
        <nova:flavor name="2d2dab36-****-****-****-246e9****110">
          <nova:memory>2048</nova:memory>
          <nova:disk>12</nova:disk>
          <nova:swap>2048</nova:swap>
          <nova:ephemeral>0</nova:ephemeral>
          <nova:vcpus>1</nova:vcpus>
        </nova:flavor>
        <nova:owner>
          <nova:user uuid="********************">****</nova:user>
          <nova:project uuid="********************">****</nova:project>
        </nova:owner>
        <nova:root type="image" uuid="4496a420-****-****-****-b50f****ada3"/>
      </nova:instance>
    </metadata>
    <memory unit='KiB'>2097152</memory>
    <currentMemory unit='KiB'>2097152</currentMemory>
    <vcpu placement='static'>1</vcpu>
    <cputune>
      <shares>1024</shares>
      <vcpupin vcpu='0' cpuset='27'/>
      <emulatorpin cpuset='27'/>
    </cputune>
    <numatune>
      <memory mode='strict' nodeset='1'/>
      <memnode cellid='0' mode='strict' nodeset='1'/>
    </numatune>
    <resource>
      <partition>/machine</partition>
    </resource>
    <sysinfo type='smbios'>
      <system>
        <entry name='manufacturer'>Fedora Project</entry>
        <entry name='product'>OpenStack Nova</entry>
        <entry name='version'>13.1.2-1.el7</entry>
        <entry name='serial'>64ab0e89-****-****-****-05312ef66983</entry>
        <entry name='uuid'>7f5a66a5-****-****-****-75decaf82bbb</entry>
        <entry name='family'>Virtual Machine</entry>
      </system>
    </sysinfo>
    <os>
      <type arch='x86_64' machine='pc-i440fx-rhel7.3.0'>hvm</type>
      <boot dev='hd'/>
      <smbios mode='sysinfo'/>
    </os>
    <features>
      <acpi/>
      <apic/>
    </features>
    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>IvyBridge</model>
      <topology sockets='1' cores='1' threads='1'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='arat'/>
      <feature policy='require' name='xsaveopt'/>
      <numa>
        <cell id='0' cpus='0' memory='2097152' unit='KiB'/>
      </numa>
    </cpu>
    <clock offset='utc'>
      <timer name='pit' tickpolicy='delay'/>
      <timer name='rtc' tickpolicy='catchup'/>
      <timer name='hpet' present='no'/>
    </clock>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>destroy</on_crash>
    <devices>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <disk type='file' device='disk'>
        <driver name='qemu' type='raw' cache='none'/>
        <source 
file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk'/>
        <backingStore/>
        <target dev='vda' bus='virtio'/>
        <alias name='virtio-disk0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' 
function='0x0'/>
      </disk>
      <disk type='file' device='disk'>
        <driver name='qemu' type='raw' cache='none'/>
        <source 
file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk.swap'/>
        <backingStore/>
        <target dev='vdb' bus='virtio'/>
        <alias name='virtio-disk1'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x05' 
function='0x0'/>
      </disk>
      <disk type='file' device='cdrom'>
        <driver name='qemu' type='raw' cache='none'/>
        <source 
file='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/disk.config'/>
        <backingStore/>
        <target dev='hdd' bus='ide'/>
        <readonly/>
        <alias name='ide0-1-1'/>
        <address type='drive' controller='0' bus='1' target='0' unit='1'/>
      </disk>
      <controller type='usb' index='0' model='piix3-uhci'>
        <alias name='usb'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x2'/>
      </controller>
      <controller type='pci' index='0' model='pci-root'>
        <alias name='pci.0'/>
      </controller>
      <controller type='ide' index='0'>
        <alias name='ide'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x1'/>
      </controller>
      <interface type='bridge'>
        <mac address='fa:16:3e:a6:ea:4f'/>
        <source bridge='brq52c66dc3-64'/>
        <bandwidth>
          <inbound average='102400'/>
          <outbound average='102400'/>
        </bandwidth>
        <target dev='tapa29e94e5-42'/>
        <model type='virtio'/>
        <alias name='net0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x0'/>
      </interface>
      <serial type='file'>
        <source 
path='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/console.log'/>
        <target type='isa-serial' port='0'>
          <model name='isa-serial'/>
        </target>
        <alias name='serial0'/>
      </serial>
      <serial type='pty'>
        <source path='/dev/pts/10'/>
        <target type='isa-serial' port='1'>
          <model name='isa-serial'/>
        </target>
        <alias name='serial1'/>
      </serial>
      <console type='file'>
        <source 
path='/data/instances/7f5a66a5-****-****-****-75decaf82bbb/console.log'/>
        <target type='serial' port='0'/>
        <alias name='serial0'/>
      </console>
      <input type='tablet' bus='usb'>
        <alias name='input0'/>
        <address type='usb' bus='0' port='1'/>
      </input>
      <input type='mouse' bus='ps2'>
        <alias name='input1'/>
      </input>
      <input type='keyboard' bus='ps2'>
        <alias name='input2'/>
      </input>
      <graphics type='vnc' port='5910' autoport='yes' listen='0.0.0.0' 
keymap='en-us'>
        <listen type='address' address='0.0.0.0'/>
      </graphics>
      <video>
        <model type='cirrus' vram='16384' heads='1' primary='yes'/>
        <alias name='video0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x0'/>
      </video>
      <memballoon model='virtio'>
        <stats period='10'/>
        <alias name='balloon0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x06' 
function='0x0'/>
      </memballoon>
    </devices>
    <seclabel type='dynamic' model='dac' relabel='yes'>
      <label>+107:+107</label>
      <imagelabel>+107:+107</imagelabel>
    </seclabel>
  </domain>

  CPU Usage Report inside VM:
  # sar -u -P 0 1 5
  Linux 3.10.0-957.el7.x86_64 (******)  05/20/2019      _x86_64_        (1 CPU)

  11:34:40 AM     CPU     %user     %nice   %system   %iowait    %steal     
%idle
  11:34:41 AM       0      0.00      0.00      0.00      0.00      0.00    
100.00
  11:34:42 AM       0      0.00      0.00      0.00      0.00      0.00    
100.00
  11:34:43 AM       0      0.00      0.00      0.00      0.00      0.00    
100.00
  11:34:44 AM       0      0.00      0.00      0.00      0.00      0.00    
100.00
  11:34:45 AM       0      0.00      0.00      0.00      0.00      0.00    
100.00
  Average:          0      0.00      0.00      0.00      0.00      0.00    
100.00

  CPU Usage Report ON HOST(the vm's cpu is pinned on host's no.27 physic cpu):
  # sar -u -P 27 1 5
  Linux 3.10.0-862.el7.x86_64 (******)  05/20/2019      _x86_64_        (48 CPU)

  11:34:40 AM     CPU     %user     %nice   %system   %iowait    %steal     
%idle
  11:34:41 AM      27    100.00      0.00      0.00      0.00      0.00      
0.00
  11:34:42 AM      27    100.00      0.00      0.00      0.00      0.00      
0.00
  11:34:43 AM      27    100.00      0.00      0.00      0.00      0.00      
0.00
  11:34:44 AM      27    100.00      0.00      0.00      0.00      0.00      
0.00
  11:34:45 AM      27    100.00      0.00      0.00      0.00      0.00      
0.00
  Average:         27    100.00      0.00      0.00      0.00      0.00      
0.00

  clocksource inside VM:
  # cat /sys/devices/system/clocksource/clocksource0/current_clocksource
  kvm_clock

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1829696/+subscriptions