[Qemu-devel] Help with Qemu/KVM performance counters

[Abdullah Alhaddad]

Hi all,

I hope you're all doing well.

As i was looking for a solution for a particular problem in Qemu/KVM
virtulization.

My issue is that I have a virtual machine that runs well in VMware and when
I migrated that to Qemu/KVM-enabled environment, it didn't work! I figured
out that under VMware hypervisor, VMware supplies CPU TSC and Performance
Counters values to the guest VM with the option
"monitor_control.pseudo_perfctr = TRUE" set the vmx configuration file,
Ref.: https://www.vmware.com/pdf/vmware_timekeeping.pdf

My question is, is there any similar option in Qemu/KVM-enabled environment
that I can use to get my VM working the same way as in the VMware
environment?

I almost tried all options in Qemu with regards to CPU but no avail.

To elaborate more, the VM I'm trying to port under Qemu/KVM environment is
a an old version of Cisco virtual ASA Firewall. The VM image is actually
meant to be run under VMware ESXi and with that
"*monitor_control.pseudo_perfctr
= TRUE*" option it can also run in Vware Workstation as well. *Yes, this
option that makes it run under VMware and if it's removed from the
configuration vmx file then the VM boots half way and crashes the same way
it crashes under Qemu*. That dictates it's the option in interest that
needs to be found in Qemu/KVM. I have a copy of this VM in the below link
in case you would like to try its behavior in under VMware. I downloaded it
from a youtube previously to test it out:

https://drive.google.com/open?id=1SEXws18hoj2sWGk8iFqqH8RpBZsBNpRH

Once you power on the VM you can telnet to 127.0.0.1 on port 3000 to see
the boot process. If you remove that option i mentioned to you and boot the
VM again you'll see the crashing in process.


I've converted that vmdk disk images into Qemu disks "qcow2" format and i
ran them using the below command line on Ubuntu:

/opt/qemu/bin/qemu-system-x86_64 -L -nographic -device
e1000-82545em,netdev=net0,mac=50:00:00:6a:00:00 -netdev
tap,id=net0,ifname=vunl0_33_0,script=no -device
e1000-82545em,netdev=net1,mac=50:00:00:6a:00:01 -netdev
tap,id=net1,ifname=vunl0_33_1,script=no -device
e1000-82545em,netdev=net2,mac=50:00:00:6a:00:02 -netdev
tap,id=net2,ifname=vunl0_33_2,script=no -device
e1000-82545em,netdev=net3,mac=50:00:00:6a:00:03 -netdev
tap,id=net3,ifname=vunl0_33_3,script=no -machine type=pc-1.0  *-cpu
host,migratable=off,invtsc=on,pmu=on* -m 4096 -hda hda.qcow2 -hdb hdb.qcow2
-serial telnet:0.0.0.0:3000,server,nowait -monitor
tcp:127.0.0.1:42379,server,nowait
-nographic -display none -enable-kvm


Once you power on the VM you can telnet to xx.xx.xx.xx 3000 (where the xx
IP is the Ubuntu machine IP) to see the crashing in process. You may need
to wait for a while for the status messages to appear in the terminal
window.

I assume it's a cpu issue because in page 9 of the Vmware pdf reference
file; it says there are machine instructions become available when this
option "*monitor_control.pseudo_perfctr = TRUE*" is enabled:

The following machine instructions then become available:

Instruction    Time Value    Returned
rdpmc           0x10000       Physical host TSC
rdpmc           0x10001       Elapsed real time in ns
rdpmc           0x10002       Elapsed apparent time in ns

Therefore, I used many of the Qemu cpu options such as these:

-cpu host,migratable=no,+invtsc (ref: https://wiki.qemu.org/ChangeLog/2.1)
-cpu host, tsc-frequency=xxxx (ref: https://lists.gnu.org/archive/
html/qemu-devel/2017-01/msg03555.html)
 -cpu host,migratable=off,invtsc=true,pmu=true

Not sure if i'm hitting the wrong option!

The log I'm getting when the VM boots up looks like the following crash
happens at the blue colored log:

----------------------------------------------------------------------------------------------------------------------------
Loading...

Starting image verification
Hash Computation:    100% Done!
Computed Hash   SHA2: 63c1e8aa9de3d0c6e738dc91be8e1784
                      5caf64af4cf06cf6a3c5da7200d478dd
                      938d380d2b1064f6a349401c7860f50e
                      cc4eeb98a0ae16c097dbc9447d4d6626

Get key records from key storage: Primary, key_store_type: 2
Embedded Hash   SHA2: 63c1e8aa9de3d0c6e738dc91be8e1784
                      5caf64af4cf06cf6a3c5da7200d478dd
                      938d380d2b1064f6a349401c7860f50e
                      cc4eeb98a0ae16c097dbc9447d4d6626

The digital signature of the running image verified successfully
Processor memory 3183296512, Reserved memory: 0

Total NICs found: 4
i82545EM rev03 Gigabit Ethernet @ irq10 dev 6 index 03 MAC: 5000.006a.0003
i82545EM rev03 Gigabit Ethernet @ irq10 dev 5 index 02 MAC: 5000.006a.0002
i82545EM rev03 Gigabit Ethernet @ irq11 dev 4 index 01 MAC: 5000.006a.0001
i82545EM rev03 Gigabit Ethernet @ irq11 dev 3 index 00 MAC: 5000.006a.0000

Thread Name: lina_flash_init_thread
Page fault: Unknown
        r8 0x0000000000000790
        r9 0x00007fff3fa8b000
       r10 0x0000000000000001
       r11 0x000000000210e130
       r12 0x00000000062ebfc0
       r13 0x0000000000010001
       r14 0x0000000000000000
       r15 0x00000000062ebfc0
       rdi 0x00000000062ebfc0
       rsi 0x0000000006c17c20
       rbp 0x00007fff4056f4e0
       rbx 0x00000000062ebfc0
       rdx 0x00007fff40566f10
       rax 0x0000000000000001
       rcx 0x0000000000010001
       rsp 0x00007fff4056f4b0
       rip 0x0000000001581130
    eflags 0x0000000000013202
    csgsfs 0x0000000000000033
error code 0x0000000000000000
    vector 0x000000000000000d
  old mask 0xffffffde3e3b5a05
       cr2 0x0000000000000000

Cisco Adaptive Security Appliance Software Version 9.3(1)

Compiled on Wed 23-Jul-14 18:16 PDT by builders
Hardware:   ASAv
Crashinfo collected on 03:42:24.059 UTC Tue Nov 28 2017

Traceback:
0: 0x0000000000422118
1: 0x0000000000422152
2: 0x0000000000424331
3: 0x00000000015874a9
4: 0x00007ffffecd55f0
5: 0x0000000000558d85
6: 0x00000000008f5a2b
7: 0x00000000008fd361
8: 0x0000000000428a15
Stack dump: base:0x00007fff4056f2e0 size:178, active:178
     entries above '==': return PC preceded by input parameters
     entries below '==': local variables followed by saved regs
                 '==Fn': stack frame n, contains next stack frame
                    '*': stack pointer at crash
 rdi rsi rdx rcx r8 r9 : Arguments 1 through 6 to leaf function
 For example:
    0x00007fffeeeeef00: 0x0000000000000009     : arg9
    0x00007fffeeeeeefc: 0x0000000000000008     : arg8
    0x00007fffeeeeeef8: 0x0000000000000007     : arg7
    0x00007fffeeeeeef4: 0x0000000000000abc     : return PC
    0x00007fffeeeeeef0: 0x00007fffeeeeef20 ==F2: stack frame F2
    0x00007fffeeeeeeec: 0x0000000000000def     : local variable
    0x00007fffeeeeeee8: 0x0000000000000123     : local variable or saved reg
    0x00007fffeeeeeee4: 0x0000000000000456     : local variable or saved reg
    0x00007fffeeeeeee0: 0x0000000000000789     : local variable or saved reg
0x00007fff4056f870: 0x00007fff4056f7e0
0x00007fff4056f868: 0x0000000000000000
0x00007fff4056f860: 0x00000038a11c0123
0x00007fff4056f858: 0x0000000000000083
0x00007fff4056f850: 0x00007fff16a864c8
0x00007fff4056f848: 0x0000000000000000
0x00007fff4056f840: 0x00000000a11ccdef
0x00007fff4056f838-0x00007fff4056f808: 0x0000000000000000
0x00007fff4056f800: 0x0000000000429867
0x00007fff4056f7f8: 0x00007fff4056f860
0x00007fff4056f7f0: 0x00007fff40567100
0x00007fff4056f7e8: 0x0000000000000000
0x00007fff4056f7e0: 0x00000030a11c0123
0x00007fff4056f7d8: 0x0000000000000083
0x00007fff4056f7d0: 0x00007fff16a864c8
0x00007fff4056f7c8: 0x0000000000000000
0x00007fff4056f7c0: 0x00000000a11ccdef
0x00007fff4056f7b8: 0x0fffffff0fffffff
0x00007fff4056f7b0-0x00007fff4056f7a8: 0x0000000000000000
0x00007fff4056f7a0: 0x00000000062cc8a0
0x00007fff4056f798: 0x0000000000000000
0x00007fff4056f790: 0x00007fff4056f6e0
0x00007fff4056f788: 0x00007fff4056f758
0x00007fff4056f780: 0x0000000000000000
0x00007fff4056f778: 0x00007fff3ff48620
0x00007fff4056f770-0x00007fff4056f730: 0x0000000000000000
0x00007fff4056f728: 0x0000000004d14940
0x00007fff4056f720: 0x000000000041d690
0x00007fff4056f718: 0x0000000002777640
0x00007fff4056f710: 0x0000000200010010
0x00007fff4056f708: 0x0000000006c17d40
0x00007fff4056f700: 0x00007fff4056f6e0
0x00007fff4056f6f8: 0x00007fff40150e80
0x00007fff4056f6f0: 0x000000000638e598
0x00007fff4056f6e8: 0x00007fff3ff48620
0x00007fff4056f6e0: 0x00007fff4056f778
0x00007fff4056f6d8: 0x00000000deadfeed
0x00007fff4056f6d0-0x00007fff4056f6c8: 0x0000000000000000
0x00007fff4056f6c0: 0x000000000041e1f6
0x00007fff4056f6b8: 0x00007fff40571fd0
0x00007fff4056f6b0: 0x00007fff40560cf0
0x00007fff4056f6a8: 0x0000000000000000
0x00007fff4056f6a0: 0x000000f0a11c0123
0x00007fff4056f698: 0x0000000000000143
0x00007fff4056f690: 0x00007fff16a864c8
0x00007fff4056f688: 0x0000000000000000
0x00007fff4056f680: 0x00000000a11ccdef
0x00007fff4056f678-0x00007fff4056f660: 0x0000000000000000 ==F5
0x00007fff4056f658: 0x000000009abcdef0
0x00007fff4056f650-0x00007fff4056f5b8: 0x123456789abcdef0
0x00007fff4056f5b0: 0x0000000000428a01
0x00007fff4056f5a8: 0x00007fff4056f570
0x00007fff4056f5a0-0x00007fff4056f590: 0x0000000000000000
0x00007fff4056f588: 0xffffffffffffdf98
0x00007fff4056f580: 0x00007fff4056f670
0x00007fff4056f578: 0x00007fff3ff48370
0x00007fff4056f570: 0x0000000000000000
0x00007fff4056f568: 0x0000000000428a15
0x00007fff4056f560: 0x00007fff4056f670 ==F4
0x00007fff4056f558: 0x00000000008fd361
0x00007fff4056f550: 0x00007fff4056f560 ==F3
0x00007fff4056f548: 0x00000000008f5a2b
0x00007fff4056f540: 0x00007fff4056f550 ==F2
0x00007fff4056f538: 0x0000000000000000
0x00007fff4056f530: 0xffffffffffffdf98
0x00007fff4056f528: 0x00007fff3ff48370
0x00007fff4056f520: 0x00000000008fba90
0x00007fff4056f518: 0x00000000008fb908
0x00007fff4056f510: 0x00007fff4056f550
0x00007fff4056f508: 0x00000000008fb87e
0x00007fff4056f500: 0x00007fff4056f510
0x00007fff4056f4f8: 0x0000000000000000
0x00007fff4056f4f0: 0xffffffffffffdf98
0x00007fff4056f4e8: 0x0000000000558d85
0x00007fff4056f4e0: 0x00007fff4056f540 ==F1
0x00007fff4056f4d8-0x00007fff4056f4d0: 0x0000000000000000
0x00007fff4056f4c8: 0x0000000000000001
0x00007fff4056f4c0-0x00007fff4056f4b8: 0x00000000062ebfc0
0x00007fff4056f4b0: 0x0000000000000000 *
0x00007fff4056f4a8: 0x00000000008fd973
0x00007fff4056f4a0: 0x00007fff4056f4d0
0x00007fff4056f498: 0x00007fff40563908
0x00007fff4056f490: 0x00007fff4056f4d0
0x00007fff4056f488: 0x00000000009d4b01
0x00007fff4056f480: 0x00007fff4056f4a0
0x00007fff4056f478-0x00007fff4056f470: 0x0000000000000000
0x00007fff4056f468: 0x00007fff418d6390
0x00007fff4056f460: 0x0000000000000000
0x00007fff4056f458: 0x000000000201b9f8
0x00007fff4056f450: 0x00007fff4056f480
0x00007fff4056f448: 0x00007fff40563908
0x00007fff4056f440: 0x0000000000000001
0x00007fff4056f438: 0x00007fff405619a0
0x00007fff4056f430: 0x00007fff40563908
0x00007fff4056f428: 0x0000000000000001
0x00007fff4056f420: 0x0000000000000000
0x00007fff4056f418: 0x0000000001627125
0x00007fff4056f410: 0x00007fff4056f450
0x00007fff4056f408: 0x00007fff3fa8b010
0x00007fff4056f400: 0x00007fff46505845
0x00007fff4056f3f8-0x00007fff4056f3c8: 0x0000000000000000
0x00007fff4056f3c0: 0x0000000000000003
0x00007fff4056f3b8-0x00007fff4056f3a8: 0x0000000000000000
0x00007fff4056f3a0: 0x0000000000000240
0x00007fff4056f398: 0x0000000000000003
0x00007fff4056f390: 0x0000024446505853
0x00007fff4056f388-0x00007fff4056f310: 0x0000000000000000
0x00007fff4056f308: 0x424b7e25fece8fc2
0x00007fff4056f300: 0x2cc4f98473045e95
0x00007fff4056f2f8: 0x18fa9b6c57ca0e78
0x00007fff4056f2f0: 0x081e2a254ab96aa4
0x00007fff4056f2e8: 0x0000000300000000

Begin to dump crashinfo to flash....

core0: An internal error occurred.  Specifically, a programming assertion
was
violated.  Copy the error message exactly as it appears, and get the
output of the show version command and the contents of the configuration
file.  Then call your technical support representative.

assertion "_vf_mode_init" failed: file "vf_api.c", line 136
core0 same core snap_count=1 signo=6 RIP=7ffffecd43fb


-----------------------------------------------
Traceback output aborted.
Flushing first exception frame:
Page fault: Unknown
        r8 0x0000000000000790
        r9 0x00007fff3fa8b000
       r10 0x0000000000000001
       r11 0x000000000210e130
       r12 0x00000000062ebfc0
       r13 0x0000000000010001
       r14 0x0000000000000000
       r15 0x00000000062ebfc0
       rdi 0x00000000062ebfc0
       rsi 0x0000000006c17c20
       rbp 0x00007fff4056f4e0
       rbx 0x00000000062ebfc0
       rdx 0x00007fff40566f10
       rax 0x0000000000000001
       rcx 0x0000000000010001
       rsp 0x00007fff4056f4b0
       rip 0x0000000001581130
    eflags 0x0000000000013202
    csgsfs 0x0000000000000033
error code 0x0000000000000000
    vector 0x000000000000000d
  old mask 0xffffffde3e3b5a05
       cr2 0x0000000000000000
Nested traceback attempted via signal, from:
Abort: Unknown
        r8 0x000000000000003c
        r9 0x0000000005097a27
       r10 0x00007fff4056de28
       r11 0x0000000000003206
       r12 0x0000000000000001
       r13 0x00007fff4056df80
       r14 0x0000000000000000
       r15 0x0000000000000006
       rdi 0x0000000000000008
       rsi 0x00007fff4056df80
       rbp 0x00007fff4056dfc0
       rbx 0x00007fff29f6b780
       rdx 0x0000000000000010
       rax 0x0000000000000010
       rcx 0xffffffffffffffff
       rsp 0x00007fff4056df50
       rip 0x00007ffffecd43fb
    eflags 0x0000000000003206
    csgsfs 0x1234000000000033
error code 0x0000000000000000
    vector 0x000000000000000d
  old mask 0xffffffde3e3b5a05
       cr2 0x0000000000000000

Cisco Adaptive Security Appliance Software Version 9.3(1)

Compiled on Wed 23-Jul-14 18:16 PDT by builders
Hardware:   ASAv
Crashinfo collected on 03:42:24.059 UTC Tue Nov 28 2017

Traceback:
0: 0x0000000000422118
1: 0x00000000004221f8
2: 0x000000000042226d
3: 0x0000000001587076
4: 0x00007ffffecd55f0
5: 0x00000000015820a0
6: 0x000000000212d482
7: 0x000000000139f304
8: 0x000000000213f315
9: 0x0000000001460873
10: 0x0000000001488625
11: 0x0000000000423e7a
12: 0x00000000004244dc
13: 0x00000000015874a9
14: 0x00007ffffecd55f0
15: 0x0000000000558d85
16: 0x00000000008f5a2b
17: 0x00000000008fd361
18: 0x0000000000428a15
-----------------------------------------------
Process shutdown finished
Rebooting.....

Thanks in advance for your help! :)

Regards,
Abdullah Alhaddad