qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Qemu-devel] Nested KVM L2 guest hangs


From: Ariel Zeitlin
Subject: [Qemu-devel] Nested KVM L2 guest hangs
Date: Fri, 9 Jan 2015 16:08:53 +0200

Hi, all
I am trying to use nested virtualization setup and experiencing L2 guest (Windows) hanging during boot. 
I have seen some threads about it, but did not manage to make sense of how can I work around it.

Would appreciate any guidance for a solution or workaround.

Thanks,
Ariel

Some details are attached.

L0 logs look like this:
[117402.803802] kvm [11772]: vcpu0 unhandled rdmsr: 0x345
[117402.823629] kvm_set_msr_common: 54 callbacks suppressed
[117402.823631] kvm [11772]: vcpu0 unhandled wrmsr: 0x680 data 0
[117402.833796] kvm [11772]: vcpu0 unhandled wrmsr: 0x6c0 data 0
[117402.843967] kvm [11772]: vcpu0 unhandled wrmsr: 0x681 data 0
[117402.854106] kvm [11772]: vcpu0 unhandled wrmsr: 0x6c1 data 0
[117402.864272] kvm [11772]: vcpu0 unhandled wrmsr: 0x682 data 0
[117402.874429] kvm [11772]: vcpu0 unhandled wrmsr: 0x6c2 data 0
[117402.884613] kvm [11772]: vcpu0 unhandled wrmsr: 0x683 data 0
[117402.893675] kvm [11772]: vcpu0 unhandled wrmsr: 0x6c3 data 0
[117402.902550] kvm [11772]: vcpu0 unhandled wrmsr: 0x684 data 0
[117402.911442] kvm [11772]: vcpu0 unhandled wrmsr: 0x6c4 data 0

At the time of the hanging this line is added:
nested_vmx_run: VMCS MSR_{LOAD,STORE} unsupported

L1 logs look like this:
Jan  9 11:34:43 hpvm---2 kernel: [  217.975781] kvm [2106]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff


The setup details are as follows:
------------------------------------------------
L0 - Ubuntu 14.04, kernel 3.13.0-43:

CPUID
Vendor ID: "GenuineIntel"; CPUID level 13

Intel-specific functions:
Version 000206d7:
Type 0 - Original OEM
Family 6 - Pentium Pro
Model 13 -
Stepping 7
Reserved 8

Extended brand string: "       Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz"
CLFLUSH instruction cache line size: 8
Initial APIC ID: 39
Hyper threading siblings: 32

Feature flags bfebfbff:
FPU    Floating Point Unit
VME    Virtual 8086 Mode Enhancements
DE     Debugging Extensions
PSE    Page Size Extensions
TSC    Time Stamp Counter
MSR    Model Specific Registers
PAE    Physical Address Extension
MCE    Machine Check Exception
CX8    COMPXCHG8B Instruction
APIC   On-chip Advanced Programmable Interrupt Controller present and enabled
SEP    Fast System Call
MTRR   Memory Type Range Registers
PGE    PTE Global Flag
MCA    Machine Check Architecture
CMOV   Conditional Move and Compare Instructions
FGPAT  Page Attribute Table
PSE-36 36-bit Page Size Extension
CLFSH  CFLUSH instruction
DS     Debug store
ACPI   Thermal Monitor and Clock Ctrl
MMX    MMX instruction set
FXSR   Fast FP/MMX Streaming SIMD Extensions save/restore
SSE    Streaming SIMD Extensions instruction set
SSE2   SSE2 extensions
SS     Self Snoop
HT     Hyper Threading
TM     Thermal monitor
31     reserved

TLB and cache info:
5a: unknown TLB/cache descriptor
03: Data TLB: 4KB pages, 4-way set assoc, 64 entries
76: unknown TLB/cache descriptor
ff: unknown TLB/cache descriptor
b2: unknown TLB/cache descriptor
f0: unknown TLB/cache descriptor
ca: unknown TLB/cache descriptor
Processor serial: 0002-06D7-0000-0000-0000-0000

L1 - Ubuntu 14.04, kernel 3.13.0-43

CPUID
CPU 0:
   vendor_id = "GenuineIntel"
   version information (1/eax):
      processor type  = primary processor (0)
      family          = Intel Pentium Pro/II/III/Celeron/Core/Core 2/Atom, AMD Athlon/Duron, Cyrix M2, VIA C3 (6)
      model           = 0xa (10)
      stepping id     = 0x1 (1)
      extended family = 0x0 (0)
      extended model  = 0x2 (2)
      (simple synth)  = Intel Core i5-2000 / Core i7-2000 / Mobile Core i3-2000 / Mobile Core i5-2000 / Mobile Core i7-2000 / Pentium G500/G600/G800 / Celeron G400/G500/700/800/B800 / Xeon E3-1200 (Sandy Bridge), 32nm
   miscellaneous (1/ebx):
      process local APIC physical ID = 0x0 (0)
      cpu count                      = 0x0 (0)
      CLFLUSH line size              = 0x8 (8)
      brand index                    = 0x0 (0)
   brand id = 0x00 (0): unknown
   feature information (1/edx):
      x87 FPU on chip                        = true
      virtual-8086 mode enhancement          = true
      debugging extensions                   = true
      page size extensions                   = true
      time stamp counter                     = true
      RDMSR and WRMSR support                = true
      physical address extensions            = true
      machine check exception                = true
      CMPXCHG8B inst.                        = true
      APIC on chip                           = true
      SYSENTER and SYSEXIT                   = true
      memory type range registers            = true
      PTE global bit                         = true
      machine check architecture             = true
      conditional move/compare instruction   = true
      page attribute table                   = true
      page size extension                    = true
      processor serial number                = false
      CLFLUSH instruction                    = true
      debug store                            = false
      thermal monitor and clock ctrl         = false
      MMX Technology                         = true
      FXSAVE/FXRSTOR                         = true
      SSE extensions                         = true
      SSE2 extensions                        = true
      self snoop                             = true
      hyper-threading / multi-core supported = false
      therm. monitor                         = false
      IA64                                   = false
      pending break event                    = false
   feature information (1/ecx):
      PNI/SSE3: Prescott New Instructions     = true
      PCLMULDQ instruction                    = true
      64-bit debug store                      = false
      MONITOR/MWAIT                           = false
      CPL-qualified debug store               = false
      VMX: virtual machine extensions         = true
      SMX: safer mode extensions              = false
      Enhanced Intel SpeedStep Technology     = false
      thermal monitor 2                       = false
      SSSE3 extensions                        = true
      context ID: adaptive or shared L1 data  = false
      FMA instruction                         = false
      CMPXCHG16B instruction                  = true
      xTPR disable                            = false
      perfmon and debug                       = false
      process context identifiers             = true
      direct cache access                     = false
      SSE4.1 extensions                       = true
      SSE4.2 extensions                       = true
      extended xAPIC support                  = true
      MOVBE instruction                       = false
      POPCNT instruction                      = true
      time stamp counter deadline             = true
      AES instruction                         = true
      XSAVE/XSTOR states                      = true
      OS-enabled XSAVE/XSTOR                  = true
      AVX: advanced vector extensions         = true
      F16C half-precision convert instruction = false
      RDRAND instruction                      = false
      hypervisor guest status                 = true
   cache and TLB information (2):
      0x7d: L2 cache: 2M, 8-way, sectored, 64 byte lines
      0x30: L1 cache: 32K, 8-way, 64 byte lines
      0x2c: L1 data cache: 32K, 8-way, 64 byte lines
   processor serial number: 0002-06A1-0000-0000-0000-0000
   deterministic cache parameters (4):
      --- cache 0 ---
      cache type                           = data cache (1)
      cache level                          = 0x1 (1)
      self-initializing cache level        = true
      fully associative cache              = false
      extra threads sharing this cache     = 0x0 (0)
      extra processor cores on this die    = 0x0 (0)
      system coherency line size           = 0x3f (63)
      physical line partitions             = 0x0 (0)
      ways of associativity                = 0x7 (7)
      WBINVD/INVD behavior on lower caches = true
      inclusive to lower caches            = false
      complex cache indexing               = false
      number of sets - 1 (s)               = 63
      --- cache 1 ---
      cache type                           = instruction cache (2)
      cache level                          = 0x1 (1)
      self-initializing cache level        = true
      fully associative cache              = false
      extra threads sharing this cache     = 0x0 (0)
      extra processor cores on this die    = 0x0 (0)
      system coherency line size           = 0x3f (63)
      physical line partitions             = 0x0 (0)
      ways of associativity                = 0x7 (7)
      WBINVD/INVD behavior on lower caches = true
      inclusive to lower caches            = false
      complex cache indexing               = false
      number of sets - 1 (s)               = 63
      --- cache 2 ---
      cache type                           = unified cache (3)
      cache level                          = 0x2 (2)
      self-initializing cache level        = true
      fully associative cache              = false
      extra threads sharing this cache     = 0x0 (0)
      extra processor cores on this die    = 0x0 (0)
      system coherency line size           = 0x3f (63)
      physical line partitions             = 0x0 (0)
      ways of associativity                = 0xf (15)
      WBINVD/INVD behavior on lower caches = true
      inclusive to lower caches            = false
      complex cache indexing               = false
      number of sets - 1 (s)               = 4095
   MONITOR/MWAIT (5):
      smallest monitor-line size (bytes)       = 0x0 (0)
      largest monitor-line size (bytes)        = 0x0 (0)
      enum of Monitor-MWAIT exts supported     = true
      supports intrs as break-event for MWAIT  = true
      number of C0 sub C-states using MWAIT    = 0x0 (0)
      number of C1 sub C-states using MWAIT    = 0x0 (0)
      number of C2 sub C-states using MWAIT    = 0x0 (0)
      number of C3 sub C-states using MWAIT    = 0x0 (0)
      number of C4 sub C-states using MWAIT    = 0x0 (0)
      number of C5 sub C-states using MWAIT    = 0x0 (0)
      number of C6 sub C-states using MWAIT    = 0x0 (0)
      number of C7 sub C-states using MWAIT    = 0x0 (0)
   Thermal and Power Management Features (6):
      digital thermometer                     = false
      Intel Turbo Boost Technology            = false
      ARAT always running APIC timer          = false
      PLN power limit notification            = false
      ECMD extended clock modulation duty     = false
      PTM package thermal management          = false
      digital thermometer thresholds          = 0x0 (0)
      ACNT/MCNT supported performance measure = false
      ACNT2 available                         = false
      performance-energy bias capability      = false
   extended feature flags (7):
      FSGSBASE instructions                    = false
      IA32_TSC_ADJUST MSR supported            = false
      BMI instruction                          = false
      HLE hardware lock elision                = false
      AVX2: advanced vector extensions 2       = false
      SMEP supervisor mode exec protection     = false
      BMI2 instructions                        = false
      enhanced REP MOVSB/STOSB                 = false
      INVPCID instruction                      = false
      RTM: restricted transactional memory     = false
      QM: quality of service monitoring        = false
      deprecated FPU CS/DS                     = false
      intel memory protection extensions       = false
      AVX512F: AVX-512 foundation instructions = false
      RDSEED instruction                       = false
      ADX instructions                         = false
      SMAP: supervisor mode access prevention  = false
      Intel processor trace                    = false
      AVX512PF: prefetch instructions          = false
      AVX512ER: exponent & reciprocal instrs   = false
      AVX512CD: conflict detection instrs      = false
      SHA instructions                         = false
      PREFETCHWT1                              = false
   Direct Cache Access Parameters (9):
      PLATFORM_DCA_CAP MSR bits = 0
   Architecture Performance Monitoring Features (0xa/eax):
      version ID                               = 0x2 (2)
      number of counters per logical processor = 0x4 (4)
      bit width of counter                     = 0x30 (48)
      length of EBX bit vector                 = 0x7 (7)
   Architecture Performance Monitoring Features (0xa/ebx):
      core cycle event not available           = false
      instruction retired event not available  = false
      reference cycles event not available     = false
      last-level cache ref event not available = false
      last-level cache miss event not avail    = false
      branch inst retired event not available  = false
      branch mispred retired event not avail   = false
   Architecture Performance Monitoring Features (0xa/edx):
      number of fixed counters    = 0x3 (3)
      bit width of fixed counters = 0x30 (48)
   XSAVE features (0xd/0):
      XCR0 lower 32 bits valid bit field mask = 0x00000007
      bytes required by fields in XCR0        = 0x00000340 (832)
      bytes required by XSAVE/XRSTOR area     = 0x00000340 (832)
      XCR0 upper 32 bits valid bit field mask = 0x00000000
   YMM features (0xd/2):
      YMM save state byte size                = 0x00000100 (256)
      YMM save state byte offset              = 0x00000240 (576)
   LWP features (0xd/0x3e):
      LWP save state byte size                = 0x00000000 (0)
      LWP save state byte offset              = 0x00000000 (0)
   hypervisor_id = "KVMKVMKVM   "
   hypervisor features (0x40000001/eax):
      kvmclock available at MSR 0x11          = true
      delays unnecessary for PIO ops          = true
      mmu_op                                  = false
      kvmclock available a MSR 0x4b564d00     = true
      async pf enable available by MSR        = true
      steal clock supported                   = true
      guest EOI optimization enabled          = true
      stable: no guest per-cpu warps expected = true
   extended feature flags (0x80000001/edx):
      SYSCALL and SYSRET instructions        = true
      execution disable                      = true
      1-GB large page support                = true
      RDTSCP                                 = true
      64-bit extensions technology available = true
   Intel feature flags (0x80000001/ecx):
      LAHF/SAHF supported in 64-bit mode     = true
      LZCNT advanced bit manipulation        = false
      3DNow! PREFETCH/PREFETCHW instructions = false
   brand = "Intel Xeon E312xx (Sandy Bridge)"
   L1 TLB/cache information: 2M/4M pages & L1 TLB (0x80000005/eax):
      instruction # entries     = 0xff (255)
      instruction associativity = 0x1 (1)
      data # entries            = 0xff (255)
      data associativity        = 0x1 (1)
   L1 TLB/cache information: 4K pages & L1 TLB (0x80000005/ebx):
      instruction # entries     = 0xff (255)
      instruction associativity = 0x1 (1)
      data # entries            = 0xff (255)
      data associativity        = 0x1 (1)
   L1 data cache information (0x80000005/ecx):
      line size (bytes) = 0x40 (64)
      lines per tag     = 0x1 (1)
      associativity     = 0x2 (2)
      size (Kb)         = 0x40 (64)
   L1 instruction cache information (0x80000005/edx):
      line size (bytes) = 0x40 (64)
      lines per tag     = 0x1 (1)
      associativity     = 0x2 (2)
      size (Kb)         = 0x40 (64)
   L2 TLB/cache information: 2M/4M pages & L2 TLB (0x80000006/eax):
      instruction # entries     = 0x0 (0)
      instruction associativity = L2 off (0)
      data # entries            = 0x0 (0)
      data associativity        = L2 off (0)
   L2 TLB/cache information: 4K pages & L2 TLB (0x80000006/ebx):
      instruction # entries     = 0x200 (512)
      instruction associativity = 4-way (4)
      data # entries            = 0x200 (512)
      data associativity        = 4-way (4)
   L2 unified cache information (0x80000006/ecx):
      line size (bytes) = 0x40 (64)
      lines per tag     = 0x1 (1)
      associativity     = 16-way (8)
      size (Kb)         = 0x200 (512)
   L3 cache information (0x80000006/edx):
      line size (bytes)     = 0x0 (0)
      lines per tag         = 0x0 (0)
      associativity         = L2 off (0)
      size (in 512Kb units) = 0x0 (0)
   Advanced Power Management Features (0x80000007/edx):
      temperature sensing diode      = false
      frequency ID (FID) control     = false
      voltage ID (VID) control       = false
      thermal trip (TTP)             = false
      thermal monitor (TM)           = false
      software thermal control (STC) = false
      100 MHz multiplier control     = false
      hardware P-State control       = false
      TscInvariant                   = false
   Physical Address and Linear Address Size (0x80000008/eax):
      maximum physical address bits         = 0x28 (40)
      maximum linear (virtual) address bits = 0x30 (48)
      maximum guest physical address bits   = 0x0 (0)
   Logical CPU cores (0x80000008/ecx):
      number of CPU cores - 1 = 0x0 (0)
      ApicIdCoreIdSize        = 0x0 (0)
   SVM Secure Virtual Machine (0x8000000a/eax):
      SvmRev: SVM revision = 0x0 (0)
   SVM Secure Virtual Machine (0x8000000a/edx):
      nested paging                 = false
      LBR virtualization            = false
      SVM lock                      = false
      NRIP save                     = false
      MSR based TSC rate control    = false
      VMCB clean bits support       = false
      flush by ASID                 = false
      decode assists                = false
      SSSE3/SSE5 opcode set disable = false
      pause intercept filter        = false
      pause filter threshold        = false
   NASID: number of address space identifiers = 0x0 (0):
   (multi-processing synth): none
   (multi-processing method): Intel leaf 1/4
   (APIC widths synth): CORE_width=0 SMT_width=16
   (APIC synth): PKG_ID=8 CORE_ID=8 SMT_ID=0
   (synth) = Intel Xeon E3-1200 (Sandy Bridge), 32nm

L2 - Windows 2003 (but experience same hanging, although not in boot time, in L2 - Linux guest)


The execution command for L1 is:
qemu-system-x86_64 -machine accel=kvm:tcg -name instance-00000001 -S -machine pc-i440fx-1.5,accel=kvm,usb=off -cpu SandyBridge,+pdpe1gb,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 16384 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid 897abedb-a2c2-4140-b994-1b4ee303a54b -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2013.2.3,serial=44454c4c-4800-1030-8057-b3c04f395931,uuid=897abedb-a2c2-4140-b994-1b4ee303a54b -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000001.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/897abedb-a2c2-4140-b994-1b4ee303a54b/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:30:0e:85,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/897abedb-a2c2-4140-b994-1b4ee303a54b/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5


The execution command for L2 is:
/usr/bin/kvm -name SP-Containers -S -machine pc-i440fx-2.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 431ce7cc-173e-3bd0-988d-94f504b7ca4c -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/SP-Containers.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/hdd/linux_dsm.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:33:50:56,bus=pci.0,addr=0x6 -netdev tap,fd=23,id=hostnet1 -device rtl8139,netdev=hostnet1,id=net1,mac=52:54:00:35:34:50,bus=pci.0,addr=0x7 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x3 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on

reply via email to

[Prev in Thread] Current Thread [Next in Thread]