[Qemu-devel] Steal time MSR not set properly during live migration?

[Apollon Oikonomopoulos]

Hi,

I'm trying to debug an issue we're having with some debian.org machines 
running in QEMU 2.1.2 instances (see [1] for more background). In short, 
after a live migration guests running Debian Jessie (linux 3.16) stop 
accounting CPU time properly. /proc/stat in the guest shows no increase 
in user and system time anymore (regardless of workload) and what stands 
out are extremely large values for steal time:

 % cat /proc/stat
 cpu  2400 0 1842 650879168 2579640 0 25 136562317270 0 0
 cpu0 1366 0 1028 161392988 1238598 0 11 383803090749 0 0
 cpu1 294 0 240 162582008 639105 0 8 39686436048 0 0
 cpu2 406 0 338 163331066 383867 0 4 333994238765 0 0
 cpu3 332 0 235 163573105 318069 0 1 1223752959076 0 0
 intr 355773871 33 10 0 0 0 0 3 0 1 0 0 36 144 0 0 1638612 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 5001741 41 0 8516993 0 3669582 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 ctxt 837862829
 btime 1431642967
 processes 8529939
 procs_running 1
 procs_blocked 0
 softirq 225193331 2 77532878 172 7250024 819289 0 54 33739135 176552 105675225
 
Reading the memory pointed to by the steal time MSRs pre- and 
post-migration, I can see that post-migration the high bytes are set to 
0xff:

(qemu) xp /8b 0x1fc0cfc0
000000001fc0cfc0: 0x94 0x57 0x77 0xf5 0xff 0xff 0xff 0xff

The "jump" in steal time happens when the guest is resumed on the 
receiving side.

I've also been able to consistently reproduce this on a Ganeti cluster 
at work, using QEMU 2.1.3 and kernels 3.16 and 4.0 in the guests. The 
issue goes away if I disable the steal time MSR using `-cpu 
qemu64,-kvm_steal_time`.

So, it looks to me as if the steal time MSR is not set/copied properly 
during live migration, although AFAICT this should be the case after 
917367aa968fd4fef29d340e0c7ec8c608dffaab.

Any ideas?

Regards,
Apollon

[1] https://bugs.debian.org/785557