Re: [Qemu-devel] [pve-devel] QEMU LIve Migration - swap_free: Bad swap file entry

[Stefan Priebe - Profihost AG]

Hi,
Am 07.02.2014 13:21, schrieb Dr. David Alan Gilbert:
> * Stefan Priebe - Profihost AG (address@hidden) wrote:
>> Hi,
>>
>> i was able to reproduce with a longer running test VM running the google
>> stress test.
> 
> Hmm that's quite a fun set of differences; I think I'd like
> to understand whether the pattern is related to the pattern of what
> the test is doing.
> 
> Can you just give an explanation of exactly how you ran that test?
>    What you installed, how exactly you ran it.

While migrating i've still no reliable way to reproduce but i'll try to.

I can force the problem without migration when start with:
bin/stressapptest -s 3600 -m 20 -i 20 -C 20 --force_errors

=> inject false errors to test error handling

Stefan

> Then Marcin and I can try and replicate it.
> 
> Dave
> 
>> And it happens exacly when the migration finishes it does not happen
>> while the migration is running.
>>
>> Google Stress Output displays Memory errors:
>> ------------------------------------------------
>> Page Error: miscompare on CPU 5(0xffff) at 0x7f52431341c0(0x0:DIMM
>> Unknown): read:0x00000040000000bf, reread:0x00000040000000bf
>> expected:0x00000040ffffffbf
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Page Error: miscompare on CPU 5(0xffff) at 0x7f52431341c8(0x0:DIMM
>> Unknown): read:0x00000020000000df, reread:0x00000020000000df
>> expected:0x00000020ffffffdf
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34020(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34028(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34060(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34068(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340a0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340a8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340e0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c340e8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34120(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34128(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34160(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34168(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341a0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341a8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341e0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c341e8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34220(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34228(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34260(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34268(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342a0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342a8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342e0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c342e8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34320(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34328(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34360(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c34368(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c343a0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 571s
>> Hardware Error: miscompare on CPU 3(0xffff) at 0x7f4fd5c343a8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235030(0x0:DIMM
>> Unknown): read:0xfffffffd000000fd, reread:0xfffffffd000000fd
>> expected:0xfffffffdfffffffd
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235070(0x0:DIMM
>> Unknown): read:0xfffffff7000000f7, reread:0xfffffff7000000f7
>> expected:0xfffffff7fffffff7
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2350b0(0x0:DIMM
>> Unknown): read:0xffffffdf000000df, reread:0xffffffdf000000df
>> expected:0xffffffdfffffffdf
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2350f0(0x0:DIMM
>> Unknown): read:0xffffff7f0000007f, reread:0xffffff7f0000007f
>> expected:0xffffff7fffffff7f
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235130(0x0:DIMM
>> Unknown): read:0xfffffdff0000fd00, reread:0xfffffdff0000fd00
>> expected:0xfffffdfffffffdff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235170(0x0:DIMM
>> Unknown): read:0xfffff7ff0000f700, reread:0xfffff7ff0000f700
>> expected:0xfffff7fffffff7ff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2351b0(0x0:DIMM
>> Unknown): read:0xffffdfff0000df00, reread:0xffffdfff0000df00
>> expected:0xffffdfffffffdfff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2351f0(0x0:DIMM
>> Unknown): read:0xffff7fff00007f00, reread:0xffff7fff00007f00
>> expected:0xffff7fffffff7fff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235230(0x0:DIMM
>> Unknown): read:0xfffdffff00fd0000, reread:0xfffdffff00fd0000
>> expected:0xfffdfffffffdffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235270(0x0:DIMM
>> Unknown): read:0xfff7ffff00f70000, reread:0xfff7ffff00f70000
>> expected:0xfff7fffffff7ffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2352b0(0x0:DIMM
>> Unknown): read:0xffdfffff00df0000, reread:0xffdfffff00df0000
>> expected:0xffdfffffffdfffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2352f0(0x0:DIMM
>> Unknown): read:0xff7fffff007f0000, reread:0xff7fffff007f0000
>> expected:0xff7fffffff7fffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235330(0x0:DIMM
>> Unknown): read:0xfdfffffffd000000, reread:0xfdfffffffd000000
>> expected:0xfdfffffffdffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235370(0x0:DIMM
>> Unknown): read:0xf7fffffff7000000, reread:0xf7fffffff7000000
>> expected:0xf7fffffff7ffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2353b0(0x0:DIMM
>> Unknown): read:0xdfffffffdf000000, reread:0xdfffffffdf000000
>> expected:0xdfffffffdfffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2353f0(0x0:DIMM
>> Unknown): read:0x7fffffff7f000000, reread:0x7fffffff7f000000
>> expected:0x7fffffff7fffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235430(0x0:DIMM
>> Unknown): read:0xdfffffffdf000000, reread:0xdfffffffdf000000
>> expected:0xdfffffffdfffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235470(0x0:DIMM
>> Unknown): read:0xf7fffffff7000000, reread:0xf7fffffff7000000
>> expected:0xf7fffffff7ffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2354b0(0x0:DIMM
>> Unknown): read:0xfdfffffffd000000, reread:0xfdfffffffd000000
>> expected:0xfdfffffffdffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2354f0(0x0:DIMM
>> Unknown): read:0xff7fffff007f0000, reread:0xff7fffff007f0000
>> expected:0xff7fffffff7fffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235530(0x0:DIMM
>> Unknown): read:0xffdfffff00df0000, reread:0xffdfffff00df0000
>> expected:0xffdfffffffdfffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235570(0x0:DIMM
>> Unknown): read:0xfff7ffff00f70000, reread:0xfff7ffff00f70000
>> expected:0xfff7fffffff7ffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2355b0(0x0:DIMM
>> Unknown): read:0xfffdffff00fd0000, reread:0xfffdffff00fd0000
>> expected:0xfffdfffffffdffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2355f0(0x0:DIMM
>> Unknown): read:0xffff7fff00007f00, reread:0xffff7fff00007f00
>> expected:0xffff7fffffff7fff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235630(0x0:DIMM
>> Unknown): read:0xffffdfff0000df00, reread:0xffffdfff0000df00
>> expected:0xffffdfffffffdfff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235670(0x0:DIMM
>> Unknown): read:0xfffff7ff0000f700, reread:0xfffff7ff0000f700
>> expected:0xfffff7fffffff7ff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2356b0(0x0:DIMM
>> Unknown): read:0xfffffdff0000fd00, reread:0xfffffdff0000fd00
>> expected:0xfffffdfffffffdff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c2356f0(0x0:DIMM
>> Unknown): read:0xffffff7f0000007f, reread:0xffffff7f0000007f
>> expected:0xffffff7fffffff7f
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235730(0x0:DIMM
>> Unknown): read:0xffffffdf000000df, reread:0xffffffdf000000df
>> expected:0xffffffdfffffffdf
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 2(0xffff) at 0x7f535c235770(0x0:DIMM
>> Unknown): read:0xfffffff7000000f7, reread:0xfffffff7000000f7
>> expected:0xfffffff7fffffff7
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834020(0x0:DIMM
>> Unknown): read:0x00000010000000ef, reread:0x00000010000000ef
>> expected:0x00000010ffffffef
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834028(0x0:DIMM
>> Unknown): read:0x00000020000000df, reread:0x00000020000000df
>> expected:0x00000020ffffffdf
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834030(0x0:DIMM
>> Unknown): read:0x00000040000000bf, reread:0x00000040000000bf
>> expected:0x00000040ffffffbf
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834038(0x0:DIMM
>> Unknown): read:0x000000800000007f, reread:0x000000800000007f
>> expected:0x00000080ffffff7f
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834060(0x0:DIMM
>> Unknown): read:0x000010000000efff, reread:0x000010000000efff
>> expected:0x00001000ffffefff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834068(0x0:DIMM
>> Unknown): read:0x000020000000dfff, reread:0x000020000000dfff
>> expected:0x00002000ffffdfff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834070(0x0:DIMM
>> Unknown): read:0x000040000000bfff, reread:0x000040000000bfff
>> expected:0x00004000ffffbfff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834078(0x0:DIMM
>> Unknown): read:0x0000800000007fff, reread:0x0000800000007fff
>> expected:0x00008000ffff7fff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340a0(0x0:DIMM
>> Unknown): read:0x0010000000ef00ff, reread:0x0010000000ef00ff
>> expected:0x00100000ffefffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340a8(0x0:DIMM
>> Unknown): read:0x0020000000df00ff, reread:0x0020000000df00ff
>> expected:0x00200000ffdfffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340b0(0x0:DIMM
>> Unknown): read:0x0040000000bf00ff, reread:0x0040000000bf00ff
>> expected:0x00400000ffbfffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340b8(0x0:DIMM
>> Unknown): read:0x00800000007f00ff, reread:0x00800000007f00ff
>> expected:0x00800000ff7fffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340e0(0x0:DIMM
>> Unknown): read:0x10000000ef0000ff, reread:0x10000000ef0000ff
>> expected:0x10000000efffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340e8(0x0:DIMM
>> Unknown): read:0x20000000df0000ff, reread:0x20000000df0000ff
>> expected:0x20000000dfffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340f0(0x0:DIMM
>> Unknown): read:0x40000000bf0000ff, reread:0x40000000bf0000ff
>> expected:0x40000000bfffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848340f8(0x0:DIMM
>> Unknown): read:0x800000007f0000ff, reread:0x800000007f0000ff
>> expected:0x800000007fffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834120(0x0:DIMM
>> Unknown): read:0x04000000fb00ff00, reread:0x04000000fb00ff00
>> expected:0x04000000fbffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834128(0x0:DIMM
>> Unknown): read:0x02000000fd00ff00, reread:0x02000000fd00ff00
>> expected:0x02000000fdffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834130(0x0:DIMM
>> Unknown): read:0x01000000fe00ff00, reread:0x01000000fe00ff00
>> expected:0x01000000feffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834138(0x0:DIMM
>> Unknown): read:0x00800000007fff00, reread:0x00800000007fff00
>> expected:0x00800000ff7fffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834160(0x0:DIMM
>> Unknown): read:0x0004000000fbff00, reread:0x0004000000fbff00
>> expected:0x00040000fffbffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834168(0x0:DIMM
>> Unknown): read:0x0002000000fdff00, reread:0x0002000000fdff00
>> expected:0x00020000fffdffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834170(0x0:DIMM
>> Unknown): read:0x0001000000feff00, reread:0x0001000000feff00
>> expected:0x00010000fffeffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f5184834178(0x0:DIMM
>> Unknown): read:0x0000800000007f00, reread:0x0000800000007f00
>> expected:0x00008000ffff7fff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341a0(0x0:DIMM
>> Unknown): read:0x000004000000fb00, reread:0x000004000000fb00
>> expected:0x00000400fffffbff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341a8(0x0:DIMM
>> Unknown): read:0x000002000000fd00, reread:0x000002000000fd00
>> expected:0x00000200fffffdff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341b0(0x0:DIMM
>> Unknown): read:0x000001000000fe00, reread:0x000001000000fe00
>> expected:0x00000100fffffeff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341b8(0x0:DIMM
>> Unknown): read:0x000000800000ff7f, reread:0x000000800000ff7f
>> expected:0x00000080ffffff7f
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341e0(0x0:DIMM
>> Unknown): read:0x000000040000fffb, reread:0x000000040000fffb
>> expected:0x00000004fffffffb
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 15(0xffff) at 0x7f51848341e8(0x0:DIMM
>> Unknown): read:0x000000020000fffd, reread:0x000000020000fffd
>> expected:0x00000002fffffffd
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00008(0x0:DIMM
>> Unknown): read:0xfffffff7000000fb, reread:0xfffffff7000000fb
>> expected:0xfffffff7fffffffb
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00018(0x0:DIMM
>> Unknown): read:0xffffff7f000000bf, reread:0xffffff7f000000bf
>> expected:0xffffff7fffffffbf
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00028(0x0:DIMM
>> Unknown): read:0xfffff7ff0000fb00, reread:0xfffff7ff0000fb00
>> expected:0xfffff7fffffffbff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00038(0x0:DIMM
>> Unknown): read:0xffff7fff0000bf00, reread:0xffff7fff0000bf00
>> expected:0xffff7fffffffbfff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00048(0x0:DIMM
>> Unknown): read:0xfff7ffff00fb0000, reread:0xfff7ffff00fb0000
>> expected:0xfff7fffffffbffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00058(0x0:DIMM
>> Unknown): read:0xff7fffff00bf0000, reread:0xff7fffff00bf0000
>> expected:0xff7fffffffbfffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00068(0x0:DIMM
>> Unknown): read:0xf7fffffffb000000, reread:0xf7fffffffb000000
>> expected:0xf7fffffffbffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00078(0x0:DIMM
>> Unknown): read:0x7fffffffbf000000, reread:0x7fffffffbf000000
>> expected:0x7fffffffbfffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00088(0x0:DIMM
>> Unknown): read:0xf7ffffffef000000, reread:0xf7ffffffef000000
>> expected:0xf7ffffffefffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00098(0x0:DIMM
>> Unknown): read:0xff7ffffffe000000, reread:0xff7ffffffe000000
>> expected:0xff7ffffffeffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000a8(0x0:DIMM
>> Unknown): read:0xfff7ffff00ef0000, reread:0xfff7ffff00ef0000
>> expected:0xfff7ffffffefffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000b8(0x0:DIMM
>> Unknown): read:0xffff7fff00fe0000, reread:0xffff7fff00fe0000
>> expected:0xffff7ffffffeffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000c8(0x0:DIMM
>> Unknown): read:0xfffff7ff0000ef00, reread:0xfffff7ff0000ef00
>> expected:0xfffff7ffffffefff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000d8(0x0:DIMM
>> Unknown): read:0xffffff7f0000fe00, reread:0xffffff7f0000fe00
>> expected:0xffffff7ffffffeff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000e8(0x0:DIMM
>> Unknown): read:0xfffffff7000000ef, reread:0xfffffff7000000ef
>> expected:0xfffffff7ffffffef
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c000f8(0x0:DIMM
>> Unknown): read:0xffffffff000000fe, reread:0xffffffff000000fe
>> expected:0xfffffffffffffffe
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00108(0x0:DIMM
>> Unknown): read:0xfffffff7000000fb, reread:0xfffffff7000000fb
>> expected:0xfffffff7fffffffb
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00118(0x0:DIMM
>> Unknown): read:0xffffff7f000000bf, reread:0xffffff7f000000bf
>> expected:0xffffff7fffffffbf
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00128(0x0:DIMM
>> Unknown): read:0xfffff7ff0000fb00, reread:0xfffff7ff0000fb00
>> expected:0xfffff7fffffffbff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00138(0x0:DIMM
>> Unknown): read:0xffff7fff0000bf00, reread:0xffff7fff0000bf00
>> expected:0xffff7fffffffbfff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00148(0x0:DIMM
>> Unknown): read:0xfff7ffff00fb0000, reread:0xfff7ffff00fb0000
>> expected:0xfff7fffffffbffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00158(0x0:DIMM
>> Unknown): read:0xff7fffff00bf0000, reread:0xff7fffff00bf0000
>> expected:0xff7fffffffbfffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00168(0x0:DIMM
>> Unknown): read:0xf7fffffffb000000, reread:0xf7fffffffb000000
>> expected:0xf7fffffffbffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00178(0x0:DIMM
>> Unknown): read:0x7fffffffbf000000, reread:0x7fffffffbf000000
>> expected:0x7fffffffbfffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00188(0x0:DIMM
>> Unknown): read:0xf7ffffffef000000, reread:0xf7ffffffef000000
>> expected:0xf7ffffffefffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c00198(0x0:DIMM
>> Unknown): read:0xff7ffffffe000000, reread:0xff7ffffffe000000
>> expected:0xff7ffffffeffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001a8(0x0:DIMM
>> Unknown): read:0xfff7ffff00ef0000, reread:0xfff7ffff00ef0000
>> expected:0xfff7ffffffefffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001b8(0x0:DIMM
>> Unknown): read:0xffff7fff00fe0000, reread:0xffff7fff00fe0000
>> expected:0xffff7ffffffeffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001c8(0x0:DIMM
>> Unknown): read:0xfffff7ff0000ef00, reread:0xfffff7ff0000ef00
>> expected:0xfffff7ffffffefff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 4(0xffff) at 0x7f50d1c001d8(0x0:DIMM
>> Unknown): read:0xffffff7f0000fe00, reread:0xffffff7f0000fe00
>> expected:0xffffff7ffffffeff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200010(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200018(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200050(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200058(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200090(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200098(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72000d0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72000d8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200110(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200118(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200150(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200158(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200190(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200198(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72001d0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72001d8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200210(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200218(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200250(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200258(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200290(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200298(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72002d0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df72002d8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200310(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200318(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200350(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200358(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200390(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Hardware Error: miscompare on CPU 8(0xffff) at 0x7f4df7200398(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00000(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00008(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00010(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00018(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00040(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00048(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00050(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00058(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00080(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00088(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00090(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00098(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000c0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000c8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000d0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa000d8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00100(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00108(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00110(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00118(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00140(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00148(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00150(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00158(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00180(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00188(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00190(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa00198(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa001c0(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Report Error: miscompare : DIMM Unknown : 1 : 572s
>> Page Error: miscompare on CPU 7(0xffff) at 0x7f4d8aa001c8(0x0:DIMM
>> Unknown): read:0x0000000000000000, reread:0x0000000000000000
>> expected:0xffffffffffffffff
>> Log: Thread 61 found 85568 hardware incidents
>> Log: Thread 62 found 169344 hardware incidents
>> Log: Thread 63 found 44544 hardware incidents
>> Log: Thread 64 found 149504 hardware incidents
>> Log: Thread 65 found 131968 hardware incidents
>> Log: Thread 66 found 150528 hardware incidents
>> Log: Thread 67 found 144384 hardware incidents
>> Log: Thread 68 found 149888 hardware incidents
>> Stats: Found 1025728 hardware incidents
>> Stats: Completed: 9176812.00M in 524.63s 17491.92MB/s, with 1025728
>> hardware incidents, 0 errors
>> Stats: Memory Copy: 4890244.00M at 9402.74MB/s
>> Stats: File Copy: 0.00M at 0.00MB/s
>> Stats: Net Copy: 0.00M at 0.00MB/s
>> Stats: Data Check: 0.00M at 0.00MB/s
>> Stats: Invert Data: 4286568.00M at 8242.44MB/s
>> Stats: Disk: 0.00M at 0.00MB/s
>>
>> Status: FAIL - test discovered HW problems
>> -----------------------------------------------------------------------------------
>>
>> Stefan
>> Am 07.02.2014 10:37, schrieb Stefan Priebe - Profihost AG:
>>>
>>> Am 07.02.2014 10:31, schrieb Dr. David Alan Gilbert:
>>>> * Stefan Priebe - Profihost AG (address@hidden) wrote:
>>>>>
>>>>> Am 07.02.2014 10:15, schrieb Dr. David Alan Gilbert:
>>>>>> * Stefan Priebe - Profihost AG (address@hidden) wrote:
>>>>>>> Am 07.02.2014 09:15, schrieb Alexandre DERUMIER:
>>>>>>>>
>>>>>>>> do you use xbzrle for live migration ?
>>>>>>>
>>>>>>> no - i'm really stucked right now with this. Biggest problem i can't
>>>>>>> reproduce with test machines ;-(
>>>>>>
>>>>>> Only being able to test on your production VMs isn't fun;
>>>>>> is it possible or you to run an extra program on these VMs - e.g.
>>>>>> if we came up with a simple (userland) memory test?
>>>>>
>>>>> You mean to reproduce?
>>>>
>>>> I'm more interested in seeing what type of corruption is happening;
>>>> if you've got a test VM that corrupts memory and we can run a program
>>>> in that vm that writes a known pattern into memory and checks it
>>>> then see what changed after migration, it might give a clue.
>>>>
>>>> But obviously this would only be of any use if run on the VM that actually
>>>> fails.
>>>
>>> Right that makes sense - sadly i still don't know how to reproduce? Any
>>> app ideas i can try?
>>>
>>>
>>>>> I already tried https://code.google.com/p/stressapptest/ while migrating
>>>>> on a test VM but this works fine.
>>>>>
>>>>> I also tried running mysql bench while migrating on a test vm and this
>>>>> works too ;-(
>>>>
>>>>
>>>> Dave
>>>> --
>>>> Dr. David Alan Gilbert / address@hidden / Manchester, UK
>>>>
> --
> Dr. David Alan Gilbert / address@hidden / Manchester, UK
>