Re: [Qemu-devel] xbzrle migration cache size advise for high memory changes workload ?

[Orit Wasserman]

On 09/03/2012 04:10 PM, Alexandre DERUMIER wrote:
> Hi,
> I'm trying to implement xbzrle live migration,
> 
> But i'm having non finishing migration with high memory changes in guest.
> (simply playing a youtube video in the guest). 
> At the end of the migration, the remaining memory to transfert goes up and 
> down.
XBZRLE can't always help migration to converge, for some workload you need to 
increase migration speed and
migration downtime. What values are you using for those ? 
> 
> I have tried to dynamicaly add memory to cache (max = guest memory), but it 
> doesn't help.
> 
> 
> What is the best size recommendation for cache ? (percent of guest memory ?)
It depends on the guest workload , XBZRLE is good for guest that dirties the 
same memory pages over and
over. so if you notice that the number of cache misses is growing increasing 
the cache size can help.
> Can I tune it based on some values ? (cachemiss, overflow ?)
if you have lots of overflow this means the pages are not suitable for 
compression so increasing the cache size
won't help.
> 
> Can we desactivate xbzrle during the migration ?
not at the moment , I plan to add support to it.

Cheers,
Orit
> 
> 
> 
> 
> Some logs counters of the migration:
> 
> Sep 03 15:00:32 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 15158 overflow 0 deltacachemiss 15158
> Sep 03 15:00:34 migration status: active (transferred 119318244, remaining 
> 1987846144), total 2114387968)
> Sep 03 15:00:34 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 29130 overflow 0 deltacachemiss 13972
> Sep 03 15:00:36 migration status: active (transferred 156261310, remaining 
> 1342005248), total 2114387968)
> Sep 03 15:00:36 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 38113 overflow 0 deltacachemiss 8983
> Sep 03 15:00:38 migration status: active (transferred 195909631, remaining 
> 786370560), total 2114387968)
> Sep 03 15:00:38 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 47762 overflow 0 deltacachemiss 9649
> Sep 03 15:00:40 migration status: active (transferred 264583501, remaining 
> 716410880), total 2114387968)
> Sep 03 15:00:40 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 64528 overflow 0 deltacachemiss 16766
> Sep 03 15:00:42 migration status: active (transferred 336759138, remaining 
> 644227072), total 2114387968)
> Sep 03 15:00:42 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 82149 overflow 0 deltacachemiss 17621
> Sep 03 15:00:44 migration status: active (transferred 399641876, remaining 
> 577712128), total 2114387968)
> Sep 03 15:00:44 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 97501 overflow 0 deltacachemiss 15352
> Sep 03 15:00:46 migration status: active (transferred 446500707, remaining 
> 528441344), total 2114387968)
> Sep 03 15:00:46 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 108941 overflow 0 deltacachemiss 11440
> Sep 03 15:00:48 migration status: active (transferred 501567884, remaining 
> 471134208), total 2114387968)
> Sep 03 15:00:48 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 122385 overflow 0 deltacachemiss 13444
> Sep 03 15:00:50 migration status: active (transferred 566289154, remaining 
> 404901888), total 2114387968)
> Sep 03 15:00:50 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 138186 overflow 0 deltacachemiss 15801
> Sep 03 15:00:52 migration status: active (transferred 631297144, remaining 
> 338370560), total 2114387968)
> Sep 03 15:00:52 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 154057 overflow 0 deltacachemiss 15871
> Sep 03 15:00:54 migration status: active (transferred 695854441, remaining 
> 272830464), total 2114387968)
> Sep 03 15:00:54 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 169818 overflow 0 deltacachemiss 15761
> Sep 03 15:00:56 migration status: active (transferred 763447412, remaining 
> 202059776), total 2114387968)
> Sep 03 15:00:56 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 186320 overflow 0 deltacachemiss 16502
> Sep 03 15:00:58 migration status: active (transferred 828361066, remaining 
> 136142848), total 2114387968)
> Sep 03 15:00:58 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 202168 overflow 0 deltacachemiss 15848
> Sep 03 15:01:00 migration status: active (transferred 879799289, remaining 
> 82030592), total 2114387968)
> Sep 03 15:01:00 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 214726 overflow 0 deltacachemiss 12558
> Sep 03 15:01:02 migration status: active (transferred 940920124, remaining 
> 19603456), total 2114387968)
> Sep 03 15:01:02 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 229648 overflow 0 deltacachemiss 14922
> Sep 03 15:01:03 migration status: active (transferred 949001101, remaining 
> 69087232), total 2114387968)
> Sep 03 15:01:03 migration xbzrle cachesize: 67108864 transferred 0 pages 0 
> cachemiss 231620 overflow 0 deltacachemiss 1972
> Sep 03 15:01:03 migration status: active (transferred 959106075, remaining 
> 58187776), total 2114387968)
> Sep 03 15:01:03 migration xbzrle cachesize: 67108864 transferred 100 pages 2 
> cachemiss 234087 overflow 0 deltacachemiss 2467
> Sep 03 15:01:03 migration status: active (transferred 969236581, remaining 
> 43577344), total 2114387968)
> Sep 03 15:01:03 migration xbzrle cachesize: 67108864 transferred 100 pages 2 
> cachemiss 236560 overflow 0 deltacachemiss 2473
> Sep 03 15:01:04 migration status: active (transferred 979374186, remaining 
> 33427456), total 2114387968)
> Sep 03 15:01:04 migration xbzrle cachesize: 67108864 transferred 100 pages 2 
> cachemiss 239035 overflow 0 deltacachemiss 2475
> Sep 03 15:01:04 migration status: active (transferred 989479026, remaining 
> 23302144), total 2114387968)
> Sep 03 15:01:04 migration xbzrle cachesize: 67108864 transferred 100 pages 2 
> cachemiss 241502 overflow 0 deltacachemiss 2467
> Sep 03 15:01:04 migration status: active (transferred 999747708, remaining 
> 28950528), total 2114387968)
> Sep 03 15:01:04 migration xbzrle cachesize: 67108864 transferred 100 pages 2 
> cachemiss 244009 overflow 0 deltacachemiss 2507
> Sep 03 15:01:04 migration status: active (transferred 1006616716, remaining 
> 22032384), total 2114387968)
> Sep 03 15:01:04 migration xbzrle cachesize: 67108864 transferred 100 pages 2 
> cachemiss 245686 overflow 0 deltacachemiss 1677
> Sep 03 15:01:05 migration status: active (transferred 1020085038, remaining 
> 18358272), total 2114387968)
> Sep 03 15:01:05 migration xbzrle cachesize: 67108864 transferred 1905397 
> pages 991 cachemiss 248381 overflow 128 deltacachemiss 2695
> Sep 03 15:01:05 migration status: active (transferred 1030213871, remaining 
> 14524416), total 2114387968)
> Sep 03 15:01:05 migration xbzrle cachesize: 67108864 transferred 6103206 
> pages 3227 cachemiss 249646 overflow 311 deltacachemiss 1265
> Sep 03 15:01:05 migration status: active (transferred 1040306669, remaining 
> 7348224), total 2114387968)
> Sep 03 15:01:05 migration xbzrle cachesize: 67108864 transferred 11579783 
> pages 6075 cachemiss 250496 overflow 588 deltacachemiss 850
> Sep 03 15:01:06 migration status: active (transferred 1047695771, remaining 
> 13287424), total 2114387968)
> Sep 03 15:01:06 migration xbzrle cachesize: 67108864 transferred 15286549 
> pages 9455 cachemiss 251271 overflow 712 deltacachemiss 775
> Sep 03 15:01:06 migration status: active (transferred 1054790825, remaining 
> 8896512), total 2114387968)
> Sep 03 15:01:06 migration xbzrle cachesize: 67108864 transferred 18600974 
> pages 12483 cachemiss 252020 overflow 886 deltacachemiss 749
> Sep 03 15:01:06 migration status: active (transferred 1064883713, remaining 
> 3870720), total 2114387968)
> Sep 03 15:01:06 migration xbzrle cachesize: 67108864 transferred 24180048 
> pages 15326 cachemiss 252856 overflow 1152 deltacachemiss 836
> Sep 03 15:01:07 migration status: active (transferred 1073205318, remaining 
> 13889536), total 2114387968)
> Sep 03 15:01:07 migration xbzrle cachesize: 67108864 transferred 28106603 
> pages 18135 cachemiss 253769 overflow 1312 deltacachemiss 913
> Sep 03 15:01:07 migration status: active (transferred 1083269753, remaining 
> 8450048), total 2114387968)
> Sep 03 15:01:07 migration xbzrle cachesize: 67108864 transferred 33685887 
> pages 20865 cachemiss 254445 overflow 1731 deltacachemiss 676
> Sep 03 15:01:07 migration status: active (transferred 1093363118, remaining 
> 13656064), total 2114387968)
> Sep 03 15:01:07 migration xbzrle cachesize: 67108864 transferred 39318685 
> pages 23895 cachemiss 255182 overflow 2083 deltacachemiss 737
> Sep 03 15:01:08 migration status: active (transferred 1104228774, remaining 
> 3948544), total 2114387968)
> Sep 03 15:01:08 migration xbzrle cachesize: 67108864 transferred 45621367 
> pages 27392 cachemiss 255988 overflow 2391 deltacachemiss 806
> Sep 03 15:01:08 migration status: active (transferred 1112828342, remaining 
> 4714496), total 2114387968)
> Sep 03 15:01:08 migration xbzrle cachesize: 67108864 transferred 50624630 
> pages 29895 cachemiss 256597 overflow 2660 deltacachemiss 609
> Sep 03 15:01:08 migration status: active (transferred 1121646599, remaining 
> 5951488), total 2114387968)
> Sep 03 15:01:08 migration xbzrle cachesize: 67108864 transferred 54638264 
> pages 32011 cachemiss 257423 overflow 3007 deltacachemiss 826
> Sep 03 15:01:09 migration status: active (transferred 1131774276, remaining 
> 14245888), total 2114387968)
> Sep 03 15:01:09 migration xbzrle cachesize: 67108864 transferred 59924446 
> pages 34904 cachemiss 258311 overflow 3301 deltacachemiss 888
> Sep 03 15:01:09 migration status: active (transferred 1139357344, remaining 
> 12529664), total 2114387968)
> Sep 03 15:01:09 migration xbzrle cachesize: 67108864 transferred 62850320 
> pages 39537 cachemiss 259385 overflow 3364 deltacachemiss 1074
> Sep 03 15:01:09 migration status: active (transferred 1147848486, remaining 
> 12972032), total 2114387968)
> Sep 03 15:01:09 migration xbzrle cachesize: 67108864 transferred 67237245 
> pages 43153 cachemiss 260229 overflow 3522 deltacachemiss 844
> Sep 03 15:01:10 migration status: active (transferred 1157968777, remaining 
> 7729152), total 2114387968)
> Sep 03 15:01:10 migration xbzrle cachesize: 67108864 transferred 72753611 
> pages 45998 cachemiss 260890 overflow 3985 deltacachemiss 661
> Sep 03 15:01:10 migration status: active (transferred 1168065017, remaining 
> 3137536), total 2114387968)
> Sep 03 15:01:10 migration xbzrle cachesize: 67108864 transferred 78000165 
> pages 48646 cachemiss 261660 overflow 4399 deltacachemiss 770
> Sep 03 15:01:10 migration status: active (transferred 1176461216, remaining 
> 5459968), total 2114387968)
> Sep 03 15:01:10 migration xbzrle cachesize: 67108864 transferred 81931708 
> pages 50605 cachemiss 262364 overflow 4785 deltacachemiss 704
> Sep 03 15:01:11 migration status: active (transferred 1187721613, remaining 
> 15183872), total 2114387968)
> Sep 03 15:01:11 migration xbzrle cachesize: 67108864 transferred 87588758 
> pages 53584 cachemiss 263313 overflow 5204 deltacachemiss 949
> Sep 03 15:01:11 migration status: active (transferred 1196325449, remaining 
> 15134720), total 2114387968)
> Sep 03 15:01:11 migration xbzrle cachesize: 67108864 transferred 92063809 
> pages 55810 cachemiss 263933 overflow 5592 deltacachemiss 620
> Sep 03 15:01:11 migration status: active (transferred 1206419175, remaining 
> 7557120), total 2114387968)
> Sep 03 15:01:11 migration xbzrle cachesize: 67108864 transferred 97500362 
> pages 58460 cachemiss 264676 overflow 5986 deltacachemiss 743
> Sep 03 15:01:11 migration status: active (transferred 1216510694, remaining 
> 15560704), total 2114387968)
> Sep 03 15:01:11 migration xbzrle cachesize: 67108864 transferred 102480030 
> pages 61212 cachemiss 265552 overflow 6358 deltacachemiss 876
> Sep 03 15:01:12 migration status: active (transferred 1225162926, remaining 
> 14811136), total 2114387968)
> Sep 03 15:01:12 migration xbzrle cachesize: 67108864 transferred 107155027 
> pages 63456 cachemiss 266234 overflow 6647 deltacachemiss 682
> Sep 03 15:01:12 migration status: active (transferred 1235228927, remaining 
> 12378112), total 2114387968)
> Sep 03 15:01:12 migration xbzrle cachesize: 67108864 transferred 112440978 
> pages 66311 cachemiss 267101 overflow 6947 deltacachemiss 867
> Sep 03 15:01:12 migration status: active (transferred 1245060361, remaining 
> 14848000), total 2114387968)
> Sep 03 15:01:12 migration xbzrle cachesize: 67108864 transferred 116886157 
> pages 68943 cachemiss 268239 overflow 7124 deltacachemiss 1138
> Sep 03 15:01:13 migration status: active (transferred 1255151722, remaining 
> 14151680), total 2114387968)
> Sep 03 15:01:13 migration xbzrle cachesize: 67108864 transferred 121058631 
> pages 71389 cachemiss 269475 overflow 7333 deltacachemiss 1236
> Sep 03 15:01:13 migration status: active (transferred 1265145124, remaining 
> 12636160), total 2114387968)
> Sep 03 15:01:13 migration xbzrle cachesize: 67108864 transferred 125804843 
> pages 74069 cachemiss 270559 overflow 7530 deltacachemiss 1084
> Sep 03 15:01:13 migration status: active (transferred 1274160338, remaining 
> 9003008), total 2114387968)
> Sep 03 15:01:13 migration xbzrle cachesize: 67108864 transferred 130109480 
> pages 76352 cachemiss 271460 overflow 7779 deltacachemiss 901
> Sep 03 15:01:14 migration status: active (transferred 1285539796, remaining 
> 12382208), total 2114387968)
> Sep 03 15:01:14 migration xbzrle cachesize: 67108864 transferred 136704693 
> pages 79967 cachemiss 272399 overflow 8008 deltacachemiss 939
> Sep 03 15:01:14 migration status: active (transferred 1294465950, remaining 
> 10989568), total 2114387968)
> Sep 03 15:01:14 migration xbzrle cachesize: 67108864 transferred 141522529 
> pages 82691 cachemiss 273221 overflow 8189 deltacachemiss 822
> Sep 03 15:01:14 migration status: active (transferred 1304111247, remaining 
> 12431360), total 2114387968)
> Sep 03 15:01:14 migration xbzrle cachesize: 67108864 transferred 146256619 
> pages 85117 cachemiss 274137 overflow 8472 deltacachemiss 916
> Sep 03 15:01:15 migration status: active (transferred 1313714825, remaining 
> 9031680), total 2114387968)
> Sep 03 15:01:15 migration xbzrle cachesize: 67108864 transferred 150961349 
> pages 87828 cachemiss 275101 overflow 8704 deltacachemiss 964
> Sep 03 15:01:15 migration status: active (transferred 1321615036, remaining 
> 8843264), total 2114387968)
> Sep 03 15:01:15 migration xbzrle cachesize: 67108864 transferred 155568345 
> pages 90258 cachemiss 275753 overflow 8856 deltacachemiss 652
> Sep 03 15:01:15 migration status: active (transferred 1332266783, remaining 
> 4997120), total 2114387968)
> Sep 03 15:01:15 migration xbzrle cachesize: 67108864 transferred 161419460 
> pages 93534 cachemiss 276690 overflow 9091 deltacachemiss 937
> Sep 03 15:01:16 migration status: active (transferred 1340961412, remaining 
> 4333568), total 2114387968)
> Sep 03 15:01:16 migration xbzrle cachesize: 67108864 transferred 165833625 
> pages 96166 cachemiss 277571 overflow 9255 deltacachemiss 881
> Sep 03 15:01:16 migration status: active (transferred 1349569887, remaining 
> 4313088), total 2114387968)
> Sep 03 15:01:16 migration xbzrle cachesize: 67108864 transferred 170194521 
> pages 98568 cachemiss 278366 overflow 9497 deltacachemiss 795
> Sep 03 15:01:16 migration status: active (transferred 1357212762, remaining 
> 5083136), total 2114387968)
> Sep 03 15:01:16 migration xbzrle cachesize: 67108864 transferred 173651067 
> pages 100693 cachemiss 279206 overflow 9679 deltacachemiss 840
> Sep 03 15:01:16 migration status: active (transferred 1362924614, remaining 
> 12259328), total 2114387968)
> Sep 03 15:01:16 migration xbzrle cachesize: 67108864 transferred 175983690 
> pages 102350 cachemiss 279936 overflow 9774 deltacachemiss 730
> Sep 03 15:01:17 migration status: active (transferred 1369555421, remaining 
> 17477632), total 2114387968)
> Sep 03 15:01:17 migration xbzrle cachesize: 67108864 transferred 178796811 
> pages 103929 cachemiss 280702 overflow 9940 deltacachemiss 766
> Sep 03 15:01:17 migration status: active (transferred 1377394106, remaining 
> 17633280), total 2114387968)
> Sep 03 15:01:17 migration xbzrle cachesize: 67108864 transferred 181986406 
> pages 106134 cachemiss 281646 overflow 10131 deltacachemiss 944
> Sep 03 15:01:17 migration status: active (transferred 1383071812, remaining 
> 5853184), total 2114387968)
> Sep 03 15:01:17 migration xbzrle cachesize: 67108864 transferred 184276689 
> pages 107825 cachemiss 282388 overflow 10216 deltacachemiss 742
> Sep 03 15:01:18 migration status: active (transferred 1389248230, remaining 
> 12910592), total 2114387968)
> Sep 03 15:01:18 migration xbzrle cachesize: 67108864 transferred 187049184 
> pages 109409 cachemiss 283037 overflow 10398 deltacachemiss 649
> Sep 03 15:01:18 migration status: active (transferred 1395386430, remaining 
> 20598784), total 2114387968)
> Sep 03 15:01:18 migration xbzrle cachesize: 67108864 transferred 189005134 
> pages 110884 cachemiss 283908 overflow 10548 deltacachemiss 871
> Sep 03 15:01:18 migration status: active (transferred 1402994779, remaining 
> 6443008), total 2114387968)
> Sep 03 15:01:18 migration xbzrle cachesize: 67108864 transferred 191456304 
> pages 112393 cachemiss 284917 overflow 10798 deltacachemiss 1009
> Sep 03 15:01:19 migration status: active (transferred 1409267086, remaining 
> 13611008), total 2114387968)
> Sep 03 15:01:19 migration xbzrle cachesize: 67108864 transferred 193247446 
> pages 113783 cachemiss 285846 overflow 10963 deltacachemiss 929
> Sep 03 15:01:19 migration status: active (transferred 1420299927, remaining 
> 6414336), total 2114387968)
> Sep 03 15:01:19 migration xbzrle cachesize: 67108864 transferred 197795949 
> pages 116951 cachemiss 287280 overflow 11112 deltacachemiss 1434
> Sep 03 15:01:19 migration status: active (transferred 1430776107, remaining 
> 9695232), total 2114387968)
> Sep 03 15:01:19 migration xbzrle cachesize: 67108864 transferred 202680859 
> pages 120761 cachemiss 288557 overflow 11200 deltacachemiss 1277
> Sep 03 15:01:20 migration status: active (transferred 1440821167, remaining 
> 2637824), total 2114387968)
> Sep 03 15:01:20 migration xbzrle cachesize: 67108864 transferred 208318417 
> pages 125073 cachemiss 289534 overflow 11299 deltacachemiss 977
> Sep 03 15:01:20 migration status: active (transferred 1447117251, remaining 
> 12091392), total 2114387968)
> Sep 03 15:01:20 migration xbzrle cachesize: 67108864 transferred 211300697 
> pages 127309 cachemiss 290244 overflow 11398 deltacachemiss 710
> Sep 03 15:01:20 migration status: active (transferred 1456026090, remaining 
> 12906496), total 2114387968)
> Sep 03 15:01:20 migration xbzrle cachesize: 67108864 transferred 216105094 
> pages 131039 cachemiss 291149 overflow 11495 deltacachemiss 905
> Sep 03 15:01:21 migration status: active (transferred 1463954394, remaining 
> 2822144), total 2114387968)
> Sep 03 15:01:21 migration xbzrle cachesize: 67108864 transferred 220531007 
> pages 134323 cachemiss 291906 overflow 11593 deltacachemiss 757
> Sep 03 15:01:21 migration status: active (transferred 1470931218, remaining 
> 11534336), total 2114387968)
> Sep 03 15:01:21 migration xbzrle cachesize: 67108864 transferred 223866256 
> pages 137986 cachemiss 292723 overflow 11665 deltacachemiss 817
> Sep 03 15:01:21 migration status: active (transferred 1477743090, remaining 
> 10235904), total 2114387968)
> Sep 03 15:01:21 migration xbzrle cachesize: 67108864 transferred 227683667 
> pages 141111 cachemiss 293368 overflow 11751 deltacachemiss 645
> Sep 03 15:01:21 migration status: active (transferred 1486951605, remaining 
> 10551296), total 2114387968)
> Sep 03 15:01:21 migration xbzrle cachesize: 67108864 transferred 232771294 
> pages 145280 cachemiss 294269 overflow 11856 deltacachemiss 901
> Sep 03 15:01:22 migration status: active (transferred 1493416356, remaining 
> 14299136), total 2114387968)
> Sep 03 15:01:22 migration xbzrle cachesize: 67108864 transferred 235959068 
> pages 148181 cachemiss 294953 overflow 11972 deltacachemiss 684
> Sep 03 15:01:22 migration status: active (transferred 1502498989, remaining 
> 10592256), total 2114387968)
> Sep 03 15:01:22 migration xbzrle cachesize: 67108864 transferred 240400566 
> pages 151598 cachemiss 295980 overflow 12078 deltacachemiss 1027
> Sep 03 15:01:22 migration status: active (transferred 1511006667, remaining 
> 10240000), total 2114387968)
> Sep 03 15:01:22 migration xbzrle cachesize: 67108864 transferred 244648073 
> pages 155655 cachemiss 296923 overflow 12175 deltacachemiss 943
> Sep 03 15:01:23 migration status: active (transferred 1521022229, remaining 
> 10424320), total 2114387968)
> Sep 03 15:01:23 migration xbzrle cachesize: 67108864 transferred 250362498 
> pages 159932 cachemiss 297842 overflow 12306 deltacachemiss 919
> Sep 03 15:01:23 migration status: active (transferred 1530294933, remaining 
> 9367552), total 2114387968)
> Sep 03 15:01:23 migration xbzrle cachesize: 67108864 transferred 255219350 
> pages 163947 cachemiss 298815 overflow 12411 deltacachemiss 973
> Sep 03 15:01:23 migration status: active (transferred 1540134732, remaining 
> 9908224), total 2114387968)
> Sep 03 15:01:23 migration xbzrle cachesize: 67108864 transferred 260909578 
> pages 168294 cachemiss 299727 overflow 12512 deltacachemiss 912
> Sep 03 15:01:24 migration status: active (transferred 1547801099, remaining 
> 2134016), total 2114387968)
> Sep 03 15:01:24 migration xbzrle cachesize: 67108864 transferred 263844849 
> pages 171252 cachemiss 300772 overflow 12622 deltacachemiss 1045
> Sep 03 15:01:24 migration status: active (transferred 1554064971, remaining 
> 12468224), total 2114387968)
> Sep 03 15:01:24 migration xbzrle cachesize: 67108864 transferred 267032335 
> pages 173533 cachemiss 301438 overflow 12707 deltacachemiss 666
> Sep 03 15:01:24 migration status: active (transferred 1565158149, remaining 
> 11444224), total 2114387968)
> Sep 03 15:01:24 migration xbzrle cachesize: 67108864 transferred 273009204 
> pages 178082 cachemiss 302561 overflow 12833 deltacachemiss 1123
> Sep 03 15:01:25 migration status: active (transferred 1574725942, remaining 
> 11866112), total 2114387968)
> Sep 03 15:01:25 migration xbzrle cachesize: 67108864 transferred 278128445 
> pages 182505 cachemiss 303555 overflow 12925 deltacachemiss 994
> Sep 03 15:01:25 migration status: active (transferred 1584030411, remaining 
> 10473472), total 2114387968)
> Sep 03 15:01:25 migration xbzrle cachesize: 67108864 transferred 282722425 
> pages 187028 cachemiss 304611 overflow 13019 deltacachemiss 1056
> Sep 03 15:01:25 migration status: active (transferred 1592843701, remaining 
> 2134016), total 2114387968)
> Sep 03 15:01:25 migration xbzrle cachesize: 67108864 transferred 287443772 
> pages 192031 cachemiss 305549 overflow 13080 deltacachemiss 938
> Sep 03 15:01:26 migration status: active (transferred 1601793792, remaining 
> 6676480), total 2114387968)
> Sep 03 15:01:26 migration xbzrle cachesize: 67108864 transferred 292445279 
> pages 196112 cachemiss 306407 overflow 13186 deltacachemiss 858
> Sep 03 15:01:26 migration status: active (transferred 1609953585, remaining 
> 12259328), total 2114387968)
> Sep 03 15:01:26 migration xbzrle cachesize: 67108864 transferred 296742516 
> pages 199610 cachemiss 307294 overflow 13242 deltacachemiss 887
> Sep 03 15:01:26 migration status: active (transferred 1618150798, remaining 
> 4358144), total 2114387968)
> Sep 03 15:01:26 migration xbzrle cachesize: 67108864 transferred 301634218 
> pages 203248 cachemiss 308022 overflow 13321 deltacachemiss 728
> Sep 03 15:01:26 migration status: active (transferred 1625047015, remaining 
> 11665408), total 2114387968)
> Sep 03 15:01:26 migration xbzrle cachesize: 67108864 transferred 305642729 
> pages 206530 cachemiss 308695 overflow 13353 deltacachemiss 673
> Sep 03 15:01:27 migration status: active (transferred 1631583544, remaining 
> 8450048), total 2114387968)
> Sep 03 15:01:27 migration xbzrle cachesize: 67108864 transferred 309520928 
> pages 209601 cachemiss 309308 overflow 13389 deltacachemiss 613
> Sep 03 15:01:27 migration status: active (transferred 1637654847, remaining 
> 7917568), total 2114387968)
> Sep 03 15:01:27 migration xbzrle cachesize: 67108864 transferred 313118229 
> pages 212178 cachemiss 309876 overflow 13425 deltacachemiss 568
> Sep 03 15:01:27 migration status: active (transferred 1644608997, remaining 
> 6324224), total 2114387968)
> Sep 03 15:01:27 migration xbzrle cachesize: 67108864 transferred 317303461 
> pages 215070 cachemiss 310504 overflow 13473 deltacachemiss 628
> Sep 03 15:01:28 migration status: active (transferred 1652796103, remaining 
> 4423680), total 2114387968)
> Sep 03 15:01:28 migration xbzrle cachesize: 67108864 transferred 321243002 
> pages 218055 cachemiss 311465 overflow 13549 deltacachemiss 961
> Sep 03 15:01:28 migration status: active (transferred 1662238886, remaining 
> 12513280), total 2114387968)
> Sep 03 15:01:28 migration xbzrle cachesize: 67108864 transferred 326180167 
> pages 221148 cachemiss 312524 overflow 13590 deltacachemiss 1059
> Sep 03 15:01:28 migration status: active (transferred 1670338088, remaining 
> 5959680), total 2114387968)
> Sep 03 15:01:28 migration xbzrle cachesize: 67108864 transferred 330945203 
> pages 224318 cachemiss 313295 overflow 13633 deltacachemiss 771
> Sep 03 15:01:29 migration status: active (transferred 1682607732, remaining 
> 5373952), total 2114387968)
> Sep 03 15:01:29 migration xbzrle cachesize: 67108864 transferred 339032796 
> pages 228942 cachemiss 314248 overflow 13701 deltacachemiss 953
> Sep 03 15:01:29 migration status: active (transferred 1691288677, remaining 
> 5660672), total 2114387968)
> Sep 03 15:01:29 migration xbzrle cachesize: 67108864 transferred 344379577 
> pages 232454 cachemiss 315033 overflow 13730 deltacachemiss 785
> Sep 03 15:01:29 migration status: active (transferred 1699318970, remaining 
> 13336576), total 2114387968)
> Sep 03 15:01:29 migration xbzrle cachesize: 67108864 transferred 349546739 
> pages 235705 cachemiss 315700 overflow 13762 deltacachemiss 667
> Sep 03 15:01:30 migration status: active (transferred 1709411503, remaining 
> 7815168), total 2114387968)
> Sep 03 15:01:30 migration xbzrle cachesize: 67108864 transferred 355408081 
> pages 238683 cachemiss 316531 overflow 13964 deltacachemiss 831
> Sep 03 15:01:30 migration status: active (transferred 1719503532, remaining 
> 12898304), total 2114387968)
> Sep 03 15:01:30 migration xbzrle cachesize: 67108864 transferred 361350845 
> pages 241661 cachemiss 317275 overflow 14233 deltacachemiss 744
> Sep 03 15:01:30 migration status: active (transferred 1728857886, remaining 
> 6778880), total 2114387968)
> Sep 03 15:01:30 migration xbzrle cachesize: 67108864 transferred 366736158 
> pages 244405 cachemiss 317948 overflow 14529 deltacachemiss 673
> Sep 03 15:01:31 migration status: active (transferred 1738122997, remaining 
> 2527232), total 2114387968)
> Sep 03 15:01:31 migration xbzrle cachesize: 67108864 transferred 372220641 
> pages 246999 cachemiss 318550 overflow 14850 deltacachemiss 602
> Sep 03 15:01:31 migration status: active (transferred 1745051297, remaining 
> 2850816), total 2114387968)
> Sep 03 15:01:31 migration xbzrle cachesize: 67108864 transferred 376056452 
> pages 248933 cachemiss 319032 overflow 15123 deltacachemiss 482
> Sep 03 15:01:31 migration status: active (transferred 1754719918, remaining 
> 12759040), total 2114387968)
> Sep 03 15:01:31 migration xbzrle cachesize: 67108864 transferred 381239933 
> pages 251517 cachemiss 319744 overflow 15506 deltacachemiss 712
> Sep 03 15:01:32 migration status: active (transferred 1763103223, remaining 
> 9637888), total 2114387968)
> Sep 03 15:01:32 migration xbzrle cachesize: 67108864 transferred 385265065 
> pages 254298 cachemiss 320538 overflow 15776 deltacachemiss 794
> Sep 03 15:01:32 migration status: active (transferred 1771003331, remaining 
> 3596288), total 2114387968)
> Sep 03 15:01:32 migration xbzrle cachesize: 67108864 transferred 388708690 
> pages 257558 cachemiss 321498 overflow 15904 deltacachemiss 960
> Sep 03 15:01:32 migration status: active (transferred 1778157089, remaining 
> 12443648), total 2114387968)
> Sep 03 15:01:32 migration xbzrle cachesize: 67108864 transferred 392561052 
> pages 260534 cachemiss 322166 overflow 16042 deltacachemiss 668
> Sep 03 15:01:33 migration status: active (transferred 1788495031, remaining 
> 10276864), total 2114387968)
> Sep 03 15:01:33 migration xbzrle cachesize: 67108864 transferred 398295061 
> pages 264848 cachemiss 323101 overflow 16231 deltacachemiss 935
> Sep 03 15:01:33 migration status: active (transferred 1794009984, remaining 
> 11046912), total 2114387968)
> Sep 03 15:01:33 migration xbzrle cachesize: 67108864 transferred 400795337 
> pages 266649 cachemiss 323687 overflow 16381 deltacachemiss 586
> Sep 03 15:01:33 migration status: active (transferred 1802929181, remaining 
> 5738496), total 2114387968)
> Sep 03 15:01:33 migration xbzrle cachesize: 67108864 transferred 406032198 
> pages 269339 cachemiss 324245 overflow 16722 deltacachemiss 558
> Sep 03 15:01:33 migration status: active (transferred 1813019717, remaining 
> 13381632), total 2114387968)
> Sep 03 15:01:33 migration xbzrle cachesize: 67108864 transferred 412084058 
> pages 272447 cachemiss 324891 overflow 17062 deltacachemiss 646
> Sep 03 15:01:34 migration status: active (transferred 1822507168, remaining 
> 3809280), total 2114387968)
> Sep 03 15:01:34 migration xbzrle cachesize: 67108864 transferred 416672658 
> pages 275640 cachemiss 325836 overflow 17313 deltacachemiss 945
> Sep 03 15:01:34 migration status: active (transferred 1829060930, remaining 
> 10371072), total 2114387968)
> Sep 03 15:01:34 migration xbzrle cachesize: 67108864 transferred 418147336 
> pages 279826 cachemiss 327002 overflow 17387 deltacachemiss 1166
> Sep 03 15:01:34 migration status: active (transferred 1835035082, remaining 
> 11374592), total 2114387968)
> Sep 03 15:01:34 migration xbzrle cachesize: 67108864 transferred 419374185 
> pages 284812 cachemiss 328106 overflow 17442 deltacachemiss 1104
> Sep 03 15:01:35 migration status: active (transferred 1840925069, remaining 
> 11190272), total 2114387968)
> Sep 03 15:01:35 migration xbzrle cachesize: 67108864 transferred 420520965 
> pages 289916 cachemiss 329221 overflow 17485 deltacachemiss 1115
> Sep 03 15:01:35 migration status: active (transferred 1846770014, remaining 
> 9371648), total 2114387968)
> Sep 03 15:01:35 migration xbzrle cachesize: 67108864 transferred 421458853 
> pages 294580 cachemiss 330372 overflow 17532 deltacachemiss 1151
> Sep 03 15:01:35 migration status: active (transferred 1853314839, remaining 
> 2514944), total 2114387968)
> Sep 03 15:01:35 migration xbzrle cachesize: 67108864 transferred 423964993 
> pages 298173 cachemiss 331222 overflow 17668 deltacachemiss 850
> Sep 03 15:01:36 migration status: active (transferred 1862231519, remaining 
> 15454208), total 2114387968)
> Sep 03 15:01:36 migration xbzrle cachesize: 67108864 transferred 428937200 
> pages 301153 cachemiss 331999 overflow 17854 deltacachemiss 777
> Sep 03 15:01:36 migration status: active (transferred 1870584409, remaining 
> 11419648), total 2114387968)
> Sep 03 15:01:36 migration xbzrle cachesize: 67108864 transferred 431326236 
> pages 304655 cachemiss 333333 overflow 17976 deltacachemiss 1334
> Sep 03 15:01:36 migration status: active (transferred 1878153329, remaining 
> 4648960), total 2114387968)
> Sep 03 15:01:36 migration xbzrle cachesize: 67108864 transferred 434057751 
> pages 307541 cachemiss 334375 overflow 18115 deltacachemiss 1042
> Sep 03 15:01:37 migration status: active (transferred 1885289154, remaining 
> 12533760), total 2114387968)
> Sep 03 15:01:37 migration xbzrle cachesize: 67108864 transferred 438228048 
> pages 310420 cachemiss 334915 overflow 18299 deltacachemiss 540
> Sep 03 15:01:37 migration status: active (transferred 1892622878, remaining 
> 4374528), total 2114387968)
> Sep 03 15:01:37 migration xbzrle cachesize: 67108864 transferred 441490323 
> pages 313985 cachemiss 335735 overflow 18473 deltacachemiss 820
> Sep 03 15:01:37 migration status: active (transferred 1901073971, remaining 
> 13205504), total 2114387968)
> Sep 03 15:01:37 migration xbzrle cachesize: 67108864 transferred 446529428 
> pages 316687 cachemiss 336305 overflow 18736 deltacachemiss 570
> Sep 03 15:01:38 migration status: active (transferred 1910878152, remaining 
> 9048064), total 2114387968)
> Sep 03 15:01:38 migration xbzrle cachesize: 67108864 transferred 451586319 
> pages 319720 cachemiss 337190 overflow 19010 deltacachemiss 885
> Sep 03 15:01:38 migration status: active (transferred 1918738540, remaining 
> 10661888), total 2114387968)
> Sep 03 15:01:38 migration xbzrle cachesize: 67108864 transferred 455338404 
> pages 321664 cachemiss 337933 overflow 19270 deltacachemiss 743
> Sep 03 15:01:38 migration status: active (transferred 1928827689, remaining 
> 6860800), total 2114387968)
> Sep 03 15:01:38 migration xbzrle cachesize: 67108864 transferred 460549198 
> pages 324277 cachemiss 338738 overflow 19656 deltacachemiss 805
> Sep 03 15:01:38 migration status: active (transferred 1937151384, remaining 
> 3428352), total 2114387968)
> Sep 03 15:01:38 migration xbzrle cachesize: 67108864 transferred 465043115 
> pages 326494 cachemiss 339336 overflow 19993 deltacachemiss 598
> Sep 03 15:01:39 migration status: active (transferred 1946239164, remaining 
> 13590528), total 2114387968)
> Sep 03 15:01:39 migration xbzrle cachesize: 67108864 transferred 469952951 
> pages 328931 cachemiss 339977 overflow 20372 deltacachemiss 641
> Sep 03 15:01:39 migration status: active (transferred 1955868111, remaining 
> 2822144), total 2114387968)
> Sep 03 15:01:39 migration xbzrle cachesize: 67108864 transferred 474850991 
> pages 332609 cachemiss 340857 overflow 20647 deltacachemiss 880
> Sep 03 15:01:39 migration status: active (transferred 1965033342, remaining 
> 7884800), total 2114387968)
> Sep 03 15:01:39 migration xbzrle cachesize: 67108864 transferred 479064122 
> pages 336578 cachemiss 341853 overflow 20860 deltacachemiss 996
> Sep 03 15:01:40 migration status: active (transferred 1972991062, remaining 
> 8335360), total 2114387968)
> Sep 03 15:01:40 migration xbzrle cachesize: 67108864 transferred 482389235 
> pages 340369 cachemiss 342906 overflow 20938 deltacachemiss 1053
> Sep 03 15:01:40 migration status: active (transferred 1978818556, remaining 
> 10371072), total 2114387968)
> Sep 03 15:01:40 migration xbzrle cachesize: 67108864 transferred 483760246 
> pages 344727 cachemiss 343955 overflow 20977 deltacachemiss 1049
> Sep 03 15:01:40 migration status: active (transferred 1984832873, remaining 
> 2367488), total 2114387968)
> Sep 03 15:01:40 migration xbzrle cachesize: 67108864 transferred 485600711 
> pages 348240 cachemiss 344922 overflow 21029 deltacachemiss 967
> Sep 03 15:01:41 migration status: active (transferred 1992114731, remaining 
> 4120576), total 2114387968)
> Sep 03 15:01:41 migration xbzrle cachesize: 67108864 transferred 489704053 
> pages 351282 cachemiss 345551 overflow 21176 deltacachemiss 629
> Sep 03 15:01:41 migration status: active (transferred 2001071596, remaining 
> 12619776), total 2114387968)
> Sep 03 15:01:41 migration xbzrle cachesize: 67108864 transferred 495339032 
> pages 354218 cachemiss 346153 overflow 21385 deltacachemiss 602
> Sep 03 15:01:41 migration status: active (transferred 2011152290, remaining 
> 5206016), total 2114387968)
> Sep 03 15:01:41 migration xbzrle cachesize: 67108864 transferred 501454775 
> pages 357298 cachemiss 346854 overflow 21652 deltacachemiss 701
> Sep 03 15:01:42 migration status: active (transferred 2019787845, remaining 
> 4804608), total 2114387968)
> Sep 03 15:01:42 migration xbzrle cachesize: 67108864 transferred 505945161 
> pages 359691 cachemiss 347502 overflow 22016 deltacachemiss 648
> Sep 03 15:01:42 migration status: active (transferred 2029872444, remaining 
> 12754944), total 2114387968)
> Sep 03 15:01:42 migration xbzrle cachesize: 67108864 transferred 510999852 
> pages 362419 cachemiss 348359 overflow 22387 deltacachemiss 857
> Sep 03 15:01:42 migration status: active (transferred 2039963167, remaining 
> 5341184), total 2114387968)
> Sep 03 15:01:42 migration xbzrle cachesize: 67108864 transferred 516535788 
> pages 365431 cachemiss 349089 overflow 22769 deltacachemiss 730
> Sep 03 15:01:42 migration status: active (transferred 2046790518, remaining 
> 13049856), total 2114387968)
> Sep 03 15:01:42 migration xbzrle cachesize: 67108864 transferred 519742235 
> pages 368455 cachemiss 349752 overflow 22990 deltacachemiss 663
> Sep 03 15:01:43 migration status: active (transferred 2056576433, remaining 
> 6127616), total 2114387968)
> Sep 03 15:01:43 migration xbzrle cachesize: 67108864 transferred 524764472 
> pages 371518 cachemiss 350495 overflow 23410 deltacachemiss 743
> Sep 03 15:01:43 migration status: active (transferred 2070030632, remaining 
> 5545984), total 2114387968)
> Sep 03 15:01:43 migration xbzrle cachesize: 67108864 transferred 532054161 
> pages 375458 cachemiss 351436 overflow 23974 deltacachemiss 941
> Sep 03 15:01:43 migration status: active (transferred 2080122845, remaining 
> 16842752), total 2114387968)
> Sep 03 15:01:43 migration xbzrle cachesize: 67108864 transferred 537186091 
> pages 378300 cachemiss 352183 overflow 24438 deltacachemiss 747
> Sep 03 15:01:44 migration status: active (transferred 2089596231, remaining 
> 19959808), total 2114387968)
> Sep 03 15:01:44 migration xbzrle cachesize: 67108864 transferred 541281406 
> pages 380898 cachemiss 353159 overflow 24775 deltacachemiss 976
> Sep 03 15:01:44 migration status: active (transferred 2098744287, remaining 
> 3334144), total 2114387968)
> Sep 03 15:01:44 migration xbzrle cachesize: 67108864 transferred 544596639 
> pages 382825 cachemiss 354171 overflow 25187 deltacachemiss 1012
> Sep 03 15:01:44 migration status: active (transferred 2107347747, remaining 
> 8888320), total 2114387968)
> Sep 03 15:01:44 migration xbzrle cachesize: 67108864 transferred 547735772 
> pages 384605 cachemiss 355136 overflow 25556 deltacachemiss 965
> Sep 03 15:01:45 migration status: active (transferred 2117427470, remaining 
> 6184960), total 2114387968)
> Sep 03 15:01:45 migration xbzrle cachesize: 67108864 transferred 551515759 
> pages 386875 cachemiss 356245 overflow 25985 deltacachemiss 1109
> Sep 03 15:01:45 migration status: active (transferred 2127522180, remaining 
> 6643712), total 2114387968)
> Sep 03 15:01:45 migration xbzrle cachesize: 67108864 transferred 555519519 
> pages 389206 cachemiss 357355 overflow 26362 deltacachemiss 1110
> Sep 03 15:01:45 migration status: active (transferred 2137180181, remaining 
> 6840320), total 2114387968)
> Sep 03 15:01:45 migration xbzrle cachesize: 67108864 transferred 559197228 
> pages 391393 cachemiss 358357 overflow 26820 deltacachemiss 1002
> Sep 03 15:01:46 migration status: active (transferred 2146653892, remaining 
> 7704576), total 2114387968)
> Sep 03 15:01:46 migration xbzrle cachesize: 67108864 transferred 563133130 
> pages 393677 cachemiss 359363 overflow 27166 deltacachemiss 1006
> Sep 03 15:01:46 migration status: active (transferred 2156170786, remaining 
> 8384512), total 2114387968)
> Sep 03 15:01:46 migration xbzrle cachesize: 67108864 transferred 566870481 
> pages 395661 cachemiss 360401 overflow 27539 deltacachemiss 1038
> Sep 03 15:01:46 migration status: active (transferred 2162116052, remaining 
> 18784256), total 2114387968)
> Sep 03 15:01:46 migration xbzrle cachesize: 67108864 transferred 569030821 
> pages 396971 cachemiss 361078 overflow 27786 deltacachemiss 677
> Sep 03 15:01:47 migration status: active (transferred 2167440600, remaining 
> 10747904), total 2114387968)
> Sep 03 15:01:47 migration xbzrle cachesize: 67108864 transferred 570951584 
> pages 397932 cachemiss 361557 overflow 28138 deltacachemiss 479
> Sep 03 15:01:47 migration status: active (transferred 2173189492, remaining 
> 25628672), total 2114387968)
> Sep 03 15:01:47 migration xbzrle cachesize: 67108864 transferred 572772376 
> pages 399193 cachemiss 362323 overflow 28331 deltacachemiss 766
> Sep 03 15:01:47 migration status: active (transferred 2181582500, remaining 
> 8200192), total 2114387968)
> Sep 03 15:01:47 migration xbzrle cachesize: 67108864 transferred 575605975 
> pages 400618 cachemiss 363152 overflow 28859 deltacachemiss 829
> Sep 03 15:01:47 migration status: active (transferred 2189847788, remaining 
> 11202560), total 2114387968)
> Sep 03 15:01:47 migration xbzrle cachesize: 67108864 transferred 578513648 
> pages 402362 cachemiss 364245 overflow 29074 deltacachemiss 1093
> Sep 03 15:01:48 migration status: active (transferred 2195877255, remaining 
> 14094336), total 2114387968)
> Sep 03 15:01:48 migration xbzrle cachesize: 67108864 transferred 580246400 
> pages 404116 cachemiss 365082 overflow 29286 deltacachemiss 837
> Sep 03 15:01:48 migration status: active (transferred 2202598243, remaining 
> 12357632), total 2114387968)
> Sep 03 15:01:48 migration xbzrle cachesize: 67108864 transferred 581462307 
> pages 407577 cachemiss 366385 overflow 29327 deltacachemiss 1303
> Sep 03 15:01:48 migration status: active (transferred 2210798256, remaining 
> 2859008), total 2114387968)
> Sep 03 15:01:48 migration xbzrle cachesize: 67108864 transferred 583350325 
> pages 413084 cachemiss 367860 overflow 29393 deltacachemiss 1475
> Sep 03 15:01:49 migration status: active (transferred 2217322581, remaining 
> 7753728), total 2114387968)
> Sep 03 15:01:49 migration xbzrle cachesize: 67108864 transferred 586159554 
> pages 415753 cachemiss 368500 overflow 29660 deltacachemiss 640
> Sep 03 15:01:49 migration status: active (transferred 2222843364, remaining 
> 10366976), total 2114387968)
> Sep 03 15:01:49 migration xbzrle cachesize: 67108864 transferred 589009723 
> pages 417307 cachemiss 368910 overflow 29902 deltacachemiss 410
> Sep 03 15:01:49 migration status: active (transferred 2229668707, remaining 
> 12546048), total 2114387968)
> Sep 03 15:01:49 migration xbzrle cachesize: 67108864 transferred 592545964 
> pages 419113 cachemiss 369404 overflow 30211 deltacachemiss 494
> Sep 03 15:01:50 migration status: active (transferred 2238229988, remaining 
> 13078528), total 2114387968)
>