Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent

[Chunguang Li]

> -----Original Messages-----
> From: "Li, Liang Z" <address@hidden>
> Sent Time: Friday, November 4, 2016
> To: "Chunguang Li" <address@hidden>
> Cc: "Dr. David Alan Gilbert" <address@hidden>, "Amit Shah" <address@hidden>, 
> "address@hidden" <address@hidden>, "address@hidden" <address@hidden>, 
> "address@hidden" <address@hidden>, "address@hidden" <address@hidden>
> Subject: RE: RE: [Qemu-devel] Migration dirty bitmap: should only mark pages 
> as dirty after they have been sent
> 
> > > > > > > > I think this is "very" wasteful. Assume the workload writes
> > > > > > > > the pages
> > > > dirty randomly within the guest address space, and the transfer
> > > > speed is constant. Intuitively, I think nearly half of the dirty
> > > > pages produced in Iteration 1 is not really dirty. This means the
> > > > time of Iteration 2 is double of that to send only really dirty pages.
> > > > > > >
> > > > > > > It makes sense, can you get some perf numbers to show what
> > > > > > > kinds of workloads get impacted the most?  That would also
> > > > > > > help us to figure out what kinds of speed improvements we can
> > expect.
> > > > > > >
> > > > > > >
> > > > > > >           Amit
> > > > > >
> > > > > > I have picked up 6 workloads and got the following statistics
> > > > > > numbers of every iteration (except the last stop-copy one) during
> > precopy.
> > > > > > These numbers are obtained with the basic precopy migration,
> > > > > > without the capabilities like xbzrle or compression, etc. The
> > > > > > network for the migration is exclusive, with a separate network for
> > the workloads.
> > > > > > They are both gigabit ethernet. I use qemu-2.5.1.
> > > > > >
> > > > > > Three (booting, idle, web server) of them converged to the
> > > > > > stop-copy
> > > > phase,
> > > > > > with the given bandwidth and default downtime (300ms), while the
> > > > > > other three (kernel compilation, zeusmp, memcached) did not.
> > > > > >
> > > > > > One page is "not-really-dirty", if it is written first and is
> > > > > > sent later (and not written again after that) during one
> > > > > > iteration. I guess this would not happen so often during the
> > > > > > other iterations as during the 1st iteration. Because all the
> > > > > > pages of the VM are sent to the dest node
> > > > during
> > > > > > the 1st iteration, while during the others, only part of the pages 
> > > > > > are
> > sent.
> > > > > > So I think the "not-really-dirty" pages should be produced
> > > > > > mainly during the 1st iteration , and maybe very little during the 
> > > > > > other
> > iterations.
> > > > > >
> > > > > > If we could avoid resending the "not-really-dirty" pages,
> > > > > > intuitively, I think the time spent on Iteration 2 would be
> > > > > > halved. This is a chain
> > > > reaction,
> > > > > > because the dirty pages produced during Iteration 2 is halved,
> > > > > > which
> > > > incurs
> > > > > > that the time spent on Iteration 3 is halved, then Iteration 4, 5...
> > > > >
> > > > > Yes; these numbers don't show how many of them are false dirty
> > though.
> > > > >
> > > > > One problem is thinking about pages that have been redirtied, if
> > > > > the page is
> > > > dirtied
> > > > > after the sync but before the network write then it's the
> > > > > false-dirty that you're describing.
> > > > >
> > > > > However, if the page is being written a few times, and so it would
> > > > > have
> > > > been written
> > > > > after the network write then it isn't a false-dirty.
> > > > >
> > > > > You might be able to figure that out with some kernel tracing of
> > > > > when the
> > > > dirtying
> > > > > happens, but it might be easier to write the fix!
> > > > >
> > > > > Dave
> > > >
> > > > Hi, I have made some new progress now.
> > > >
> > > > To tell how many false dirty pages there are exactly in each
> > > > iteration, I malloc a buffer in memory as big as the size of the
> > > > whole VM memory. When a page is transferred to the dest node, it is
> > > > copied to the buffer; During the next iteration, if one page is
> > > > transferred, it is compared to the old one in the buffer, and the
> > > > old one will be replaced for next comparison if it is really dirty.
> > > > Thus, we are now able to get the exact number of false dirty pages.
> > > >
> > > > This time, I use 15 workloads to get the statistic number. They are:
> > > >
> > > >   1. 11 benchmarks picked up from cpu2006 benchmark suit. They are
> > > > all scientific
> > > >      computing workloads like Quantum Chromodynamics, Fluid Dynamics,
> > etc.
> > > > I pick
> > > >      up these 11 benchmarks because compared to others, they have
> > > > bigger memory
> > > >      occupation and higher memory dirty rate. Thus most of them
> > > > could not converge
> > > >      to stop-and-copy using the default migration speed (32MB/s).
> > > >   2. kernel compilation
> > > >   3. idle VM
> > > >   4. Apache web server which serves static content
> > > >
> > > >   (the above workloads are all running in VM with 1 vcpu and 1GB
> > > > memory, and the
> > > >    migration speed is the default 32MB/s)
> > > >
> > > >   5. Memcached. The VM has 6 cpu cores and 6GB memory, and 4GB are
> > > > used as the cache.
> > > >      After filling up the 4GB cache, a client writes the cache at a 
> > > > constant
> > speed
> > > >      during migration. This time, migration speed has no limit, and is 
> > > > up to
> > the
> > > >      capability of 1Gbps Ethernet.
> > > >
> > > > Summarize the results first: (and you can read the precise number
> > > > below)
> > > >
> > > >   1. 4 of these 15 workloads have a big proportion (>60%, even >80%
> > > > during some iterations)
> > > >      of false dirty pages out of all the dirty pages since iteration 2 
> > > > (and the
> > big
> > > >      proportion lasts during the following iterations). They are
> > cpu2006.zeusmp,
> > > >      cpu2006.bzip2, cpu2006.mcf, and memcached.
> > > >   2. 2 workloads (idle, webserver) spend most of the migration time
> > > > on iteration 1, even
> > > >      though the proportion of false dirty pages is big since
> > > > iteration 2, the space to
> > > >      optimize is small.
> > > >   3. 1 workload (kernel compilation) only have a big proportion
> > > > during iteration 2, not
> > > >      in the other iterations.
> > > >   4. 8 workloads (the other 8 benchmarks of cpu2006) have little
> > > > proportion of false
> > > >      dirty pages since iteration 2. So the spaces to optimize for them 
> > > > are
> > small.
> > > >
> > > > Now I want to talk a little more about the reasons why false dirty
> > > > pages are produced.
> > > > The first reason is what we have discussed before---the mechanism to
> > > > track the dirty pages.
> > > > And then I come up with another reason. Here is the situation: a
> > > > write operation to one memory page happens, but it doesn't change
> > > > any content of the page. So it's "write but not dirty", and kernel
> > > > still marks it as dirty. One guy in our lab has done some
> > > > experiments to figure out the proportion of "write but not dirty"
> > > > operations, and he uses the cpu2006 benchmark suit. According to his
> > > > results, general workloads has a little proportion (<10%) of "write
> > > > but not dirty" out of all the write operations, while few workloads
> > > > have higher proportion (one even as high as 50%). Now we are not
> > > > sure why "write but not dirty" would happen, it just happened.
> > > >
> > > > So these two reasons contribute to the false dirty pages. To
> > > > optimize, I compute and store the SHA1 hash before transferring each
> > > > page. Next time, if one page needs retransmission, its
> > > > SHA1 hash is computed again, and compared to the old hash. If the
> > > > hash is the same, it's a false dirty page, and we just skip this
> > > > page; Otherwise, the page is transferred, and the new hash replaces
> > > > the old one for next comparison.
> > > > The reason to use SHA1 hash but not byte-by-byte comparison is the
> > > > memory overheads. One SHA1 hash is 20 bytes. So we need extra
> > > > 20/4096 (<1/200) memory space of the whole VM memory, which is
> > > > relatively small.
> > > > As far as I know, SHA1 hash is widely used in the scenes of
> > > > deduplication for backup systems.
> > > > They have proven that the probability of hash collision is far
> > > > smaller than disk hardware fault, so it's secure hash, that is, if
> > > > the hashes of two chunks are the same, the content must be the same.
> > > > So I think the SHA1 hash could replace byte-to-byte comparison in
> > > > the VM memory scenery.
> > > >
> > > > Then I do the same migration experiments using the SHA1 hash. For
> > > > the 4 workloads which have big proportions of false dirty pages, the
> > > > improvement is remarkable. Without optimization, they either can not
> > > > converge to stop-and-copy, or take a very long time to complete.
> > > > With the
> > > > SHA1 hash method, all of them now complete in a relatively short time.
> > > > For the reason I have talked above, the other workloads don't get
> > > > notable improvements from the optimization. So below, I only show
> > > > the exact number after optimization for the 4 workloads with
> > > > remarkable improvements.
> > > >
> > > > Any comments or suggestions?
> > >
> > > Maybe you can compare the performance of your solution as that of
> > XBZRLE to see which one is better.
> > > The merit of using SHA1 is that it can avoid data copy as that in XBZRLE, 
> > > and
> > need less buffer.
> > > How about the overhead of calculating the SHA1? Is it faster than copying 
> > > a
> > page?
> > >
> > > Liang
> > >
> > >
> > 
> > Yes, XBZRLE is able to handle the false dirty pages. However, if we want to
> > avoid transferring all of the false dirty pages using XBZRLE, we need a 
> > buffer
> > as big as the whole VM memory, while SHA1 needs a much small buffer. Of
> > course, if we have a buffer as big as the whole VM memory using XBZRLE, we
> > could transfer less data on network than SHA1, because XBZRLE is able to
> > compress similar pages. In a word, yes, the merit of using SHA1 is that it
> > needs much less buffer, and leads to nice improvement if there are many
> > false dirty pages.
> > 
> 
> The current implementation of XBZRLE begins to buffer page from the second 
> iteration,
> Maybe it's worth to make it start to work from the first iteration based on 
> your finding.
> 
> > In terms of the overhead of calculating the SHA1 compared with transferring
> > a page, it's related to the CPU and network performance. In my test
> > environment(Intel Xeon
> > E5620 @2.4GHz, 1Gbps Ethernet), I didn't observe obvious extra computing
> > overhead caused by calculating the SHA1, because the throughput of
> > network (got by "info migrate") remains almost the same.
> 
> You can check the CPU usage, or to measure the time spend on a local live 
> migration
>  which use SHA1/ XBZRLE.
> 
> Liang
> 
> 

I compare SHA1 with XBZRLE. I use XBZRLE in two ways: 
1. Begins to buffer pages from iteration 1;
2. As current implementation, begins to buffer pages from iteration 2.

I post the results of three workloads: cpu2006.zeusmp, cpu2006.mcf, memcached. 
I set the cache size as 256MB for zeusmp & mcf (they run in VM with 1GB ram),
and set the cache size as 1GB for memcached (it run in VM with 6GB ram, and 
memcached takes 4GB as cache).

As you can read from the data below, beginning to buffer pages from iteration 1
is better than the current implementation(from iteration 2), because the total
migration time is shorter.

SHA1 is better than the XBZRLE with the cache size I choose, because it leads 
to shorter
migration time, and consumes far less memory overhead (<1/200 of the total VM 
memory).


1. zeusmp
(1) XBZRLE 256MB cache, begins to buffer pages from iteration 1
Iteration   1, duration:  21402 ms , transferred pages:   266450 (dup:    
91456, n:   174994, x:        0) , new dirty pages:   129225 , remaining dirty 
pages:   129225
Iteration   2, duration:   2295 ms , transferred pages:   101471 (dup:    
77921, n:    16665, x:     6885) , new dirty pages:    76125 , remaining dirty 
pages:    77424
Iteration   3, duration:   2000 ms , transferred pages:    56092 (dup:    
36345, n:    11249, x:     8498) , new dirty pages:   111498 , remaining dirty 
pages:   112836
Iteration   4, duration:   1604 ms , transferred pages:    87335 (dup:    
69441, n:    10018, x:     7876) , new dirty pages:    19982 , remaining dirty 
pages:    19982
Iteration   5, duration:    302 ms , transferred pages:    19850 (dup:    
16718, n:     2547, x:      585) , new dirty pages:    14084 , remaining dirty 
pages:    14084
Iteration   6, duration:    194 ms , transferred pages:    13403 (dup:    
12338, n:      846, x:      219) , new dirty pages:     3900 , remaining dirty 
pages:     4243
Iteration   7, duration:      8 ms , transferred pages:     3938 (dup:     
3425, n:      239, x:      274) , new dirty pages:      372 , remaining dirty 
pages:      372
Iteration   8, duration:     71 ms , transferred pages:        0 (dup:        
0, n:        0, x:        0) , new dirty pages:        0 , remaining dirty 
pages:      372
total time: 27891 milliseconds

(2) XBZRLE 256MB cache, begins to buffer pages from iteration 2
    can not converge
Iteration   1, duration:  21698 ms , transferred pages:   266331 (dup:    
89009, n:   177322, x:        0) , new dirty pages:   125990 , remaining dirty 
pages:   126109
Iteration   2, duration:   5909 ms , transferred pages:   126109 (dup:    
77248, n:    48861, x:        0) , new dirty pages:   124870 , remaining dirty 
pages:   124870
Iteration   3, duration:   3197 ms , transferred pages:   110583 (dup:    
75471, n:    23129, x:    11983) , new dirty pages:   118035 , remaining dirty 
pages:   118035
Iteration   4, duration:   3195 ms , transferred pages:   102787 (dup:    
72708, n:    22158, x:     7921) , new dirty pages:    86576 , remaining dirty 
pages:    86773
Iteration   5, duration:   3111 ms , transferred pages:    79563 (dup:    
52073, n:    21289, x:     6201) , new dirty pages:    97402 , remaining dirty 
pages:    97402
Iteration   6, duration:   2407 ms , transferred pages:    79567 (dup:    
56415, n:    16013, x:     7139) , new dirty pages:   101193 , remaining dirty 
pages:   101193
Iteration   7, duration:   2896 ms , transferred pages:    83278 (dup:    
55778, n:    20652, x:     6848) , new dirty pages:    90683 , remaining dirty 
pages:    92977
Iteration   8, duration:   2701 ms , transferred pages:    89112 (dup:    
62579, n:    18699, x:     7834) , new dirty pages:   109827 , remaining dirty 
pages:   110008
Iteration   9, duration:   3602 ms , transferred pages:    95866 (dup:    
61631, n:    25632, x:     8603) , new dirty pages:    94551 , remaining dirty 
pages:    96227
Iteration  10, duration:   3802 ms , transferred pages:    83693 (dup:    
50558, n:    26427, x:     6708) , new dirty pages:   123537 , remaining dirty 
pages:   124170
Iteration  11, duration:   3399 ms , transferred pages:   108770 (dup:    
75144, n:    23952, x:     9674) , new dirty pages:   103934 , remaining dirty 
pages:   104981
Iteration  12, duration:   2700 ms , transferred pages:    91080 (dup:    
62981, n:    16600, x:    11499) , new dirty pages:    88314 , remaining dirty 
pages:    88948
Iteration  13, duration:   3102 ms , transferred pages:    78406 (dup:    
50165, n:    21409, x:     6832) , new dirty pages:    73586 , remaining dirty 
pages:    74025
Iteration  14, duration:    806 ms , transferred pages:    66530 (dup:    
51013, n:     3973, x:    11544) , new dirty pages:    67941 , remaining dirty 
pages:    67941
Iteration  15, duration:   2398 ms , transferred pages:    53117 (dup:    
33312, n:    18436, x:     1369) , new dirty pages:   116502 , remaining dirty 
pages:   118956
Iteration  16, duration:   3200 ms , transferred pages:   103009 (dup:    
71642, n:    21378, x:     9989) , new dirty pages:    81777 , remaining dirty 
pages:    83724
Iteration  17, duration:   3005 ms , transferred pages:    73096 (dup:    
45738, n:    19016, x:     8342) , new dirty pages:   116671 , remaining dirty 
pages:   118397
Iteration  18, duration:   3302 ms , transferred pages:   101507 (dup:    
67290, n:    22721, x:    11496) , new dirty pages:   104163 , remaining dirty 
pages:   105921
Iteration  19, duration:   3705 ms , transferred pages:    90516 (dup:    
56932, n:    26394, x:     7190) , new dirty pages:   118139 , remaining dirty 
pages:   120170
Iteration  20, duration:   3903 ms , transferred pages:   102710 (dup:    
67623, n:    25811, x:     9276) , new dirty pages:   103608 , remaining dirty 
pages:   105496

(3) SHA1
Iteration   1, duration:  21601 ms , transferred pages:   266450 (dup:    
89731, rd:   176719) , new dirty pages:   139843 , remaining dirty pages:   
139843
Iteration   2, duration:   1747 ms , transferred pages:    92077 (dup:    
78364, rd:    13713) , new dirty pages:    90945 , remaining dirty pages:    
90945
Iteration   3, duration:   1592 ms , transferred pages:    62253 (dup:    
49435, rd:    12818) , new dirty pages:    76929 , remaining dirty pages:    
76929
Iteration   4, duration:    992 ms , transferred pages:    44837 (dup:    
37886, rd:     6951) , new dirty pages:    71331 , remaining dirty pages:    
72916
Iteration   5, duration:    998 ms , transferred pages:    55229 (dup:    
47150, rd:     8079) , new dirty pages:    21703 , remaining dirty pages:    
23302
Iteration   6, duration:    211 ms , transferred pages:    20337 (dup:    
18516, rd:     1821) , new dirty pages:    14500 , remaining dirty pages:    
14500
Iteration   7, duration:     31 ms , transferred pages:    12933 (dup:    
12627, rd:      306) , new dirty pages:     1520 , remaining dirty pages:     
1520
Iteration   8, duration:     30 ms , transferred pages:        0 (dup:        
0, rd:        0) , new dirty pages:        4 , remaining dirty pages:     1524
total time: 27225 milliseconds

2. mcf
(1) XBZRLE 256MB cache, begins to buffer pages from iteration 1
Iteration   1, duration:  31706 ms , transferred pages:   266325 (dup:     
7032, n:   259293, x:        0) , new dirty pages:   238215 , remaining dirty 
pages:   238340
Iteration   2, duration:  21807 ms , transferred pages:   186619 (dup:      
335, n:   176826, x:     9458) , new dirty pages:   226886 , remaining dirty 
pages:   228857
Iteration   3, duration:  21300 ms , transferred pages:   181925 (dup:      
201, n:   172974, x:     8750) , new dirty pages:   202288 , remaining dirty 
pages:   204100
Iteration   4, duration:  17300 ms , transferred pages:   148972 (dup:       
38, n:   141113, x:     7821) , new dirty pages:   136220 , remaining dirty 
pages:   137992
Iteration   5, duration:  13699 ms , transferred pages:   118247 (dup:       
38, n:   112030, x:     6179) , new dirty pages:    48397 , remaining dirty 
pages:    50466
Iteration   6, duration:   4499 ms , transferred pages:    41719 (dup:       
24, n:    36790, x:     4905) , new dirty pages:     3753 , remaining dirty 
pages:     5690
Iteration   7, duration:    399 ms , transferred pages:     3826 (dup:        
4, n:     3265, x:      557) , new dirty pages:     1261 , remaining dirty 
pages:     2437
Iteration   8, duration:     72 ms , transferred pages:        0 (dup:        
0, n:        0, x:        0) , new dirty pages:        0 , remaining dirty 
pages:     2437
total time: 110812 milliseconds

(2) XBZRLE 256MB cache, begins to buffer pages from iteration 2
Iteration   1, duration:  31606 ms , transferred pages:   266450 (dup:     
7267, n:   259183, x:        0) , new dirty pages:   233582 , remaining dirty 
pages:   233582
Iteration   2, duration:  28413 ms , transferred pages:   231693 (dup:       
89, n:   231604, x:        0) , new dirty pages:   216962 , remaining dirty 
pages:   218851
Iteration   3, duration:  18618 ms , transferred pages:   159936 (dup:        
3, n:   151579, x:     8354) , new dirty pages:   216400 , remaining dirty 
pages:   218790
Iteration   4, duration:  18621 ms , transferred pages:   159665 (dup:        
0, n:   152102, x:     7563) , new dirty pages:   209860 , remaining dirty 
pages:   211611
Iteration   5, duration:  17709 ms , transferred pages:   151672 (dup:        
4, n:   144493, x:     7175) , new dirty pages:   146273 , remaining dirty 
pages:   148006
Iteration   6, duration:   9911 ms , transferred pages:    86971 (dup:        
2, n:    80842, x:     6127) , new dirty pages:   118364 , remaining dirty 
pages:   120396
Iteration   7, duration:  14212 ms , transferred pages:   117460 (dup:        
0, n:   116149, x:     1311) , new dirty pages:   213993 , remaining dirty 
pages:   216107
Iteration   8, duration:  22913 ms , transferred pages:   213698 (dup:        
4, n:   161520, x:    52174) , new dirty pages:   217947 , remaining dirty 
pages:   219955
Iteration   9, duration:  23808 ms , transferred pages:   217375 (dup:        
3, n:   152315, x:    65057) , new dirty pages:   172615 , remaining dirty 
pages:   174859
Iteration  10, duration:  15099 ms , transferred pages:   131265 (dup:        
0, n:   123463, x:     7802) , new dirty pages:   113946 , remaining dirty 
pages:   116026
Iteration  11, duration:  10002 ms , transferred pages:    88477 (dup:        
8, n:    81753, x:     6716) , new dirty pages:    97006 , remaining dirty 
pages:    99110
Iteration  12, duration:   6898 ms , transferred pages:    62861 (dup:        
4, n:    56392, x:     6465) , new dirty pages:    45164 , remaining dirty 
pages:    47297
Iteration  13, duration:   3601 ms , transferred pages:    35360 (dup:        
0, n:    29390, x:     5970) , new dirty pages:    24581 , remaining dirty 
pages:    26779
Iteration  14, duration:   1902 ms , transferred pages:    19794 (dup:        
0, n:    15475, x:     4319) , new dirty pages:    66153 , remaining dirty 
pages:    67850
Iteration  15, duration:   5504 ms , transferred pages:    50369 (dup:        
0, n:    44902, x:     5467) , new dirty pages:    49279 , remaining dirty 
pages:    51198
Iteration  16, duration:   3699 ms , transferred pages:    36519 (dup:        
2, n:    30184, x:     6333) , new dirty pages:    23672 , remaining dirty 
pages:    25914
Iteration  17, duration:   1601 ms , transferred pages:    17628 (dup:        
0, n:    12972, x:     4656) , new dirty pages:     8685 , remaining dirty 
pages:    10646
Iteration  18, duration:    599 ms , transferred pages:     7835 (dup:        
0, n:     4825, x:     3010) , new dirty pages:     6167 , remaining dirty 
pages:     6266
Iteration  19, duration:    200 ms , transferred pages:     3590 (dup:        
0, n:     1576, x:     2014) , new dirty pages:     3873 , remaining dirty 
pages:     5709
Iteration  20, duration:    200 ms , transferred pages:     4134 (dup:        
0, n:     1574, x:     2560) , new dirty pages:     3609 , remaining dirty 
pages:     4099
Iteration  21, duration:    100 ms , transferred pages:     2785 (dup:        
0, n:      787, x:     1998) , new dirty pages:     1900 , remaining dirty 
pages:     2585
Iteration  22, duration:      9 ms , transferred pages:     2191 (dup:        
0, n:      539, x:     1652) , new dirty pages:      596 , remaining dirty 
pages:      596
Iteration  23, duration:     41 ms , transferred pages:        0 (dup:        
0, n:        0, x:        0) , new dirty pages:        0 , remaining dirty 
pages:      596
total time: 235286 milliseconds

(3) SHA1
Iteration   1, duration:  31711 ms , transferred pages:   266450 (dup:     
6831, rd:   259619) , new dirty pages:   240209 , remaining dirty pages:   
240209
Iteration   2, duration:   6250 ms , transferred pages:    51244 (dup:      
211, rd:    51033) , new dirty pages:   226651 , remaining dirty pages:   228571
Iteration   3, duration:   4395 ms , transferred pages:    36008 (dup:       
80, rd:    35928) , new dirty pages:   110719 , remaining dirty pages:   111478
Iteration   4, duration:   3390 ms , transferred pages:    28068 (dup:       
28, rd:    28040) , new dirty pages:   185172 , remaining dirty pages:   185172
Iteration   5, duration:   2986 ms , transferred pages:    23780 (dup:       
45, rd:    23735) , new dirty pages:    64357 , remaining dirty pages:    66305
Iteration   6, duration:   2727 ms , transferred pages:    22800 (dup:       
12, rd:    22788) , new dirty pages:    61675 , remaining dirty pages:    61675
Iteration   7, duration:   2372 ms , transferred pages:    18943 (dup:       
13, rd:    18930) , new dirty pages:    55144 , remaining dirty pages:    55265
Iteration   8, duration:   2100 ms , transferred pages:    17189 (dup:       
11, rd:    17178) , new dirty pages:    55244 , remaining dirty pages:    55668
Iteration   9, duration:   2003 ms , transferred pages:    16371 (dup:       
11, rd:    16360) , new dirty pages:   107058 , remaining dirty pages:   108014
Iteration  10, duration:   2132 ms , transferred pages:    17825 (dup:       
24, rd:    17801) , new dirty pages:   126214 , remaining dirty pages:   126214
Iteration  11, duration:   2229 ms , transferred pages:    18156 (dup:       
22, rd:    18134) , new dirty pages:    65725 , remaining dirty pages:    65725
Iteration  12, duration:   2315 ms , transferred pages:    18651 (dup:       
21, rd:    18630) , new dirty pages:    52575 , remaining dirty pages:    53903
Iteration  13, duration:   2147 ms , transferred pages:    17435 (dup:       
16, rd:    17419) , new dirty pages:    46652 , remaining dirty pages:    47260
Iteration  14, duration:   2000 ms , transferred pages:    16371 (dup:       
11, rd:    16360) , new dirty pages:    42721 , remaining dirty pages:    43266
Iteration  15, duration:   1901 ms , transferred pages:    15552 (dup:       
10, rd:    15542) , new dirty pages:    38593 , remaining dirty pages:    40792
Iteration  16, duration:   1801 ms , transferred pages:    14735 (dup:       
11, rd:    14724) , new dirty pages:    54252 , remaining dirty pages:    55639
Iteration  17, duration:   1708 ms , transferred pages:    13860 (dup:        
2, rd:    13858) , new dirty pages:    72379 , remaining dirty pages:    74170
Iteration  18, duration:   1923 ms , transferred pages:    15442 (dup:       
12, rd:    15430) , new dirty pages:   101911 , remaining dirty pages:   103547
Iteration  19, duration:   2311 ms , transferred pages:    18823 (dup:        
9, rd:    18814) , new dirty pages:    80534 , remaining dirty pages:    82521
Iteration  20, duration:   2081 ms , transferred pages:    17156 (dup:       
34, rd:    17122) , new dirty pages:    36054 , remaining dirty pages:    36054
Iteration  21, duration:   1665 ms , transferred pages:    13777 (dup:       
10, rd:    13767) , new dirty pages:    29624 , remaining dirty pages:    29624
Iteration  22, duration:   1657 ms , transferred pages:    13290 (dup:        
7, rd:    13283) , new dirty pages:    25949 , remaining dirty pages:    28265
Iteration  23, duration:   1599 ms , transferred pages:    13088 (dup:        
0, rd:    13088) , new dirty pages:    22356 , remaining dirty pages:    24813
Iteration  24, duration:   1500 ms , transferred pages:    12280 (dup:       
10, rd:    12270) , new dirty pages:    21181 , remaining dirty pages:    22608
Iteration  25, duration:   1400 ms , transferred pages:    11457 (dup:        
5, rd:    11452) , new dirty pages:    18657 , remaining dirty pages:    20311
Iteration  26, duration:   1200 ms , transferred pages:     9822 (dup:        
6, rd:     9816) , new dirty pages:    15690 , remaining dirty pages:    17294
Iteration  27, duration:   1201 ms , transferred pages:     9822 (dup:        
6, rd:     9816) , new dirty pages:    14810 , remaining dirty pages:    15936
Iteration  28, duration:   1000 ms , transferred pages:     8183 (dup:        
3, rd:     8180) , new dirty pages:    15387 , remaining dirty pages:    16423
Iteration  29, duration:    900 ms , transferred pages:     7372 (dup:       
10, rd:     7362) , new dirty pages:    13303 , remaining dirty pages:    15292
Iteration  30, duration:   1000 ms , transferred pages:     8181 (dup:        
1, rd:     8180) , new dirty pages:    17879 , remaining dirty pages:    18457
Iteration  31, duration:    951 ms , transferred pages:     8140 (dup:        
9, rd:     8131) , new dirty pages:    21738 , remaining dirty pages:    23304
Iteration  32, duration:    946 ms , transferred pages:     6946 (dup:        
1, rd:     6945) , new dirty pages:    15815 , remaining dirty pages:    15815
Iteration  33, duration:    747 ms , transferred pages:     6192 (dup:        
0, rd:     6192) , new dirty pages:     6249 , remaining dirty pages:     7670
Iteration  34, duration:    501 ms , transferred pages:     4090 (dup:        
0, rd:     4090) , new dirty pages:     6163 , remaining dirty pages:     8422
Iteration  35, duration:    600 ms , transferred pages:     4910 (dup:        
2, rd:     4908) , new dirty pages:     3673 , remaining dirty pages:     5222
Iteration  36, duration:    300 ms , transferred pages:     2454 (dup:        
0, rd:     2454) , new dirty pages:     2132 , remaining dirty pages:     4337
Iteration  37, duration:    200 ms , transferred pages:     1637 (dup:        
1, rd:     1636) , new dirty pages:      544 , remaining dirty pages:     2251
Iteration  38, duration:      0 ms , transferred pages:        0 (dup:        
0, rd:        0) , new dirty pages:        0 , remaining dirty pages:     2251
total time: 97919 milliseconds

3. memcached
(1) XBZRLE 1024MB cache, begins to buffer pages from iteration 1
Iteration   1, duration:  40763 ms , transferred pages:  1570149 (dup:   
404139, n:  1166010, x:        0) , new dirty pages:   526462 , remaining dirty 
pages:   533483
Iteration   2, duration:  15741 ms , transferred pages:   461867 (dup:     
4070, n:   437704, x:    20093) , new dirty pages:   256841 , remaining dirty 
pages:   265501
Iteration   3, duration:   7874 ms , transferred pages:   231950 (dup:      
280, n:   207569, x:    24101) , new dirty pages:   153526 , remaining dirty 
pages:   160865
Iteration   4, duration:   4260 ms , transferred pages:   135181 (dup:      
135, n:   116768, x:    18278) , new dirty pages:   100298 , remaining dirty 
pages:   107278
Iteration   5, duration:   2506 ms , transferred pages:    87596 (dup:      
180, n:    67600, x:    19816) , new dirty pages:    63685 , remaining dirty 
pages:    71790
Iteration   6, duration:   1373 ms , transferred pages:    51800 (dup:      
128, n:    37336, x:    14336) , new dirty pages:    38785 , remaining dirty 
pages:    46064
Iteration   7, duration:    872 ms , transferred pages:    32015 (dup:       
56, n:    23414, x:     8545) , new dirty pages:    23580 , remaining dirty 
pages:    31629
Iteration   8, duration:    527 ms , transferred pages:    21833 (dup:       
40, n:    14372, x:     7421) , new dirty pages:    16624 , remaining dirty 
pages:    23482
Iteration   9, duration:    291 ms , transferred pages:    14917 (dup:       
16, n:     6572, x:     8329) , new dirty pages:    10039 , remaining dirty 
pages:    16753
Iteration  10, duration:    113 ms , transferred pages:     6082 (dup:      
111, n:     3300, x:     2671) , new dirty pages:     4081 , remaining dirty 
pages:    12703
Iteration  11, duration:    119 ms , transferred pages:     3970 (dup:       
16, n:     2953, x:     1001) , new dirty pages:     3824 , remaining dirty 
pages:    11936
Iteration  12, duration:     51 ms , transferred pages:     3585 (dup:        
0, n:     1154, x:     2431) , new dirty pages:     1711 , remaining dirty 
pages:     9900
Iteration  13, duration:     62 ms , transferred pages:     2945 (dup:        
0, n:     1589, x:     1356) , new dirty pages:     1909 , remaining dirty 
pages:     8503
Iteration  14, duration:      2 ms , transferred pages:        0 (dup:        
0, n:        0, x:        0) , new dirty pages:        1 , remaining dirty 
pages:     8504
total time: 74738 milliseconds

(2) XBZRLE 1024MB cache, begins to buffer pages from iteration 2
Iteration   1, duration:  40375 ms , transferred pages:  1570347 (dup:   
415923, n:  1154424, x:        0) , new dirty pages:   511859 , remaining dirty 
pages:   518682
Iteration   2, duration:  17580 ms , transferred pages:   510145 (dup:     
5970, n:   504175, x:        0) , new dirty pages:   291686 , remaining dirty 
pages:   300223
Iteration   3, duration:   8259 ms , transferred pages:   253656 (dup:      
929, n:   230020, x:    22707) , new dirty pages:   166721 , remaining dirty 
pages:   174231
Iteration   4, duration:   4733 ms , transferred pages:   147925 (dup:      
257, n:   132454, x:    15214) , new dirty pages:   103965 , remaining dirty 
pages:   111436
Iteration   5, duration:   2587 ms , transferred pages:    90734 (dup:      
251, n:    70008, x:    20475) , new dirty pages:    61266 , remaining dirty 
pages:    69202
Iteration   6, duration:   1377 ms , transferred pages:    51416 (dup:       
55, n:    37776, x:    13585) , new dirty pages:    45236 , remaining dirty 
pages:    52106
Iteration   7, duration:   1126 ms , transferred pages:    40020 (dup:      
259, n:    30064, x:     9697) , new dirty pages:    28433 , remaining dirty 
pages:    35358
Iteration   8, duration:    574 ms , transferred pages:    23754 (dup:       
40, n:    16066, x:     7648) , new dirty pages:    18067 , remaining dirty 
pages:    26353
Iteration   9, duration:    395 ms , transferred pages:    17607 (dup:       
16, n:     9463, x:     8128) , new dirty pages:    11507 , remaining dirty 
pages:    18488
Iteration  10, duration:    171 ms , transferred pages:     8195 (dup:       
40, n:     4726, x:     3429) , new dirty pages:     5482 , remaining dirty 
pages:    13898
Iteration  11, duration:    116 ms , transferred pages:     6594 (dup:       
16, n:     2679, x:     3899) , new dirty pages:     3884 , remaining dirty 
pages:     9581
Iteration  12, duration:     54 ms , transferred pages:     1793 (dup:        
0, n:     1634, x:      159) , new dirty pages:     1515 , remaining dirty 
pages:     9189
Iteration  13, duration:     62 ms , transferred pages:     1793 (dup:        
0, n:     1643, x:      150) , new dirty pages:     1657 , remaining dirty 
pages:     8871
Iteration  14, duration:      3 ms , transferred pages:        0 (dup:        
0, n:        0, x:        0) , new dirty pages:        1 , remaining dirty 
pages:     8872
total time: 77578 milliseconds

(3) SHA1
Iteration   1, duration:  40664 ms , transferred pages:  1569037 (dup:   
405940, rd:  1163097) , new dirty pages:   506846 , remaining dirty pages:   
514979
Iteration   2, duration:   8032 ms , transferred pages:   161130 (dup:     
4007, rd:   157123) , new dirty pages:   153479 , remaining dirty pages:   
153479
Iteration   3, duration:   2620 ms , transferred pages:    65260 (dup:       
20, rd:    65240) , new dirty pages:    64014 , remaining dirty pages:    67100
Iteration   4, duration:   1160 ms , transferred pages:    30227 (dup:       
60, rd:    30167) , new dirty pages:    34031 , remaining dirty pages:    41414
Iteration   5, duration:    648 ms , transferred pages:    18700 (dup:       
56, rd:    18644) , new dirty pages:    18375 , remaining dirty pages:    25536
Iteration   6, duration:    389 ms , transferred pages:    11399 (dup:       
55, rd:    11344) , new dirty pages:    12536 , remaining dirty pages:    17516
Iteration   7, duration:    292 ms , transferred pages:     8197 (dup:        
0, rd:     8197) , new dirty pages:     8387 , remaining dirty pages:    16802
Iteration   8, duration:    171 ms , transferred pages:     4931 (dup:       
39, rd:     4892) , new dirty pages:     6182 , remaining dirty pages:    14060
Iteration   9, duration:    163 ms , transferred pages:     4355 (dup:       
16, rd:     4339) , new dirty pages:     5530 , remaining dirty pages:    11973
Iteration  10, duration:    104 ms , transferred pages:     3266 (dup:        
0, rd:     3266) , new dirty pages:     2893 , remaining dirty pages:    11014
Iteration  11, duration:     52 ms , transferred pages:     1153 (dup:        
0, rd:     1153) , new dirty pages:     1586 , remaining dirty pages:    10516
Iteration  12, duration:     52 ms , transferred pages:     1921 (dup:       
39, rd:     1882) , new dirty pages:     1619 , remaining dirty pages:     8842
Iteration  13, duration:     62 ms , transferred pages:     1537 (dup:        
0, rd:     1537) , new dirty pages:     2052 , remaining dirty pages:     8871
Iteration  14, duration:     58 ms , transferred pages:     1665 (dup:        
0, rd:     1665) , new dirty pages:     1947 , remaining dirty pages:     7989
Iteration  15, duration:      2 ms , transferred pages:        0 (dup:        
0, rd:        0) , new dirty pages:        0 , remaining dirty pages:     7989
total time: 54693 milliseconds