Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent

[Chunguang Li]

> -----原始邮件-----
> 发件人: "Amit Shah" <address@hidden>
> 发送时间: 2016年9月30日 星期五
> 收件人: "Chunguang Li" <address@hidden>
> 抄送: "Dr. David Alan Gilbert" <address@hidden>, address@hidden, 
> address@hidden, address@hidden, address@hidden
> 主题: Re: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as 
> dirty after they have been sent
> 
> On (Mon) 26 Sep 2016 [22:55:01], Chunguang Li wrote:
> > 
> > 
> > 
> > > -----原始邮件-----
> > > 发件人: "Dr. David Alan Gilbert" <address@hidden>
> > > 发送时间: 2016年9月26日 星期一
> > > 收件人: "Chunguang Li" <address@hidden>
> > > 抄送: address@hidden, address@hidden, address@hidden, address@hidden, 
> > > address@hidden
> > > 主题: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as 
> > > dirty after they have been sent
> > > 
> > > * Chunguang Li (address@hidden) wrote:
> > > > Hi all!
> > > > I have some confusion about the dirty bitmap during migration. I have 
> > > > digged into the code. I figure out that every now and then during 
> > > > migration, the dirty bitmap will be grabbed from the kernel space 
> > > > through ioctl(KVM_GET_DIRTY_LOG), and then be used to update qemu's 
> > > > dirty bitmap. However I think this mechanism leads to resendness of 
> > > > some NON-dirty pages.
> > > > 
> > > > Take the first iteration of precopy for instance, during which all the 
> > > > pages will be sent. Before that during the migration setup, the 
> > > > ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to 
> > > > produce the dirty bitmap from this moment. When the pages "that haven't 
> > > > been sent" are written, the kernel space marks them as dirty. However I 
> > > > don't think this is correct, because these pages will be sent during 
> > > > this and the next iterations with the same content (if they are not 
> > > > written again after they are sent). It only makes sense to mark the 
> > > > pages which have already been sent during one iteration as dirty when 
> > > > they are written.
> > > > 
> > > > 
> > > > Am I right about this consideration? If I am right, is there some 
> > > > advice to improve this?
> > > 
> > > I think you're right that this can happen; to clarify I think the
> > > case you're talking about is:
> > > 
> > >   Iteration 1
> > >     sync bitmap
> > >     start sending pages
> > >     page 'n' is modified - but hasn't been sent yet
> > >     page 'n' gets sent
> > >   Iteration 2
> > >     sync bitmap
> > >        'page n is shown as modified'
> > >     send page 'n' again
> > >
> > 
> > Yes，this is right the case I am talking about.
> >  
> > > So you're right that is wasteful; I guess it's more wasteful
> > > on big VMs with slow networks where the length of each iteration
> > > is large.
> > 
> > I think this is "very" wasteful. Assume the workload writes the pages dirty 
> > randomly within the guest address space, and the transfer speed is 
> > constant. Intuitively, I think nearly half of the dirty pages produced in 
> > Iteration 1 is not really dirty. This means the time of Iteration 2 is 
> > double of that to send only really dirty pages.
> 
> It makes sense, can you get some perf numbers to show what kinds of
> workloads get impacted the most?  That would also help us to figure
> out what kinds of speed improvements we can expect.
> 
> 
>               Amit

Yes, I can pick up some workloads to get some perf numbers. 
However, I don't know how to get the quantity of non-dirty pages we are
resending in each iteration. Instead, I can  get the numbers below:

1. The time consuming of each iteration;
2. The quantity of pages transferred during each iteration;
3. The quantity of dirty pages (including not-really-dirty pages) produced 
   during each iteration.

With these numbers, we can only estimate the quantity of not-really-dirty
pages to some extent. How do you think of this test plan? Any suggestions?

Chunguang

--
Chunguang Li, Ph.D. Candidate
Wuhan National Laboratory for Optoelectronics (WNLO)
Huazhong University of Science & Technology (HUST)
Wuhan, Hubei Prov., China