Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty after they have been sent

[Chunguang Li]

> -----原始邮件-----
> 发件人: "Dr. David Alan Gilbert" <address@hidden>
> 发送时间: 2016年9月26日 星期一
> 收件人: "Chunguang Li" <address@hidden>
> 抄送: address@hidden, address@hidden, address@hidden, address@hidden, 
> address@hidden
> 主题: Re: [Qemu-devel] Migration dirty bitmap: should only mark pages as dirty 
> after they have been sent
> 
> * Chunguang Li (address@hidden) wrote:
> > Hi all!
> > I have some confusion about the dirty bitmap during migration. I have 
> > digged into the code. I figure out that every now and then during 
> > migration, the dirty bitmap will be grabbed from the kernel space through 
> > ioctl(KVM_GET_DIRTY_LOG), and then be used to update qemu's dirty bitmap. 
> > However I think this mechanism leads to resendness of some NON-dirty pages.
> > 
> > Take the first iteration of precopy for instance, during which all the 
> > pages will be sent. Before that during the migration setup, the 
> > ioctl(KVM_GET_DIRTY_LOG) is called once, so the kernel begins to produce 
> > the dirty bitmap from this moment. When the pages "that haven't been sent" 
> > are written, the kernel space marks them as dirty. However I don't think 
> > this is correct, because these pages will be sent during this and the next 
> > iterations with the same content (if they are not written again after they 
> > are sent). It only makes sense to mark the pages which have already been 
> > sent during one iteration as dirty when they are written.
> > 
> > 
> > Am I right about this consideration? If I am right, is there some advice to 
> > improve this?
> 
> I think you're right that this can happen; to clarify I think the
> case you're talking about is:
> 
>   Iteration 1
>     sync bitmap
>     start sending pages
>     page 'n' is modified - but hasn't been sent yet
>     page 'n' gets sent
>   Iteration 2
>     sync bitmap
>        'page n is shown as modified'
>     send page 'n' again
>

Yes，this is right the case I am talking about.
 
> So you're right that is wasteful; I guess it's more wasteful
> on big VMs with slow networks where the length of each iteration
> is large.

I think this is "very" wasteful. Assume the workload writes the pages dirty 
randomly within the guest address space, and the transfer speed is constant. 
Intuitively, I think nearly half of the dirty pages produced in Iteration 1 is 
not really dirty. This means the time of Iteration 2 is double of that to send 
only really dirty pages.

Thanks,

Chunguang

> 
> Fixing it is not easy, because you have to be really careful
> never to miss a page modification, even if the page is sent
> about the same time it's dirtied.
> 
> One way would be to sync the dirty log from the kernel
> in smaller chunks.
> 
> Dave
> 
> 
> > 
> > Thanks,
> > Chunguang Li
> --
> Dr. David Alan Gilbert / address@hidden / Manchester, UK


--
Chunguang Li, Ph.D. Candidate
Wuhan National Laboratory for Optoelectronics (WNLO)
Huazhong University of Science & Technology (HUST)
Wuhan, Hubei Prov., China