Re: [Qemu-devel] QEMU MicroCheckpointing Pause & Resume Latency

[Paolo Bonzini]

On 09/03/2017 18:06, Dr. David Alan Gilbert wrote:
> (cc'ing in Paolo since he knows our barrier code)
> 
> * FENG, Jiasheng (address@hidden) wrote:
>> Dear David,
>>
>> Really appreciate your feedback.
>>
>> I have proceeded the experiments in both conditions, and no matter the
>> vCPUs are in idle or busy situation, there is no difference that smp_wmb()
>> will consume a lot of time to proceed its work.
>>
>> In your opinion, may I know that what is the alternative way to minimize
>> the time consumption of smp_wmb() or any other system setting could speed
>> up smp_wmb()?
>>
>> Thanks in advance for your assistance and hope to receive your feedback soon
> 
> Just checking, is this on a normal x86 PC?
> Your numbers of 3-5ms just seem quite high to me but I've not tried timing 
> that
> code.

smp_wmb does not produce a single machine instruction, so this is
probably a fluke in the profiling tool.

The most expensive part of vm_stop_force_state is going to be
bdrv_drain_all/bdrv_flush_all.  bdrv_flush_all is definitely not needed
for checkpointing purposes.

Paolo

>>>
>>>> Please kindly refer to migration/checkpoint.c file, in function
>>>> capture_checkpoint, I proceeded a test to see the time consumption
>>> between
>>>> vm_stop_force_state and vm_start. I found out that even if the system is
>>>> idle, there are still 12-20ms latency recorded ( mem=2G, vCPU=4 ).
>>>> Moreover, latency will be increased while more cpus equipped by my
>>> virtual
>>>> machine. I have done some research on that and I realized that it is
>>>> related to the Memory Barrier in KVM kernel. Each cpu will proceed a
>>>> smp_wmb() request during pause & resume and it takes about  3-5ms to
>>> finish
>>>> the request ( mem=2G, vCPU=4 ).
>>>>
>>>>
>>>>
>>>> Therefore, I would like to ask 3 questions regarding on the above issue:
>>>>
>>>>
>>>> 1. What is your consideration with calling smp_wmb() in checkpoint
>>> period;
>>>>
>>>> 2. Is it any other solution to minimize the latency to improve the
>>>> performance in checkpoint period;
>>>>
>>>> 3. Is smp_wmb() able to be safely disabled during the checkpoint period
>>>
>>> Well you'd have to understand where it's used; but for example, when taking
>>> a checkpoint you'd want to be sure that the checkpoint data contained
>>> a consistent copy of the last write data from all of the vCPUs; so I think
>>> a wmb would be needed to make sure it's consistent.
>>>
>>> I'm surprised that the smp_wmb is such a big chunk of your total checkpoint
>>> time, and that it's quite so long.
>>> Are the vCPUs idle or are they busy - does it make difference?
>>>
>>> Dave
>>>
>>>> Really appreciate your help with my problems and hope to receive your
>>>> feedback soon.
>>>>
>>>>
>>>> Thanks again for your contribution to QEMU and it is such a masterpiece.
>>>
>>> Dave
>>>
>>>>
>>>>
>>>>
>>>> Thanks and best regards,
>>>>
>>>> Niko Jiasheng Feng
>>>>
>>>> University of Hong Kong
>>>>
>>>> --
>>>> *Niko Jiasheng *
>>>> *Feng **Computer Science(General Stream), Faculty of Engineering, The
>>>> University of Hong Kong*
>>>> Contact:  （852）97908620
>>>> Address: Pokfulam Road, The University of Hong Kong
>>>> Email:      address@hidden / address@hidden
>>> --
>>> Dr. David Alan Gilbert / address@hidden / Manchester, UK
>>>
>>
>>
>>
>> -- 
>> *Niko Jiasheng *
>> *Feng **Computer Science(General Stream), Faculty of Engineering, The
>> University of Hong Kong*
>> Contact:  （852）97908620
>> Address: Pokfulam Road, The University of Hong Kong
>> Email:      address@hidden / address@hidden
> --
> Dr. David Alan Gilbert / address@hidden / Manchester, UK
>