|
From: | YoungJoon Lee |
Subject: | [Qemu-devel] I have a question about qemu’s li ve migration function. |
Date: | Sat, 22 Dec 2012 21:31:22 +0900 |
Hello, mailing lists.
I have a question about qemu’s live migration function.
First, Let me introduce my source versions. I don’t know my Qemu’s version. Because I am using Kemari Projects newest version. And, my host kernel version is Linux kernel 3.6.0.
I want to know Qemu’s migration process.(Especially, arch_init.c ‘s ram_save_live function.)
Q1.
I am trying to know, brief algorythm’s process of my version. I paste my version’s code below of mail.
I guess, this process is occur.
1. Every time, this functions is called, We sync Qemu’s ram_list’s dirty bitmap to KVM’s. But, Null is returned at first time, because cpu_physical_memory_set_dirty_tracking(1); is not setted. In Second iteration’s, Qemu’s dirty bitmap is synchronized to KVM’s dirty bitmap.
2. In first call(after migration command is typed), cpu_physical_sync_dirty_bitmap doesn’t synchronized our qemu’s ram_list’s dirty bitmap. So, We must set all ram_list’s dirty bitmap to 1(Dirty).
3. Enable, KVM can return dirty bitmap. after this code, cpu_physical_sync_dirty_bitmap can work.
4. (Maybe) put metadata(not data) to Qemu file.
5. Another thread change stage to 2, and Iteratively copy dirty bytes.
6. Stage is shift to 3 and, Stop system to prevent dirty memory bit is generated. And, put system status to Qemu files.(Stop-And-Copy phase)
Is this right?
Q2.
I am trying to check my guess is right. But, my guess is not right. I want to know why.
Test 1.
If I disable 2(Set all block’s dirtybitmap to 0xFFFFFFFFUL), System transfer only dirty data and, Stop-And-Copy phase is going to very long time. So, I comment out this code.
#if 0
/* Make sure all dirty bits are set */
QLIST_FOREACH(block, &ram_list.blocks, next)
{
for (addr = block->offset; addr < block->offset + block->length;
addr += TARGET_PAGE_SIZE)
{
if (!cpu_physical_memory_get_dirty(addr,MIGRATION_DIRTY_FLAG))
{
cpu_physical_memory_set_dirty(addr);
//printf("addr is %lu",addr);
}
}
}
#endif
result1.
remaining data is going bigger.
Test2.
If I reset dirty bitmap before stage2’s put code, I can check stage2’s data transfer rate to very small. So I insert next code in stage2.(line 315 of below code.) I insert first_trans flag, to execute this code one time.
//Make sure stage2's first iteration doesn't send anything.
if(stage == 2 && first_trans)
{
RAMBlock *block;
first_trans=0;
QLIST_FOREACH(block, &ram_list.blocks, next)
{
for (addr = block->offset; addr < block->offset + block->length;
addr += TARGET_PAGE_SIZE)
{
if (!cpu_physical_memory_get_dirty(addr,MIGRATION_DIRTY_FLAG))
{
//cpu_physical_memory_set_dirty(addr);
cpu_physical_memory_reset_dirty(addr, addr+TARGET_PAGE_SIZE,MIGRATION_DIRTY_FLAG);
cpu_physical_memory_reset_dirty(addr, addr+TARGET_PAGE_SIZE,MASTER_DIRTY_FLAG);
}
}
}
}
Result2.
Stage2’s transfer data is more bigger.
Q3.
I create my own bitmap(Using hyper call and guest’s daemon) and sent it to QEMU. I am trying to reset bitmaps using this bitmap.(Ignore some Guest page’s copying)
I wrote next code, but it doesn’t work. I want to know why.
for (j=0; j<ACTIVE_NSLOTS; j++)// ACTIVE_NSLOTS means my KVM slot number.(0 to 2)
{
const target_phys_addr_t end_addr=(base_gfn[j]+npages[j])<<12;
start_addr=base_gfn[j]<<12;//base_gfn/npages of KVM slot j
while(start_addr < end_addr)
{
const target_phys_addr_t current = get_start_addr(start_addr, end_addr);
if(current == NULL)
{
break;
}
size_t size = get_memory_size(current, end_addr);//get memory size of first slot overllaped in current~end_addr
if (current + size > end_addr)//entered here after second call
{
size=(end_addr - current);
}
const size_t len = ((size / TARGET_PAGE_SIZE) + HOST_LONG_BITS - 1) / HOST_LONG_BITS;//len is number of bitmaps
for (i=0; i<len; ++i)
{
const size_t index = i +count;
const size_t page_number = index * HOST_LONG_BITS;
const target_phys_addr_t addr1 = page_number * TARGET_PAGE_SIZE;
const target_phys_addr_t addr2 = current + addr1;
const ram_addr_t ram_addr = cpu_get_physical_page_desc(addr2);
cpu_physical_memory_reset_dirty(ram_addr, ram_addr+TARGET_PAGE_SIZE,MIGRATION_DIRTY_FLAG);
cpu_physical_memory_reset_dirty(ram_addr, ram_addr+TARGET_PAGE_SIZE,MASTER_DIRTY_FLAG);
cpu_physical_memory_set_dirty_range(ram_addr, leul_to_cpu(pagecache_bitmap[j][index]));
cpu_physical_memory_set_dirty_range_mig(ram_addr, leul_to_cpu(pagecache_bitmap[j][index]));
}
start_addr = current + size;
count += len;
}
}
first_trans=0;
}
Doesn’t work. I think I have to know algorythm’s of Qemu migration.
I’m looking forward to information to solve this proble. Thankyou.
269 int ram_save_live(Monitor *mon, QEMUFile *f, int stage, void *opaque)
270 {
271 ram_addr_t addr;
272 uint64_t bytes_transferred_last;
273 double bwidth = 0;
274 uint64_t expected_time = 0;
276 if (stage < 0) {
277 cpu_physical_memory_set_dirty_tracking(0);
278 return 0;
279 }
281 if (cpu_physical_sync_dirty_bitmap(0, TARGET_PHYS_ADDR_MAX) != 0) {
283 return 0;
284 }
286 if (stage == 1) {
288 bytes_transferred = 0;
289 last_block = NULL;
290 last_offset = 0;
291 sort_ram_list();
293 /* Make sure all dirty bits are set */
294 QLIST_FOREACH(block, &ram_list.blocks, next) {
295 for (addr = block->offset; addr < block->offset + block->length;
296 addr += TARGET_PAGE_SIZE) {
297 if (!cpu_physical_memory_get_dirty(addr,
299 cpu_physical_memory_set_dirty(addr);
300 }
301 }
302 }
304 /* Enable dirty memory tracking */
305 cpu_physical_memory_set_dirty_tracking(1);
307 qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE);
309 QLIST_FOREACH(block, &ram_list.blocks, next) {
310 qemu_put_byte(f, strlen(block->idstr));
311 qemu_put_buffer(f, (uint8_t *)block->idstr, strlen(block->idstr));
312 qemu_put_be64(f, block->length);
313 }
314 }
316 bytes_transferred_last = bytes_transferred;
317 bwidth = qemu_get_clock_ns(rt_clock);
319 while (!qemu_file_rate_limit(f)) {
320 int bytes_sent;
322 bytes_sent = ram_save_block(f);
323 bytes_transferred += bytes_sent;
324 if (bytes_sent == 0) { /* no more blocks */
325 break;
326 }
327 }
329 bwidth = qemu_get_clock_ns(rt_clock) - bwidth;
330 bwidth = (bytes_transferred - bytes_transferred_last) / bwidth;
332 /* if we haven't transferred anything this round, force expected_time to a
333 * a very high value, but without crashing */
334 if (bwidth == 0) {
335 bwidth = 0.000001;
336 }
338 /* try transferring iterative blocks of memory */
339 if (stage == 3) {
340 int bytes_sent;
342 /* flush all remaining blocks regardless of rate limiting */
343 while ((bytes_sent = ram_save_block(f)) != 0) {
344 bytes_transferred += bytes_sent;
345 }
346 cpu_physical_memory_set_dirty_tracking(0);
347 }
349 qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
351 expected_time = ram_save_remaining() * TARGET_PAGE_SIZE / bwidth;
353 return (stage == 2) && (expected_time <= migrate_max_downtime());
354 }
[Prev in Thread] | Current Thread | [Next in Thread] |