Re: [Qemu-devel] [BUG] Regression of exec migration

[Pierre Riteau]

On 27 août 09, at 18:24, Anthony Liguori wrote:

> Pierre Riteau wrote:
> > On 27 août 09, at 16:13, Anthony Liguori wrote:
> >
> > > Pierre Riteau wrote:
> > > > [Sorry Chris, resending without the giant attachments.]
> > > >
> > > > Commit 907500095851230a480b14bc852c4e49d32cb16d makes exec  
> > > > migration much slower than before.
> > > > I'm running the latest HEAD of qemu, on Debian Lenny 5.0.2.
> > > >
> > > > I'm migrating a fully booted Linux VM (also running Lenny) with  
> > > > 128MB of RAM to a file, using the following command: migrate  
> > > > "exec: cat > vmimage". The resulting file has a size of 57MB  
> > > > (because we save only what is allocated from the 128MB).
> > > > With the current HEAD, it takes from 15 to 40 seconds (it's  
> > > > variable) to perform the migration to the file.
> > > > With commit 907500095851230a480b14bc852c4e49d32cb16d reverted (or  
> > > > just commenting the "socket_set_nonblock(s->fd);" statement), it  
> > > > takes about 3 seconds.
> > >
> > > Without that changeset, it wasn't a live migration.  The better  
> > > way to compare would be to issue stop before doing the migrate and  
> > > compare that time with the previous time.
> > >
> > > When a migration is live, it's iterative which means there's more  
> > > work to do.
> >
> > I tried with stop too, and I get the same results. It's an idle VM  
> > so only a small number of pages are being modified while the  
> > migration is going on.
> > I agree that the changeset seems good, the code it replaces was  
> > obviously wrong.
> > But I think there is something wrong somewhere else, unless it is  
> > considered normal that it takes so much time for an exec migration.
> > To compare, using the same setup with one more machine and a  
> > Gigabit network, a tcp migration capped at 35m (the slowest speed  
> > I've measured from the disk, it can be way faster) takes about the  
> > same time, between 2 and 4 seconds.
>
> I don't think the difference between 3 seconds and 15 seconds is  
> significant.
>
> Can you try a different workload that will result in a migration  
> that takes much longer (say multiple minutes)?  That is, I'd like to  
> know whether there's a fixed greater cost of exec: migration vs.  
> factor of 5.
>
> I expect exec: to be slower because there is more copying but not by  
> a factor of 5.  I expect that it's going to be a combination of  
> relatively small constant factor + relatively small constant fixed  
> cost.
>
> Regards,
>
> Anthony Liguori


I did more tests, now with a 1024MB VM. Before launching the migration  
I run a simple program that allocates 900MB of memory and fills it  
with random data, then sleeps.
The results are:
  TCP migration: ~ 30s
  exec migration to hard drive: ~ 4 min 20s
  exec migration to netcat (replicating the setup used with the TCP  
migration): ~ 5 min 40s

Now, what's interesting is that I modified the program to do an  
infinite loop after writing the 900MB instead of sleeping.
With this new program the results for exec migration are quite  
different from the previous experiment:
  TCP migration: ~ 30s
  exec migration to hard drive:  ~ 30s
  exec migration to netcat: ~ 30s

Could it be that, when the VM is idling, qemu sleeps for some time and  
misses important events like "the popen'ed process is ready to read"?.

--
Pierre Riteau -- http://perso.univ-rennes1.fr/pierre.riteau/