Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Paolo Bonzini]

Il 25/09/2013 16:32, Lei Li ha scritto:
> This RFC patch series tries to introduce a mechanism using side
> channel pipe for RAM via SCM_RIGHTS with unix domain socket
> protocol migration.
> 
> This side channel will be used for the page flipping by vmsplice,
> which will be the internal mechanism for localhost migration that
> we are trying to add. The previous patch series for localhost migration
> as link,
> 
> http://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg02916.html
> 
> After this series, will adjust the process of current migration for
> the localhost migration and involve the vmsplice based on the previous
> patch set as link above.
> 
> Please let me know if it is the proper way for it or there is anything
> need to be improved. Your suggestions and comments are very welcome, and
> thanks for Paolo for his review and useful suggestions.
> 
> 
> Lei Li (8):
>   migration-local: add pipe protocol for QEMUFileOps
>   migration-local: add qemu_fopen_pipe()
>   migration-local: add send_pipefd()
>   migration-local: add recv_pipefd()
>   QAPI: introduce magration capability unix_page_flipping
>   migration: add migrate_unix_page_flipping()
>   migration-unix: side channel support on unix outgoing
>   migration-unix: side channel support on unix incoming
> 
>  Makefile.target               |    1 +
>  include/migration/migration.h |    3 +
>  include/migration/qemu-file.h |    4 +
>  migration-local.c             |  247 
> +++++++++++++++++++++++++++++++++++++++++
>  migration-unix.c              |   48 +++++++-
>  migration.c                   |    9 ++
>  qapi-schema.json              |    8 +-
>  7 files changed, 315 insertions(+), 5 deletions(-)
>  create mode 100644 migration-local.c
> 

Yes, this is much closer!

There are two problems to be fixed, but it is getting there.

First, it breaks migration from old QEMU to new QEMU, and also migration
where the source uses "unix:" and the destination uses "fd:" migration
(this should work as long as page flipping is disabled).  The problem is
that recv_pipefd() "eats" one byte, and old versions of QEMU do not send
that byte.

The second problem is that you are not really using a side channel; you
are still using the QEMUFile and relying on the normal migration code to
send pages on the pipe.  This will not be possible when you use vmsplice.

Both problems can be addressed with a single change in your approach:
always use the Unix socket QEMUFile but, if page flipping is enabled,
only transmit page addresses on the socket; page data will be on the
pipe.  You can use hooks such as before_ram_iterate, save_page and
hook_ram_load to do all your customizations: send the pipe file
descriptor, read the pipe file descriptor, and use the pipe as a side
channel.

To fix the first problem, you can use the before_ram_iterate callback to
send the fd, and the hook_ram_load callback to receive it.  The
before_ram_iterate callback can write a special 8-byte record (with the
RAM_SAVE_FLAG_HOOK set) that will trigger the hook, followed by
send_pipefd().  The load_hook callback is called after the first 8-byte
record is sent, and can just do recv_pipefd().

To fix the second problem, and really use the pipe as a side channel,
you can use the save_page QEMUFile callback on the send side.  This
callback must return RAM_SAVE_CONTROL_NOT_SUPP if page flipping is
disabled.  If it is enabled, it should write another 8-byte record with
the RAM_SAVE_FLAG_HOOK bit, this time with the address of the page on
the Unix socket; then write the page data on the pipe, and return 0.  On
the receive side, the 8-byte page address will once more cause the
load_hook callback to be called.  This time you already have a file
descriptor, so you do not need to call recv_pipefd(): you just extract
the page address from the 8-byte record and read the page data from the
pipe.

The basis of your code will still be the socket-based QEMUFile, but
you'll need your own QEMUFile since you're adding Unix-specific
functionality.  For this it is not a problem to have two copies the
QEMUFile code for sockets, one in savevm.c and one in migration-unix.c.
 It's a very small amount of code.

Paolo