[RFC PATCH v3 00/30] migration: File based migration with multifd and fixed-ram

[Fabiano Rosas]

Hi,

In this v3:

Added support for the "file:/dev/fdset/" syntax to receive multiple
file descriptors. This allows the management layer to open the
migration file beforehand and pass the file descriptors to QEMU. We
need more than one fd to be able to use O_DIRECT concurrently with
unaligned writes.

Dropped the auto-pause capability. That discussion was kind of
stuck. We can revisit optimizations for non-live scenarios once the
series is more mature/merged.

Changed the multifd incoming side to use a more generic data structure
instead of MultiFDPages_t. This allows multifd to restore the ram
using larger chunks.

The rest are minor changes, I have noted them in the patches
themselves.

Thanks

CI run: https://gitlab.com/farosas/qemu/-/pipelines/1086786947

v2:
https://lore.kernel.org/r/20231023203608.26370-1-farosas@suse.de
v1:
https://lore.kernel.org/r/20230330180336.2791-1-farosas@suse.de

Fabiano Rosas (24):
  io: fsync before closing a file channel
  migration/ram: Introduce 'fixed-ram' migration capability
  migration: Add fixed-ram URI compatibility check
  migration/ram: Add incoming 'fixed-ram' migration
  migration/multifd: Allow multifd without packets
  migration/multifd: Allow QIOTask error reporting without an object
  migration/multifd: Add outgoing QIOChannelFile support
  migration/multifd: Add incoming QIOChannelFile support
  io: Add a pwritev/preadv version that takes a discontiguous iovec
  multifd: Rename MultiFDSendParams::data to compress_data
  migration/multifd: Decouple recv method from pages
  migration/multifd: Allow receiving pages without packets
  migration/ram: Ignore multifd flush when doing fixed-ram migration
  migration/multifd: Support outgoing fixed-ram stream format
  migration/multifd: Support incoming fixed-ram stream format
  tests/qtest: Add a multifd + fixed-ram migration test
  migration: Add direct-io parameter
  tests/qtest: Add a test for migration with direct-io and multifd
  monitor: Honor QMP request for fd removal immediately
  monitor: Extract fdset fd flags comparison into a function
  monitor: fdset: Match against O_DIRECT
  docs/devel/migration.rst: Document the file transport
  migration: Add support for fdset with multifd + file
  tests/qtest: Add a test for fixed-ram with passing of fds

Nikolay Borisov (6):
  io: add and implement QIO_CHANNEL_FEATURE_SEEKABLE for channel file
  io: Add generic pwritev/preadv interface
  io: implement io_pwritev/preadv for QIOChannelFile
  migration/qemu-file: add utility methods for working with seekable
    channels
  migration/ram: Add outgoing 'fixed-ram' migration
  tests/qtest: migration-test: Add tests for fixed-ram file-based
    migration

 docs/devel/migration.rst            |  43 ++++
 include/exec/ramblock.h             |   8 +
 include/io/channel.h                | 109 ++++++++
 include/migration/qemu-file-types.h |   2 +
 include/qemu/bitops.h               |  13 +
 include/qemu/osdep.h                |   2 +
 io/channel-file.c                   |  69 +++++
 io/channel.c                        | 128 ++++++++++
 migration/file.c                    | 191 +++++++++++++-
 migration/file.h                    |   5 +
 migration/migration-hmp-cmds.c      |  11 +
 migration/migration.c               |  38 ++-
 migration/multifd-zlib.c            |  22 +-
 migration/multifd-zstd.c            |  22 +-
 migration/multifd.c                 | 376 ++++++++++++++++++++++------
 migration/multifd.h                 |  30 ++-
 migration/options.c                 |  70 ++++++
 migration/options.h                 |   4 +
 migration/qemu-file.c               |  82 ++++++
 migration/qemu-file.h               |   7 +-
 migration/ram.c                     | 291 ++++++++++++++++++++-
 migration/ram.h                     |   1 +
 migration/savevm.c                  |   1 +
 monitor/fds.c                       |  27 +-
 qapi/migration.json                 |  24 +-
 tests/qtest/migration-helpers.c     |  42 ++++
 tests/qtest/migration-helpers.h     |   1 +
 tests/qtest/migration-test.c        | 206 +++++++++++++++
 util/osdep.c                        |   9 +
 29 files changed, 1686 insertions(+), 148 deletions(-)

-- 
2.35.3