[Qemu-commits] [qemu/qemu] 6c1e39: configure: add --disable-colo-proxy option

[Richard Henderson]

  Branch: refs/heads/staging
  Home:   https://github.com/qemu/qemu
  Commit: 6c1e3906ce0bb414478d9c2c698bcbb6d2d28c63
      
https://github.com/qemu/qemu/commit/6c1e3906ce0bb414478d9c2c698bcbb6d2d28c63
  Author: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
  Date:   2023-05-18 (Thu, 18 May 2023)

  Changed paths:
    M meson_options.txt
    M net/meson.build
    M scripts/meson-buildoptions.sh
    A stubs/colo-compare.c
    M stubs/meson.build

  Log Message:
  -----------
  configure: add --disable-colo-proxy option

Add option to not build filter-rewriter and colo-compare when
they are not needed.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Zhang Chen <chen.zhang@intel.com>
Message-Id: <20230515130640.46035-2-vsementsov@yandex-team.ru>
Signed-off-by: Juan Quintela <quintela@redhat.com>


  Commit: dd42ce24a3cda4be3c839aceb91fdf85e31c194f
      
https://github.com/qemu/qemu/commit/dd42ce24a3cda4be3c839aceb91fdf85e31c194f
  Author: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
  Date:   2023-05-18 (Thu, 18 May 2023)

  Changed paths:
    M migration/colo.c
    M migration/migration.c
    M migration/migration.h
    M migration/rdma.c

  Log Message:
  -----------
  migration: split migration_incoming_co

Originally, migration_incoming_co was introduced by
25d0c16f625feb3b6
   "migration: Switch to COLO process after finishing loadvm"
to be able to enter from COLO code to one specific yield point, added
by 25d0c16f625feb3b6.

Later in 923709896b1b0
 "migration: poll the cm event for destination qemu"
we reused this variable to wake the migration incoming coroutine from
RDMA code.

That was doubtful idea. Entering coroutines is a very fragile thing:
you should be absolutely sure which yield point you are going to enter.

I don't know how much is it safe to enter during qemu_loadvm_state()
which I think what RDMA want to do. But for sure RDMA shouldn't enter
the special COLO-related yield-point. As well, COLO code doesn't want
to enter during qemu_loadvm_state(), it want to enter it's own specific
yield-point.

As well, when in 8e48ac95865ac97d
 "COLO: Add block replication into colo process" we added
bdrv_invalidate_cache_all() call (now it's called activate_all())
it became possible to enter the migration incoming coroutine during
that call which is wrong too.

So, let't make these things separate and disjoint: loadvm_co for RDMA,
non-NULL during qemu_loadvm_state(), and colo_incoming_co for COLO,
non-NULL only around specific yield.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Message-Id: <20230515130640.46035-3-vsementsov@yandex-team.ru>
Signed-off-by: Juan Quintela <quintela@redhat.com>


  Commit: d0a14a2ba01c7b200e6ce3e7979e1ed3ede1d5c7
      
https://github.com/qemu/qemu/commit/d0a14a2ba01c7b200e6ce3e7979e1ed3ede1d5c7
  Author: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
  Date:   2023-05-18 (Thu, 18 May 2023)

  Changed paths:
    M include/migration/colo.h
    M migration/colo.c
    M migration/migration.c
    M stubs/colo.c

  Log Message:
  -----------
  migration: process_incoming_migration_co(): move colo part to colo

Let's make better public interface for COLO: instead of
colo_process_incoming_thread and not trivial logic around creating the
thread let's make simple colo_incoming_co(), hiding implementation from
generic code.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Message-Id: <20230515130640.46035-4-vsementsov@yandex-team.ru>
Signed-off-by: Juan Quintela <quintela@redhat.com>


  Commit: 8e4b2a70599b3700b12b5a6059c819f81da9588c
      
https://github.com/qemu/qemu/commit/8e4b2a70599b3700b12b5a6059c819f81da9588c
  Author: Juan Quintela <quintela@redhat.com>
  Date:   2023-05-18 (Thu, 18 May 2023)

  Changed paths:
    M migration/migration-stats.h
    M migration/migration.c
    M migration/qemu-file.c

  Log Message:
  -----------
  migration: Don't use INT64_MAX for unlimited rate

Define and use RATE_LIMIT_DISABLED instead.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Harsh Prateek Bora <harshpb@linux.ibm.com>
Message-Id: <20230515195709.63843-2-quintela@redhat.com>


  Commit: de37f8b9c21e1c6ef98eebb0b05bd83e5867bc6f
      
https://github.com/qemu/qemu/commit/de37f8b9c21e1c6ef98eebb0b05bd83e5867bc6f
  Author: Juan Quintela <quintela@redhat.com>
  Date:   2023-05-18 (Thu, 18 May 2023)

  Changed paths:
    M migration/qemu-file.c

  Log Message:
  -----------
  qemu-file: Account for rate_limit usage on qemu_fflush()

That is the moment we know we have transferred something.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Cédric Le Goater <clg@kaod.org>
Message-Id: <20230515195709.63843-5-quintela@redhat.com>


  Commit: e1fde0e038bafd0bd05db7d43305b9b2f03c0683
      
https://github.com/qemu/qemu/commit/e1fde0e038bafd0bd05db7d43305b9b2f03c0683
  Author: Juan Quintela <quintela@redhat.com>
  Date:   2023-05-18 (Thu, 18 May 2023)

  Changed paths:
    M hw/ppc/spapr.c
    M hw/s390x/s390-stattrib.c
    M include/migration/qemu-file-types.h
    M migration/block-dirty-bitmap.c
    M migration/block.c
    M migration/meson.build
    M migration/migration-stats.c
    M migration/migration-stats.h
    M migration/migration.c
    M migration/multifd.c
    M migration/options.c
    M migration/options.h
    M migration/qemu-file.c
    M migration/qemu-file.h
    M migration/ram.c
    M migration/savevm.c

  Log Message:
  -----------
  migration: Move rate_limit_max and rate_limit_used to migration_stats

These way we can make them atomic and use this functions from any
place.  I also moved all functions that use rate_limit to
migration-stats.

Functions got renamed, they are not qemu_file anymore.

qemu_file_rate_limit -> migration_rate_exceeded
qemu_file_set_rate_limit -> migration_rate_set
qemu_file_get_rate_limit -> migration_rate_get
qemu_file_reset_rate_limit -> migration_rate_reset
qemu_file_acct_rate_limit -> migration_rate_account.

Reviewed-by: Harsh Prateek Bora <harshpb@linux.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-Id: <20230515195709.63843-6-quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>


  Commit: 99319e2dafaf2ad05412c8f119efcdf24a3baf04
      
https://github.com/qemu/qemu/commit/99319e2dafaf2ad05412c8f119efcdf24a3baf04
  Author: Juan Quintela <quintela@redhat.com>
  Date:   2023-05-18 (Thu, 18 May 2023)

  Changed paths:
    M migration/migration-stats.c
    M migration/migration-stats.h
    M migration/migration.c

  Log Message:
  -----------
  migration: Move migration_total_bytes() to migration-stats.c

Once there rename it to migration_transferred_bytes() and pass a
QEMUFile instead of a migration object.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Cédric Le Goater <clg@kaod.org>
Message-Id: <20230515195709.63843-7-quintela@redhat.com>


  Commit: 3db9c05a9006550da3a73600cec89bb995420325
      
https://github.com/qemu/qemu/commit/3db9c05a9006550da3a73600cec89bb995420325
  Author: Juan Quintela <quintela@redhat.com>
  Date:   2023-05-18 (Thu, 18 May 2023)

  Changed paths:
    M migration/migration-stats.c
    M migration/trace-events

  Log Message:
  -----------
  migration: Add a trace for migration_transferred_bytes

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Cédric Le Goater <clg@kaod.org>
Message-Id: <20230515195709.63843-8-quintela@redhat.com>


  Commit: 813cd61669e45ee6d5db09a83d03df8f0c6eb5d2
      
https://github.com/qemu/qemu/commit/813cd61669e45ee6d5db09a83d03df8f0c6eb5d2
  Author: Juan Quintela <quintela@redhat.com>
  Date:   2023-05-18 (Thu, 18 May 2023)

  Changed paths:
    M migration/migration-stats.c
    M migration/migration-stats.h
    M migration/migration.c

  Log Message:
  -----------
  migration: Use migration_transferred_bytes() to calculate rate_limit

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Cédric Le Goater <clg@kaod.org>
Message-Id: <20230515195709.63843-9-quintela@redhat.com>


  Commit: bd7ceaf6d584b77dd6dbd8af7949d1e91a7c0537
      
https://github.com/qemu/qemu/commit/bd7ceaf6d584b77dd6dbd8af7949d1e91a7c0537
  Author: Juan Quintela <quintela@redhat.com>
  Date:   2023-05-18 (Thu, 18 May 2023)

  Changed paths:
    M migration/migration-stats.c
    M migration/migration-stats.h
    M migration/multifd.c
    M migration/qemu-file.c

  Log Message:
  -----------
  migration: We don't need the field rate_limit_used anymore

Since previous commit, we calculate how much data we have send with
migration_transferred_bytes() so no need to maintain this counter and
remember to always update it.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Cédric Le Goater <clg@kaod.org>
Message-Id: <20230515195709.63843-10-quintela@redhat.com>


  Commit: cbec7eb76879d419e7dbf531ee2506ec0722e825
      
https://github.com/qemu/qemu/commit/cbec7eb76879d419e7dbf531ee2506ec0722e825
  Author: Juan Quintela <quintela@redhat.com>
  Date:   2023-05-18 (Thu, 18 May 2023)

  Changed paths:
    M migration/multifd.c

  Log Message:
  -----------
  migration/multifd: Compute transferred bytes correctly

In the past, we had to put the in the main thread all the operations
related with sizes due to qemu_file not beeing thread safe.  As now
all counters are atomic, we can update the counters just after the
do the write.  As an aditional bonus, we are able to use the right
value for the compression methods.  Right now we were assuming that
there were no compression at all.

Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Message-Id: <20230515195709.63843-17-quintela@redhat.com>


  Commit: ba9d2cbc01b4e33f9a97edcd77247831a333eac2
      
https://github.com/qemu/qemu/commit/ba9d2cbc01b4e33f9a97edcd77247831a333eac2
  Author: Juan Quintela <quintela@redhat.com>
  Date:   2023-05-18 (Thu, 18 May 2023)

  Changed paths:
    M migration/meson.build

  Log Message:
  -----------
  migration: Fix duplicated included in meson.build

This is the commint with the merge error (not in the submited patch).

commit 52623f23b0d114837a0d6278180b3e3ae8947117
Author: Lukas Straub <lukasstraub2@web.de>
Date:   Thu Apr 20 11:48:35 2023 +0200

    ram-compress.c: Make target independent

    Make ram-compress.c target independent.

Fixes: 52623f23b0d114837a0d6278180b3e3ae8947117
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Message-Id: <20230509170217.83246-1-quintela@redhat.com>


  Commit: 146f515110e86aefe3bc2e8eb581ab724614060f
      
https://github.com/qemu/qemu/commit/146f515110e86aefe3bc2e8eb581ab724614060f
  Author: Richard Henderson <richard.henderson@linaro.org>
  Date:   2023-05-18 (Thu, 18 May 2023)

  Changed paths:
    M hw/ppc/spapr.c
    M hw/s390x/s390-stattrib.c
    M include/migration/colo.h
    M include/migration/qemu-file-types.h
    M meson_options.txt
    M migration/block-dirty-bitmap.c
    M migration/block.c
    M migration/colo.c
    M migration/meson.build
    M migration/migration-stats.c
    M migration/migration-stats.h
    M migration/migration.c
    M migration/migration.h
    M migration/multifd.c
    M migration/options.c
    M migration/options.h
    M migration/qemu-file.c
    M migration/qemu-file.h
    M migration/ram.c
    M migration/rdma.c
    M migration/savevm.c
    M migration/trace-events
    M net/meson.build
    M scripts/meson-buildoptions.sh
    A stubs/colo-compare.c
    M stubs/colo.c
    M stubs/meson.build

  Log Message:
  -----------
  Merge tag 'migration-20230518-pull-request' of 
https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request

Hi

Based on latest reviewed parts of migration:
- Disable colo (vladimir)
- Migration atomic counters (juan)

Please apply.

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCAAdFiEEGJn/jt6/WMzuA0uC9IfvGFhy1yMFAmRmXJUACgkQ9IfvGFhy
# 1yNRAxAAjDYJELL34Qovt/WE9qKhYJEvIUGTl1IMWJ22YMFnqIFKRdka57dWoU3P
# 7EK1BHmokEEtzGT7Fe1ecERXsOwQIJDIkDTJ5g8Oc8Jt1iqY1AC8h5T+LghijCar
# mbZ6qWHaSjsg2lmek/xc9quymzFGGK36PSyB5WkaLRviKQn4RIkEDpUaWny7nDbA
# Q8zJJpBqNFqKfC5/DN0ePa3QQscXQJhey3nxqFd8hYp8RFNIV5UJVW5Lf6ombtK7
# atgdWC4ckkfO2z3OsghKeo/UiMFWpPktgBVVMhDLmk+P/E6czc2gfzD6SCvrPKTj
# XowI8hro22HVmq9bEY8PtbjMOfpxrAxer+tM2KR/0O9l3UzUacFsi7KGqCJ1/trQ
# 1tSDjlgyczb8GOgLwwxj8XE+jPHPfVrzCNfDqrBKBNxz6nnZSdZUwhV5mG8FdVtm
# oVVV96BIrNXLl/lIxYIFD/Zyvl8/lrSWQdLkEHTzihYQeXaQfyvPVbV/dOLT4sii
# YUuGCuEhF+DW/qz43G1krwq5/bfxsiZoQzrMV/Odtf0wYQKkabA3KNBIda/vxBCR
# dsLQ7QtmOwKmCzjqw4LUov9vDNYOYr98o7ZqwJ3qeKL4QgFwtEZUFO3VW6UR8fnF
# arVXiTn9wVlkTpu4sT5hLm9400iadhX4Fppji7Ce0tUpLbWbghA=
# =3x32
# -----END PGP SIGNATURE-----
# gpg: Signature made Thu 18 May 2023 10:12:53 AM PDT
# gpg:                using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [undefined]
# gpg:                 aka "Juan Quintela <quintela@trasno.org>" [undefined]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg:          There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03  4B82 F487 EF18 5872 D723

* tag 'migration-20230518-pull-request' of 
https://gitlab.com/juan.quintela/qemu:
  migration: Fix duplicated included in meson.build
  migration/multifd: Compute transferred bytes correctly
  migration: We don't need the field rate_limit_used anymore
  migration: Use migration_transferred_bytes() to calculate rate_limit
  migration: Add a trace for migration_transferred_bytes
  migration: Move migration_total_bytes() to migration-stats.c
  migration: Move rate_limit_max and rate_limit_used to migration_stats
  qemu-file: Account for rate_limit usage on qemu_fflush()
  migration: Don't use INT64_MAX for unlimited rate
  migration: process_incoming_migration_co(): move colo part to colo
  migration: split migration_incoming_co
  configure: add --disable-colo-proxy option

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>


Compare: https://github.com/qemu/qemu/compare/297e8182194e...146f515110e8