[Qemu-devel] [RFC v4 00/28] Base enabling patches for MTTCG

[Alex Bennée]

This is the fourth iteration of the RFC patch set which aims to
provide the basic framework for MTTCG. I hope this will provide a good
base for discussion at KVM Forum later this month.

Prerequisites
=============

This tree has been built on top of two other series of patches:

  - Reduce lock contention on TCG hot-path (v5, in Paolo's tree)
  - cpu-exec: Safe work in quiescent state (v5, in my tree)

You can find the base tree (based off -rc0) at:

  https://github.com/stsquad/qemu/tree/mttcg/async-safe-work-v5

Changes
=======

Since the last posting there have been a number of updates to the
original patches:

   - more updates to docs/multi-thread-tcg.txt design document
   - clean ups of sleep handling (and safe work integration)
   - split the big enable-multi-thread patch
   - split some re-factoring movement stuff into individual patches

As usual the patches themselves have a revision summary under the ---

In addition I've brought forward a number of changes from the original
ARM enabling patches to support the various cputlb operations which
are basically generic anyway. These include:

   - making cross-vCPU tlb_flush operations use async_run_on_cpu
   - making tlb_reset_dirty_range atomically apply the TLB_NOTDIRTY flag

A copy of the tree can be found at:

  https://github.com/stsquad/qemu/tree/mttcg/base-patches-v4

The series includes all the generic work needed and in theory just
needs MTTCG aware atomics and memory barriers for the various
host/guest combinations to be enabled by default.

In practice the memory barrier problems don't show up with an x86
host. In fact I have created a tree which merges in the Emilio's
cmpxchg atomics which happily boots ARMv7 Debian systems without any
additional changes. You can find that at:

  
https://github.com/stsquad/qemu/tree/mttcg/base-patches-v4-with-cmpxchg-atomics-v2


Testing
=======

I've tested this boots ARMv7 Debian and all both ARMv7 and v8 kvm-unit-tests 
with:

  -accel tcg,thread=single

In addition I've tested ARMv7 and ARMv8 kvm-unit-tests of the tcg and
tlbflush group with:

  -accel tcg,thread=multi

These tests are safe as they don't rely on atomics to be work but do
exercise the parallel execution, invalidation and flushing of code.
The full invocation of all the tests is:

  echo "Running all tests in Single Thread Mode"
  ./run_tests.sh -t -o "-accel tcg,thread=single -name debug-threads=on"
  echo "Running tlbflush in Multi Thread Mode"
  ./run_tests.sh -t -g tlbflush -o "-accel tcg,thread=multi -name 
debug-threads=on"
  echo "Running TCG in Multi Thread Mode"
  ./run_tests.sh -t -g tcg -o "-accel tcg,thread=multi -name debug-threads=on"


Performance
===========

You can't do full work-load testing on this tree due to the lack of
atomic support (but I will run some numbers on
mttcg/base-patches-v4-with-cmpxchg-atomics-v2). However you certainly
see a run time improvement with the kvm-unit-tests TCG group.

  retry.py called with ['./run_tests.sh', '-t', '-g', 'tcg', '-o', '-accel 
tcg,thread=single']
  run 1: ret=0 (PASS), time=1047.147924 (1/1)
  run 2: ret=0 (PASS), time=1071.921204 (2/2)
  run 3: ret=0 (PASS), time=1048.141600 (3/3)
  Results summary:
  0: 3 times (100.00%), avg time 1055.737 (196.70 varience/14.02 deviation)
  Ran command 3 times, 3 passes
  retry.py called with ['./run_tests.sh', '-t', '-g', 'tcg', '-o', '-accel 
tcg,thread=multi']
  run 1: ret=0 (PASS), time=303.074210 (1/1)
  run 2: ret=0 (PASS), time=304.574991 (2/2)
  run 3: ret=0 (PASS), time=303.327408 (3/3)
  Results summary:
  0: 3 times (100.00%), avg time 303.659 (0.65 varience/0.80 deviation)
  Ran command 3 times, 3 passes

The TCG tests run with -smp 4 on my system. While the TCG tests are
purely CPU bound they do exercise the hot and cold paths of TCG
execution (especially when triggering SMC detection). However there is
still a benefit even with a 50% overhead compared to the ideal 263
second elapsed time.

Alex

Alex Bennée (23):
  cpus: make all_vcpus_paused() return bool
  translate_all: DEBUG_FLUSH -> DEBUG_TB_FLUSH
  translate-all: add DEBUG_LOCKING asserts
  cpu-exec: include cpu_index in CPU_LOG_EXEC messages
  docs: new design document multi-thread-tcg.txt (DRAFTING)
  linux-user/elfload: ensure mmap_lock() held while setting up
  translate-all: Add assert_(memory|tb)_lock annotations
  target-arm/arm-powerctl: wake up sleeping CPUs
  tcg: move tcg_exec_all and helpers above thread fn
  tcg: cpus rm tcg_exec_all()
  tcg: add kick timer for single-threaded vCPU emulation
  tcg: rename tcg_current_cpu to tcg_current_rr_cpu
  cpus: re-factor out handle_icount_deadline
  tcg: remove global exit_request
  tcg: move locking for tb_invalidate_phys_page_range up
  cpus: tweak sleeping and safe_work rules for MTTCG
  tcg: enable tb_lock() for SoftMMU
  tcg: enable thread-per-vCPU
  atomic: introduce cmpxchg_bool
  cputlb: add assert_cpu_is_self checks
  cputlb: tweak qemu_ram_addr_from_host_nofail reporting
  cputlb: make tlb_reset_dirty safe for MTTCG
  cputlb: make tlb_flush_by_mmuidx safe for MTTCG

Jan Kiszka (1):
  tcg: drop global lock during TCG code execution

KONRAD Frederic (3):
  tcg: protect TBContext with tb_lock.
  tcg: add options for enabling MTTCG
  cputlb: introduce tlb_flush_* async work.

Paolo Bonzini (1):
  tcg: comment on which functions have to be called with tb_lock held

 bsd-user/mmap.c           |   5 +
 cpu-exec-common.c         |  19 +-
 cpu-exec.c                |  41 ++--
 cpus.c                    | 510 +++++++++++++++++++++++++++++-----------------
 cputlb.c                  | 279 ++++++++++++++++++-------
 docs/multi-thread-tcg.txt | 310 ++++++++++++++++++++++++++++
 exec.c                    |  28 +++
 hw/i386/kvmvapic.c        |   4 +
 include/exec/cputlb.h     |   2 -
 include/exec/exec-all.h   |   5 +-
 include/qemu/atomic.h     |   9 +
 include/qom/cpu.h         |  27 +++
 include/sysemu/cpus.h     |   2 +
 linux-user/elfload.c      |   4 +
 linux-user/mmap.c         |   5 +
 memory.c                  |   2 +
 qemu-options.hx           |  20 ++
 qom/cpu.c                 |  10 +
 softmmu_template.h        |  17 ++
 target-arm/Makefile.objs  |   2 +-
 target-arm/arm-powerctl.c |   2 +
 target-i386/smm_helper.c  |   7 +
 tcg/tcg.h                 |   2 +
 translate-all.c           | 175 +++++++++++++---
 vl.c                      |  48 ++++-
 25 files changed, 1227 insertions(+), 308 deletions(-)
 create mode 100644 docs/multi-thread-tcg.txt

-- 
2.7.4