Re: [Qemu-devel] [RFC PATCH v2 00/49] Series short description

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:address@hidden On Behalf Of Paolo Bonzini
> Il 17/07/2014 13:01, Pavel Dovgalyuk ha scritto:
> > This set of patches is related to the reverse execution and deterministic
> > replay of qemu execution  Our implementation of deterministic replay can
> > be used for deterministic and reverse debugging of guest code through gdb
> > remote interface.
> >
> > Execution recording writes non-deterministic events log, which can be later
> > used for replaying the execution anywhere and for unlimited number of times.
> > It also supports checkpointing for faster rewinding during reverse 
> > debugging.
> > Execution replaying reads the log and replays all non-deterministic events
> > including external input, hardware clocks, and interrupts.
> 
> From a first look:
> 
> - patches 2 to 13 probably should try to use subsections, so that VMs
> that do not use the devices try not to save the extra data and keep
> backwards migration compatibility (at least try to)

 Could you give me and example?
 As I know, subsection is loaded when some predicate function returns true. How 
can I 
construct such a function for integratorcp module? What kind of condition 
should it check?
 In this module I just added missed vmstates (it does not saved/restored at all
by the master version).

> - patch 16 should also use subsections, and perhaps apply to all other
> CPUs too?

 We implemented replay only for i386 and ARM. If we'll change other targets, it 
will not
add record/replay capabilities to them, but can confuse the reviewers.

> - patches 23-24-25 perhaps could try using icount, like Konrad's patch do?

 Using faster icount (like in Konrad's patches) is the our next aim. It 
obviously will 
increase the speed of recording process. But now I submitted slower, but more 
conservative
version of icount which we had already tested.

> - patch 27 makes sense but VIRTUAL is used to skip blinking when the VM
> is stopped

 Right, this is kind of hack. I haven't found better solution yet.

> - the others I haven't yet looked at, but they look like something that
> would bitrot really, really fast.  For patch 33, I think changing
> INSERT_HEAD to INSERT_TAIL would be just fine, and I wonder if it's the
> same for other patches here.  How do you plan on testing them and
> keeping them up to date?

 We're constantly keeping these patches up to date, because we are using 
deterministic replay
and reverse debugging for solving our tasks. We re-test all the features when 
pulling new
patches from the master branch.

Pavel Dovgalyuk

> 
> > Reverse execution has the following features:
> >  * Deterministically replays whole system execution and all contents of the 
> > memory,
> >    state of the hadrware devices, clocks, and screen of the VM.
> >  * Writes execution log into the file for latter replaying for multiple 
> > times
> >    on different machines.
> >  * Supports i386, x86_64, and ARM hardware platforms.
> >  * Performs deterministic replay of all operations with keyboard, mouse, 
> > network adapters,
> >    audio devices, serial interfaces, and physical USB devices connected to 
> > the emulator.
> >  * Provides support for gdb reverse debugging commands like reverse-step 
> > and reverse-
> continue.
> >  * Supports auto-checkpointing for convenient reverse debugging.
> >
> > Usage of the record/replay:
> >  * First, record the execution, by adding '-record fname=replay.bin' to the
> >    command line.
> >  * Then you can replay it for the multiple times by using another command
> >    line option: '-replay fname=replay.bin'
> >  * Virtual machine should have at least one virtual disk, which is used to
> >    store checkpoints. If you want to enable automatic checkpointing, simply
> >    add ',period=XX' to record options, where XX is the checkpointing period
> >    in seconds.
> >  * Using of the network adapters in record/replay mode is possible with
> >    the following command-line options:
> >    - '-net user' (or another host adapter) in record mode
> >    - '-net replay' in replay mode. Every host network adapter should be
> >      replaced by 'replay' when replaying the execution.
> >  * Reverse debugging can be used through gdb remote interface.
> >    reverse-stepi and reverse-continue commands are supported. Other reverse
> >    commands should also work, because they reuse these ones.
> >  * Monitor is extended by the following commands:
> >    - replay_info - prints information about replay mode and current step
> >      (number of instructions executed)
> >    - replay_break - sets "breakpoint" at the specified instructions count.
> >    - replay_seek - rewinds (using the checkpoints, if possible) to the
> >      specified step of replay log.
> >
> > Paper with short description of deterministic replay implementation:
> > http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html
> >
> > Modifications of qemu include:
> >  * adding missed fields of the virtual devices' states to the vmstate
> >    structures to allow deterministic saving and restoring the VM state
> >  * adding virtual clock-based timers to vmstate structures, because virtual
> >    clock is the part of the virtual machine state
> >  * modification of block layer to support automatic creation of the overlay
> >    files to store the changes and snapshots while recording
> >  * disabling of system reset while loading VM state to avoid generating of
> >    interrupts by reset handlers
> >  * adding warpers for clock and time functions to save their return
> >    values in the log
> >  * saving different asynchronous events (e.g. system shutdown) into the log
> >  * synchronization of the bottom halves execution
> >  * synchronization of the threads from thread pool
> >  * recording/replaying user input (mouse and keyboard), input from virtual
> >    serial ports, incoming network packets, input from connected USB devices
> >  * adding HMP/QMP commands to monitor for controlling replay execution
> >
> > v2 changes:
> >  * Patches are split to be reviewable and bisectable (as suggested by 
> > Kirill Batuzov)
> >  * Added QMP versions of replay commands (as suggested by Eric Blake)
> >  * Removed some optional features of replay to make patches cleaner
> >  * Minor changes and code cleanup were made
> >
> > ---
> >
> > Pavel Dovgalyuk (49):
> >       acpi: accurate overflow check
> >       integratorcp: adding vmstate for save/restore
> >       pcspk: adding vmstate for save/restore
> >       fdc: adding vmstate for save/restore
> >       parallel: adding vmstate for save/restore
> >       serial: fixing vmstate for save/restore
> >       kvmapic: fixing loading vmstate
> >       hpet: fixing saving and loading process
> >       pckbd: adding new fields to vmstate
> >       rtl8139: adding new fields to vmstate
> >       piix: do not raise irq while loading vmstate
> >       mc146818rtc: add missed field to vmstate
> >       pl031: add missed field to vmstate
> >       ide pci: reset status field before loading the vmstate
> >       softmmu: fixing usage of cpu_st/ld* from helpers
> >       target: save cpu state fields
> >       target-i386: update fp status fix
> >       migration: add vmstate for int8 and char arrays
> >       replay: global variables and function stubs
> >       block: add suffix parameter to bdrv_open functions
> >       sysemu: system functions for replay
> >       replay: internal functions for replay log
> >       cpu: invent instruction count for accurate replay
> >       target-arm: instructions counting code for replay
> >       target-i386: instructions counting code for replay
> >       replay: interrupts and exceptions
> >       vga: do not use virtual clock for blinking cursor
> >       replay: asynchronous events infrastructure
> >       replay: recording and replaying clock ticks
> >       replay: recording and replaying different timers
> >       replay: shutdown event
> >       replay: checkpoints
> >       replay: bottom halves
> >       replay: replay aio requests
> >       replay: thread pool
> >       pl031: vmstate in replay mode
> >       replay: initialization and deinitialization
> >       replay: command line options
> >       replay: snapshotting the virtual machine
> >       replay: recording of the user input
> >       tap-win32: destroy the thread at exit
> >       replay: network packets record/replay
> >       replay: audio data record/replay
> >       replay: serial port
> >       replay: USB passthrough
> >       replay: replay_info command
> >       replay: replay_break command
> >       replay: replay_seek_step command
> >       gdbstub: reverse debugging
> >
> >
> >  Makefile.target                  |    1
> >  arch_init.c                      |    8
> >  async.c                          |   45 ++
> >  audio/audio.c                    |   14 +
> >  audio/audio_win_int.h            |    3
> >  audio/winwaveaudio.c             |  167 +++++++--
> >  block.c                          |  134 ++++++-
> >  block/blkdebug.c                 |    2
> >  block/blkverify.c                |    4
> >  block/cow.c                      |    2
> >  block/qcow.c                     |    2
> >  block/qcow2.c                    |   10 -
> >  block/qed.c                      |    2
> >  block/raw-posix.c                |    6
> >  block/raw-win32.c                |    4
> >  block/sheepdog.c                 |    4
> >  block/vmdk.c                     |    8
> >  block/vvfat.c                    |    2
> >  blockdev.c                       |   11 -
> >  cpu-exec.c                       |   34 +-
> >  cpus.c                           |   81 ++++
> >  dma-helpers.c                    |   10 -
> >  exec.c                           |   12 +
> >  gdbstub.c                        |   79 +++-
> >  hmp-commands.hx                  |   41 ++
> >  hw/acpi/core.c                   |    7
> >  hw/arm/integratorcp.c            |   38 ++
> >  hw/audio/pcspk.c                 |   19 +
> >  hw/block/fdc.c                   |   11 -
> >  hw/block/virtio-blk.c            |   10 -
> >  hw/char/parallel.c               |   22 +
> >  hw/char/serial.c                 |  115 ++++--
> >  hw/display/vga.c                 |    4
> >  hw/i386/kvmvapic.c               |   22 +
> >  hw/ide/ahci.c                    |    4
> >  hw/ide/atapi.c                   |    9
> >  hw/ide/core.c                    |   18 +
> >  hw/ide/pci.c                     |   10 +
> >  hw/input/pckbd.c                 |    3
> >  hw/intc/apic_common.c            |    5
> >  hw/net/rtl8139.c                 |    5
> >  hw/pci-host/piix.c               |   22 +
> >  hw/timer/arm_timer.c             |    2
> >  hw/timer/hpet.c                  |   13 -
> >  hw/timer/mc146818rtc.c           |    5
> >  hw/timer/pl031.c                 |   35 +-
> >  hw/usb/hcd-uhci.c                |    2
> >  hw/usb/host-libusb.c             |  525 ++++++++++++++++++---------
> >  include/block/aio.h              |   17 +
> >  include/block/block.h            |   21 +
> >  include/block/thread-pool.h      |    4
> >  include/exec/cpu-defs.h          |    1
> >  include/exec/cpu_ldst_template.h |   28 +
> >  include/exec/exec-all.h          |   31 ++
> >  include/hw/host-libusb.h         |  105 +++++
> >  include/migration/vmstate.h      |   13 +
> >  include/qemu-common.h            |    3
> >  include/qemu/main-loop.h         |    1
> >  include/qemu/timer.h             |   48 ++
> >  include/qom/cpu.h                |    4
> >  include/sysemu/char.h            |   25 +
> >  include/sysemu/cpus.h            |    1
> >  include/sysemu/sysemu.h          |    2
> >  include/ui/input.h               |    2
> >  main-loop.c                      |    5
> >  monitor.c                        |   49 +++
> >  net/Makefile.objs                |    1
> >  net/clients.h                    |    3
> >  net/dump.c                       |    6
> >  net/hub.c                        |    1
> >  net/net-replay.c                 |   66 +++
> >  net/net.c                        |    7
> >  net/slirp.c                      |   14 +
> >  net/socket.c                     |   35 ++
> >  net/tap-win32.c                  |   25 +
> >  net/tap.c                        |   23 +
> >  net/vde.c                        |   14 +
> >  qapi-schema.json                 |   62 +++
> >  qemu-char.c                      |   55 +++
> >  qemu-img.c                       |    6
> >  qemu-io-cmds.c                   |    2
> >  qemu-io.c                        |    4
> >  qemu-nbd.c                       |    2
> >  qemu-options.hx                  |   28 +
> >  qemu-timer.c                     |   45 ++
> >  qmp-commands.hx                  |   60 +++
> >  replay/Makefile.objs             |   11 +
> >  replay/replay-audio.c            |  228 ++++++++++++
> >  replay/replay-char.c             |   99 +++++
> >  replay/replay-debug.c            |  148 ++++++++
> >  replay/replay-events.c           |  381 ++++++++++++++++++++
> >  replay/replay-input.c            |  107 ++++++
> >  replay/replay-internal.c         |  159 ++++++++
> >  replay/replay-internal.h         |  224 ++++++++++++
> >  replay/replay-net.c              |  190 ++++++++++
> >  replay/replay-qmp.c              |   57 +++
> >  replay/replay-time.c             |  181 +++++++++
> >  replay/replay-usb.c              |  188 ++++++++++
> >  replay/replay.c                  |  735 
> > ++++++++++++++++++++++++++++++++++++++
> >  replay/replay.h                  |  218 +++++++++++
> >  savevm.c                         |   32 +-
> >  slirp/slirp.c                    |    9
> >  softmmu_template.h               |   18 +
> >  stubs/Makefile.objs              |    1
> >  stubs/replay.c                   |   42 ++
> >  target-arm/Makefile.objs         |    1
> >  target-arm/helper.h              |    3
> >  target-arm/machine.c             |    5
> >  target-arm/replay_helper.c       |   38 ++
> >  target-arm/translate.c           |   62 +++
> >  target-i386/Makefile.objs        |    1
> >  target-i386/cpu.c                |    1
> >  target-i386/cpu.h                |    8
> >  target-i386/fpu_helper.c         |    5
> >  target-i386/helper.h             |    3
> >  target-i386/machine.c            |    7
> >  target-i386/replay_helper.c      |   38 ++
> >  target-i386/translate.c          |   91 ++++-
> >  tests/test-thread-pool.c         |    7
> >  thread-pool.c                    |   53 ++-
> >  trace-events                     |    2
> >  translate-all.c                  |   15 +
> >  ui/input.c                       |   79 +++-
> >  util/iov.c                       |    4
> >  vl.c                             |  108 +++++-
> >  vmstate.c                        |    6
> >  126 files changed, 5433 insertions(+), 543 deletions(-)
> >  create mode 100755 include/hw/host-libusb.h
> >  create mode 100755 net/net-replay.c
> >  create mode 100755 replay/Makefile.objs
> >  create mode 100755 replay/replay-audio.c
> >  create mode 100755 replay/replay-char.c
> >  create mode 100755 replay/replay-debug.c
> >  create mode 100755 replay/replay-events.c
> >  create mode 100755 replay/replay-input.c
> >  create mode 100755 replay/replay-internal.c
> >  create mode 100755 replay/replay-internal.h
> >  create mode 100755 replay/replay-net.c
> >  create mode 100755 replay/replay-qmp.c
> >  create mode 100755 replay/replay-time.c
> >  create mode 100755 replay/replay-usb.c
> >  create mode 100755 replay/replay.c
> >  create mode 100755 replay/replay.h
> >  create mode 100755 stubs/replay.c
> >  create mode 100755 target-arm/replay_helper.c
> >  create mode 100755 target-i386/replay_helper.c
> >