[PATCH v5 00/50] Initial support for multi-process qemu

[Jagannathan Raman]

Hello
  
Started with the presentation in October 2017 made by Marc-Andre (Red Hat)
and Konrad Wilk (Oracle) [1], and continued by Jag's BoF at KVM Forum 2018,
the multi-process project is now available and presented in this patchset.
This first series enables the emulation of lsi53c895a in a separate process.

We posted the Proof Of Concept patches [2] before the BoF session in 2018.
Subsequently, we posted RFC v1 [3], RFC v2 [4], RFC v3 [5] and RFC v4 [6].

John & Elena presented the status of this project in KVM Forum 2019. We
appreciate the in-person and email feedback we received to improve this
patchset. We also received valuable feedback and direction on future
improvements from the bi-weekly KVM community conference. We have
incorporated all the feedback in the current version of the series, v5.

Following people contributed to this patchset:

John G Johnson <address@hidden>
Jagannathan Raman <address@hidden>
Elena Ufimtseva <address@hidden>
Kanth Ghatraju <address@hidden>
Konrad Wilk <address@hidden>

For full concept writeup about QEMU disaggregation, refer to
docs/devel/qemu-multiprocess.rst. Please refer to
docs/qemu-multiprocess.txt for usage information.

We are planning on making the following improvements in the future to the 
experimental
Qemu multi-process:
 - Asynchronous communication channel;
 - Performance improvements;
 - Libvirt support;
 - Enforcement of security policies and privileges control;

We welcome all your ideas, concerns, and questions for this patchset.

Thank you!


[1]: 
http://events17.linuxfoundation.org/sites/events/files/slides/KVM%20FORUM%20multi-process.pdf
[1]: https://www.youtube.com/watch?v=Kq1-coHh7lg
[2]: https://www.mail-archive.com/address@hidden/msg566538.html
[3]: https://www.mail-archive.com/address@hidden/msg602285.html
[4]: https://www.mail-archive.com/address@hidden/msg624877.html
[5]: https://www.mail-archive.com/address@hidden/msg642000.html
[6]: https://www.mail-archive.com/address@hidden/msg655118.html

Elena Ufimtseva (23):
  multi-process: add a command line option for debug file
  multi-process: introduce proxy object
  mutli-process: build remote command line args
  multi-process: configure remote side devices
  multi-process: add qdev_proxy_add to create proxy devices
  multi-process: remote: add setup_devices msg processing
  multi-process: remote: use fd for socket from parent process
  multi-process: remote: add create_done condition
  multi-process: add processing of remote device command line
  multi-process: refractor vl.c code
  multi-process: add remote option
  multi-process: add remote options parser
  multi-process: add parse_cmdline in remote process
  multi-process: send heartbeat messages to remote
  multi-process: handle heartbeat messages in remote process
  multi-process/mon: choose HMP commands based on target
  multi-process/mig: Load VMSD in the proxy object
  multi-process/mig: refactor runstate_check into common file
  multi-process/mig: Synchronize runstate of remote process
  multi-process/mig: Restore the VMSD in remote process
  multi-process: Enable support for multiple devices in remote
  multi-process: Validate incoming commands from Proxy
  multi-process: add configure and usage information

Jagannathan Raman (26):
  multi-process: memory: alloc RAM from file at offset
  multi-process: Refactor machine_init and exit notifiers
  multi-process: Add stub functions to facilate build of multi-process
  multi-process: Add config option for multi-process QEMU
  multi-process: build system for remote device process
  multi-process: define mpqemu-link object
  multi-process: add functions to synchronize proxy and remote endpoints
  multi-process: setup PCI host bridge for remote device
  multi-process: setup a machine object for remote device process
  multi-process: setup memory manager for remote device
  multi-process: remote process initialization
  multi-process: PCI BAR read/write handling for proxy & remote
    endpoints
  multi-process: Synchronize remote memory
  multi-process: create IOHUB object to handle irq
  multi-process: Retrieve PCI info from remote process
  multi-process: Introduce build flags to separate remote process code
  multi-process: Use separate MMIO communication channel
  multi-process: perform device reset in the remote process
  multi-process/mon: stub functions to enable QMP module for remote
    process
  multi-process/mon: enable QMP module support in the remote process
  multi-process/mon: Refactor monitor/chardev functions out of vl.c
  multi-process/mon: Initialize QMP module for remote processes
  multi-process: prevent duplicate memory initialization in remote
  multi-process/mig: build migration module in the remote process
  multi-process/mig: Enable VMSD save in the Proxy object
  multi-process/mig: Send VMSD of remote to the Proxy object

John G Johnson (1):
  multi-process: add the concept description to
    docs/devel/qemu-multiprocess

 Makefile                         |    2 +
 Makefile.objs                    |   45 ++
 Makefile.target                  |  105 +++-
 accel/Makefile.objs              |    2 +
 accel/stubs/kvm-stub.c           |    5 +
 accel/stubs/tcg-stub.c           |  107 ++++
 backends/Makefile.objs           |    2 +
 block/Makefile.objs              |    2 +
 chardev/char.c                   |   14 +
 configure                        |   15 +
 docs/devel/index.rst             |    1 +
 docs/devel/qemu-multiprocess.rst | 1102 ++++++++++++++++++++++++++++++++++++++
 docs/qemu-multiprocess.txt       |   86 +++
 exec.c                           |   31 +-
 hmp-commands-info.hx             |   10 +
 hmp-commands.hx                  |   25 +-
 hw/Makefile.objs                 |    7 +
 hw/block/Makefile.objs           |    2 +
 hw/core/Makefile.objs            |   19 +
 hw/nvram/Makefile.objs           |    2 +
 hw/pci/Makefile.objs             |    4 +
 hw/proxy/memory-sync.c           |  212 ++++++++
 hw/proxy/qemu-proxy.c            |  906 +++++++++++++++++++++++++++++++
 hw/scsi/Makefile.objs            |    2 +
 include/chardev/char.h           |    1 +
 include/exec/address-spaces.h    |    2 +
 include/exec/ram_addr.h          |    4 +-
 include/hw/pci/pci_ids.h         |    3 +
 include/hw/proxy/memory-sync.h   |   37 ++
 include/hw/proxy/qemu-proxy.h    |  107 ++++
 include/hw/qdev-core.h           |    2 +
 include/io/mpqemu-link.h         |  218 ++++++++
 include/monitor/monitor.h        |    2 +
 include/monitor/qdev.h           |   26 +
 include/qemu-common.h            |    8 +
 include/qemu/log.h               |    1 +
 include/qemu/mmap-alloc.h        |    3 +-
 include/remote/iohub.h           |   50 ++
 include/remote/machine.h         |   32 ++
 include/remote/memory.h          |   20 +
 include/remote/pcihost.h         |   45 ++
 include/sysemu/runstate.h        |    3 +
 include/sysemu/sysemu.h          |    2 +
 io/Makefile.objs                 |    2 +
 io/mpqemu-link.c                 |  410 ++++++++++++++
 memory.c                         |    2 +-
 migration/Makefile.objs          |   13 +
 migration/savevm.c               |   63 +++
 migration/savevm.h               |    3 +
 monitor/Makefile.objs            |    4 +
 monitor/misc.c                   |   84 +--
 monitor/monitor-internal.h       |   38 ++
 monitor/monitor.c                |   40 +-
 net/Makefile.objs                |    2 +
 qapi/Makefile.objs               |    2 +
 qdev-monitor.c                   |  259 ++++++++-
 qemu-options.hx                  |   21 +
 qom/Makefile.objs                |    4 +
 remote/Makefile.objs             |    6 +
 remote/iohub.c                   |  148 +++++
 remote/machine.c                 |   99 ++++
 remote/memory.c                  |   63 +++
 remote/pcihost.c                 |   64 +++
 remote/remote-main.c             |  625 +++++++++++++++++++++
 remote/remote-opts.c             |  115 ++++
 remote/remote-opts.h             |   15 +
 rules.mak                        |    2 +-
 runstate.c                       |   41 ++
 scripts/hxtool                   |   44 +-
 softmmu/vl.c                     |  299 ++++-------
 stubs/audio.c                    |   12 +
 stubs/gdbstub.c                  |   23 +
 stubs/machine-init-done.c        |    4 +
 stubs/migration.c                |  211 ++++++++
 stubs/monitor.c                  |   72 +++
 stubs/net-stub.c                 |  121 +++++
 stubs/qapi-misc.c                |   43 ++
 stubs/qapi-target.c              |   55 ++
 stubs/replay.c                   |   26 +
 stubs/runstate-check.c           |    3 +
 stubs/ui-stub.c                  |  130 +++++
 stubs/vl-stub.c                  |  195 +++++++
 stubs/vmstate.c                  |   19 +
 stubs/xen-mapcache.c             |   22 +
 ui/Makefile.objs                 |    2 +
 util/Makefile.objs               |    2 +
 util/log.c                       |    2 +
 util/mmap-alloc.c                |    7 +-
 util/notify.c                    |   43 ++
 util/oslib-posix.c               |    2 +-
 vl-parse.c                       |  164 ++++++
 vl.h                             |   43 ++
 92 files changed, 6698 insertions(+), 245 deletions(-)
 create mode 100644 docs/devel/qemu-multiprocess.rst
 create mode 100644 docs/qemu-multiprocess.txt
 create mode 100644 hw/proxy/memory-sync.c
 create mode 100644 hw/proxy/qemu-proxy.c
 create mode 100644 include/hw/proxy/memory-sync.h
 create mode 100644 include/hw/proxy/qemu-proxy.h
 create mode 100644 include/io/mpqemu-link.h
 create mode 100644 include/remote/iohub.h
 create mode 100644 include/remote/machine.h
 create mode 100644 include/remote/memory.h
 create mode 100644 include/remote/pcihost.h
 create mode 100644 io/mpqemu-link.c
 create mode 100644 remote/Makefile.objs
 create mode 100644 remote/iohub.c
 create mode 100644 remote/machine.c
 create mode 100644 remote/memory.c
 create mode 100644 remote/pcihost.c
 create mode 100644 remote/remote-main.c
 create mode 100644 remote/remote-opts.c
 create mode 100644 remote/remote-opts.h
 create mode 100644 runstate.c
 mode change 100644 => 100755 scripts/hxtool
 create mode 100644 stubs/audio.c
 create mode 100644 stubs/migration.c
 create mode 100644 stubs/net-stub.c
 create mode 100644 stubs/qapi-misc.c
 create mode 100644 stubs/qapi-target.c
 create mode 100644 stubs/ui-stub.c
 create mode 100644 stubs/vl-stub.c
 create mode 100644 stubs/xen-mapcache.c
 create mode 100644 vl-parse.c
 create mode 100644 vl.h

-- 
1.8.3.1