[Qemu-ppc] [RFC PATCH v3 00/24] CPU and Memory hotplug for PowerPC sPAPR guests

[Bharata B Rao]

This is v3 of the CPU and Memory hotplug patches for PowerPC sPAPR.

- CPU hotplug implementation here is based on device_add command and has
  nothing to do with the existing cpu-add QEMU monitor command.
  (qemu) device_add powerpc64-cpu-socket,id=sock1
  (qemu) device_del sock1
- This version adds a full socket in response to device_add. Though for
  PowerPC, the term socket really doesn't make sense, it is used as a
  container of CPU cores here. Unless sockets= option is explicitly specified,
  QEMU assigns one core per socket by default for PowerPC and the hotplug
  implementation too sticks to this behaviour.
- Currently hotplugging a socket populates the cores and threads based on
  the topology specified at boot time. i,e., there isn't a way to have add
  partially populated sockets resulting in non-homogeneous configurations.
  I have kept it this way since I don't want to invent and implement
  any new semantics that hasn't been agreed upon.
- Andreas has been mentioning about having link<> properties for the machine
  defined at boot time and populated at hotplug time by CPU objects that get
  created. I am not following that here as its need wasn't felt.
- The CPU hotplug semantics discussion is still underway but I am posting
  this version with v2 review comments addressed and more fixes incorporated.

Changes in v3
-------------
CPU
---
- Minor reworks on CPU device tree code reorganzation. Refer to patch
  description in 05/24 to see how the DT generation code flow looks like
  from different call sites.
- Don't create ibm,my-drc-index property when CPU DR isn't enabled. (12/24)
- Enable reuse of device ID string after unplug by releasing core and socket
  object when a vCPU is destroyed. (17/24) (Removed David's Reviewed-by
  from 17/24 since the patch got changed)
- Releasing cores and sockets during vcpu destroy needed a new QOM API (16/24).
- Deduce socket, core and thread numbers correctly even when none or only
  some of them are explicitly specified. (11/24) 
- Remove vCPU thread only after sending the unplug notification to guest and
  getting ACK from it. (20/24)
- Add CPU device coldplug support which allows CPU to be specified on QEMU
  cmdline like -device powerpc64-cpu-socket,id=sock1. This is in addition
  to the CPUs that are specified using -smp X. Coldplug support is needed
  for migration. (12/24)
- Setup CPU DR connectors early before boot CPUs are setup. This is needed
  to support coldplug. (02/24 - Removed David's Reviewed-by since the patch
  got changed)
- Ensure hot and cold plug of CPUs are handled correctly for machines where
  CPU DR isn't enabled. (12/24)
- Ensure NUMA node information is set for cold pluggged CPUs too. (12/24)

Memory
------
- Use ldl_phys instead of rtas_ld as suggested by David. (23/24)
- Support cold plugging of memory device via QEMU cmdline like
  -object memory-backend-ram,id=ram1,size=1G -device pc-dimm,id=dimm,memdev=ram1
  Ensure SPAPR_LMB_FLAGS_ASSIGNED flags are set for cold-plugged memory
  too. Cold plugging support enables migration. (23/24)
- New API to lookup NUMA node by address. (22/24)
- Removed unused enforced-aligned-dimm property.
- Enforce memory hotplug to node 0 only since pseries kernel still doesn't
  allow updation of associativity index for hotplugged LMBs. (24/24)
- Do vm_unregister_ram() and memory_region_del_subregion() when hot adding
  of memory fails. (24/24)
- Fail hotplug and unplug for machine versions that don't support memory
  DR (24/24)
- Resorted to fdt_setprop instead of fdt_setprop64 since the latter isn't
  present yet in the DTC submodule of QEMU. (23/24)
- Use uint64_t for lmb_size so that memory > 4G is handled correctly by
  the guest (23/24)
 
Known issues and limitations
----------------------------
- Hot removal of CPUs in random order breaks migration. Fixing this needs
  invention of new semantics.
- Still not able to fail the hotplug and do proper cleanup for hot plug
  requests on machines which don't support CPU DR. Resorting to silent
  failure without raising an error to the user.
- Memory can be hotplugged to Node 0 only currently. IIUC, to support hotplug
  to other NUMA nodes, guest kernel support for sending configure-connector
  call during hotplug is needed.
- Guest NUMA nodes come up with flat distance after supporting
  ibm,dynamic-reconfiguration-memory node. Still debugging this.
- DRC states are still not migrated, this is needed to support migration
  with hotplug correctly. Without it there are limitations:
        - VM hotplugged with CPUs can be migrated and hotplugged CPUs
          can be removed at the target (supported by a hack)
        - VM hotplugged with memory can be migrated however hot memory
          addition to a migrated VM isn't possible yet.
  These oddities will go away after DRC state migration is supported.
- David's long standing review comment about sharing code b/n x86 and
  PowerPC memory hotplug handlers is yet to be addressed.

Dependencies
------------
- For CPU and Memory hotplug to work, latest powerpc-utils and ppc64-diag
  packages are needed in the guest. version 1.2.25 of powerpc-utils
  and latest git master (unreleased) of ppc64-diag are required.

Previous versions
-----------------
v2: http://lists.nongnu.org/archive/html/qemu-devel/2015-03/msg04737.html
v1: http://lists.gnu.org/archive/html/qemu-devel/2015-01/msg00611.html
v0: http://lists.nongnu.org/archive/html/qemu-devel/2014-09/msg00752.html

Git tree for this implementation
--------------------------------
- These patches apply against spapr-hotplug-pci-v7 branch of Michael Roth's
  PCI hotplug tree (git://github.com/mdroth/qemu)
- The current patchset can be fetched from
  spapr-hotplug branch at https://github.com/bharata/qemu/


Andreas Färber (1):
  cpu: Prepare Socket container type

Bharata B Rao (21):
  spapr: Add DRC dt entries for CPUs
  spapr: Consider max_cpus during xics initialization
  spapr: Support ibm,lrdr-capacity device tree property
  spapr: Reorganize CPU dt generation code
  spapr: Consolidate cpu init code into a routine
  ppc: Prepare CPU socket/core abstraction
  spapr: Add CPU hotplug handler
  ppc: Update cpu_model in MachineState
  ppc: Create sockets and cores for CPUs
  spapr: CPU hotplug support
  cpus: Add Error argument to cpu_exec_init()
  cpus: Convert cpu_index into a bitmap
  ppc: Move cpu_exec_init() call to realize function
  qom: Introduce object_has_no_children() API
  xics_kvm: Don't enable KVM_CAP_IRQ_XICS if already enabled
  xics_kvm: Add cpu_destroy method to XICS
  spapr: CPU hot unplug support
  spapr: Initialize hotplug memory address space
  numa: API to looking NUMA node by address
  spapr: Support ibm,dynamic-reconfiguration-memory
  spapr: Memory hotplug support

Gu Zheng (1):
  cpus, qom: Reclaim vCPU objects

Michael Roth (1):
  spapr: enable PHB/CPU/LMB hotplug for pseries-2.3

 cpus.c                            |   67 +++
 default-configs/ppc64-softmmu.mak |    1 +
 docs/specs/ppc-spapr-hotplug.txt  |   66 +++
 exec.c                            |   39 +-
 hw/cpu/Makefile.objs              |    2 +-
 hw/cpu/socket.c                   |   21 +
 hw/intc/xics.c                    |   12 +
 hw/intc/xics_kvm.c                |   19 +
 hw/ppc/Makefile.objs              |    1 +
 hw/ppc/cpu-core.c                 |   65 +++
 hw/ppc/cpu-socket.c               |   68 +++
 hw/ppc/mac_newworld.c             |   10 +-
 hw/ppc/mac_oldworld.c             |    7 +-
 hw/ppc/ppc440_bamboo.c            |    7 +-
 hw/ppc/prep.c                     |    7 +-
 hw/ppc/spapr.c                    | 1054 ++++++++++++++++++++++++++++++-------
 hw/ppc/spapr_events.c             |   11 +-
 hw/ppc/spapr_hcall.c              |   51 +-
 hw/ppc/spapr_rtas.c               |   29 +-
 hw/ppc/virtex_ml507.c             |    7 +-
 include/exec/exec-all.h           |    2 +-
 include/hw/cpu/socket.h           |   14 +
 include/hw/ppc/cpu-core.h         |   32 ++
 include/hw/ppc/cpu-socket.h       |   32 ++
 include/hw/ppc/spapr.h            |   37 +-
 include/hw/ppc/xics.h             |    3 +
 include/qom/cpu.h                 |   19 +
 include/qom/object.h              |   11 +
 include/sysemu/kvm.h              |    1 +
 include/sysemu/numa.h             |    3 +
 kvm-all.c                         |   57 +-
 kvm-stub.c                        |    5 +
 numa.c                            |   61 +++
 qom/object.c                      |   12 +
 target-alpha/cpu.c                |    8 +-
 target-arm/cpu.c                  |    3 +-
 target-cris/cpu.c                 |    8 +-
 target-i386/cpu.c                 |    8 +-
 target-lm32/cpu.c                 |    8 +-
 target-m68k/cpu.c                 |    8 +-
 target-microblaze/cpu.c           |    8 +-
 target-mips/cpu.c                 |    8 +-
 target-moxie/cpu.c                |    8 +-
 target-openrisc/cpu.c             |    8 +-
 target-ppc/cpu.h                  |    1 +
 target-ppc/translate_init.c       |   65 ++-
 target-s390x/cpu.c                |    3 +-
 target-sh4/cpu.c                  |    8 +-
 target-sparc/cpu.c                |    3 +-
 target-tricore/cpu.c              |    7 +-
 target-unicore32/cpu.c            |    8 +-
 target-xtensa/cpu.c               |    8 +-
 52 files changed, 1747 insertions(+), 264 deletions(-)
 create mode 100644 hw/cpu/socket.c
 create mode 100644 hw/ppc/cpu-core.c
 create mode 100644 hw/ppc/cpu-socket.c
 create mode 100644 include/hw/cpu/socket.h
 create mode 100644 include/hw/ppc/cpu-core.h
 create mode 100644 include/hw/ppc/cpu-socket.h

-- 
2.1.0