[Qemu-ppc] CPU hotplug

[David Gibson]

Hi,

It seems to me we're getting rather bogged down in how to proceed with
an improved CPU hotplug (and hot unplug) interface, both generically
and for ppc in particular.

So here's a somewhat more concrete suggestion of a way forward, to see
if we can get some consensus.

The biggest difficulty I think we're grappling with is that device-add
is actually *not* a great interface to cpu hotplug.  Or rather, it's
not great as the _only_ interface: in order to represent the many
different constraints on how cpus can be plugged on various platforms,
it's natural to use a heirarchy of cpu core / socket / package types
specific to the specific platform or real-world cpu package being
modeled.  However, for the normal case of a regular homogenous (and at
least slightly para-virtualized) server, that interface is nasty for
management layers because they have to know the right type to
instantiate.

To address this, I'm proposing this two layer interface:

Layer 1: Low-level, device-add based

    * a new, generic cpu-package QOM type represents a group of 1 or
      more cpu threads which can be hotplugged as a unit
    * cpu-package is abstract and can't be instantiated directly
    * archs and/or individual platforms have specific subtypes of
      cpu-package which can be instantiated
    * for platforms attempting to be faithful representations of real
      hardware these subtypes would match the specific characteristics
      of the real hardware devices.  In addition to the cpu threads,
      they may have other on chip devices as sub-objects.
    * for platforms which are paravirtual - or which have existing
      firmware abstractions for cpu cores/sockets/packages/whatever -
      these could be more abstract, but would still be tied to that
      platform's constraints
    * Depending on the platform the cpu-package object could have
      further internal structure (e.g. a package object representing a
      socket contains package objects representing each core, which in
      turn contain cpu objects for each thread)
        * Some crazy platform that has multiple daughterboards each with
          several multi-chip-modules each with several chips, each
          with several cores each with several threads could represent
          that too.

What would be common to all the cpu-package subtypes is:
    * A boolean "present" attribute ("realized" might already be
      suitable, but I'm not certain)
    * A generic means of determining the number of cpu threads in the
      package, and enumerating those
    * A generic means of determining if the package is hotpluggable or
      not
    * They'd get listed in a standard place in the QOM tree

This interface is suitable if you want complete control over
constructing the system, including weird cases like heterogeneous
machines (either totally different cpu types, or just different
numbers of threads in different packages).

The intention is that these objects would never look at the global cpu
type or sockets/cores/threads numbers.  The next level up would
instead configure the packages to match those for the common case.

Layer 2: Higher-level

    * not all machine types need support this model, but I'd expect
      all future versions of machine types designed for production use
      to do so
    * machine types don't construct cpu objects directly
    * instead they create enough cpu-package objects - of a subtype
      suitable for this machine - to provide maxcpus threads
    * the machine type would set the "present" bit on enough of the
      cpu packages to provide the base number of cpu threads

Management layers can then manage hotplug without knowing platform
specifics by using qmp to toggle the "present" bit on packages.
Platforms that allow thread-level pluggability can expose a package
for every thread, those that allow core-level expose a package per
core, those that have even less granularity expose a package at
whatever grouping they can do hotplug on.

Examples:

For use with pc (or q35 or whatever) machine type, I'd expect a
cpu-package subtype called, say "acpi-thread" which represents a
single thread in the ACPI sense.  Toggling those would trigger ACPI
hotplug events as cpu_add does now.

For use with pseries, I'd expect a "papr-core" cpu-package subtype,
which represents a single (paravirtual) core.  Toggling present on
this would trigger the PAPR hotplug events.  A property would control
the number of threads in the core (only settable before enabling
present).

For use with the powernv machine type (once ready for merge) I'd
expect "POWER8-package" type which represents a POWER8 chip / module
as close to the real hardware as we can get.  It would have a fixed
number of cores and threads within it as per the real hardware, and
would also include xscoms and other per-module logic.

From here to there:

A suggested order of implementation to get there without too much risk
of breaking things.

  1. Fix bugs with creation / removal of CPU objects (Bharata's cpu
     hotplug series already has this)
  2. Split creation and realization of CPU objects, so machine types
     must explicitly perform both steps (Bharata's series has this
     too)
  3. Add the abstract cpu-package type, and define the generic
     interfaces it needs (Bharata's series has something that could be
     changed to this fairly easily)
  4. For each machine type we care to convert:
      4.1. Add platform suitable cpu-package subtypes
      4.2. Convert the (latest version) machine type to instantiate packages 
instead of
           cpu threads directly
      4.3. Add any necessary backwards compat goo
  5. Teach libvirt how to toggle cpu-packages

-- 
David Gibson                    | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
                                | _way_ _around_!
http://www.ozlabs.org/~dgibson

signature.asc
Description: PGP signature