Re: [Qemu-devel] CPU hotplug

[David Gibson]

On Tue, Feb 02, 2016 at 04:33:27PM -0200, Eduardo Habkost wrote:
> On Mon, Feb 01, 2016 at 04:35:17PM +1100, David Gibson wrote:
> > Hi,
> > 
> > It seems to me we're getting rather bogged down in how to proceed with
> > an improved CPU hotplug (and hot unplug) interface, both generically
> > and for ppc in particular.
> > 
> > So here's a somewhat more concrete suggestion of a way forward, to see
> > if we can get some consensus.
> > 
> > The biggest difficulty I think we're grappling with is that device-add
> > is actually *not* a great interface to cpu hotplug.  Or rather, it's
> > not great as the _only_ interface: in order to represent the many
> > different constraints on how cpus can be plugged on various platforms,
> > it's natural to use a heirarchy of cpu core / socket / package types
> > specific to the specific platform or real-world cpu package being
> > modeled.  However, for the normal case of a regular homogenous (and at
> > least slightly para-virtualized) server, that interface is nasty for
> > management layers because they have to know the right type to
> > instantiate.
> > 
> > To address this, I'm proposing this two layer interface:
> > 
> > Layer 1: Low-level, device-add based
> > 
> >     * a new, generic cpu-package QOM type represents a group of 1 or
> >       more cpu threads which can be hotplugged as a unit
> >     * cpu-package is abstract and can't be instantiated directly
> >     * archs and/or individual platforms have specific subtypes of
> >       cpu-package which can be instantiated
> >     * for platforms attempting to be faithful representations of real
> >       hardware these subtypes would match the specific characteristics
> >       of the real hardware devices.  In addition to the cpu threads,
> >       they may have other on chip devices as sub-objects.
> >     * for platforms which are paravirtual - or which have existing
> >       firmware abstractions for cpu cores/sockets/packages/whatever -
> >       these could be more abstract, but would still be tied to that
> >       platform's constraints
> >     * Depending on the platform the cpu-package object could have
> >       further internal structure (e.g. a package object representing a
> >       socket contains package objects representing each core, which in
> >       turn contain cpu objects for each thread)
> >         * Some crazy platform that has multiple daughterboards each with
> >           several multi-chip-modules each with several chips, each
> >       with several cores each with several threads could represent
> >       that too.
> 
> What exactly in this approach makes it depend on device-add? We
> could have something very similar based on creation of QOM
> objects, for example.

Uh.. I guess it doesn't.  device_add just seemed the obvious thing to
me.

> > What would be common to all the cpu-package subtypes is:
> >     * A boolean "present" attribute ("realized" might already be
> >       suitable, but I'm not certain)
> 
> "realized" might be suitable, but I am not even sure we want
> cpu-package to be a TYPE_DEVICE subclass. It could be a simple
> QOM class or even a QOM interface (machines could choose to
> implement it as TYPE_DEVICE, or not).

Yeah, I think doing it as a QOM interface makes sense.

> >     * A generic means of determining the number of cpu threads in the
> >       package, and enumerating those
> 
> This could be based on QOM links.

Yes, that makes sense.

> >     * A generic means of determining if the package is hotpluggable or
> >       not
> 
> Isn't this a machine attribute, instead of a package attribute?

Not necessarily.  I was thinking of cases where for architectural
reasons you can't hotplug chip/cpu/module 0 but can plug or unplug all
the rest.

> >     * They'd get listed in a standard place in the QOM tree
> 
> If we allow CPU thread enumeration and package enumeration be
> based in QOM links, we can let machines implement those
> interfaces without introducing QOM hierarchy requirements.
> 
> We have one example where we would need to make this flexible
> enough about QOM hierarchy, below (thread-based hotplug in x86).

Makes sense.

> 
> > 
> > This interface is suitable if you want complete control over
> > constructing the system, including weird cases like heterogeneous
> > machines (either totally different cpu types, or just different
> > numbers of threads in different packages).
> > 
> > The intention is that these objects would never look at the global cpu
> > type or sockets/cores/threads numbers.  The next level up would
> > instead configure the packages to match those for the common case.
> > 
> > Layer 2: Higher-level
> > 
> >     * not all machine types need support this model, but I'd expect
> >       all future versions of machine types designed for production use
> >       to do so
> >     * machine types don't construct cpu objects directly
> >     * instead they create enough cpu-package objects - of a subtype
> >       suitable for this machine - to provide maxcpus threads
> >     * the machine type would set the "present" bit on enough of the
> >       cpu packages to provide the base number of cpu threads
> 
> Sounds interesting, and very simple for management code. What I
> don't see is: what exactly makes it easier to implement just
> Layer 1 and not Layer 2?


> Implementing Layer 1 looks more difficult to me, because it
> requires supporting creation of cpu-package objects on the fly,
> using device_add (or whatever mechanism we choose for cpu-package
> creation). Layer 2 lets the implementation choose freely when/how
> exactly the other objects will be created and how exactly they
> will appear in the device tree. They just need to do the right
> thing when the "present" property is flipped.

Hmm.. good point.  Ok, how about this revised plan:

  1. Implement the QOM backend structures for cpu packages, but don't
     allow them to be user instantiated
  2. Implement Layer 2 in terms of (1)
  3. When/if we need it, add the extra stuff necessary to allow direct
     instantiation of the cpu packages

> > Management layers can then manage hotplug without knowing platform
> > specifics by using qmp to toggle the "present" bit on packages.
> > Platforms that allow thread-level pluggability can expose a package
> > for every thread, those that allow core-level expose a package per
> > core, those that have even less granularity expose a package at
> > whatever grouping they can do hotplug on.
> > 
> > Examples:
> > 
> > For use with pc (or q35 or whatever) machine type, I'd expect a
> > cpu-package subtype called, say "acpi-thread" which represents a
> > single thread in the ACPI sense.  Toggling those would trigger ACPI
> > hotplug events as cpu_add does now.
> 
> You have a good point here: I remember seeing suggestions of
> making CPU hotplug tied to the socket/core/thread hierarchy
> somehow. But this won't change the fact that x86 allows hotplug
> of individual CPU threads.

Right, this seems to be where we're bogged down - we seem to be going
back and forth betweeh core level, socket level, thread level
proposals without really looking at the big picture to come up with a
scheme that works for all platforms.

> In other words, if we make a /machine/socket/core/thread QOM
> hierarchy, the cpu-packages for x86 won't necessarily represent
> CPU sockets (but in other architectures, they might).  The
> interface needs to be generic enough to not assume anything about
> the CPU topology level where CPU hotplug happens.

Exactly.  That's what I see as the key advantage of this proposal over
earlier ones.

> > For use with pseries, I'd expect a "papr-core" cpu-package subtype,
> > which represents a single (paravirtual) core.  Toggling present on
> > this would trigger the PAPR hotplug events.  A property would control
> > the number of threads in the core (only settable before enabling
> > present).
> > 
> > For use with the powernv machine type (once ready for merge) I'd
> > expect "POWER8-package" type which represents a POWER8 chip / module
> > as close to the real hardware as we can get.  It would have a fixed
> > number of cores and threads within it as per the real hardware, and
> > would also include xscoms and other per-module logic.
> > 
> > From here to there:
> > 
> > A suggested order of implementation to get there without too much risk
> > of breaking things.
> > 
> >   1. Fix bugs with creation / removal of CPU objects (Bharata's cpu
> >      hotplug series already has this)
> >   2. Split creation and realization of CPU objects, so machine types
> >      must explicitly perform both steps (Bharata's series has this
> >      too)
> >   3. Add the abstract cpu-package type, and define the generic
> >      interfaces it needs (Bharata's series has something that could be
> >      changed to this fairly easily)
> >   4. For each machine type we care to convert:
> >       4.1. Add platform suitable cpu-package subtypes
> >       4.2. Convert the (latest version) machine type to instantiate 
> > packages instead of
> >            cpu threads directly
> 
> Machines could even have the freedom to instantiate CPU threads
> directly and then set up package objects for them. Reusing
> generic code is useful, but it doesn't even need to be mandatory,
> as long as the objects are available at the right place in the
> QOM hierarchy.

Ah, yes, I guess so.

> 
> >       4.3. Add any necessary backwards compat goo
> >   5. Teach libvirt how to toggle cpu-packages
> 
> This is different from the very flexible QOM object
> building/linking approach Andreas was talking about in last KVM
> Forum.

So, I know this stuff was discussed at KVM Forum, but unfortunately I
never got a clear picture of what the outcome was.

> But while I would love to have the ability to build
> arbitrary QOM hierarchies with complex links between CPUs
> sockets, cores, threads, etc, I believe we need an interface that
> is: 1) generic enough for multiple architectures and machines to
> implement them; 2) simple enough so that libvirt can use it
> easily without requiring more arch-specific code.
> 
> Also, your approach doesn't prevent the simple cpu-package
> interface from having a complex QOM hierarchy hidden behind it.

Exactly.


-- 
David Gibson                    | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
                                | _way_ _around_!
http://www.ozlabs.org/~dgibson

signature.asc
Description: PGP signature