Re: [Qemu-devel] CPU hotplug

[Eduardo Habkost]

On Wed, Feb 03, 2016 at 12:50:02PM +1100, David Gibson wrote:
> On Tue, Feb 02, 2016 at 04:33:27PM -0200, Eduardo Habkost wrote:
> > On Mon, Feb 01, 2016 at 04:35:17PM +1100, David Gibson wrote:
> > > Hi,
> > > 
> > > It seems to me we're getting rather bogged down in how to proceed with
> > > an improved CPU hotplug (and hot unplug) interface, both generically
> > > and for ppc in particular.
> > > 
> > > So here's a somewhat more concrete suggestion of a way forward, to see
> > > if we can get some consensus.
> > > 
> > > The biggest difficulty I think we're grappling with is that device-add
> > > is actually *not* a great interface to cpu hotplug.  Or rather, it's
> > > not great as the _only_ interface: in order to represent the many
> > > different constraints on how cpus can be plugged on various platforms,
> > > it's natural to use a heirarchy of cpu core / socket / package types
> > > specific to the specific platform or real-world cpu package being
> > > modeled.  However, for the normal case of a regular homogenous (and at
> > > least slightly para-virtualized) server, that interface is nasty for
> > > management layers because they have to know the right type to
> > > instantiate.
> > > 
> > > To address this, I'm proposing this two layer interface:
> > > 
> > > Layer 1: Low-level, device-add based
> > > 
> > >     * a new, generic cpu-package QOM type represents a group of 1 or
> > >       more cpu threads which can be hotplugged as a unit
> > >     * cpu-package is abstract and can't be instantiated directly
> > >     * archs and/or individual platforms have specific subtypes of
> > >       cpu-package which can be instantiated
> > >     * for platforms attempting to be faithful representations of real
> > >       hardware these subtypes would match the specific characteristics
> > >       of the real hardware devices.  In addition to the cpu threads,
> > >       they may have other on chip devices as sub-objects.
> > >     * for platforms which are paravirtual - or which have existing
> > >       firmware abstractions for cpu cores/sockets/packages/whatever -
> > >       these could be more abstract, but would still be tied to that
> > >       platform's constraints
> > >     * Depending on the platform the cpu-package object could have
> > >       further internal structure (e.g. a package object representing a
> > >       socket contains package objects representing each core, which in
> > >       turn contain cpu objects for each thread)
> > >         * Some crazy platform that has multiple daughterboards each with
> > >           several multi-chip-modules each with several chips, each
> > >     with several cores each with several threads could represent
> > >     that too.
> > 
> > What exactly in this approach makes it depend on device-add? We
> > could have something very similar based on creation of QOM
> > objects, for example.
> 
> Uh.. I guess it doesn't.  device_add just seemed the obvious thing to
> me.
> 
> > > What would be common to all the cpu-package subtypes is:
> > >     * A boolean "present" attribute ("realized" might already be
> > >       suitable, but I'm not certain)
> > 
> > "realized" might be suitable, but I am not even sure we want
> > cpu-package to be a TYPE_DEVICE subclass. It could be a simple
> > QOM class or even a QOM interface (machines could choose to
> > implement it as TYPE_DEVICE, or not).
> 
> Yeah, I think doing it as a QOM interface makes sense.
> 
> > >     * A generic means of determining the number of cpu threads in the
> > >       package, and enumerating those
> > 
> > This could be based on QOM links.
> 
> Yes, that makes sense.
> 
> > >     * A generic means of determining if the package is hotpluggable or
> > >       not
> > 
> > Isn't this a machine attribute, instead of a package attribute?
> 
> Not necessarily.  I was thinking of cases where for architectural
> reasons you can't hotplug chip/cpu/module 0 but can plug or unplug all
> the rest.

Right, that makes sense too. But you can't query the cpu-package
object unless it was already created. I guess that's another
reason to start with the Layer 2 solution (where there's no need
for on-the-fly creation of cpu-package objects).

> 
> > >     * They'd get listed in a standard place in the QOM tree
> > 
> > If we allow CPU thread enumeration and package enumeration be
> > based in QOM links, we can let machines implement those
> > interfaces without introducing QOM hierarchy requirements.
> > 
> > We have one example where we would need to make this flexible
> > enough about QOM hierarchy, below (thread-based hotplug in x86).
> 
> Makes sense.
> 
> > 
> > > 
> > > This interface is suitable if you want complete control over
> > > constructing the system, including weird cases like heterogeneous
> > > machines (either totally different cpu types, or just different
> > > numbers of threads in different packages).
> > > 
> > > The intention is that these objects would never look at the global cpu
> > > type or sockets/cores/threads numbers.  The next level up would
> > > instead configure the packages to match those for the common case.
> > > 
> > > Layer 2: Higher-level
> > > 
> > >     * not all machine types need support this model, but I'd expect
> > >       all future versions of machine types designed for production use
> > >       to do so
> > >     * machine types don't construct cpu objects directly
> > >     * instead they create enough cpu-package objects - of a subtype
> > >       suitable for this machine - to provide maxcpus threads
> > >     * the machine type would set the "present" bit on enough of the
> > >       cpu packages to provide the base number of cpu threads
> > 
> > Sounds interesting, and very simple for management code. What I
> > don't see is: what exactly makes it easier to implement just
> > Layer 1 and not Layer 2?
> 
> 
> > Implementing Layer 1 looks more difficult to me, because it
> > requires supporting creation of cpu-package objects on the fly,
> > using device_add (or whatever mechanism we choose for cpu-package
> > creation). Layer 2 lets the implementation choose freely when/how
> > exactly the other objects will be created and how exactly they
> > will appear in the device tree. They just need to do the right
> > thing when the "present" property is flipped.
> 
> Hmm.. good point.  Ok, how about this revised plan:
> 
>   1. Implement the QOM backend structures for cpu packages, but don't
>      allow them to be user instantiated
>   2. Implement Layer 2 in terms of (1)
>   3. When/if we need it, add the extra stuff necessary to allow direct
>      instantiation of the cpu packages

Makes sense to me. With generic mechanisms to let management
enumerate the socket/core/thread IDs contained/available in each
cpu-package slot, this should be good enough for all
architectures and still give management enough information to
decide how to plug/unplug specific CPUs.

I have another suggestion: if we are going to expose these
objects where new CPUs could be plugged by just flipping a
property, what about naming them "cpu-slot"? Because they
wouldn't represent an actual CPU package, but just a slot where a
CPU package could be plugged in. The initial (simpler) solution
could involve just flipping a property in the cpu-slot.  Later,
we may allow more complex scenarios where complex QOM objects are
created before linking them to the cpu-slot.

> 
> > > Management layers can then manage hotplug without knowing platform
> > > specifics by using qmp to toggle the "present" bit on packages.
> > > Platforms that allow thread-level pluggability can expose a package
> > > for every thread, those that allow core-level expose a package per
> > > core, those that have even less granularity expose a package at
> > > whatever grouping they can do hotplug on.
> > > 
> > > Examples:
> > > 
> > > For use with pc (or q35 or whatever) machine type, I'd expect a
> > > cpu-package subtype called, say "acpi-thread" which represents a
> > > single thread in the ACPI sense.  Toggling those would trigger ACPI
> > > hotplug events as cpu_add does now.
> > 
> > You have a good point here: I remember seeing suggestions of
> > making CPU hotplug tied to the socket/core/thread hierarchy
> > somehow. But this won't change the fact that x86 allows hotplug
> > of individual CPU threads.
> 
> Right, this seems to be where we're bogged down - we seem to be going
> back and forth betweeh core level, socket level, thread level
> proposals without really looking at the big picture to come up with a
> scheme that works for all platforms.
> 
> > In other words, if we make a /machine/socket/core/thread QOM
> > hierarchy, the cpu-packages for x86 won't necessarily represent
> > CPU sockets (but in other architectures, they might).  The
> > interface needs to be generic enough to not assume anything about
> > the CPU topology level where CPU hotplug happens.
> 
> Exactly.  That's what I see as the key advantage of this proposal over
> earlier ones.

Right.

> 
> > > For use with pseries, I'd expect a "papr-core" cpu-package subtype,
> > > which represents a single (paravirtual) core.  Toggling present on
> > > this would trigger the PAPR hotplug events.  A property would control
> > > the number of threads in the core (only settable before enabling
> > > present).
> > > 
> > > For use with the powernv machine type (once ready for merge) I'd
> > > expect "POWER8-package" type which represents a POWER8 chip / module
> > > as close to the real hardware as we can get.  It would have a fixed
> > > number of cores and threads within it as per the real hardware, and
> > > would also include xscoms and other per-module logic.
> > > 
> > > From here to there:
> > > 
> > > A suggested order of implementation to get there without too much risk
> > > of breaking things.
> > > 
> > >   1. Fix bugs with creation / removal of CPU objects (Bharata's cpu
> > >      hotplug series already has this)
> > >   2. Split creation and realization of CPU objects, so machine types
> > >      must explicitly perform both steps (Bharata's series has this
> > >      too)
> > >   3. Add the abstract cpu-package type, and define the generic
> > >      interfaces it needs (Bharata's series has something that could be
> > >      changed to this fairly easily)
> > >   4. For each machine type we care to convert:
> > >       4.1. Add platform suitable cpu-package subtypes
> > >       4.2. Convert the (latest version) machine type to instantiate 
> > > packages instead of
> > >            cpu threads directly
> > 
> > Machines could even have the freedom to instantiate CPU threads
> > directly and then set up package objects for them. Reusing
> > generic code is useful, but it doesn't even need to be mandatory,
> > as long as the objects are available at the right place in the
> > QOM hierarchy.
> 
> Ah, yes, I guess so.
> 
> > 
> > >       4.3. Add any necessary backwards compat goo
> > >   5. Teach libvirt how to toggle cpu-packages
> > 
> > This is different from the very flexible QOM object
> > building/linking approach Andreas was talking about in last KVM
> > Forum.
> 
> So, I know this stuff was discussed at KVM Forum, but unfortunately I
> never got a clear picture of what the outcome was.

I didn't see a clear conclusion after the KVM Forum discussions,
except for some ideas to use QOM object creation/linking to
implement something flexible enough for all architectures. But I
don't think we were moving towards something generic enough that
would allow management to implement CPU hotplug without
arch-specific code to setup CPU
sockets/packages/cores/threads/etc.

> > But while I would love to have the ability to build
> > arbitrary QOM hierarchies with complex links between CPUs
> > sockets, cores, threads, etc, I believe we need an interface that
> > is: 1) generic enough for multiple architectures and machines to
> > implement them; 2) simple enough so that libvirt can use it
> > easily without requiring more arch-specific code.
> > 
> > Also, your approach doesn't prevent the simple cpu-package
> > interface from having a complex QOM hierarchy hidden behind it.
> 
> Exactly.

-- 
Eduardo