Re: [Qemu-devel] [PATCH 0/8] [PATCH RFC v3] s390 cpu hotplug

[Andreas Färber]

Am 05.09.2013 12:40, schrieb Christian Borntraeger:
> On 04/09/13 14:45, Andreas Färber wrote:
>> Hello,
>>
>> Am 01.08.2013 16:12, schrieb Jason J. Herne:
>>> From: "Jason J. Herne" <address@hidden>
>>>
>>> Latest code for cpu Hotplug on S390 architecture.   This one is vastly 
>>> simpler
>>> than v2 as we have decided to avoid the command line specification 
>>> of -device s390-cpu.
>>>
>>> The last version can be found here:
>>> http://lists.gnu.org/archive/html/qemu-devel/2013-06/msg01183.html
>>>
>>> There is also a patch in this series to add cpu-add to the Qemu monitor
>>> interface.
>>>
>>> Hotplugged cpus are created in the configured state and can be used by the
>>> guest after the guest onlines the cpu by: 
>>> "echo 1 > /sys/bus/cpu/devices/cpuN/online"
>>>
>>> Hot unplugging is currently not implemented by this code. 
>>
>> We have been having several off-list discussions since then that I'll
>> try to briefly summarize here, please correct or extend as needed:
>>
>> 1) CPU topology for QOM
>>
>> Physically a System z machine may have an MCM with, e.g., 6 CPUs with 6
>> cores each. But unlike x86, there is PR/SM, LPAR and possibly z/VM in
>> between Linux and hardware, so we do actually want to be able to
>> hot-plug in quantities of 1 and not by 6 on s390x for the foreseeable
>> future. We seem willing to set a QOM ABI in stone based on that assumption.
> 
> Just stepping in, Jason is on vacation this week.

Everyone is welcome to comment. :)

> To summarize my understanding:
> You were thinking if CPU model needs topology (e.g. -device mcm,id=m1, 
> -device cpu,mcm=m1)
> and s390 was the only arch left, that you were not sure about if topology is 
> needed? 
> All other platforms dont need topology for cpu hotplug?

No, on the contrary: I don't want s390x to blindly copy x86 cpu-add,
because for x86 we know that what we have is a hack to make it work
today, but there we know we want to do device_add Xeon-X42-4242 instead,
which then hot-plugs the 6 cores x 2 threads at once that a physical
hot-plug would do and not hot-add individual threads.

So the question of topology is not about what is below KVM but about
what is inside QEMU, since x86 emulates i440fx/q35 based hardware.
The understanding I reached on IRC is that s390x (similar to sPAPR)
tries to emulate LPAR / z/VM layer rather than the hardware below them,
thus no applicable concept of "real" hardware and arbitrary quantities.

> Yes, we want to be able to hotplug single cores (not chips, not MCMs). 
> It is pretty hard to pin the vCPUs to a given real topology for KVM. You need 
> to
> pin on LPAR and KVM. Libvirt could  do some pinning of guest vCPUs to host 
> CPUs and
> LPAR can have dedicated CPUs. But pinning a full chip (6cores) would only make
> sense in very rare cases.

Last time I looked into this, the post-add hook was solely for overall
ccw initialization. So we can use device_add s390-cpu today, can't we?

The question that I still need to investigate is how the
always-incrementing CPU address interacts with maxcpus. Consider
maxcpus=6 and smp_cpus=2. 4x device_add should work. Now if we did 1x
device_del, then 1x device_add should work again IMO. cpu-add checks the
user-supplied id against maxcpus though iirc.

Therefore my saying in multiple contexts that we should get the QEMU and
KVM CPU count checks into the CPU realizefn so that we get the checks
irrespective of the call site with nice error reporting.

>> => s390-cpu (or future subtypes) to be used with device_add.
>> => Flat /machine/cpu[n] list in composition tree a possibility.
>>
>> 1a) CPU topology for guests
>>
>> STSI instruction topology support not implemented yet.
> 
> Right not implemented yet, but we certainly want to be able to define the 
> guest
> visible topology at some point in time (grouping of cores basically). 
> But I guess this does not mean that we have to go away from the flat list of 
> CPUs.

So STSI would show what real LPAR/CPU we are running on? But QEMU would
have /machine/cpu[0]? Or do we need /machine/cpugroup[0]/cpu[0]? The
latter is my concern here, to decide about child<> vs. link<> properties.

To cope with device_add s390-cpu adding the device to
/machine/peripheral/<id> or /machine/peripheral-anon/device[0] I *think*
we'll need link<>, which would then translate back to ipi_states array
as backend and the remaining question would be where to expose those
properties in the composition tree - i.e. /machine/cpu[n] or
/machine/ipi/cpu[n] or something - please suggest. Similarly if those
become link<> properties then the CPUs created by the machine via
smp_cpus need a canonical path as well; quite obviously both cannot be
the same.

Background is that long-term Anthony would like x86 CPU hot-plug to
become setting/unsetting some /machine/cpu-socket[n] link<> property of
the machine, and the ipi_states array seems a close equivalent on s390x.

>> => Guest unaware of any emulated topology today.
> 
> An additional problem is, that for the normal case (linux scheduler, no 
> pinning, also
> no gang scheduling) the topology would change too fast. The guest would be 
> busy rebuilding
> the scheduler domains all the time.
[snip]

Regards,
Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg