Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Erlon Cruz]

> On Wed, Sep 12, 2012 at 6:48 PM, Benjamin Herrenschmidt <address@hidden> 
> wrote:
> On Wed, 2012-09-12 at 17:53 +0200, Alexander Graf wrote:
>> On 09/12/2012 04:54 PM, Erlon Cruz wrote:
>> > Hi all,
>> >
>> > We are planning to implement DLPAR capacity on QEMU pSeries. As we
>>
>> What is DLPAR? Hotplug support?
>
> Yes.
>
>> > lack of experience in the internals of the arch we would like you guys
>> > to give us some design directions
>> > and confirm if we going in the right direction. Our first idea is:
>> >
>> >      1 - to patch 'spapr.c' so it can dynamically insert/remove basic
>> > items into the device tree.
>>
>> What exactly would you like to patch into it? We already do have support
>> for dynamic dt creation with the spapr target.
>
> No we don't. We don't have the necessary bits and pieces to pass the DT
> updates down to the guest. PAPR defines a mechanism using RTAS calls
> which we need to implement, but there are some issues remaining:

Do we need any patching on SLOF to make this possible?

>  - We don't have a way to "initiate" a DLPAR operation. This is
> currently done by proprietary tools that communicate with the HMC. We
> want to invent some kind of hotplug "interrupt" (using existing RTAS
> event facilities). All it needs to do is indicate the DT path (ie.
> connector) where something is to be plugged to or unplugged, which can
> then trigger the relevant configure-connector calls to retrieve the DT
> bits.

I think that a device/guest driver will work for this purpose. It will
get interrupted when something on DT changes and trigger the kernel
routines that in the actual implementation are called by those
proprietary tool.

>  - We have a problem with PCI. Currently, the content of the PCI
> bus(ses) is discovered by SLOF running inside the guest. Not by qemu.
> It's SLOF that assigns the BARs and create the device-tree nodes for the
> various PCI devices. However, with hotplug, the guest expects to get
> fully populated DT nodes for hotplugged PCI devices and fully assigned
> BARS. Under pHyp that works because under the hood, RTAS contains an OFW
> implementation which does all the assignment before passing the stuff to
> the OS, but under qemu, RTAS is actually in qemu. This means we'll
> probably have to move back the PCI device node creation and resource
> assignment to qemu (like it was in the very early versions of the spapr
> support).

AFIK in the first versions of spapr there was no PCI support, right?
So I'm guessing that you refer to the first implementation of PCI.
Would we have to remove the PCI discovering functions from SLOF? I
have no idea on how to code into SLOF.

>> >      2 - create a host side device that will be used with a guest side
>> > driver to perform guest side operations and communicate changes from
>> > host to the guest (like DynamicRM does in PowerVM LPARs). We are not
>>
>> Why not just use hypercalls?
>
> Actually there are existing RTAS calls to use for the actual passing of
> the device-tree bits, the problem is purely how to "initiate" an
> operation to trigger the guest code that will then perform the
> appropriate calls.
> qemu-ga is an option. But I was thinking more along the lines of adding
> some new RTAS events, maybe EPOW style, a bit like ACPI does.
>
>> > planning to use powerpc-tools and want to make resource management
>> > transparent (i.e. no need to run daemons or userspace programs in the
>> > guest, only this kernel driver).
>> >      3 - create bindings to support adding/removal  ibmvscsi devices
>> >      4 - create bindings to support adding/removal  ibmveth devices
>> >      5 - create bindings to support adding/removal PCI devices
>> >      6 - create bindings to support adding/removal of memory
>
> There's already large parts of the necessary bits using RTAS in the
> kernel (in recent kernels that is, older stuff really needed it all done
> in userspace). The trigger mostly is missing.
>> This is going to be the hardest part. I don't think QEMU supports memory
>> hotplug yet.
>
> Missing from the above list is also CPU hotplug.
>
>> >          - Do we need to do this the way PowerVM does? We have tested
>> > virtio ballooning and it can works with a few endiannes corrections.
>>
>> I don't know how PowerVM works. But if normal ballooning is all you
>> need, you should certainly just enable virtio-balloon.
>
> Does virtio-balloon needs endian fixes ? We though it was just working !
> Feel free to submit patches :)
>
>> >      7 - create bindings to support adding/removal  CPUs
>> >          - is SMP supported already? I tried to run SMP in a x86 host
>> > and the guest stuck when SMP is enabled
>>
>> SMP should work just fine, yes. Where exactly does it get stuck?
>
> Right,it works fine as far as I can tell.
>
>> >          - would be possible to work on this without a P7 baremetal
>> > machine?
>>
>> At least for device hotplug, it should be perfectly possible to use an
>> old G5 with PR KVM. I haven't gotten around to patch all the pieces of
>> the puzzle to make -M pseries work with PR KVM when it's running on top
>> of pHyp yet, so that won't work.
>>
>> > We have a P7 8205-E6B, is that possible to kick PHYP out?
>>
>> Ben?
>
> Probably not. You need a 7R2.
>
>> > Any ideia on how much effort (time/people) the hole thing would take?
>> > Any consideration about this is much appreciated :)
>>
>> Phew. It's hard to tell. Depends heavily on how good your people are :).
>>
>
> Cheers,
> Ben.
>
>
>

Cheers,
Erlon