Re: [Qemu-devel] [PATCH] hw/pci: disable pci-bridge's shpc by default

[David Gibson]

On Thu, Nov 24, 2016 at 11:39:25AM +0200, Marcel Apfelbaum wrote:
> On 11/24/2016 06:06 AM, David Gibson wrote:
> > On Wed, Nov 23, 2016 at 01:08:46PM +0200, Marcel Apfelbaum wrote:
> > > On 11/22/2016 10:25 PM, Laurent Vivier wrote:
> > > > 
> > > > 
> > > > On 22/11/2016 18:26, Marcel Apfelbaum wrote:
> > > > > On 11/18/2016 05:52 PM, Andrew Jones wrote:
> > > > > > On Wed, Nov 16, 2016 at 07:05:25PM +0200, Marcel Apfelbaum wrote:
> > > > > > > On 11/16/2016 06:44 PM, Andrew Jones wrote:
> > > > > > > > On Sat, Nov 05, 2016 at 06:46:34PM +0200, Marcel Apfelbaum 
> > > > > > > > wrote:
> > > > > > > > > On 11/03/2016 09:40 PM, Michael S. Tsirkin wrote:
> > > > > > > > > > On Thu, Nov 03, 2016 at 01:05:44PM +0200, Marcel Apfelbaum 
> > > > > > > > > > wrote:
> > > > > > > > > > > On 11/03/2016 06:18 AM, Michael S. Tsirkin wrote:
> > > > > > > > > > > > On Wed, Nov 02, 2016 at 05:16:42PM +0200, Marcel 
> > > > > > > > > > > > Apfelbaum wrote:
> > > > > > > > > > > > > The shpc component is optional while  ACPI hotplug is 
> > > > > > > > > > > > > used
> > > > > > > > > > > > > for hot-plugging PCI devices into a PCI-PCI bridge.
> > > > > > > > > > > > > Disabling the shpc by default will make slot 0 usable 
> > > > > > > > > > > > > at boot time
> > > > > > > > > > > 
> > > > > > > > > > > Hi Michael
> > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > at the cost of breaking all hotplug for all non-acpi 
> > > > > > > > > > > > users.
> > > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > Do we have a non-acpi user that is able to use the shpc 
> > > > > > > > > > > component
> > > > > > > > > > > as-is today?
> > > > > > > > > > 
> > > > > > > > > > power and some arm systems I guess?
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Adding Andrew , maybe he can give us an answer.
> > > > > > > > 
> > > > > > > > Not really :-) My lack of PCI knowledge makes that difficult. 
> > > > > > > > I'd be
> > > > > > > > happy
> > > > > > > > to help with an experiment though. Can you give me command line
> > > > > > > > arguments,
> > > > > > > > qmp commands, etc. that I should use to try it out? I imagine I 
> > > > > > > > should
> > > > > > > > just boot an ARM guest using DT (instead of ACPI) and then 
> > > > > > > > attempt to
> > > > > > > > hotplug a PCI device. I'm not sure, however, what, if any, 
> > > > > > > > special
> > > > > > > > configuration I need in order to ensure I'm testing what you're
> > > > > > > > interested in.
> > > > > > > > 
> > > > > > > 
> > > > > > > Hi Drew,
> > > > > > > 
> > > > > > > 
> > > > > > > Just run QEMU with '-device pci-bridge,chassis_nr=1,id=bridge1
> > > > > > > -monitor stdio'
> > > > > > > with an ARM guest using DT and wait until the guest finish 
> > > > > > > booting.
> > > > > > > 
> > > > > > > Then run at hmp:
> > > > > > > device_add virtio-net-pci,bus=bridge1,id=net2
> > > > > > > 
> > > > > > > Next run lspci in the guest to see the new device.
> > > > > > 
> > > > > > Thanks for the instructions Marcel. Here's the results
> > > > > > 
> > > > > >  $QEMU -machine virt,accel=$ACCEL -cpu $CPU -nographic -m 4096 -smp 
> > > > > > 8 \
> > > > > >        -bios /usr/share/AAVMF/AAVMF_CODE.fd \
> > > > > >        -device pci-bridge,chassis_nr=1,id=bridge1 \
> > > > > >        -drive file=$FEDORA_IMG,if=none,id=dr0,format=qcow2 \
> > > > > >        -device 
> > > > > > virtio-blk-pci,bus=bridge1,addr=01,drive=dr0,id=disk0 \
> > > > > >        -netdev user,id=hostnet0 \
> > > > > >        -device 
> > > > > > virtio-net-pci,bus=bridge1,addr=02,netdev=hostnet0,id=net0
> > > > > > 
> > > > > >  # lspci
> > > > > >  00:00.0 Host bridge: Red Hat, Inc. Device 0008
> > > > > >  00:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
> > > > > >  01:01.0 SCSI storage controller: Red Hat, Inc Virtio block device
> > > > > >  01:02.0 Ethernet controller: Red Hat, Inc Virtio network device
> > > > > > 
> > > > > >  (qemu) device_add virtio-net-pci,bus=bridge1,id=net2
> > > > > >  Unsupported PCI slot 0 for standard hotplug controller. Valid 
> > > > > > slots are
> > > > > >  between 1 and 31.
> > > > > > 
> > > > > > (Tried again giving addr=03)
> > > > > > 
> > > > > >  (qemu) device_add virtio-net-pci,bus=bridge1,id=net2,addr=03
> > > > > > 
> > > > > > (Seemed to work, but...)
> > > > > > 
> > > > > >  # lspci
> > > > > >  00:00.0 Host bridge: Red Hat, Inc. Device 0008
> > > > > >  00:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
> > > > > >  01:01.0 SCSI storage controller: Red Hat, Inc Virtio block device
> > > > > >  01:02.0 Ethernet controller: Red Hat, Inc Virtio network device
> > > > > > 
> > > > > > (Doesn't show up in lscpi. So I guess it doesn't work)
> > > > > > 
> > > > > 
> > > > > Hi Drew,
> > > > > Thanks for confirming that it doesn't work.
> > > > > 
> > > > > Michael asked if we can check the same for powerpc before
> > > > > disabling the shpc by default.
> > > > > 
> > > > > Adding David, Thomas and Laurrent, maybe they have time
> > > > > to check it for powerpc.
> > > > 
> > > > With this patch:
> > > > 
> > > > # lspci
> > > > 00:00.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
> > > > 00:03.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
> > > > address@hidden ~]# QEMU 2.7.90 monitor - type 'help' for more 
> > > > information
> > > > (qemu) device_add virtio-net-pci,bus=bridge1,id=net2
> > > > Bus 'bridge1' does not support hotplugging
> > > > (qemu) device_add virtio-net-pci,bus=bridge1,id=net2,addr=3
> > > > Bus 'bridge1' does not support hotplugging
> > > > 
> > > > Without this patch:
> > > > 
> > > > # lspci
> > > > 00:00.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
> > > > 00:03.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
> > > > address@hidden ~]# QEMU 2.7.90 monitor - type 'help' for more 
> > > > information
> > > > (qemu) device_add virtio-net-pci,bus=bridge1,id=net2
> > > > Unsupported PCI slot 0 for standard hotplug controller. Valid slots are
> > > > between 1 and 31.
> > > > (qemu) device_add virtio-net-pci,bus=bridge1,id=net2,addr=3
> > > > (qemu) [   37.342908] shpchp 0000:00:00.0: Latch close on Slot(3)
> > > > [   37.342963] shpchp 0000:00:00.0: Button pressed on Slot(3)
> > > > [   37.343003] shpchp 0000:00:00.0: Card present on Slot(3)
> > > > [   37.344186] shpchp 0000:00:00.0: PCI slot #3 - powering on due to
> > > > button press
> > > > [   43.361827] shpchp 0000:00:00.0: No new device found
> > > 
> > > 
> > > Hi Laurrent,
> > > Thank you for the fast reply!
> > > 
> > > What I see from the log is that ppc does use the shpc for hotplug,
> > > but there is some implementation bug preventing it to work.
> > 
> > That doesn't sound right.  ppc, or more correctly the pseries machine
> > type, has it's own hotplug notification mechanism.  If we're trying to
> > invoke shpc, I think something is wrong.
> > 
> 
> Hi David,
> 
> As you can see the shpc driver claims the pci-bridge slots and it responds
> to the hot-plug events.

Yeah.. that's a bug of itself.  shpc can't work properly on PAPR
guests (or even on IBM POWER hardware hosts, for related though not
identical reasons).

In addition to PAPR having its own hotplug mechanism, the shpc stuff
apparently conflicts with the (also POWER specific) EEH error recovery
mechanism.

> Anyway, what is the hotplug mechanism preferred by the pseries machine ?

The PAPR specification which defines the firmware and hypervisor
interfaces that guests on POWER operate under, has its own hotplug
mechanism, using hypervisor events to deliver updated device tree
fragments to the guest.

> Side note: what we want to understand is if using pci-bridges's shpc  by 
> default
> would help non x86 archs (since QEMU x86 machine uses ACPI based
> hot-plug).

Certainly not for pseries or powernv.  It might be correct for (some?)
embedded ppc machines with PCI-E

> 
> Thanks,
> Marcel
> 
> > > Drew, may I trouble one last time for the dmesg?
> > > I think we would see exactly the same log.
> > > 
> > > 
> > > Thanks,
> > > Marcel
> > > 
> > > > [   43.361867] shpchp 0000:00:00.0: Cannot add device at 0000:01:03
> > > > [   43.363277] shpchp 0000:00:00.0: Latch open on Slot(3)
> > > > [   43.363320] shpchp 0000:00:00.0: Button pressed on Slot(3)
> > > > [   43.363360] shpchp 0000:00:00.0: Card not present on Slot(3)
> > > > [   43.363506] shpchp 0000:00:00.0: PCI slot #3 - powering on due to
> > > > button press
> > > > [   48.371835] shpchp 0000:00:00.0: No adapter on slot(3)
> > > > 
> > > > Laurent
> > > > 
> > > 
> > 
> 

-- 
David Gibson                    | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
                                | _way_ _around_!
http://www.ozlabs.org/~dgibson

signature.asc
Description: PGP signature