Re: [Qemu-devel] KVM call minutes 2013-01-29 - Port I/O

[Benjamin Herrenschmidt]

On Wed, 2013-01-30 at 15:39 -0600, Anthony Liguori wrote:
> Benjamin Herrenschmidt <address@hidden> writes:

> > There also exists P2P bridges doing such substractive
> > decoding, this used to be fairly common with transparent bridges used for
> > laptop docking.
> 
> I'm not sure I understand how this would work.  How can two devices on
> the same PCI domain both do subtractive decoding?  Indeed, the PCI spec
> even says:
> 
> "Subtractive decoding can be implemented by only one device on the bus
>  since it accepts all accesses not positively decoded by some other
>  agent."

They would typically be the only one at *that* level, though of course
some other device can do it underneath them.

> >> 2) There are special hacks in most PCI chipsets to route very specific
> >>    addresses ranges to certain devices.  Namely, legacy VGA IO transactions
> >>    go to the first VGA device.  Legacy IDE IO transactions go to the first
> >>    IDE device.  This doesn't need to be programmed in the BARs.  It will
> >>    just happen.
> >
> > This is also mostly not a hack in the chipset. It's a well defined behaviour
> > for legacy devices, sometimes call hard decoding. Of course often those 
> > devices
> > are built into the chipset but they don't have to. Plug-in VGA devices will
> > hard decode legacy VGA regions for both IO and MMIO by default (this can be
> > disabled on most of them nowadays) for example. This has nothing to do with
> > the chipset.
> 
> So I understand what you're saying re: PCI because the devices actually
> assert DEVSEL to indicate that they handle the transaction.

Yes.

> But for PCI-E, doesn't the controller have to expressly identify what
> the target is?  Is this done with the device class?

No.

PCI-E is point to point. So until you have switches, a device will get
any transaction and can decide to accept or reject it (which boils down
from a high level SW perspective to the same thing as asserting DEVSEL
or not, the rejection typically causing a master abort style error).

If you have a switch, it's the bridge windows that define where a
transaction goes, and those still, I believe, support the bit to
indicate forwarding of legacy ISA in addition to the usual bridge
windows.
 
> > There's a specific bit in P2P bridge to control the forwarding of legacy
> > transaction downstream (and VGA palette snoops), this is also fully 
> > specified
> > in the PCI spec.
> 
> Ack.
> 
> >
> >> 3) As it turns out, all legacy PIIX3 devices are positively decoded and
> >>    sent to the ISA-bridge (because it's faster this way).
> >
> > Chipsets don't "send to a bridge". It's the bridge itself that
> > decodes.
> 
> With PCI...

Right. And PCI-E :-)

> >> Notice the lack of the word "ISA" in all of this other than describing
> >> the PCI class of an end point.
> >
> > ISA is only relevant to the extent that the "legacy" regions of IO space
> > originate from the original ISA addresses of devices (VGA, IDE, etc...)
> > and to the extent that an ISA bus might still be present which will get
> > the transactions that nothing else have decoded in that space.
> 
> Ack.
> 
> >  
> >> So how should this be modeled?
> >> 
> >> On x86, the CPU has a pio address space.  That can propagate down
> >> through the PCI bus which is what we do today.
> >> 
> >> On !x86, the PCI controller ought to setup a MemoryRegion for
> > downstream
> >> PIO that devices can use to register on.
> >> 
> >> We probably need to do something like change the PCI VGA devices to
> >> export a MemoryRegion and allow the PCI controller to device how to
> >> register that as a subregion.
> >
> > The VGA device should just register fixed address port IOs the same way
> > it would register an IO BAR. Essentially, hard coded IO addresses (or
> > memory, VGA does memory too, don't forget that) are equivalent to having
> > an invisible BAR with a fixed value in it.
> 
> Ack.
> 
> >
> > There should be no "global port IO" because that concept is broken on
> > real multi-domain setups. Those "legacy" address ranges are just
> > hard-wired sub regions of the normal PCI space on which the device sits
> > on (unless you start doing real non-PCI ISA x86).
> 
> So, I think what you're suggesting (and I agree with), is that each PCI
> device should export one or more MemoryRegions and indicate what the
> MemoryRegions are for.
> 
> Potential options are:
> 
>  - MMIO BAR
>  - PIO BAR
>  - IDE hard decode
>  - VGA hard decode
>  - subtractive decode

Simpler:

 - MMIO BAR with variable address (normal BAR)
 - MMIO BAR with fixed address (legacy)
 - PIO BAR with variable address
 - PIO BAR with fixed address

That's for devices. For bridge, add substractive decode option.

IE. No need to make "IDE" or "VGA" special. There are other cases of
fixed address decoding anyway, some legacy serial cards (PCI based)
still hard decode by default for example, etc...

Also some devices actually use BARs for hard decode. IE. It's common
for IDE devices to actually have all BARs for PIO but they start
read-only and set to the legacy addresses, and you can change them to
RW & configurable by writing to ProgIf to switch from legacy to native
(it also has the side effect of changing the irq from edge to level on
some which is a horribly gross hack from hell since PCI interrupts
aren't supposed to be edge but heh ... welcome to x86).

> I'm very much in agreement if that's what you're suggesting.

I think we are roughly on the same line :-)

Cheers,
Ben.

> Regards,
> 
> Anthony Liguori
> 
> >
> > Cheers,
> > Ben.