Re: [Qemu-devel] Re: [PATCH 1/6] Make config space accessor host bus trapable

[Michael S. Tsirkin]

On Tue, Jan 05, 2010 at 12:08:19AM +0100, Alexander Graf wrote:
> 
> On 04.01.2010, at 23:59, Michael S. Tsirkin wrote:
> 
> > On Tue, Jan 05, 2010 at 09:51:48AM +1100, Benjamin Herrenschmidt wrote:
> >> On Tue, 2010-01-05 at 00:25 +0200, Michael S. Tsirkin wrote:
> >>> On Tue, Jan 05, 2010 at 08:53:52AM +1100, Benjamin Herrenschmidt wrote:
> >>>> 
> >>>>> Yes, but I think how you program your host to pci bridge is platform 
> >>>>> specific,
> >>>>> the standard (mostly) applies to what happens below the bridge.  There's
> >>>>> no real standard for how PCI host bridge is connected to processor
> >>>>> AFAIK, it's by luck we can share code there at all.
> >>>> 
> >>>> Well, yes and no ... there's a standard on how a PCI host bridge is
> >>>> connected in the sense that how normal MMIO accesses go through in term
> >>>> of endianness is well defined.
> >>>> 
> >>> 
> >>> Go through where? From processor to PCI?
> >>> Which spec do you refer to?
> >> 
> >> The PCI spec from memory :-) Byte swizzling for MMIO between a processor
> >> and PCI bus is well defined.
> > 
> > Couldn't find it in spec.
> > 
> >> Now of course, since issuing config space cycles tend to be host-bridge
> >> chip specific, everything is possible there :-) In -most- cases though,
> >> they use a mechanism similar to x86 using the cf8/cfc ports or
> >> equivalent mapped wherever the PIO region is mapped in MMIO space for
> >> non-x86 processors, and thus end up with the exact same byte swizzling.
> >> 
> >> In fact, this is true of accesses to PCI devices as well. Take for
> >> example, a device that has a 32-bit MMIO register. This register is
> >> meant to appear as little endian (well, unless the device itself plays
> >> tricks but most don't) whatever the host processor is. Thus an x86 host
> >> will need no byteswap but a powerpc host (assuming the ppc is running in
> >> BE mode) will.
> >> 
> >> This is why for example the base readl and writel function in Linux do
> >> byteswap on powerpc.
> >> 
> >> This is important to understand that this is a property of how the PCI
> >> bridge is connected to the host processor, such that the PCI "native"
> >> byte order is preserved along with address invariance for sub-32-bit
> >> quantities.
> >> 
> >> The endianness of the host bridge config space access register is thus
> >> most of the time just a natural side effect of said register being part
> >> of the bridge PIO space and thus getting the natural byteswap explained
> >> above for a 32-bit LE register on PCI.
> >> 
> >> Cheers,
> >> Ben.
> > 
> > So, it appears that this is not the case for many platforms: bridge
> > itself does a byteswap to make devices behind it work according to spec,
> > but this does not apply to programming bridge itself.
> > 
> > This seems common on BE platforms, this is why qemu has
> > ifdef TARGET_WORDS_BIGENDIAN there IIUC.
> 
> IIRC qemu's mmio functions just pass the register value the guest had at that 
> moment to the mmio function.
> 
> While that is pretty simple, it means that we behave different from real 
> hardware. Real hardware has 2 components:
> 
> 1) CPU
> 2) Device
> 
> The CPU sends an MMIO request in local byte order to the device. The device 
> also has a native endianness - either BE or LE.
> So what the byte swap here does is simply converting from wrong LE (what 
> Linux thought the device needs) to a proper variable.
> 
> I'm not sure what the correct solution to this is. I'm inclined to suggest 
> that mmio callbacks should get called with tswap'ed values. But then again 
> the code as is works in quite a lot of cases and I don't want to spend months 
> getting it back to where it is just because of a cosmetic change.
> 
> Alex

If I understand correctly this is just a byteswap to emulate how host
bridge is connected, unrelated to local byte order.  By anyway, I agree:
let's not spend months on this.