[Qemu-devel] Re: [PATCH 1/2] Pad iommu with an empty slot (necessary for SunOS 4.1.4)

[Artyom Tarasenko]

2010/5/25 Blue Swirl <address@hidden>:
> On Tue, May 25, 2010 at 5:00 PM, Artyom Tarasenko
> <address@hidden> wrote:
>> 2010/5/21 Blue Swirl <address@hidden>:
>>> On Fri, May 21, 2010 at 5:23 PM, Artyom Tarasenko
>>> <address@hidden> wrote:
>>>> 2010/5/10 Blue Swirl <address@hidden>:
>>>>> On 5/10/10, Artyom Tarasenko <address@hidden> wrote:
>>>>>> 2010/5/10 Blue Swirl <address@hidden>:
>>>>>>
>>>>>> > On 5/10/10, Artyom Tarasenko <address@hidden> wrote:
>>>>>>  >> 2010/5/9 Blue Swirl <address@hidden>:
>>>>>>  >>  > On 5/9/10, Artyom Tarasenko <address@hidden> wrote:
>>>>>>  >>  >> 2010/5/9 Blue Swirl <address@hidden>:
>>>>>>  >>  >>
>>>>>>  >>  >> > On 5/8/10, Artyom Tarasenko <address@hidden> wrote:
>>>>>>  >>  >>  >> On the real hardware (SS-5, LX) the MMU is not padded, but 
>>>>>> aliased.
>>>>>>  >>  >>  >>  Software shouldn't use aliased addresses, neither should it 
>>>>>> crash
>>>>>>  >>  >>  >>  when it uses (on the real hardware it wouldn't). Using 
>>>>>> empty_slot
>>>>>>  >>  >>  >>  instead of aliasing can help with debugging such accesses.
>>>>>>  >>  >>  >
>>>>>>  >>  >>  > TurboSPARC Microprocessor User's Manual shows that there are
>>>>>>  >>  >>  > additional pages after the main IOMMU for AFX registers. So 
>>>>>> this is
>>>>>>  >>  >>  > not board specific, but depends on CPU/IOMMU versions.
>>>>>>  >>  >>
>>>>>>  >>  >>
>>>>>>  >>  >> I checked it on the real hw: on LX and SS-5 these are aliased 
>>>>>> MMU addresses.
>>>>>>  >>  >>  SS-20 doesn't have any aliasing.
>>>>>>  >>  >
>>>>>>  >>  > But are your machines equipped with TurboSPARC or some other CPU?
>>>>>>  >>
>>>>>>  >>
>>>>>>  >> Good point, I must confess, I missed the word "Turbo" in your first
>>>>>>  >>  answer. LX and SS-20 don't.
>>>>>>  >>  But SS-5 must have a TurboSPARC CPU:
>>>>>>  >>
>>>>>>  >>  ok cd /FMI,MB86904
>>>>>>  >>  ok .attributes
>>>>>>  >>  context-table            00 00 00 00 03 ff f0 00 00 00 10 00
>>>>>>  >>  psr-implementation       00000000
>>>>>>  >>  psr-version              00000004
>>>>>>  >>  implementation           00000000
>>>>>>  >>  version                  00000004
>>>>>>  >>  cache-line-size          00000020
>>>>>>  >>  cache-nlines             00000200
>>>>>>  >>  page-size                00001000
>>>>>>  >>  dcache-line-size         00000010
>>>>>>  >>  dcache-nlines            00000200
>>>>>>  >>  dcache-associativity     00000001
>>>>>>  >>  icache-line-size         00000020
>>>>>>  >>  icache-nlines            00000200
>>>>>>  >>  icache-associativity     00000001
>>>>>>  >>  ncaches                  00000002
>>>>>>  >>  mmu-nctx                 00000100
>>>>>>  >>  sparc-version            00000008
>>>>>>  >>  mask_rev                 00000026
>>>>>>  >>  device_type              cpu
>>>>>>  >>  name                     FMI,MB86904
>>>>>>  >>
>>>>>>  >>  and still it behaves the same as TI,TMS390S10 from the LX. This is 
>>>>>> done on SS-5:
>>>>>>  >>
>>>>>>  >>  ok 10000000 20 spacel@ .
>>>>>>  >>  4000009
>>>>>>  >>  ok 14000000 20 spacel@ .
>>>>>>  >>  4000009
>>>>>>  >>  ok 14000004 20 spacel@ .
>>>>>>  >>  23000
>>>>>>  >>  ok 1f000004 20 spacel@ .
>>>>>>  >>  23000
>>>>>>  >>  ok 10000008 20 spacel@ .
>>>>>>  >>  4000009
>>>>>>  >>  ok 14000028 20 spacel@ .
>>>>>>  >>  4000009
>>>>>>  >>  ok 1000000c 20 spacel@ .
>>>>>>  >>  23000
>>>>>>  >>  ok 10000010 20 spacel@ .
>>>>>>  >>  4000009
>>>>>>  >>
>>>>>>  >>
>>>>>>  >>  LX is the same except for the IOMMU-version:
>>>>>>  >>
>>>>>>  >>  ok 10000000 20 spacel@ .
>>>>>>  >>  4000005
>>>>>>  >>  ok 14000000 20 spacel@ .
>>>>>>  >>  4000005
>>>>>>  >>  ok 18000000 20 spacel@ .
>>>>>>  >>  4000005
>>>>>>  >>  ok 1f000000 20 spacel@ .
>>>>>>  >>  4000005
>>>>>>  >>  ok 1ff00000 20 spacel@ .
>>>>>>  >>  4000005
>>>>>>  >>  ok 1fff0004 20 spacel@ .
>>>>>>  >>  1fe000
>>>>>>  >>  ok 10000004 20 spacel@ .
>>>>>>  >>  1fe000
>>>>>>  >>  ok 10000108 20 spacel@ .
>>>>>>  >>  41000005
>>>>>>  >>  ok 10000040 20 spacel@ .
>>>>>>  >>  41000005
>>>>>>  >>  ok 1fff0040 20 spacel@ .
>>>>>>  >>  41000005
>>>>>>  >>  ok 1fff0044 20 spacel@ .
>>>>>>  >>  1fe000
>>>>>>  >>  ok 1fff0024 20 spacel@ .
>>>>>>  >>  1fe000
>>>>>>  >>
>>>>>>  >>
>>>>>>  >>  >>  At what address the additional AFX registers are located?
>>>>>>  >>  >
>>>>>>  >>  > Here's complete TurboSPARC IOMMU address map:
>>>>>>  >>  >  PA[30:0]          Register          Access
>>>>>>  >>  > 1000_0000       IOMMU Control         R/W
>>>>>>  >>  > 1000_0004    IOMMU Base Address       R/W
>>>>>>  >>  > 1000_0014   Flush All IOTLB Entries    W
>>>>>>  >>  > 1000_0018        Address Flush         W
>>>>>>  >>  > 1000_1000  Asynchronous Fault Status  R/W
>>>>>>  >>  > 1000_1004 Asynchronous Fault Address  R/W
>>>>>>  >>  > 1000_1010  SBus Slot Configuration 0   R/W
>>>>>>  >>  > 1000_1014  SBus Slot Configuration 1   R/W
>>>>>>  >>  > 1000_1018  SBus Slot Configuration 2   R/W
>>>>>>  >>  > 1000_101C  SBus Slot Configuration 3   R/W
>>>>>>  >>  > 1000_1020  SBus Slot Configuration 4   R/W
>>>>>>  >>  > 1000_1050     Memory Fault Status     R/W
>>>>>>  >>  > 1000_1054    Memory Fault Address     R/W
>>>>>>  >>  > 1000_2000     Module Identification    R/W
>>>>>>  >>  > 1000_3018      Mask Identification      R
>>>>>>  >>  > 1000_4000      AFX Queue Level         W
>>>>>>  >>  > 1000_6000      AFX Queue Level         R
>>>>>>  >>  > 1000_7000      AFX Queue Status        R
>>>>>>  >>
>>>>>>  >>
>>>>>>  >>
>>>>>>  >> But if I read it correctly 0x12fff294 (which makes SunOS crash with 
>>>>>> -m 32) is
>>>>>>  >>  well above this limit.
>>>>>>  >
>>>>>>  > Oh, so I also misread something. You are not talking about the
>>>>>>  > adjacent pages, but 16MB increments.
>>>>>>  >
>>>>>>  > Earlier I sent a patch for a generic address alias device, would it be
>>>>>>  > useful for this?
>>>>>>
>>>>>>
>>>>>> Should do as well. But I thought empty_slot is less overhead and
>>>>>>  easier to debug.
>>>>>>
>>>>
>>>> Also the aliasing patch would require one more parameter: the size of
>>>> area which has to be aliased. Except we implement stubs for all
>>>> missing devices and and do aliasing of the connected port ranges. And
>>>> then again, SS-20 doesn't have aliasing in this area at all.
>>>>
>>>> What do you think about this (empty_slot) solution (except that I
>>>> missed the SoB line)? Meanwhile it's tested with SunOS 4.1.3U1 too.
>>>
>>> I'm slightly against it, of course it would help for this but I think
>>> we may be missing a bigger problem.
>>>
>>>>>>> Maybe we have a general design problem, perhaps unassigned access
>>>>>>> faults should only be triggered inside SBus slots and ignored
>>>>>>> elsewhere. If this is true, generic Sparc32 unassigned access handler
>>>>>>> should just ignore the access and special fault generating slots
>>>>>>> should be installed for empty SBus address ranges.
>>>>
>>>> Agreed that they should be special for SBus, because SS-20 OBP is
>>>> not happy with the fault we are currently generating. But otherwise I 
>>>> think qemu
>>>> does it correct. On SS-5:
>>>>
>>>> ok f7ff0000 2f spacel@ .
>>>> Data Access Error
>>>> ok sfar@ .
>>>> f7ff0000
>>>> ok 20000000 2f spacel@ .
>>>> Data Access Error
>>>> ok sfar@ .
>>>> 20000000
>>>> ok 40000000 20 spacel@ .
>>>> Data Access Error
>>>> ok sfar@ .
>>>> 40000000
>>>>
>>>> Neither ff7ff0000 nor f20000000, nor 40000000 are in SBus range,  right?
>>>
>>> 40000000 is on SS-5.
>>
>> Ah. I was only aware of the control space. What ranges does SBus take?
>
> On SS-5, 30000000 to 7fffffff, each slot taking 10000000. There's AFX
> bus on 20000000.
>
> OBP property '/iommu/sbus/ranges' shows these (also other ranges)
>
>>
>>> So is the SBus Control Space in 0x10000000 to
>>> 0x1fffffff the only area besides DRAM where the accesses won't trap?
>>
>> At least some area after ROM is aliased too. Also on SS-10 with a
>> non-active frame buffer
>> writing to SX registers makes no visible effect and reading from them
>> produces no fault but a NMI.
>
> Then we should cover the whole area after IOMMU with empty slot
> device. ROM probably doesn't matter.

You mean up to sbus (the example above has shown that 40000000
produced a fault)?
But that's what the patch is actually doing. More examples on SS-5:

ok 1ffffffc 20 spacel@ .
3fe000
ok 20000000 20 spacel@ .
c8840140
ok 20010000 20 spacel@ .
c8840140
ok 2f010000 20 spacel@ .
c8840140
ok 30000000 20 spacel@ .
Data Access Error
ok sfar@ . sfsr@ .
30000000 836
ok 40000000 20 spacel@ .
Data Access Error
ok sfar@ . sfsr@ .
40000000 836
ok 50000000 20 spacel@ .
fd03774a
ok 60000000 20 spacel@ .
Data Access Error
ok 70000000 20 spacel@ .
10802f66

ok show-sbus
SBus slot 5 ledma le SUNW,bpp espdma esp
SBus slot 4 power-management SUNW,CS4231
SBus slot 1
SBus slot 2
SBus slot 3 cgsix
SBus slot 0

The ranges 20000000,50000000 and 70000000 don't produce a fault cause
there are devices attached. The areas 30000000,40000000 and  60000000
do cause there is nothing.
So qemu does more or less the right thing. It's even more right in
case of SS-20 cause iommu there is not aliased and accessing the
addresses after iommu also produces faults.

Was going to put some more empty slots into SS-10/20 (VSIMMs, SX)
after we are done with SS-5 (due to technical limitations I can switch
access from one real SS model to another one once a few days only).

-- 
Regards,
Artyom Tarasenko

solaris/sparc under qemu blog: http://tyom.blogspot.com/