Re: [PULL 32/41] target/mips: Use cpu_*_mmuidx_ra instead of MMU_MODE*_SUFFIX

[Philippe Mathieu-Daudé]

On 2/1/20 11:07 PM, Philippe Mathieu-Daudé wrote:
> On 2/1/20 9:12 PM, Philippe Mathieu-Daudé wrote:
>> Hi Richard,
>>
>> On 1/8/20 4:45 AM, Richard Henderson wrote:
>>> The separate suffixed functions were used to construct
>>> some do_##insn function switched on mmu_idx.  The interface
>>> is exactly identical to the *_mmuidx_ra functions.  Replace
>>> them directly and remove the constructions.
>>>
>>> Cc: Aurelien Jarno <address@hidden>
>>> Cc: Aleksandar Rikalo <address@hidden>
>>> Tested-by: Philippe Mathieu-Daudé <address@hidden>
>>
>> I'm seeing a regression since commit 7dd547e5a (however I'm not sure
>> this particular commit introduced the regression, or instead revealed it).
>>
>> I get a hang when booting a I6400 CPU on Malta:
>>
>> $ qemu-system-mips64el -cpu I6400 -serial stdio \
>>   -kernel vmlinux \
>>   -append "clocksource=GIC console=ttyS0 printk.time=1"
> 
> FYI this binary was built using:
> 
>   ./configure --target-list=mips64el-softmmu --disable-xen
> --disable-spice --disable-vnc --disable-rdma --disable-vde --disable-rbd
> --disable-smartcard --disable-libusb --disable-glusterfs --disable-tpm
> --disable-vhdx --disable-tools --disable-guest-agent --disable-docs
> --disable-vhost-vsock --extra-cflags='-Wno-format-truncation
> -Wno-stringop-truncation -Wno-address-of-packed-member'
> 
>> Linux version 4.7.0-rc1 (@859affe05464) (gcc version 6.3.0 20170516
>> (Debian 6.3.0-18) ) #2 SMP Sat Feb 1 18:16:17 UTC 2020
>> GCRs appear to have been moved (expected them at 0x1fbf8000)!
>> earlycon: uart8250 at I/O port 0x3f8 (options '38400n8')
>> bootconsole [uart8250] enabled
>> MIPS CPS SMP unable to proceed without a CM
>> CPU0 revision is: 0001a900 (MIPS I6400)
>> FPU revision is: 20f30300
>> MSA revision is: 00000300
>> MIPS: machine is mti,malta
>> Software DMA cache coherency enabled
>> Determined physical RAM map:
>>  memory: 0000000008000000 @ 0000000000000000 (usable)
>> Zone ranges:
>>   DMA      [mem 0x0000000000000000-0x0000000000ffffff]
>>   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
>>   Normal   empty
>> Movable zone start for each node
>> Early memory node ranges
>>   node   0: [mem 0x0000000000000000-0x0000000007ffffff]
>> Initmem setup node 0 [mem 0x0000000000000000-0x0000000007ffffff]
>> Primary instruction cache 64kB, VIPT, 4-way, linesize 64 bytes.
>> Primary data cache 64kB, 4-way, VIPT, no aliases, linesize 64 bytes
>> percpu: Embedded 5 pages/cpu @980000000107c000 s29664 r8192 d44064 u81920
>> Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 8163
>> Kernel command line: clocksource=GIC console=ttyS0 printk.time=1
>> [    0.000000] PID hash table entries: 512 (order: -2, 4096 bytes)
>> [    0.000000] Dentry cache hash table entries: 16384 (order: 3, 131072
>> bytes)
>> [    0.000000] Inode-cache hash table entries: 8192 (order: 2, 65536 bytes)
>> [    0.000000] Writing ErrCtl register=00000000
>> [    0.000000] Readback ErrCtl register=00000000
>> [    0.000000] MAAR configuration:
>> [    0.000000]   [0]: 0x0000000000010000-0x0000000007ffffff speculate
>> [    0.000000]   [1]: disabled
>> [    0.000000]   [2]: disabled
>> [    0.000000]   [3]: disabled
>> [    0.000000]   [4]: disabled
>> [    0.000000]   [5]: disabled
>> [    0.000000]   [6]: disabled
>> [    0.000000]   [7]: disabled
>> [    0.000000] Memory: 121728K/131072K available (5253K kernel code,
>> 380K rwdata, 1276K rodata, 304K init, 278K bss, 9344K reserved, 0K
>> cma-reserved)
>> [    0.000000] Hierarchical RCU implementation.
>> [    0.000000]  Build-time adjustment of leaf fanout to 64.
>> [    0.000000]  RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=1.
>> [    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=1
>> [    0.000000] NR_IRQS:256
>> [    0.000000] CPU frequency 200.00 MHz
>> [    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles:
>> 0xffffffff, max_idle_ns: 19112598155 ns
>> [    0.000223] sched_clock: 32 bits at 100MHz, resolution 9ns, wraps
>> every 21474829307ns
>> [    0.008011] Console: colour dummy device 80x25
>> [    0.012181] Calibrating delay loop... 950.27 BogoMIPS (lpj=4751360)
>> [    0.164015] pid_max: default: 32768 minimum: 301
>> [    0.165920] Mount-cache hash table entries: 2048 (order: 0, 16384 bytes)
>> [    0.166244] Mountpoint-cache hash table entries: 2048 (order: 0,
>> 16384 bytes)
>> [    0.202498] Brought up 1 CPUs
>> [    0.213199] devtmpfs: initialized
>> [    0.228877] clocksource: jiffies: mask: 0xffffffff max_cycles:
>> 0xffffffff, max_idle_ns: 19112604462750000 ns
>> [    0.236181] NET: Registered protocol family 16
>> [    0.253496] clocksource: pit: mask: 0xffffffff max_cycles:
>> 0xffffffff, max_idle_ns: 1601818034827 ns
>> [    0.254012] pm-cps: no CM, non-coherent states unavailable
>> [    0.257508] ------------[ cut here ]------------
>> [    0.258085] WARNING: CPU: 0 PID: 1 at fs/kernfs/dir.c:502
>> __kernfs_remove+0x250/0x258
>> [    0.258489] Modules linked in:
>> [   21.494186] INFO: rcu_sched detected stalls on CPUs/tasks:
>> [   21.494530]  (detected by 0, t=2102 jiffies, g=-299, c=-300, q=1)
>> [   21.494921] All QSes seen, last rcu_sched kthread activity 2101
>> (4294939419-4294937318), jiffies_till_next_fqs=1, root ->qsmask 0x0

I have been using this test:

---
class MaltaMachineI6400(Test):

    count = 15
    timeout = 50 * count

    def do_test_mips_malta_i6400_stall(self):
        """
        :avocado: tags=arch:mips64el
        :avocado: tags=machine:malta
        :avocado: tags=cpu:i6400
        """
        kernel_url = ('https://github.com/philmd/qemu-testing-blob/raw/'
                      'a5966ca4b5/mips/malta/mips64el/'
                      'vmlinux-4.7.0-rc1.I6400.gz')
        kernel_hash = '096f50c377ec5072e6a366943324622c312045f6'
        kernel_path_gz = self.fetch_asset(kernel_url,
asset_hash=kernel_hash)
        kernel_path = self.workdir + "vmlinux"
        archive.gzip_uncompress(kernel_path_gz, kernel_path)

        self.vm.set_console()
        kernel_command_line = ('printk.time=1 ' +
                               'clocksource=GIC console=tty0 console=ttyS0')
        self.vm.add_args('-kernel', kernel_path,
                         '-cpu', 'I6400',
                         '-smp', '8',
                         '-append', kernel_command_line)
        self.vm.launch()
        success_message = 'VFS: Cannot open root device "(null)"'
        wait_for_console_pattern(self, success_message,
                                 failure_message='detected stall')
        self.vm.shutdown()

    def test_mips_malta_i6400_stall(self):
        """
        :avocado: tags=arch:mips64el
        :avocado: tags=machine:malta
        :avocado: tags=cpu:i6400
        """
        logger = logging.getLogger('console')
        for i in range(self.count):
            logger.debug('\n' + '=' * 80)
            self.do_test_mips_malta_i6400_stall()
---

I started bisecting from current master to v2.9.1.

With v4.0.0 I get the stall:
console: [   23.066200] INFO: rcu_sched detected stalls on CPUs/tasks:

With v3.1.0 I couldn't.
PASS (305.11 s)

To be able to build and test multiple commits in parallel, I used the
'-p qemu_bin' option:

$ avocado --show=app,console run -t cpu:i6400 \
  -p qemu_bin=v4.0.0/mips64el-softmmu/qemu-system-mips64el \
  tests/acceptance/

Stats for the day:

$ ccache -s
stats zero time                     Sat Feb  1 13:48:09 2020
cache hit (direct)                 12001
cache hit (preprocessed)            7463
cache miss                         39856
cache hit rate                     32.81 %
called for link                    16945
called for preprocessing             187
compile failed                        65
preprocessor error                   210
couldn't find the compiler             1
unsupported compiler option          429
no input file                        293
cleanups performed                     0
files in cache                    104406
cache size                           7.9 GB

I'll stop commenting in this thread and create a launchpad bug.

Regards,

Phil.