Re: [Qemu-devel] QEMU+Linux ARMv7A current state

[Peter Crosthwaite]

On Sat, Oct 3, 2015 at 5:21 PM, Guenter Roeck <address@hidden> wrote:
> On 10/03/2015 02:31 PM, Peter Crosthwaite wrote:
>>
>> Hi,
>>
>> I have done an audit of the ARMv7 boards to see what can boot a
>> vanilla linux kernel. The basic approach is to build ARM
>> multi_v7_defconfig kernel and boot QEMU using the DTBs built out by
>> the kernel. The intersection of what mainline Linux has a DTB for and
>> what QEMU models is tested. The boards that do/should work include:
>>
>>      vexpress-a9
>>      vexpress-a15
>>      smdkc210 (exynos)
>>      xilinx-zynq-a9
>>      cubieboard (allwinner)
>>      highbank
>>      midway
>>      virt
>>
>> Have I missed anything?
>>
>> The basic rule is, no source code, config, or DTB edits allowed (i.e.
>> what is a day0 user of QEMU+Linux going to experience?).
>>
>> So going board-by-board, these are the issues.
>>
>> highbank/midway:
>>
>> The kernel expects a secure monitor to be present which does at-least
>> two things:
>>
>> * PSCI
>> * Some cache-maintainance ops.
>>
>> QEMU has PSCI support, but the 0.1 command codings specified in the
>> DTB are different to QEMUs 0.1 encodings. So I switched monitor mode
>> for the CPUs on (it was conservatively set to off by the original
>> authors of the QEMU ARM monitor support) and turned PSCI on. I had to
>> write some patches to remap the PSCI encodings to highbank's custom
>> ones and then the kernel at least entry-points all 4 cpus. It then
>> deadlocks on something that I am yet to figure out. So the PSCI SMP
>> stuff is shelved.
>>
>> The cache-maintenance is a bigger issue, that stops even a single core
>> boot. The kernel cache controller driver is doing an smc that QEMU
>> goes haywire on (as there simply is no firmware to catch the smc). But
>> caches in QEMU are a nop, so I wrote some monitor firmware that just
>> erets the smc without action and it works.
>>
>> I am using Sata (sysbus-ahci) for the boot media which works straight
>> out of the box.
>>
>> So single core highbank works with the dummy monitor firmware and the
>> monitor mode restrictor removed.
>>
> I have highbank working with multi_v7_defconfig. It generates a warning,
> though.
>
> WARNING: CPU: 0 PID: 0 at arch/arm/mach-highbank/highbank.c:60
> highbank_l2c310_write_sec+0x34/0x58()
> [    0.000000] Highbank L2C310: ignoring write to reg 0x104
>

Yes I am ignoring this.

> I think this is a kernel problem.
>
> It also generates an SMC request.
>
> SMC cmd 0x102 val 0x1
>
> My "fix" for that is to simply log and ignore all SMC requests in qemu.
> Not really feasible for production qemu, but after all I am testing the
> kernel,
> not qemu. That also helped for various other HW. Maybe your monitor firmware
> can help me get rid of that.
>

Yes. My change doesn't touch target-arm and keeps it all in highbank.c.

> On top of that, I had to enable EL3. I don't remember why, just that it was
> needed.
>

To bring your SMC printf into play most probably. It would just undef
instruction otherwise. I have the same change in my tree.

>
>> cubieboard:
>>
>> The kernel expects a system level clock controller device that isn't
>> modelled. A dummy model which hardcodes all registers to a know reset
>> value and just lets the kernel read and write whatever it wants
>> resolves. I'm not 100% sure whether this is needed but the kernel
>> prints multiple errors with many pages of traces without. So it is at
>> least needed to reduce the dmesg noise.
>>
>> QEMU cubieboard has no usable storage media, but the real hardware
>> does have AHCI sata. I added sysbus-ahci at the right place but turns
>> out the SATA controller has some custom power/clock (not really
>> sure??) registers specific to this SoC. It sets/clears bits then polls
>> them back expecting them to change to the other value asynchronously.
>> The kernel device probe then times-out. So I subclassed sysbus-ahci
>> and added the missing registers and forced the polled registers to the
>> "I'm done" state. It works.
>>
>> I am using meta-sunxi Yocto-layer to build out the allwinner custom
>> kernel/rootfs etc, and with the clock and Sata changes I get a boot.
>> But when I change to the unedited kernel+dtb+rootfs I get stuck. RTC
>> messages are around the point of failure which is not modelled in
>> QEMU, so that is suspect.
>>
>> So with a dummy clock-controller and a slightly modified Sata
>> controller (as a new device) cubieboard progresses, but it is still
>> stuck on something that looks RTC related.
>>
>> xilinx-zynq-a9:
>>
>> The kernel wants to use cpufreq to set the CPU frequency. The kernel
>> assumes ownership of the CPU clock divider but not the PLLs. QEMU uses
>> the HW default for the silicon (x26) for the PLL which is incompatible
>> with the dtb-set CPU frequency. The kernel boot asserts this and
>> crashes. The ideal solution is to provide some sort of pre-boot
>> firmware to set it to a compatible value. In real HW, this is the FSBL
>> bootloader.
>>
>> TBH I think this is a kernel bug, as since cpufreq can be deconfigured
>> it is ultimately optional. This means it shouldn't be a showstopper on
>> a boot. The kernel should just switch off cpufreq and proceed and
>> behave as if the feature was never configured in the kernel.
>>
>> There is also a critical bug in the SLCR model (my bad) where write to
>> registers are ignored. I have a patch.
>>
>> More reading about the SLCR issues here (thanks to Guenter for getting
>> the ball rolling here):
>>
>> https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg03374.html
>>
> Seemed to be going nowhere, though :-(.
>

I'll get there :)

>> I have a patch with sets up the needed SLCR values from software but
>> it needs more work.
>>
>> The kernel also expect the ADC device which is absent. Thanks to
>> Guenter who patched this in which is under review:
>>
>> https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg03375.html
>>
> Still working on the review comments.
>
>
>> I am using SD card for the rootfs media. SD has a bug where it cannot
>> handle a backing file of a non-round size. I am just passing the ext3
>> image in unchanged (so unpadded) to QEMU and QEMU rounds down the
>> image to the SD block size (512k!!), so I am losing us to 512k off the
>> end of my ext3 filesystem. I have a hacky patch that changes this to
>> round-up.
>>
>> With the minimal firmware, slcr bugfix, Guenters ADC, the SD hack a
>> vanilla Zynq will be in good shape.
>>
>> virt:
>>
>> Virt largely works, but there are no immediately obvious
>> storage/network options that are supported by the kernel. So I have to
>> make some exceptions to the kernel config rule, that is, I add:
>>
>> CONFIG_VIRTIO_PCI=y"
>> CONFIG_VIRTIO_BLK=y"
>> CONFIG_VIRTIO_NET=y"
>>
>> To get disks and network for the virt board.
>>
>> Could we add these configs to mainline Linux's defconfig? Note, we
>> don't need to add a DTS for this board, as QEMU's bootloader will
>> generate and provide it for us.
>>
>> There is the known highmem PCI issue that is currently under discussion:
>>
>> https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg06529.html
>>
>> So aside from the missing VIRTIO kernel configs and highmem bug, virt
>> is in good shape.
>>
>> vexpress:
>>
>> vexpress boots up to rootfs probing, however the only storage media
>> that seems to be supported is flash, which doesn't have the needed
>> kernel configs to boot from (not 100% sure on that but I gave up
>> quickly). What are people who use this board in real hardware doing?
>> It may just work best with initrd style boots.
>>
> I use it with
>         -drive file=rootfs.ext3,format=raw,if=sd \
>         -append "root=/dev/mmcblk0 ..."
>

I now have the same. Did you run into the 512k bug? or are your
padding your rootfs images?

> with multi_v7_defconfig and vexpress-v2p-ca9.dtb. This works for both
> vexpress-a9 and vexpress-a15 (with vexpress-v2p-ca15-tc1.dtb).
>
>> exynos:
>>
>> QEMU models an Exynos skdkc210 but there is only a 310 available in
>> QEMU dtbs. QEMU exynos however does not model a lot on the board level
>> so we can largely get away with booting 310 dtb on 210.
>>
> I needed a couple of fixes to get the smdkc210 machine to work with
> exynos4210-smdkv310.dtb.
>
> - CMU support was submitted a long time ago by Maksim Kozlov, but never
>   accepted. I took the code and merged the absolutely necessary parts of
>   (I dropped clock support).
> - exynos4210_mct needs a one-line bug fix to initialize a timer before
>   it is used.
> - dma support for exynos4 must be enabled (pl330 must be instantiated).
> - If I recall correctly, smdkc210 also needed SMC support.
>
> I have to change the configuration file, though. QEMU currently does not
> implement exynos cpuidle, so I have to disable CONFIG_CPU_IDLE and
> CONFIG_ARM_EXYNOS_CPUIDLE.
>
> I test both multi_v7_defconfig and exynos_defconfig with -M smdkc210.
>

Are your changes only needed for the exynos_defconfig?

>> Similar to vexpress, the exynos board has a lack of usable boot media.
>> The board does model USB (EHCI), however the DTB disables it as the
>> smdk310 board does not support it. So the alternative would be to use
>> one of the more full-featured exynos DTBs  (with way more peripherals)
>> from mainline Linux, but the kernel is liable to then probe all sorts
>> of missing hardware.
>>
>> Exynos SDHCI support was on list for a few revs and quite some time,
>> merging it and using SD may solve the problem.
>>
>> I have not attempted the SMP support.
>>
> I run smdkc210 with "-smp 2". If I recall correctly, this is mandatory,
> but I may be wrong.
>

Makes sense. I saw a warning to that effect.

>> Unlike all other boards, Exynos is quite consuming of RAM. All other
>> boards work with Yocto's default of 128MB but Exynos crashes unless
>> you jack it up. So users need to be careful of the -m option.
>>
>> realview:
>>
>> Realview is not tested, but I discuss it here as it is a bit of a red
>> herring. The kernel has dtb support for the realview-1176 board,
>> however this is not modelled by QEMU. I creating a Realview variant
>> with 1176 in QEMU, however the memory maps of all the peripherals were
>> mismatched.
>>
>> Turns, out, QEMU is is modelling the realview FPGA emulation platform
>> which has a different peripheral map to the 1176 realview board
>> proper. So QEMU doesn't support the realview boards, just the FPGA
>> emulation platform, for which I can't find a kernel dtb. Should we
>> upstream one with some dtsi's for the CPU variation supported in QEMU?
>>
>> So Realview does not work within my testing parameters with no
>> immediate prospects.
>>
> I have realview-pb-a8, realview-eb-mpcore, and realview-eb working,
> but not with a default configuration.
>

So if it can be reduced to a "ARCH=arm make some_defconfig; make -j;"
I think we should go for it. Your repo has what looks like
QEMU-specific realview defconfigs. Upstream has a realview_defconfig,
is this likely to work or are there issues?

>> how I am testing:
>>
>> So this is all powered by the Yocto project (Poky). I got some good
>> help from Nathan Rossi. I have a poky fork which changes the qemuarm
>> target to build my mainline (4.2.1) kernel (multi_v7_defconfig +
>> VIRTIO), all the DTBs and a usable rootfs. You can then specify the
>> type of machine (from the list above) to yocto's runqemu command. The
>> command sets up boot media and network automatically.
>>
>> To use it:
>>
>> git clone https://github.com/pcrost/poky.git
>> cd poky
>> source ./oe-init-build-env
>> MACHINE=qemuarm bitbake core-image-minimal #this takes a while
>> QEMUBIN=/path/to/qemu-arm MACHINE_SUBTYPE=virt runqemu qemuarm slirp
>>
> Just for reference, my code is at github.
>
> https://github.com/groeck/linux-build-test
> https://github.com/groeck/qemu
>

There's some cool stuff in your QEMU branch. I made some high level
comments on the github. The IMX timer will need a bit of a rebase
following Jean-Christophe's refactoring.

Regards,
Peter

> Guenter
>