Re: Possible race condition in aspeed ast2600 smp boot on TCG QEMU

[Stephen Longfield]

Sort of resolved: We were able to find a good-enough workaround. In case anyone else is running into this, here's what we did:

By dropping to the uboot console and running the command

```

mw.l 0x1e6e2188 0xbabecafe

```

The magic number is set in the SCU regardless of how the race goes, and the 2nd core gets released from its mailbox polling loop.

On Thu, Jan 11, 2024 at 9:38 AM Stephen Longfield <slongfield@google.com> wrote:

We’ve noticed inconsistent behavior when running a large number of aspeed ast2600 executions, that seems to be tied to a race condition in the smp boot when executing on TCG-QEMU, and were wondering what a good mediation strategy might be.

The problem first shows up as part of SMP boot. On a run that’s likely to later run into issues, we’ll see something like:

```
[    0.008350] smp: Bringing up secondary CPUs ...
[    1.168584] CPU1: failed to come online
[    1.187277] smp: Brought up 1 node, 1 CPU
```

Compared to the more likely to succeed:

```
[    0.080313] smp: Bringing up secondary CPUs ...
[    0.093166] smp: Brought up 1 node, 2 CPUs
[    0.093345] SMP: Total of 2 processors activated (4800.00 BogoMIPS).
```

It’s somewhat reliably reproducible by running the ast2600-evb with an OpenBMC image, using ‘-icount auto’ to slow execution and make the race condition more frequent (it happens without this, just easier to debug if we can reproduce):


```
./aarch64-softmmu/qemu-system-aarch64 -machine ast2600-evb -nographic -drive file=~/bmc-bin/image-obmc-ast2600,if=mtd,bus=0,unit=0,snapshot=on -nic user -icount auto
```

Our current hypothesis is that the problem comes up in the platform uboot.  As part of the boot, the secondary core waits for the smp mailbox to get a magic number written by the primary core:

https://github.com/AspeedTech-BMC/u-boot/blob/aspeed-master-v2019.04/arch/arm/mach-aspeed/ast2600/platform.S#L168

However, this memory address is cleared on boot:

https://github.com/AspeedTech-BMC/u-boot/blob/aspeed-master-v2019.04/arch/arm/mach-aspeed/ast2600/platform.S#L146

The race condition occurs if the primary core runs far ahead of the secondary core: if the primary core gets to the point where it signals the secondary core’s mailbox before the secondary core gets past the point where it does the initial reset and starts waiting, the reset will clear the signal, and then the secondary core will never get past the point where it’s looping in `poll_smp_mbox_ready`.

We’ve observed this race happening by dumping all SCU reads and writes, and validated that this is the problem by using a modified `platform.S` that doesn’t clear the =SCU_SMP_READY mailbox on reset, but would rather not have to use a modified version of SMP boot just for QEMU-TCG execution.

Is there a way to have QEMU insert a barrier synchronization at some point in the bootloader?  I think getting both cores past the =SCU_SMP_READY reset would get rid of this race, but I’m not aware of a way to do that kind of thing in QEMU-TCG.

Thanks for any insights!

--Stephen

---

P.S. Additional note about the aspeed platform.S:

Clearing the mailbox was added in this patch:

https://github.com/AspeedTech-BMC/u-boot/commit/55825c55d1dabc00e37999a38495ed05c901bec2

At the time, the write to what was then known as `=AST_SMP_MBOX_FIELD_READY` (now `=SCU_SMP_READY`) happened after `scu_unlock`.  But, when the boot flow was revised in

https://github.com/AspeedTech-BMC/u-boot/commit/46a48bbe56c1e790c9bd1794364db86ec609c48e the scu_unlock was moved to primary core boot, so, unless the primary core wins the race, it doesn’t seem like the mailbox ready clear actually will have any effect, since it’ll be writing while the SCU is locked.