Re: [RFC PATCH v2 00/23] NXP i.MX RT595, ARM SVD and device model unit tests

[Peter Maydell]

On Sat, 17 Aug 2024 at 11:26, Octavian Purdila <tavip@google.com> wrote:
>
> This patch set adds support for NXP's RT500 MCU [1] and the RT595
> EVK[2]. More RT500 device models will be submitted in future patch sets.
>
> The goal of this first patch set is to provide a minimal set that
> allows running the NXP MCU SDK hello world example[4].
>
> The patch set introduces a (python) tool that generates C header files
> from ARM SVD files[3]. This significantly reduces the effort to write
> a new device model by automatically generating: register definitions
> and layout (including bit fields), register names for easier debugging
> and tracing, reset register values, register write masks, etc.
>
> The generated files are commited and not generated at compile
> time. Build targets are created so that they can be easily regenerated
> if needed.
>
> It also introduces unit tests for device models. To allow accessing
> registers from unit tests a system bus mock is created.
>
> This can potentially introduce maintainance issues, due to mocks or
> unit tests getting outdated when code is refactored. However, I think
> this is not an issue in this case because the APIs we mocked (system
> bus MMIO access) or directly used (irq APIs, chardev APIs, clock tree
> APIs) to interact with device models are stable at this
> point. Anecdotally, our experience seems to confirm this: we only run
> into one (trivially fixed) breaking upstream change (gpio getting
> removed from hwcore) in the last three years.

My main issue with the mocking is that it introduces a
completely different way of testing devices that is
not the same as what we use for any existing device.
QEMU already has too many places where there are multiple
different ways or styles of doing something, so adding a
new one should be a high bar (e.g. "this lets us test XYZ
that would be impossible in the old way") and preferably
also have a transition plan for how we would be
deprecating and dropping the old way of doing things.

So my inclination here is to say "you said that you could
do the testing of this device with qtest, so use qtest".
If we were designing a "test devices" framework from
scratch, using mocks would probably be a strong candidate
for that design, but we aren't starting from scratch.

thanks
-- PMM