Re: [PATCH v3 1/2] tests/avocado: sbsa-ref: add FreeBSD tests

[Philippe Mathieu-Daudé]

On 1/7/24 15:37, Peter Maydell wrote:
> On Mon, 24 Jun 2024 at 17:20, Marcin Juszkiewicz
> <marcin.juszkiewicz@linaro.org> wrote:
> > FreeBSD has longer support cycle for stable release (14.x EoL in 2028)
> > than OpenBSD (7.3 we used is already EoL). Also bugfixes are backported
> > so we can stay on 14.x for longer. Tests done on OpenBSD will now be
> > done using FreeBSD.
> >
> > OpenBSD 7.3 stays with Cortex-A57 test for local runs only.
> >
> > Moved from Neoverse-N1 to Neoverse-N2 as sbsa-ref defaults were changed.
> >
> > Timeout messages expanded to mention being affected by PAuth emulation.
> >
> > Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
> > ---
> >   tests/avocado/machine_aarch64_sbsaref.py | 75 ++++++++++++++++++++++++--------
> >   1 file changed, 56 insertions(+), 19 deletions(-)
> >
> > diff --git a/tests/avocado/machine_aarch64_sbsaref.py 
> > b/tests/avocado/machine_aarch64_sbsaref.py
> > index 6bb82f2a03..bb2c098aac 100644
> > --- a/tests/avocado/machine_aarch64_sbsaref.py
> > +++ b/tests/avocado/machine_aarch64_sbsaref.py
> > @@ -1,4 +1,4 @@
> > -# Functional test that boots a Linux kernel and checks the console
> > +# Functional test that boots a kernel and checks the console
> >   #
> >   # SPDX-FileCopyrightText: 2023-2024 Linaro Ltd.
> >   # SPDX-FileContributor: Philippe Mathieu-Daudé <philmd@linaro.org>
> > @@ -8,7 +8,7 @@
> >
> >   import os
> >
> > -from avocado import skipUnless
> > +from avocado import skipIf, skipUnless
> >   from avocado.utils import archive
> >
> >   from avocado_qemu import QemuSystemTest
> > @@ -165,7 +165,8 @@ def test_sbsaref_alpine_linux_max_pauth_impdef(self):
> >           """
> >           self.boot_alpine_linux("max,pauth-impdef=on")
> >
> > -    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
> > +    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'),
> > +                'Test might timeout due to PAuth emulation')
> >       def test_sbsaref_alpine_linux_max(self):
> >           """
> >           :avocado: tags=cpu:max
> > @@ -173,7 +174,6 @@ def test_sbsaref_alpine_linux_max(self):
> >           """
> >           self.boot_alpine_linux("max")
> >
> > -
> >       # This tests the whole boot chain from EFI to Userspace
> >       # We only boot a whole OS for the current top level CPU and GIC
> >       # Other test profiles should use more minimal boots
> > @@ -200,6 +200,9 @@ def boot_openbsd73(self, cpu):
> >                                    "Welcome to the OpenBSD/arm64"
> >                                    " 7.3 installation program.")
> >
> > +
> > +    # we keep OpenBSD 7.3 on Cortex-A57 by request
> > +    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
>
> Don't skip stuff if GITLAB_CI, please. If we don't want to
> run it on the CI then we shouldn't be running it by default
> when a local user runs check-avocado either.

I did request that.

> Any skip should have a clear description of the reason why
> we're skipping it.

Right. Maybe replace by:

  @skipUnless(os.getenv('AVOCADO_TEST_LEGACY_OS'), 'OpenBSD 7.3 is EoL')

Or AVOCADO_TEST_LEGACY_OPENBSD... Naming is hard.

Regards,

Phil.