Re: [RFC] gitlab: introduce s390x wasmtime job

[Alex Bennée]

Ilya Leoshkevich <iii@linux.ibm.com> writes:

> On Tue, 2022-07-05 at 15:40 +0100, Peter Maydell wrote:
>> On Tue, 5 Jul 2022 at 15:37, Ilya Leoshkevich <iii@linux.ibm.com>
>> wrote:
>> > 
>> > On Tue, 2022-07-05 at 14:57 +0100, Peter Maydell wrote:
>> > > On Tue, 5 Jul 2022 at 14:04, Daniel P. Berrangé
>> > > <berrange@redhat.com>
>> > > wrote:
>> > > > If we put this job in QEMU CI someone will have to be able to
>> > > > interpret the results when it fails.
>> > > 
>> > > In particular since this is qemu-user, the answer is probably
>> > > at least some of the time going to be "oh, well, qemu-user isn't
>> > > reliable
>> > > if you do complicated things in the guest". I'd be pretty wary of
>> > > our
>> > > having
>> > > a "pass a big complicated guest code test suite under linux-user
>> > > mode"
>> > > in the CI path.
>> 
>> > Actually exercising qemu-user is one of the goals here: just as an
>> > example, one of the things that the test suite found was commit
>> > 9a12adc704f9 ("linux-user/s390x: Fix unwinding from signal
>> > handlers"),
>> > so it's not only about the ISA.
>> > 
>> > At least for s390x, we've noticed that various projects use
>> > qemu-user-based setups in their CI (either calling it explicitly,
>> > or
>> > via binfmt-misc), and we would like these workflows to be reliable,
>> > even if they try complicated (within reason) things there.
>> 
>> I also would like them to be reliable. But I don't think
>> *testing* these things is the difficulty: it is having
>> people who are willing to spend time on the often quite
>> difficult tasks of identifying why something intermittently
>> fails and doing the necessary design and implementation work
>> to correct the problem. Sometimes this is easy (as in the
>> s390 regression above) but quite often it is not (eg when
>> multiple threads are in use, or the guest wants to do
>> something complicated with clone(), etc).
>> 
>> thanks
>> -- PMM
>> 
>
> For what it's worth, we can help analyzing and fixing failures detected
> by the s390x wasmtime job. If something breaks, we will have to look at
> it anyway, and it's better to do this sooner than later.

Sorry for necroing an old thread but I just wanted to add my 2p.

I think making 3rd party test suites easily available to developers is a worthy
goal and there are a number that I would like to see including LTP and
kvm-unit-tests. As others have pointed out I'm less sure about adding it
to the gating CI.

If we want to go forward with this we should probably think about how we
would approach this generally:

  - tests/third-party-suites/FOO?
  - should we use avocado as a wrapper or something else?
    - make check-?
  - ensuring the suites output tap for meson
  - document in docs/devel/testing.rst

Also I want to avoid adding stuff to tests/docker/dockerfiles that
aren't directly related to check-tcg and the cross builds. I want to
move away from docker.py so for 3rd party suites lets just call
docker/podman directly.

>
> Best regards,
> Ilya


-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro