Re: [RFC PATCH-for-5.2] gitlab-ci: Do not automatically run Avocado integration tests anymore

[Daniel P . Berrangé]

On Mon, Nov 30, 2020 at 10:03:35AM +0100, Thomas Huth wrote:
> On 27/11/2020 19.46, Philippe Mathieu-Daudé wrote:
> > On 11/27/20 7:29 PM, Thomas Huth wrote:
> >> On 27/11/2020 18.57, Philippe Mathieu-Daudé wrote:
> >>> On 11/27/20 6:47 PM, Thomas Huth wrote:
> >>>> On 27/11/2020 18.41, Philippe Mathieu-Daudé wrote:
> >>>>> We lately realized that the Avocado framework was not designed
> >>>>> to be regularly run on CI environments. Therefore, as of 5.2
> >>>>> we deprecate the gitlab-ci jobs using Avocado. To not disrupt
> >>>>> current users, it is possible to keep the current behavior by
> >>>>> setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable
> >>>>> (see [*]).
> >>>>> From now on, using these jobs (or adding new tests to them)
> >>>>> is strongly discouraged.
> >>>>>
> >>>>> Tests based on Avocado will be ported to new job schemes during
> >>>>> the next releases, with better documentation and templates.
> >>>>
> >>>> Why should we disable the jobs by default as long as there is no 
> >>>> replacement
> >>>> available yet?
> >>>
> >>> Why keep it enabled if it is failing randomly
> >>
> >> We can still disable single jobs if they are not stable, but that's no
> >> reason to disable all of them by default, is it?
> >>
> >>> if images hardcoded
> >>> in tests are being removed from public servers, etc...?
> >>
> >> That's independent from Avocado, you'll always have that problem if you 
> >> want
> >> to test with external images, unless you mirror them into a repository on
> >> the same server (ie. gitlab), which, however, might not always be 
> >> possible...
> >>
> >>> They are not disabled, they are still runnable manually or setting
> >>> QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE...
> >>
> >> And who do you think is going to set that variable? Hardly anybody, I 
> >> guess.
> > 
> > Does that mean nobody cares about these tests?
> 
> It's like with all the other tests: Most of the people do not really care
> about them (if they are not the author of a test) unless the test fails
> during "make check" / the gating CI of Peter. So IMHO the right way to go is
> to finally get these in the gating CI, otherwise, if you now even disable
> them in the gitlab-CI by default, they will bitrot completely.

That people don't care, and ignore it until Peter hits the failure during
merge is a tragedy of the commons in itself.

I think we need to set expectations that caring about tests is a key part
of every contributor's responsibility, with subsystem maintainers leading
by example:

   https://lists.gnu.org/archive/html/qemu-devel/2020-11/msg04897.html

We do need tests to be reliable though when we're treating them as gating.

Hiding unreliable tests behind an env variable you have to opt-in to
setting is not going to help that. IMHO unreliable tests should be
just disabled entirely. If someone genuinely does care about the test
then they can fix it and re-enable it at the same time. 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|