Re: [PATCH] Ignore flaky ob-python tests

[Christian Köstlin]

On Wed, Nov 2, 2022 at 7:52 AM Ihor Radchenko <yantar92@posteo.net> wrote:
>
> Christian Köstlin <christian.koestlin@gmail.com> writes:
>
> > The new async feature of python seems to be implemented in a way
> > that can break with high CPU load. This patch still runs the 3 tests in
> > question, but does not fail the complete testsuite if any of those fail.
>
> Thanks for the patch, but I'm afraid that it will drive the tests useless.
> After the patch, even if the tests fail for a valid reason, it will not
> be visible upon running make test, AFAIU.
I think we know of the async problem, and you reported that as well,
so I would expect
if someone is able to fix it, he will also move the tests to "sharp" again.
With my patch the tests are still failing but the testsuite continues
to run and returns
with 0 as status code (one can see the failed test though).

> Maybe we can instead provide some variable that can turn-on ignoring
> some problematic tests only during CI tests?
Yes ... that would also be possible e.g. skip the test or test nothing
if it's running on the ci.
(I am talking only about the 3 tests!).

Kind regards,
Christian