octave-maintainers
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: xtest vs test


From: Carnë Draug
Subject: Re: xtest vs test
Date: Sun, 31 Jul 2016 15:43:06 +0100

On 3 July 2016 at 07:17, LachlanA <address@hidden> wrote:
> Greetings all,
>
> In the context of bug 45507 [http://savannah.gnu.org/bugs/?45507], JWE said
> " I don't think anyone likes to see tests that fail, but removing the tests
> is definitely not what I'd like to see...
>
> I'm not even sure I like to see xtests because then people just get used to
> the idea that the tests fail, that they are expected to fail, and then don't
> even look to see why or whether they are problems that should really be
> fixed."
>
> I think this issue applies to many more bugs, so I'm replying here.
>
> I think there are four classes of people.
> 1. "Typical users" who just use the release, and trust that it works.
> 2. Those who compile a release from source, and use "make check" to see
> whether or not to trust the release.
> 3. Those who compile from dev sources, and report bugs.
> 4. Developers.
>
> Group 1 doesn't care about xtest vs test.
>
> To my understanding, xtest is to reassure group 2 that we are aware of the
> bug, and know that they can still trust Octave to work.
>
> For group 3, we'll reinstate a proper (not "x") test as soon as 4.2.0 is
> forked, and missing feedback for a few weeks doesn't matter.
>
> For group 4, we should treat xtest failures as candidates for fixing anyway.
>
> All of this means that I think that all* tests that we know will fail on
> relatively common systems, but do not mean that Octave becomes unreliable,
> should be xtests for the release (and reverted as soon as the 4.2.0 branch
> is forked from default).
>
> * That is an extreme position for argument's sake.  Obviously there will be
> many exceptions.
>
> What do others think?
>

I could swear that the purpose of xtests was for tests that sometimes fail
and couldn't be easily turned into conditional tests.  So I checked the
manual and your premise was correct, xtests are meant to be used on known
bugs that cause the test to fail every time:


    Sometimes during development there is a test that should work but is
    known to fail.  You still want to leave the test in because when the
    final code is ready the test should pass, but you may not be able to
    fix it immediately.  To avoid unnecessary bug reports for these known
    failures, mark the block with xtest rather than test


However, I disagree with "reassure group 2 that we are aware of the bug,
and know that they can still trust Octave to work".  They obviously can't
trust Octave to work if the failing test is on the feature they need.
Using xtest will mean they won't even look into it.

Also, you said "all* tests that we know will fail [...] but do not mean that
Octave becomes unreliable ..." but any failing test makes Octave unreliable.
A test checks the expected output and a failing test means it's not doing
what it should.  It is unreliable.

I also disagree with the manual on the use of xtest (seems like jwe does too).

A test either passes or it fails.  An expected failure is still a failure,
and we shouldn't have a special place for those.  It's one thing to skip
tests because Octave was built without support for X, another thing is to
silence a test because we know of the bug.

The issue xtest addresses is people building Octave and expecting it to be
perfect and fail no test.  The fix for that should be on the expectations
of those people.  There are bugs in Octave, whether we have a test for it
or not.  A failing test will be more informative and also shows more respect
for the user because we should inform them of that issue.

So here's a counter proposal.  Let's change all xtest into failing tests
(xtests on statistical tests can set rand seed), and add tests that we know
are failing from the bug tracker.

Carnë



reply via email to

[Prev in Thread] Current Thread [Next in Thread]