Re: xtest vs test

[Carlo De Falco]

On 3 Jul 2016, at 08:17, LachlanA <address@hidden> wrote:

> Greetings all,
> 
> In the context of bug 45507 [http://savannah.gnu.org/bugs/?45507], JWE said
> " I don't think anyone likes to see tests that fail, but removing the tests
> is definitely not what I'd like to see...
> 
> I'm not even sure I like to see xtests because then people just get used to
> the idea that the tests fail, that they are expected to fail, and then don't
> even look to see why or whether they are problems that should really be
> fixed."
> 
> I think this issue applies to many more bugs, so I'm replying here.
> 
> I think there are four classes of people.
> 1. "Typical users" who just use the release, and trust that it works.
> 2. Those who compile a release from source, and use "make check" to see
> whether or not to trust the release.
> 3. Those who compile from dev sources, and report bugs.
> 4. Developers.
> 
> Group 1 doesn't care about xtest vs test.
> 
> To my understanding, xtest is to reassure group 2 that we are aware of the
> bug, and know that they can still trust Octave to work.
> 
> For group 3, we'll reinstate a proper (not "x") test as soon as 4.2.0 is
> forked, and missing feedback for a few weeks doesn't matter.
> 
> For group 4, we should treat xtest failures as candidates for fixing anyway.
> 
> All of this means that I think that all* tests that we know will fail on
> relatively common systems, but do not mean that Octave becomes unreliable,
> should be xtests for the release (and reverted as soon as the 4.2.0 branch
> is forked from default).
> 
> * That is an extreme position for argument's sake.  Obviously there will be
> many exceptions.
> 
> What do others think?
> 
> Cheers,
> Lachlan

I like your suggestions! 

What about also adding an option to test such as 'treat_xtest_as_test' 
and set it on for development and off for release?

This would make the procedure you suggest more automatic ...

c.