Re: xtest vs test

[Carnë Draug]

On 31 July 2016 at 16:19, John W. Eaton <address@hidden> wrote:
> On 07/31/2016 10:43 AM, Carnë Draug wrote:
>
>> but any failing test makes Octave unreliable.
>> A test checks the expected output and a failing test means it's not doing
>> what it should.  It is unreliable.
>
>
> I think it depends on the reason for the failure.  If it is checking for a
> feature and the feature is missing, it doesn't mean that Octave is
> unreliable, it just means that there is a feature that is not implemented
>
> To me, unreliable means that the program may silently give incorrect
> answers, lose data, or exit unexpectedly.  That is much different from a
> missing feature.
>

Yes, I agree.  But those are already not xtest (expected failures).  They
are "test if" (skipped tests).  Which is why I wrote:

    [...] It's one thing to skip
    tests because Octave was built without support for X, another thing is to
    silence a test because we know of the bug.

I know of some cases where they are marked as xtest and didn't want to
mention them since that's a separate issue and I didn't want to cloud
this one.  There should be a way to check at runtime if something is
not available (last resort would be if the test fails by throwing an
"Octave:disabled-feature" error id --- which does not exist at the
moment).

On 31 July 2016 at 16:31, John W. Eaton <address@hidden> wrote:
> On 07/31/2016 10:43 AM, Carnë Draug wrote:
>
>> So here's a counter proposal.  Let's change all xtest into failing tests
>> (xtests on statistical tests can set rand seed), and add tests that we
>> know
>> are failing from the bug tracker.
>
>
> To avoid a flood of bug reports about the failing tests that we will waste a
> lot of time marking as duplicate and closing, I think we will also need to
> tag these tests with a bug report number and make it VERY obvious that they
> are known failures.  The test summary should be modified to display the
> failures as known, and there should be some indication of what it means to
> see a known failure.  Maybe we can handle this by modifying the xtest
> feature so that it can accept a bug report number or URL.
>
> Showing known/unknown failures would be helpful to me because I look at the
> summary to see whether a change I just made introduced any NEW problems.  If
> there are suddenly many failing tests, I don't think I'll be able to
> remember what the "expected" number of failing tests is, especially if new
> failing tests are constantly being added to the test suite.  Then it will be
> more difficult to know whether a change has screwed something up, or if new
> failing tests have been added.
>

If we change the xtest into failing tests that are marked as known failures,
then we just leave xtest as it is now in place.  Instead, let's just change
the wording in the summary from "expected failures" to "known failures".