Re: help needed to tag (x)failing tests with bug numbers

[John W. Eaton]

On 08/25/2016 01:16 PM, Colin Macdonald wrote:
> On 25/08/16 05:39, John W. Eaton wrote:
> > Note that %!test with a message behaves the same as %!xtest, so I
> > converted the %!xtest instances to %!test.
>
> I like the extra information in that "x".  (The pattern "%!test
> <BUG_ID>" does not inform me that is expected to fail b/c for example
> your second change adds a bug id to at least one test which is currently
> passing.)

Calling these tests "expected failures" was probably the wrong choice. 
These tests are not really "expected to fail".  The intent was more 
like, "this test is known to fail but we don't want to alarm you by 
saying that there is a failure here".  But there IS a failure and it 
should be fixed.  Unfortunately, what happens when you call it an 
"expected failure" instead of a "known bug" is that people ignore it and 
don't even think that it is something that should be examined or fixed.

When running "make check" the tests with bug ids attached to them will 
still be reported as XFAIL.  So there is no change there.

It's fine to have bug ids for tests that pass.  That just means that the 
bug is fixed and the bug id can be ignored.  But if the test fails 
again, the old bug id will be reported.  The failure might not be due to 
exactly the same problem, but maybe it will help.  Existing tests that 
begin to fail are usually noticed fairly quickly so I'm not too worried 
that the messages point to old bug reports.  These bug report ids can 
also give some clues about why a test was added.  It probably wouldn't 
be a bad idea to add the bug id to new tests that are added because of a 
bug report.

> In principle, keeping the "x" would allow us to modify our tooling so
> that passing an xtest is fatal.  This is how the Doctest pkg behaves.

For that case maybe you really want to use "%!fail"?

> Anyway, I assume packages can keep using xtest.

For now it's not going away.  I'd just like to discourage its use.

jwe