Re: Test suite regressions vs expected failures

[John W. Eaton]

On 08/21/2017 08:02 PM, Mike Miller wrote:
> On Sat, Aug 19, 2017 at 22:24:51 -0700, Rik wrote:
> > When I added the tests they were not tagged as regressions, just regular
> > tests.  The '*' tag should be removed from these BIST tests.
>
> To be clear, the '*' tag does not mark the test as a regression. The '*'
> tag is supposed to mean that the bug number has been marked as fixed on
> the bug tracker. No '*' means that the bug is still open. If I see a
> test marked with "<12345>", that means there may still be work to be
> done.
>
> If a test fails it is flagged as either an expected failure or a
> regression, depending on whether we thought the bug was fixed or not.
>
> Any new tests that are added for bugs when they are fixed should include
> the '*' marker.

Yes, so the problem is that these tests are tagged with a report number 
that has been closed as fixed.

If we intend to fix these problems, then either the bug report should 
not be closed as fixed, or we need a new report that is not closed.

If we close the report as "won't fix", then should we have tests that 
fail forever?  Or should we have another way to flag them in the tests 
so that they are reported as "issues we know about but don't plan to 
fix"?  That was the question I was originally trying to ask.

jwe