Re: Test suite regressions vs expected failures

[Rik]

On 08/16/2017 05:45 PM, Mike Miller wrote:
> Rik,
>
> You may have missed the discussion about a month ago [1], but there are
> four tests that you added to the test script "for.tst" that are failing
> with the new "regression" status. You added these tests as part of the
> fix for bug #50893 [2].

When I added the tests they were not tagged as regressions, just regular
tests.  The '*' tag should be removed from these BIST tests.

>
> I'm not completely sure what the intent was, were these tests supposed
> to pass as part of the bug fix? Or was your intent to add them as xtests
> that fail because of behavior we are intentionally not following (a testb
> of a "won't fix" feature)?
>
> In [1] we didn't really come to any conclusions about whether we want to
> add tests for things that are "won't fix", or whether to mark them
> somehow so they aren't flagged as regressions. But that's how they look
> for now.
>
> [1]: 
> https://lists.gnu.org/archive/html/octave-maintainers/2017-07/msg00040.html
> [2]: https://savannah.gnu.org/bugs/?50893
>
For bug #50893 we need to decide what level of Matlab compatibility we are
striving for.  The tests in for.tst contain actual expected results as
determined under Matlab.  For example, this test

-- Code --
%! cnt = 0;
%! for k = zeros (0,3);
%!   cnt++;
%! endfor
%! assert (cnt, 0);
%! assert (k, zeros (0,1));
-- End Code --

Both Octave and Matlab now create an empty variable k which is the more
important thing. Of lesser importance, but up for discussion here on the
Maintainer's List, is whether we want exact compatibility.  Octave executes
the initialization statement exactly as written in the for loop so the
variable k is of size 0x3.  Matlab creates the empty variable k but the
size is 0x1.

--Rik