Re: Mauve test question

[Archie Cobbs]

Stephen Crawley wrote:
> I think (d) is mis-characterized.  If a test fails but it isn't a VM
> or Classpath bug, then >>surely<< it must be a fault in the testcase
> and/or the Mauve framework.  Please let's be clear about this.
>
> For example:
>
>   1) Tests for behavior that is beyond the scope of "the standard".
>      For example, testcases that test the names of classes returned by 
>      certain factory methods, or that test exceptions message strings
>      are fundamentally broken.
>
>   2) Tests that have been designed to match (Sun acknowledged) buggy
>      JDK behavior or buggy specs / javadocs.
>
>   3) Tests that assume some VM-specific functionality or behavior.  
>      For example, almost any test for GC-related behavior is inevitably 
>      VM-specific, since there is no VM-independent way to force the GC 
>      to run.
>
>   4) Tests that depend on environmental conditions.  For example,
>      some existing tests assume that the host machine can open socket 
>      connections to random machines/ports on other network.  These
>      break for me because I am behind a firewall.
>
> IMO, all of the above are actually bugs in the Mauve testcases, and/or
> problems in the way that they are organized / classified.

Yep, that's a good classification. My point is simply that even with these 
problems, the tests are still useful, and people want to run them, so we 
need some way of handling them.

-Archie

__________________________________________________________________________
Archie Cobbs      *        CTO, Awarix        *      http://www.awarix.com