Re: Mauve test question

[Stephen Crawley]

address@hidden said:
> The point is that every test has a known state, one of:
> (a) don't even bother trying it (doesn't compile or we know it fails)
> (b) it is a known failure (i.e., a Classpath bug) 
> (c) it should pass (i.e., if it fails, it must be a JVM bug) 
> (d) it should pass but there can be false negatives depending on the JVM

Ahem ...

I think (d) is mis-characterized.  If a test fails but it isn't a VM
or Classpath bug, then >>surely<< it must be a fault in the testcase
and/or the Mauve framework.  Please let's be clear about this.

For example:

  1) Tests for behavior that is beyond the scope of "the standard".
     For example, testcases that test the names of classes returned by 
     certain factory methods, or that test exceptions message strings
     are fundamentally broken.

  2) Tests that have been designed to match (Sun acknowledged) buggy
     JDK behavior or buggy specs / javadocs.

  3) Tests that assume some VM-specific functionality or behavior.  
     For example, almost any test for GC-related behavior is inevitably 
     VM-specific, since there is no VM-independent way to force the GC 
     to run.

  4) Tests that depend on environmental conditions.  For example,
     some existing tests assume that the host machine can open socket 
     connections to random machines/ports on other network.  These
     break for me because I am behind a firewall.

IMO, all of the above are actually bugs in the Mauve testcases, and/or
problems in the way that they are organized / classified.

-- Steve