Re: classpath+jamvm+mauve Howto?

[David Gilbert]

Hi Norman,

Norman Hendrich wrote:

> Hello all,
>
> [I recently checked out Mauve CVS with the good intention of first
> running the testsuite, learning from the existing testcases, and
> then converting my own bug-report testcases into some nice Mauve
> regression tests.]
>  
Excellent news!

> Unfortunately, running Mauve under classpath+jamvm is less fun than
> I expected. Running the default make check:
>
> * works fine with JDK 1.4.2 (129 fails, 6341 tests total), 
>
> * while JDK 1.5.0 dies with a compiler error:
>  gnu/testlet/java/math/BigInteger/compareTo.java:79: 
>    compareTo(java.math.BigInteger) in java.math.BigInteger 
>      cannot be applied to (java.lang.Integer)
>        a.compareTo(new Integer(1));
>
> * and classpath with jamvm simply crashes, with make exiting with 
>  'error 139' or jamvm dying with OutOfMemoryExceptions.
>  
Mauve is a bit of a beast, for sure.   I know next to nothing about 
'make' et al, and I've never run Mauve that way, even though that's how 
it is *meant* to work.  Instead, I run Mauve from Eclipse (run 
gnu.testlet.SimpleTestHarness) and tweak the command line arguments to 
point to a 'test.txt' file that I generate myself.  Right now, the 
command line arguments I'm using are:

-verbose -debug -file MetalFileChooserUITests.txt

...and MetalFileChooserUITests.txt contains all the tests I've written 
for the MetalFileChooserUI class.  It's a short list that looks like this:

gnu.testlet.javax.swing.plaf.metal.MetalFileChooserUI.createActionMap
gnu.testlet.javax.swing.plaf.metal.MetalFileChooserUI.getApproveButton
gnu.testlet.javax.swing.plaf.metal.MetalFileChooserUI.getBottomPanel
gnu.testlet.javax.swing.plaf.metal.MetalFileChooserUI.getButtonPanel
gnu.testlet.javax.swing.plaf.metal.MetalFileChooserUI.getFileName
gnu.testlet.javax.swing.plaf.metal.MetalFileChooserUI.getMaximumSize
gnu.testlet.javax.swing.plaf.metal.MetalFileChooserUI.getMinimumSize
gnu.testlet.javax.swing.plaf.metal.MetalFileChooserUI.getPreferredSize
gnu.testlet.javax.swing.plaf.metal.MetalFileChooserUI.setFileName

This is quite a good way to develop Mauve tests, since you only need to 
run the tests you are working on.

When I want to run ALL the tests, I have a Java program (attached) that 
scans the source files and generates a text file containing the test 
names.  There is a similar program in CVS: 
gnu.testlet.runner.CreateTags, but I haven't tried that one yet.

> (1) Can anyone please provide some feedback whether mauve+classpath+jamvm
>    is supposed to work out-of-the-box?  
>    
>    If it is not, we should somehow tweak mauve to make running the
>    testsuite easier. It will be very hard to convince developers to run
>    the mauve tests at all, if this requires a lot of extra effort.
I agree that the 'new to Mauve' experience should be easier.  I don't 
have the skills to do anything about the 'make' tools, but I'm certainly 
happy to document how I currently run Mauve tests (even though my 
methods are a little primitive!).

>    Given that many people seem to use jamvm for classpath hacking,
>    I guess jamvm would also be useful as a 'reference VM' for mauve.
>    
> (2) If anyone has a current list of unsupported tests and tweaks to
>    get Mauve running with jamvm, please post them...  A list of XFAILS
>    would also be nice, because I have no idea whether some tests (like
>    the korean? locale tests) are expected to pass or fail.
>    
>    I would also appreciate a short howto or command lines for the mauve
>    configure and make check scripts, including magic options like
>    initial and maximum memory sizes, etc.
>  
The list at the bottom of:

http://www.object-refinery.com/classpath/mauve/report/

...are the only tests that wouldn't run for me with JamVM.  That report 
is a couple of weeks old, and I will update it soon.

> About Mauve: (please excuse me for asking this here, but I guess that
> many more people are subscribed to the classpath mailing list than the
> mauve-discuss list, and one question should not produce to much noise):
>
> (3) several of the GUI tests leave their windows lying around... 
>    is this intentional or just an oversight?  (I did run the tests on
>    my normal desktop X server, not the extra hidden test X server)
That happens for me sometimes too.  I've managed to ignore the problem so far.    

> (5) Are the results linked from the classpath homepage really the best 
>    we can do?
>    
>    http://www.object-refinery.com/classpath/mauve/report/
>  
It seems to be, so far.  I've no doubt the job could be done better, but 
no one else has stepped up to volunteer yet, so I decided to publish 
what I have.  It beats the '404 Page not found' error that was there before.

>    
> (6) How do I enable all 32157 tests listed in the above report, when
>    the default cvs checkout (see above) only runs 6341 tests with 
>    the JDK?
>  
I've attached the program I ran to generate the text file containing the 
list of all test cases that appear in the report in (5) above.  When I 
ran these tests on jamvm, I had to increase the memory to get it to run 
through.  For the first few runs, the process would hang.  When that 
happened, I commented out the name of the test that caused the hang, and 
reran the report.  You can see the names of the tests that I commented 
out at the end of the report.

Mauve is important, so I'm glad you are taking an interest in it.  
Please ask me if you have any questions about what I've written above.

Regards,

Dave Gilbert


> - Norman
>
>
>
> _______________________________________________
> Classpath mailing list
> address@hidden
> http://lists.gnu.org/mailman/listinfo/classpath
>
>
>  

// Copyright (C) 2004 by Object Refinery Limited
// Written by David Gilbert (address@hidden)

// This file is part of Mauve Reporter.

// Mauve Reporter is free software; you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation; either version 2 of the License, or
// (at your option) any later version.

// Mauve Reporter is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
// GNU General Public License for more details.

// You should have received a copy of the GNU General Public License
// along with Mauve Reporter; if not, write to the Free Software
// Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA

package org.jfree.test;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.io.LineNumberReader;
import java.io.PrintWriter;
import java.util.ArrayList;
import java.util.List;

/**
 * A utility class that searches the Mauve test source files and extracts the 
 * names of the classes containing tests with a specific tag.
 */
public class MakeTestClassList 
{

  /** The source directory. */
  private String sourceDir;
  
  /** The package prefix. */
  private String prefix;
  
  /**
   * Creates a new instance.
   * 
   * @param sourceDir  the source directory.
   * @param prefix  the package prefix.
   */
  public MakeTestClassList(String sourceDir, String prefix) 
  {      
    this.sourceDir = sourceDir;
    this.prefix = prefix;
  }

  /**
   * Returns the source directory.
   * 
   * @return The source directory.
   */
  public String getSourceDir() 
  {
    return this.sourceDir;    
  }
  
  /**
   * Returns the package prefix.
   * 
   * @return The package prefix.
   */
  public String getPrefix() {
      return this.prefix;
  }
  
  /**
   * Returns a list containing the names of the tests in the specified
   * directory that have the specified tag.
   * 
   * @param dir  the directory.
   * @param tags  the tags.
   * 
   * @return A list of class names.
   */
  public List getTests(File dir, String[] tags) 
  {
    List result = new ArrayList();
    result = processDirectory(dir, tags, result, true);
    return result;
  }
    
  /**
   * Processes the contents of a directory looking for Mauve test files 
   * containing a particular tag.
   * 
   * @param dir  the directory.
   * @param tags  the tags.
   * @param tests  the list of tests to append to.
   * @param recurseSubDirectories  a flag that controls whether subdirectories 
   *                               are processed.
   * 
   * @return A list of class names.
   */
  public List processDirectory(File dir, String[] tags, List tests, 
                               boolean recurseSubDirectories) 
  {
    String[] files = dir.list();
    for (int i = 0; i < files.length; i++) 
    {
      System.out.println(files[i]);
      File f = new File(dir, files[i]);
      if (f.isDirectory()) 
      {
        processDirectory(f, tags, tests, true);
      }
      else 
      {
        // do we need to add to the list
        if (f.getName().endsWith(".java")) 
        {
          processFile(f, tags, tests);
        }
      }
    }
    return tests;
  }
   
  private boolean containsTag(String line, String[] tags) 
  {
    boolean result = false;
    for (int i = 0; i < tags.length; i++) 
    {
      if (line.indexOf(tags[i]) >= 0) {
          result = true;
      }
    }
    return result;
  }
  
  private boolean containsTag(File f, String[] tags) 
  {
    try
    {
      FileReader source = new FileReader(f);
      LineNumberReader r = new LineNumberReader(source);
      boolean done = false;
      while (r.ready() && !done)
      {
        String line = r.readLine();
        if (line.startsWith("package"))
        {
          done = true;
        }
        else
        {
          if (containsTag(line, tags)) 
          {
            return true;
          }
        }
      }
    }
    catch (FileNotFoundException e)
    {
      e.printStackTrace();
    }
    catch (IOException e) 
    {
      e.printStackTrace();
    }
    return false;
      
  }
  
  /**
   * Processes a single file.
   * 
   * @param f  the file.
   * @param tags  the tags.
   * @param tests  the list of tests to append to
   * 
   * @return The list of tests.
   */
  public List processFile(File f, String[] tags, List tests) {
    if (containsTag(f, tags)) 
    {
      String fullName = f.getParent() + "/" + f.getName();
      String shortName = fullName.substring(
        this.sourceDir.length(), fullName.length() - 5
      );
      String className = shortName.replace('/', '.');
      tests.add(className);
    }
    return tests;
  }
    
  /**
   * Runs the test list generator.  Currently the values are hard-coded, they 
   * should become command line parameters.
   * 
   * @param args  the command line arguments (first argument should be Mauve 
   *              root directory, second should be the output filename).
   */
  public static void main(String[] args) {
    MakeTestClassList app = new MakeTestClassList(args[0], "gnu/testlet");
    File rootDirectory = new File(app.getSourceDir() + app.getPrefix());
    List tests = app.getTests(
      rootDirectory, new String[] {
        "JDK1.0", "JDK1.1", "JDK1.2", "JDK1.3", "JDK1.4"
      }
    );
    File f = new File(args[1]);
    try
    {
      PrintWriter out = new PrintWriter(new FileWriter(f));
      for (int i = 0; i < tests.size(); i++) 
      {
        out.println(tests.get(i));
      }
      out.close();
    }
    catch (IOException e) 
    {
      e.printStackTrace();
    }
  }
}