[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Converting DejaGNU tests to Paul's test/assert infrastructure
|
From: |
David Bateman |
|
Subject: |
Re: Converting DejaGNU tests to Paul's test/assert infrastructure |
|
Date: |
Wed, 26 Oct 2005 11:22:03 +0200 |
|
User-agent: |
Mozilla Thunderbird 0.8 (X11/20040923) |
Paul Kienzle wrote:
On Oct 24, 2005, at 9:20 AM, David Bateman wrote:
However, nested functions are not permitted with Paul's test
infrastructure. I haven't looked at test.m yet to see, but how
complicated would it be to include nested functions in test.m?
You should now be able to build test function blocks with the function
name being the name of a shared variable which will last until the end
of the test or until the next 'shared' block.
Another issue I have is that although "%!error" exists there is no
equivalent "%!warning" for the warning conditions. Again how
complicated would it be to add this?
Octave can only test the first warning since lastwarn isn't updated
after the first warning. Presumably it resets itself after returning
to the prompt. Also the warnings will be displayed during the test.
Regardless, I've added a warning block similar to the error block:
%!warning warning("message text")
If you want to check the contents of the warning or error message,
then use the new syntax '%!error <pattern> code'. For example:
%!error <nonconformant> [1,2,3]*[4,5,6]
%!warning <division by zero> 5/0
Pattern can be any regular expression on the string with leading
'error: ' and trailing '\n' stripped off.
I first extended the 'fail' command to allow
fail(code,'warning',pattern) which will check the value of lastwarn
against the pattern, but I decided the tests looked too ugly.
- Paul
Attached find my updated perl script that uses these new features. Note
that I only expect to run this script once in the end and so don't care
abut the quality of the script itself or that it misses a few tests that
can be converted by hand. I also used a small patch to the DejaGNU tests
to simplify the job of doing the conversion.
Things that really still have issues are
* tests for warning messages aren't work. This needs changes to octave
as discussed
* The functions in the test scripts are converted to funtion handles and
so things like
fsolve("f", ....) won't work. Paul are the function handles really
needed, or is there a way to
get the same functionality in the test scripts without nested functions?
* The converted umask-1.m causes problems with the path in test.m that I
don't understand and so have disabled this test.
* The test suite itself need to be converted to be acceptable for octave
itself; texinfo help, coding style(?) and documentation in the manual
itself. For the documentation, I'm inclined to steal the help from
test.m and rewrite a shorter help for test.m. Another issue in this
conversion is what to do with regex, as Paul points out that the syntax
of the regex can be improved.
It seems to me there is not much more profit to advance this script any
further as it converts correctly about 90% of the tests, and the others
I'll fix by hand. Where the effort can be more productive is in the
points above than in the writing of the script itself. If you want to
try the script do
cd octave/
convert_test test/octave.test new_test
cd new_test
octave
fntests
Once the issue of warnings and function handles in test are addressed I
expect that the script will convert all but about 5% of the tests correctly.
Regards
D.
--
David Bateman address@hidden
Motorola Labs - Paris +33 1 69 35 48 04 (Ph)
Parc Les Algorithmes, Commune de St Aubin +33 1 69 35 77 01 (Fax)
91193 Gif-Sur-Yvette FRANCE
The information contained in this communication has been classified as:
[x] General Business Information
[ ] Motorola Internal Use Only
[ ] Motorola Confidential Proprietary
#! /usr/bin/perl
use strict;
use File::Find;
use File::Basename;
use Text::Wrap;
use FileHandle;
use IPC::Open3;
my $in_dir = @ARGV[0];
my $out_dir = @ARGV[1];
# locate all *.exp files in $in_dir
my @exp_files = ();
find(\&exp_files_in_dir, $in_dir);
sub exp_files_in_dir { # {{{1 populates global array @exp_files
return unless -f and /\.(exp)$/; # .exp files
my $path = "$File::Find::dir/$_";
$path =~ s|^[.]/||;
push @exp_files, $path;
} # 1}}}
my $unconverted = 0;
foreach my $fexp ( @exp_files ) {
my $fbase = basename($fexp,('.exp'));
my $fdir = dirname($fexp);
die "Null DejaGNU expect file?? [$fexp]\n" unless $fbase;
my $fout = sprintf("%s/test_%s.m", $out_dir, $fbase);
if (open(OUT,">$fout")) {
print OUT "%% Automatically generated from DejaGNU files\n\n";
if (open(IN,$fexp)) {
print STDOUT "Converting $fexp to $fout\n";
while (<IN>) {
# Look for line of the form /^set test [a-zA-Z].*/ to identify
# the test
next unless /^set test [a-zA-Z].*/;
# Identify output of the test
my $test = <IN>;
$test =~ s/^set prog_output "(.*)";*$/$1/;
$test =~ s/^\\n//;
my $mtest = <IN>;
$mtest =~ s/do_test\s*//;
$mtest =~ s/\s*\n//;
## XXX FIXME XXX Fix the umask-1 test as it messes up path
next if $mtest =~ /umask-1/;
$mtest = sprintf("%s/%s", $fdir, $mtest);
# Is the output an error message
if ($test =~ /^\^usage/ || $test =~ /^\^*\.\.\. [a-zA-Z].*$/ ||
$test =~ /^\^error/ || $test =~ /^parse error/ ||
$test =~ /^\^*warning/) {
$test =~ s/\s*\n//;
print OUT "%% $mtest\n";
if (open(MIN,$mtest)) {
my $line = <MIN>;
$line =~ s/^\s*//;
$line =~ s/\s*\n$//;
while (<MIN>) {
s/^\s*//;
s/\s*\n$//;
$line .= $_;
}
if ($test =~ /^\^usage/ || $test =~ /^\^error/) {
if ($line =~ /;/) {
print OUT "%!error eval('$line');\n\n";
} else {
print OUT "%!error $line;\n\n";
}
} elsif ($test =~ /^\^warning/) {
if ($line =~ /;/) {
print OUT "%!warning eval('$line');\n\n";
} else {
print OUT "%!warning $line;\n\n";
}
} else {
if ($test =~ /parse error/) {
print OUT "%!error <syntax error> eval('$line');\n\n";
} elsif ($line =~ /;/) {
print OUT "%!error <$test> eval('$line');\n\n";
} else {
print OUT "%!error <$test> $line;\n\n";
}
}
close (MIN);
} else {
print STDERR "Could not open test file ($mtest): $!\n";
}
} elsif ($test =~ /^\^*[a-zA-Z0-9]*/) {
my $var = $test;
$var =~ s/^\^*([a-zA-Z0-9]*) = .*\n$/$1/;
my $val = $test;
$val =~ s/^\^*ans = (.*)\n$/$1/;
if ($val =~ /\\n/ || $val =~ /\*/) {
# $val is a matrix to stdout. Manipulate it to something nicer
$val =~ s/^\s*\**[\\n]*\s*\**//;
$val =~ s/[\\n]+\s*\**/;/g;
$val =~ s/\s*\*/,/g;
$val = sprintf("[%s]",$val);
}
print OUT "%% $mtest\n";
if (open(MIN,$mtest)) {
while (<MIN>) {
last unless /^[%#]/;
s/[\%#]*/\%\%/;
print OUT $_;
}
} else {
print STDERR "Could not open test file ($mtest): $!\n";
}
my @funcs = ();
if (open(MIN,$mtest)) {
# Pass through the code looking for functions
while (<MIN>) {
next if /^#/;
next if /^%/;
next unless /function/;
next if /endfunction/;
if (/^\s*function.*=\s*.*\(.*$/) {
s/^\s*function.*=\s*(.*)\(.*$/$1/;
push @funcs, $_;
} else {
s/^\s*function\s*(.*)\(.*$/$1/;
push @funcs, $_;
}
}
close(MIN);
if (@funcs) {
# If we have functions, create a shared block to contain them
print OUT "%!shared";
my $first = 0;
foreach my $func (@funcs) {
$func =~ s/\s*\n//;
if ($first == 0) {
$first = 1;
print OUT " $func";
} else {
print OUT ",$func";
}
}
print OUT "\n";
# Now repass through the file gathering the functions into the
# shared block
if (open(MIN,$mtest)) {
my $infunc = 0;
while (<MIN>) {
last unless /^[%#]/;
}
if (/function/) {
$infunc = 1;
print OUT "%!$_";
}
while (<MIN>) {
if ($infunc == 1) {
# Assume function is closed with endfunction only!!
s/printf/printf_assert/;
if (/endfunction/) {
$infunc = 0;
} else {
print OUT "%!$_";
}
} else {
next unless /function/;
$infunc = 1;
print OUT "%!$_";
}
}
close (MIN);
}
}
} else {
print STDERR "Could not open file ($mtest): $!\n";
}
if (open(MIN,$mtest)) {
while (<MIN>) {
last unless /^[%#]/;
}
my $line = $_;
if ($line =~ /^1;$/) {
$line = <MIN>;
}
$line =~ s/^\s*//;
$line =~ s/\s*\n$//;
if ($line !~ /\;$/) {
while (<MIN>) {
s/^\s*//;
s/\s*\n$//;
$line = sprintf("%s\n%! %s", $line, $_);
last if ($line =~ /\;$/);
}
}
$line =~ s/printf/printf_assert/;
my $s = "";
if ($line =~ /\;$/ || $line =~ /end(for|function|if|switch)?$/) {
$s = " ";
print OUT "%!test\n";
my $infunc = 0;
if ($line =~ /function/) {
$infunc = 1;
} else {
print OUT "%! $line\n";
}
while ($line = <MIN>) {
if ($infunc == 1) {
if ($line =~ /endfunction/) {
$infunc = 0;
}
next;
} else {
if ($line =~ /function/) {
$infunc = 1;
next;
}
}
$line =~ s/^\s*//;
$line =~ s/\s*\n$//;
if ($line !~ /\;$/ && $line !~ /^$/ &
$line !~ /end(for|function|if|switch)?$/) {
while (<MIN>) {
s/^\s*//;
s/\s*\n$//;
if (/function/) {
$infunc = 1;
if ($line !~ /^$/) {
$line = sprintf("%s;", $line);
}
last;
}
$line = sprintf("%s\n%! %s", $line, $_);
last if ($line =~ /\;$/);
}
}
$line =~ s/printf/printf_assert/;
if ($line =~ /\;$/ || $line =~ /^$/ ||
$line =~ /end(for|function|if|switch)?$/) {
foreach my $func (@funcs) {
my $rep = sprintf("^%s\\s*;",$func);
my $wth = sprintf("%s ();",$func);
if ($line =~ /$rep/) {
$line =~ s/$rep/$wth/;
}
}
print OUT "%! $line\n";
} else {
last;
}
}
}
if ($test =~ /^\^*[a-zA-Z][a-zA-Z0-9]* = [^;]*$/) {
if ($line =~ /any\s*\(/ || $line =~ /all\s*\(/ ||
$line =~ /\&\&/ || $line =~ /\|\|/ || $line =~ /\=\=/ ||
$line =~ /\>/ || $line =~ /\</ || $line =~ /^str/ ||
$line =~ /^is/ || $line =~ /^finite/) {
if ($val =~ /^1$/) {
if ($var !~ /^ans$/) {
print OUT "%! $line;\n";
print OUT "%! assert($var);\n\n";
} else {
print OUT "%!${s}assert($line);\n\n";
}
} else {
if ($var !~ /^ans$/) {
print OUT "%! $line;\n";
print OUT "%! assert(!($var));\n\n";
} else {
print OUT "%!${s}assert(!($line));\n\n";
}
}
} else {
if ($var !~ /^ans$/) {
print OUT "%! $line;\n";
print OUT "%! assert($var,$val);\n\n";
} else {
print OUT "%!${s}assert($line,$val);\n\n";
}
}
} else {
$var =~ s/\s*\n$//;
if ($line) {
foreach my $func (@funcs) {
my $rep = sprintf("^%s\\s*\$",$func);
my $wth = sprintf("%s ()",$func);
if ($line =~ /$rep/) {
$line =~ s/$rep/$wth/;
}
}
print OUT "%! $line;\n";
}
print OUT "%! assert(prog_output_assert(\"$var\"));\n\n";
}
close(MIN);
}
} else {
$test =~ s/\n$//;
print STDERR "Can't use $mtest: \"$test\"\n";
$unconverted = $unconverted + 1;
}
}
close (IN);
} else {
print STDERR "Could not open file ($fexp): $!\n";
}
close (OUT);
} else {
print STDERR "Could not open file ($fout): $!\n";
}
}
if ($unconverted != 0) {
print STDERR "Number of unconverted tests: $unconverted\n";
}
# Create the main script for the tests
my $fscript = sprintf ("%s/fntests.m", $out_dir);
if (open(OUT,">$fscript")) {
print OUT "pso = page_screen_output;\n";
print OUT "try\n";
print OUT " page_screen_output =0;\n";
print OUT " fid=fopen('fntests.log','wt');";
print OUT " if (fid < 0)\n";
print OUT " error('could not open fntests.log for writing');\n";
print OUT " endif\n";
print OUT "\n %% Shared functions to simplify the conversion\n";
print OUT " function printf_assert(varargin)\n";
print OUT " global _assert_printf;\n";
print OUT " _assert_printf=cat(2,_assert_printf,sprintf(varargin{:}));\n";
print OUT " endfunction\n";
print OUT " function ret = prog_output_assert(str)\n";
print OUT " global _assert_printf;\n";
print OUT " if (_assert_printf(end) == \"\\n\")\n";
print OUT " ret = strcmp(_assert_printf(1:(end-1)),str);\n";
print OUT " else\n";
print OUT " ret = strcmp(_assert_printf,str);\n";
print OUT " endif\n";
print OUT " _assert_printf = \"\";\n";
print OUT " endfunction\n\n";
print OUT " test('','explain',fid);\n";
print OUT " dp=dn=0;\n";
print OUT " lst=dir('.');\n";
print OUT " for i = 1:length(lst)\n";
print OUT " nm = lst(i).name;\n";
print OUT " if (length(nm) > 5 && strcmp(nm(1:5),'test_'))\n";
print OUT " [p,n] = test(nm(1:(end-2)),'quiet',fid);\n";
print OUT " printf(' %s: passes %d out of %d tests\\n',nm,p,n);\n";
print OUT " dp +=p;\n";
print OUT " dn += n;\n";
print OUT " endif\n";
print OUT " endfor\n";
print OUT " if dp==dn, printf('%-10s ---> success',''); else\n";
print OUT " printf('%-10s ---> passes %d out of %d tests','',dp,dn); end\n";
print OUT " disp('');printf('see fntests.log for details');disp('');\n";
print OUT " fclose(fid);\n";
print OUT "catch\n";
print OUT " page_screen_output = pso;\n";
print OUT "end\n";
close(OUT);
} else {
print STDERR "Could not open script file ($fscript): $!\n";
}
*** ./test/octave.test/args/args-10.m.orig 2005-10-25 17:57:10.000000000
+0200
--- ./test/octave.test/args/args-10.m 2005-10-25 17:59:36.000000000 +0200
***************
*** 1,5 ****
--- 1,9 ----
1;
function [varargout] = f (varargin)
printf ("nargin: %d, nargout: %d\n", nargin, nargout);
+ varargout{1} = 1;
+ varargout{2} = 2;
+ varargout{3} = 3;
+ varargout{4} = 4;
endfunction
[s, t, u, v] = f (1, 2, 3);
*** ./test/octave.test/contin/contin-11.m.orig 2005-10-25 20:26:43.185830308
+0200
--- ./test/octave.test/contin/contin-11.m 2005-10-25 20:30:27.634348046
+0200
***************
*** 1,11 ****
1;
! function f (a,...
! b, ...
! c, ... % comments ok
! x, # continuation characters not required in parens
! y, \# but they should work too.
! z)
! 1
! end
f ()
--- 1,11 ----
1;
! function y = f (a,...
! b, ...
! c, ... % comments ok
! x, # continuation characters not required in parens
! y, \# but they should work too.
! z)
! y = 1;
! endfunction
f ()
*** ./test/octave.test/poly/residue-1.m.orig 2002-05-01 06:19:34.000000000
+0200
--- ./test/octave.test/poly/residue-1.m 2005-10-24 17:07:37.000000000 +0200
***************
*** 2,7 ****
a = [1, -5, 8, -4];
[r, p, k, e] = residue (b, a);
(abs (r - [-2; 7; 3]) < 1e-6
! && abs (p - [2; 2; 1]) < 1e-7)
&& isempty (k)
&& e == [1; 2; 1])
--- 2,7 ----
a = [1, -5, 8, -4];
[r, p, k, e] = residue (b, a);
(abs (r - [-2; 7; 3]) < 1e-6
! && abs (p - [2; 2; 1]) < 1e-7
&& isempty (k)
&& e == [1; 2; 1])
*** ./test/octave.test/system/tic-toc-1.m.orig 2005-10-24 17:46:19.000000000
+0200
--- ./test/octave.test/system/tic-toc-1.m 2005-10-24 17:46:26.000000000
+0200
***************
*** 1,3 ****
tic ();
! sleep (2)
toc () > 0
--- 1,3 ----
tic ();
! sleep (2);
toc () > 0
- Re: Converting DejaGNU tests to Paul's test/assert infrastructure, (continued)