[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Autotest: loops of tests
From: |
Ralf Wildenhues |
Subject: |
Re: Autotest: loops of tests |
Date: |
Mon, 15 Aug 2005 18:26:29 +0200 |
User-agent: |
Mutt/1.4.1i |
This is continuing an old old thread from the autoconf list:
http://lists.gnu.org/archive/html/autoconf/2005-03/msg00117.html
I moved it to autoconf-patches now.
* Noah Misch wrote on Thu, Mar 24, 2005 at 03:47:45AM CET:
> On Tue, Mar 22, 2005 at 10:05:04AM +0100, Ralf Wildenhues wrote:
> >
> > I'm not too happy with the replication they cause in `testsuite' -- one
> > of my ideas was to test on the order of 2^8 link flag combinations
> > eventually, and I would like to keep the blow-up as small as possible.
>
> I understand. Current Autotest does not provide a clean way to avoid that
> bloat. That is a problem worth addressing; the damage to non-Autoconf users
> of
> Autotest is particularly bad since you distribute `testsuite' itself.
I hacked something up. Rather ugly for Autoconf standards, but it
does exactly what I think I need: it avoids all bloat. If you can
make something nice out of it, the better. I would really like to
see functionality comparable to this in autotest.
Currently, AT_CHECK(cmd, ignore, , , fail_commands)
is somewhat superfluous, as fail_commands will never get executed
because we ignore the return value. The patch below changes that.
I regard this change in semantics as ok because of the nonsensical
behavior this combination had before, IMVHO.
It passes the test suite, and even makes `tests/testsuite' 33 bytes
smaller, by using a bit more m4 compile time. :)
Do you like it? It allows me to do this (as an example, a failure
is generated when $i is nonempty):
### snip loop.at ###
dnl autom4te -l autotest loop.at > loop; chmod +x loop; ./loop
m4_define([AT_PACKAGE_STRING],[loop])
m4_define([AT_PACKAGE_BUGREPORT],[devnull])
AT_INIT
AT_SETUP([loop])
AT_CAPTURE_FILE([failures])
for i in '' one '' two '' three; do
dnl Stylistic issue: we overwrite `failures' here to avoid quadratic
dnl growth of `testsuite.log' in the number of failures.
dnl This is because many tests might fail for the same reason.
AT_CHECK([test -z "$i"], ignore,ignore,ignore, [echo "$i" > failures])
done
dnl next command generates the real failure:
AT_CHECK([test -s failures],1,ignore,ignore)
AT_CLEANUP
### snip ###
The design decision whether or not to capture files in the case in
question is not clear to me -- I'd probably prefer yes, like I've
done now. That way allows more flexibility for the commands.
Cheers,
Ralf
* lib/autotest/general.m4 (AT_CHECK, _AT_CHECK):
If RUN-IF-FAIL is given, execute it upon nonzero status, even
if STATUS is set to `ignore'. Capture files then, too.
* doc/autoconf.texi (Writing testsuite.at): Updated.
Index: doc/autoconf.texi
===================================================================
RCS file: /cvsroot/autoconf/autoconf/doc/autoconf.texi,v
retrieving revision 1.925
diff -u -r1.925 autoconf.texi
--- doc/autoconf.texi 6 Jul 2005 21:39:31 -0000 1.925
+++ doc/autoconf.texi 14 Aug 2005 14:06:36 -0000
@@ -16168,8 +16180,10 @@
The @var{commands} @emph{must not} redirect the standard output, nor the
standard error.
-If @var{status}, or @var{stdout}, or @var{stderr} is @samp{ignore}, then
-the corresponding value is not checked.
+If @var{stdout} or @var{stderr} is @samp{ignore}, then the corresponding value
+is not checked. If @var{status} is @samp{ignore}, a non-zero return value
+does not cause the test group to fail, but nevertheless causes
address@hidden to be executed, and files to be captured.
The special value @samp{expout} for @var{stdout} means the expected
output of the @var{commands} is the content of the file @file{expout}.
Index: lib/autotest/general.m4
===================================================================
RCS file: /cvsroot/autoconf/autoconf/lib/autotest/general.m4,v
retrieving revision 1.189
diff -u -r1.189 general.m4
--- lib/autotest/general.m4 27 Jul 2005 14:20:32 -0000 1.189
+++ lib/autotest/general.m4 14 Aug 2005 14:06:36 -0000
@@ -1207,7 +1207,9 @@
# STDERR contents. Shell metacharacters in STDOUT and STDERR are
# _not_ processed by the shell, but are treated as string literals.
#
-# STATUS, STDOUT, and STDERR are not checked if equal to `ignore'.
+# STDOUT and STDERR are not checked if equal to `ignore'.
+# If STATUS is equal to `ignore', non-zero return value does not cause
+# the test group to fail, but does RUN-IF-FAIL to be executed.
#
# If STDOUT is `expout', then stdout is compared to the content of the file
# `expout'. Likewise for STDERR and `experr'.
@@ -1437,7 +1439,7 @@
[ 77) echo 77 > "$at_status_file"; exit 77;;
])dnl
m4_if([$2], [ignore],
- [ *);;],
+ [ 0);; *) at_failed=:;;],
[ m4_default([$2], [0])) ;;
*) echo "$at_srcdir/AT_LINE: exit code was $at_status, expected
m4_default([$2], [0])"
at_failed=:;;])
@@ -1446,7 +1448,8 @@
m4_ifdef([AT_capture_files],
[for file in AT_capture_files
do echo "$file:"; sed 's/^/> /' "$file"; done])
- echo 1 > "$at_status_file"
- exit 1], [$6])
+ m4_if([$2], [ignore], [at_failed=false],
+ [echo 1 > "$at_status_file"
+ exit 1])], [$6])
$at_traceon
])# _AT_CHECK
- Re: Autotest: loops of tests,
Ralf Wildenhues <=