Re: more Autotest thoughts

[Ralf Wildenhues]

Hi Noah,

* Noah Misch wrote on Tue, Apr 26, 2005 at 03:16:23AM CEST:
> On Fri, Apr 22, 2005 at 09:16:48AM +0200, Ralf Wildenhues wrote:
> > It would be nice if autotest put each test (or test group) into a
> > separate file, and then source or exec it in a subshell.  This is
> 
> I once considered doing this as part of a scheme to run several tests in
> parallel.

Good.  The parallel thingy might also be a nice idea.

[ splitting up ]
> > Is there an m4 limitation that prevents a fix for this?
> 
> M4 has no support for creating multiple output files, but it would be easy to
> mark file boundaries in output via magic separators and do the splitting in
> autom4te or another, dedicated script.

OK.

> > Furthermore, the AT_CHECK docs are not fully clear to me: If COMMANDS
> > fail, but RUN_IF_FAIL is given, then AT_CHECK still only acts upon the
> > COMMANDS exit status.  In other words: it is not possible to change
> > the exit status of the test group from within RUN_IF_FAIL (thinking
> > this may possible may be a flaw of mine -- or suboptimal docs).  Also,
> 
> The existing behavior makes sense to me.  How could we improve the
> semantics or documentation?

Improve the documentation, the semantics are OK AFAICS.  After reading
the comments above the implementation of the respective macros, they
make much more sense to me.

> > the interaction of COMMANDS exit status and STDOUT and STDERR
> > expectations are IMHO not described accurately.  Namely, if COMMANDS
> > exit with 77, but also produce unwanted output, the test fails.  This
> > is a valid, but unfortunate combination for me, as it means I might
> > have to discard or ignore output.
> 
> Can you post a sample test group that this upsets?

Oh, that description of mine was a followup error for my failure to
understand what STDOUT=stdout really meant.  I'm sorry.  I'll go and fix
the corresponding bug in my libtool test now.

Maybe some background is of help:

When cross-compiling, there are several possible situations:
- You are on a multi-ABI system (like linux/x86_64), and $host binaries
  can actually be run on $build without any problems.
- $host binaries cannot be run at all, so produce outright failure in $?
  when executed.
- you have a wrapper for execution, but it's not trivial, it might
  output something or even require user interaction.  Examples are
  binfmt_support in combination with wine.  Wine might output something,
  so I can't match STDOUT or STDERR directly.

Now, in order to still get reasonable results while trying to minimize
both false positives and false negatives, my current, somewhat naive approach
is something like

# LT_AT_EXEC_CHECK(EXECUTABLE, [STATUS = 0], [STDOUT], [STDERR])
# --------------------------------------------------------------
m4_define([LT_AT_EXEC_CHECK],
[AT_CHECK([if $1; then :;
           elif test "X$host" != "X$build" &&
                { test -x "$1" || test -x "$1"$EXEEXT; }
           then (exit 77); fi],[$2],[$3],[$4])
])

for testing built executables.  The SKIP is a compromise here.  What
I would like in a series of AT_CHECKS and LT_AT_EXEC_CHECKS, that the
test group would keep going on, but I still have an indication that not
all went prefectly.  Plus I'd like to be able to match output exactly in
the non-crosscompile case.  The latter requirement is not so important,
as I could change that interface to require subsequent grepping of
`stdout' and/or `stderr', which would be more resistent to wrappers.

Does this make any sense to you?

[ The two tests should probably be merged into one, the necessity for
both is probably a bug in the Libtool testsuite, but it's not my main
concern ATM. ]

> > AT_DATA doc should include the fact that no shell expansion will take
> > place on the second argument, but on the first.  I would like a macro
> 
> Yes, all of these facts about AT_DATA should be documented.

OK.

> > similar to AT_DATA which shell-expands its second argument.  But
> > frankly, I really don't see the advantage of either command over just
> > using
> > 
> > | for i in 1 2; do
> > |   cat >expanded_file$i <<EOF
> 
> Use `cat' and friends.  I see no need for a special macro that populates a 
> file.

That's fine.  I merely wonder what AT_DATA was invented for then.

> > The apparent beauty of m4 macros seems IMVHO to be traded for increased
> > output size more often than is technically necessary.
> 
> Which macros?

AT_DATA, for above reason.

Regards, and thank you for your comments,
Ralf