bug#70638: Automake custom-driver interface documentation doesn't match implementation

[Mark Wooding]

This is sort of a follow-up on #22445; alas, nobody involved with that
bug noticed the actual scale of the problem.  I'm sorry to be the bearer
of bad news.

In essence, #22445 observed that the usage message printed by
`test-driver' showed option arguments attached to their options with an
`=' sign, while the program itself required the argument in the next
argument word.  The bug was resolved by fixing the usage message to
match the program's behaviour.

The command-line syntax accepted by `test-driver' is not idiosyncratic.
Rather, it's part of a defined protocol, also implemented by
`tap-driver.sh', and `check.am' and `check2.am', and defined in the
manual at (automake)Command-line arguments for test drivers.  The usage
message for `tap-driver.sh' and the manual agree with the old usage for
`test-driver' that option arguments should be attached to their options
with `='.  However, the implementation of `tap-driver.sh' uses code
almost identical to `test-driver', and requires option arguments to be
passed in separate words:

        [gibson ~/src/ext/automake (master)]git describe
        v1.16i-41-gefb9f7b29
        [gibson ~/src/ext/automake (master)]lib/tap-driver.sh --test-name=splat
        tap-driver.sh: invalid option: '--test-name=splat'
        Usage:
          tap-driver.sh --test-name=NAME --log-file=PATH --trs-file=PATH
        ...

And the code to invoke test drivers, in `check2.am', assumes that this
is acceptable:

        ?GENERIC?       $(am__set_b); \
                $(am__check_pre) %DRIVER% --test-name "$$f" \
                --log-file $$b.log --trs-file $$b.trs \
                $(am__common_driver_flags) %DRIVER_FLAGS% -- %COMPILE% \
                "$$tst" $(AM_TESTS_FD_REDIRECT)

The default setting of `am__common_driver_flags' in `check.am' concurs,
unsurprisingly:

        # Default flags passed to test drivers.
        am__common_driver_flags = \
          --color-tests "$$am__color_tests" \
          --enable-hard-errors "$$am__enable_hard_errors" \
          --expect-failure "$$am__expect_failure"

But all of this is contrary to the manual:

        A custom driver can rely on various command-line options and
        arguments being passed to it automatically by the
        Automake-generated test harness.  It is _mandatory_ that it
        understands all of them (even if the exact interpretation of the
        associated semantics can legitimately change between a test
        driver and another, and even be a no-op in some drivers).

        Here is the list of options:

        `--test-name=NAME'
             The name of the test, with VPATH prefix (if any) removed.
             This can have a suffix and a directory component (as in
             e.g., `sub/foo.test'), and is mostly meant to be used in
             console reports about testsuite advancements and results
             (see Testsuite progress output).
        ...

It seems that the documentation has never matched the implementation
since the former was written in 2011.

I don't see any particularly satisfactory resolution to this.  Usually,
I'd consider the manual as normative, especially when it's written in
such a proscriptive style.  But, as far as I know, Automake itself is
the only software which invokes test drivers to any significant extent,
and its current behaviour is to pass arguments in separate words, it
seems to me that the least bad approach is to correct the documentation
and usage messages to describe it accurately, with a historical note in
the manual explaining the long-standing error in previous versions of
the documentation.

It seems that I'm the only person in nearly 13 years to have noticed
this.  I guess that hardly anyone writes Automake test drivers. :-/
(Yes, I am considering doing this.)

-- [mdw]