[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
bug#20715: tap-driver.sh 'missing test plan' message contradicts the TAP
From: |
Arthur Schwarz |
Subject: |
bug#20715: tap-driver.sh 'missing test plan' message contradicts the TAP Standard |
Date: |
Tue, 2 Jun 2015 11:08:25 -0700 |
On 2015-06-02 09:40 -0700, Arthur Schwarz wrote:
> [Nick Bowler wrote]
> The plan line has to be mandatory for this feature to work as
> intended (i.e., for the TAP consumer to determine whether a
> producer has run to completion or not). An optional plan would
> be useless.
>
> Sorry to be such a pest.
>
> Your statement that "... determine whether a producer has run to
completion
> or not" is puzzling. The TAP Standard specifically states that if the test
> plan is 1..N and the number of test lines are k < N, then the k+1 .. N
> missing test lines are to be considered as skipped.
Where does it say that? It is not correct to treat missing test results
as skips.
From http://testanything.org/tap-version-13-specification.html
Skipping a few
The following listing plans on running 5 tests. However, our program
decided to not run tests 2 thru 5 at all. To properly report this, the tests
are marked as being skipped.
TAP version 13
1..5
ok 1 - approved operating system
# $^0 is solaris
ok 2 - # SKIP no /sys directory
ok 3 - # SKIP no /sys directory
ok 4 - # SKIP no /sys directory
ok 5 - # SKIP no /sys directory
It is an error for the number of actual test results to differ from the
plan. This is the whole point of the plan: the consumer always knows
how many test results are expected. This enables the consumer to tell
the difference between a producer that crashed before completing all the
tests, and a producer that ran normally.
See above.
> There is one issue not addressed in the TAP Standard and not addressed in
> the Automake Manual. What happens if k > N?
If k > N, then the number of test cases does not match the plan, so
this is an error.
TAP allows k < N (see above). The question of what happens if k > N seems
reasonable. As an extension, the Automake Manual does not state what will
happen if k > N (nor if k < N). This also reflects of the TAP Protocol
statement that the test plan can be at the beginning or end of the test. The
issue is easy to address if the test plan is at the beginning of the
document but will require that an implementation use deferred test line
posting if the test plan is at the end of the document. I think that the TAP
Protocol is poor in this regard.
art
Regards,
--
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)