[bug#68855] wording for user on test suite failure?

[Karl Berry]

I was thinking about making the message when the test suite failure more
explicit about what to do, as in below. If any comments or criticisms,
let me know.

I was actually inspired to do this not so much to tweak these words, but
because I was thinking of adding yet more instructions about how to help
debug the subsecond mtime stuff (for automake only, of course), i.e.,
setting am_cv_sleep_fractional_seconds=false. But with the new
possibility that checking make's behavior will be enough to obviate
those random failures, that doesn't seem necessary/desirable now.

But having gone to all the trouble of figuring out how to change the
text and which tests need to be changed to keep them in sync, I guess I
still hope people think it's worth it, or at least ok, to make this
change. The output would look like this:

============================================================================
See tests/test-suite.log for debugging.
Some test(s) failed.  Please report this to <address>,
together with the test-suite.log file (gzipped) and your system
information.  Thanks.
============================================================================

Hopefully I did not break the colorization, but I don't have a good way
to test that. --thanks, karl.

-----------------------------------------------------------------------------
test: make bug-reporting message on test failure more explicit.

* NEWS: mention this.
* lib/am/check.am ($(TEST_SUITE_LOG)) <please_report>: new fn,
new wording.
* t/ax/test-defs.in (failure_footer_text): new wording.
(failure_footer_text_colorized): add red.
* t/ax/tap-summary-aux.sh (failure_footer): use failure_footer_text*.
* t/tap-doc2.sh: change grep.
* t/testsuite-summary-color.sh: likewise.
* t/testsuite-summary-count-many.sh: likewise.
* t/testsuite-summary-count.sh: likewise.
* t/testsuite-summary-reference-log.sh: likewise.
diff --git a/NEWS b/NEWS
index 153cb7733..964d82e8c 100644
--- a/NEWS
+++ b/NEWS
@@ -44,6 +44,10 @@ New in 1.17:
   - The missing script also supports autoreconf, autogen, and perl.
     (https://lists.gnu.org/archive/html/automake-patches/2015-08/msg00000.html)

+  - test-suite.log now contains basic system information, and the
+    console message about bug reporting on failure has a bit more detail.
+    (bug#68746)
+
 * Bugs fixed

   - Generated file timestamp checks handle filesystems with sub-second
diff --git a/lib/am/check.am b/lib/am/check.am
index d210172a6..b6fa3aaad 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -246,6 +246,7 @@ am--force-recheck:
 ## Exists only to be overridden.  See bug#11745.
 AM_TESTSUITE_SUMMARY_HEADER = ' for $(PACKAGE_STRING)'

+## emacs-page
 $(TEST_SUITE_LOG): $(TEST_LOGS)
        @$(am__set_TESTS_bases); \
 ## Helper shell function, tells whether a path refers to an existing,
@@ -360,7 +361,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
          result_count $$1 "ERROR:" $$error "$$mgn"; \
        }; \
 ##
-## Write some basic system info in the log. 
+## Function to write some basic system info in the log. 
        output_system_information () \
        { \
 ## Omit the hostname for privacy.  In practice it's a single word?
@@ -372,6 +373,16 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
            cat /etc/issue;                        \
          fi;                                      \
        }; \
+##
+## Function to output the "please report bugs" message.
+       please_report () \
+       { \
+echo "Some test(s) failed.  Please report this to $(PACKAGE_BUGREPORT),";    \
+echo "together with the test-suite.log file (gzipped) and your system";      \
+echo "information.  Thanks.";                                                \
+       }; \
+##
+## emacs-page
 ## Write "global" testsuite log.
        {                                                               \
          echo "$(PACKAGE_STRING): $(subdir)/$(TEST_SUITE_LOG)" |       \
@@ -379,8 +390,10 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
          create_testsuite_report --no-color;                           \
          output_system_information;                                    \
          echo;                                                         \
-## What is this line for? It seems to be intended for machine parsing,
-## but I can't find any other reference to it. --karl, 27jan24.
+## What is this line for? It's a Sphinx reStructuredText directive that
+## would insert a table of contents that goes two levels deep on headings.
+## But why is it in test-suite.log? No other references to it are evident.
+## Please inform automake@gnu.org.
          echo ".. contents:: :depth: 2";                               \
          echo;                                                         \
          for b in $$bases; do echo $$b; done                           \
@@ -403,9 +416,9 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
        create_testsuite_report --maybe-color;                          \
        echo "$$col$$br$$std";                                          \
        if $$success; then :; else                                      \
-         echo "$${col}See $(subdir)/$(TEST_SUITE_LOG)$${std}";         \
+         echo "$${col}See $(subdir)/$(TEST_SUITE_LOG) for debugging.$${std}";\
          if test -n "$(PACKAGE_BUGREPORT)"; then                       \
-           echo "$${col}Please report to $(PACKAGE_BUGREPORT)$${std}"; \
+           please_report | sed -e "s/^/$${col}/" -e s/'$$'/"$${std}"/; \
          fi;                                                           \
          echo "$$col$$br$$std";                                        \
        fi;                                                             \
diff --git a/t/ax/tap-summary-aux.sh b/t/ax/tap-summary-aux.sh
index d1e2888c3..a709f1ef3 100644
--- a/t/ax/tap-summary-aux.sh
+++ b/t/ax/tap-summary-aux.sh
@@ -87,8 +87,17 @@ if test $use_colors = yes; then
   mgn="$esc[0;35m"
   brg="$esc[1m"
   std="$esc[m"
+  failure_footer="\
+${red}${br}${std}
+$failure_footer_text_colorized
+${red}${br}${std}"
+
 else
   red= grn= lgn= blu= mgn= brg= std=
+  failure_footer="\
+${red}${br}${std}
+$failure_footer_text
+${red}${br}${std}"
 fi

   success_header="\
@@ -101,12 +110,6 @@ ${grn}${br}${std}"
   failure_header="\
 ${red}${br}${std}
 ${red}Testsuite summary for GNU AutoTAP 5.12${std}
-${red}${br}${std}"
-
-  failure_footer="\
-${red}${br}${std}
-${red}See ./test-suite.log${std}
-${red}Please report to bug-automake@gnu.org${std}
 ${red}${br}${std}"

 $ACLOCAL
diff --git a/t/ax/test-defs.in b/t/ax/test-defs.in
index 8ea842355..c1ead418e 100644
--- a/t/ax/test-defs.in
+++ b/t/ax/test-defs.in
@@ -215,6 +215,20 @@ sleep="sleep $MTIME_RESOLUTION"
 # in UTC; see <https://www.gnu.org/gnu/initial-announcement.html>.
 old_timestamp=198309271735.59

+# We need this in multiple tests and the wording is subject to change.
+# Don't include the ${br} lines since br is not defined at this point.
+#
+failure_footer_text="\
+See ./test-suite.log for debugging.
+Some test(s) failed.  Please report this to bug-automake@gnu.org,
+together with the test-suite.log file (gzipped) and your system
+information.  Thanks."
+#
+# And with color.
+red=''
+std='';
+failure_footer_text_colorized="`echo \"$failure_footer_text\" | sed -e 
"s/^/${red}/" -e s/'$'/"${std}"/`"
+
 # Make our wrapper script (or installed scripts, if running under
 # "installcheck") accessible by default.  And avoid uselessly
 # extending $PATH multiple times if this file is sourced multiple times.
diff --git a/t/tap-doc2.sh b/t/tap-doc2.sh
index fdca66ad8..54c69b631 100644
--- a/t/tap-doc2.sh
+++ b/t/tap-doc2.sh
@@ -102,7 +102,7 @@ cat exp
 cat got
 diff exp got

-grep '^Please report to bug-automake@gnu\.org$' stdout
+grep 'Please report this to bug-automake@gnu\.org' stdout

 run_make -O check \
   TESTS='foo.test baz.test' \
diff --git a/t/testsuite-summary-color.sh b/t/testsuite-summary-color.sh
index 7b6254f61..ffbe5c158 100644
--- a/t/testsuite-summary-color.sh
+++ b/t/testsuite-summary-color.sh
@@ -46,8 +46,7 @@ ${red}${br}${std}"

 failure_footer="\
 ${red}${br}${std}
-${red}See ./test-suite.log${std}
-${red}Please report to bug-automake@gnu.org${std}
+$failure_footer_text_colorized
 ${red}${br}${std}"

 do_check '' <<END
diff --git a/t/testsuite-summary-count-many.sh 
b/t/testsuite-summary-count-many.sh
index 2560ef82c..3cb5a500c 100644
--- a/t/testsuite-summary-count-many.sh
+++ b/t/testsuite-summary-count-many.sh
@@ -34,7 +34,7 @@ ${br}"

 footer="\
 ${br}
-See ./test-suite.log
+See ./test-suite.log for debugging.
 ${br}"

 echo AC_OUTPUT >> configure.ac
diff --git a/t/testsuite-summary-count.sh b/t/testsuite-summary-count.sh
index c8f15389d..9cd2fa58e 100644
--- a/t/testsuite-summary-count.sh
+++ b/t/testsuite-summary-count.sh
@@ -32,8 +32,7 @@ success_footer=${br}

 failure_footer="\
 ${br}
-See ./test-suite.log
-Please report to bug-automake@gnu.org
+$failure_footer_text
 ${br}"

 # Corner cases.
diff --git a/t/testsuite-summary-reference-log.sh 
b/t/testsuite-summary-reference-log.sh
index 64cc4eb6c..f27e6fc39 100644
--- a/t/testsuite-summary-reference-log.sh
+++ b/t/testsuite-summary-reference-log.sh
@@ -46,11 +46,11 @@ cd build
 ../configure

 run_make -O -e FAIL check
-grep '^See \./my_test_suite\.log$' stdout
+grep '^See \./my_test_suite\.log for debugging\.$' stdout

 mkdir bar
 run_make -O -e FAIL TEST_SUITE_LOG=bar/bar.log check
-grep '^See \./bar/bar\.log$' stdout
+grep '^See \./bar/bar\.log for debugging\.$' stdout

 cd ..

@@ -70,13 +70,13 @@ $AUTOMAKE

 ./configure
 run_make -O -e FAIL check
-grep '^See sub/test-suite\.log$' stdout
+grep '^See sub/test-suite\.log for debugging\.$' stdout
 cd sub
 run_make -O -e FAIL check
-grep '^See sub/test-suite\.log$' stdout
+grep '^See sub/test-suite\.log for debugging\.$' stdout
 cd ..

 run_make -O -e FAIL TEST_SUITE_LOG=foo.log check
-grep '^See sub/foo\.log$' stdout
+grep '^See sub/foo\.log for debugging\.$' stdout

 :

compile finished at Wed Jan 31 10:56:25 2024