Fail verbosely when test fails in process_test

[Tom de Vries]

[ was: Re: Check if test has reportable output in process_test ]

On 06/03/2017 05:59 PM, Tom de Vries wrote:
> Hi,
>
> I found a problem in the dejagnu testsuite, with process_test.
>
> If there is a problem running a .test file, then process_test is silent 
> about it.
>
> This minimal demonstrator patch adds basic reporting if there is a 
> problem, by checking that at least a single clause in the expect command 
> triggered. Using this patch we're able to demonstrate that there are 
> problems in target.test and remote.test.

This patch tries to be verbose about what exactly goes wrong when 
running a .test file fails.

[ As we now can see, what currently goes wrong in both target.test and 
remote.test, is that load_lib is called, but not available. ]

OK?

Thanks,
- Tom

Fail verbosely when test fails in process_test

Result:
--------------------------------------------------------
Running ../src/testsuite/runtest.all/libs.exp ...
ERROR: ../src/testsuite/runtest.all/target.test failed
../src/testsuite/runtest.all/target.test stdout: ...
ERROR: (DejaGnu) proc "load_lib telnet.exp" does not exist.

                === foobar Summary ===

... done
../src/testsuite/runtest.all/target.test stderr: ...
The error code is TCL LOOKUP COMMAND load_lib
The info on the error is:
invalid command name "load_lib"
    while executing
"::tcl_unknown load_lib telnet.exp"
    ("uplevel" body line 1)
    invoked from within
"uplevel 1 ::tcl_unknown $args"
can't read "test_counts(PASS,total)": no such variable
    while executing
"set val $test_counts($x,$which)"
    (procedure "log_summary" line 59)
    invoked from within
"log_summary total"
    (procedure "log_and_exit" line 5)
    invoked from within
"log_and_exit"
    (procedure "::unknown" line 15)
    invoked from within
"load_lib "telnet.exp""
    (file "../src/testsuite/../lib/remote.exp" line 23)
    invoked from within
"source $file"
    invoked from within
"if [ file exists $file] {
    source $file
} else {
    puts "ERROR: $file doesn't exist"
}"
    (file "../src/testsuite/runtest.all/target.test" line 31)

... done
ERROR: ../src/testsuite/runtest.all/remote.test failed
../src/testsuite/runtest.all/remote.test stdout: ...

... done
../src/testsuite/runtest.all/remote.test stderr: ...
invalid command name "load_lib"
    while executing
"load_lib "telnet.exp""
    (file "../src/testsuite/../lib/remote.exp" line 23)
    invoked from within
"source $file"
    invoked from within
"if [ file exists $file] {
    source $file
} else {
    puts "ERROR: $file doesn't exist"
}"
    (file "../src/testsuite/runtest.all/remote.test" line 17)

... done
--------------------------------------------------------

2017-06-03  Tom de Vries  <address@hidden>

        * testsuite/runtest.all/libs.exp (process_test): Dump entire output when
        there's no reportable output.

---
 testsuite/runtest.all/libs.exp | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/testsuite/runtest.all/libs.exp b/testsuite/runtest.all/libs.exp
index 7520e12..b716007 100644
--- a/testsuite/runtest.all/libs.exp
+++ b/testsuite/runtest.all/libs.exp
@@ -31,7 +31,31 @@ proc process_test { test } {
 
     if [file exists $test] {
        verbose "Processing test $test" 2
-       spawn -open  [open "|$EXPECT $test $srcdir $subdir [pwd]" r]
+
+       set res [catch {
+           exec -ignorestderr \
+               $EXPECT $test $srcdir $subdir [pwd] \
+               > OUTPUT 2>ERROR
+       }]
+       if { $res } {
+           perror "$test failed" 0
+
+           set fp [open "OUTPUT" r]
+           set output [read $fp]
+           close $fp
+           puts "$test stdout: ..."
+           puts "$output"
+           puts "... done"
+
+           set fp [open "ERROR" r]
+           set error [read $fp]
+           close $fp
+           puts "$test stderr: ..."
+           puts "$error"
+           puts "... done"
+       }
+
+       spawn -open [open "|cat OUTPUT" r]
        expect {
            "No such file or directory" {
                perror "$test wouldn't run" 0
@@ -68,6 +92,7 @@ proc process_test { test } {
                verbose "All Done" 3
            }
        }
+       file delete OUTPUT ERROR
     } else {
        perror "$test doesn't exist" 0
     }