Re: [lmi] Patches for stderr output (was: Automated GUI testing, revisited)

[Vadim Zeitlin]

On Sun, 09 Nov 2014 03:34:28 +0000 Greg Chicares <address@hidden> wrote:

GC> On 2014-11-09 01:21Z, Greg Chicares wrote:
GC> [...]
GC> > 00:36:58: Starting automatic tests:
GC> > 00:36:58:     about_dialog_version: ok (14ms)
GC> > 00:36:58:     benchmark_census: ERROR (Assertion failure: A message box 
"Sorry, could not open this file." was shown unexpectedly. [file 
/opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, in ReportFailure()].)
GC> > 00:37:02:     calculation_summary: ok (4671ms)
GC> 
GC> That output, BTW, was from the log window. Are the timestamps just an
GC> artifact of the use of a wx log class? They're somewhat interesting
GC> at the moment, while 'wx_test' is being developed, but ultimately I
GC> suspect we'll want to forgo them, because they make it less simple
GC> to compare output from one run to the next.
GC> 
GC> BTW, we were talking off the mailing list about doing away with the
GC> log window and just writing test output to either
GC>  (1) a flat text file, or
GC>  (2) stdout or stderr as the case may be.
GC> Do you have a preference between those? I'm leaning toward (2).

 Here are preliminary patches implementing (2). The first patch redirects
the output to stderr and disables the time stamps. It also changes the test
behaviour: it now closes the main window and exits the application after
finishing running all the tests, making the test fully non-interactive.

 The second patch changes the output format to be more greppable, notably
it splits the timing information onto separate lines, so that it can be
easily filtered out. Example of the output with this patch:

        % ./wx_test.exe --data_path=/opt/lmi/data -t pdf\*
        [TEST] NOTE: starting the test suite
        [TEST] pdf_census: started
        [TEST] time=5476ms (for pdf_census)
        [TEST] pdf_census: ok
        [TEST] pdf_illustration: started
        [TEST] time=1656ms (for pdf_illustration)
        [TEST] pdf_illustration: ok
        [TEST] time=7133ms (for all tests)
        [TEST] SUCCESS: 2 tests successfully completed.

or, without the timings:

        % ./wx_test.exe --data_path=/opt/lmi/data -t pdf\* |& grep -v '^\[TEST] 
time='
        [TEST] NOTE: starting the test suite
        [TEST] pdf_census: started
        [TEST] pdf_census: ok
        [TEST] pdf_illustration: started
        [TEST] pdf_illustration: ok
        [TEST] SUCCESS: 2 tests successfully completed.

Notice that the "started" lines seem to be redundant, but they can be
useful if there is any output produced during the test, and I decided that
it was better to produce more output now that it's not meant to be examined
by a human but mostly just diff'd with the output from the previous runs.
But it can be easily changed, of course, if you think it's superfluous.


 Finally, the third patch, not really related to the first two, but which I
made while testing them, provides more information for the benchmark tests
failures.


 Please let me know what do you think about these patches, thanks in
advance!
VZ

0001-Send-GUI-test-suite-output-to-stderr-instead-of-show.patch
Description: Text document

0002-Make-GUI-test-suite-output-more-suitable-for-automat.patch
Description: Text document

0003-Show-more-information-when-benchmarking-unit-test-fa.patch
Description: Text document