Re: [lmi] Manual test of automated GUI test

[Greg Chicares]

On 2014-10-18 22:44Z, Greg Chicares wrote:
> I've built lmi with all the latest GUI-testing patches, using a
> wx snapshot as of 2014-10-18T16:40Z.

This time I added the secret backdoor command-line argument.
It was fun to watch. Here's the log in its entirety:

22:45:17: Starting automatic tests:
22:45:17:     about_dialog_version: ok (14ms)
22:45:17: Error: can't open file 'wx_test.conf' (error 2: the system cannot 
find the file specified.)
22:45:17:     benchmark_census: ERROR (Assertion failure: can't read from 
closed file [file ../src/common/ffile.cpp, line 128, in Read()].)
22:45:23:     calculation_summary: ok (5391ms)
22:45:23:     configurable_settings: ERROR (Assertion 
'fs::exists("/etc/opt/lmi/configurable_settings.xml")' failed. [file 
/lmi/src/lmi/wx_test_config_settings.cpp, line 37] )
22:45:23:     create_open_census: ok (143ms)
22:45:24:     create_open_database: ok (1007ms)
22:45:24:     create_open_gpt: ok (352ms)
22:45:25:     create_open_illustration: ok (1080ms)
22:45:26:     create_open_mec: ok (249ms)
22:45:26:     create_open_policy: ok (180ms)
22:45:26:     create_open_rounding: ok (381ms)
22:45:27:     create_open_strata: ok (475ms)
22:45:27:     create_open_text: ok (50ms)
22:45:27:     default_input: ERROR (Assertion '(effective_date) == 
(first_of_month)' failed (expected 2456932 vs observed 2456949). [file 
/lmi/src/lmi/wx_test_default_input.cpp, line 44] )
22:45:27:     default_update: ERROR (Assertion failure: A message box "Unable 
to read file 'C:/etc/opt/lmi/default.ill'. [file 
/lmi/src/lmi/illustration_document.cpp, line 110] " was shown unexpectedly, 
expected wxDialog. [file /opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, 
in ReportFailure()].)
22:45:27:     expiry_dates: ERROR (Assertion '(end) == (end_of_month)' failed 
(expected 2450480 vs observed 2458849). [file 
/lmi/src/lmi/wx_test_expiry_dates.cpp, line 77] )
22:45:27:     extract: ERROR (Assertion failure: Expected wxMessageDialog 
dialog was not shown. [file /opt/lmi/local/include/wx-3.1/wx/testing.h, line 
315, in ReportFailure()].)
22:45:27:     input_sequences: ERROR (Assertion failure: A message box "Sorry, 
could not open this file." was shown unexpectedly. [file 
/opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, in ReportFailure()].)
22:45:27:     input_validation: ERROR (Assertion failure: A message box "Sorry, 
could not open this file." was shown unexpectedly. [file 
/opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, in ReportFailure()].)
22:45:28:     paste_census: ok (1017ms)
22:45:36:     pdf_census: ok (8717ms)
In file ../src/msw/mimetype.cpp at line 272: 'AssocQueryString' failed with 
error 0x80070002 (the system cannot find the file specified.).
22:45:39:     pdf_illustration: ok (2362ms)
22:45:40:     validate_output_illustration: ERROR (Assertion failure: Expected 
wxDialog dialog was not shown. [file 
/opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, in ReportFailure()].)
22:45:40:     validate_output_mec: ERROR (Assertion failure: A message box 
"Sorry, could not open this file." was shown unexpectedly, expected wxDialog. 
[file /opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, in 
ReportFailure()].)
22:45:40: FAILURE: 10 out of 24 tests failed.

and here are some comments on a few particular lines in the log
[I understand that you're still working with Wendy on refining
the specifications, but I'll try to say here why some tests
that may be useful in other contexts don't succeed for me]:

22:45:23:     configurable_settings: ERROR (Assertion 
'fs::exists("/etc/opt/lmi/configurable_settings.xml")' failed. [file 
/lmi/src/lmi/wx_test_config_settings.cpp, line 37] )

There must be a 'configurable_settings.xml' file somewhere, but
/etc/opt/lmi/ = "C:\etc\opt\lmi" is only the first location inspected.

[The other one is not as well defined as I would like:

    std::string filename = "/etc/opt/lmi/" + configuration_filename();
    if(0 != access(filename.c_str(), R_OK))
        {
        // TODO ?? At this point, AddDataDir() refers to the directory
        // where the wx binary resides. A configurable_settings object
        // apparently exists before ProcessCommandLine() is called.
        filename = AddDataDir(configuration_filename());

I habitually install lmi binaries in /opt/lmi/bin and data in /opt/lmi/data
and normally run the illustration system like this:
  /opt/lmi/bin[0]$./lmi_wx_shared --ineffable_password --data_path=/opt/lmi/data
and that seems to pick up the settings file in bin/ rather than data/
because of the problem noted above. Grepping for 'configurable_settings'
in 'workhorse.make' finds apparently regrettable workarounds for this
defect.]

It appears that the 'wrap_fardel' makefile target used in assembling
production distributions uses the copy in /opt/lmi/bin :

        @$(CP) $(bin_dir)/configurable_settings.xml .

I normally don't have a copy in /etc (as noted in this makefile comment):

# If this file:
#   /etc/opt/lmi/configurable_settings.xml
# exists, it overrides files with the same name in other directories.
# Developers may have reason to prefer that it not exist; therefore,
# local copies are provided for as needed.

...for example, because I sometimes run old archived distributions
that may expect a different version of this settings file.

22:45:27:     default_update: ERROR (Assertion failure: A message box "Unable 
to read file 'C:/etc/opt/lmi/default.ill'. [file 
/lmi/src/lmi/illustration_document.cpp, line 110] " was shown unexpectedly, 
expected wxDialog. [file /opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, 
in ReportFailure()].)

For the same reason, I normally don't have this file.

22:45:27:     default_input: ERROR (Assertion '(effective_date) == 
(first_of_month)' failed (expected 2456932 vs observed 2456949). [file 
/lmi/src/lmi/wx_test_default_input.cpp, line 44] )
22:45:27:     expiry_dates: ERROR (Assertion '(end) == (end_of_month)' failed 
(expected 2450480 vs observed 2458849). [file 
/lmi/src/lmi/wx_test_expiry_dates.cpp, line 77] )

I suppose the intention here is to test the range of dates for which
a distribution is valid, and that is a useful test when a distribution
is being assembled. For my own local purposes, I set the expiry date
a couple dozen years into the future:

$cat /opt/lmi/data/expiry
2450449 2458849%
  [looks like [1997-001,2019-365] ]
  [trailing '%' because there's no newline, deliberately]

but the 'expiry' file we distribute is much more restrictive.

Examining the code, I wonder whether there should be an assertion
testing the first value in 'expiry' too, and I'm not entirely sure
whether the default_input test is relevant for me personally...but,
again, specifications are still being discussed in detail, so let's
keep those discussions separate.

The other failing tests may indicate actual problems. Instead of
looking into them now, let's wait to see what results others
report, especially with a production distribution (unless there's
something obvious that you want to talk about right away). Once
the current release is ready to send, I'll commit the GUI testing
changes and the wx upgrade; that'll be within a week or two.