Re: [lmi] Manual test of automated GUI test

[Vadim Zeitlin]

On Sun, 19 Oct 2014 00:02:50 +0000 Greg Chicares <address@hidden> wrote:

GC> On 2014-10-18 22:44Z, Greg Chicares wrote:
GC> > I've built lmi with all the latest GUI-testing patches, using a
GC> > wx snapshot as of 2014-10-18T16:40Z.
GC> 
GC> This time I added the secret backdoor command-line argument.
GC> It was fun to watch. Here's the log in its entirety:
GC> 
GC> 22:45:17: Starting automatic tests:
GC> 22:45:17:     about_dialog_version: ok (14ms)
GC> 22:45:17: Error: can't open file 'wx_test.conf' (error 2: the system cannot 
find the file specified.)
GC> 22:45:17:     benchmark_census: ERROR (Assertion failure: can't read from 
closed file [file ../src/common/ffile.cpp, line 128, in Read()].)

 These two errors are redundant but I'm not sure if it's worth improving
this as this file is supposed to exist in the normal case. Do you think it is?

GC> 22:45:23:     calculation_summary: ok (5391ms)
GC> 22:45:23:     configurable_settings: ERROR (Assertion 
'fs::exists("/etc/opt/lmi/configurable_settings.xml")' failed. [file 
/lmi/src/lmi/wx_test_config_settings.cpp, line 37] )

 This is pretty self-explanatory, I think...

GC> 22:45:23:     create_open_census: ok (143ms)
GC> 22:45:24:     create_open_database: ok (1007ms)
GC> 22:45:24:     create_open_gpt: ok (352ms)
GC> 22:45:25:     create_open_illustration: ok (1080ms)
GC> 22:45:26:     create_open_mec: ok (249ms)
GC> 22:45:26:     create_open_policy: ok (180ms)
GC> 22:45:26:     create_open_rounding: ok (381ms)
GC> 22:45:27:     create_open_strata: ok (475ms)
GC> 22:45:27:     create_open_text: ok (50ms)
GC> 22:45:27:     default_input: ERROR (Assertion '(effective_date) == 
(first_of_month)' failed (expected 2456932 vs observed 2456949). [file 
/lmi/src/lmi/wx_test_default_input.cpp, line 44] )
GC> 22:45:27:     default_update: ERROR (Assertion failure: A message box 
"Unable to read file 'C:/etc/opt/lmi/default.ill'. [file 
/lmi/src/lmi/illustration_document.cpp, line 110] " was shown unexpectedly, 
expected wxDialog. [file /opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, 
in ReportFailure()].)
GC> 22:45:27:     expiry_dates: ERROR (Assertion '(end) == (end_of_month)' 
failed (expected 2450480 vs observed 2458849). [file 
/lmi/src/lmi/wx_test_expiry_dates.cpp, line 77] )

 As are all those.

GC> 22:45:27:     extract: ERROR (Assertion failure: Expected wxMessageDialog 
dialog was not shown. [file /opt/lmi/local/include/wx-3.1/wx/testing.h, line 
315, in ReportFailure()].)
GC> 22:45:27:     input_sequences: ERROR (Assertion failure: A message box 
"Sorry, could not open this file." was shown unexpectedly. [file 
/opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, in ReportFailure()].)
GC> 22:45:27:     input_validation: ERROR (Assertion failure: A message box 
"Sorry, could not open this file." was shown unexpectedly. [file 
/opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, in ReportFailure()].)

 I'm not totally sure about those, but I think they're due to the previous
failures as well because they all use configurable_settings.

GC> 22:45:28:     paste_census: ok (1017ms)
GC> 22:45:36:     pdf_census: ok (8717ms)
GC> In file ../src/msw/mimetype.cpp at line 272: 'AssocQueryString' failed with 
error 0x80070002 (the system cannot find the file specified.).

 This probably indicates that you don't have a PDF viewer (i.e. a program
registered for opening files with .pdf extension) installed?

GC> 22:45:39:     pdf_illustration: ok (2362ms)
GC> 22:45:40:     validate_output_illustration: ERROR (Assertion failure: 
Expected wxDialog dialog was not shown. [file 
/opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, in ReportFailure()].)
GC> 22:45:40:     validate_output_mec: ERROR (Assertion failure: A message box 
"Sorry, could not open this file." was shown unexpectedly, expected wxDialog. 
[file /opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, in 
ReportFailure()].)

 Those use configurable_settings as well.


GC> 22:45:23:     configurable_settings: ERROR (Assertion 
'fs::exists("/etc/opt/lmi/configurable_settings.xml")' failed. [file 
/lmi/src/lmi/wx_test_config_settings.cpp, line 37] )
GC> 
GC> There must be a 'configurable_settings.xml' file somewhere, but
GC> /etc/opt/lmi/ = "C:\etc\opt\lmi" is only the first location inspected.

 Yes, I'm aware about this, but I thought the point of this test was to
check for its existence at this particular location. Especially as it was
the very first item of the test specification, quoting Wendy's email from
2013-11-13T14:51:30Z:

>       [Specifications begin]
>
>       1. Inspect 'configurable_settings.xml' for the following:
>
>         A. The file exists in '/etc/opt/lmi'

GC> I normally don't have a copy in /etc (as noted in this makefile comment):

 To be honest, neither did I, before I started working on this. But I had
to create a copy there just to allow this test to pass locally. I could,
and would probably be glad to, to relax this test and just check that the
configurable settings file is available somewhere where it can be found.
But, to be honest, I don't really see the point of this test then as the
absence of configurable_settings.xml would result in a fatal error anyhow,
whether we test for it explicitly or not.

 To summarize, I'm not sure what am I supposed to do here? Modify the test?
Remove it entirely? Something else?

 I'd just like to remind you that you can exclude this test if you don't
want to run it by using "-t -configurable_settings" command line option.


GC> 22:45:27:     default_input: ERROR (Assertion '(effective_date) == 
(first_of_month)' failed (expected 2456932 vs observed 2456949). [file 
/lmi/src/lmi/wx_test_default_input.cpp, line 44] )
GC> 22:45:27:     expiry_dates: ERROR (Assertion '(end) == (end_of_month)' 
failed (expected 2450480 vs observed 2458849). [file 
/lmi/src/lmi/wx_test_expiry_dates.cpp, line 77] )
GC> 
GC> I suppose the intention here is to test the range of dates for which
GC> a distribution is valid, and that is a useful test when a distribution
GC> is being assembled. For my own local purposes, I set the expiry date
GC> a couple dozen years into the future:

 Again, so do (or at least did) I, but I thought the intention of the test
was to check that the dates were updated correctly for the distribution.
Here I'm almost certain that the checks should not be relaxed and IMHO
using "-t -expiry_dates" would be a good enough workaround, wouldn't it?

GC> Examining the code, I wonder whether there should be an assertion
GC> testing the first value in 'expiry' too, and I'm not entirely sure
GC> whether the default_input test is relevant for me personally...but,
GC> again, specifications are still being discussed in detail, so let's
GC> keep those discussions separate.

 I don't think this issue was mentioned in the other thread, so perhaps it
would be better to either follow up there with at least a note about this
or just finish discussing it here, otherwise I'm afraid it will just get
lost.

GC> The other failing tests may indicate actual problems. Instead of
GC> looking into them now, let's wait to see what results others
GC> report, especially with a production distribution (unless there's
GC> something obvious that you want to talk about right away).

 No, I still think/hope that they are all just due to the missing/misplaced
files. At least the tests continue to stubbornly pass for me here...

 Thanks for your testing!
VZ