Re: [lmi] Automated GUI testing, revisited

[Greg Chicares]

On 2014-11-14 03:02Z, Vadim Zeitlin wrote:
> On Wed, 12 Nov 2014 12:54:52 +0000 Greg Chicares <address@hidden> wrote:
> 
> GC> This raises the question where these test files should
> GC> reside.
> 
>  I am not sure who is supposed to answer this question. If I had to do it,
> I'd say that this directory probably needs to be configurable as these
> files seem to be located in an environment-dependent directory and I don't
> see how can we accommodate everybody with anything fixed.

Agreed: it needs to be configurable.

> GC> My configuration directory, as seen above, is
> GC>   C:/etc/opt/lmi
> GC> and I don't want any input files in /etc/ because that
> GC> directory is only for things like configuration files.
> GC> If someone else wants to place the default input file
> GC> elsewhere, and put input files there, too, then I don't
> GC> need to object to that practice; but I do object to
> GC> following it myself. Is there any reason why files used
> GC> by 'wx_test' should be put in that particular directory
> GC> instead of the directory where 'wx_test' resides?
> 
>  Assuming there is a reason to keep them there (i.e. I guess that this is
> how things are in the "distribution" environment), the only way I see to
> still make it work for both you and this case is to simply look in the
> current working directory -- and run wx_test from different locations. This
> would be also relatively convenient to me when running MSVC-built
> wx_test.exe.

There is no reason to keep those files there. Therefore, we don't need
the $PWD workaround above--which you proceed to recommend against:

>  But I can't earnestly recommend relying on the current working directory,
> this risks being too fragile and surprising.

Kim and I had considered specifying the path for GUI-test files in
'configurable_settings.?pp'. That's somewhat appealing because it's
a natural extension of that facility, but we dislike it because it
intrudes GUI testing into the production system. You have devised a
solution that we like much better:

> So my, complicated but
> reliable, solution would be to add some --test-data-path command line
> option to specify this directory explicitly.

Yes, would you please provide a patch to do that? I propose
  --gui_test_path
as the name of the argument (and it's okay with me not to offer
any '-g' short alternative), and I propose
  /opt/lmi/gui_test
as the hardcoded default value to be used if no path is explicitly
specified.

Would it also be a good idea to use that same directory for any
output files that are created as a side effect of running the
GUI test? Hmm...maybe we should postpone that question. Some files
are written to particular locations, like
  configurable_settings::print_directory()
and it seems dubious to override that. At least some other files
should be deleted by 'wx_test' upon exit. We may perhaps conclude
that all output files fall into those two categories.

>  Or, unless we discard the idea of having the configuration file
> completely, make this configurable via the same file. This has the
> advantage of being much more flexible and extensible, of course.

I'd much rather avoid using any configuration file. We don't have to
get rid of it right away, but let's take steps (above) toward that
goal, and then we can reevaluate whether it still seems useful.