Re: [lmi] Automated GUI testing, revisited

[Vadim Zeitlin]

On Wed, 12 Nov 2014 01:55:21 +0000 Greg Chicares <address@hidden> wrote:

GC> but at least the "benchmark" test fails later:
GC>   benchmark_census: ERROR (Assertion 'std::fabs(diff_in_percents) < 10' 
failed
GC>   (expected 11000ms, but actually took 456ms, i.e. -95.85%).
GC>   [file /lmi/src/lmi/wx_test_benchmark_census.cpp, line 101] )
GC> 
GC> Vadim--I believe I copied 'wx_test.conf' from an old email that
GC> you had sent. Was "time_disk=11000" an actual measurement?

 Just for completeness: yes, it was. I think the discrepancy in timing is
due to the fact that no printing is actually done in your run because of
the previously (but in an answer to a later message...) discussed change of
the menu item accelerator.

GC> Now I'm wondering whether 'wx_test.conf' is really necessary.

 The current file was added solely for the purpose of storing
machine-specific timing statistics, so if we don't need them any more, it's
indeed not necessary.

 However a local config file might be useful for other things which are
specific to a particular machine or a particular installation, e.g. as I
proposed yesterday, we could use it to specify the machine-specific
directory containing the test files. So I'm not so sure we should hurry to
get rid of it.

GC> I think a similar workflow for 'wx_test' makes the most sense.
GC> We'll run 'wx_test' often (maybe whenever we run a system test),
GC> and save a series of results, e.g. as 'wx_test-20141022T2055Z'
GC> etc. to parallel 'md5sums-20141022T2055Z' above. Comparing a new
GC> file side by side against a saved touchstone[0] lets us see what
GC> changed, e.g.
GC> + MSEC_0.cns  run=434 disk=11000 spreadsheet=710
GC> - MSEC_0.cns  run=441 disk=11022 spreadsheet=708

 The trouble is that this output is the same on all machines while the
execution times are not. I don't know how much variance can there be, but I
strongly suspect it can go well beyond 10%.

GC> If we feel it's necessary, we can read those values and compare
GC> them numerically, naming those that differ by more than a given
GC> tolerance, or stating that none differ materially. That can be
GC> done in an external program,

 Yes, it would have to be, because I am not aware of any diff program that
would be capable of ignoring changes to numbers if they're not too far from
each other. Possibly it could be achieved by writing some post (or pre?)
processor for diff, but it still would only take care of the expected small
(although IME it's not so small neither, I had to bump up the tolerance to
10% after regularly getting false positives with 5%) run-time variations
between the runs on the same machine, it wouldn't help at all with
comparing outputs from different machines.

GC> so we don't need timings like this
GC>   time_run=434
GC>   time_disk=11000
GC>   time_spreadsheet=710
GC> in 'wx_test.conf'. But then the configuration file isn't needed
GC> at all.

 We definitely can implement it like you suggest above and if the tests, or
at least this particular test involving the timings, will only ever run on
a single machine, then I agree it's a good solution. But if we want to be
able to check on any arbitrary machine that the timings haven't changed too
much, e.g. after making some change to the code, then it isn't and I still
believe that some kind of a local configuration file is needed.

 Please let me know what is your decision about this.


GC> Vadim--In the office, we use a shareware tool, but I'm slowly
GC> preparing to move to a free OS; do you have any recommendation
GC> for a free (as in freedom) GUI file-comparison tool? Here are
GC> a few ideas:
GC>   
http://stackoverflow.com/questions/112932/graphical-diff-programs-for-linux

 I'm afraid I'm a wrong person to ask: as for everything else involving
text files, I just use Vim for this. FWIW Vim diff mode is great because
you can easily edit the files being compared, e.g. to get rid of
insignificant differences such as timestamps in the first column, and also
because you can use "diffget" and "diffput" commands to synchronize chunks
between files while viewing the diff. But this is not a viable solution
unless you're already a vi(m) user, of course.

 I do know that both Meld and KDiff3 are two standard suggestions for
people looking for a GUI frontend for "git merge", so I think they must be
pretty good. In addition to them I also know of ECMerge
(http://www.elliecomputing.com/en/Products/merge_overview.asp) but just
because it's written in wxWidgets and I know its author, and while it seems
nice, it's not open source.

 On a somewhat different note, for simple/quick diffs I like to use "git
diff" which is much nicer than plain diff because it uses colours and pager
by default and because it has a very nice --color-words option. Notice that
"git diff" is one of a few of git commands that can be used outside a git
repository, i.e. it works with any files, not just those under git control.
And, as a final piece of advice, if you don't use git at all, you can
achieve something similar with cdiff (https://pypi.python.org/pypi/cdiff/0.9.6).

 Hope this helps at least a little,
VZ