[lmi] Automated GUI testing

[Greg Chicares]

Vadim--would you please take a look at this and tell me what you
think?

We've discussed automated GUI testing before, and now that I've
put together a prototype, I think it can be much simpler for lmi
than I had ever suspected. What's really crucial is to test the
wxNotebook-based input, which handles control enablement etc.
automatically as recently documented here:
  
http://savannah.nongnu.org/cgi-bin/viewcvs/*checkout*/lmi/skeleton/xml_notebook.hpp?rev=1.9&content-type=text/plain
in a large '///' (doxygen) comment block. The testing prototype is:
  
http://savannah.nongnu.org/cgi-bin/viewcvs/*checkout*/lmi/skeleton/mvc_test.cpp?rev=1.1&content-type=text/plain
and I'd like to ask you some questions about mvc_test::Test()
in that file.

First, the dialog has to be activated:

    // Use Show() instead of ShowModal() so that the dialog can be
    // manipulated while it's shown.

A modal dialog would suspend the application's message loop, but
here we need to keep control in the test routine, which I've set
up as a friend of the dialog (actually, wxNotebook-derived) class.
The testing code could have been embedded in the class it tests,
but I think tests want to be separate. Now, a question:

    // TODO ?? Why doesn't Show(false) work here?
    // Use Show(true) because Show(false) seems to prevent the
    // 'mvc' notebook window from responding.
    mvc.Show(true);

['mvc' previously declared as 'XmlNotebook mvc;']

Show(false) demonstrably prevents the test from working, at least
with msw. I guess that's reasonable, because dialogs are usually
for user interaction, and you can't interact with it if you can't
see it. Is that a good guess?

Is there a simple way to make the test invisible? Here's why I ask:
a complete test might be so fast that it would be okay to run it
every time the application is opened, but a normal user wouldn't
want to know that it's running.

    // WX !! There seems to be no way to do this by xrc name, e.g.
    //   notebook.SetSelection(XRCID("ignore_panel"));
    // But the pages could be enumerated and their names read.
    notebook.SetSelection(0);

Has anyone ever considered implementing a function to select a
wxNotebook tab based on the tab's (string) name? If not, then I
understand that it's an exotic need; but I thought I'd ask before
writing code to do it myself.

    // Change control values, expecting the framework to react by
    // enabling other controls.
    check0.SetValue(true);
    check1.SetValue(true);
    // The framework reacts through its EVT_UPDATE_UI handler, which
    // must be called explicitly.

In the lmi MVC framework, here's what happens. The Model knows that
'check2' must be enabled iff both 'check0' and 'check1' are checked.
The Controller sees that the condition is true, and updates the View
accordingly, through the Controller's EVT_UPDATE_UI handler.

Changing a checkbox's state with wxCheckBox::SetValue() is actually
correct here: the MVC framework is data-driven. An event-driven
approach would of course need to send an event, and might have to
worry about synchronizing its events with the application's event
handlers. That's one reason why this is so simple to test: it handles
only a few events (EVT_UPDATE_UI and a few that affect focus). Still,
though, this test routine needs to let the MVC framework do its work
asynchronously, and I've tried several different methods, not all of
which work.

    // Call wxSafeYield() to see whether EVT_UPDATE_UI is handled.
    wxSafeYield();

This doesn't work. My guess is that wxSafeYield() only gives up the
application's timeslice to other processes. What I need to do is
let this application's background processing happen--let it react to
everything in its message queue.

    // Call wxSleep(1) to see whether EVT_UPDATE_UI is handled.
    wxSleep(1);

This doesn't work, either. My guess is that it puts the whole
application to sleep.

    // Call the EVT_UPDATE_UI handler explicitly and retest.
    mvc.OnUpdateGUI(update);

This does work, at least as far as I can determine. But it doesn't
seem ideal. I really want to yield to wx idle processing, and test
the results when that's finished. For users, it happens so fast
that it doesn't matter, but an automated test can and should run
as quickly as possible.

There's another complication: OnUpdateGUI() might switch the Model
to an unstable state initially, requiring another call or calls to
OnUpdateGUI() before it stabilizes at what I suppose mathematicians
would call a 'fixed point': a state that maps to itself. But I don't
know how many calls to OnUpdateGUI() may be needed, or even whether
more than one is needed (this test code will let me find out). So
I'm thinking of having OnUpdateGUI() raise a semaphore on entry, and
clear it only when it knows that it has found a fixed point. But
please let me know if you see a better way.

I will soon add tests that require generating some events, so a
general YieldToWxIdleProcessing() function will be needed in lmi.
For instance, we need to trap invalid numbers (e.g. '2.718.28') in
a wxTextCtrl, and hold focus there until the invalid entry is
corrected or Cancel is pressed. We already have code to do that
(it blocks OK, focusing another control on the same notebook page,
and page changes), but it needs an automated test.

I'm doing this development in the 'skeleton' branch. I like to keep
this 'application skeleton' there, and it's much smaller than lmi,
so it builds much faster; and I can change it without any risk of
destabilizing our production build. Probably that's inconvenient for
you because that branch isn't autotoolized, but I imagine you can
give some advice just by reading the code quoted above.