[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Gnash-commit] gnash/doc/C internals.xml [testing-documentation-refactor
|
From: |
Ann Barcomb |
|
Subject: |
[Gnash-commit] gnash/doc/C internals.xml [testing-documentation-refactor] |
|
Date: |
Tue, 19 Dec 2006 15:46:30 +0000 |
CVSROOT: /sources/gnash
Module name: gnash
Branch: testing-documentation-refactor
Changes by: Ann Barcomb <ann> 06/12/19 15:46:29
Modified files:
doc/C : internals.xml
Log message:
This commit starts the process of splitting the test
documentation to appeal to three different audiences:
users, Gnash developers, and Flash developers. The
users section has been written, while the remaining
two sections have been outlined.
This commit is being made in a branch created for this
purpose so that Sandro will have a chance to comment
on these changes before they become part of the standard
checkout. Once they are merged with the main branch I
will add a comment to the ChangeLog.
CVSWeb URLs:
http://cvs.savannah.gnu.org/viewcvs/gnash/doc/C/internals.xml?cvsroot=gnash&only_with_tag=testing-documentation-refactor&r1=1.47&r2=1.47.2.1
Patches:
Index: internals.xml
===================================================================
RCS file: /sources/gnash/gnash/doc/C/internals.xml,v
retrieving revision 1.47
retrieving revision 1.47.2.1
diff -u -b -r1.47 -r1.47.2.1
--- internals.xml 12 Dec 2006 22:44:29 -0000 1.47
+++ internals.xml 19 Dec 2006 15:46:29 -0000 1.47.2.1
@@ -1903,612 +1903,163 @@
</sect2>
<sect2 id="testing">
- <title>Testing Support</title>
+ <title>Testing</title>
- <sect3 id="testtools">
- <title>Testing Tools</title>
+ <sect3 id="runtests">
+ <title>Running Tests</title>
<para>
- Currently Gnash uses three other tools to help with
- testing. Two of these are free compilers for the Flash
- format. This lets us write simple test cases for Gnash to test
- specific features, and to see how the features operate.
+ Gnash users are encouraged to run the test suite and report
+ failing tests to the development team.
</para>
+<!--
<para>
- The primary compiler used at this time is <ulink type="http"
- url="http://ming.sf.net";>Ming</ulink>. Since release 0.3,
- <emphasis>Ming</emphasis> includes a command-line compiler,
- <emphasis>makeswf</emphasis>. This allows test case development
- to be done entirely with free tools.
+ If a test fails, please report it by following the
+ <link linkend="bugreport">instructions for reporting a bug</link>.
</para>
+-->
+ <sect4 id="dejagnu">
+ <title>Using DejaGnu</title>
<para>
- The other tools are optional.
+ The easiest way to run Gnash's test suite is to install
+ <emphasis>
<ulink type="http"
- url="http://www.gnu.org/software/dejagnu";>DejaGnu</ulink>
- is used to run multiple test cases in an automated
- manner. <emphasis>DejaGnu</emphasis> is used by many other <ulink
- type="http" url="http://www.gnu.org";>GNU</ulink> projects like
- <ulink type="http" url="http://gcc.gnu.org";>GCC</ulink> and
- <ulink type="http" url="http://www.samba.org";>Samba</ulink>.
- </para>
-
- </sect3>
-
- <sect3 id="testcases">
- <title>Test Cases</title>
-
- <para>
- ActionScript test cases are located under testsuite/actionscript.all/;
- these are organized in one file for the ActionScript class.
- Other Ming-generated tests are under testsuite/ming-misc.all/;
- these are typically used to test specific tag types.
- Full movies are located in testsuite/movies.all/ and
- sample movies are found in testsuite/samples/.
- Other directories in testsuite/ are (or shall be) used for other
- kind of tests.
- </para>
-
- </sect3>
-
- <sect3 id="writeastests">
- <title>Writing ActionScript Tests</title>
-
- <para>
- Writing ActionScript tests is very simple. The
- <emphasis>makeswf</emphasis> compiler makes use of the C preprocessor,
- thus allowing the inclusion of definitions for macros and external
- files. We use these feature to provide common utilities
- for test units.
- </para>
-
- <para>
- Each test unit sets an <emphasis>rcsid</emphasis> variable, includes the
- <emphasis>check.as</emphasis> file and performs some checks using
- the provided macros. Here is an example:
-
+
url="http://www.gnu.org/software/dejagnu";>DejaGnu</ulink></emphasis>.
+ After installing DejaGnu, change to the
+ <emphasis>testsuite</emphasis> directory and run:
<programlisting>
-
- // This variable will be used by check.as
- // to show testcase info as part of the test runs.
- rcsid="Name and version of this testcase, usually the RCS id";
-
- #include "check.as"
-
- // Test object creation
- check(new Object() instanceOf Object);
-
- // Test parseInt
- check(isNaN(parseInt('none')));
-
- // Test assignment
- var a = 1;
- check_equals(a, 1);
-
- // .. your tests here ...
+ make check
</programlisting>
</para>
<para>
- The check(expr) macro will <emphasis>trace</emphasis> PASSED or FAILED
- together with the expression being evaluated and the line number
- of the check. This is the format expected by DejaGnu.
- </para>
-
- <para>
- The <emphasis>check_equals(obtained, expected)</emphasis> macro uses
equality operator
- <emphasis>==</emphasis> to check for equality. When possible, use of the
- <emphasis>check_equals()</emphasis> macro is preferred over
<emphasis>check()</emphasis>
- because it shows what the actual result was in case of a failure.
- DejaGnu.
- </para>
-
- <para>
- Additionally, the check.as file provides a transparent way to send
- results to a TextField rather then using trace. This is very useful
- when you use a flash player without tracing support.
- </para>
-
- <para>
- Test units are built by running <emphasis>make TestName.swf</emphasis>.
- This will use TestName.as as source. To build a "visual" tracing
- version you'd run <emphasis>make TestName.vswf</emphasis>.
- </para>
-
- <para>
- Note that if you get a syntax error from the compiler, the line
- number will refer to the pre-processed file. This file is called
- <emphasis>TestName.as.pp</emphasis> and it's not thrown away by
- <emphasis>makeswf</emphasis> to make debugging easier.
- </para>
-
- <para>
- Sometimes an expression is only supported by a specific SWF
- version, or it's evaluated differently by different SWF versions.
- For this purpose the framework provides an OUTPUT_VERSION macro
- that you can use to switch code based on output version. For example:
-
+ If you encounter a problem with a test, increasing the
+ verbosity may make the issue easier to spot.
+ Additional details are visible when
+ <emphasis>RUNTESTFLAGS</emphasis> are used to add the
+ <emphasis>verbose</emphasis> and <emphasis>all</emphasis> options.
+ Verbose prints more information about the testing process, while
+ all includes details on passing tests.
<programlisting>
-
- #if OUTPUT_VERSION >= 7
- check(_root.getSWFVersion == OUTPUT_VERSION);
- #endif
-
+ make check RUNTESTFLAGS="-v -a"
</programlisting>
</para>
- </sect3>
-
- <sect3 id="writemingtests">
- <title>Writing Ming-based self-contained SWF tests</title>
-
- <para>
- Ming-based test cases are located in testsuite/misc-ming.all
- and contain a test generator and a test runner.
- The test generator (usually a C program) is used to produce the SWF
- file, while the test runner (a C++ program) will run it using a
- MovieTester class.
- Note that only the test generator needs Ming, not the test
- runner, so if Ming isn't installed on the user's host,
- the test cases can still be run as long as SWF has been distributed.
- </para>
-
- <para>
- Producing tests using Ming has the advantage that you can easily see
- and modify the full source code for the SWF movie, and you can use
- some <link linkend="ming_testgenerator_facilities">facilities</link>
- provided by the Gnash testing framework to easily run tests.
- </para>
-
- <para>
- For generic Ming API documentation, see <ulink type="http"
- url="http://www.libming.org/";>http://www.libming.org</ulink>.
- </para>
-
- <sect4 id="ming_testgenerator_facilities">
- <title>Using Ming-based test generators facilities</title>
-
- <para>
- Ming-based test generator facilities, which might be moved into
- a loadable SWF in the future, can be currently used by your test
- generator by including the ming_utils.h file and calling the
- appropriate functions.
- </para>
-
- <para>
- The most useful facility provided for Ming-based SWF test generators
- is a Dejagnu-like TestState ActionScript class.
- In order to use this facility you must call 'add_dejagnu_functions()'
- right after Movie creation.
- The function takes an SWFMovie object and some parameters specifying
- depth and location of the "visual" trace textfield; it instantiates
- a global 'TestState' ActionScript object to keep track of test's state.
- </para>
<para>
- You will <emphasis>not</emphasis> need to directly invoke the
- TestState object created by the 'add_dejagnu_functions()' routine,
- rather you will be using other C functions which hide its complexity:
-
+ It is possible to run just a particular test, or group of tests,
+ by specifying the directory or compiled test file. For instance,
+ the following runs the 'clip_as_button2-TestRunner' test from
+ the top source directory:
<programlisting>
-
- check(SWFMovie mo, const char* expr, int expect_failure)
-
- Evaluate an ActionScript expression.
- If 'expect_failure' is true a failure is expected
- (for cases where the call exposes a known bug).
-
- check_equals(SWFMovie mo, const char* obtained,
- const char* expected, int expect_failure)
-
- Evaluate an ActionScript expression against an
- expected output.
- If 'expect_failure' is true a failure is expected
- (for cases where the call exposes a known bug).
-
- print_tests_summary(SWFMovie mo)
-
- This will print a summary of tests run, and should be
- called as the last step in your SWF generator.
-
+ make -C testsuite/samples/clip_as_button2-TestRunner check
</programlisting>
-
- </para>
-
- <para>
- Test cases generated using Ming and the provided
- <link linkend="ming_testgenerator_facilities">facilities</link>
- will be self-contained, which means they can be used as tests
- by simply running them with whatever Player you might have.
- Any 'check' or 'check_equals' result will be both traced and
- printed in a textfield. You can use 'gprocessor -v' to have
- Gnash use them as tests.
- </para>
-
- <para>
- See section <link linkend="writing_test_runners">Writing Test
Runners</link>
- for information about writing SWF test runners.
</para>
-
</sect4>
-
- </sect3>
-
- <sect3 id="writing_dejagnu_so_tests">
- <title>Writing self-contained SWF tests with other compilers</title>
-
- <para>
- If you want/need to use a different compiler for your test cases
(there's
- plenty of open source tools for generating SWF out there), you can still
- make use of a loadable SWF utility provided as part of the Gnash
testsuite
- to let your test consistent with the rest of the suite.
- </para>
+ <sect4 id="manually">
+ <title>Running The Tests Manually</title>
<para>
- The loadable module is called <code>Dejagnu.swf</code> and is built
during
- <code>make check</code> under testsuite/misc-ming.all. In order to use
it
- you will need to load it into your SWF. We currently load it with an
IMPORT
- tag for our ActionScript based test cases, but you can probably also use
- loadMovie or whatever works in the target SWF you're generating. Just
make
- sure that the module is initialized before using it. You can check this
by
- inspecting the <code>dejagnu_module_initialized</code> variable, which
will
- be set to 'true' when all initialization actions contained in the
- <code>Dejagnu.swf</code> file are executed.
+ You may also run test cases by hand, which can be useful if you
+ want to see all the debugging output from the test case. Often
+ the messages which come from deep within Gnash are most useful for
+ development.
</para>
<para>
- Once the module is loaded you will be able to invoke the following
functions,
- all registered against the <code>_root</code> sprite (effects of
<code>_lockroot</code>
- untested):
+ The first step is to compile the test case. Ming's
+ <emphasis>makeswf</emphasis> program is used to compile the test
+ case into Flash. By default, no options are required.
+ Running <emphasis>makeswf</emphasis> looks like this:
<programlisting>
-
- check(expression);
-
- Evaluate the expression.
- Trace result (PASSED: expression / FAILED: expression).
- If fails, *visually* trace the failure.
-
- check_equals(obtained, expected)
-
- Evaluate an expression against an expected output.
- Trace result (PASSED: obtained == expected / FAILED: expected
X, obtained Y)
- If fails, *visually* trace the failure.
-
- xcheck(expression);
-
- Evaluate the expression.
- Trace result (XPASSED: expression / XFAILED: expression).
- If fails, *visually* trace the failure.
-
- xcheck_equals(obtained, expected)
-
- Evaluate an expression against an expected output.
- Trace result (XPASSED: obtained == expected / XFAILED: expected
X, obtained Y)
- If fails, *visually* trace the failure.
-
- note(string)
-
- Print string, both as debugging and *visual* trace.
-
- totals()
-
- Print a summary of tests run, both as debugging and *visual*
traces.
-
+ shellprompt> makeswf XML.as
+ Output file name: out.swf
+ Output compression level: 9
+ Output SWF version: 6
+ Preprocessing XML.as... done.
+ Compiling `XML.as.pp' into frame 1... done.
+ Saving output to out.swf... done.
+ shellprompt>
</programlisting>
</para>
<para>
- Visual traces are lines of text pushed to a textarea defined by the
<code>Dejagnu.swf</code> module.
- The textarea is initially placed at <code>0, 50</code> and is
<code>600x800</code> in size.
- You can resize/move the clip after loading it.
- Also, you can completely make the clip invisible if that bothers you.
The important thing is the
- *debuggin* trace (call to the trace function). The latter will be used
by the testing framework.
- </para>
-
- <para>
- See section <link linkend="writing_test_runners">Writing Test
Runners</link>
- for information about writing a test runners for your self-contained
tests.
- </para>
-
- </sect3>
-
- <sect3 id="writing_test_runners">
- <title>Writing Test Runners</title>
-
- <para>
- Test runners are executables that run one or more tests,
- writing results in Dejagnu form to standard output.
- </para>
-
- <para>
- The Dejagnu form uses a standard set of labels when printing test
- results. These are:
- <informaltable frame="all">
- <?dbhtml table-width="75%" ?>
- <tgroup cols="2">
- <thead>
- <row>
- <entry valign="top">
- <para>Label</para>
- </entry>
- <entry valign="top">
- <para>Meaning</para>
- </entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry valign="top" align="left">
- <para>PASSED</para>
- </entry>
- <entry valign="top" align="left">
- <para>The test succeeded.</para>
- </entry>
- </row>
- <row>
- <entry valign="top" align="left">
- <para>FAILED</para>
- </entry>
- <entry valign="top" align="left">
- <para>The test failed.</para>
- </entry>
- </row>
- <row>
- <entry valign="top" align="left">
- <para>XPASSED</para>
- </entry>
- <entry valign="top" align="left">
- <para>The test succeeded, but was expected to fail.</para>
- </entry>
- </row>
- <row>
- <entry valign="top" align="left">
- <para>XFAILED</para>
- </entry>
- <entry valign="top" align="left">
- <para>The test failed, and was expected to fail.</para>
- </entry>
- </row>
- <row>
- <entry valign="top" align="left">
- <para>UNRESOLVED</para>
- </entry>
- <entry valign="top" align="left">
- <para>The results of the test could not be automatically
- parsed.</para>
- </entry>
- </row>
- <row>
- <entry valign="top" align="left">
- <para>UNTESTED</para>
- </entry>
- <entry valign="top" align="left">
- <para>This test case is not complete.</para>
- </entry>
- </row>
- <row>
- <entry valign="top" align="left">
- <para>UNSUPPORTED</para>
- </entry>
- <entry valign="top" align="left">
- <para>The test case relies on a conditional feature which
- is not present in your environment.</para>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </para>
-
- <para>
- The following labels may also appear:
- <informaltable frame="all">
- <?dbhtml table-width="75%" ?>
- <tgroup cols="2">
- <thead>
- <row>
- <entry valign="top">
- <para>Label</para>
- </entry>
- <entry valign="top">
- <para>Meaning</para>
- </entry>
- </row>
- </thead>
- <tbody>
- <row>
- <entry valign="top" align="left">
- <para>ERROR</para>
- </entry>
- <entry valign="top" align="left">
- <para>There was a serious error in running the test. </para>
- </entry>
- </row>
- <row>
- <entry valign="top" align="left">
- <para>WARNING</para>
- </entry>
- <entry valign="top" align="left">
- <para>There may have been a problem with running the
- test.</para>
- </entry>
- </row>
- <row>
- <entry valign="top" align="left">
- <para>NOTE</para>
- </entry>
- <entry valign="top" align="left">
- <para>There was some additional information given about
- the test.</para>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </para>
-
- <sect4 id="generic_test_runner">
- <title>Using the generic test runner for self-contained SWF
tests</title>
-
- <para>
- The simplest test runner is one that simply invokes Gnash
- in verbose mode against a self-contained SWF test movie.
- Self-contained SWF test movies are the ones that print
- the PASSED/FAILED etc. lines using ActionScript (traces).
- By invoking Gnash in verbose mode this movie will behave
- as a compliant "Test Runner".
- </para>
-
- <para>
- A generator for simple test runners can be found in
- <code>testsuite/generic-testrunner.sh</code>.
- The script can be invoked by passing it <code>$(top_builddir)</code>
- as the first argument and the name of the SWF file (without the path)
- as the second argument. This will create a specific runner for your
- test in the current build directory.
- A simple Makefile.am rule for doing this follows:
+ Once you have the flash movie version of the test case, you can
+ run it through the Gnash standalone player with some options
+ which indicate that it is a simple test case. The unit level
+ tests for ActionScript classes are run without the associated
+ graphics display.
<programlisting>
-MyTest-Runner: $(srcdir)/../generic-testrunner.sh MyTest.swf
- sh $(srcdir)/../generic-testrunner.sh $(top_builddir) MyTest.swf > $@
- chmod +x $@
+ shellprompt> gprocessor -v out.swf
+ PASSED: XML::XML() constructor
+ PASSED: XML::addRequestHeader() exists
+ PASSED: XML::appendChild() exists
+ ...
</programlisting>
</para>
-
</sect4>
+ </sect3>
- <sect4 id="writing_movie_testers">
- <title>Writing Movie testers</title>
-
- <para>
- There are some parts of Gnash that can NOT be tested
- by only using ActionScript tests. Examples include: frame
- advancements, actual actions execution, gui events and so on.
- </para>
+ <sect3 id="writing_tests">
+ <title>Writing Tests</title>
<para>
- In this case you might want to use the MovieTester class to
- implement a C++ test runner. Be aware that you can *mix* tests in
- the MovieTester-based class with *self-contained*
- tests in the SWF file as long as you activate verbosity for
- the debug logfile. This is done, for example, for the
- DefineEditTextVariableNameTest.swf file. The corresponding
- test runner (DefineEditTextVariableNameTest-Runner) is a C++
- runner based on MovieTester class. If you run the runner you
- see two kinds of test results: the ones coming from the ActionScript
- engine, and the ones coming from the test runner. You can
- distinguish between the two because the former contains an additional
- timestamp and the latter does not. Also, you'll see two final
- summaries for the two test sets. The 'make check' rule, which uses
- the testsuite/simple.exp output parser as its work-horse, will
- count test results from both test sets.
+ Experienced Flash developers can help the Gnash development process
+ by creating test movies.
</para>
+ <sect4 id="ming_tests">
+ <title>Using Ming</title>
<para>
- Movie testers are executables which load an SWF, generate events
- (both user or system) on it, and check its state using
- a standard interface.
+ <emphasis><ulink type="http"
+ url="http://ming.sf.net";>Ming</ulink></emphasis> is the
+ primary compiler used at this time. As of release 0.3,
+ <emphasis>Ming</emphasis> includes a command-line compiler,
+ <emphasis>makeswf</emphasis>. This allows test case development
+ to be done entirely with free tools.
</para>
- <para>
- To help this process a MovieTester class is defined in the
- testsuite/MovieTester.{h,cpp} files; see Doxygen documentation
- for more information.
- </para>
+ <sect5 id="ming_actionscript_tests">
+ <title>Writing ActionScript Tests</title>
<para>
- Note that you do NOT need access to the SWF source code in order
- to implement a Movie tester for it. Some knowledge about the
- expected behavior suffices.
+ ActionScript test cases are located in
+ 'testsuite/actionscript.all'. They are organized into
+ one file per ActionScript class.
</para>
+ </sect5>
- </sect4>
-
- </sect3>
-
- <sect3 id="runtests">
- <title>Running The Tests</title>
-
- <sect4 id="dejagnu">
- <title>Using DejaGnu</title>
+ <sect5 id="ming_selfcontained_tests">
+ <title>Writing Ming Self-Contained SWF Tests</title>
<para>
- The simple way to run the tests is to install
- <emphasis>DejaGnu</emphasis>, and use that to run the tests. That
- handles all the details to compile and execute the tests. To
- run the tests using DejaGnu, change to the
- <emphasis>testsuite</emphasis> directory and type:
- <programlisting>
- make check
- </programlisting>
+ Ming-generated tests which do not involve ActionScript
+ are located in 'testsuite/ming-misc.all' and are typically
+ used to test specific tag types.
</para>
+ </sect5>
+ </sect4>
- <para>
- You can get more details by adding command line option when
- invoking <emphasis>make</emphasis>. The <emphasis>make
check</emphasis> target
- in the Makefile supports a variable, <emphasis>RUNTESTFLAGS</emphasis>
- that gets passed to DejaGnu when it's invoked by
<emphasis>make</emphasis>.
-
- <programlisting>
- make check RUNTESTFLAGS="-v -a"
- </programlisting>
-
- This adds the <emphasis>verbose (-v)</emphasis> option and the
- <emphasis>all (-a)</emphasis> option. Verbose prints much more
- information about how DejaGnu is running the test. It may be
- too much information, but if you think you are having a
- problem with running a test case, this is a good way to track
- it down. The <emphasis>all</emphasis> option tells DejaGnu to print
- all the tests that <emphasis>PASS</emphasis>, as well as those that
- <emphasis>FAIL</emphasis>.
- </para>
+ <sect4 id="othercompiler_tests">
+ <title>Using Another Compiler</title>
</sect4>
+ </sect3>
- <sect4 id="manually">
- <title>Manually Running Tests</title>
- <para>
- You can also run all the test cases by hand, which is useful
- if you want to see all the debug output from the test
- case. Often the debug messages which come deep from within
- Gnash are the most useful during development.
- </para>
+ <sect3 id="writing_test_runners">
+ <title>Writing Test Runners</title>
<para>
- The first step is to compile the test case. Ming's
- <emphasis>makeswf</emphasis> program is used to compile the test
- case into Flash. By default, no options are
- required. Running <emphasis>makeswf</emphasis> looks like this:
-
- <programlisting>
- shellprompt> makeswf XML.as
- Output file name: out.swf
- Output compression level: 9
- Output SWF version: 6
- Preprocessing XML.as... done.
- Compiling `XML.as.pp' into frame 1... done.
- Saving output to out.swf... done.
- </programlisting>
+ Gnash developers may wish to write test runners, which are
+ executables which run one or more tests, writing results in
+ <emphasis>Dejagnu</emphasis> format to standard output.
</para>
+ </sect3>
- <para>
- Once you have the flash movie version of the test case, you
- can run it through the Gnash standalone player with a few
- options which enable a simple test cases to be run. The
- unit level tests for an ActionScript class are run without
- an associated graphics display.
- </para>
- <programlisting>
- shellprompt> gprocessor -v out.swf
- PASSED: XML::XML() constructor
- PASSED: XML::addRequestHeader() exists
- PASSED: XML::appendChild() exists
- ...
- </programlisting>
- </sect4>
- </sect3>
</sect2>
</sect1>
- [Gnash-commit] gnash/doc/C internals.xml [testing-documentation-refactor],
Ann Barcomb <=