[Monotone-devel] Sneak preview: net.venge.monotone.parallel-tests

[Zack Weinberg]

I've been working on making the test suite parallelizable.
Unfortunately, I am leaving town for the week with it only 90% done,
but this is interesting enough in itself that I wanted to call it to
folks' attention.

On the branch net.venge.monotone.parallel-tests:

* Multiple instances of ./tester can run in parallel without
clobbering each other.  The per-test directories go in a subdirectory
of tester_dir, named the same as the testsuite directory (so "tests"
for the main testsuite and "tester-tests" for the self-tests); the
master logfile for a given suite is named $subdir.log (where $subdir
is the aforementioned subdirectory).

* The unit tests are run under the direction of a tester instance.
The unit_tests binary (renamed unit_tester for collision avoidance)
now just spits out a list of all the tests it can run if invoked with
no arguments.  If invoked with one argument, it runs that test.  A Lua
testsuite is synthesized on the fly when you do ./run_unit_tests.  I
did this because I wanted to write the actual parallelization code
only once, but it has a couple of nice side effects: the unit tests no
longer drop "x" and "y" directories in the build dir, and the
gargantuan unit_tests.log is automatically pruned down to just the
bits relevant to failing test cases.

* ./tester forks off a subprocess for each and every test case.  At
present, those subprocesses are serialized.  (The remaining 10% of the
work on this branch will be adding code to tester.cc to parallelize
them.)  This obviously makes it *possible* to parallelize things
further, but also has good side effects: test cases now cannot affect
each other via persistent Lua state and cannot scribble on stdout.
That's also a minor curse - tester-tests/cleanup-[12] are no longer
testing what they used to, and I can't think of any good way to prove
that the cleanup hook ran.  Ideas appreciated.

* Tangentially, I hacked the Makefile to not waste time recompiling
object files two or three times with exactly the same options for
inclusion in 'mtn', 'unit_tester', and 'tester'.

* Also tangentially, I fixed it so the unit tests' unit tests can
actually be run by default.

The only real downsides of all this, as I see it, are increased
overhead for serial testing (due to forking, re-execing, and
re-initializing tester for every test case - this can be reduced, at
least on Unix) and a sadly necessary change to the progress scroll: we
cannot report the currently running test case anymore.  I anticipate
hefty wall-time speedups when I finish the code to parallelize between
test cases, but already I can demonstrate nice effects just from being
able to parallelize between test suites.  On my fairly old but still
decent two-way hyperthreaded i386 box:

# complete build including unit_tester and tester

trunk $ time make -j2 all unit_tester tester >& build.log
real 21m17.054s user 40m6.026s sys 1m21.757s

parallel-tests $ time make -j2 all unit_tester tester >& build.log
real 12m21.754s user 22m31.220s sys 0m50.295s

# serial test
trunk $ time make check
real 20m19.893s user 14m23.214s sys 1m16.325s

parallel-tests $ time make check
real 24m55.132s user 19m10.168s sys 1m25.081s

# parallel test invocation (can't do this via the Makefile at present,
stupid automake)

parallel-tests $ ( ( ./run_tester_tests ; ./run_unit_tests ) &
./run_lua_tests & wait ) >& check2.log & time wait
real 14m55.427s user 20m30.497s sys 1m39.158s

See y'all Monday -
zw