Re: [PATCH v4 1/3] parallel-tests: add auxiliary script 'pt-driver', refactor

[Stefano Lattarini]

On Friday 17 June 2011, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Thu, Jun 16, 2011 at 10:00:31AM CEST:
> > This refactoring should cause no API of functionality change,
> > and is meant only to simplify the future implementation of TAP
> > and SubUnit testsuite drivers.  More precisely, our roadmap is
> > to move most of the "testsuite driving" features out of the
> > Automake-generated Makefiles, and into external scripts with
> > well-defined interfaces.  This will allow the user to define
> > its own personalized testsuite drivers, and will also offer us
> > a framework upon which to implement our new TAP and SubUnit
> > drivers, all in a very unobtrusive way and retaining an high
> > degree of code reuse and backward-compatibility.
> 
> I generally like the direction this is taking.  The point of best
> separation between which code goes into Makefile.in and which into
> the driver scripts can be fine-tuned when we have more than one such
> script.
> 
> Actually, yes, before deciding on this for real I really do want to see
> a nontrivial other driver script.  There is no point in hardcoding
> too much in several driver scripts if it all needs to be the same
> anyway.
> 
> Please measure the time overhead your changes introduce into the current
> code, for a trivial testsuite (say, 50 tests running 'true'), and a
> nontrivial one like Automake's and one with faster tests.  Thanks.
> 
Done.

I've run each of the testsuites repeated times, in the hope of shaking
out most of the noise.

I've run the Automake testsuite four times on a fast Solaris 10 system
(quadcore 3 GHz, 20 Gb of RAM) in low priority (a niceness of 19) and
with 32 processes concurrently ("make -j 32 check")

Then, I've run the Coreutils testsuite three times, on my Debian desktop
(single core 1.5 Ghz, 768 Mb of RAM), with one job at the time (i.e.,
plain "make check", no `-j' option).

Finally, I've run the "dummy" testsuite on my desktop ten times -- five
in serial mode ("make check") and five in concurrent mode, with four jobs
at the time ("make -j4 check").

-*-*-*-

Here are the results:


Automake Testsuite
==================

  master branch
  -------------

  real    76m46.661s
  user    127m54.170s
  sys     136m31.526s

  test-harness branch
  -------------------

  real    77m42.103s
  user    126m7.108s
  sys     140m26.200s


Coreutils Testsuite
===================

  master branch
  -------------

  real    34m55.552s
  user    4m23.216s
  sys     6m20.312s

  test-harness branch
  -------------------

  real    35m42.560s
  user    4m29.453s
  sys     6m27.740s


Dummy Testsuite
===============

  master branch
  -------------

  real    1m3.158s
  user    0m11.045s
  sys     0m17.421s

  test-harness branch
  -------------------

  real    1m11.025s
  user    0m16.249s
  sys     0m21.073s


-*-*-*-

Regads,
  Stefano