Re: parallel compression in 'make dist'

[Peter Breitenlohner]

On Mon, 12 Oct 2009, Ralf Wildenhues wrote:

> * Peter Breitenlohner wrote on Wed, Sep 16, 2009 at 10:11:58AM CEST:
> > Another completely unrelated item:
> >
> > it would be nice if Automake could run gzip and bzip2 (and others) in
> > parallel when creating the distribution tarballs.  In a parallel make that
> > could save quite some time when running dist(check) for a largish project.
> >
> > Feasible?
>
> I think so.  Can you try this patch and check whether it improves `make
> dist' time for you significantly?  Thanks.

Hi Ralf,

this looks very nice. Here my timing results for 'make -j dist' in the TeX
Live tree:
          A      B      C
  real:  2:59   2:21   2:23
  user:  2:43   2:40   3:27
  sys:   0:08   0:06   0:06
where A creates .gz and .xz with Automake-1.11, B the same with your patch
"applied" manually to Makefile{.in,}, and C the same for .gz, .bz2, and .xz. 
The timing for the individual compressors is 0:28 for gzip (exactly the
difference in real time between A and B), 0:44 for bizp2 (about the
difference in user time between B and C), and 2:09 for xz.

All tests on a (more or less idle) Intel Core2 Quad with i686-linux-gnu.

> I did some quick .....
>
> indicating that the extra make recursion seems to be acceptable, and the
> optimization worthwhile for some packages....

I fully agree. For small and flat packages everything is fast, and for large
and/or deep packages this extra recursion ought to be negligible.

> ... Notably, -j2 isn't helpful
> for GM, as it starts the slowest encoder late only.

I could think of one more improvement for -j2: running the slowest
encoder first (provided that can be achieved easily).

> BTW, this is definitely a 1.12-only change:

Of course.

Regards
Peter Breitenlohner <address@hidden>