Re: [PATCH] build: Speed up .go compilation.

[Ludovic Courtès]

address@hidden (Taylan Ulrich "Bayırlı/Kammer") skribis:

> address@hidden (Ludovic Courtès) writes:
>
>> So, what speedup to you get compared to ‘make -jN’?
>
> It should have the same improvement as 'guix pull' since it does
> basically the same thing.  I measured to make things concrete, and on my
> machine the run time of 'make -j4' decreases from ~26m to ~6m.

On my 4-core machine, it takes a bit less than 2mn.

> Here the lambda cannot entirely be factored out since the 'mutex' from
> the lexical scope is needed.  I still factored out the actual code and
> pass the mutex as an argument to the procedure in a minimal lambda, as
> seen in the new patch below; I hope it's readable this way.

It is.

>> It would be awesome if you could check that ‘make distcheck’ still
>> passes, and also make sure things behave correctly when modifying just
>> one file and running ‘make’, things like that.
>
> I'm having headaches with distcheck.  Currently it bails out because I'm
> missing tex.  Debian's version is apparently too old, and Guix's version
> is huge and has been downloading for many hours now.  I'll report back
> again when I'm done with that.

OK.

> Other notable changes in this version of the patch:
>
> - I noticed guix/config.scm and guix/tests.scm are not in MODULES in
>   Makefile.am (intentionally?), so I added them to the make-go rule.

Good catch.

This is indeed intentional, because $(MODULES) goes to the distribution,
but we wouldn’t want to distribute config.scm, which is derived from
config.scm.in at configure-time; tests.scm is not in $(MODULES) because
it is not installed.

> - I removed the MKDIR_P loop in the make-go rule, and do the equivalent
>   in the Scheme code now.
>
> - The target host and top source directory are now passed to the script
>   via lowercase environment variables, which makes the code a little
>   simpler.  I hope this is stylistically fine.

Good!

> From 697950b82ea86f7b7438e586bbf4efae3e87d8f8 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
>  <address@hidden>
> Date: Thu, 5 Nov 2015 23:42:45 +0100
> Subject: [PATCH] build: Speed up .go compilation.
>
> * build-aux/compile-all.scm: New file.
> * Makefile.am (EXTRA_DIST): Add it.
> (%.go, make-go): New rules.

The code looks good to me.  I’ve tested it, it works fine indeed, and
it’s much faster than ‘make -j4’!

I have one concern: while running it peaked at 300 MiB resident in
‘top’, which is OK on many machines but still quite a lot.  I realize we
have the same problem with ‘guix pull’ now, but that’s not great.  Part
of the problem may be that modules are not GC’d.  But anyway, that’s a
scalability problem.

What do people think?

A lesser concern is the long command-line passed to compile-all.scm,
notably because Emacs’ compilation-mode runs regexps on each line, and
that takes time proportional to the length of the line, so that leads
Emacs to hang for a second when it sees that line.  Silly.  ;-)

Thanks!
Ludo’.