Re: [PATCH] build: pull: Compile .scm files in one process.

[Ludovic Courtès]

address@hidden (Taylan Ulrich "Bayırlı/Kammer") skribis:

> With that and your further help on IRC, I ended up with the following
> patch.
>
> This takes about 2 minutes on my system.  The CPU gets fried during the
> parallel compilation, as expected. ;-)

Nice.  :-)

> From 0eaf7d4c264dc531718a4c0b933323f48ea91930 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
>  <address@hidden>
> Date: Thu, 5 Nov 2015 23:43:20 +0100
> Subject: [PATCH] build: pull: Compile .scm files in one process.
>
> * guix/build/pull.scm (call-with-process): Removed procedure.
> (report-build-progress): Removed procedure.
> (p-for-each): Removed procedure.

Rather:

… (call-with-process, report-build-progress, p-for-each): Remove.

> +        (let ((mutex (make-mutex))
> +              (completed 0))
> +          (par-for-each
> +           (lambda (file)
> +             (with-mutex mutex
> +               (display #\cr log-port)
> +               (format log-port "compiling...\t~5,1f% of ~d files" ;FIXME: 
> i18n
> +                       (* 100. (/ completed total)) total)
> +               (force-output log-port)
> +               (format debug-port "~%compiling '~a'...~%" file))
> +             (let ((go (string-append (string-drop-right file 4) ".go")))
> +               (compile-file file
> +                             #:output-file go
> +                             #:opts %auto-compilation-options))
> +             (with-mutex mutex
> +               (set! completed (+ 1 completed))))
> +           files)))))

Does it actually work reliably?  :-)

I think we’re in trouble if DEBUG-PORT is a real port because ports
aren’t thread-safe in 2.0, and ‘compile-file’ might write to it.  Void
ports seem to be OK because their ‘write’ method doesn’t do anything.

So I think we have to do things sequentially when DEBUG-PORT is
non-void.

I’m also concerned about modules accessed by concurrent calls to
‘compile-file’ because modules and hash tables aren’t thread-safe
either.

WDYT?

Thanks to the two of you!  Sounds like we’ll soon have a less slow or
even a fast ‘guix pull’.  ;-)

Ludo’.