[PATCH] DRAFT: build: Compile scheme modules in batches (was Re: Release!)

[Mark H Weaver]

Ricardo Wurmus <address@hidden> writes:

> Hi Ludo,
>
>>> Here are some important items I can think of:
> […]
>>>   • Guile 2.2 compiler terrible issue:
>>>     <https://lists.gnu.org/archive/html/guile-devel/2017-05/msg00033.html>.
>
> One way to side-step this issue for the upcoming release is to use one
> Guile process per file when running “guix pull”.  This will make it run
> a lot slower, but that would be better than the current situation.

I've attached a workaround that I've been using for the last 6 weeks on
my MIPS-based Yeeloong running GuixSD, since it only has 1 GB of RAM and
otherwise it would not be able to successfully build the 'guix' package.

Note that I never use 'guix pull', so I'm not sure off-hand whether this
solves the problem there, but it certainly greatly reduces the memory
needed to run 'make' and thus to build the 'guix' package.

This patch modifies build-aux/compile-all.scm to work as follows: after
loading all modules in the parent process, it then forks off a child and
compiles 20 modules in parallel within that child while the parent
waits.  When the child is finished compiling those 20 modules, the child
exits (thus freeing the memory leaked during compilation), and then the
parent spawns a new child to compile the next 20 modules, and so on,
until all the modules are compiled.

We should probably consider applying this to master.  Thoughts?

      Mark


>From 05d5581ff71eb3b48773a5d46b612202de0492fb Mon Sep 17 00:00:00 2001
From: Mark H Weaver <address@hidden>
Date: Tue, 22 Aug 2017 03:26:10 -0400
Subject: [PATCH] DRAFT: build: Compile scheme modules in batches.

---
 build-aux/compile-all.scm | 48 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 41 insertions(+), 7 deletions(-)

diff --git a/build-aux/compile-all.scm b/build-aux/compile-all.scm
index 147bb8019..96658e069 100644
--- a/build-aux/compile-all.scm
+++ b/build-aux/compile-all.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2016 Taylan Ulrich Bayırlı/Kammer <address@hidden>
 ;;; Copyright © 2016, 2017 Ludovic Courtès <address@hidden>
+;;; Copyright © 2017 Mark H Weaver <address@hidden>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -19,6 +20,7 @@
 
 (use-modules (system base target)
              (system base message)
+             (srfi srfi-1)
              (ice-9 match)
              (ice-9 threads)
              (guix build utils))
@@ -118,13 +120,45 @@
   ((_ . files)
    (let ((files (filter file-needs-compilation? files)))
      (for-each load-module-file files)
-     (let ((mutex (make-mutex)))
-       ;; Make sure compilation related modules are loaded before starting to
-       ;; compile files in parallel.
-       (compile #f)
-       (par-for-each (lambda (file)
-                       (compile-file* file mutex))
-                     files)))))
+     ;; Make sure compilation related modules are loaded before starting to
+     ;; compile files in parallel.
+     (compile #f)
+     ;; Flush all ports before entering the fork loop, to avoid flushing them
+     ;; more than once within the child processes created below.
+     (flush-all-ports)
+
+     ;; FIXME The following loop works around the apparent memory leak in the
+     ;; compiler of guile-2.2.2, where compiling scheme modules requires
+     ;; increasing amounts of memory, up to nearly 2 gigabytes when all guix
+     ;; sources are compiled within a single process.
+     ;;
+     ;; Ideally, we would simply apply 'par-for-each' to the entire set of
+     ;; files.  For now, to work around the memory leak, we spawn subprocesses
+     ;; to compile the files in batches of up to 20 files each.
+     (let fork-loop ((files files))
+       (unless (null? files)
+         (call-with-values (lambda ()
+                             (split-at files (min 20 (length files))))
+           (lambda (current-batch remaining-files)
+             ;; IMPORTANT: as noted in the Guile manual, it is unsafe to fork a
+             ;; process that has multiple threads running.  Here we avoid this
+             ;; difficulty by spawning threads only within the child processes,
+             ;; which never call fork.
+             (match (primitive-fork)
+               (0
+                ;; This is the child.  It spawns threads but never forks.
+                (let ((mutex (make-mutex)))
+                  (par-for-each (lambda (file)
+                                  (compile-file* file mutex))
+                                current-batch))
+                (primitive-exit))
+               (child-pid
+                ;; This is the parent.  It forks but never spawns threads.
+                (match (waitpid child-pid)
+                  ((_ . 0)
+                   (fork-loop remaining-files))
+                  ((_ . status)
+                   (primitive-exit (or (status:exit-val status) 1)))))))))))))
 
 ;;; Local Variables:
 ;;; eval: (put 'with-target 'scheme-indent-function 1)
-- 
2.14.1