Re: [PATCH] build: pull: Compile .scm files in one process.

[Taylan Ulrich Bayırlı/Kammer]

Andy Wingo <address@hidden> writes:

> On Fri 06 Nov 2015 16:41, address@hidden (Taylan Ulrich "Bayırlı/Kammer") 
> writes:
>
>> Andy Wingo <address@hidden> writes:
>>
>>> On Thu 05 Nov 2015 17:10, address@hidden (Taylan Ulrich "Bayırlı/Kammer") 
>>> writes:
>>>
>>>> It used to max out every CPU core, now just one. :-)
>>>> It used to take ~18 minutes on my machine, now less than 3.
>>>
>>> If you compile within a par-for-each you should be able to peg your CPU
>>> core again, but actually reduce the time :)
>>
>> From what I understand, that would probably ignite the bug again.
>>
>> We need to ensure that as soon as a module file is compiled, it's also
>> explicitly loaded before anything else is compiled (which might import
>> it), otherwise that compilation will import the "degenerate" version of
>> the module that results from compiling but not loading it.
>>
>> But that's really just my shallow high-level understanding of the bug,
>> and could be way off.  If you have any insights on what's really going
>> on, that would be greatly appreciated. :-)
>
> AFAIU the problem that makes compilation slow is that *expansion* is
> slow.  When all your Scheme files are fresh, compiling 1 module has to
> expand all N modules.  Using one process to compile avoids this N^2
> penalty, just paying the O(N) cost up-front and then the marginal
> compilation cost is O(1).

Right, I was conflating expansion and compilation.

> There is benefit to compiling support modules before compiling (gnu
> packages) so that Guix's macros run compiled and not interpreted, but if
> you already have all of the modules expanded and loaded I don't think
> there is any advantage to loading the freshly compiled .go files.
>
> I do not understand what you mean by "degenerate" modules :)  An
> interpreted module should act the same as a compiled module.  I am
> interested to hear what difference you can perceive between the two.

The relevant bug report is <http://bugs.gnu.org/15602>.

According to Ludo's explanation, compiling a module file leads to the
module being created in the runtime, but with syntax bindings only, and
runtime bindings missing.  That's what I mean with "degenerate" module
for lack of a better term.  Loading the same file explicitly afterwards
(or using load-compiled on the generated .go) seems to solve the issue.

Taylan