Re: Storing serialised graph along with packages

[Ludovic Courtès]

Hi,

Ricardo Wurmus <address@hidden> skribis:

> Yes, indeed.  My goal is to get a *better* approximation than what the
> references database currently gives us.

I think the problem is that this would remain an approximation; people
might get a false sense that they can “decompile” a store item to a
package object and then be disappointed.

> Out of curiosity I’ve been playing with serialisation on the train ride
> and build systems are indeed a problem.  In my tests I just skipped
> them until I figured something out.
>
> I played with cutting out the sources for the package expression (using
> “package-location”) and compiling the record to a file.  Unfortunately,
> this won’t work for packages that are the result of generator procedures
> (like “gfortran”).
>
> My current approach is just to go through each field of a package record
> to generate an S-expression representing the package object, and then to
> compile that.  In a clean environment I can load that module along with
> copies of the modules under the “guix” directory that implement things
> like “url-fetch” or the search-path-specifications record.
>
> To be able to traverse the dependency graph, one must load additional
> modules for each of the store items making up the package closure.
> (This would require that in addition to just embedded references we
> would need to record the store items that were present at build time,
> but that’s easy.)

‘source-module-closure’ might be helpful:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix)
scheme@(guile-user)> ,use(guix modules)
scheme@(guile-user)> (length (source-module-closure '((gnu packages gcc))))
$2 = 272
--8<---------------cut here---------------end--------------->8---

>> The safe way to achieve what you want would be to store the whole Guix
>> tree (+ GUIX_PACKAGE_PATH), or a pointer to that (a Git commit).
>>
>> There’s a also the problem of bit-for-bit reproducibility: there’s an
>> infinite set of source trees that can lead to a given store item.  If we
>> stored along with, say, Emacs, the Guix source tree/commit that led to
>> it, then we’d effectively remove that equivalence (whitespace would
>> become significant, for instance[*].)
>
> Hmm, that’s true.  And it’s not just a problem of sources.  We might
> still introduce unimportant differences if we only serialised the
> compiled objects and completely excluded the plain text source code,
> e.g. when we refactor supporting code that has no impact on the value of
> the result but which would lead to a change in the compiled module.
>
> Can we separate the two?  Instead of installing modules (or the whole
> Guix tree) into the output directory of a store item, could we instead
> treat them like a table in the database?  Building that part would not
> be part of the package derivation; it would just be a pre- or
> post-processing step, like registering the references in the database.

To me the source/store mapping should be a separate service.

I imagine we could have some sort of a ledger that maps Git commits to
sets of store items (we could even call that a “blockchain” and be
buzzword-compliant ;-)).  Guix could come with a library to maintain
such a database, and ‘guix publish’ could even publish it.  We’d have
tools to query that database both for mappings and reverse-mappings,
things like that.

(There are also connections with the “binary transparency” ledger
discussed at the R-B summit.)

WDYT?

Ludo’.