Re: [PATCH] Prototype register-path

[Chris Marusich]

Hi Caleb,

I'm sure Ludo may have more to say, but I think I can help you with a
few of your questions.

Caleb Ristvedt <address@hidden> writes:

>> Though something went from with your email client.  :-)
>
> Aye, turns out M-x epa-mail-sign doesn't work well with attachments. I
> posted another message that should be signed correctly, as well as this
> one.

Looks like your latest email was signed correctly.  I see that it was
signed using your Savannah key, which has the following fingerprint:

74D6 A930 F44B 9B84 9EA5  5606 C166 AA49 5F7F 189C

> - A derivation is a node in a (acyclic?) graph of "what depends on what" or,
>   alternately put, what is input to what.

I don't think there's a simple answer for the question of "what is a
derivation"?  I'll explain what I know, and hopefully it'll help you.

No, derivations cannot depend on one another [1].  I believe it is
correct to think of a derivation as a node in an acyclic graph of
inter-dependent software components; that is one way to think about
derivations.  Perhaps Ludo can say more about that.

The word "derivation" is a bit overloaded.  Fundamentally (and vaguely),
it means "a component build action, which *derives* the component from
its inputs" [2].

More specifically, there are various interpretations of the word
"derivation", depending on the context.  For example, there are structs
in the Nix C++ source code that represent derivations [3].  In the Nix
expression language, derivations are represented as attribute sets [4].
There are also "store derivations", which are files in the store that
precisely encode the build actions that will be executed when the
derivation is built [5].

Some of this carries over to Guix.  The general meaning of the word
"derivation" appears to be the same as in Nix: a derivation basically
takes inputs and produces an output.  Guix also uses "store derivations"
because we use (a modified version of) the Nix daemon.  Note that the
Guix manual sometimes uses the word "derivation" to refer to "store
derivations", which can be confusing.

As you probably know, Guix eschews the Nix expression language in favor
of Guile for composing components.  Therefore, Guix has defined
abstractions in Guile to represent derivations [6].  Guix has also
defined additional abstractions to represent things like packages [7],
which are higher level abstractions than derivations.  There are a lot
of abstractions like this in Guix, and they interact with each other in
various ways.  To get an idea of what I mean, you can look at the
'(guix) Invoking guix graph' section in the Guix manual.  Derivations
are a general concept taken from Nix, but as far as I know, things like
bags and packages are Guix-specific concepts.

> - "Building" a derivation involves first ensuring that all of its inputs are
>   "built" (in the case of inputs that are derivations, this means ensuring its
>   output exists as a store item, and if not, producing it by this same
>   process. In the case of plain file inputs, this means just ensuring that 
> they
>   exist as a store item), and then preparing a build environment. The 
> "builder"
>   script is then executed in a chroot rooted under the build tree. After it 
> has
>   completed, the output (where is it put during the build process?) from
>   building the derivation is placed in the store (and registered).

Note that sometimes the word "build" is used to refer to the act of
creating a derivation, but not executing its build actions [8].

The process you've described sounds basically right to me; Ludo probably
knows more.  The Nix thesis might be out of date by now, but it
describes (described?) these kinds of things in pretty good detail [9].
It also describes the invariants that must be upheld by the entire
process.

> - "preparing a build environment" involves first creating a directory
>in which
>   the build will be run, then populating it with certain expected files and
>   directories (as detailed in 2.4.1 of the manual). Also, the store items
>   produced from building the inputs need to somehow be made available (is 
> there
>   a gnu/store/ under the build tree? I feel like I don't quite understand 
> where
>   everything goes in the build tree). Also, set any environment variables
>   associated with the derivation.

I believe that inputs and outputs are, from the a derivation's builder's
perspective, just paths in the store which can be accessed via
environment variables that Nix sets up during the build.  I think this
is the same in Guix as in Nix.  However, in Guix, I believe there are
additional ways to refer to inputs when defining a builder, e.g. by
using G-Expressions, which hide details like this from the user so that
the user can reason about builds in terms of higher-level abstractions.

As for how the store items produced from building the inputs are "made
available," my understanding is that the builder just needs to write
files to the special "output" path, which (I think) is in the store.
That special output path is made available to the builder during the
build via an environment variable (the variable is named "out" or
"output"; I can't remember exactly).  If you're using G-Expressions to
define a builder, the output path can be accessed via "#$output" (see
'(guix) G-Expressions' for details).

> Assuming I've got the gist of it sort of right, that leaves one more
>question:
> how are hashes for the paths computed? It's not sha256 like the database 
> hashes,
> it seems to be 160 bits in size. How are all the inputs hashed together? Are 
> all
> the inputs put in one big directory and that is hashed in nar form, or what? 
> And
> what about store items that don't have any inputs (for example, something 
> placed
> in there with add-text-to-store)? I suppose in that case they are their own
> input, perhaps?

The hashing mechanisms, the relevant invariants, and the motivation
behind these design decisions are described in the Nix thesis [10].  I
think some (maybe all) of this hashing stuff is implemented in Guix in
guix/derivations.scm (check the 'derivation' procedure) and
guix/store.scm (check the procedures that 'derivation' calls).

Hope that helps!  Good luck with your project.

Footnotes: 
[1]  See footnote 4 on p. 29 of Eelco Dolstra's Ph. D. thesis.

[2]  See p. 27 of the thesis.  There are alternate (and equally vague)
definitions in the Guix and Nix manuals.  They all have the same spirit:
a derivation is like a function from software components to software
components.

[3]  See Derivation in derivations.hh in the Nix source code.

[4]  The following blog entry explains this in detail:
https://lethalman.blogspot.co.uk/2014/07/nix-pill-6-our-first-derivation.html

[5]  See Section 2.4 "Store derivations" in the thesis for a great
explanation of store derivations, including why it is useful to
represent derivations in this way.

[6]  See the <derivation> record type in guix/derivations.scm.  Looks
like Guix also implements some of the actual logic for serializing a
<derivation> into a store derivation (see write-derivation in the same
module).

[7]  See <package> guix/packages.scm.

[8]  See for example the docstring for the 'derivation' procedure defined
in guix/derivations.scm, which claims to "build a derivation" when in
fact it only creates the store derivation without executing its build
actions.

[9]  See for example section 5.5 "Building store derivations", which
describes how store derivations are built in the extensional model (the
intensional model, discussed later in the thesis, has not been
implemented in Nix or Guix).  Figure 5.11, box #79 shows that the
builder is executed in a temporary directory; presumably the builder
writes the component's output directly into the output path (which is in
the store).  You can probably compare this to the actual build algorithm
in the C++ source code to figure out what's really going on.

[10] See sections 5.1 "Cryptographic hashes" and 5.2.2 "Store paths" in
the Nix thesis.  Various kinds of hashes are used in various places;
it's all pretty complicated, but the thesis describes much of the
details and motivation.  There may be other relevant sections, depending
on what exactly you want to know.  Of course, this thesis may be out of
date by now, so the C++ source code is the place to look to find actual
implementation.  But it's helpful to know the intent behind the code.

-- 
Chris

signature.asc
Description: PGP signature