Re: [PATCH] Prototype register-path

[Ludovic Courtès]

Hello!

Caleb Ristvedt <address@hidden> skribis:

> Well, I took a look at libstore/build.cc, and that is a lot of code. I
> hope I can get away with only reading as much as is necessary to clarify
> details. To that end, I'd like to make sure that my high-level
> understanding of the build process is accurate.
>
> - A derivation is a node in a (acyclic?) graph of "what depends on what" or,
>   alternately put, what is input to what.

Yes, a directed acyclic graph.

> - Derivations can depend on other derivations or plain files (can those files 
> be
>   directories?), where "plain files" cannot depend on anything else.

Right, “plain files” are referred to as “sources” in the code (e.g., the
‘sources’ field of <derivation>).  “Sources” are content-addressable
things added with ‘add-text-to-store’, ‘add-to-store’, and similar.

> - "Building" a derivation involves first ensuring that all of its inputs are
>   "built" (in the case of inputs that are derivations, this means ensuring its
>   output exists as a store item, and if not, producing it by this same
>   process. In the case of plain file inputs, this means just ensuring that 
> they
>   exist as a store item), and then preparing a build environment. The 
> "builder"
>   script is then executed in a chroot rooted under the build tree. After it 
> has
>   completed, the output (where is it put during the build process?) from
>   building the derivation is placed in the store (and registered).

Exactly.

> - "preparing a build environment" involves first creating a directory in which
>   the build will be run, then populating it with certain expected files and
>   directories (as detailed in 2.4.1 of the manual). Also, the store items
>   produced from building the inputs need to somehow be made available (is 
> there
>   a gnu/store/ under the build tree? I feel like I don't quite understand 
> where
>   everything goes in the build tree). Also, set any environment variables
>   associated with the derivation.

Yes.  In the build environment, there’s the subset of /gnu/store that is
explicitly declared as a dependency, directly or indirectly.  For
example, we can see it here:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix)
scheme@(guile-user)> ,enter-store-monad
store-monad@(guile-user) [1]> (gexp->derivation "contents"
                                                #~(begin
                                                    (use-modules (ice-9 ftw))
                                                    (call-with-output-file 
#$output
                                                      (lambda (port)
                                                        (write (scandir 
"/gnu/store") port)))))
$2 = #<derivation /gnu/store/8r9l9n9vv8vsnjy1b7w15bp8liz9bcvh-contents.drv => 
/gnu/store/a1z90bzb7xl6glp033jkxjc1n3w0lw7w-contents 41847d0>
store-monad@(guile-user) [1]> (built-derivations (list $2))
$3 = #t
store-monad@(guile-user) [1]> ,q
scheme@(guile-user)> ,pp (call-with-input-file (derivation->output-path $2) 
read)
$4 = ("."
 ".."
 "02426nwiy32cscm4h83729vn5ws1gs2i-bash-static-4.4.12"
 "3lsfrwlp1qa345x71yw5w49i2mpp0vxm-guile-2.0.14"
 "6czmriicly50bljbp6p34py5a63wzf05-contents-builder"
 "9f66pvz4vf1d9k8iir6asdp3l8k58cnn-libatomic-ops-7.4.4"
 "a1z90bzb7xl6glp033jkxjc1n3w0lw7w-contents"
 "b7w6bjp602qvhbsjs535dfax8v7wy8s8-gmp-6.1.2"
 "dhc2iy059hi91fk55dcv79z09kp6500y-gcc-5.4.0-lib"
 "dj9w9y66ncmn7qpnahz5fhqjjwqrnbjm-ncurses-6.0"
 "hvyk1qyph1hihfmym1w271ygp84adb0v-readline-7.0"
 "k7029k5va68lkapbzcycdzj7m5bjb4b8-bash-4.4.12"
 "kndl3vllk4bdq1xd3p8k67h8jrsq8xbq-readline-7.0"
 "lcmcm4c0zjv3sa9amdrhaszd7vwwxjh1-libltdl-2.4.6"
 "px46g18zg3sjgndwdcsgm6027w7s5gbc-pkg-config-0.29.1"
 "q1x4v3x8v2g59d244hl7k0i1n4h83c9a-ncurses-6.0"
 "rmjlycdgiq8pfy5hfi42qhw3k7p6kdav-glibc-2.25"
 "v4h4qw8a95479capaq08vs4zdyxdijhv-libunistring-0.9.7"
 "wqx8sxqjvz323vk9xalrhqk5f35yd42f-libffi-3.2.1"
 "yr7m8ldp3n40mrzjax91cj9hjw1k2a58-libgc-7.6.0")
--8<---------------cut here---------------end--------------->8---

What we see above is Guile and its dependencies, along with the output
and its build script.

> Assuming I've got the gist of it sort of right, that leaves one more question:
> how are hashes for the paths computed? It's not sha256 like the database 
> hashes,
> it seems to be 160 bits in size. How are all the inputs hashed together? Are 
> all
> the inputs put in one big directory and that is hashed in nar form, or what? 
> And
> what about store items that don't have any inputs (for example, something 
> placed
> in there with add-text-to-store)? I suppose in that case they are their own
> input, perhaps?

See (guix derivations) for how these hashes are computed, and also
‘store-path’, ‘output-path’, and ‘fixed-output-path’ in (guix store).
Normally you won’t have to reimplement any of these.

> Some of those answers might be in the manual, but I didn't manage to
> find them.

The manual doesn’t tell much about internal APIs.  :-)

HTH!

Ludo’.