Re: Proposal: prefetch tarballs in a batch

[Ludovic Courtès]

Nikita Karetnikov <address@hidden> skribis:

> 1. It doesn’t seem to prefetch all the needed dependencies.  ‘guix build
>    hello’ (without network access) fails after prefetching the said
>    package.

Fails how?

> 2. The substituter fails from time to time.  Note that eight tests fail
>    on the machine I used: ‘builders.scm’, ‘utils.scm’, ‘packages.scm’,
>    ‘store.scm’, ‘monads.scm’, ‘gexp.scm’, ‘guix-package.sh’,
>    ‘guix-register.sh’.  Perhaps we ought to fix the mentioned failures
>    first.  Which log files would you like to see?

The SRFI-64 log files of the failing tests, plus test-suite.log.  Test
failures must not remain uncorrected!  ;-)

When there are so many failures, it’s likely that there’s a setup issue,
like socket names are too long.

>    $ ./pre-inst-env guix prefetch -n icecat
>    substitute-binary: guile: hashtab.c:137: vacuum_weak_hash_table: Assertion 
> `removed <= len' failed.

What Guile and libgc version is this, and what platform?

>    I’ve also seen this one.  In case it matters, that was before running
>    ‘chgrp 1001 /gnu/store; chmod 1775 /gnu/store’.
>
>    $ ./pre-inst-env guix prefetch -n gnunet
>    Backtrace:
>    In ice-9/boot-9.scm:
>     157: 17 [catch #t #<catch-closure 9319ef0> ...]

[...]

>    In guix/derivations.scm:
>     175: 3 [derivation-prerequisites-to-build # # # ...]
>    In guix/store.scm:
>     695: 2 [substitutable-paths # #]
>     392: 1 [process-stderr #<build-daemon 256.14 9a29660> #f]
>    In guix/serialization.scm:
>      51: 0 [read-int #<input-output: socket 11>]
>
>    guix/serialization.scm:51:4: In procedure read-int:
>    guix/serialization.scm:51:4: In procedure bv-u32-ref: Wrong type argument 
> in position 1 (expecting bytevector): #<eof>

That seems similar no?

Does the installation seem sane, basically?  Do ‘guix build’, ‘guix
package’ etc. work somehow, or not even?

> 3. When using the substituter, the command takes much more time.  Do
>    we even need it in this case?  I seem to recall that the GNUnet
>    tarball was served by Hydra, but I forgot the details.

There’s a local cache of substituter hits/failures in
/var/guix/substitute-binary (or similar.)  When that cache is empty or
outdated, a lot of HTTP queries are made to hydra.gnu.org, which can
take a bit of time.

When using a recent Guile, these queries are made in parallel.  On my
machine, it goes like this:

--8<---------------cut here---------------start------------->8---
$ sudo rm -rf /var/guix/substitute-binary/cache/

$ time guix build emacs -n

real    0m30.551s
user    0m2.519s
sys     0m0.154s

$ time guix build emacs -n

real    0m3.381s
user    0m2.039s
sys     0m0.123s
--8<---------------cut here---------------end--------------->8---

The cost is high when starting from an empty cache, but afterwards it’s
small.

> +(define (show-help)

Add call to ‘show-build-options-help’ from (guix scripts build).

> +(define %options
> +  ;; Specification of the command-line options.
> +  (list (option '("no-substitutes") #f #f
> +                (lambda (opt name arg result . rest)
> +                  (apply values
> +                         (alist-cons 'substitutes? #f
> +                                     (alist-delete 'substitutes? result))
> +                         rest)))

Remove this option, and concatenate with ‘%standard-build-options’.

> +                       (filter-map (match-lambda
> +                                    (('argument . value)
> +                                     (identity  ; discard the second value

The extra value gets truncated here, so it’s not strictly needed.

Thanks!

Ludo’.