bug#41669: Cross-compiled powerpc64-linux bootstrap-tarballs not reproducible

[Chris Marusich]

Hi Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> You can update (gnu packages bootstrap) accordingly.

Thank you for uploading the little-endian bootstrap binaries!  I've
downloaded them, and I confirm that they are identical to the ones I
built.  Do you also plan to upload the big-endian bootstrap binaries?  I
think we were hoping to try both system types.

I'm afraid I've hit a snag using what you've uploaded, though.  It looks
like we'll need to extract bash, mkdir, tar, and xz from
static-binaries-0-powerpc64le-linux-gnu.tar.xz and place a copy of each
in the following locations:

- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/bash
- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/mkdir
- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/tar
- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/xz

Could you do that?  The reason why it's necessary is described below.

I've started making changes locally to (gnu packages bootstrap).  So
far, with the attached patch, I'm able to do the following things on my
POWER9 machine running ppc64le Debian (unstable).

I can successfully build Guix from source using Debian packages (and a
manually-built copy of guile-avahi and guile-gnutls).  With my patch, I
still have to supply the "--with-courage" configure option.  Although
"make" succeeded, "make check" failed on the following tests:

- tests/build-utils.scm
- test/challenge.scm
- tests/containers.scm
- tests/debug-link.scm

The failures fell into two categories:

- Some tests couldn't download the bootstrap bash.
- In tests/containers.scm, call-with-container evaluated to #f when it
  wasn't supposed to.

Nevertheless, I created the necessary build users and started the
guix-daemon via pre-inst-env.  I then tried building a simple package:

  ./pre-inst-env guix build -e '(@@ (gnu packages bootstrap) 
%bootstrap-coreutils&co)'

The result was promising, but it quickly failed like some of the tests
did - it couldn't download the bash bootstrap binary:

--8<---------------cut here---------------start------------->8---
marusich@suzaku:~/repos/guix$ ./pre-inst-env guix build -e '(@@ (gnu packages 
bootstrap) %bootstrap-coreutils&co)'
substitute: guix substitute: warning: ACL for archive imports seems to be 
uninitialized, substitutes may be unavailable
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/y9868ab6a4wjcvwzj4ln5fzk1y4y6zpz-bootstrap-binaries-0.drv
   /gnu/store/2nwml4l272qzq034hwf5icv9vxi813ja-xz.drv
   /gnu/store/c1v9lmsh0awbhpai72mzw4qv02rrbpw6-module-import-compiled.drv
   /gnu/store/n5hf44bybvqmsybjvnn61pkpmcdcrlbd-guile-bootstrap-2.0.drv
   /gnu/store/ix4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv
   /gnu/store/jxh9xn77flxarwzcjga485pgrkjknrgb-tar.drv
   
/gnu/store/yd1mib8s1f38qwdn61zj16ijx8p0ryzm-guile-static-stripped-2.0.14-powerpc64le-linux-gnu.tar.xz.drv
   /gnu/store/z66wc9z4qvffn60q4jdx7in6rxpswhx3-mkdir.drv
   
/gnu/store/m4qv668s851v2ndzns3xwzg5rga9fhff-static-binaries-0-powerpc64le-linux-gnu.tar.xz.drv
building 
/gnu/store/m4qv668s851v2ndzns3xwzg5rga9fhff-static-binaries-0-powerpc64le-linux-gnu.tar.xz.drv...

Starting download of 
/gnu/store/2phdifnfw6i989rqbav04zakxx7qb165-static-binaries-0-powerpc64le-linux-gnu.tar.xz
From 
https://ftp.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/static-binaries-0-powerpc64le-linux-gnu.tar.xz...
download failed 
"https://ftp.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/static-binaries-0-powerpc64le-linux-gnu.tar.xz";
 404 "Not Found"

Starting download of 
/gnu/store/2phdifnfw6i989rqbav04zakxx7qb165-static-binaries-0-powerpc64le-linux-gnu.tar.xz
From 
https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/static-binaries-0-powerpc64le-linux-gnu.tar.xz...
downloading from 
https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/static-binaries-0-powerpc64le-linux-gnu.tar.xz
 ...
 static-binaries-0-powerpc64le-linux-gnu.tar.xz  4.4MiB 2.8MiB/s 00:02 
[##################] 100.0%
successfully built 
/gnu/store/m4qv668s851v2ndzns3xwzg5rga9fhff-static-binaries-0-powerpc64le-linux-gnu.tar.xz.drv
building /gnu/store/ix4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv...

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From 
https://git.savannah.gnu.org/cgit/guix.git/plain/gnu/packages/bootstrap/powerpc64le-linux/20210106/bash...
download failed 
"https://git.savannah.gnu.org/cgit/guix.git/plain/gnu/packages/bootstrap/powerpc64le-linux/20210106/bash";
 404 "Not found"

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From http://lilypond.org/janneke/guix/powerpc64le-linux/20210106/bash...
download failed 
"http://lilypond.org/janneke/guix/powerpc64le-linux/20210106/bash"; 404 "Not 
Found"

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From 
https://ci.guix.gnu.org/file/bash/sha256/1kiw7n6mkdy2x9in97646nb7aiayxr090ws1hbrlazah3fjqi6nj...
download failed 
"https://ci.guix.gnu.org/file/bash/sha256/1kiw7n6mkdy2x9in97646nb7aiayxr090ws1hbrlazah3fjqi6nj";
 404 "Not Found"

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From 
https://tarballs.nixos.org/sha256/1kiw7n6mkdy2x9in97646nb7aiayxr090ws1hbrlazah3fjqi6nj...
download failed 
"https://tarballs.nixos.org/sha256/1kiw7n6mkdy2x9in97646nb7aiayxr090ws1hbrlazah3fjqi6nj";
 404 "Not Found"

Starting download of /gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash
From 
https://archive.softwareheritage.org/api/1/content/sha256:d29a88a51b507d45f38241739040ee5e45759635c49c6463eac2b7598d3d3cce/raw/...
download failed 
"https://archive.softwareheritage.org/api/1/content/sha256:d29a88a51b507d45f38241739040ee5e45759635c49c6463eac2b7598d3d3cce/raw/";
 404 "Not Found"
failed to download "/gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash" from 
("https://git.savannah.gnu.org/cgit/guix.git/plain/gnu/packages/bootstrap/powerpc64le-linux/20210106/bash";
 "http://lilypond.org/janneke/guix/powerpc64le-linux/20210106/bash";)
builder for `/gnu/store/ix4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv' failed to 
produce output path `/gnu/store/0kaj6l1ccw0qd0289hii7qhr828s71sv-bash'
build of /gnu/store/ix4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv failed
View build log at 
'/var/log/guix/drvs/ix/4mpvzxfi1hrmvdmmpgqhm9x1cdh347-bash.drv.gz'.
cannot build derivation 
`/gnu/store/n5hf44bybvqmsybjvnn61pkpmcdcrlbd-guile-bootstrap-2.0.drv': 1 
dependencies couldn't be built
building /gnu/store/jxh9xn77flxarwzcjga485pgrkjknrgb-tar.drv...
cannot build derivation 
`/gnu/store/y9868ab6a4wjcvwzj4ln5fzk1y4y6zpz-bootstrap-binaries-0.drv': 1 
dependencies couldn't be built
guix build: error: build of 
`/gnu/store/y9868ab6a4wjcvwzj4ln5fzk1y4y6zpz-bootstrap-binaries-0.drv' failed
--8<---------------cut here---------------end--------------->8---

To resolve this issue, it seems that we must also extract the bootstrap
bash, mkdir, tar, and xz from
static-binaries-0-powerpc64le-linux-gnu.tar.xz and place a copy of each
in the following locations:

- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/bash
- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/mkdir
- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/tar
- https://alpha.gnu.org/gnu/guix/bootstrap/powerpc64le-linux/20210106/xz

After you've uploaded these files, I'll also need to adjust my patch so
it adds alpha.gnu.org to %bootstrap-executable-base-urls in
gnu/packages/bootstrap.scm.  That's how Guix finds these raw, unpackaged
binaries.

By the way, I've noticed that the other architectures don't seem to have
"raw" binaries at all on alpha.gnu.org.  Maybe you already knew this,
but it seems that the "raw" binaries are actually downloaded from this
specific URL (see the bootstrap-executable-file-name procedure):

https://git.savannah.gnu.org/cgit/guix.git/plain/gnu/packages/bootstrap/?id=44f07d1dc6806e97c4e9ee3e6be883cc59dc666e

Is that intended?  I was surprised to discover that we store these four
"raw" binaries in a totally separate place.  That seems like it would
make it easy for someone to accidentally forget to update the "raw"
binaries when they update an architecture's bootstrap tarballs.

In any case, for powerpc64le-linux, the right thing to do is probably
just to add copies of these "raw" binaries like I mentioned above.
Could you do that when you have the time?

Thank you,

-- 
Chris

0001-gnu-bootstrap-Add-support-for-powerpc64le-linux.patch
Description: Text Data

signature.asc
Description: PGP signature