Hash mismatch for Python 2.7.10 (x86_64-linux)

[Ludovic Courtès]

Leo Famulari <address@hidden> skribis:

> Can you take a look at python-2.7.10?
>
> Specifically, /gnu/store/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv-python-2.7.10
>
> This is from "ifur" on #guix:
> http://paste.lisp.org/display/311051

This is an “interesting” case.  There’s indeed a hash mismatch between
that narinfo and the nar (hydra.gnu.org and mirror.guixsd.org provide
the exact same narinfo and nar though):

--8<---------------cut here---------------start------------->8---
$ wget -q -O - https://hydra.gnu.org/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv.narinfo 
|grep Hash
NarHash: sha256:1ifhxz7ng64ckrsfp0cy6rcybwrdf2mgq7p9w73ynbwy1w2frhp5
$ wget -q -O - 
https://hydra.gnu.org/nar/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv-python-2.7.10 | 
bunzip2 | guix hash /dev/stdin
1rb34mfv32zj6ir9nkrmw2mrc00n2gf9p2chihh1qpl9fmsxhhif
$ guix hash -r /gnu/store/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv-python-2.7.10
1ifhxz7ng64ckrsfp0cy6rcybwrdf2mgq7p9w73ynbwy1w2frhp5
--8<---------------cut here---------------end--------------->8---

To spot the differences, I did:

--8<---------------cut here---------------start------------->8---
$ wget -O - 
https://mirror.guixsd.org/nar/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv-python-2.7.10 | 
bunzip2 | guix archive -x /tmp/python
$ diff -ur --no-dereference /tmp/python 
/gnu/store/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv-python-2.7.10
Binary files /tmp/python/lib/python2.7/_abcoll.pyc and 
/gnu/store/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv-python-2.7.10/lib/python2.7/_abcoll.pyc
 differ
Binary files /tmp/python/lib/python2.7/abc.pyc and 
/gnu/store/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv-python-2.7.10/lib/python2.7/abc.pyc 
differ
Binary files /tmp/python/lib/python2.7/codecs.pyc and 
/gnu/store/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv-python-2.7.10/lib/python2.7/codecs.pyc
 differ
Binary files /tmp/python/lib/python2.7/copy_reg.pyc and 
/gnu/store/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv-python-2.7.10/lib/python2.7/copy_reg.pyc
 differ
Binary files /tmp/python/lib/python2.7/encodings/aliases.pyc and 
/gnu/store/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv-python-2.7.10/lib/python2.7/encodings/aliases.pyc
 differ
Binary files /tmp/python/lib/python2.7/encodings/__init__.pyc and 
/gnu/store/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv-python-2.7.10/lib/python2.7/encodings/__init__.pyc
 differ
Binary files /tmp/python/lib/python2.7/encodings/utf_8.pyc and 
/gnu/store/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv-python-2.7.10/lib/python2.7/encodings/utf_8.pyc
 differ
[...]
$ diffoscope --html t.html /tmp/python/lib/python2.7 
/gnu/store/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv-python-2.7.10/lib/python2.7/
--8<---------------cut here---------------end--------------->8---

Diffoscope shows several hundred of bytes of differences in several .pyc
files (so it’s not just 32-byte timestamps as discussed in
<http://bugs.gnu.org/22010>, IIRC.)

That the differences are so important is fishy.  Could you check what
you have on your machine?  Is anyone able to make an independent build
of this thing?


For non-deterministic packages, something that could in theory happen
(although it seems unlikely) is this:

  1. nginx caches the narinfo of a first build with hash X;

  2. nobody downloads the corresponding nar, and the result is
     eventually GC’d from hydra;

  3. Hydra rebuilds the thing, obtains hash Y; the nar with hash Y is
     downloaded and cached by nginx, but, no luck, it doesn’t match the
     hash in the narinfo that was cached in step #1.

That could happen, but it seems pretty far-fetched, especially for a
package like Python.


On my machine I looked for the corresponding “derivers”:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> ,use(guix)
scheme@(guile-user)> (define s (open-connection))
scheme@(guile-user)> (valid-derivers s 
"/gnu/store/4bs4mx6xyx0jx9827vmmbxyhjln7cmcv-python-2.7.10")
$4 = ("/gnu/store/a5s3mkl387wwkklj3ss5hw34ib9rb2p6-python-2.7.10.drv" 
"/gnu/store/pvsdsjn15rj1ciwr37vl2vqprvl57szg-python-2.7.10.drv")
--8<---------------cut here---------------end--------------->8---

And then run:

--8<---------------cut here---------------start------------->8---
$ guix build "/gnu/store/pvsdsjn15rj1ciwr37vl2vqprvl57szg-python-2.7.10.drv" 
--check  
--8<---------------cut here---------------end--------------->8---

However that’s not very helpful since it unsurprisingly fails with a
“may not be deterministic” error.

Ludo’.