Re: Python and propagation

[Federico Beffa]

Ricardo Wurmus <address@hidden> writes:

> 1) print a warning when a collision is expected to happen, not when a
> collision has happened.

+1

> 2) avoid PYTHONPATH, patch all Python files invasively!
>
> Python does not have any feature that is comparable to RUNPATH.  It is
> only concerned with finding libraries/modules by *name* in one of the
> directories specified by the PYTHONPATH environment variable.
>
> But actually the PYTHONPATH variable is not the only means to affect the
> search path for modules.  It is possible to change the search path
> programmatically:
>
>     import sys
>     sys.path.append("/gnu/store/cabba9e...-numpy.../lib/...")
>     import numpy
>
> The first two lines add an explicit store item path to the search path;
> the third line just imports the numpy module by name (as usual).  Even
> without setting the PYTHONPATH to include the numpy in the profile the
> third line won’t fail.
>
> I wonder if we could design a phase that — very much like the
> “wrap-program” phase — modifies *every* Python file and injects lines
> like the first two in the example above, appending explicit store item
> paths, so that all dependent Python libraries can be found without the
> need to have them installed in the same profile and without the need to
> set PYTHONPATH.
>
> Maybe this is crazy, and maybe this causes other annoying problems, but
> I think it is the closest we can get to a RUNPATH-like feature in
> Python.
>
> What do you think?  Do I need a long vacation or is this something we
> might realistically do in an automated fashion?

Isn't the problem left of how the python interpreter can find those
patched libraries?

Maybe something along this lines could do:

* Instead of installing all files related to a module into profiles,
  only install a .pth file in 'site-packages' with a unique name. The
  name could even include the Guix package hash. This file should
  include the full path to the store ('real') package. This could maybe
  be achieved with 'python-build-system' creating two derivations: a
  derivation including only the .pth file and the 'real' package
  derivation (similar to a wrapped program pointing to the real one).

  For example

  $cd ~/tmp/ttmp
  $echo -e "aspell\nmechanics" > test.pth
  $cd
  $python
  >>> import sys
  >>> import site
  >>> site.addsitedir("/home/beffa/tmp/ttmp")
  >>> sys.path[-3:]
  ['/home/beffa/tmp/ttmp', '/home/beffa/tmp/ttmp/aspell',
  '/home/beffa/tmp/ttmp/mechanics']

* Then you can install many different versions of a package and they
  will not mask each other.

  To select a specific version we could then possibly use pkg_resources.

  For example:

  >>> import pkg_resources
  >>> pkg_resources.require("numpy==1.9.1")
  [numpy 1.9.1 
(/gnu/store/hbdzccnvnlf54mflgcigqw2jv4ybippv-profile/lib/python3.4/site-packages)]
  >>> import numpy

  If this always works (I'm not sure), we could patch this info into the
  packages at compile time.

Regards,
Fede