Re: Prototype for using Guix with python packages

[Christopher Baines]

On 07/09/16 15:01, Ludovic Courtès wrote:
>> For a few months now, I've been assembling a prototype for how packages
>> could be produced for software released as Python source distributions
>> (sdists) [1].
> 
> Woow, quite an achievement!  Do you know how many of the automatically
> generated packages built from source flawlessly?

Not exactly, I disabled quite a few test suites, and made some
requirements stricter in some cases among other things.

> IIRC, ‘guix import pypi’ currently produces templates that can require
> extra tweaks, although that was improved by reading metadata from
> Wheels.
> 
> How does sdist metadata differ from PyPI or Wheels metadata?  Is it
> generally more complete, or of better quality?

sdists normally contain a .egg-info directory (e.g. sentry.egg-info),
which contains a requires.txt file, which as far as I can tell contains
the same information as the metadata.json in a wheel.

I use that, as it means I can get the data without having to rely on the
existence of a wheel.

Its still really poor quality though, as it does not include
dependencies required to run the tests, nor build system dependencies
(e.g. setuptools or otherwise) which may be found in a different file [1].

1: https://www.python.org/dev/peps/pep-0518/

>> I will also try to look at how what I have been doing could be used to
>> improve the Guix PyPI importer and the Python build system, as well as
>> submit the packages that are currently just in the guix-env.scm file in
>> the repository (pyguile, libsolv) for inclusion in Guix.
> 
> That would be awesome!  Anything towards making the output of ‘guix
> import’ work out of the box would be great.

I think the next steps towards this would be:
 - Switch from downloading wheels to just using the requires.txt in the
.egg-info directory
 - Add support for https://www.python.org/dev/peps/pep-0518/
 - Identify when PBR (Python Build Reasonableness) is in use, as it may
mean that the test requirements are available (in test-requirements.txt)

In my attempt to do this in a very automated way, I've only done the
first point, but I build the package and parse the build log for signs
of missing dependencies if the build fails, and then repeat the build.
This is done first with the tests disabled, and then if it eventually
builds, again with the tests enabled. I'm not sure if you would want
guix import to do this?

>> I'm quite excited by the possibilities offered by approaches like this,
>> and was wondering if anyone has feedback, questions, opinions, or if
>> anyone is working on something similar?
> 
> That reminds me of the “recursive importer” that has been discussed a
> few times and is currently implemented (but not in master yet) for the
> CRAN and NPM importers.  Perhaps your work could be somehow integrated
> to get a recursive PyPI/sdist importer?
> 
> An idea would be to automatically put all the imported packages into,
> say, ~/.cache/guix/python, and add that to GUIX_PACKAGE_PATH.  Then we
> could run “guix import pypi --update” to update it, or something like
> that.

The main reason why I didn't just improve the importer is that I was
looking for a way to collaborate around getting multiple versions of
python packages building and working, and as far as I am aware, Guix
only contains multiple versions of the same piece of software in some
special cases?

I have implemented a recursive importer, but, I'm not trying to generate
Guix package records, but instead records describing the versioned
requirements, as from this, build graphs can be generated. I don't like
this approach in that it duplicates a few packages in the Guix
repository, but having multiple versions can be really useful in some
circumstances.

Thanks for your feedback Ludo, and it would be really interesting to
hear your thoughts on the multiple version issue.

signature.asc
Description: OpenPGP digital signature