Improved NPM importer with blacklist (Was: Re: NPM importer)

[swedebugia]

Hi :)

On 2018-11-11 16:37, Julien Lepiller wrote:
> I improved a bit over jlicht's work here, but there still a few tgings we want 
> co work on: https://framagit.org/tyreunom/guix/tree/npm
>
> There is an importer and a build system as well as a few packages. One of tge 
> issue is that the importer is not recursive, so it doesn't get the specified 
> version, and the packages aren't tested because the tests depend on cylic 
> dependencies (and sometimes very big circles).

There is a stub of a recursive importer, but it does not seem to work.

--

I improved on Julliens work and added a blacklister. After much sweat it 
now works and the blacklist is populated with a lot of unneeded 
development dependencies and complex packages.

I added the version to all imported npm-packages and to the inputs as it 
does not make sense to reference the packages without a version given 
all the cyclic dependencies.

Thanks for all the help!

TODO:
* make npm-recursive-import work by not fetching blacklisted packages

* implement keyword blacklisting based on the descriptions

* match not just the whole string of blacklisted packages:
  e.g. match also "rollup-plugin" when "rollup" is in the blacklist.

* get the tarballs from npm-registry instead as they are never missing
  (githubs sometimes are) and likely reproducible.

* Output a (define-public <guixname> (inherit <guixname>-<version>)) for
  all imported npm-packages.

* Make it possible to specify a specific version to import (and perhaps
  the latest of all minor versions of a package :D).
(For async that would be "0.1.22", "0.2.10", "0.3.0", etc all the way up 
to "2.6.1" which is the current beast. This would mean that we in total 
import about 477.000 packages times the number of minor releases (mean 
~10?) that equals 4,7 mio. npm-packages :p) Then we will definitely need 
to speed up guile. My guess is that we will have to import at least 1,5 
versions for every npm package to mitigate cyclic dependencies (this 
means 477.000*1,5 = 715.500 npm-package-versions).

* Make it easy to analyze a given npm-package to see when deps/devdeps 
were added. In the case async, propose we import 0.9.0 first which is 
the last version without lodash as devdep. From 1.0.0 more devdeps were 
added. (source: https://registry.npmjs.org/async)

Perhaps some kind of tree output for these complex packages with 
versions as branches and dependencies as subbranches would be nice?
--

See the files I changed attached.

--
Cheers
Swedebugia

npm.scm.bz2
Description: application/bzip

npm-blacklist.scm
Description: Text Data