Re: jquery 3.1.1

[Jelle Licht]

Hello Catonano,

2017-01-19 21:48 GMT+01:00 Catonano <address@hidden>:

I made a crawler and I let it loose on the jquery 3.1.1 dependencies on registry.npmjs.com

It recursevely fetched the dependencies of jquery 3.1.1, then the dependencies of the dependencies, then the dependencies of the dependencies of the dependencies... and so on

Until there were no more dependencies to fetch.

It stored all such dependencies in a graph database made by the amazing amz3 for their project, Culturia. They wrote about such project several times on the guile-users mailing list.

It took days to download. And it took me months to produce a working version of this code. Because I don't know Guile very well and maybe I'm not that smart overall.

Anyway, now I have a COMPLETE graph of the dependencies of jquery 3.1.1

These pictures are very informative indeed. I will try to be brief, but I quickly wanted to share
that I find your efforts (and results!) amazing.
 

It's made of

47311 vertices and

324569 edges

I made a graph of a subset (graphviz chocked a bit on this ;-) )

It's here
http://catonano.altervista.org/grafo.svg

There are 448 "broken" packages. Probably I should explain how I classified packages as "broken". Maybe not in this email.

Anyway, these broken packages pose a challenge to the mission of porting Jquery into Guix, in my opinion, and they should be considered with some attention.

Here's the list
http://catonano.altervista.org/broken_packages.txt

 

Something of note: a big chunk of the packages you classified as being 'broken' are

making me recall some (unpleasant) memories; From my own crawling experiments,

which were not nearly as complete as this one, I also ran into a lot of the same

show-stoppers. In a very real sense, resolving node dependencies quickly devolves into

resolving dependencies for most of the popular build systems as well as plugins like
broccoli-funnel. What I am trying to get at here is that fixing and packaging these 448 packages

will likely contribute a lot towards making an npm-enriched guix possible.
 

There are 1314 packages with NO dependencies that could be used as starting points in porting Jquery into Guix.

Here's the list
http://catonano.altervista.org/broken_packages.txt

These could probably make use of the npm importer I worked on earlier. Do you make use of your own? Otherwase I'll get to

rebasing my version on the current guix master branch.
 

If there's anyone interested, I can give you the data folder so you can try all the queries you want on these data without having to to run this thing for a bunch of hours

If possible, yes please. What would be the most convenient way you to share this data?
 

In the future, I'd like to run this thing on some other package and merge the graphs so I will be able to investigate which are the common fundamental dependencies for SEVERAL important packages in Nodejs.

So if someone wants to dedicate time to porting Nodejs stuff in Guix they will be able to select most urgent packages to start from.

The same could be said of broken packages taht affect several important packages.

The porting of Nodejs in Guix cannot be done with brute strength. A data oriented approach can help, in my opinion.

Indeed.

 

The ideal would be to have something that, like bitcoin, coordinates a swarm in such a way that every node can contribute a tiniy bit of data to a common data structure, so all the nodes would have a complete copy of the database.

Collecting a mantaining of datasets should be freed of the client server model too. Not only the social media.

I have no idea what you are referring to. Could you please elaborate a bit at a later point in time?

 

But that's more than I can handle, anyway.

I am already thankful

I'd like to talk about the stumbling blocks I run into to discuss Guile and my knowledge of it.

For example, I can't use that thing in the autotools that processes configure.am files so I just forked amz3's project and added my files in there. As guests. Thanks amz3 !

I'd also like to describe te screw ups in the format I put the data into. I realized my mistakes when hours of crunching had already been done.

They can be migrated to a better format though

If you don't mind, I will discuss these issue in the future, not now.

The code is here
https://gitlab.com/humanitiesNerd/Culturia

One last fun fact: while I was watching the output flowing in my terminal, I saw a package called

"broccoli-funnel"

No, really. It's here
https://registry.npmjs.org/broccoli-funnel

Ok, that's all for now.

Thanks again for your efforts on this. I am looking forward to working with your data.

Regards,

Jelle Licht