Non-bootstrappable NPM packages

[Timothy Sample]

Hi Guix,

(I’m CC’ing swedebugia and Jelle on this, since I promised both of them
off-list that I would publish this work eventually.)

Most of us know that the JavaScript situation with Guix is a little
dire.  Particularly, some JavaScript packages have circular dependencies
and most are not bootstrappable.  In practice, this has led to Guix
having almost no JavaScript packages, making it not suitable for
JavaScript-heavy Web development.  It also means that packaging
high-profile, Web-based programs like GitLab or Mastodon is so difficult
as to be impossible.

Given these obstacles, I thought it would be interesting to explore how
far you can get if you stop worrying about bootstrapping the packages,
and just use the pre-built ones.  It turns out that you can go as far as
you like this way.  In fact, I’ve built a recursive importer and
build-system for NPM that can import and “build” large and complicated
packages such as Babel.  It does this by downloading the pre-built
tarballs from NPM, and then installing them using the “npm” command with
some dependency rewriting.  It’s actually pretty simple. The main trick
was finding the right flags to pass to NPM, and figuring out how to
trick it into pulling dependencies from the store rather than the Web.

The code is at <https://git.ngyro.com/guix-npm>.

I know that this is not great for Guix proper, but I thought that this
might be a good use-case for channels.  In a channel we could build out
the JavaScript ecosystem in Guix on top of a bunch of “binary” packages,
and then try to reduce this set of “binary” packages gradually.  I think
that having a useful JavaScript ecosystem, even if it is problematic,
might help motivate folks to work out the problems.  For example, it
would be a lot more exciting to work on bootstrapping TypeScript if it
were the only thing standing in the way of getting a Mastodon service in
Guix.

That’s the high-level overview. The rest of this message is the
technical details of how to try out the importer, build packages, etc.

Here’s how to make “@babel/cli” work with a Babel plugin called
“transform-arrow-functions”.

Clone the repo and build it like you would build Guix normally, except
for where you would usually use “guix environment guix”.  Instead, use

    guix environment guix --ad-hoc -l guile-semver.scm

to ensure that the Semantic Versioning library required by the importer
is available.  After this, continue as usual.

Once built, create a new file at “gnu/packages/node-babel.scm” with the
following contents:

    (define-module (gnu packages node-babel)
      #:use-module (guix build-system other-node)
      #:use-module (guix download)
      #:use-module ((guix licenses) #:prefix license:)
      #:use-module (guix packages))

Then, you can run

    ./pre-inst-env guix import npm-binary -r @babel/cli \
        | tee -a gnu/packages/node-babel.scm

(using “tee” is nice ’cause you can watch the output as it goes – it has
a lot of dependencies, so it takes a little while).

Get the plugin, too:

    ./pre-inst-env guix import npm-binary -r \
        @babel/plugin-transform-arrow-functions \
        | tee -a gnu/packages/node-babel.scm


There’s a package in there that is macOS specific, so it has to be
removed.  It’s called “fsevents”, and it is dependency of “chokidar”.
Everything works fine if it is simply deleted.

At this point, you can launch an environment with the packages (be sure
to include “node”, as that reminds Guix to set “NODE_PATH”):

    ./pre-inst-env guix environment --ad-hoc node node-babel-cli--fiio \
        node-babel-plugin-transform-arrow-functions--fiio

(For the curious, “fiio” is an initialism for a phrase used in
Christopher Lemmer Webber’s article about JavaScript packaging:
<http://dustycloud.org/blog/javascript-packaging-dystopia/>.  It serves
as a reminder that these are not nice packages.)

Unfortunately, Babel only looks for plugins relative to the current
directory, so you’ll need a symlink: “ln -s $NODE_PATH”.

With all this in place, it’s time for a test!

    echo '() => { return true; };' > test.js
    babel --plugins=@babel/plugin-transform-arrow-functions test.js

This prints:

    (function () {
      return true;
    });

which is good, since it is transforming the modern lambda syntax with
the arrow “=>” into the old syntax with the “function” keyword.  Hooray!

Now for some more notes.

I’ve tried a handful of packages, and have had good results so far.
Sometimes things need a little fiddling (like “fsevents”), but the
results are usable.

Both the importer and the build system are mostly new code.

For the importer, the main difference is that it uses the NPM version
constraints to make sure the recursively imported packages have the
expected versions.  It does this via a Guile library that I wrote.  One
limitation is that the recursive importer does not check versions for
existing Guix packages.  If there is a package with the right name, it
is accepted even if it doesn’t satisfy the version constraints.  Guix’s
“recursive-import” function would have to be changed to fix this.

The build system uses “npm” for everything, rather than trying to
emulate “npm” in Scheme (like the recently added build system does).
I’m not sure if this is better or worse, really, but it works nicely for
this particular use-case.  It works in theory to build packages from
source, too.  One test I did was importing all of jQuery’s development
dependencies as binary packages, and then building jQuery from source.

I’ve come to think that bootstrapping JavaScript might be easier than it
looks.  As time goes on, Node gets better at the newer JavaScript
features.  This removes the need for things like Babel or Rollup, since
with some care, Node can run the source directly with out any
transformations or bundling.  That being said, TypeScript looks to be a
major issue, as it is used in many fundamental JavaScript packages and
it is not bootstrappable.

I’m not sure in what capacity I want to pursue this.  It’s been sitting
dormant on my computer for while, so I thought sharing it would be
better than letting it fall by the wayside.  I hope it proves useful one
way or another.

If you got this far, thanks for reading!  :)


-- Tim