[GSoC] Integrating npm into the Guix ecosystem

[Jelle Licht]

Greetings Guix hackers,

It has been some time since my last mail to this list, so I wanted to
share what I have been up to. For the people who might want to watch
along after today, I will be posting the changes that should not break
everything immediately to [1]. (Apologies for it being web-only, my
sysadminfu is still lacking. If you know how to easily and most of all
securely expose a git repo for read-only anonymous checkouts, please do
tell).

These first two weeks, I have mostly been keeping busy with writing some
of the features that will quickly allow me to identify problems with the
import and build procedures as I envision them. I was in luck; there
were many of these problems as I found out ;-).

One of the things that kind of worried me for a while was the fact that
npm was broken after my first attempts at writing _anything_. At this
moment, still being somehow intimidated by the guix code base, I had
quite some trouble finding out what was happening; it turns out
somewhere along the way, the way Node was built since version 6 (using
guix, at least) has changed, making the `npm' in the PATH of your guix
profile a _copy_ instead of a symlink to the actual executable
`npm-cli.js' script.

A patch to fix this is currently still brewing, so expect to see it
soon. In order to continue a bit (and implement something that actually
works), up till now I have worked with the 5.10 release of node.

After toiling a bit to find a nice way of getting node to find
dependencies, I started out with the NODE_PATH variable. Although some
alarming messages I read some time ago indicated this method of allowing
node to load modules is discouraged, but not deprecated.

At first I wanted to make use of a variant of the npm command to
actually take care of installing node modules to the store outputs; some
of the advantages include:

- The 'correct' files are ignored/always included
- The 'correct' symlinks are generated to executable scripts, man-pages
  and other documentation.
- Automatically up to date with new conventions in npm-land

While that all seemed (and still seems) quite nice, it happens to be the
case that npm is quite particular about dependencies that it will
accept. Patching the actual dependencies and versions in the
`package.json' file was another option, but after some consideration and
advice from my mentors I decided this would not work nicely in the long
term. Instead, I want to implement the relevant set of behaviors in
guix, of course allowing for our particular kind of dependency
management.

As such, the current installation phase mostly involves copying files to
their expected locations. I added support (as an exercise in build
system arguments) for 'global' installs[3], which as far as my current
bare-bones implementation is concerned means properly symlinking
executable scripts. In the (near-)future, I am looking into properly
wrapping these scripts to make sure the amount of propagated inputs is
kept to a minimum. As a sanity test, I packaged [4] in my own little
scratchpad. If you want to follow along, it only takes 4 `guix import
npm' invocations ;).

The current importer functions as expected for the "90%" of packages
that I tried. A problem that I ran into that I could not recognize as
easily in other importers is the fact that the npm community only
distributes the artifacts that you need to run the npm modules, but not
to build them. In most trivial cases, there are literally no
differences, but especially for more complicated packages involving
transpilation steps, this poses a problem.

As such, the importer can not actually 'know' of the location of the
source. Right now it uses some limited heuristics to probe GitHub
repositories for a tarball release, and if these are not found or the
sources are hosted at non-GH sites, it tries to check out a tag
according to the npm packaging conventions (SemVer).

The most important thing that needs to happen right now would be to
extend the range of packages that are buildable by the build system. A
combination of working towards having working 'large' packages and test
frameworks should help me quickly identify problems. This will be my
main focus for the next week.

I am aware that the current importer is quite brittle, so this needs to
change as well. As I run into problems (or someone brings them to my
attention), I will improve the quality of this subsystem as well.

If you have some advice, questions or problems, please do not hesitate
to reply. The same goes if you have some npm packages you would love to
see packaged.

Thanks a bunch for reading this WoT :)
- Jelle

[1]https://gitweb.jlicht.org/?p=guix.git;a=shortlog;h=refs/heads/gsoc-npm
[2]https://github.com/nodejs/node/issues/1627
[3]https://docs.npmjs.com/getting-started/installing-npm-packages-globally
[4]https://www.npmjs.com/package/package-json-validator