npm

Thu, 07 Apr 2011 00:40:33 UTC - Isaac Schlueter - npm

npm 1.0 is in release candidate mode. Go get it!

In npm 0.x, there was a command called link. With it, you could “link-install” a package so that changes would be reflected in real-time. This is especially handy when you’re actually building something. You could make a few changes, run the command again, and voila, your new code would be run without having to re-install every time.

Of course, compiled modules still have to be rebuilt. That’s not ideal, but it’s a problem that will take more powerful magic to solve.

In npm 0.x, this was a pretty awful kludge. Back then, every package existed in some folder like:

prefix/lib/node/.npm/my-package/1.3.6/package

and the package’s version and name could be inferred from the path. Then, symbolic links were set up that looked like:

prefix/lib/node/my-package@1.3.6 -> ./.npm/my-package/1.3.6/package

It was easy enough to point that symlink to a different location. However, since the package.json file could change, that meant that the connection between the version and the folder was not reliable.

At first, this was just sort of something that we dealt with by saying, “Relink if you change the version.” However, as more and more edge cases arose, eventually the solution was to give link packages this fakey version of “9999.0.0-LINK-hash” so that npm knew it was an impostor. Sometimes the package was treated as if it had the 9999.0.0 version, and other times it was treated as if it had the version specified in the package.json.

A better way

For npm 1.0, we backed up and looked at what the actual use cases were. Most of the time when you link something you want one of the following:

  1. globally install this package I’m working on so that I can run the command it creates and test its stuff as I work on it.
  2. locally install my thing into some other thing that depends on it, so that the other thing can require() it.

And, in both cases, changes should be immediately apparent and not require any re-linking.

Also, there’s a third use case that I didn’t really appreciate until I started writing more programs that had more dependencies:

  1. Globally install something, and use it in development in a bunch of projects, and then update them all at once so that they all use the latest version.

Really, the second case above is a special-case of this third case.

The first step is to link your local project into the global install space. (See global vs local installation for more on this global/local business.)

I do this as I’m developing node projects (including npm itself).

cd ~/dev/js/node-tap  # go into the project dir
npm link              # create symlinks into {prefix}

Because of how I have my computer set up, with /usr/local as my install prefix, I end up with a symlink from /usr/local/lib/node_modules/tap pointing to ~/dev/js/node-tap, and the executable linked to /usr/local/bin/tap.

Of course, if you set your paths differently, then you’ll have different results. (That’s why I tend to talk in terms of prefix rather than /usr/local.)

When you want to link the globally-installed package into your local development folder, you run npm link pkg where pkg is the name of the package that you want to install.

For example, let’s say that I wanted to write some tap tests for my node-glob package. I’d first do the steps above to link tap into the global install space, and then I’d do this:

cd ~/dev/js/node-glob  # go to the project that uses the thing.
npm link tap           # link the global thing into my project.

Now when I make changes in ~/dev/js/node-tap, they’ll be immediately reflected in ~/dev/js/node-glob/node_modules/tap.

Let’s say I have 15 sites that all use express. I want the benefits of local development, but I also want to be able to update all my dev folders at once. You can globally install express, and then link it into your local development folder.

npm install express -g  # install express globally
cd ~/dev/js/my-blog     # development folder one
npm link express        # link the global express into ./node_modules
cd ~/dev/js/photo-site  # other project folder
npm link express        # link express into here, as well

                        # time passes
                        # TJ releases some new stuff.
                        # you want this new stuff.

npm update express -g   # update the global install.
                        # this also updates my project folders.

Caveat: Not For Real Servers

npm link is a development tool. It’s awesome for managing packages on your local development box. But deploying with npm link is basically asking for problems, since it makes it super easy to update things without realizing it.

Caveat 2: Sorry, Windows!

I highly doubt that a native Windows node will ever have comparable symbolic link support to what Unix systems provide. I know that there are junctions and such, and I've heard legends about symbolic links on Windows 7.

When there is a native windows port of Node, if that native windows port has fs.symlink and fs.readlink support that is exactly identical to the way that they work on Unix, then this should work fine.

But I wouldn't hold my breath. Any bugs about this not working on a native Windows system (ie, not Cygwin) will most likely be closed with wontfix.

Aside: Credit where Credit’s Due

Back before the Great Package Management Wars of Node 0.1, before npm or kiwi or mode or seed.js could do much of anything, and certainly before any of them had more than 2 users, Mikeal Rogers invited me to the Couch.io offices for lunch to talk about this npm registry thingie I’d mentioned wanting to build. (That is, to convince me to use CouchDB for it.)

Since he was volunteering to build the first version of it, and since couch is pretty much the ideal candidate for this use-case, it was an easy sell.

While I was there, he said, “Look. You need to be able to link a project directory as if it was installed as a package, and then have it all Just Work. Can you do that?”

I was like, “Well, I don’t know… I mean, there’s these edge cases, and it doesn’t really fit with the existing folder structure very well…”

“Dude. Either you do it, or I’m going to have to do it, and then there’ll be another package manager in node, instead of writing a registry for npm, and it won’t be as good anyway. Don’t be python.”

The rest is history.

Thu, 24 Mar 2011 06:07:13 UTC - Isaac Schlueter - npm

npm 1.0 is in release candidate mode. Go get it!

More than anything else, the driving force behind the npm 1.0 rearchitecture was the desire to simplify what a package installation directory structure looks like.

In npm 0.x, there was a command called bundle that a lot of people liked. bundle let you install your dependencies locally in your project, but even still, it was basically a hack that never really worked very reliably.

Also, there was that activation/deactivation thing. That’s confusing.

Two paths

In npm 1.0, there are two ways to install things:

  1. globally —- This drops modules in {prefix}/lib/node_modules, and puts executable files in {prefix}/bin, where {prefix} is usually something like /usr/local. It also installs man pages in {prefix}/share/man, if they’re supplied.
  2. locally —- This installs your package in the current working directory. Node modules go in ./node_modules, executables go in ./node_modules/.bin/, and man pages aren’t installed at all.

Which to choose

Whether to install a package globally or locally depends on the global config, which is aliased to the -g command line switch.

Just like how global variables are kind of gross, but also necessary in some cases, global packages are important, but best avoided if not needed.

In general, the rule of thumb is:

  1. If you’re installing something that you want to use in your program, using require('whatever'), then install it locally, at the root of your project.
  2. If you’re installing something that you want to use in your shell, on the command line or something, install it globally, so that its binaries end up in your PATH environment variable.

When you can't choose

Of course, there are some cases where you want to do both. Coffee-script and Express both are good examples of apps that have a command line interface, as well as a library. In those cases, you can do one of the following:

  1. Install it in both places. Seriously, are you that short on disk space? It’s fine, really. They’re tiny JavaScript programs.
  2. Install it globally, and then npm link coffee-script or npm link express (if you’re on a platform that supports symbolic links.) Then you only need to update the global copy to update all the symlinks as well.

The first option is the best in my opinion. Simple, clear, explicit. The second is really handy if you are going to re-use the same library in a bunch of different projects. (More on npm link in a future installment.)

You can probably think of other ways to do it by messing with environment variables. But I don’t recommend those ways. Go with the grain.

Slight exception: It’s not always the cwd.

Let’s say you do something like this:

cd ~/projects/foo     # go into my project
npm install express   # ./node_modules/express
cd lib/utils          # move around in there
vim some-thing.js     # edit some stuff, work work work
npm install redis     # ./lib/utils/node_modules/redis!? ew.

In this case, npm will install redis into ~/projects/foo/node_modules/redis. Sort of like how git will work anywhere within a git repository, npm will work anywhere within a package, defined by having a node_modules folder.

Test runners and stuff

If your package's scripts.test command uses a command-line program installed by one of your dependencies, not to worry. npm makes ./node_modules/.bin the first entry in the PATH environment variable when running any lifecycle scripts, so this will work fine, even if your program is not globally installed:

{ "name" : "my-program"
, "version" : "1.2.3"
, "dependencies": { "express": "*", "coffee-script": "*" }
, "devDependencies": { "vows": "*" }
, "scripts":
  { "test": "vows test/*.js"
  , "preinstall": "cake build" } }

Fri, 18 Mar 2011 06:22:17 UTC - Isaac Schlueter - npm

This is the first in a series of hopefully more than 1 posts, each detailing some aspect of npm 1.0.

In npm 0.x, the ls command was a combination of both searching the registry as well as reporting on what you have installed.

As the registry has grown in size, this has gotten unwieldy. Also, since npm 1.0 manages dependencies differently, nesting them in node_modules folder and installing locally by default, there are different things that you want to view.

The functionality of the ls command was split into two different parts. search is now the way to find things on the registry (and it only reports one line per package, instead of one line per version), and ls shows a tree view of the packages that are installed locally.

Here’s an example of the output:

$ npm ls
npm@1.0.0 /Users/isaacs/dev-src/js/npm
├── semver@1.0.1 
├─┬ ronn@0.3.5 
│ └── opts@1.2.1 
└─┬ express@2.0.0rc3 extraneous 
  ├─┬ connect@1.1.0 
  │ ├── qs@0.0.7 
  │ └── mime@1.2.1 
  ├── mime@1.2.1 
  └── qs@0.0.7

This is after I’ve done npm install semver ronn express in the npm source directory. Since express isn’t actually a dependency of npm, it shows up with that “extraneous” marker.

Let’s see what happens when we create a broken situation:

$ rm -rf ./node_modules/express/node_modules/connect
$ npm ls
npm@1.0.0 /Users/isaacs/dev-src/js/npm
├── semver@1.0.1 
├─┬ ronn@0.3.5 
│ └── opts@1.2.1 
└─┬ express@2.0.0rc3 extraneous 
  ├── UNMET DEPENDENCY connect >= 1.1.0 < 2.0.0
  ├── mime@1.2.1 
  └── qs@0.0.7

Tree views are great for human readability, but some times you want to pipe that stuff to another program. For that output, I took the same datastructure, but instead of building up a treeview string for each line, it spits out just the folders like this:

$ npm ls -p
/Users/isaacs/dev-src/js/npm
/Users/isaacs/dev-src/js/npm/node_modules/semver
/Users/isaacs/dev-src/js/npm/node_modules/ronn
/Users/isaacs/dev-src/js/npm/node_modules/ronn/node_modules/opts
/Users/isaacs/dev-src/js/npm/node_modules/express
/Users/isaacs/dev-src/js/npm/node_modules/express/node_modules/connect
/Users/isaacs/dev-src/js/npm/node_modules/express/node_modules/connect/node_modules/qs
/Users/isaacs/dev-src/js/npm/node_modules/express/node_modules/connect/node_modules/mime
/Users/isaacs/dev-src/js/npm/node_modules/express/node_modules/mime
/Users/isaacs/dev-src/js/npm/node_modules/express/node_modules/qs

Since you sometimes want a bigger view, I added the --long option to (shorthand: -l) to spit out more info:

$ npm ls -l
npm@1.0.0 
│ /Users/isaacs/dev-src/js/npm
│ A package manager for node
│ git://github.com/isaacs/npm.git
│ http://npmjs.org/
├── semver@1.0.1 
│   ./node_modules/semver
│   The semantic version parser used by npm.
│   git://github.com/isaacs/node-semver.git
├─┬ ronn@0.3.5 
│ │ ./node_modules/ronn
│ │ markdown to roff and html converter
│ └── opts@1.2.1 
│     ./node_modules/ronn/node_modules/opts
│     Command line argument parser written in the style of commonjs. To be used with node.js
└─┬ express@2.0.0rc3 extraneous 
  │ ./node_modules/express
  │ Sinatra inspired web development framework
  ├─┬ connect@1.1.0 
  │ │ ./node_modules/express/node_modules/connect
  │ │ High performance middleware framework
  │ │ git://github.com/senchalabs/connect.git
  │ ├── qs@0.0.7 
  │ │   ./node_modules/express/node_modules/connect/node_modules/qs
  │ │   querystring parser
  │ └── mime@1.2.1 
  │     ./node_modules/express/node_modules/connect/node_modules/mime
  │     A comprehensive library for mime-type mapping
  ├── mime@1.2.1 
  │   ./node_modules/express/node_modules/mime
  │   A comprehensive library for mime-type mapping
  └── qs@0.0.7 
      ./node_modules/express/node_modules/qs
      querystring parser

$ npm ls -lp
/Users/isaacs/dev-src/js/npm:npm@1.0.0::::
/Users/isaacs/dev-src/js/npm/node_modules/semver:semver@1.0.1::::
/Users/isaacs/dev-src/js/npm/node_modules/ronn:ronn@0.3.5::::
/Users/isaacs/dev-src/js/npm/node_modules/ronn/node_modules/opts:opts@1.2.1::::
/Users/isaacs/dev-src/js/npm/node_modules/express:express@2.0.0rc3:EXTRANEOUS:::
/Users/isaacs/dev-src/js/npm/node_modules/express/node_modules/connect:connect@1.1.0::::
/Users/isaacs/dev-src/js/npm/node_modules/express/node_modules/connect/node_modules/qs:qs@0.0.7::::
/Users/isaacs/dev-src/js/npm/node_modules/express/node_modules/connect/node_modules/mime:mime@1.2.1::::
/Users/isaacs/dev-src/js/npm/node_modules/express/node_modules/mime:mime@1.2.1::::
/Users/isaacs/dev-src/js/npm/node_modules/express/node_modules/qs:qs@0.0.7::::

And, if you want to get at the globally-installed modules, you can use ls with the global flag:

$ npm ls -g
/usr/local
├─┬ A@1.2.3 -> /Users/isaacs/dev-src/js/A
│ ├── B@1.2.3 -> /Users/isaacs/dev-src/js/B
│ └─┬ npm@0.3.15 
│   └── semver@1.0.1 
├─┬ B@1.2.3 -> /Users/isaacs/dev-src/js/B
│ └── A@1.2.3 -> /Users/isaacs/dev-src/js/A
├── glob@2.0.5 
├─┬ npm@1.0.0 -> /Users/isaacs/dev-src/js/npm
│ ├── semver@1.0.1 
│ └─┬ ronn@0.3.5 
│   └── opts@1.2.1 
└── supervisor@0.1.2 -> /Users/isaacs/dev-src/js/node-supervisor

$ npm ls -gpl
/usr/local:::::
/usr/local/lib/node_modules/A:A@1.2.3::::/Users/isaacs/dev-src/js/A
/usr/local/lib/node_modules/A/node_modules/npm:npm@0.3.15::::/Users/isaacs/dev-src/js/A/node_modules/npm
/usr/local/lib/node_modules/A/node_modules/npm/node_modules/semver:semver@1.0.1::::/Users/isaacs/dev-src/js/A/node_modules/npm/node_modules/semver
/usr/local/lib/node_modules/B:B@1.2.3::::/Users/isaacs/dev-src/js/B
/usr/local/lib/node_modules/glob:glob@2.0.5::::
/usr/local/lib/node_modules/npm:npm@1.0.0::::/Users/isaacs/dev-src/js/npm
/usr/local/lib/node_modules/npm/node_modules/semver:semver@1.0.1::::/Users/isaacs/dev-src/js/npm/node_modules/semver
/usr/local/lib/node_modules/npm/node_modules/ronn:ronn@0.3.5::::/Users/isaacs/dev-src/js/npm/node_modules/ronn
/usr/local/lib/node_modules/npm/node_modules/ronn/node_modules/opts:opts@1.2.1::::/Users/isaacs/dev-src/js/npm/node_modules/ronn/node_modules/opts
/usr/local/lib/node_modules/supervisor:supervisor@0.1.2::::/Users/isaacs/dev-src/js/node-supervisor

Those -> flags are indications that the package is link-installed, which will be covered in the next installment.

← Page 1