prades

Prades is a tool to embed large binary objects in npm packages, it works using S3 as store and a signing microservice in the npm registry.

We need to install private npm packages with large binaries. Npm has an internal limitation of ~250Mb for packages. There are some solutions to distribute binaries (like node-pre-gyp) but all of them assume the packages are public. In this scenario, AWS S3 is a good solution to distribute private binaries with signed URLs.

Prades rely on the existence of a signing request microservice in the registry server. For example https://registry-node.starbreeze.com has a /-/releases endpoint that signs request (if user is authorized) and redirects to this signed S3 url.

How to use

Configure package.json

package.json contains three relevant section for prades

where we specify details about the binary file and location

  "binary": {
    "file": "{package_name}/{package_version}/{node_abi}-{platform}-{arch}.tar.gz",
    "path": "bin",
    "host": "https://registry-node.starbreeze.com/-/releases"
  }

more on package.json's binary format options below.

the dependency to prades

  "dependencies": {
    "@sb/prades": "*"
  }

the instruction to run a prades install when installing the package

  "scripts": {
    "install": "prades install"
  }

When publishing a package

When (after or before doesn't matter) we publish an npm package that contains large binaries, then we have to upload the binaries to S3.

Prades has a helper to do that through the same mechanism of signed requests.

./node_modules/.bin/prades publish

When installing a package

When we install a package we don't have to do anything special. The install script in the package.json will run the prades install and it will download the binaries and put them in the specified target directory.

npm install has an strange behaviour. There is no way to tell when you run npm install to install dependencies to build and publish, or when npm is installing as part of a depencency. When you checkout a git repo and try to build a package you will encounter errors because prades install is run as part of npm install but the binaries packages do not exist yet.

To prevent this catch, prades install will do nothing if there exists a .git folder. This behaviour can be disabled using -f (--force) command line option.

package.json options

Matching rules for path option

The path package.json inside the binary section allows specifying what files have to be included in the binary tarball. It is often impractical to specify all source filepaths individually, so prades supports filename expansion.

Prades uses grunt file blobbing and matching. Check grunt globbing patterns documentation for exhaustive info.

This options accepts either comma separated globbing patterns or an array of globbing patterns. Paths matching patterns that begin with ! will be excluded from the returned array. Patterns are processed in order, so inclusion and exclusion order is significant.

While this isn't a comprehensive tutorial on globbing patterns, know that in a filepath:

* matches any number of characters, but not / ? matches a single character, but not / ** matches any number of characters, including /, as long as it's the only thing in a path part {} allows for a comma-separated list of "or" expressions ! at the beginning of a pattern will negate the match

.gitignore and .npmignore

Prades honors .gitignore or .npmignore files. .npmignore takes precedence, if it exists .gitignore is not used.

os and cpu options

There are two optional 'os' and 'cpu' properties in the binary section of the package.json that allow customizing the behavior for different platforms.

publish:

• works if there is no os or cpu
• works if the os and cpu are enabled
• gives an error if the os is NOT enabled
• gives an error if the cpu is NOT enabled

install:

• installs if there is no os or cpu
• installs if the os and cpu are enabled
• skips install if the os in NOT enabled
• skips install if the cpu in NOT enabled

example:

 "binary": {
   "file": "{package_name}/{package_version}/{platform}-{arch}.tar.gz",
   "path": "bin",
   "host": "https://registry-node.starbreeze.com/-/releases",
   "os": ["win32"],
   "cpu": ["x64"]
 }

os

The host operating system is determined by process.platform You can specify which operating systems prades binaries will be installed on: "os" : [ "darwin", "linux" ]

cpu

The cpu architecture is determined by process.arch If your code only runs on certain cpu architectures, you can specify which ones. "cpu" : [ "x64", "ia32" ]

Relation with npm publish

If npm package is not published

• prades can publish binaries

If npm package is published

• prades cannot publish binaries
• prades can force publish binaries ('-f, --force' option)
• prades cannot unpublish binaries
• prades can force unpublish binaries ('-f, --force' option)

prades info

Running prades info provides information about the published status.

If nothing is published

• it shows nothing is published

If npm package is published, but no prades binary is

• it shows npm info

If prades binary packages are published, but npm is not published

• shows prades info (information about published binaries), example:

[ { file: 'sb/prades_test_1/0.0.1/node_46-darwin-x64.tar.gz',
    last_modified: '2016-05-12T01:38:50.000Z',
    size: 373,
    md5: '030dd2b937ba1d1e9b51c6cb1b060c7c' },
  { file: 'sb/prades_test_1/0.0.1/node_46-win32-x64.tar.gz',
    last_modified: '2016-05-13T09:08:10.000Z',
    size: 340,
    md5: '10519f392dc80b5f45ba797803c8dd2f' },
  { file: 'sb/prades_test_1/0.0.1/node_47-linux-x64.tar.gz',
    last_modified: '2016-10-20T20:02:26.000Z',
    size: 375,
    md5: '9ead89ddc369fd683f06d46879ea40ef' } ]