dynamic-module-loader

1.0.8 • Public • Published

dynamic-module-loader Build Status

The dynamic module loader library allows code to retrieve Node modules from a web server, install them locally and serve them up as though they'd been manually deployed to the running server. This allows you to retrieve content from remote sources when necessary and manage updates dynamically. Internally the dynamic module loader uses cluster-lock to ensure that only one download request happens at any time for a single module. In other words, if you have a cluster on a single machine and multiple processes request the same module at the same time, only one will download and install the module. The others will wait until the process is complete before serving the module up from its local location.

The Packages

Packages must adhere to the following structure:

package-dir
|
+- package.json

The Javascript files in the package may live anywhere, in any directory. You need only be able to require them. The rule is that you should follow the same rules for a dynamic module as you would for a Node.js library module.

The package.json file must contain the following:

{
    "main":"<path to main module"
}

If you use other Node.js libraries, you must include the standard NPM dependencies blocks. The actual libraries shown below are for illustrative purposes only; you can put whatever you want in the dependencies section, as long as they can be accessed by the runtime server when NPM installs the package:

{
    "main":"./index.js",

    "dependencies":{
        .
        .
        .
        underscore":"1.3.3",
        "futures":"2.1.0",
        "underscore.string":"2.2.0rc",
        "jsclass":"3.0.9",
        .
        .
        .
    }
}

The Web Server

Packages may be served by the web server in tar.gz or zip form. Zip packages may have the package.json file at the root level or in a directory stored at the root level. So you can have a package called my-package.zip with either this internal structure:

my-package
|
+- package.json
+- main.js
+- <whatever>

or with no root folder, like this:

+- package.json
+- main.js
+- <whatever>

The DynamicModuleLoader will find the location of the package.json file and use that as the root of the module.

You need not pre-install the package using NPM. In other words, you needn't include the node_modules dependencies in the package.

The DynamicModuleLoader will make GET requests for packages using URLs of the form:

http://the-server/module-name.tar.gz

Where http://the-server is a configuration value and module-name is the name of the module requested at runtime.

How To Use

OK, enough blabbing. Here's how you create and configure a DynamicModuleLoader:

// Require all of the needful.
var LockManager = require('hurt-locker').LockManager;
var DynamicModuleLoader = require('dynamic-module-loader').DynamicModuleLoader;

// Create our loader.
var dynamicModuleLoader = new DynamicModuleLoader();

// Configure the directories needed by the module loader.  These methods also have accessors (getXXDir()).

// Set the download directory.  Default is ./downloads
dynamicModuleLoader.setDownloadDir(path.normalize("/some/accessible/location/downloads"));

// Set the installed modules directory.  Default is ./installed-modules
dynamicModuleLoader.setModuleInstallationDir(path.normalize("/somewhere/else/accessible/installed-modules"));

// Configure the package web sever URL.  URL can be anything.  Default is http://localhost.
dynamicModuleLoader.setModulePackageServerUrl("http://gattacus");

// Provide the loader with a lock manager.  If you don't do this it will create its own lock manager and use
// ./locks as the lock directory.
var lockManager = new LockManager();
lockManager.setLockDir(path.normalize("/another/accessible/location/for/lock/files");
dynamicModuleLoader.setLockManager(lockManager);

Now that it's been configured we can download and run some packages. But not so fast big boy! Hold your horses! We need a package first.

Let's say we have a package that consists of just two files: the package.json file and the main package file. To make it interesting, the package uses Futures. Here are the files in the package:

package.json

{
    "name":"test-dynamic-module",
    "version":"0.0.1",
    "main":"./index.js",
    "dependencies":{
        "futures":"2.1.0"
    }
}

index.js

var Future = require('futures').future;

module.exports.helloWorld = function()
{
    var future = new Future();

    // No need to bother with anything fancy.  We can pre-fulfill this future with a success message.  The point is
    // only to include NPM dependencies in our test dynamic module to make sure everything works.
    future.fulfill(undefined, "hello world");

    return future;
};

We can call the helloWorld function like this:

var moduleResult = dynamicModuleLoader.load('test-dynamic-module');
result.when(function(err, module)
            {
                if (err)
                {
                    // Failed to download the module.  You can try again or give up like a sissi.
                    return;
                }


                // Do something with the module.
                console.log(module.helloWorld());
            });

Changing Defaults

By default the DynamicModuleLoader assumes you will be downloading packages in tar.gz form. You can change the default like this:

var dynamicModuleLoader = new DynamicModuleLoader();
dynamicModuleLoader.setDefaultRemoteServerPackageFileExtension('.zip');

// Will assume a .zip extension when requesting the package from the web server.
var moduleResult = dynamicModuleLoader.load('test-dynamic-module');
.
.
.

You can always override the default by specifying the extension when calling load:

// By default assume .tar.gz.
var dynamicModuleLoader = new DynamicModuleLoader();

// Will assume a .tar.gz extension when requesting the package from the web server.
var moduleResult = dynamicModuleLoader.load('test-dynamic-module');
.
.
.

// Will assume a .zip extension when requesting the package from the web server.
var moduleResult = dynamicModuleLoader.load('test-dynamic-module', '.zip');
.
.
.

External Dependencies

DynamicModuleLoader depends on the existence of npm and unzip on the machine on which it is deployed. You can specify the paths to these executables like this:

var dynamicModuleLoader = new DynamicModuleLoader();
dynamicModuleLoader.setNpmExecutablePath('/wherever/npm/is/installed');
dynamicModuleLoader.setUnzipExecutablePath('/wherever/unzip/ins/installed');

By default, DynamicModuleLoader assumes that npm and unzip paths are /usr/local/bin/npm and usr/bin/unzip respectively.

Customizing NPM Options

You can customize the NPM options. For example, you can set NPM to perform a verbose install:

var dynamicModuleLoader = new DynamicModuleLoader({
    npmOptions: ['--production', '--verbose']
});
dynamicModuleLoader.setNpmInstallVerbose(true);

Skipping Installation

You can configure the dynamic downloader such that it doesn't execute the 'npm install' phase. In this configuration, the downloader assumes that modules have been pre-installed and that executing npm is not necessary. The loader still fires the same events in this configuration but you need not specify the location of NPM. Here's how to skip the installation process:

dynamicModuleLoader.setNpmSkipInstall(true);

You can get the configuration settings like this:

dynamicModuleLoader.getNpmSkipInstall();

By default, NPM install is turned on, meaning that the "NPMSkipInstall" property is false.

Events

DynamicModuleLoader is an event emitter. Here's how to listen for the events it fires:

var dynamicModuleLoader = new DynamicModuleLoader();
dynamicModuleLoader.on(dynamicModuleLoader.events.moduleDownloaded, function(moduleName, downloadedFile, next)
{
    // Called when a module is downloaded from a remote source.

    // moduleName - The name of the module downloaded.
    // downloadedFile - The package file that was downloaded.
    // next - A function that proceeds to the next stage in the event workflow (see below).
});
dynamicModuleLoader.on(dynamicModuleLoader.events.moduleExtracted, function(moduleName, extractLocation, next)
{
    // Called after a module has been downloaded and the package has been extracted to disk.

    // moduleName - The name of the module extracted.
    // extractLocation - The location to which the package contents were extracted.
    // next - A function that proceeds to the next stage in the event workflow (see below).
});
dynamicModuleLoader.on(dynamicModuleLoader.events.moduleInstalled, function(moduleName, installationLocation, next)
{
    // Called after a module has been extracted and NPM has installed it.

    // moduleName - The name of the module installed.
    // extractLocation - The location to which the package contents were extracted.
    // next - A function that proceeds to the next stage in the event workflow (see below).
});
dynamicModuleLoader.on(dynamicModuleLoader.events.moduleLoaded, function(moduleName, next)
{
    // Called after a module has been NPM installed and loaded.

    // moduleName - The name of the module loaded.
    // next - A function that proceeds to the next stage in the event workflow (see below).
});
dynamicModuleLoader.on(dynamicModuleLoader.events.moduleEvicted, function(moduleName)
{
    // Called whenever a module has been evicted from memory

    // moduleName - The name of the module evicted.
});
dynamicModuleLoader.on(dynamicModuleLoader.events.moduleDestroyed, function(moduleName)
{
    // Called after the system has called the destroy method of the module (just before the eviction)

    // moduleName - The name of the module loaded.
});

Next Function

The "next" function passed to event methods is a counted callback. That means that processing will continue when all event listeners call "next". If one of the listeners fails to call it, processing will stall, so be sure you invoke next()!

The next function takes the following parameters:

  • err An error object. Pass 'undefined' if there was no error.
  • all other parameters.

Evict Function

The evict function that takes a package name (same one provided during the load) in parameter will evict this module from the cache. It means that if you're doing another dynamicLoader.load of this package, the system will have to reload the package from the disk instead of using the require.cache. It can be very useful for when you need to load lot or big packages and do not use them for long.

The evict function returns a future object (same kind as the load function) so you can add a callback to it using the function when on the future object. See example below :

 
    var moduleResult = dynamicModuleLoader.load('test-dynamic-module');
 
    //Do some processing
 
    dynamicModuleLoader.evict('test-dynamic-module').when(function (err, packageName) {
        //Package 'packageName' correctly evicted from the cache if there is no 'err'
    });
 

One particularity of this evict function is that it will try to call a function 'destroy(callback)' of your module so you can on cleanup any resource you want to cleanup (especially useful when using c++ native library). Here is an example of how it work :

You need to export inside your module a function named destroy which takes 1 argument, the callback function :

module.exports.destroy = function(callback) {
 //Implement what you want to do just before the module is evicted
 callback(); //This function call is mandatory as it allows the system to know when you're done and continue with the eviction
}

If the module doesn't contains a function named destroy then the eviction of the module will be done directly.

All Properties and Defaults

Here are the properties published by DynamicModuleLoader along with their respective default values:

Property Name Default Value
downloadDir path.normalize('./downloads')
moduleInstallationDir path.normalize('./installed-modules')
modulePackageServerUrl http://localhost
npmExecutablePath /usr/local/bin/npm
npmOptions An array of options supplied to npm. By default contains ['--production']. See NPM options https://npmjs.org/doc/.
npmSkipInstall false
lockManager new lock manager, lock dir set to path.normalize('./locks')
downloadLockTimeout 30000
lockOwner {id:'DynamicModuleLoader'}
defaultRemoteServerPackageFileExtension .tar.gz
unzipExecutablePath /usr/bin/unzip

Readme

Keywords

none

Package Sidebar

Install

npm i dynamic-module-loader

Weekly Downloads

3

Version

1.0.8

License

none

Last publish

Collaborators

  • kdunsmore