memcouch

tiny in-memory CouchDB wannabe

npm install memcouch
6 downloads in the last week
12 downloads in the last month

memcouch

This is an experiment: assuming a CouchDB dataset small enough to replicate to the client, why not simply keep it in memory?

Access the data with synchronous calls (including a simple "full table scan" style query mechanism) while still keeping basic track of changes. Then let some other code handle the "eventual consistency" part in the background — asynchronously replicating* to something like PouchDB or CouchDB in the background.

What could possibly go wrong?

* Note that it would take not-insignificant more code complexity here to participate in full replication. Perhaps a more accurate way of describing what memcouch does right now is "slaving", to a single persistent/remote database.

Example usage

First <script src="memcouch.js"></script> or npm install memcouch as needed. Here's an example of usage:

var memcouch = require('memcouch');
memcouch.slaveToPouch = require('memcouch.pouchdb').slaveToPouch;
// ^^^ don't use above in browser

var db = memcouch.db();
db.put({_id:'zero', number:0});
db.put({_id:'aaaa', number:3});
db.put({number:2});
db.put({number:1});

// get all documents, sorted by number (pass `true` or a custom comparator)
db.query(function (doc) { this.emit(doc.number); }, true);

// array of all long (in this case, autogenerated) document _ids
var min = 4;
db.query(function (doc) { if (doc._id.length > min) db.emit(); }).map(function (row) { return row.doc._id; });

var lastSeq = null;
function watcher(changeResult) {
    lastSeq = changeResult.seq;
    console.log(changeResult.doc._id + " changed!");
}
db.watch(watcher);
var doc = db.get('zero');
doc.number = Infinity;
db.put(doc);      // will log
db.clear(watcher);

doc.number = 0;
db.put(doc);
db.since(lastSeq);      // array of one changeResult

var status = Pouch("idb://metakaolin", function (e, pouch) {
    memcouch.slaveToPouch(model, db);
});
window.addEventListener('beforeunload', function (e) {
    if (status.changesPending) return (e.returnValue = "Not all of your changes have been stored yet.");
}, false);

API Documentation

Memcouch provides a sychronous API to an in-memory collection of documents. The basic API is as follows:

  • var db = memcouch.db() — returns a new in-memory document collection
  • db.put(doc) — saves new document or increments revision, if no _id one will be generated
  • db.get(id) — returns document with given _id, or undefined if missing. This object may be shared with other callers.
  • db.del(id) — removes all non-internal fields and adds "_deleted": true. will no longer show up from .get or .query

Documents are given a _seq field, which is set to the database update sequence (see below) when the document was last saved with .put(). Currently the _rev field is left as-is, so eventual writes back to a mirrored database will succeed unless the document has changed remotely.

There are also some Couch-like ways to retrieve and monitor data:

  • db.update_seq — starts at zero, increments each time a change is made
  • db.query(map[, cmp]) — returns array of what map function yields via db.emit(key, value) (db is passed as this argument). Results will not be sorted unless a comparator (or true for default) is passed. Using the default comparator iswill return similar results as a CouchDB view, albeit relying on range filtering within the map function [may be extended later] and reduction using the ES5 built-in Array.prototype.reduce method.
  • db.since(seq) — returns a _changes-like array since a given update_seq. Note that memcouch does not do _rev tracking, and so no "changes":[{"rev":"…"] field is provided; just {seq,doc,id[,deleted]} fields.
  • db.watch(cb[,seq]) — during all subsequent changes, the callback function cb will be passed a change object like those provided within the .since() array. If you provide the optional seq parameter, all current changes since that db.update_seq will be fed to cb before this call returns. The future callbacks are also syncronous in a sense, i.e. it happens before control is returned to the caller of .put().
  • db.clear(cb) — removes callback provided to previous .watch() call, does nothing if original function is not found.

Using the optional memcouch.pouch helper function, you can slave a memcouch instance to PouchDB (and therefore CouchDB via its HTTP adapter):

  • var status memcouch.slaveToPouch(db, pouch) — sends all existing and future changes from memcouch db to pouch instance, and vice versa. Returns a status object including a changesPending:0 field, which will be incremented while asyncronous write(s) from memcouch to PouchDB are in progress. No conflict handling is done, the memcouch database is simply overwritten with recent changes sent from PouchDB.

Since memcouch does not track document revision trees, this does not implement the "true" CouchDB masterless replication algorithm. It is only intended to provide something of a buffer between your app's active code and a single underyling persistent store. Any multi-peer or filtered replications should be set up at the PouchDB/CouchDB level.

TODO

  1. Implement default comparison for emitted arrays/objects
  2. Fix up some missing since/watch features.
  3. Profit!
npm loves you