simplenosql
An NoSQL data store built using Cloudant but hiding some of Cloudant's more advanced features:
- Changes Feeds
- Replication
- Design Documents & MapReduce
- MVCC (revision tokens)
- Attachments
This library concentrates on creating datbases and creating, updating & deleting documents. It also allows the data to be queried and aggregated easily without ever seeing a design document databases.
The format of the data you are returned is simplified: revision tokens are removed and complex aggregate JSON structures are pared down to a minimum.
Get started storing, querying and aggregating your data using simplenosql.
Installation
npm install --save simplenosql
Using in your application
Start up the library by passing the URL of your Cloudant database:
var url = 'https://username:password@myhost.cloudant.com';var animals = url 'animals';
The URL should allow admin access to your Cloudant account.
Alternatively, a single parameter with the URL of the database can be supplied:
var url = 'https://username:password@myhost.cloudant.com/animals';var animals = url;
This library uses Promises so function calls made on simplenosql object will be of this form:
animals <FUNCTION CALL HERE> ;
Some of the following code samples omit the Promise then
and catch
for brevity,
but all database operations are asynchronous.
When the await
JavaScript operator is supported in Node, it will be possible to use this library like so:
var data = await dball;
CRUD operations
Creating a database
Before a database can be used, it must be created once with the create
function:
animals ;// {ok:true}
This creates the database in Cloudant. If you are just connecting to a database that simplenosql created for you
last time, then there is no need for the create
step.
Adding documents
Add a single document to a database with the insert
function:
animals ;// { ok: true, _id: 'dog1' }
Documents have a key field _id
which must be unique across the database. It can
either be supplied by you in the object you pass in or can be omitted and one will be generated for you:
animals ;// { ok: true, _id: "f03bb0361f1a507d3dc68d0e860675b6" }
We can insert arrays of documents for bulk inserts:
var somecats = _id:'cat1' name:'Paws' colour:'tabby' collection:'cats' cost:102 weight:24 _id:'cat2' name:'Fluffy' colour:'white' collection:'cats' cost:82 weight:21 _id:'cat3' name:'Snowy' colour:'white' collection:'cats' cost:52 weight:60 _id:'cat4' name:'Mittens' colour:'black' collection:'cats' cost:45 weight:18;animals ;// { ok: true, success: 4, failed: 0 }
Arrays of documents are written in batches of 500 at a time with up to 5 write operations going on in parallel.
Fetching documents by id
Retrieve a single document with the get
function:
animals ;// { _id: 'cat1', name: 'Paws', colour: 'tabby', collection: 'cats', cost:102, weight:2.4 }
or by supplying multiple document ids to get an array of documents in reply:
animals ;// [ { _id: 'cat1', name: 'Paws', colour: 'tabby', collection: 'cats', cost:102, weight:2.4 },// { _id: 'cat2', name: 'Fluffy', colour: 'white', collection: 'cats', cost:82, weight:2.1 } ]
Updating documents
A document can be replaced with a new document by supplying its _id
and the new document body:
var id = 'dog1';var newdoc = name:'Bobbie' colour:'black' collection:'dogs' cost:45 weight:64;animals ;// {ok:true}
or by passing in a single object that contains an _id
in the new body:
var newdoc = _id: 'dog1' name:'Bobbie' colour:'black' collection:'dogs' cost:45 weight:64;animals ;// {ok:true}
Even if the document id doesn't already exist, simplenosql will write a new document, so in a sense the update
function is rather like an "upsert" operation: either update and replace the existing document or create a new one.
For this reason, an upsert
function also exists that is a synonym of the update
function.
Deleting documents
A document can be deleted by supplying its _id
:
var id = 'dog1';animals ;// {ok:true}
Fetching all the documents
All documents can be retrieved with the all
function:
animals all ;// [ { _id: 'cat1', name: 'Paws', colour: 'tabby', collection: 'cats', cost:102, weight:2.4 },// { _id: 'cat2', name: 'Fluffy', colour: 'white', collection: 'cats', cost:82, weight:2.1 },// { _id: 'cat3', name: 'Snowy', colour: 'white', collection: 'cats', cost:52, weight:6.0 },// { _id: 'cat4', name: 'Mittens', colour: 'black', collection: 'cats', cost:45, weight:1.8 },// { _id: 'f03bb0361f1a507d3dc68d0e860675b6', name: 'Sam', colour: 'grey', collection: 'dogs', cost:72, weight: 5.2 } ]
For larger data sets, the document list is retrieved in blocks of 100 and a skip
option can be supplied to retrieve
documents deeper in the data set:
// fetch records 300 to 400animalsallskip:300
Querying the database
A database can be queried by passing a query object to query
function:
// get animals that are whiteanimals ;// [ { _id: 'cat3', name: 'Snowy', colour: 'white', collection: 'cats', cost:52, weight:6.0 },// { _id: 'cat2', name: 'Fluffy', colour: 'white', collection: 'cats', cost:82, weight:2.1 } ]
where query is key/value pairs that match the source documents. The key/value pairs are AND'd together:
// get documents that black in are in the 'cats' collectionanimals ;// [ { _id: 'cat4', name: 'Mittens', colour: 'black', collection: 'cats', cost:45, weight:1.8 } ]
or the object can contain Cloudant Query Selector operators:
// get animals that are called Paws or that are blackanimals ;// [ { _id: 'cat1', name: 'Paws', colour: 'tabby', collection: 'cats', cost:102, weight:2.4 },// { _id: 'cat4', name: 'Mittens', colour: 'black', collection: 'cats', cost:45, weight:1.8 } ]
The optional second parameter can be used for sorting with a sort
property:
// retrieve black animals and sort by name, in ascending orderanimals
or multi-dimensional sorting with an array of objects:
// get animals that are black, sorted by name and cost in reverse orderanimals;
See Clouant Query documentation for details on the full sort syntax.
The optional second parameter is an object that can contain one or more of:
- sort - an array of sort parameters e.g.
[{'name:string':'desc'},{'cost:number':'desc'}]
- limit - the number of search results to return. Defaults to 100
- skip - the number of results to skip in the result set. Defaults to 0
- fields - either a string representing the document property to return or an array of properties e.g.
['name','cost','collection']
. If omitted, the whole document is returned.
e.g.
// animal names, 100 to 200animals;// name and cost of cats sorted by price - highest firstanimals;
Aggregating data
Counting
The number of documents in a database can be obtained with the count
function:
animals ;// 5
Passing a string to count
returns the number of occurences of that field's values:
// get counts of animals by colouranimals ;// { black: 1, grey: 1, tabby: 1, white: 2 }
Values from deeper within your document can be accessed using object notation:
address.postcode
socialmedia.facebook.email
Passing an array to count
causes multi-dimensional counting:
// get counts of animals, grouped by colleciton and colouranimals ;// { 'cats/black': 1,// 'cats/tabby': 1,// 'cats/white': 2,// 'dogs/grey': 1 }
Summing
To get totals of values from your documents call the sum
function passing in the field you would
like to aggregate:
// get totals on an animals' costanimals ;// 353
The sum of multiple properties can be calculated by passing an array of strings:
// get stats on animals' cost & weightanimals ;// { cost: 353, weight: 17.5 }
The totals can also be grouped by another field by providing a second parameter:
// get sum of animals' cost, grouped by collectionanimals ;// { cats: 281, dogs: 72 } // get sum of animals' cost & weight, grouped by collectionanimals ;// { // cats: { cost: 281, weight: 12.3 },// dogs: { cost: 72, weight: 5.2 } // }
Stats
To get the statistics on values from your documents, call the stats
function passing in the
field you would like statistics on:
// get stats on an animals' costanimals ;// { sum: 353, count: 5, min: 45, max: 102, mean: 70.6, variance: 423.840, stddev: 20.587 }
Multiple values can be analysed using an array of fields:
// get stats on animals' cost & weightanimals ;// { // cost: { sum: 353, count: 5, min: 45,max: 102, mean: 70.6, variance: 423.840, stddev: 20.587 }// weight: { sum: 17.5, count: 5, min: 1.8, max: 6, mean: 3.5, variance: 3.040, stddev: 1.7435 } // }
The stats can also be grouped by another field by providing a second parameter:
// get stats on animals' cost - grouped by collectionanimals ;// { // cats: { sum: 281, count: 4, min: 45, max: 102, mean: 70.25, variance: 529.1875, stddev: 23.004 },// dogs: { sum: 72, count: 1, min: 72, max: 72, mean: 72, variance: 0, stddev: 0 } // }
Arrays work for grouping too:
// get stats on animals' cost & weight - grouped by collectionanimals ;// { // cats: {// cost: { sum: 281, count: 4, min: 45, max: 102, mean: 70.25, variance: 529.1875, stddev: 23.004 },// weight: { sum: 12.3, count: 4, min: 1.8, max: 6, mean: 3.075, variance: 2.896, stddev: 1.7020 } // },// dogs: { // cost: { sum: 72, count: 1, min: 72, max: 72, mean: 72, variance: 0, stddev: 0 },// weight: { sum: 5.2, count: 1, min: 5.2, max: 5.2, mean: 5.2, variance: 0, stddev: 0 } // } // }
Debugging
To see the HTTP requests being made set an environment variable DEBUG
before running your code:
DEBUG=simplenosql node myapp.js
Notes
This library uses Cloudant as its storage engine. It hides some of the complexities of working with Cloudant but if you intend to use Cloudant in earnest, you may want to be aware of some of the compromises and design decisions that this library has made to make it simpler for a first-time user.
Please note:
- the library hides
_rev
tokens from you. They still exist, but you don't see them in returned documents or API calls, nor are they required when updating or deleting documents. You may want to familiarise yourself with Cloudant's Multi-version Concurrency Control mechanism to prevent loss of data when the same document is updated in different ways at the same time in a distributed system. - when this library creates a database with the
create
function, it also creates a Cloudant Query index instructing Cloudant to index all fields with a Lucene-based index. This is convenient but probably not what you want to do in a production system. It's much better to only index the fields you need. - it is still possible to get document conflicts when using this library. Be careful when updating or deleting documents.
- calls to the count/sum/stats function result in a Design Document being generated for every combination of keys/values you supply. In a production system, MapReduce views are usually grouped together with several views per design document.
- with very large data sets, it's not efficient to page through the result set with the
all
function using 'skip' and 'limit' parameters. It's better to use the startkey_docid parameter - when using this library to communicate with CouchDB 2.0, the
create
function will throw an error because it will fail to create a Cloudant Query (Mango) text index. After that, the other functions work although thequery
function will be slow because of the lack of an index to support it.
It's anticipated that you start using Cloudant with this library and switch to the Official Cloudant Node.js library when you're ready to build some production code.