bucket-maker

To reduce I/O cost, bucket-maker makes bucket of sliced timline for each unit you requested.
Name of the bucket is defined by its day and slot.

• day is a number of days from unix epoch
• slot is a number of unit from 00:00
• unit is defined by you

This is calculated like as follows:

unit = 5000;                                                // msec
days = Date.now() / (60*60*24*1000);                        // 17764.309366608795
day  = Math.floor(days);                                    // 17764
slot = Math.ceil( (days - day) * (60*60*24*1000) / unit );  // 5346

Demo

$ git clone https://github.com/RayKitajima/Bucket-Maker.git
$ cd Bucket-Maker

$ node demo/timestamp.js

demo/timestamp.js write out timestamp every sec into the bucket.
The bucket is made for every five sec.
So the bucket contains five entries in it.

At a grance

// get 5sec(msec) bucket maker
let bucket = require('bucket-maker').createBucket({
    unit: 5000
});

// put timestamp every sec
setInterval(function(){
    bucket.put(Date.now());
},1000);

// will write out
$ ls 
17764.4761.bucket  17764.4762.bucket

// will contain
$ cat 17764.4762.bucket
1534833405881
1534833406883
1534833407888
1534833408893
1534833409897

Options

Internally bucket is an array. And it will periodically dump and write them by dumper and writer.

name	what is
prefix	prefix for output bucket name
suffix	suffix for output bucket name. default <suffix> is '.bucket'
unit	bucket unit as msec
spool	where to write out bucket file
dumper	how to dump elements in the bucket. by default assume them as stirng and join() them
writer	how to write out dumped object. by default write out to file named as <prefix><day>.<slot><suffix>.
start	whether to start writing at created
cron_time	frequency of checking bucket
log	how to system log

Practice

const pid  = process.pid;
const host = os.hostname();
const prefix = pid + '.' + host + '.';

let bucket = require('bucket-maker').createBucket({
    prefix : prefix,
    suffix : '.bucket',
    unit   : 5000,
    spool  : '/path/to/buckets',
    dumper : function(items){ return items.map( x => JSON.stringify(x) ).join('\n') },
});

setInterval(function(){
    bucket.put(os.loadavg());
},1000);

Reporting load avarage every sec. And make 5sec bucket.

$ ls 
4825.localhost.17764.4761.bucket  4825.localhost.17764.4762.bucket

More reducing I/O cost with redis

const client = require('redis').createClient();

const writer = function(maker){
    let names = Object.keys(maker.Buckets);
    let now = Date.now();
    for( let i=0; i<names.length; i++ ){
        if( maker.Buckets[names[i]].cutoff > now ){ continue; }   // skip: still putting into the bucket
        let dump = maker.dumper(maker.Buckets[names[i]].buffer);
        let key  = name[i];                                       // wait:17764.4762.bucket
        client.set(key,dump,function(err,res){
            if( err ){
                maker.log(err);
            }else{
                delete maker.Buckets[names[i]];
            }
        });
    }
};

let bucket = require('bucket-maker').createBucket({
    prefix : 'wait:',
    unit   : 5000,
    dumper : function(items){ return items.map( x => JSON.stringify(x) ).join('\n') },
    writer : writer,
});

setInterval(function(){
    bucket.put(os.loadavg());
},1000);

This writes out loadavg report into redis.

const client = require('redis').createClient();

const drain = function(){
    client.keys('wait:*',function(err,keys){
        for( let i=0; i<keys.length; i++ ){
            client.get(keys[i],function(err,content){
                let bucket_name = keys[i].split(':')[1]; // 17764.4762.bucket
                fs.writeFileSync(bucket_name,content);
                client.del(keys[i]);
            });
        }
    });
};

setInterval(drain,5000);

This drains bucket to the file system.

If you would like to implement your own writer, you have to know about internal code, and should write several idiom to define what to write out.

License

MIT