batcher

A stream that batches data sent to it

npm install batcher
5 downloads in the last day
57 downloads in the last week
675 downloads in the last month

Batcher

Batching your streams since 2012.

Usage

Your logging service takes a stream of JSON objects. You don't want to stream objects as they come, but rather batch them and send them every minute, or when a hundred comes by or some such thing.

var Batcher = require('batcher');

var b = new Batcher({
  batchSize: 100,
  batchTimeMs: 60000,
  encoder: function(items) {
    return items.map(JSON.stringify).join('');
  }
});

b.pipe(net.createConnection(host: 'loggingservice.com', port: 1337));

More frequently, you don't need to pipe it but just accumulate:

b.on('data', function(payload) {
  upload('/somewhere', payload);
});

You can also manually flush the batch:

b.write('one');
b.flush(function() {
  console.log('one has been sent');
});

Batchers support write(), pause(), resume(), flush(), end() and destroy().

pause()/resume() is useful when you need to send with retries, but want to avoid the thundering herd when the server comes back online.

var b = new Batcher({
  batchSize: 1,
  maxBatchSize: 100,
  encoder: function(items) {
    b.pause();

    function send() {
      request({
        url: some_url,
        json: items,
      }, function(err) {
        if (err) return setTimeout(send, 1000);
        b.resume();
      });
    }

    send();
  }
});

batchSize of 1 means send them as they come. maxBatchSize of 100 means if there are more available, send as many as 100. setTimeout is used to retry sending every second.

npm loves you