node-chunking-streams
A set of Node.js streams to process data in chunks
- LineCounter
- SeparatorChunker
- SizeChunker
- GzipChunker
- S3MultipartUploader
To intstall with NPM
npm install chunking-streams
To use with Node.JS
var chunkingStreams = ; var LineCounter = chunkingStreamsLineCounter;var SeparatorChunker = chunkingStreamsSeparatorChunker;var SizeChunker = chunkingStreamsSizeChunker; // ad so on...
LineCounter
Simple TransformStream
which counts lines (\n
is a separator) and emit data chunks contains exactly specified number
of them.
Configuration
numLines: 1 // number of lines in a single output chunk. 1 is default flushTail: false // on stream end, flush or not remaining buffer. false is default;
SeparatorChunker
Split incoming data into chunks based on specified separator. After each separator found data chunk is emitted.
By default separator sequence is set to \n
, so it is equals to LineCounter with numLines: 1
Configuration
separator: '\n' // separator sequence flushTail: false // on stream end, flush or not remaining buffer. false is default;
SizeChunker
Split streams into chunks having exactly specified number of bytes. It is object mode stream! Each data chunk is an object with the following fields:
- id: number of chunk (starts from 0)
- data:
Buffer
with data
SizeChunker has 2 additional events:
- chunkStart: emitted on each chunk start.
- chunkEnd: emitted on each chunk finish.
Both event handlers must accept two arguments:
- id: number of chunk
- done: callback function, must be called then processing is completed
If some tail data is not fit fully into specified chunk size, it can be emitted at the end if flushTail
flag is set.
Configuration
chunkSize: 1024 // must be a number greater than zero. flushTail: true // flush or not remainder of an incoming stream. Defaults to false;
Example
var input = fs chunker = chunkSize: 1024 output; chunker; chunker; chunker; input;
S3MultipartUploader
INCOMPLETE