Time-based Streaming Chunking
A node.js deplux stream that buffers input for a while before outputting it. This is useful if you have an readable-stream producing many small chunks that you want to pipe to a writable-stream and the writable-stream only handles large chunks efficiently.
Consider the following setup:
streamIn
, produces a lot of small chunks (e.g. output from a subprocess),streamOut
, takes a long time to write a single chunk regardless of size (e.g. sends the chunk over network to a database and commits)
Now, streamIn.pipe(streamOut)
would be a bad idea, because streamOut
has a
large overhead for each chunk regardless of size. Instead using
streamIn.pipe(new TimeChunkedStream({timeout: 5000})).pipe(streamOut)
would
buffer the chunks for 5 seconds and then write them to streamOut
.
This is mainly useful for logging and thing where you want to commit and the commit overhead is significant. If you want to optimizing file-writes or cases with plenty of data consider using buffered-stream instead.
License
The time-chunked-stream
library is released on the MIT license, see the
LICENSE
for complete license.