text-pipe
Transform stream enumeratee generators for stream-driven data extract and transformation (i.e. ETL).
$ npm install text-pipe
text-pipe is a node module that includes functions for creating enumeratees that serve as transform streams. This allows you to iteratively compose complex data processing workflows via simple map, reduce, and filter-like functions. Check out the examples folder for working code samples. In order to run the code samples, you must do the following:
% git clone https://github.com/wilmoore/text-pipe.git
% cd text-pipe
% npm install
% cd examples
% cat stackoverflow.txt
% node extract-question-ids
NOTE: read the comments and the code in the examples to get a feel for how it works.
API
.map
aliases: .transform
For each value, return result of calling transformer function to given value.
var transform = map(function transformer(string) {
return /\d+/.exec(string)[0];
});
stream.pipe(transform);
.select
aliases: .filter, .keep
For each value, return only values that pass a truth test.
var transform = select(function (string) {
return /^question_id:/.exec(string);
});
stream.pipe(transform);
.reduce
aliases: .fold, .inject
Array.prototype.reduce as a through stream.
var max = reduce(function(memo, val) {
return Number(val) > Number(memo) ? val : memo;
}, 0)
stream.pipe(max);