❤
npm
Sign UpSign In

streamy-data

0.2.4 • Public • Published

streamy-data

Inspired by gulp.js, this library helps you work on data collection in the form of Node Stream (in object mode). If you are not familiar with gulp.js, here is an introduction:

No Need To Grunt, Take A Gulp Of Fresh Air

The pros of this approach are similar to gulp.js:

  • Leverage the power of Stream.
    • No intermidiate file I/O.
    • Highly granular plugin functions.
    • Straightforward spec. (They're just object mode streams.)
  • More code, less config.

The cons are:

  • Require basic knowledge of Stream.

Install

npm install streamy-data

Usage

var streamy = require('streamy-data');
 
streamy.ptt.board('food', { limit: 2 }) // start with a readable stream which emits post URLs
    .pipe(streamy.ptt.post()) // a transform stream which map the URL to post content
    .on('data', function (data) {
        // do something in an explicit handler
    })
    .on('end', function () {
        // or do something when the stream ends
        console.log('= END =');
    });

API

streamy.array(array)

Return a readable string which iterates through the array.

streamy.array(['A', 'B', 'C'])
    .on('data', function (data) {
        console.log(data);
    });

array

Type: Array

The source array.

streamy.map(func)

Return an asynchronous transform stream.

someReadableStream
    .pipe(streamy.map(function (data, callback) {
        // do something
        if (error) {
            callback(error);
            return;
        }
        // do some transformation
        // data = data + 1
        callback(null, data);
    }))
    .on('data', function (data) {
        console.log(data);
    });

func

Type: Function

The transform function.

streamy.map.sync(func)

The synchronous version of streamy.map.

someReadableStream
    .pipe(streamy.map.sync(function (data) {
        // do some transformation
        // data = data + 1
        return data;
    }))
    .on('data', function (data) {
        console.log(data);
    });

func

Type: Function: * -> *

The transform function.

streamy.http()

Return a transform stream which maps input to the result of its http request.

Input format:

See the first parameter of npm request.

Output format:

{
    url: (input url),
    response: (the response object),
    body: (html body string)
}

streamy.ptt.board(name, [options])

Return a readable stream which emits post links in the PTT board, in descending order.

streamy.ptt.board('food', { limit: 2 })
    .on('data', function (data) {
        console.log(data);
    });

Output format:

{
    board: (board name string),
    author: (author id string),
    post: (post id string),
    title: (post title string),
    href: (post link url string)
}

name

The board name.

options.start

The start page index (high, inclusive).

options.end

The end page index (low, exclusive).

options.limit

The number of pages to scrape. When limit and end are both specified, both conditions are respected.

streamy.ptt.post(options)

Return a transform stream which maps board-post pair to post content.

Input format: (compatible with streamy.ptt.board() output)

{
    board: (board name string),
    post: (post id string)
}

Output format:

{
    board: (board name string),
    post: (post id string),
    title: (post title string),
    raw: (the html body string), // if the raw flag is true
    meta: [
        { tag: '作者', value: ... },
        { tag: '站內', value: ... },
        { tag: '標題', value: ... },
        { tag: '時間', value: ... }
    ],
    content: [
        (content),
        { type: comment, subtype: (1/2/3), author: ..., content: ... },
        { type: comment, subtype: (1/2/3), author: ..., content: ... }
        ...
    ]
}
streamy.ptt.board('food', { limit: 2 })
    .pipe(streamy.ptt.post())
    .on('data', function (data) {
        console.log(data);
    });

options.raw

Type: boolean, default: false

The raw field is included in the output if the value true.

See also

Readme

Keywords

none

Package Sidebar

Install

npm i streamy-data

Repository

github.com/simonpai/streamy-data

Homepage

github.com/simonpai/streamy-data

Weekly Downloads

0

Version

0.2.4

License

none

Last publish

Collaborators

  • simonpai
Try on RunKit
Report malware