birdeater

0.0.3 • Public • Published

Using Node.js and JQuery to Crawl Public Tweets

A talk by @benjamincoe

Birdeater

Birdeater is a command-line tool for backing up a user's public Tweets in JSON format.

Usage

First, make sure you have node package manager installed:

curl http://npmjs.org/install.sh | sh

Then run the following commands to install and run Birdeater:

npm install birdeater -g
birdeater --user=shitmydadsays

Be mindful when running it, as Twitter limits the number of requests that a single client can make per hour.

How Birdeater Works

Birdeater does not use Twitter's API. It was built as a demonstration of an approach I like to use for parsing structured information from unstructured HTML. Here's how it works:

An http connection is made to a user's public timeline using the request library:

User.prototype.loadStatuses = function(callback) {
    request({
        url: 'https://twitter.com/...',
        headers: {accept: 'application/json'}
    }, function(err, res, body) {
        var error = err || res.statusCode != 200;
        // handle error, or.
        callback( body );
    });
};

This returns an HTML representation of the tweets. JQuery is used to extract structured information from this:

jquery(body).find('.tweet').each(function() {
    var tweet = jquery(this);
    _this.tweets.push({
        text: jquery.trim( tweet.find('.js-tweet-text').text() ),
        timestamp: parseInt( tweet.find('._timestamp').attr('data-time') ),
        mentions: mentions.length > 0 ? tweet.attr('data-mentions').split(' ') : [],
        urls: urls
    });
});

I find that Node.js, coupled with JQuery, works great for building web crawlers:

  • Node.js offers a non-blocking evented paradigm. This is awesome for a web-crawler, which will tend to be I/O bound.
  • JQuery is a champ at parsing HTML. I find it much more intuitive than other libraries that I've used, such as Beautiful Soup.
  • First and foremost, it's easy to do (take a look at lib/user.js Birdeater clocks in at about 100 lines of code).

This approach has become my hammer when web scraping tasks come up.

Package Sidebar

Install

npm i birdeater

Weekly Downloads

1

Version

0.0.3

License

none

Last publish

Collaborators

  • bcoe