crawler-request

1.2.2 • Public • Published

crawler-request

HTTP request module customized for crawlers.

version downloads node status

Installation

npm install crawler-request

Usage

Simple Request

const crawler = require('crawler-request');
 
crawler("https://stackoverflow.com/").then(function(response){
    // handle response
    console.log(response.text.lenght);
});

PDF Parse

const crawler = require('crawler-request');
 
crawler("http://careers.stackoverflow.com/stack_overflow_careers.pdf").then(function(response){
    // handle response
    console.log(response.text.lenght);
});

Extend

const crawler = require('crawler-request');
 
function response_text_size(response){
    response["size"] = response.text.length;
    return response;
}
 
crawler("https://stackoverflow.com/",response_text_size).then(function(response){
    // handle response
    console.log(response.size);
});

Test

mocha or npm test

Check test folder and quickstart.js for extra usages.

Support

I use this package actively myself, so it has my top priority. You can chat on WhatsApp about any infos, ideas and suggestions.

WhatsApp

Submitting an Issue

If you find a bug or a mistake, you can help by submitting an issue to GitLab Repository

Creating a Merge Request

GitLab calls it merge request instead of pull request.

License

MIT licensed and all it's dependencies are MIT or BSD licensed.

Dependencies (6)

Dev Dependencies (4)

Package Sidebar

Install

npm i crawler-request

Weekly Downloads

2,039

Version

1.2.2

License

MIT

Last publish

Collaborators

  • mehmet.kozan