raspar
A simple to use Promise-based web scraper with local caching.
Usage
const { fetch } = require('raspar');
fetch('http://www.google.com/humans.txt').then(content => console.log(content));const { fetch } = require('raspar');
fetch([
'http://www.google.com/humans.txt',
'http://www.google.com/robots.txt'
]).then(content => console.log(content[0]));Options
const { fetch } = require('raspar');
const options = {
cacheDirectory: 'temp/cache/',
requestOptions: {
headers: {
'User-Agent': 'request'
},
method: 'POST'
},
ttl: 1800
};
fetch('http://www.google.com/humans.txt', options).then(content =>
console.log(content)
);| Name | Description | Default Value |
|---|---|---|
| cacheDirectory | Directory to store cache. | temp/cache/ |
| requestOptions | Request options object. Read more github.com/node-fetch/node-fetch | {} |
| ttl | TTL (Time to live) in seconds. | 1800 |