Redis Spidey - Distributed Web Scraping Solution Powered by Redis
RedisSpidey is a powerful tool that combines the capabilities of Spidey and Redis to enable efficient distributed crawling and web scraping. Leveraging the advanced features of Redis, RedisSpidey features a distributed architecture that supports parallel operation of multiple instances, all listening to the same queue. Additionally, RedisSpidey pushes scraped data back to Redis queues for easy distributed post-processing, enhancing the overall efficiency of the scraping process.
Features
- Distributed Crawling: RedisSpidey enables seamless operation of multiple instances of crawlers, all listening to the same queue, for efficient distributed crawling.
- RedisPipeline: RedisSpidey provides support to push crawled data back to Redis queues for distributed post-processing
Installation
npm install spidey-redis
Options
RedisSpidey supports all Spidey options in addition to the following specific options.
| Configuration | Type | Description | Default | Required |
|---|---|---|---|---|
redisUrl |
string |
Redis url such as redis://localhost:6379
|
null |
Yes |
urlsKey |
string |
Redis input queue name such as urls:queue
|
null |
Yes |
dataKey |
string |
Redis output queue name such as data:queue
|
null |
Yes if using RedisPipeline |
sleepDelay |
number |
Wait for new items in queue if empty | 5000ms |
No |
Usage
import { RedisSpidey, RedisPipeline } from 'spidey-redis';
class AmazonSpidey extends RedisSpidey {
constructor() {
super({
// spidey options ...
redisUrl: 'redis://localhost:6379',
// Input queue
urlsKey: 'amazon:urls',
// Output queue
dataKey: 'amazon:data',
// Redis pipeline to push crawled data to data queue
pipelines: [RedisPipeline],
});
}
}Conclusion
RedisSpidey is the ultimate solution for distributed web scraping and crawling, offering unparalleled performance, scalability, and flexibility. With RedisSpidey, you can easily handle large-scale web scraping tasks with ease, while taking advantage of advanced Redis and Spidey technology for efficient distributed crawling and post-processing of data.
License
Spidey is licensed under the MIT License.