Browse by Keyword: "spider"
cc98-node-api This is an API that parses www.cc98.org into useful informations in json.
claw Simple web scraper chassis, scrapes fields from a list of web pages and dumps the results to JSON/CSV files.
cobweb Web auditing and analysis framework
cobweb-queue Adds queuing functionality to Cobweb
console-crawler A simple web crawler that keeps to the domain
crawler Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously. Scraping should be simple and fun!
crawler-hq Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously. Scraping should be simple and fun!
crawler-mod based on node-crawler
crawljs A basic nodejs crawler.
dollar Web scraper with a jQuery like wrapper
easyspider mini spider.
elsewhere A node project that aims to replicate the functionality of the Google Social Graph API
extrae A web scraping framework written in coffeescript
flexible Easily build flexible, scalable, and distributed, web crawlers.
funnelweb Detect search engine crawlers by their User-Agent strings.
gretel Follows and collects breadcrumbs accross the web
huntsman Super configurable async web spider
img-crawler A module to download images from a given URL
isrobot a small module that determines based on user-agent if the request came from a robot
kickstarter-crawler Crawls a given kickstarter project page. Returns over 40 data points.
krawler Fast and lightweight web crawler with built-in cheerio, xml and json parser.
node-crawler Node.JS Multithreaded Web Crawler with rules to parse site
node-spider Generic web crawler powered by NodeJS
node-spy Concurrent Spider with Node.js and server-side jQuery
octopus A fast & easy web scraping framework
pagemunch A node.js wrapper for the PageMunch web crawler API
phanos Simple human like stress test tool. This tools doesn't provide any stat info, or special logging functionality. This tool just walking on yours site as true user.
phantomjs-test-starter Starter Template for testing PhantomJS ‘Applications’ with Jasmine, Grunt, and Istanbul
rambler Walk a website checking for bad links
repunt Simple, configurable and extensible webcrawler
ruthless Crawl the web breadth-first from a seed url, statefully
Scrap Easy way to write a spider.
scraperrr Web crawler configured by JSON configurations defining what data fields to scrape from the visited websites using regular expressions or DOM selectors and how to export them as JSON
simplecrawler Very straigntforward web crawler. Uses EventEmitter. Generates queue statistics and has a basic cache mechanism with extensible backend.
sinew-node Sinew-Node collects structured data from web sites (screen scraping).
spidar web spider common tools, targets on chinese sites
spider-event A simple evented web scraping framework using node.js
spidertest A test framework for verifying links and HTTP requests and their responses with automated test generation.
spiderweb Crawl multiple domains using one or more entry URLs.
splitzee-crawler Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously. Scraping should be simple and fun!
tarantula nodejs crawler/spider which provides a simple interface for crawling the Web
voxel-spider blocky spider creatures for your voxel.js game
web-crawler Scalable, extensible, web crawler framework.