Browse by Keyword: "spider"

Page 1

cc98-node-api This is an API that parses www.cc98.org into useful informations in json.

claw Simple web scraper chassis, scrapes fields from a list of web pages and dumps the results to JSON/CSV files.

cobweb Web auditing and analysis framework

cobweb-queue Adds queuing functionality to Cobweb

console-crawler A simple web crawler that keeps to the domain

crawler Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously. Scraping should be simple and fun!

crawler-hq Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously. Scraping should be simple and fun!

crawler-mod based on node-crawler

crawljs A basic nodejs crawler.

dollar Web scraper with a jQuery like wrapper

doogle Node.js app for taking HTML snapshots of JavaScript pages to make your dynamic apps Google crawlable.

easyspider mini spider.

elsewhere A node project that aims to replicate the functionality of the Google Social Graph API

extrae A web scraping framework written in coffeescript

flexible Easily build flexible, scalable, and distributed, web crawlers.

funnelweb Detect search engine crawlers by their User-Agent strings.

gretel Follows and collects breadcrumbs accross the web

huntsman Super configurable async web spider

img-crawler A module to download images from a given URL

isrobot a small module that determines based on user-agent if the request came from a robot

kickstarter-crawler Crawls a given kickstarter project page. Returns over 40 data points.

krawler Fast and lightweight web crawler with built-in cheerio, xml and json parser.

node-crawler Node.JS Multithreaded Web Crawler with rules to parse site

node-spider Generic web crawler powered by NodeJS

node-spy Concurrent Spider with Node.js and server-side jQuery

octopus A fast & easy web scraping framework

pagemunch A node.js wrapper for the PageMunch web crawler API

phanos Simple human like stress test tool. This tools doesn't provide any stat info, or special logging functionality. This tool just walking on yours site as true user.

phantomjs-test-starter Starter Template for testing PhantomJS ‘Applications’ with Jasmine, Grunt, and Istanbul

rambler Walk a website checking for bad links

repunt Simple, configurable and extensible webcrawler

ruthless Crawl the web breadth-first from a seed url, statefully

Scrap Easy way to write a spider.

scraperrr Web crawler configured by JSON configurations defining what data fields to scrape from the visited websites using regular expressions or DOM selectors and how to export them as JSON

simplecrawler Very straigntforward web crawler. Uses EventEmitter. Generates queue statistics and has a basic cache mechanism with extensible backend.

sinew-node Sinew-Node collects structured data from web sites (screen scraping).

spidar web spider common tools, targets on chinese sites

spider-event A simple evented web scraping framework using node.js

spidertest A test framework for verifying links and HTTP requests and their responses with automated test generation.

spiderweb Crawl multiple domains using one or more entry URLs.

splitzee-crawler Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously. Scraping should be simple and fun!

tarantula nodejs crawler/spider which provides a simple interface for crawling the Web

voxel-spider blocky spider creatures for your voxel.js game

web-crawler Scalable, extensible, web crawler framework.

Page 1

npm loves you