Browse by Keyword: "crawl"

Page 1

amazon-products A node.js module to crawl product IDs from Amazon.

amazon-reviews A node.js module to crawl product reviews from Amazon.

bas Behaviour Assertion Sheets: CSS-like declarative syntax for client-side integration testing and quality assurance.

cameo Small-scale webpage archiver

craigslist-utils Nodejs utilities for scraping/crawling/working with craigslist

crawl Website crawler and differencer

crawlit A node.js crawler support custom crawl rules for special site with a plugin.

crawljs A basic nodejs crawler.

crawlstream Crawl websites in a streaming fashion

crawly make output crawl across your terminal all spooky like

curljs Node.js module that wraps curl functionality

gnosis A utility to traverse an object and execute a callback to transform the object, etc.

gretel Follows and collects breadcrumbs accross the web

grunt-url-image-crawler Crawl your CSS/SCSS or HTML files for img URL's and store the crawled image URL's in a local JSON file.

huntsman Super configurable async web spider

img-crawler A module to download images from a given URL

jedi-crawler Lightsabing Node/PhantomJS crawler. Crawl almost everything, including AJAX content.

node-spider Generic web crawler powered by NodeJS

phantom-crawl Web crawler for ajax applications

phantomjs-test-starter Starter Template for testing PhantomJS ‘Applications’ with Jasmine, Grunt, and Istanbul

rambler Walk a website checking for bad links

repubblica-sport-match A node.js module to scrape player points from Repubblica Sport match report

repunt Simple, configurable and extensible webcrawler

robotstxt a robotstxt parser for node.js

ruthless Crawl the web breadth-first from a seed url, statefully

scraperrr Web crawler configured by JSON configurations defining what data fields to scrape from the visited websites using regular expressions or DOM selectors and how to export them as JSON

scrowser A server-side scraping web browser

session-scraper Simple scraper for imitating browsing sessions

skrap Easily scrap web pages by providing json recipes

skrapper custom scrapping tool

spatula Spatula scrapes things! Webby things.

specter Specter is a Node.js wrapper for PhantomJS. PhantomJS is a headless browser which enables developers to load pages that rely on Javascript. Specter was made to easily access these dynamically and asynchronously created HTML pages using Node.js.

spiderweb Crawl multiple domains using one or more entry URLs.

spotlight An object crawler/property search library that works on nearly all JavaScript platforms.

upstage Utility to crawl an entry URL and produce a static copy of the site.

witchypoo Stores a unique list of domain names and their page rank at time of crawling

Page 1

npm loves you