skyscraper

Scrape the web.

npm install skyscraper
18 downloads in the last month

Skyscraper

NPM version Build Status Dependency Status

Skyscraper is an easy to use web scraper / spider for extracting arbitrary data from web pages by using simple CSS selectors. Skyscraper is super good when you want to evaluate a number of pages that all follow the same format.

Try it out for ya self!

$ npm install skyscraper --save

Synopsis

Skyscraper is based on simple schemas (we call them blueprints), somewhat inspired by Mongoose. One blueprint describes one type of page and is linked to one Skyscraper instance. After you have set up the blueprint, you can start scraping pages.

/**
 * Create a new `Skyscraper` instance.
 */
var skyscraper = new Skyscraper;

/**
 * Set up the blueprint.
 */
skyscraper.blueprint

  .property("offer.title")
  .selector("#offer h1")
  .string()
    .default("No title given")

  .property("offer.url")
  .selector("#offer h1 a", "href")
  .string()

  .property("offer.price")
  .selector("#offer span.offer-price")
  .number()
    .unsigned()
    .float()

  .property("offer.description")
  .selector("#offer span.offer-price")
  .string()
    .trim(1000)

  .property("offer.items")
  .selector("#offer ul.offer-items li")
  .array().string()

  .property("offer.features")
  .selector("#offer ul.offer-features li")
  .selector("#offer ul.offer-features li", "class")
  .object()
    .string()
    .boolean(/available/, /unavailable/)

/**
 * Scrape a page.
 */
skyscraper.get("http://www.example.com/offer/12345", function(err, model, html) {
  // Everything okay?
  if(err) throw err;

  // Show the scraped data.
  console.log(model); // { offer: { title: "No title given", price: 150.3, ... }}

  // Show the original HTML code
  console.log(html); // "<html>\n\t<head>\n\t\t<title>example.com</title>..."
});

Is it ready for production?™

Though we have not reached v1.0.0 yet, we are pretty optimistic about our Sykscraper. Here are the key facts:

  • Skyscraper is based on cheerio by Matthew Mueller, which is pretty well tested and gets constantly updated. Cheerio is probably the fastest and most forgiving jQuery-esque implementation for Node.js.

  • We follow the Semantic Versioning rules to spare you the dependency hell.

  • We heavily use Skyscraper ourself 24/7 for our web-indexing software. If we discover an issue, we want to get it fixed. Instantly.

npm loves you