jwebquery

An jQuery style web crawler(actually extend jquery).

npm install jwebquery
5 downloads in the last week
30 downloads in the last month

jwebquery

jQuery like web crawler

Note

It's my first node modules and features are not well tested now.Since it's just a crawler,use it on your own risk.

Feature

  • gzip support
  • http/https support
  • cookie support
  • can set page fetch/download interval,and max parallel download limitations.
  • http proxy support
  • GET method support ,POST is not tested and upload multipart is not supported
  • auto add http head Referer

Example

basic

var jwebquery = require("jwebquery");
(new jwebquery.Page("http://www.stackoverflow.com/")).ready(function(page){
    //get things by css selector
    //we get every summery text of index page of stackoverflow
    page.find(".question-summary .summary h3").each(function(index){
    //access jquery from jwebquery 
    console.log(index,jwebquery.$(this).text());
    })
}).on("error",function(err){
    console.log(err);
})

proxy

//use localhost:8099 as http proxy
jwebquery.setProxy("http://localhost:8099/");
npm loves you