jwebquery

0.0.7 • Public • Published

jwebquery

jQuery like web crawler

Note

It's my first node modules and features are not well tested now.Since it's just a crawler,use it on your own risk.

Feature

  • gzip support
  • http/https support
  • cookie support
  • can set page fetch/download interval,and max parallel download limitations.
  • http proxy support
  • GET method support ,POST is not tested and upload multipart is not supported
  • auto add http head Referer

Example

basic

var jwebquery = require("jwebquery");
(new jwebquery.Page("http://www.stackoverflow.com/")).ready(function(page){
    //get things by css selector
    //we get every summery text of index page of stackoverflow
    page.find(".question-summary .summary h3").each(function(index){
    //access jquery from jwebquery 
    console.log(index,jwebquery.$(this).text());
    })
}).on("error",function(err){
    console.log(err);
})
 
 

proxy

//use localhost:8099 as http proxy
jwebquery.setProxy("http://localhost:8099/");

Readme

Keywords

Package Sidebar

Install

npm i jwebquery

Weekly Downloads

8

Version

0.0.7

License

none

Last publish

Collaborators

  • nstal