Docparse Scraper API server
Parse scraped documents
Startup
To start the parseScraped worker, execute
node parseScrapedWorker.js --config test/config.json
This will create a new Parser object which can receive remote requests to start parsing documents. See app.js for the construction of a new Parser object. Also See the parseScrapedWorker.js file in the project root for details
var ParseScrapedWorker = var config = var parser = config
Parsing
The parseScrapedWorker object has a function parseScraped. This function should be called with a scraped document as the first parameter and a callback function as the second.
var inspect = ;var config = var parser = configvar scrapedDoc = supplierCode: 'HES' payload: billDate: '2011-02-23 00:00:00 +00:00' accountNumber: 'fooAccountNumber' billNumber: 'fooBillNumber' loginID: 'fooLoginID' supplierCode: 'HES' textPages: 'foo page 1' 'bar page 2' // these are the extracted text pages from the bill pdf file parser
Sockets
A scraped document is parsed differently for each supplier. Therefore in the DocParse system, there need to parsing servers online for each supplier. When the ParseScrapedWorker is initiated, it binds to a request Axon socket for each supported supplier. See lib/getRemoteParseSockets.js
for details. It also registers a service with seaport so that the supplier response sockets know where to connect
The system currently supports HES (Hess), NST (NStar), NGE (NGrid Electric), and NGA (NGrid Gas). For HES, the ParseScrapedWorker would bind to a socket as follows
var config = var seaConfig = configvar seaHost = seaConfighostvar seaPort = seaConfigportvar ports = seaportvar role = 'pushScrapedParseJobHES' // all push roles are in the format "pushScrapedParseJob<supplier code>"var port = ports // register with seaport so the remote HES parsing server knows where to connectvar socket = axon;socketsocket
This binding happens for each supported supplier. The ParseScrapedWorker keeps a reference to each supplier-specific push socket in its self.sockets property. The sockets property is an object keyed by supplierCode When the ParseScrapedWorker.parseScraped method is called, it gets the appropriate supplier-specific socket and sends out a request to the remote parsing server. The supplier-specific socket is an Axon req socket.
Parsing Scraped Documents
Each instatiated ParseScrapedWorker object has a parseScraped
function. This function is called with an unparsed scraped document and a callback. The parseScraped function gets the appropriate supplier-specific request socket and sends a new parsing request out the the remote supplier-specific parsing server
Parserprototype { var self = this var config = selfconfig var supplierCode = docsupplierCode var sockets = selfsockets var socket = socketssupplierCode socket})
In the example above, the parseScrape function actually calls lib/parseRemote
function. The functionality is the same, but parseRemote adds a some timeout logic in case the remote parsing server goes down request fails for some reason