❤
npm
Sign UpSign In

docparse-scraper-status-socket

1.0.13 • Public • Published

Scraper Status Socket

Allow node.js based scrapers to connect to a status socket so that external services can query the current status of the scraper and check if it is running or stopped.

When a scraper starts up, it registers a new service with seaport using the role scraperStatusSupplierCode. For example the NGrid Electric scraper has the supplierCode NGE so it would register with the role scraperStatusNGE

Then this module will connect to an axon rep socket on the port returned by seaport.

Querying

To query the status of a scraper, get the port of the status socket from seaport and send a status request over http

var inspect = require('eyespect').inspector();
var request = require('request')
var config = require('nconf')
var supplierCode = 'NGE'
var seaConfig = config.get('seaport')
var seaHost = seaConfig.host
var seaPort = seaConfig.port
var ports = seaport.connect(seaHost, seaPort)
var role = 'scraperStatus' + supplierCode // eg "scraperStatusNGE"
var scraperServices = ports.query(role) // an array of services
var scraperService = scraperServices[0] // use the first one as an example
var statusHost = scraperService.host
var statusPort = scraperService.port
 
var url = 'http://' + statusHost + ':' + statusPort + '/status'
var opts = {
  url: url,
  json: true
}
request(opts, function (err, res, reply) {
  if (err) {
    inspect(err, 'error getting status of scraper')
    return
  }
  inspect(reply, 'scraper status reply')
})

The reply is json and should look like

reply = {
  "supplierCode": "NGE",
  "status": "running",
  "running": true
}

Readme

Keywords

none

Package Sidebar

Install

npm i docparse-scraper-status-socket

Weekly Downloads

12

Version

1.0.13

License

BSD

Last publish

Collaborators

  • clewfirst
Try on RunKit
Report malware