cloudQ
A http message/job queue that is easy to publish, consume and complete messages...
Install from source
git clone https://github.com/twilson63/node-cloudq.gitnpm install .# configure env varsexport COUCH=http://localhost:5984export DB=cloudqexport TOKEN=fooexport SECRET=barexport PORT=8000export TIMEOUT=100000npm run-script setupnpm start
Install and Run Locally
First, you need to install couchdb, you can download couchdb at http://couchdb.apache.org/
npm install cloudq -gexport COUCH=http://localhost:5984export DB=cloudqexport TOKEN=fooexport SECRET=barexport PORT=8000export TIMEOUT=100000# run servercloudq
Usage
A job message queue server, allows your applications to push jobs to a queue, then worker applications can watch the queue and request for a job, when the worker receives the job, it does the work, then sends a complete message back to the server. Each job as a pre-defined schema that consists of two attributes:
- klass
- args
The klass attribute is a string represents the name of object that you wish to invoke. The args attribute is an array of parameters that you wish to provide to that objects perform method.
job schema
publish
publishes the job to the queue named send_mail
curl -XPOST -d '{ "job": { "klass": "Mailer", "args": [{"to": "foo@email.com", "subject": "hello"}]}}' \-H "Content-Type: application/json" \http://localhost:8000/send_mail#> { "ok": true, "id": "c3b9e16d1efc436b7e30543bcf00182a", "rev": "1-977964400b51b5a1673fa7ac76d33874" } }
consume
consumes the next highest job in the queue
curl http://localhost:8000/send_mail#>{ "klass": "Mailer", "args": [{"to": "foo@email.com", "subject": "hello"}], "id": "1", "ok": true}
complete
curl -XDELETE http://cloudq.example.com/send_mail/1#>{ "status": "success"}
Authorization
Currently authorization is done by environment varables:
TOKEN and SECRET
Theses env variables should match with basic authentication, per request:
curl http://token:secret@localhost:3000/foo
Test Successful Authentication:
curl -XPOST -d '{ "job": { "klass": "Mailer", "args": [{"to": "foo@email.com", "subject": "hello"}]}}' http://token:secret@cloudq.example.com/send_mail
Logging
CloudQ uses bunyan as the logger and returns a stream of json, but if you want to put it into a more common format, then you can use the bunyan
command to pipe the json into a readable format.
npm install bunyan -g
cloudq | bunyan
Produces:
2013-11-05T22:01:23.911Z] INFO: cloudq/4187 on thing-4.local:
0: {
"ok": true,
"id": "_design/dequeue",
"rev": "17-d66392bf5441a2cae9bf4c52700cfeff"
}
--
for a shorter format
cloudq | bunyan -o short`
NewRelic
CloudQ is NewRelic Ready, simply supply an ENV Var for your New Relic key and you should be good to go.
# enable New Relic
export NEW_RELIC_LICENSE_KEY=xkkk
export NEW_RELIC_APP_NAME=cloudq
cloudq | bunyan
Deploy
Deploy to nodejitsu
mkdir mycloudqcd mycloudqnpm init# edit package.json and set "node": "~0.6.x"npm install cloudq --saveecho 'require("cloudq/server");' >> server.jsjitsu databases create couch cloudqjitsu env set COUCH http://xxxx263878962530.iriscouch.com:5984jitsu env set DB cloudqjitsu env set TOKEN foojitsu env set SECRET barjitsu deploy
Deploy to heroku
# create an iriscouch accountmkdir mycloudqcd mycloudqnpm init# edit package.json and set "node": "~0.6.x"npm install cloudq --saveecho 'web: ./node_modules/cloudq/bin/cloudq' >> Procfileecho 'node_modules' >> .gitignoregit initgit add .git commit -am "first commit"heroku createheroku config:add COUCH=http://mydb.iriscouch.comheroku config:add DB=cloudqheroku config:add TOKEN=fooheroku config:add SECRET=bargit push heroku master
Tests
npm test
License
see LICENSE
Contributing
GOALS
- ONLY THREE CORE API METHODS
- POST /queue - PUBLISH a JOB on the QUEUE
- GET /queue - CONSUME a JOB
- DELETE /queue/id - Mark JOB as Completed
TODO
- tokens authorization
- create acl for queues, views, bulk updates
pull requests welcome