DeeToo
DeeToo makes it dead-simple to write service-providing workers in Node.js
With minimal boilerplate, you get a fast, reliable server that provides a service (or group of services) to the rest of your infrastructure. When we say minimal boilerplate, we mean it. Here's a worker that knows how to "email" and provides a public HTTP API for pushing jobs, an admin dashboard and more:
// "sendEmail" is your function to send emailsvar d2 = ;d2start;
(pro tip: check out DeeToo Skeleton)
You could use DeeToo for email sending, report generation, rendering, backups, etc... At Rafflecopter, we use it for almost everything.
It features these awesome things, right out of the box:
- Multi-protocol, authenticated API
- Admin dashboard
- Schedule jobs for later, update them before they're processed
- Graceful shutdown for easy [re-]deployments
- Garbage collection
- Extensible logging
- More awesomeness is coming! See TODO
Yeah, whatever. Tell me how to use it.
tl;dr: Read the example file
The atomic unit of work in DeeToo is a Job
. You tell DeeToo how to perform a certain type of job, and DeeToo provides an interface for your application, so other services can send jobs to it.
DeeToo is based on a job queue (specifically Kue, via Redis) - to have DeeToo perform a task, you simply push a job into the queue. DeeToo will process the job as fast as it can (or wait until a certain time, optionally)
DeeToo provides some core functions, which we'll discuss in detail:
DeeToo Module (ie. require('deetoo')
)
.init ([config])
-- returns the DeeToo singleton object
DeeToo Object
.can (jobType, [concurrency], callback)
.speaks (dialect, [dialect], [...])
.start ([callback])
.shutdown ([timeout], [deaf], callback)
What is a Job?
A job is simply a packaged set of information that your service needs to process a specific job. It's an object that contains these fields:
jobType
- what type of job is this (ie. "email", "cohort-report", "csv-export", etc...)jobData
- data specific to this job (ie. "to_address", "subject", etc...)[when]
- optional - A UTC timestamp (in milliseconds) when the job should be run[id]
- optional - if you give a scheduled Job (one w/ awhen
) an ID, you can change it before it's processed
For example, here's a job for a "subscription expiring" email to be sent @ midnight MST on 11/31/2012:
How do I get Jobs in the queue?
You can push a job to DeeToo in one of 2 ways: use the worker's public API, or push a job from w/in your code.
Using the API
DeeToo can speak any protocol, but ships w/ HTTP by default. Use it like this:
POST /api/v2/<job_type>
The POST body should be a JSON-encoded object, containing a .jobData
property, and (optionally) a .when
& .id
From w/in your code:
If you want to push a job from w/in the worker itself, you can use:
d2jobs
job
is an object containing the fields detailed above. callback
is an optional callback that's called after the job is pushed into the queue.
You said I could schedule jobs, and update them later?
Yep! Just include a when
and an id
in your job. If a job w/ that id
is already queued, it will be replaced w/ when a new job w/ the same id
is pushed. Simple!
How do I process jobs from the queue?
That's what d2.can()
is for. Here's how it works:
{ console } d2
You can access your job-specific data from job.data
job
can do some other useful things, like update its own progress (which will be displayed on the admin dashboard). To do that, use job.progress(currentFrame, totalFrames)
-- for more about using job
, see Kue's README
You can optionally pass a concurrency
parameter, which means DeeToo can process that many jobs at once. For low-CPU jobs (like sending email), set this high for a big potential speedup. For high-CPU (blocking) jobs, this won't help much. Note that this does not spawn child processes. It simply runs the jobs in parallel on one process. Example:
d2
Starting Up and Shutting Down
Starting Up
DeeToo is a standalone server, at least providing an admin-facing dashboard via HTTP. To start it, simply do:
d2start { console}
Shutting Down
One of DeeToo's strengths is that it knows how to shut itself down gently, without canceling jobs that are in-progress, or corrupting any data. Here's how to shut it down:
d2
By default, .shutdown()
only stops processing jobs. It will keep the service available by listening for incoming requests and queuing them. This is usually desired behavior! If you want to stop listening for incoming requests without calling process.exit
, you can pass an optional deaf
argument, like so:
d2
By default, DeeToo will only shut down after all in-progress jobs have finished running. This is usually desired behavior! However, sometimes there's a limit on how long you can wait. In those cases, you can use the optional timeout
parameter:
d2
If a timeout occurs, any in-progress jobs will be marked as failed
and will need to be restarted by hand. TODO: there are better ways to handle this, ie. retries
You can use both optional arguments together:
d2
SIGTERM
DeeToo will begin the shutdown process when it receives a SIGTERM
. This is not optional (yet).
Auxiliary things
You can access the raw Kue instance via d2.jobs._rawQueue
You can access DeeToo's logger via d2.log
. To learn how to use it, see the node-book project.
DeeToo shares some of the things it uses, so you can use them, too. They're accessed via d2._
d2._.www
- the Express serverd2._.redis
- the connected redis client
Configuration Options
DeeToo offers a number of configuration options. Set config options by passing an object to .init()
when you first import the module. Defaults are subject to wild change, for now.
port_www - what port should HTTP listen to? -- default: 8888
garbage_collect - remove jobs from Redis after completion? (saves RAM) -- default: true
sigterm_shutdown_timeout - timeout SIGTERM will pass to .shutdown()
-- default: 5 minutes
auth - use HTTP authentication to secure admin UI and job API -- default: undefined
- If defined, should be an object containing
user
&pass
properties
redis - where to find Redis and how to authenticate -- default: localhost, default port, no authentication
- If defined, should be an object containing
host
,port
, &pass
properties
How do I make it speak another protocol?
DOCUMENTATION COMING SOON
TODO
-
Add Cluster for better CPU utilization on multicore machines
-
Add other dialects besides HTTP (eyeing Axon, 0mq, and raw TCP)
______ ,-'//__\\`-. ,' ____ `. / / ,-.-. \ (/# /__`-'_| || || ) ||# []/()] O || || | __`------------------'__ |--| |<=={_______}=|| |--| | | |-------------|| | | | | |={_______}==>|| | | | | | |: _ :| || | | > _| |___|:===:| || |__< :| | __| |: - :| || | |: :| | ==| |: _ :| || | |: :| | ==|_|:===:|___||_| |: :| |___|_|:___:|___||_| |: :| ||| ||/_\|| ||| -| |: ;I_|||[]_||\_/|| ||| -|_I; |_ |__________________| _| | `\\\___|____|____/_//' | J : | \____/ | : L _|_: | |__| | :_|_ -/ _-_.' -/ \- `.-_- \- /______\ /______\ /______\