- Alpha One
- Tagging
- Limiting
- Browser Console / ANSI Colors
- Options
- Encryption
- 'Easy DB'
- NPM & GitHub Publishing Helpers, Module Administration
- Persisting, Responsive Long Running Processes
- Jizura Data Queries
- Paper Publishing
- Publication Helpers / Build Tools
- Git Automation - Rejected - NodeGit - Jake
- NodeJS Version Management
- Application Architecture & Design
Table of Contents generated with DocToc
Alpha One
About: This is a meta-project to collect scattered thoughts about and working solutions for recurring tasks in Web- and Backend-Application building. Do not expect polished, working code, but rather fragmentary, sketchy hints.
The project takes its name from the 1970s British TV series Moonbase Alpha / Space: 1999.
Tagging
Limiting
Rationale: We often want to restrict access to the ressources a web application publishes; for example, you want subscribers to have unlimited access, guest users to enjoy a realistic peek into your offerings without handing over the full power of your application and the data therein, and at the same time deter any automated downloads. One part of the equation that can make this happen is User Authentication; the other is identifying anonymous visitors and selectively throttle or block their access, based on automatically collected behavior details and / or manual black- and whitelisting.
-
A limiter should be completely agnostic as to the task it is limiting—it should help to generally put limits to ressource usage, not, say, be built to specifically limit HTTP requests (a drop-in middleware could do that–one that uses a generic limiter).
-
Because of its genericity, a limiter should work with arbitrary ID tokens that represent clients—these could be any piece of data (an internal user ID, an IP address, a session ID) that is fit to identify a client in the sense of the application.
-
Since a limiter should be able to base its behavior on the past behavior of a given client and on the client's ranking (say, customer vs. guest vs. rogue), it must either entertain a suitable datastructure to keep such group affiliation and usage data (or else be fed (some of) these details when called).
-
Whether or not a limiter should take care of data persistence is an open question; ideally, server restart or redirection to another server process should not impact limiter behavior (this consideration would appear to favor Redis-like data persistence plans).
-
A limiter should distinguish between and be configurable for at least three behavioral patterns: full access, deferred (throttled) access, and denial of service (the last two with timeouts and permanent).
CoffeeNode Limit is an sttempt to bring all of the above points to the backend. It currently works with a slightly patched version of node-rate-limiter.
Browser Console / ANSI Colors
-
SCRATCH/alpha-one/src/ANSI256.coffee
-
https://github.com/Guilda/console-output-colorizer/blob/master/src/jquery.colorizeConsoleOutput.js
-
SCRATCH/includer/index.html
-
http://replit.github.io/jq-console/ (relies on an outdated version of jQuery; doesn't decode ANSI colors correctly; makes strange assumption about browser console element styling)
Options
Command Line Option Parsing
-
coffeenode-options
Using path-extra
to get the canonical OS-dependent location for user-level application option files:
### https://github.com/jprichardson/node-path-extra ###njs_path = require 'path-extra'info njs_pathtempdirinfo njs_pathhomedirinfo njs_pathdatadir 'app-name-here'
Encryption
See discussion of initialization vector (iv):
-
src/encryption.coffee
General Data
Probably use openpgpjs or sjcl:
- https://github.com/chengxianga2008/node-cryptojs-aes
- http://openpgpjs.org/
- http://bitwiseshiftleft.github.io/sjcl/
- https://github.com/srijs/rusha
- https://github.com/h2non/jshashes
User Authentication & Management
The specific purpose of password encryption should probably be done using bcrypt, for which there are 100% JS implementations available. It does have the advantage of adaptable algorithmic complexity. APIs typically do not offer decryption, only comparison of data.
https://github.com/jed/authom
Jed Schmidt writes in the README:
since it doesn't handle logins, persistence, sessions, or anything past authentication, it is more of a tool and less of a framework
'Easy DB'
- SCRATCH/test-nedb.coffee
- https://github.com/louischatriot/nedb
NPM & GitHub Publishing Helpers, Module Administration
Our erstwhile solution to preparing CoffeeScript source files for both development and production, larq, is very much on the way out.
It still has one distinctive feature, though, and that is larq dev
, which compiles a JavaScript file for
each CoffeeScript file that does not contain the actual code, but rather dynamically loads (and compiles)
the CoffeeScript source; in this way, the CoffeeScript sources may change any time, while to outside
consumers, everything appears as though code was actually loaded from JavaScript files, from the locations
where those JavaScript loader files are located.
Thus, every effort is made to make the existence of CoffeeScript sources transparent; the result is
convenience during development and assurance that production versions of the same code (produced with larq pub
or coffee --compile
) are fully functional (even in the absence of the CS sources—the plan is to
publish those, too, though, so potential contributors get a chance to work on either JS or CS, whatever they
feel most comfotable with).
I started larq ('Language-Aware ReQuire') when
I
realized that the global require.extensions
hook in NodeJS has huge problems—long story short, it's a global hook, so if any of your dependencies have
different ideas on how to resolve a particular filename extensions, you're BFE. It would be fine if it was
some very local thing, but as it stands, its not, and care must be exercised.
There are some thoughts on
how to repair this,
but so far no overarching facility or best-practices solution has emerged, short of the advisory to compile
your 3rd-party-language sources down to the standard ones (*.js
, *.css
, *.html
) and only ever deal
with these compilation targets in production.
A possible, maybe makeshift solution is to use CoffeeNode Monitor or similar to watch all sources and compile on change; this may prove a better alternative than trying to do the same, but in more convoluted ways, using yet another tool (larq).
Persisting, Responsive Long Running Processes
Rationale: When we want to have a single HTTP serving process to run on a production server, we can accomplish that by daemonizing the server process. Having more than one process and processes that are resumed after abortion requires more work. On a development machine, we want to typically have a single HTTP server to be run explicitly (not automatically on system startup), and, when any of its dependencies have been modified, we want to re-run any build steps and then restart the serving process. In case of syntax or runtime errors, we typically would like the monitoring instance to wait for any subsequent changes to the codebase and then try and re-run the build steps. A server is a Long Running Process—as distinguished from, say, a backup script that runs once a day for a minute or so. A Responsive Process is one that responds to (rather, is managed so it can respond to) changes in source files. A Persisting Process is one that is resumed on termination.
- coffeenode-monitor
Jizura Data Queries
- coffeenode-mingkwai
- coffeenode-solr
- coffeenode-mojikura
Paper Publishing
- XeLaTex
- https://www.google.de/search?q=xelatex+fraktur&ie=utf-8&oe=utf-8&rls=org.mozilla:en-US:official&client=firefox-a&gws_rd=cr&ei=LyBkUqKRF8GYtQbznIHYBA
- coffeenode-tex
Publication Helpers / Build Tools
Generate TOC for your README.md
with npm install --global doctoc
:
-
cake (part of CoffeeScript)
Tested:
-
OSX GitHub app
Git Automation
Git Automation
Rejected
NodeGit
- http://www.nodegit.org/, https://github.com/nodegit/nodegit (sadly, does not compile under NodeJS 0.11.7)
-1 for not updating their examples on their homepage. -1 for the Pyramids of Doom in their examples. -1 for
an API that makes the programmatical equivalent of git add --all; commit -m "foo"
look like rocket
science. -1 for embracing asynchronicity so damn hard you'll get a hundred callbacks when you ask for a
commit history. -1 for embracing Object-Oriented Programming so damn hard that you'll never get an answer to
your question but an object whose methods you must call—to asynchronously receive another object on which to
call more methods.
Jake
Cake is better.
NodeJS Version Management
Rationale: With the advent of breaking changes in NodeJS unstable 0.11.x, many modules relying on gyp
/ waf
do not compile any more. Some tools (such as browserify) compile with 0.11.7 but not with 0.11.8.
Consequently we must be able to switch between NodeJS versions in a swift and painless manner; n by
visionmedia is one way to do that.
After the painless install: npm install --global n
, you get a new executable n
that lets you install and
/ or use new versions with commands like n latest
, n stable
or n 0.11.7
. Simply executing n
yields a
menu in the terminal that lets you choose one of the installed versions. The global node
command is always
re-bound to the last chosen version with n
; in order to get commands for specific versions, cd
into a
directory on the path and do something along the lines of:
ln -s /usr/local/n/versions/0.11.7/bin/node node0117ln -s /usr/local/n/versions/0.11.7/bin/node node-latestln -s /usr/local/n/versions/0.10.22/bin/node node-stable
Application Architecture & Design
- https://github.com/mikaelbr/node-notifier
- brew install terminal-notifier
Control Flow
And the Winner is: Yield!
NB CoffyScript is still (highly) experimental; its use in underlying libraries is especially strongly
discouraged. CyS is currently implemented as a random patch of (an earlier version of) CoffeeScript 1.6.3
which has been shown to posses some subtle bugs (especially faulty error reporting that does not point out
the source of the error). There was an attempt to upgrade to the latest bugfix editions of CoffeeScript, but
the code had already suffieciently changed to make simple copy-and-paste of the changes unviable. The CS
Redux project is, unfortunately, not yet advanced /
adopted enough for me switch over, so i decided to just wait it out: wait for yield
to become the
available in a stable Node release (i.e. 0.12 or 1.x) and wait for CS Redux to land in the master branch of
CoffeeScript.
With all the caveats, CoffyScript is already usable and has greatly helped to write some very readable, working code that feels magnitudes better than the Promises/A+ ridden code it replaced.
The Journey Continues
- caolan async
Making Things Synchronous
Sometimes the solution to async woes lies in choosing an underlying functionality provider that is
synchronous—for example, when doing application startup stuff like compiling sources and so on, you might
choose fs.writeFileSync
over fs.writeFile
because (1) it lets you write linear code, (2) you can't run
your app before this step has completed anyhow. Using synchronous IO is an antipattern in online serving,
but its much more of a best practice for a startup sequence.
In the same vein, it may be desirable to execute shell commands in a synchronous fashion. Sadly, this piece
of functionality is not included in NodeJS proper;
fortunately, there are modules to do that,
like execSync.
It compiles with warnings under NodeJS 0.11.7 on my OSX box but appears to work, so now you can do
TRM = require 'coffeenode-trm'
, result = TRM.execute 'ls'
alongside with
TRM.execute 'ls', ( error, result ) -> ...
.
Rejected
Promises
Promises, with all the
neat /A+
marketing ribbons tacked unto them,
are horrible IMHO.
They come fraught with conceptual burden (eg. promises vs. deferreds), force boilerplate on you (never forget to
call .done()
when you're done), and they urge you to wrap all of your callback-based functions so as to make them play nice (which is not nice—callbacks were there first, and they are the
conceptually simpler mechanism. You just threw out the trusty hammer, now you got a beeping flashy gadget).
And: The API of the leading promises implementation is huge. If you insist on doing promises, i'll ask you to memorize this 60+ items list:
Q.all
, Q.allSettled
, Q.async
, Q.defer
, Q.delay
, Q.denodeify
, Q.getUnhandledReasons
,
Q.isPromise
, Q.longStackSupport
, Q.makePromise
, Q.nbind
, Q.nextTick
, Q.nfapply
, Q.nfcall
,
Q.ninvoke
, Q.npost
, Q.onerror
, Q.promise
, Q.promised
, Q.reject
, Q.resetUnhandledRejections
,
Q.spawn
, Q.stopUnhandledRejectionTracking
, promise.all
, promise.allSettled
, promise.catch
,
promise.delay
, promise.delete
, promise.dispatch
, promise.done
, promise.fapply
, promise.fbind
,
promise.fcall
, promise.finally
, promise.get
, promise.inspect
, promise.invoke
,
promise.isFulfilled
, promise.isPending
, promise.isRejected
, promise.keys
, promise.nodeify
,
promise.post
, promise.progress
, promise.set
, promise.spread
, promise.then
, promise.thenReject
,
promise.thenResolve
, promise.timeout
, deferred.promise
, deferred.resolve
, deferred.reject
,
deferred.notify
, deferred.makeNodeResolver
.
You're of course not done yet—care to learn that (from the
API docs)
"Q.when( 5, onFulfilled )
is equivalent to Q( 5 ).then( onFulfilled )
"? I never wanted to know this.
One might argue that this particular API is just the result of some particular diligent programmer going somewhat over the top, but that doesn't devalidate the argument that there is more than a fair number of promises libraries out there, some of them broken, others with uncertain merits. Finding out how the ~30 libraries listed alongside the /A+ specs interact and compare in terms of features, performance and memory consumption is not something i want to spend my time with. And interaction between promises libraries is a real concern. To cite a proponent of promises (emphases mine):
As authors of Promises/A-consuming libraries, we would like to assume this statement to be true: that something that is “thenable” [meaning you can say
x.then()
on it] actually behaves as a Promises/A promise, with all the power that entails.If you can make this assumption, you can write very extensive libraries that are entirely agnostic to the implementation of the promises they accept! Whether they be from Q, when.js, or even WinJS, you can use the simple composition rules of the Promises/A spec to build on promise behavior. For example, here's a generalized retry function that works with any Promises/A implementation.
Unfortunately, libraries like jQuery break this. This necessitates ugly hacks to detect the presence of objects masquerading as promises, and who call themselves in their API documentation promises, but aren't really Promises/A promises. If the consumers of your API start trying to pass you jQuery promises, you have two choices: fail in mysterious and hard-to-decipher ways when your compositional techniques fail, or fail up-front and block them from using your library entirely. This sucks.
Interesting to hear that the jQuery team tried to implement promises and failed. Interesting, too, that
(some) promises library authors (according to the above statement) appearently try to live up to the
duck typing dream—i.e. they rationalize "this must be a
promises object!" when they get a return value x
where x.then
happens to be a function. That's about
as reliable as saying "this must be a cinema!" because "it has seats inside".
Frankly, as much as asynchronous calls and Pyramids of Doom tend to be a PITA, promises look and feel wrong.
Untested
Standalone Apps & Isomorphic Modules
- https://github.com/spikebrehm/isomorphic-tutorial
- SCRATCH/includer/index.html
- https://github.com/rogerwang/node-webkit
- /Volumes/Storage/temp/nw-sample-apps
- https://github.com/alloy/terminal-notifier
Rationale. NodeJS makes it possible to use the same code on the server and in the browser; tools like
node-webkit
make it possible to run one-file downloads as no-installation / 'portable' desktop apps (with
the VM (runtime) either bundled or installed globally on the target machine). This spells good-bye to all
(or at least most) of the many competing GUI toolkits (if and where a 70MB minimum size for the VM is
acceptable, where a given platform is supported, and where performance is sufficient).
Tested:
Untested
(Web) Application (Backend) Design
To Framework or Not To Framework?
Tero Piirainen writes under the title Technology lock-in:
If you look at the history of frameworks in any programming language, it's a history of failures. Frameworks come and go.
The Stack
Ordered from the earliest, most general, down to the latest, most specific steps, the 'stack of actions' in a web application may be sketched as follows:
- Preparation
- Limit (blocking)
- Routing & Query Parsing
- Favicon
- Static
- Authentication
- Limit (throttle)
- App
- Response Initiation
- Session Handling
- DB Access
- Template (View) Building
- Finalization
- Response Finalization
Modern Wire Communication Protocols
Chunked HTTP
- what it means for templating
WebSockets
- http://jinzhang.me/posts/2013/sockjs-redis-nodejs-tutorial/
- http://pyvideo.org/video/1798/make-more-responsive-web-applications-with-socket
- http://strongloop.com/strongblog/real-time-engines-in-node-js/
Web Application Frontend Design
Icons &c
Web Application Deployment
Untested
HTLM5 & CSS3
Flexboxes
Probably better than using float
:
Custom HTML Elements
Rationale: getting away from <div class='foo'><div class='bar'>
, moving towards <foo><bar>
.
Probably need a polyfill for IE compatibility; without namespacing, custom tags may face problems with future updates to HTML5 standard.
Alpha-One Design Philosophy (Sketches)
Templates, Layouts, Views
A template is a function that accepts a request
and a response
argument and returns HTML
as text (or as a POD—not yet implemented).
A layout is a function that accepts a request
, a response
, and a content
argument; it returns
HTML as text (or as a POD—not yet implemented).
A view is a middleware method that accepts a request
, a response
, and a done
argument; it gets
bound to a constant or parametrized HTTP route. Views are somewhat special in that they can act either in
a synchronous or a asynchronous fashion.
-
Asynchronous views must return
undefined
ornull
to signal that output should not yet be sent to the client (this is expecially important to keep in mind when using CoffeeScript, as functions may be compiled with an explicitreturn
statement, so it's good practice to end asynchronous view methods with an explicitreturn null
). -
**Asynchronous views must call
done content
when they're ready to, otherwise the client will hang on waiting indefinitely for a response. -
Synchronous views must return a text; no other return values are allowed at this point in time, although it conceivably makes sense to return an object that represents some content and may be manipulated before beign serialized and sent to the client.
After a view has returned a text, it must not call done
; this is only allowed once. Conversely, once
done
has been called, returning anything but undefined
or null
is not permitted.
-
We currently use teacup as a templating engine, but this may change in the future.
-
Typically, we have one separate file for templates called, unsurprisingly,
templates.coffee
; within that module, there is a boilerplate headerfor name_ of teacup eval "#{name_.toUpperCase()} = teacup[ #{rpr name_} ]"
that 'pulls' all teacup methods as uppercase variables into the module-local namespace.
-
Templates and views do not use
teacup.renderable
; instead, they typically have, near the top of the function, a linereturn RENDER => ...
; this has proven to work better for our purposes (it's also less surprising thanRENDERABLE
). -
There is some boilerplate near the top of each template and layout:
O = get_options request page_style = O[ 'page-style' ] ? 'plain' title = O[ 'title' ] ? 'welcome'
and a module-local function
get_options = ( request ) -> return request[ app_key ] ? {}
providing a way to get view options from the request object.
app_key
is a text variable shared between the provider (the middleware stack) and the consumers (the templates and layouts) of per-request data; it should be set to a string that can be assumed to be unique within the given middleware stack (in the example, we chosealpha-one
). -
As a matter of convention,
O[ 'layout' ]
may contain a function that should be used instead oftemplates.layout
. -
Here is how templates are turned into views.—In
app/main
, there is a functionview
:view = ( view_name ) -> return ( request, response ) -> http_ok request, response response.write templates[ view_name ] request response.end()
It is used like this:
app.get '/', view 'homepage' app.get '/welcome', view 'welcome' app.get '/goodbye', view 'goodbye'
TAINT As becomes immediately apparent, there is no support to emit any other HTTP status code but
200
at this point in time—no redirects, no 404s are possible with this kind of setup; 'all sales are final'; when the given route matches, the templates is executed and its result are sent back with an HTTP200 OK
. Also, nonext
method to skip the view is present. Such functionality may or may not get integrated into theview
function at some point in the future; right now the way to do it is to implement a middleware function yourself (it is easy) and not useview
for this purpose.
User Authentication
Redirect, Bounce, and Back-To
There are three functions concerned with HTTP redirection: redirect
, bounce
, and back_to
.
-
redirect request, response, location
is just the plain Expressresponse.redirect
function (it is conceivable that some functionality might be added to this function so maybe prefer it over the Express method). -
bounce request, response, location
is likeredirect
, but it (1) issues a flash informing the user about the redirection (which your app may choose to silently ignore), and, (2) more importantly, sets a cookie value that contains the location from where the redirect was issued. -
back_to request, response, location
is likeredirect
, but it tries to retrieve the original location from the cookie set bybounce
; if none can be found, it redirects tolocation
. So say you go to a restricted arehttp://app/nogo
, abounce
to/login
will occur; after successfully entering credentials, you will goback_to
/nogo
. If you went straight to/login
or you intentionally deleted the cookie in the process and thelogin
middleware ends with the lineback_to request, response, '/welcome'
, you will end up at a catch-all welcome page instead. (Implementation details discussed here may change; it is conceivably cleaner to store thebounce
address in a session instead of in a cookie).