restify-git-json
An API to git repos exposed as a RESTful interface.
Uses libraries to package up js-git as a set of streaming interfaces. Uses git streams to provide a RESTful API ala github to inspect, create, and manipulate git repos.
Includes an endpoint to upload files, streaming them into a uniquely identified git reference per user per namespace per upload.
Install
$ git clone git@github.com:tidepool-org/restify-git-json.git# or $ git clone git://github.com/tidepool-org/restify-git-json.git$ cd restify-git-json$ npm install
Run
$ node server.js
Environment variables
-
PORT Expects to be assigned a tcp port to serve http requests on via
PORT
environment variable. Default is6776
. -
BASE the directory housing bare git repos if using
fs-db
backend. Default value is./out
. -
GIT_BACKEND
fs-db
(default) ormemdb
.
Choosing a git backend
fs-db
Bare file database Each git database is stored on disk using the "bare" repo layout (no working directory, no files are left "checked out"). See git repository layout for more information.
The current advice is to use this one if you aren't sure.
memdb
Memory based database The git database is stored in memory, which is cool, but you can only get the
contents via the http endpoints supported by the server. (Cloning is
post-mvp.) See lib/handlers/experiments
for WIP.
API
HTTP REST Endpoints
Basic idea is to wrap a configurable git repo (backed by s3, memory, or bare fs) using js-git, providing a RESTful API identical to the github json API.
Basic pattern is:
METHOD ENDPOINT DescriptionGET /repos/:owner/:repo/git/:typeGET /repos/:owner/:repo/git/refsGET /repos/:owner/:repo/git/refs/:refGET /repos/:owner/:repo/git/refs/:refGET /repos/:owner/:repo/git/commits/:shaGET /repos/:owner/:repo/git/trees/:shaGET /repos/:owner/:repo/git/blobs/:sha
Uploads
As a bonus, we expose an additional endpoint
METHOD ENDPOINT DescriptionPOST /repos/:owner/:repo/upload
This pins uploads to a uniquely identified branch. The result can then be cloned for replication.
The semantics for working with this endpoint are the exact same as
normal browser file uploads. As a bonus feature, if a header with
content-md5
is sent, the
checksum of the file is validated
before being saved. For all cases, sha1 sums are provided for each
upload.
Caveats cloning is currently only possible when using fs bare option (default).
Example
EG
+ curl -ivs -F web=@./History.md -F kkktwo=@env.js localhost:6776/repos/me/test/upload+ json* About to * Trying 127.0.0.1... connected> POST /repos/me/test/upload HTTP/1.1> User-Agent: curl/7.22.0 libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3> Host: localhost:6776> Accept: */*> Content-Length: 2911> Expect: 100-continue> Content-Type: multipart/form-data; boundary=----------------------------124e1b80ed69> < HTTP/1.1 100 Continue} [data not shown]< HTTP/1.1 201 Created< Connection: close< Content-Type: application/json< Content-Length: 866< Date: Wed, 15 Jan 2014 05:58:03 GMT<
So each request like this creates a new set of blobs on a new uniquely identified branch, per upload per namespace (repo) per user (per server).
Installed routes
+ curl -ivs localhost:6776/help+ json* About to * Trying 127.0.0.1... connected> GET /help HTTP/1.1> User-Agent: curl/7.22.0 libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3> Host: localhost:6776> Accept: */*> < HTTP/1.1 200 OK< Connection: close< Content-Type: application/json< Content-Length: 998< Date: Wed, 15 Jan 2014 06:01:29 GMT<
Use API to fetch data
+ curl -ivs http://localhost:6776/repos/me/proof/git/refs+ json -H* About to * Trying 127.0.0.1... connected> GET /repos/me/proof/git/refs HTTP/1.1> User-Agent: curl/7.22.0 libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3> Host: localhost:6776> Accept: */*> < HTTP/1.1 200 OK< Connection: close< Content-Type: application/json< Content-Length: 2452< Date: Thu, 26 Dec 2013 00:05:36 GMT<
Get ref
+ curl -ivs http://localhost:6776/repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-57540359/36ca88+ json -H* About to * Trying 127.0.0.1... connected> GET /repos/me/proof/git/refs/heads/upload/incoming/2013-12-25-57540359/36ca88 HTTP/1.1> User-Agent: curl/7.22.0 libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3> Host: localhost:6776> Accept: */*> < HTTP/1.1 200 OK< Connection: close< Content-Type: application/json< Content-Length: 184< Date: Thu, 26 Dec 2013 00:00:56 GMT<
Get commit
+ curl -ivs http://localhost:6776/repos/me/proof/git/commits/36ca88c51023c38b135b243fe054da98d3eb6ac5+ json -H* About to * Trying 127.0.0.1... connected> GET /repos/me/proof/git/commits/36ca88c51023c38b135b243fe054da98d3eb6ac5 HTTP/1.1> User-Agent: curl/7.22.0 libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3> Host: localhost:6776> Accept: */*> < HTTP/1.1 200 OK< Connection: close< Content-Type: application/json< Content-Length: 402< Date: Thu, 26 Dec 2013 00:01:44 GMT<
Get tree
+ json -H+ curl -ivs http://localhost:6776/repos/me/proof/git/trees/1daf69f11ee1f5ff253e1a3300839cf6c7be710f* About to * Trying 127.0.0.1... connected> GET /repos/me/proof/git/trees/1daf69f11ee1f5ff253e1a3300839cf6c7be710f HTTP/1.1> User-Agent: curl/7.22.0 libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3> Host: localhost:6776> Accept: */*> < HTTP/1.1 200 OK< Connection: close< Content-Type: application/json< Content-Length: 291< Date: Thu, 26 Dec 2013 00:02:02 GMT<
Get blob
+ curl -ivs http://localhost:6776/repos/me/proof/git/blobs/ca32e189ce9fc464583de225ae5170bc7b8bf776+ json -H -C content* About to * Trying 127.0.0.1... connected> GET /repos/me/proof/git/blobs/ca32e189ce9fc464583de225ae5170bc7b8bf776 HTTP/1.1> User-Agent: curl/7.22.0 libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3> Host: localhost:6776> Accept: */*> < HTTP/1.1 200 OK< Connection: close< Content-Type: application/json< Content-Length: 691< Date: Thu, 26 Dec 2013 00:02:34 GMT<
Inspired by
- https://github.com/creationix/js-git/blob/master/examples/create.js
- https://gist.github.com/bewest/8008603
- https://github.com/substack/stream-handbook#transform
- https://github.com/dominictarr/event-stream#eventstream
TODO
the rest
- update repo metadata
- update user metadata
ROADMAP
-
get POST json api endpoints working
- doublecheck API compatibility with github
-
test with memdb
- develop module for memcache/redis
- develop module for s3 (knox, request, use proxy to s3)
-
test git http support XXX: so far I have only tested using file based git bare repo... would prefer http support for mem and s3 modules
-
develop export module
- bundle with encrypt?
-
develop import module
-
develop remotes/push module
fetch content
+ curl -ivs http://localhost:6776/repos/me/test/raw/upload/incoming/2014-01-15-79083446/489841/History.md* About to * Trying 127.0.0.1... connected> GET /repos/me/test/raw/upload/incoming/2014-01-15-79083446/489841/History.md HTTP/1.1> User-Agent: curl/7.22.0 libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3> Host: localhost:6776> Accept: */*> < HTTP/1.1 200 OK< Connection: close< Content-Type: text/plain< Content-Length: 2192< Date: Wed, 15 Jan 2014 06:06:16 GMT<
Customize profile creation
+ json+ curl -isv -XPOST 'http://localhost:6776/users/crash/create?email=tesser@tidepool.org&name=Tessy+Tidepool&handle=secret&user\[email\]=example@tidepool.org&user\[name\]=Tessy+Tidepool&author\[name\]=Foo+Bar&author\[email\]=secrete@tidepool.io&committer\[name\]=blip+on+behalf+of+secret&committer\[email\]=blip%4Atideool.io&message=my+create+message'* About to * Trying 127.0.0.1... connected> POST /users/crash/create?email=tesser@tidepool.org&name=Tessy+Tidepool&handle=secret&user[email]=example@tidepool.org&user[name]=Tessy+Tidepool&author[name]=Foo+Bar&author[email]=secrete@tidepool.io&committer[name]=blip+on+behalf+of+secret&committer[email]=blip%4Atideool.io&message=my+create+message HTTP/1.1 > User-Agent: curl/7.22.0 libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Host: localhost:6776 > Accept: */* > < HTTP/1.1 201 Created < Connection: close < Content-Type: application/json < Content-Length: 656 < Date: Sat, 18 Jan 2014 18:43:31 GMT <
"name": "Tessy Tidepool" "handle": "secret" "email": "tesser@tidepool.org" "committer": "name": "blip on behalf of secret" "email": "blipJtideool.io" "author": "name": "Foo Bar" "email": "secrete@tidepool.io" "user": "crash" "url": "http://localhost:6776/users/crash" "updated": "ref": "upload/incoming/2014-01-18-38611838/538497" "sha": "538497ec4b9db2eca4c7eb63e9acd0ec234a2c9c" "head": "commit": "538497ec4b9db2eca4c7eb63e9acd0ec234a2c9c" "tree": "tree": "728aee4bbb551213b1cc829357da3ce118d2b55a" "author": "name": "Foo Bar" "email": "secrete@tidepool.io" "committer": "name": "blip on behalf of secret" "email": "blipJtideool.io" "message": "my create message"