task-sharding
This project demonstrates sharding responsiblity between nodes in a cluster, using consistent hashing
Usage
TaskSharding class
const TaskSharding = TaskSharding; const taskSharding = selfNode: "myNodeIdentifier";; taskSharding;taskSharding;taskSharding;taskSharding;taskSharding;taskSharding;taskSharding;
Events
let currentlyAssigned = 0;taskShardingtaskSharding;
Running the example
Prerequisites
- docker-compose
- npm
Running
In order to run it, download and unzip the repository. Then run:
git clone https://github.com/LivePersonInc/task-sharding.gitcd task-shardingnpm inpm start
In the logs you can see the nodes' statements regarding their task responsablity. You can addd nodes to the cluster by opening another shell window and:
cd task-sharding/examplesdocker-compose scale app=10
In the logs you will see new nodes coming in and new work division. Then you can kill some of the nodes by cahnging the scale again:
cd task-sharding/examplesdocker-compose scale app=1
The output should look like this:
app_1 | 1 out of 12 tasks assigned - total 1 nodes: Assigned "id1"
app_1 | 2 out of 12 tasks assigned - total 1 nodes: Assigned "id246"
app_1 | 3 out of 12 tasks assigned - total 1 nodes: Assigned "id426"
app_1 | 4 out of 12 tasks assigned - total 1 nodes: Assigned "id66"
app_1 | 5 out of 12 tasks assigned - total 1 nodes: Assigned "id10"
app_1 | 6 out of 12 tasks assigned - total 1 nodes: Assigned "id11"
app_1 | 7 out of 12 tasks assigned - total 1 nodes: Assigned "id20"
app_1 | 8 out of 12 tasks assigned - total 1 nodes: Assigned "id25"
app_1 | 9 out of 12 tasks assigned - total 1 nodes: Assigned "id303"
app_1 | 10 out of 12 tasks assigned - total 1 nodes: Assigned "id111"
app_1 | 11 out of 12 tasks assigned - total 1 nodes: Assigned "id44"
app_1 | 12 out of 12 tasks assigned - total 1 nodes: Assigned "id12566"
app_1 | 12 out of 12 tasks assigned - total 1 nodes: Hashring updated.
# after docker-compose scale app=5
app_4 | 1 out of 12 tasks assigned - total 5 nodes: Assigned "id1"
app_4 | 2 out of 12 tasks assigned - total 5 nodes: Assigned "id426"
app_4 | 3 out of 12 tasks assigned - total 5 nodes: Assigned "id20"
app_4 | 4 out of 12 tasks assigned - total 5 nodes: Assigned "id111"
app_4 | 4 out of 12 tasks assigned - total 5 nodes: Hashring updated.
app_5 | 1 out of 12 tasks assigned - total 5 nodes: Assigned "id246"
app_1 | 11 out of 12 tasks assigned - total 5 nodes: Revoked id1
app_2 | 1 out of 12 tasks assigned - total 5 nodes: Assigned "id25"
app_5 | 2 out of 12 tasks assigned - total 5 nodes: Assigned "id10"
app_2 | 2 out of 12 tasks assigned - total 5 nodes: Assigned "id12566"
app_5 | 3 out of 12 tasks assigned - total 5 nodes: Assigned "id11"
app_2 | 2 out of 12 tasks assigned - total 5 nodes: Hashring updated.
app_1 | 10 out of 12 tasks assigned - total 5 nodes: Revoked id246
app_5 | 4 out of 12 tasks assigned - total 5 nodes: Assigned "id303"
app_1 | 9 out of 12 tasks assigned - total 5 nodes: Revoked id426
app_5 | 5 out of 12 tasks assigned - total 5 nodes: Assigned "id44"
app_1 | 8 out of 12 tasks assigned - total 5 nodes: Revoked id66
app_5 | 5 out of 12 tasks assigned - total 5 nodes: Hashring updated.
app_1 | 7 out of 12 tasks assigned - total 5 nodes: Revoked id10
app_1 | 6 out of 12 tasks assigned - total 5 nodes: Revoked id11
app_1 | 5 out of 12 tasks assigned - total 5 nodes: Revoked id20
app_1 | 4 out of 12 tasks assigned - total 5 nodes: Revoked id25
app_1 | 3 out of 12 tasks assigned - total 5 nodes: Revoked id303
app_1 | 2 out of 12 tasks assigned - total 5 nodes: Revoked id111
app_1 | 1 out of 12 tasks assigned - total 5 nodes: Revoked id44
app_1 | 0 out of 12 tasks assigned - total 5 nodes: Revoked id12566
app_1 | 0 out of 12 tasks assigned - total 5 nodes: Hashring updated.
app_3 | 1 out of 12 tasks assigned - total 5 nodes: Assigned "id66"
app_3 | 1 out of 12 tasks assigned - total 5 nodes: Hashring updated.
# after docker-compose scale app=1
examples_app_2 exited with code 137
examples_app_4 exited with code 137
examples_app_5 exited with code 137
examples_app_3 exited with code 137
app_1 | 1 out of 12 tasks assigned - total 1 nodes: Assigned "id1"
app_1 | 2 out of 12 tasks assigned - total 1 nodes: Assigned "id246"
app_1 | 3 out of 12 tasks assigned - total 1 nodes: Assigned "id426"
app_1 | 4 out of 12 tasks assigned - total 1 nodes: Assigned "id66"
app_1 | 5 out of 12 tasks assigned - total 1 nodes: Assigned "id10"
app_1 | 6 out of 12 tasks assigned - total 1 nodes: Assigned "id11"
app_1 | 7 out of 12 tasks assigned - total 1 nodes: Assigned "id20"
app_1 | 8 out of 12 tasks assigned - total 1 nodes: Assigned "id25"
app_1 | 9 out of 12 tasks assigned - total 1 nodes: Assigned "id303"
app_1 | 10 out of 12 tasks assigned - total 1 nodes: Assigned "id111"
app_1 | 11 out of 12 tasks assigned - total 1 nodes: Assigned "id44"
app_1 | 12 out of 12 tasks assigned - total 1 nodes: Assigned "id12566"
app_1 | 12 out of 12 tasks assigned - total 1 nodes: Hashring updated.
You can stop the servers using
cd task-shardingnpm stop
Contributing
In lieu of a formal styleguide, take care to maintain the existing coding style. Add unit tests for any new or changed functionality, lint and test your code.
-
To run lint and tests:
npm testnpm run lint