ZongJi

A MySQL binlog listener running on Node.js.

ZongJi (踪迹) is pronounced as zōng jì in Chinese.

This package is a pure JS implementation based on mysql. It has been tested to work in MySQL 5.5, 5.6, and 5.7.

Latest Release

The latest release is v0.5.0, only supports Node.js from v8.

v0.4.7 is the last release which supports Node.js v4.x.

Quick Start

let zongji = new ZongJi({ /* ... MySQL Connection Settings ... */ });
 
// Each change to the replication log results in an event
zongji.on('binlog', function(evt) {
  evt.dump();
});
 
// Binlog must be started, optionally pass in filters
zongji.start({
  includeEvents: ['tablemap', 'writerows', 'updaterows', 'deleterows']
});

For a complete implementation see example.js...

Installation

• Requires Node.js v8+

  $ npm install zongji

• Enable MySQL binlog in my.cnf, restart MySQL server after making the changes.

  From MySQL 5.6, binlog checksum is enabled by default. Zongji can work with it, but it doesn't really verify it.

  # Must be unique integer from 1-2^32
  server-id        = 1
  # Row format required for ZongJi
  binlog_format    = row
  # Directory must exist. This path works for Linux. Other OS may require
  #   different path.
  log_bin          = /var/log/mysql/mysql-bin.log
  
  binlog_do_db     = employees   # Optional, limit which databases to log
  expire_logs_days = 10          # Optional, purge old logs
  max_binlog_size  = 100M        # Optional, limit log size
  

• Create an account with replication privileges, e.g. given privileges to account zongji (or any account that you use to read binary logs)

  GRANT REPLICATION SLAVE, REPLICATION CLIENT, SELECT ON *.* TO 'zongji'@'localhost'

ZongJi Class

The ZongJi constructor accepts one argument of either:

• An object containing MySQL connection details in the same format as used by package mysql
• Or, a mysql Connection or Pool object that will be used for querying column information.

If a Connection or Pool object is passed to the constructor, it will not be destroyed/ended by Zongji's stop() method.

If there is a dateStrings mysql configuration option in the connection details or connection, ZongJi will follow it.

Each instance includes the following methods:

Some events can be emitted in different phases:

Options available:

• By default, all events and schema are emitted.
• excludeSchema and excludeEvents take precedence over includeSchema and includeEvents, respectively.

Supported Binlog Events:

Event Methods

Neither method requires any arguments.

Important Notes

• 🌟 All types allowed by mysql are supported by this package.
• 🙊 64-bit integer is supported via package big-integer(see #108). If an integer is within the safe range of JS number (-2^53, 2^53), a Number object will returned, otherwise, will return as String.
• 👉 TRUNCATE statement does not cause corresponding DeleteRows event. Use unqualified DELETE FROM for same effect.
• When using fractional seconds with DATETIME and TIMESTAMP data types in MySQL > 5.6.4, only millisecond precision is available due to the limit of Javascript's Date object.

Run Tests

• install Docker
• run docker-compose up and then ./docker-test.sh

Reference

I learnt many things from following resources while making ZongJi.

• https://github.com/mysqljs/mysql
• https://github.com/felixge/faster-than-c/
• http://intuitive-search.blogspot.co.uk/2011/07/binary-log-api-and-replication-listener.html
• https://github.com/Sannis/node-mysql-libmysqlclient
• https://kkaefer.com/node-cpp-modules/
• http://dev.mysql.com/doc/internals/en/replication-protocol.html
• http://www.cs.wichita.edu/~chang/lecture/cs742/program/how-mysql-c-api.html
• https://github.com/jeremycole/mysql_binlog (Ruby implemenation of MySQL binlog parser)
• http://dev.mysql.com/doc/internals/en/date-and-time-data-type-representation.html

License

MIT