Etherpad-lite: Migrate data from dirtydb to mysql

Recently I’ve been playing with etherpad-lite, which is a port of Etherpad (a collaborative editor a’la Google Wave acquired by Google and then open-sourced) to node.js platform.

As my node.js project is getting ready for production use, I have decided to migrate from the default dirtydb to mysql. Dirtydb is a fun little database written in JavaScript that just appends key/value pairs to a json-formatted text files and then basically reads in backwards to find a matching key if you need to query it. Super-simple and works nicely. The author claims it can handle up to 1M records.

I would happily stay with dirtydb , but etherpad-lite adds new revisions of the pad on every character typed, so 1M records can be reached pretty quickly with it, so I have decided to move to mysql as a backend.

Etherpad-lite uses a DB abstraction library ueberDB (the name probably should have been überDB , but the author didn’t have the ‘ü’ on his keyboard? ;-)), which basically mimics the dirtydb interface, but can use other backends, like mysql or sqlite for actual storage.

You configure etherpad-lite/ueberDB to use mysql in the settings.json file:

"dbType" : "mysql",
"dbSettings" :  {
    "user": "etherpad",
    "host": "localhost",
    "password": "etherpad",
    "database": "etherpad"
   },
}

This already works provided you have created the necessary mysql database and user:

- mysqladmin create etherpad -u root -p
- grant all privileges on etherpad.* to etherpad identified by 'etherpad';

But if you have been using the app with dirtydb for some time, you probably want to migrate the data from dirtydb to mysql , here is a simple script to do just that:

var dirty = require("dirty")("../var/dirty.db")
var db = require("./db/DB")
db.init(function () {
  db = db.db
  dirty.on("load", function () {
    dirty.forEach(function (key, value) {
      db.set(key, value)
    })
  })
})

Save it as [etherpad-root]/node/dirty2mysql.js, then

- cd node
- node dirty2mysql.js

If it doesn’t do anything, try installing the dirtydb module explicitly (not as a dependency of ueberDB):

- npm install dirty

And you’re done!