Tag Archives: Innostore

Hello, Bitcask

April 27, 2010

because you needed another local key/value store

One aspect of Riak that has helped development to move so quickly is pluggable per-node storage. By allowing nearly anything k/v-shaped to be used for actual persistence, progress on storage engines can occur in parallel with progress on the higher-level parts of the system.

Many such local key/value stores already exist, such as Berkeley DB, Tokyo Cabinet, and Innostore.

There are many goals we sought when evaluating which storage engines to use in Riak, including:

  • low latency per item read or written
  • high throughput, especially when writing an incoming stream of random items
  • ability to handle datasets much larger than RAM w/o degradation
  • crash friendliness, both in terms of fast recovery and not losing data
  • ease of backup and restore
  • a relatively simple, understandable (and thus supportable) code
    structure and data format
  • predictable behavior under heavy access load or large volume
  • a license that allowed for easy default use in Riak

Achieving some of these is easy. Achieving them all is less so.

None of the local key/value storage systems available (including but not limited to those written by us) were ideal with regard to all of the above goals. We were discussing this issue with Eric Brewer when he had a key insight about hash table log merging: that doing so could potentially be made as fast or faster than LSM-trees.

This led us to explore some of the techniques used in the log-structured file systems first developed in the 1980s and 1990s in a new light. That exploration led to the development of bitcask, a storage system that meets all of the above goals very well. While bitcask was originally developed with a goal of being used under Riak, it was also built to be generic and can serve as a local key/value store for other applications as well.

If you would like to read a bit about how it works, we’ve produced a short note describing bitcask’s design that should give you a taste. Very soon you should be able to expect a Riak backend for bitcask, some improvements around startup speed, information on tuning the timing of merge and fsync operations, detailed performance analysis, and more.

In the meantime, please feel free to give it a try!

- Justin and Dizzy

Using Innostore with Riak

February 22, 2010

Innostore is an Erlang application that provides an API for storing and retrieving key/value data using the InnoDB storage system. This storage system is the same one used by MySQL for reliable, transactional data storage. It’s a proven, fast system and perfect for use with Riak if you have a large amount of data to store. Let’s take a look at how you can use Innostore as a backend for Riak.

(Note: I assume that you have successfully built an instance of Riak for your platform. If you built Riak from source in ~/riak, then set $RIAK to ~/riak/rel/riak.”)

We first get started by grabbing a stable release of Innostore. You’ll need to download the source for a release from: https://github.com/basho/innostore

Looking in the “Tags & snapshots” section, you should download the source for the highest available RELEASE_* tag. In my case, RELEASE_4 is the most recent release, so I’ll grab the bz2 file associated with it.

Once I have the source code, it’s time to unpack it and build:

$ tar -xjf innostore-RELEASE_4.tar.bz2

$ cd innostore

$ make

Depending on the speed of the machine you are building on, this may take a few minutes to complete. At the end, you should see a series of unit tests run, with the output ending:

All 7 tests passed.

100222 7:43:58 InnoDB: Shutdown completed; log sequence number 90283

Cover analysis: /Users/dizzyd/src/public/innostore/.eunit/index.html

Now that we have successfully built Innostore, it’s time to install it into the Riak distribution:

$ ./rebar install target=$RIAK/lib

If you look in the $RIAK/lib directory now, you should see the innostore-4 directory alongside a bunch of .ez files and other directories which compose the Riak release.

Now, we need to tell Riak to use the Innostore driver as a backend. Make sure Riak is not running. Edit $RIAK/etc/app.config, setting the value for “storage_backend” as follows:

{storage_backend, innostore_riak},

In addition, append the configuration for the Innostore application after the SASL section:

{sasl, [ ....

]}, %% < -- make sure you add a comma here!!

{innostore, [

{data_home_dir, "data/innodb"}, %% Where data files go

{log_group_home_dir, "data/innodb"}, %% Where log files go

{buffer_pool_size, 2147483648} %% 2G in-memory buffer in bytes


You may need to adjust the directories for your data_home_dir and log_group_home_dirs to match where you want the inno data and log files to be stored. If possible, make sure that the data and log dirs are on separate disks — this can yield much better performance.

Once you’ve completed the changes to $RIAK/etc/app.config, you’re ready to start Riak:

$ $RIAK/bin/riak console

As it starts up, you should see messages from Inno that end with something like:

100220 16:36:58 InnoDB: highest supported file format is Barracuda.

100220 16:36:58 Embedded InnoDB started; log sequence number 45764

That’s it! You’re ready to start using Riak for storing truly massive amounts of data.


Dave Smith