Tag Archives: Riak

Riak 1.0 Release Party Recap

November 17, 2011

The Riak 1.0 Release Party happened just over a week ago in San Francisco. It was an exceptional evening, and we were able to bring together the Basho Team and a huge number of local Riak community members to celebrate the release.

In addition to the excellent food, drinks, company, and conversation, we had two great talks. The first was delivered by Basho’s CTO Justin Sheehy and he did about 20 minutes on the origins of Basho and Riak, and precisely how and why we got to Riak 1.0. After Justin concluded, Dave “Dizzy” Smith, Basho’s Director of Engineering, closed things up with some passionate words about where Riak and Basho are going and why he’s excited to be a part of it.

Most importantly, if you weren’t able to attend, we recorded the talks so no one would miss out on the action. They are well worth the 30 minutes and at the end of it you can call yourself a “Riak Historian”. You can find the video below. We also took some photos of the event. Those are below, too.

Enjoy, and thanks for being a part of Riak.

The Basho Team

Riak 1.0 Is Officially Released!

September 30, 2011

We are absolutely thrilled to report that as of today, Riak 1.0 is officially released and ready for your production applications!

Riak 1.0 packages are available. Go download one. And then go read the release notes because they are extensive and full of useful information highlighting the great work Basho has done since the last release.

There is already a lot of literature out there on the release, so here are the essentials to get you started.

The High Level Awesome

For those of you who need a refresher on the release, this 1.0 Slide Deck will give you a quick overview of why you should be excited about it. The big-ticket features are as follows:

Secondary Indexing

In 1.0 we added the ability to build secondary indexes on your data stored in Riak. We developed this functionality because, quite frankly, people needed a more powerful way to query their data.

Riak Pipe And Revamped MapReduce

Riak’s MapReduce functionality isn’t anything new, but we did a lot of work in this release to make the system more robust, performant, and resistant to failures. Riak Pipe is the new underlying processing layer that powers MapReduce, and you’ll be seeing a lot of cool features and functionality made possible as a result of it in the near future.

Lager

Usability is a huge focus for us right now, and logging is something that’s less-than-simple to understand in Erlang applications. To that end, we wrote a new logging framework for Erlang/OTP called Lager that is shipping with 1.0 and drastically reduces the headaches traditionally associated with Erlang logging and debugging.

Search Integration

Riak Search has been a supported Basho product for several releases now, but until 1.0 you were required to build it as a separate package. In 1.0 we’ve merged the search functionality into Riak proper. Enabling it is a simple one line change in a configuration file. Do this and you’ve got distributed, full text search capabilities on top of Riak.

Support for LevelDB

Riak provides for pluggable storage backends, and we are constantly trying to improve the options we offer to our users. Google released LevelDB some months back, and we started to investigate it as a possible addition to our suite of supported backends. After some rigorous testing, what we found is that LevelDB had some attractive functionality and performance characteristics compared to our existing offerings (mainly Innostore), and it will be shipping in 1.0. Bitcask is still the default storage engine, but LevelDB, aside from being an alternative for key/value storage, is being used as the backend behind the new Secondary Indexing functionality.

Cluster Membership

One of the most powerful components of Riak is riak_core, the distributed systems framework that, among many others things, enables Riak to scale horizontally. Riak’s scalability and operational simplicity are of paramount importance to us, and we are constantly looking to make this code and system even better. With that in mind, we did some major work in 1.0 to improve upon our cluster membership system and are happy to report that it’s now more stable and scalable than ever.

And So Much More …

Riak 1.0 is a massive accomplishment, and the features and code listed above are just the beginning of what this release has to offer. Take some time to read the lengthy release notes and you’ll see what we mean.

These improvements are many months in the making, and the bug fixes, new features, and added functionality make Riak (in our humble opinion) the best open source database available today.

Thank You, Community!

We did our best to ensure that the community was as big a part of this release as possible, and there’s no way the code and features would be this rock-solid without your help. Thanks for your usage, support, testing, debugging, and help with spreading the word about Riak and 1.0.

And 1.0 is just the beginning. We’ll continue to refine and build Riak over the coming months, and we would love for you to be a part of it if you’re not already. Some ways to get involved:

Download Riak on Your Preferred Platform

Read the Riak Wiki

Watch Riak on GitHub

Thanks for being a part of Riak!

 

Riak Pipe – the New MapReduce Power

September 19, 2011

A few months ago, I announced the opening of Riak Pipe, as well as two goals for the project. With the upcoming 1.0 release of Riak, we have achieved the first goal: new clusters will use Riak Pipe to power their MapReduce processing. Existing clusters will also be able to migrate to using Riak Pipe, with no changes needed from the client perspective.

There are a few reasons you should be excited about running your MapReduce queries on Riak Pipe. First and foremost, Riak Pipe is designed as a work distribution system, and as such, it is better able to take advantage of the parallel resources available in the cluster. One small example of how Riak Pipe achieves this is simply by splitting the “map” phase processing into two steps: fetching the object from Riak KV, and transforming it. This allows the work of each step to happen in parallel; the next input will be fetched while the transformation of the last one is in progress.

Riak Pipe also recognizes that a cluster’s resources are finite, and that sometimes it’s better to delay one pile of work in order to make progress on another. Processing phases in Riak Pipe, called fittings, provide backpressure to fittings upstream from them by means of limiting the sizes of their input queues. The upstream fittings pause their processing when the downstream queues are full, freeing up system resources (or at least not increasing their consumption) to allow those downstream processes a chance to catch up.

Input queues are another example of Riak Pipe’s parallel resource use. Inter-phase results are delivered directly from a vnode running one stage to the vnode that will process them for the next stage. Since they are not forced through a single, central process, the memory of the entire cluster can be used to move them around, instead of requiring a single machine’s memory to handle them.

The KV-object fetching stage of the new Riak Pipe MapReduce system is also much more of a well-behaved KV user. That is, the requests it makes are much more fairly balanced with respect to regular Riak KV operations (get, put, etc.). This means MapReduce on Riak Pipe should have much less impact on the performance of rest of your Riak use.

Using Riak Pipe MapReduce is simple. Make sure that the setting {mapred_system, pipe} is in the riak_kv section of your cluster’s app.config, and then … just send MapReduce queries over HTTP or Protocol Buffers as you always have. The results should be the same. There are a few knobs you can tweak, which control batching of reduce phase evaluation, but the goal of this release was a 100% compatible implementation of the existing MapReduce functionality.

There is much more on the horizon for Riak Pipe, including more efficiency gains and exposing some of the new processing statistics it tracks, not to mention exposing more of its functionality beyond Riak KV’s MapReduce. We’re very excited about the future.

If you would like to learn more about Riak Pipe, in general, and get involved, I recommend paging through the README to get an idea of its structure, and then browsing the new Riak KV MapReduce code for some examples.

-Bryan

Secondary Indexes in Riak

September 14, 2011

Developers building an application on Riak typically have a love/hate relationship with Riak’s simple key/value-based approach to storing data. It’s great that anyone can grok the basics (3 simple operations, get/put/delete) quickly. It’s convenient that you can store anything imaginable as an object’s value: an integer, a blob of JSON data, an image, an MP3. And the distributed, scalable, failure-tolerant properties that a key/value storage model enables can be a lifesaver depending on your use case.

But things get much less rosy when faced with the challenge of representing alternate keys, one-to-many relationships, or many-to-many relationships in Riak. Historically, Riak has shifted these responsibilities to the application developer. The developer is forced to either find a way to fit their data into a key/value model, or to adopt a polyglot storage strategy, maintaining data in one system and relationships in another.

This adds complexity and technical risk, as the developer is burdened with writing additional bookkeeping code and/or learning and maintaining multiple systems.

That’s why we’re so happy about Secondary Indexes. Secondary Indexes are the first step toward solving these challenges, lifting the burden from the backs of developers, and enabling more complex data modeling in Riak. And the best part is that it ships in our 1.0 release, just a few weeks from now.

How Do Secondary Indexes Work?

Update: Secondary Indexes use the new style HTTP API. See the Riak Wiki for more details.

From an application developer’s perspective, Secondary Indexes allow you to tag a Riak object with some index metadata, and later retrieve the object by querying the index, rather than the object’s primary key.

For example, let’s say you want to store a user object, accessible by username, twitter handle, or email address. You might pick the username as the primary key, while indexing the twitter handle and email address. Below is a curl command to accomplish this through the HTTP interface of a local Riak node:

bash
curl -X POST
-H 'x-riak-index-twitter_bin: rustyio'
-H 'x-riak-index-email_bin: rusty@basho.com'
-d '...user data...'

http://localhost:8098/buckets/users/keys/rustyk

Previously, there was no simple way to access an object by anything other than the primary key, the username. The developer would be forced to “roll their own indexes.” With Secondary Indexes enabled, however, you can easily retrieve the data by querying the user’s twitter handle:

Query the twitter handle…

curl localhost:8098/buckets/users/index/twitter_bin/rustyio

Response…

{“keys”:["rustyk"]}

Or the user’s email address:

Query the email address…

curl localhost:8098/buckets/users/index/email_bin/rusty@basho.com

Response…

{“keys”:["rustyk"]}

You can change an object’s indexes by simply writing the object again with the updated index information. For example, to add an index on Github handle:

bash
curl -X POST
-H 'x-riak-index-twitter_bin: rustyio'
-H 'x-riak-index-email_bin: rusty@basho.com'
-H 'x-riak-index-github_bin: rustyio'
-d '...user data...'

http://localhost:8098/buckets/users/keys/rustyk

That’s all there is to it, but that’s enough to represent a variety of different relationships within Riak.

Above is an example of assigning an alternate key to an object. But imagine that instead of a twitter_bin field, our object had an employer_bin field that matched the primary key for an object in our employers bucket. We can now look up users by their employer.

Or imagine a role_bin field that matched the primary key for an object in our security_roles bucket. This allows us to look up all users that are assigned to a specific security role in the system.

Design Decisions

Secondary Indexes maintains Riak’s distributed, scalable, and failure tolerant nature by avoiding the need for a pre-defined schema, which would be shared state. Indexes are declared on a per-object basis, and the index type (binary or integer) is determined by the field’s suffix.

Indexing is real-time and atomic; the results show up in queries immediately after the write operation completes, and all indexing occurs on the partition where the object lives, so the object and its indexes stay in sync. Indexes can be stored and queried via the HTTP interface or the Protocol Buffers interface. Additionally, index results can feed directly into a Map/Reduce operation. And our Enterprise customers will be happy to know that Secondary Indexing plays well with multi data center replication.

Indexes are declared as metadata, rather than an object’s value, in order to preserve Riak’s view that the value of your object is as an opaque document. An object can have an unlimited number of index fields of any size (dependent upon system resources, of course.) We have stress tested with 1,000 index fields, though we expect most applications won’t need nearly that many. Indexes do contribute to the base size of the object, and they also take up their own disk space, but the overhead for each additional index entry is minimal: the vector clock information (and other metadata) is stored in the object, not in the index entry. Additionally, the LevelDB backend (and, likely, most index-capable backends) support prefix-compression, further shrinking ndex size.

This initial release does have some important limitations. Only single index queries are supported, and only for exact matches or range queries. The result order is undefined, and pagination is not supported. While this offers less in the way of ad-hoc querying than other datastores, it is a solid 80% solution that allows us to focus future energy where users and customers need it most. (Trust me, we have many plans and prototypes of potential features. Building something is easy, building the right thing is harder.)

Behind The Scenes

What is happening behind the scenes? A lot, actually.

At write time, the system pulls the index fields from the incoming object, parses and validates the fields, updates the object with the newly parsed fields, and then continues with the write operation. The replicas of the object are sent to virtual nodes where the object and its indexes are persisted to disk.

At query time, the system first calculates what we call a “covering” set of partitions. The system looks at how many replicas of our data are stored and determines the minimum number of partitions that it must examine to retrieve a full set of results, accounting for any offline nodes. By default, Riak is configured to store 3 replicas of all objects, so the system can generate a full result set if it reads from one-third of the system’s partitions, as long as it chooses the right set of partitions. The query is then broadcast to the selected partitions, which read the index data, generate a list of keys, and send them back to the coordinating node.

Storing index data is very different from storing key/value data: in general, any database that stores indexes on a disk would prefer to be able to store the index in a contiguous block and in the desired
order–basically getting as near to the final result set as possible. This minimizes disk movement and other work during a query, and provides faster read operations. The challenge is that index values rarely enter the system in the right order, so the database must do some shuffling at write time. Most databases delay this shuffling, they write to disk in a slightly sub-optimal format, then go back and “fix things up” at a later point in time.

None of Riak’s existing key/value-oriented backends were a good fit for index data; they all focused on fast key/value access. During the development of Secondary Indexes we explored other options. Coincidentally, the Basho team had already begun work to adapt LevelDB–a low-level storage library from Google–as a storage engine for Riak KV. LevelDB stores data in a defined order, exactly what Secondary Indexes needed, and it is actually versatile enough to manage both the index data AND the object’s value. Plus, it is very RAM friendly. You can learn more about LevelDB from this page on Google Code.

Want To Know More?

If you want to learn more about Secondary Indexes, you can read the slides from my talk at OSCON Data 2011: Querying Riak Just Got Easier. Alternatively, you can watch the video.

You can grab a pre-release version of Riak Version 1.0 on the Basho downloads site to try the examples above. Remember to change the storage backend to riak_kv_eleveldb_backend!

Finally keep an eye out for documentation that will land on the newly re-organized Basho Wiki within the next two weeks.

Rusty

A Preview Of Cluster Membership In Riak 1.0

September 9, 2011

Being a distributed company, we make a lot of videos at Basho that are intended for internal consumption and used to educate everyone on new features, functionality, etc. Every once and a while someone makes a video that’s so valuable it’s hard not to share it with the greater community. This is one of those.

This screencast is a bit on the long side, but it’s entirely worth it. Basho Software Engineer Joe Blomstedt put it together to educate all of Basho on the new cluster membership code, features, and functionality coming in the Riak 1.0 release (due out at the end of the month). We aim to make Riak as operationally-simple as possible to operate at scale, and the choices we make and code we write around cluster membership form the crux of this simplicity.

At the end of this you’ll have a better idea of what Riak’s cluster membership is all about, its major components, how it works in production, new commands that are present Riak 1.0, and much, much more.

And, if you want to dig deeper into what Riak and cluster membership is all about, start here:

* Download Riak 1.0 Pre-release 1
* Riak Core on GitHub
* Where To Start With Riak Core
* Join the Riak Mailing List

It should be noted again that this was intended for internal consumption at Basho, so Joe’s tone and language reflect that in a few sections.

Enjoy, and thanks for being a part of Riak.

The Basho Team

Riak at Formspring

August 8, 2011

Several weeks back at the San Francisco Riak Meetup, Tim Bart of Formspring delivered a great talk to a packed house all about how he and his team are using Riak for two new features they were rolling out to their more than 24 million users. I’m happy to report that the video is online, edited, and ready for viewing.

This one runs just over 28 minutes and is an excellent overview of what Formpsring is all about, why they chose Riak, and lessons learned (both good and bad) when developing against Riak. Tim also touches on what plans they have for Riak in the future and, as any thorough user of a given technology should, makes some suggestions for functionality he and his team would like to see in Riak.

A few notes:

  • At just over a minute into the talk, Tim references the “Riak and Scala at Yammer” talk.
  • There were several questions at the end of the talk but there was a problem with the microphone so we had to cut them off. If you have any questions, leave them in the comments and Tim will answer them when he has the opportunity. (Thanks, Tim!)

Enjoy.

Mark

What's New In Riak's Python Client?

August 4, 2011

There’s a whole slew of new and noteworthy features in today’s release of the Python client for Riak, and I thought it’d be a good idea for us to sit down and look at a bunch of them so I can add more detail to what is already in the release notes.

Test Server

Ripple has had an in-memory test server for a while now, and I thought the Python client should have something similar too. By the way, a lot of the features here draw heavy inspiration from Ripple in general, so credit where credit is due.

The basic idea is that instead of using a locally installed Riak instance with file system storage you use one that stores data in memory instead. This is not only faster than storing everything on disk, it makes it much easier to just wipe all the data and start from scratch, without having to restart the service. In short, this is a neat way to integrate Riak into your test suite.

All the test server requires is a local installation to use the libraries from and to steal some files to build a second Riak installation in a temporary directory. Let’s look at an example:

“`python
from riak.test_server import TestServer

server = TestServer()
server.prepare()
server.start()
“`

This will start a Riak instance in the background, with the Python part interacting with it through the Erlang console. That allows you to do things like wiping all data to have a minty fresh and empty Riak installation for the next test run:

python
server.recycle()

The TestServer class has a default of where to look for a Riak installation, but the path could be anywhere you put a Riak build you made from an official release archive. Just point it to that Riak installation’s bin directory, and you’re good to go:

python
server = TestServer(bin_dir="/usr/local/riak/0.14.2/bin")

You can also overwrite the default settings used to generate the app.config file for the in-memory Riak instance. Just specify a keyword pointing to a dictionary for every section in the app.config like so:

python
server = TestServer(riak_core={"web_port": 8080})

By default the test server listens on ports 9000 (HTTP) and 9001 (Protocol buffers), so make sure you adapt your test code accordingly.

Using Riak Search’s Solr-compatible HTTP Interface

One of the nice things about Riak Search is its Solr-compatible HTTP interface. So far, you were only able to use Riak Search through MapReduce. New in release 1.3 of the Python client is support to directly index and query documents using Riak Search’s HTTP interface.

The upside is that you can use Riak Search with a Python app as a scalable full-text search without having to store data in Riak KV for them to be indexed.

The interface is as simple as it is straight forward, we’ve added a new method to the RiakClient class called solr() that returns a small façade object. That in turn allows you to interact with the Solr interface, e.g. to add documents to the index:

python
client.solr().add("superheroes",
{"id": "hulk",
"name": "hulk"
"skill": "Hulksmash!"})

You just specify an index and a document, which must contain a key-value pair for the id, and that’s it.

The beauty about using the Solr interface is that you can use all the available parameters for sorting, limiting result sets and setting default fields to query on, without having to do that with a set of reduce functions.

python
client.solr().query("superheroes", "hulk", df="name")

Be sure to check our documentation for the full set of supported parameters. Just pass in a set of keyword arguments for all valid parameters.

Something else that’s new on the search integration front is the ability to programmatically enable and disable indexing on a bucket by installing or removing the relevant pre-commit hook.

python
bucket = client.bucket("superheroes")
if not bucket.search_enabled():
bucket.enable_search()

Storing Large Files With Luwak

When building Riagi, the application showcased in the recent webinar, I missed Luwak support in the Python client. Luwak is Riak’s way of storing large files, chunked into smaller bits and stored across your Riak cluster. So we added it. The API consists of three simple functions, store_file, get_file, and delete_file.

“`python
client.store_file(“hulk”, “hulk.jpg”)

client.get_file(“hulk”)

client.delete_file(“hulk”)
“`

Connection Caching for Protocol Buffers and HTTP

Thanks to the fine folks at Formspring the Python client now sports easier ways to reuse protocol buffer and even HTTP connections, and to make their use more efficient. All of them are useful if you’re doing lots of requests or want to reuse connections across several requests, e.g. in the context of a single web request.

Here’s a summary of the new transports added in the new release, all of them accept the same parameters as the original transport classes for HTTP and PBC:

  • riak.transports.pbc.RiakPbcCachedTransport
    A cache that reuses a set of protocol buffer connections. You can set a boundary of connections kept in the cache by specifying a maxsize attribute when creating the object.
  • riak.transports.http.RiakHttpReuseTransport
    This transport is more efficient when reusing HTTP connections by setting SO_REUSEADDR on the underlying TCP socket. That allows the TCP stack to reuse connections before the TIME_WAIT state has passed.
  • riak.transports.http.RiakHttpPoolTransport
    Use the urllib3 connection pool to pool connections to the same host.

We’re always looking for contributors to the Python client, so keep those pull requests coming!

Mathias

Follow Up To Riak And Python Webinar

August 3, 2011

Thanks to everyone who attended yesterday’s webinar on Riak and Python. If you missed the webinar, we’ve got you covered. Find a screencast of the webinar below, or check it out directly on Vimeo. Sorry that it’s missing the questions I answered at the end. I recapped the questions in written form below to make up for it.

We made the slides available for your viewing pleasure as well. But most importantly, have a look at our Python library for Riak, which got a lot of feature love lately, and at the source code for Riagi, the sample Django application. It utilizes Riak for session and file storage, and Riak Search storing user and image metadata.

Thanks to dotCloud for providing the hosting for Riagi. A little birdie told us they’re working on supporting Riak as part of their official stack soon.

The Python client for Riak wouldn’t exist without the community, so we want to say thank you for your contributions, and we’re always ready for more pull requests!

Keep an eye out for a new release of the Python client this week, including several of the new features shown in the webinar!

The two questions asked at the end were:

  • Is there a Scala client for Riak? Links relevant to the answer in the video: Outdated Scala Library, a more recent fork, and our official Java client.
  • Is Protocol Buffers more efficient than using HTTP? Answer is detailed in the webinar video.

Basho and Riak Invade OSCON Next Week

July, 22, 2011

This year’s OSCON has lots of chances for you to hear about Riak. And, the Basho team will be presenting some great new Riak features as well. OSCON Data is a special track of the conference, dedicated to all things open and data-oriented – sure to be a hotbed of cool new open source projects and updates on more established platforms like Riak.

Be sure to check out the following sessions:

Querying Riak Just Got Easier – Introducing Secondary Indices

Consistency or Bust: Breaking a Riak Cluster

Also – be sure to stay on the lookout for Basho community manager Mark Phillips – he’s always up to hear about how people are usuing Riak, directing the curious to the right resources – oh, and he always has some cool Riak schwag on hand.

See you there!

Introducing Lager – A New Logging Framework for Erlang/OTP

July 20, 2011

Hi. My name is Andrew Thompson and I’ve been working on Riak at Basho since March. I’ve been focused primary on various technical debt issues within Riak since starting. The largest project I’ve undertaken is Lager, a new logging framework for Erlang/OTP. Lager is actually my second logging framework for Erlang, and I’ve used my previous experience to make this one even better. I think Lager has some compelling features to offer not seen in other erlang logging frameworks and I’ll go over the highlights in this post.

So, why write another log framework for Erlang? There’s already several; error_logger itself, SASL, log4erl and riak_err to name a few. One of the key goals was to make logging friendlier; both to the end-user and to the sysadmin. Lager tries very hard to hide the traditional “giant error tuple of doom” from the user (unless they go looking for it). For example, here’s what happens when a gen_server dies with a badmatch error (with SASL running):

“`text
=ERROR REPORT==== 19-Jul-2011::17:50:10 ===
** Generic server crash terminating
** Last message in was badmatch
** When Server state == {}
** Reason for termination ==
** {{badmatch,{}},
[{crash,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}

=CRASH REPORT==== 19-Jul-2011::17:50:10 ===
crasher:
initial call: crash:init/1
pid:
registered_name: crash
exception exit: {{badmatch,{}},
[{crash,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}
in function gen_server:terminate/6
ancestors: []
messages: []
links: []
dictionary: []
trap_exit: false
status: running
heap_size: 377
stack_size: 24
reductions: 127
neighbours:
“`

A little scary, no? Conversely, here’s what Lager displays when that happens:

“`text
2011-07-19 17:51:21 [error] gen_server crash terminated with reason: no match of right hand value {} in crash:handle_call/3
2011-07-19 17:51:22 [error] CRASH REPORT Process crash with 0 neighbours crashed with reason: no match of right hand value {} in crash:handle_call/3<
“`

A little more readable, eh? Now, there’s times when all that extra information is useful, and Lager doesn’t throw it away. Instead, it has a “crash log” where those messages go, and you’re free to dig through this file for any additional information you might need. Lager also borrows heavily from riak_err, such that printing large crash messages are safe. (I actually found a bug in riak_err, so Lager is even safer).

Now, those were messages coming out of error_logger, which is fine for legacy or library code, but Lager also has its own logging API that you can use. It’s actually implemented via a parse_transform so that Lager can capture the current module, function, line number and pid for inclusion in the log message. All this is done automatically, and the logging call in the code looks like this:

“`erlang

lager:error(“oh no!”)
lager:warning(“~s, ~s and ~s, oh my!”, [lions, tigers, bears])

“`

Which will be displayed like:

“`text
2011-07-19 18:02:02 [error] @test2:start:8 oh no!
2011-07-19 18:02:02 [warning] @test2:start:9 lions, tigers and bears, oh my!
“`

Note that you can easily see where the error occurred just by glancing at the line. Also notice that you don’t need to stick a newline on the end of the log message. Lager automatically (and happily) does that for you.

Why did I use a parse transform? I was originally going to use the traditional macro approach, capturing ?MODULE and ?LINE but I had a talk with Steve Vinoski, who also has some prior experience with Erlang logging, and he suggested a parse transform. A parse transform is handy in a couple different ways; we can do some compile time calculations, we can capture the current function name and in some ways its more flexible than a macro. Of course, Lager could also easily be extended to provide more traditional logging macros as well.

Now, Lager is architected much like error_logger in that its a gen_event with multiple handlers installed. Right now, there are only two provided: a console handler and a file handler. The file version supports multiple files at different levels, so you can have a log of only errors and above, and a log with informational messages as well. The loglevels are adjustable at runtime so you can turn the logging up/down for a particular backend:

“`erlang

lager:set_loglevel(lager_console_backend, debug)
lager:set_loglevel(lager_file_backend, “error.log”, warning)

“`

The first call would tell the console to display all messages at the debug level and above, and the second tells the file backend to set the “error.log” file to log any messages warning and above. You can of course set the defaults in Riak’s app.config.

Lager keeps track of which backends are consuming which levels and will very efficiently discard messages that would not be logged anywhere (they aren’t even sent to the gen_event). This means that if you have no backends consuming debug messages, you can log a million debug messages in less than half a second; they’re effectively free. Therefore you can add lots of debug messages to your code and not have to worry they’re slowing things down if you’re not looking at them.

Lager also plays well with log rotation. You simply move or delete the log file and Lager will notice and reopen the file or recreate it. This means it will work out of the box with tools like logrotate or newsyslog. It also handles situations like out of disk space or permission errors gracefully and when the situation is resolved it will resume logging.

Some further enhancements I plan to make are:

  • Internal log rotation by size and time
  • Syslog (remote and local) backends
  • Ability to filter messages by module/pid (think enabling debug messages for just a single module instead of globally)

Needless to say, I’m pretty excited about releasing this code. Lager should be merged mainline in Riak sometime this week once the integration work has been reviewed. That means that it will be part of the next major release, as well. Please let me know what you think. As usual, patches or suggestions are welcomed and encouraged.

Andrew