Tag Archives: Erlang

Kivra Built Their Secure Mailbox Service on Riak

October 14, 2013

Kivra is a Swedish company that provides secure digital mailboxes, allowing users to securely receive, upload, and store all postal mail. Their mailboxes help users organize bills and notifications while eliminating the environmental footprint created from paper mail.

At a Riak Meetup in Dublin earlier this year, Kivra CTO, Bip Thelin, spoke about how they currently use Riak. Originally, Kivra tried to build their platform on a SQL database. However, they quickly outgrew this system and decided to move their infrastructure to Erlang and Riak because of its scalability and resiliency. You can watch Bip’s full presentation below to learn more about why they chose to switch to Riak, how they built six Riak clusters, and some of the lessons they’ve learned.

Bip’s slides can also be viewed here:

For more companies that have switched from SQL to Riak, check out our Users Page.


Basho to Speak About Riak at Lambda Jam

Chicago, IL – July 8, 2013 – Throughout the Lambda Jam Conference this week, Basho will be presenting twice about various aspects of Riak, as well as hosting a workshop on Webmachine. Lambda Jam is a conference for functional programmers and features a mix of sessions and workshops. It takes place in Chicago from July 8-10.

John Daily (Technical Evangelist at Basho), will be presenting first on “Distributed Programming with Riak Core and Pipe.” During his talk, he will dive into how Riak Core and Riak Pipe can be used, both within and beyond Basho. His talk begins at 9am on Tuesday, July 9th.

On July 10th at 9:50am, Basho Architect, Steve Vinoski, will be speaking on “Addressing Network Congestion in Riak Clusters.” In this talk, he will discuss an experimental approach to alleviating network congestion effects, such as timeouts and throughput collapse for Riak clusters under extreme load. In addition to exploring network scalability issues, this talk shows how Erlang can seamlessly integrate with non-FP languages.

Finally, Sean Cribbs and Chris Meiklejohn (Software Engineers at Basho) will be hosting a workshop entitled, “Functional Web Applications with Webmachine.” This workshop will provide guidance for understanding and getting started with Webmachine. It will then gradually expose richer HTTP features, while building out an application that is used by browsers and API clients alike. Their workshop begins at 1pm on July 10th.

To see where else Basho will be speaking, please visit our Basho Events Page.

Erlang at Basho, Five Years Later

July 2, 2013

We use the Erlang/OTP programming language in building our products here at Basho. We made that choice consciously, believing that it would be a tradeoff – significant benefits balanced by a handful of costs. I am often asked if we would make the same choice all over again. To answer that question I need to address the tradeoff we thought we were making.

The single most compelling reason to choose Erlang was the attribute for which it is best known: extremely high availability. The original design goal for Erlang was to enable rapid development of highly robust concurrent systems that “run forever.” The poster child of its success (outside Riak, of course) is the AXD 301 ATM switch, which reportedly delivers at or better than “nine nines” (99.9999999%) of uptime to customers. Since when we set out to build a database for applications requiring extremely high availability, Erlang was a natural fit.

We knew that Erlang’s supervisor concept, enabling a “let it crash” program designed for resilience, would be a big help for making systems that handle unforeseen errors gracefully. We knew that lightweight processes and a “many-small-heaps” approach to garbage collection would make it easier to build systems not suffering from unpredictable pauses in production. Those features paid off exactly as expected, and helped us a great deal. Many other features that we didn’t understand the full importance of at the time (such as the ability to inspect and modify a live system at run-time with almost no planning or cost) have also helped us greatly in making systems that our users and customers trust with their most critical data.

It turns out that our assessment of the key trade-off — a more limited pool of talented engineers — is, in practice, not a problem for a company like Basho. We need to hire great software developers, and we tend to look for ones with particular skills in areas like databases and/or distributed systems. If someone is a skilled programmer in relatively arcane disciplines like those, then the ability to learn a new programming language will not be daunting. While it’s theoretically a nice bonus for someone to bring knowledge of all the tools we use, we’ve hired a significant number of engineers that had no prior Erlang experience and they’ve worked out well.

This same purported drawback is a benefit in some ways. By not just looking for “X Engineers” (where X is Java, Erlang, or anything else), we make a statement both about our own technology decision-making process and the expected levels of interesting work at Basho. To help me work on my house, I’d rather have someone who self-identifies as an “expert carpenter” or “expert plumber,” not “expert hammer wielder,” even in the cases where most of the job might involve that tool. We expect developers at Basho to exercise deep, broad interests and expertise, and for them to do highly creative work. When we mention Erlang and the other thoughtful decisions we made in building our products, they value the roadmap and leadership.

I had an entertaining and ironic conversation about this recently with a manager at a large database company. He explained to me that we had clearly made the wrong choice, and that we should have chosen Java (like his team) in order to expand the recruiting pool. Then, without breaking a stride, he asked if I could send any candidates his way, to fill his gaps in finding talented people.

We continue to grow and to bring on great new engineers.

That’s not to say that there are no downsides. Any language, runtime, and community will bring with it different constraints and freedoms, making some tasks easier and others less so. We’ve done some work over the years to participate in the highly supportive Erlang community. But the big organizational weakness that so many people thought would come with the choice? It’s simply not a problem.

That lesson, combined with the ongoing technical advantages we enjoy because of Erlang, makes it easy to answer the question:

Yes, we would absolutely choose Erlang today.

Justin Sheehy

Riak Enterprise at Velti

December 3, 2012

In 2009, mobile marketing and advertising technology provider Velti had a good (but challenging) problem on their hands. Their technology, which allows people to interact with their TV by voting, giving feedback, participating in contests, etc., had taken off. It had been adopted by nearly all of the TV broadcasters in the UK and three of the UK’s five mobile operators. As more customers began using their technology, Velti saw quick growth in (inherently spikey) traffic. Their 2003-era .NET, SQLServer platform was becoming a concern.

Because the team at Velti had been working with Erlang (what Riak is written in), in 2010 they brought in Erlang Solutions to help them architect their next generation platform. Riak was chosen for the database, and an early version of Multi-Data Center replication in Riak Enterprise was used to build two geographically separated sites to minimize potential catastrophic outages.

Velti’s new mGageTM platform is now running on 18 servers across two data centers (nine nodes in each data center), with each server running both Erlang applications as well as Riak. We’re pleased to pass along reports that the platform is redundant, queue behavior has significantly improved (especially for large queue populations), and that after Velti moved to Riak 1.2, they saw noticeable disk space utilization thanks to improvements in merge management.

Markus Kern, VP Technology at Velti summarizes, “We operate a 24/7 service for over 140 customers. We cannot afford a single minute of downtime. Riak gives us the ability to meet and exceed our requirements for scale, data, durability, and availability.” Woot!

For more details on Velti’s experience, see our case study Highly Available Mobile Platform With Riak.

Basho Team

Lingua RICON – A Guide For Language Enthusiasts

August 29, 2012

tl;dr – There will be no shortage of language-specific content at RICON when it comes to building Riak-backed applications. If you and your team working on a Riak application and have specific questions or needs around your language or framework of choice, you should be at RICON. Register here. The early bird price ends this Friday.

We are billing RICON as a “distributed systems conference dedicated to developers.” We mean this in two ways:

  1. We are raising awareness and strengthening a community around what it takes to build “distributed systems”; in which a set of physical resources that are spread over unpredictable networks cooperate to run a service in production with little or no downtime. Riak is one of a wide set of technologies that make this possible.
  2. We are delivering on a promise to simplify how developers interact with distributed systems at the language level. This is largely focused on Riak, but not entirely.

A brief look at the RICON schedule will make it quickly apparent that there is plenty of bonafide distributed systems knowledge and experience to go around. What may not be completely obvious (as was pointed out to me a few days ago by a prospective attendee and trusted advisor) is the depth of language-specific knowledge and experience that is represented in RICON’s schedule. I wanted to make sure we cleared this up.

For those of you interested what it takes to build applications with Riak (at the language level), here are the details of what will be represented in the talks. (Keep in mind that the listed speakers constitute but a tiny subset of knowledge that will be present.)

Java and the JVM

  • Comcast contributed the first ever Riak Java client some time around the beginning of 2010. Though that code has changed immensely over the past three years, Riak has spread to various teams who are now using it in production, mostly with Java on the front-end. Michael Bevilacqua-Linn’s Big Data in the Small talk will give valuable insight on how to build JVM-based services that talk to Riak.
  • George Reese’s Migrating from MySQL to Riak session will highlight their work using the Java-based Dasein persistence framework alongside Riak.
  • Brian Roach and Russell Brown, primary maintainers of the Java client, will be wandering the crowd. There will also be several community members using Riak in production with Clojure and Scala that have experience to share.


  • Riak is written in Erlang. And it follows OTP principles in that it’s composed of various Erlang applications and extensions like riak kv and riak_core. To that end, Bryan Fink’s talk on Riak Pipe, Ryan Zezeski’s Riak and Solr session, and a few other talks from the Basho Team will highlight how to build Erlang applications with Riak.
  • OpenX is using riak_core to do all sorts of crazy, amazing things. Anthony Molinaro’s talk about how he and his team are serving trillions of ads per year will go deep on building Erlang services with Riak.


  • Gary Flake is giving Day Two’s opening keynote. He and his team at Clipboard have put Riak through its paces and built a social network fronted by Node.js. He will have much advice and wisdom to pass along.
  • Matt Ranney and Voxer operate one of the biggest Node.js applications known to man. They recently open-sourced their Riak node.js client and, along with real-world experience about running Riak clusters that are creeping towards petabytes of data, his talk will be invaluable to anyone building an application with Riak and Node.

Ruby and Rails

  • The (not-yet-announced) talk from Ines Sombra and Michael Brodhead of EngineYard will include a non-trivial amount of Riak and Ruby production knowledge.
  • Sean Cribbs, original author of Riak’s Ruby client, will be on-hand, along with a handful of community members who have Ruby/Rails applications in production.


  • The team at Bump is full of talent, and they are steeped in Python experience. The first application they wrote when they switched from MongoDB to Riak was Python-based, and their talk about building a transaction log on Riak will touch on their Python usage, too.
  • Various community members who have contributed to and use the Riak Python Client will be in attendance, ready to answer questions and debate implementation details.


  • Bump’s talk will be valuable to Haskell fans, too, as they will be detailing using Riak with a custom, open-source Haskell proxy that handles client-side resolution.
  • There are a few other known applications running Riak with Haskell in the wild. They, too, will be represented among the crowd.


  • In addition to being Riak Core experts, OpenX wrote a custom C backend for Riak that will be highlighted in their talk.
  • Andy Gross, primary author of the still-beta Riak C Client, will be at RICON and is expecting to share his plan for the future with would-be contributors.

What Other Languages Enthusiasts Should Attend?

Just because there isn’t a “Building a Blog with Riak and OCaml” talk on the schedule doesn’t mean that fans of OCaml should shy away from RICON. (In fact, Dave Parfitt has been hacking on an OCaml client and I’m sure he would love your input.) Fans of languages like Perl, Clojure, Go, and Smalltalk are encouraged to join. I have no doubt that you’ll leave feeling more confident about building applications that scale in your specific domain (and as I’ve said before we’ll happily refund your admission price if leave RICON feeling less-than-enriched.)

It’s also worth noting that, along with the massive power of the 100s of non-Basho attendees, nearly every member of the Basho Team that writes code – Engineering, Developer Advocates, Architects, Evangelists – will be at RICON as both eager onlookers and Riak authorities.

Join us for RICON. We’re looking forward to seeing you in October.


Congratulations To Bryan Fink And Joseph Blomstedt

July 9, 2012

We have some good news from the Basho Engineering Team to share with the community: Bryan Fink and Joseph Blomstedt have both had papers accepted to the Eleventh ACM SIGPLAN Erlang Workshop.

Bryan’s paper, “Experience Report: Distributed Computation on Dynamo-style Distributed Storage: Riak Pipe”, details the design and internals of Riak Pipe, the distributed processing framework that forms the foundation for Riak’s MapReduce engine. Bryan is the primary author of Riak Pipe.

Joseph’s submission, “Concurrent Property-based Testing: From Prototype to Final Implementation”, is based on the work that he and the team did (and continue to do) to test and bullet-proof the resiliency of Riak. (He gave a related talk at Erlang Factory this past March.)

They will both be part of the Workshop happening September 14th in Copenhagen, Denmark.

Congratulations to Bryan and Joseph!

The Basho Team

Erlang Factory SF and Basho Bash 2012

March 26, 2012

This is a big week for Basho.

Erlang Factory

A handful of Basho employees are in San Francisco this week for Erlang Factory SF 2012, taking place March 26th – March 31st at the Marines’ Memorial Club and Hotel in Union Square.

The first three days of Erlang Factory are primarily workshops, and Daniel Reverri will be teaching a 3 day class on Building Distributed Clusters with Riak. All attendees will walk away with a clear understanding of exactly why Riak is the best distributed database you will ever run in production.>

The actual conference spans Thursday – Friday, and the talk lineup for this year’s event is exceptional. The Basho team will be well-represented. Put these talks on your calendar if you’re attending:

In addition to actual talks, Dizzy Smith will be running The Polyglot Programmer track and Steve Vinoski is hosting the Big Data portion of the Factory.

Several members of the Riak Community are also on the schedule:

  • Erlang for .NET Developers – OJ Reeves
  • Rewriting GitHub Pages with Riak Core, Riak KV, and Webmachine – Jesse Newland
  • Basho Bash West

    We’re really excited about all the success surrounding Riak in 2011 and we’re continuously building on that momentum as we move deeper into 2012. The number of Riak users and community members are growing exponentially so we decided to throw a party to celebrate. We’re calling it Basho Bash West 2012, and it’s co-sponsored by our friends at Joyent, Yammer and Voxer.

    Come join us on Thursday, March 29th, at 6:30PM. We are renting out Roe, and you won’t be allowed to pay for anything. You’ll also be leaving with some limited edition Riak swag that will make you the envy of all your friends. Various members of the Basho team will be in attendance, along with hundreds of developers, executives, and technology enthusiasts from the Bay Area. Miss this at your peril.

    You must RSVP to attend.


BashoChats 001 – Erlang and DTrace; Storm and Distributed RPC

December 21, 2011

The inaugural BashoChats was held just under a week ago at BashoWest in San Francisco. About 30 local developers came out to have a few beers on Basho’s tab and discuss distributed systems and databases. If you’re local to the Bay Area and/or want to keep an eye on what we have planned, join the group. There are some great talks in the pipeline…

Most importantly I’m happy to report that both talks from the evening are now online for your viewing pleasure.

Enjoy. Hope to see you next month.


DTrace and the Erlang VM

Andy Gross opened up the evening with just under 30 minutes on the current work happening at Basho and a few other companies to bring DTrace to Erlang VM. He starts off with some general information on both components and then goes in-depth on how they can be used to profile a running Riak installation.

Repo here on GitHub with the code he used for the examples in his presentation.

Computing Reach Using Storm Distributed RPC

After Andy concluded, Nathan Marz gave an overview of Storm, a framework he and his team at BackType built for distributed and fault tolerant realtime computation. He takes us through some Storm basics and then demonstrates how it is used to compute reach using distributed RPC.

Announcing Riaknostic, your Riak Doctor

December 15, 2011

As much as we love Riak and talk about how easy it is to put into production, the activity on the #riak IRC roommailing list and customer support issues has a way of telling us there are still some issues to be ironed out. Sometimes, things just go wrong and you want to know what they are. It can be frustrating to figure those things out if you don’t know what to look for.

We have felt this pain, too, and wished for a way that our customers and community users could discover and remedy problems on their own, without having to resort to the mailing list or IRC chat. That’s why today, we’re happy to announce the first public release of Riaknostic.

NOTE: Riaknostic is very new and still under development. It will not be fully tested and integrated until the next release of Riak. While it diagnostics are low-impact, do take care when running it on your system.

What is Riaknostic?

Riaknostic is an Erlang script (escript) that runs a series of “diagnostics” or “checks”, inspecting your operating system and Riak installation for known potential problems and then printing suggestions for how to fix those problems. Riaknostic will NOT fix those problems for you, it’s only a tool for diagnostics. Some of the things it checks are:

  • How much memory does the Riak process currently use?
  • Do Riak’s data directories have the correct permissions?
  • Did the Riak node crash in the past and leave a dump file?

Some of the checks require that the Riak node be running, but others simply work with the operating system. It has a whole bunch of other options and details that I won’t cover here, but you can read about them on the homepage or in the API documentation. Of course, we encourage you to read the source code and fork the project as well.

As is typical of Basho software, Riaknostic is free, open-source, and licensed under the Apache 2.0 license. You can use Riaknostic today, but full integration will be shipped in the next release of Riak (v1.1).

Extending Riaknostic For Fun and Wearable-Profit

We’ve constructed Riaknostic in such a way that it’s fairly easy to incorporate new diagnostics. Our goal is to get it to the point where Riaknostic can diagnose the majority of errors and misconfigurations that might befall your Riak cluster before you’re in production. Basho will continue to add to the library, but we can’t make it totally robust without your help. So, we will happily distribute various pieces of SWAG (including but not limited to t-shirts and stickers), to anyone who spends time extending it. Keep in mind that you’ll have to write a touch of Erlang to make this happen, but, if you’re new to the language, this is probably one of the best ways to learn. Take a look at the existing modules for examples and inspiration.

For precise instructions on how to Contribute to the library, have a look at the Riaknostic README on the GitHub Repo. And if you have any issues with it, send a message along to the Riak Mailing List or join us in #riak IRC.


Sean, on behalf of all Developer Advocates

Introducing Lager – A New Logging Framework for Erlang/OTP

July 20, 2011

Hi. My name is Andrew Thompson and I’ve been working on Riak at Basho since March. I’ve been focused primary on various technical debt issues within Riak since starting. The largest project I’ve undertaken is Lager, a new logging framework for Erlang/OTP. Lager is actually my second logging framework for Erlang, and I’ve used my previous experience to make this one even better. I think Lager has some compelling features to offer not seen in other erlang logging frameworks and I’ll go over the highlights in this post.

So, why write another log framework for Erlang? There’s already several; error_logger itself, SASL, log4erl and riak_err to name a few. One of the key goals was to make logging friendlier; both to the end-user and to the sysadmin. Lager tries very hard to hide the traditional “giant error tuple of doom” from the user (unless they go looking for it). For example, here’s what happens when a gen_server dies with a badmatch error (with SASL running):

=ERROR REPORT==== 19-Jul-2011::17:50:10 ===
** Generic server crash terminating
** Last message in was badmatch
** When Server state == {}
** Reason for termination ==
** {{badmatch,{}},

=CRASH REPORT==== 19-Jul-2011::17:50:10 ===
initial call: crash:init/1
registered_name: crash
exception exit: {{badmatch,{}},
in function gen_server:terminate/6
ancestors: []
messages: []
links: []
dictionary: []
trap_exit: false
status: running
heap_size: 377
stack_size: 24
reductions: 127

A little scary, no? Conversely, here’s what Lager displays when that happens:

2011-07-19 17:51:21 [error] gen_server crash terminated with reason: no match of right hand value {} in crash:handle_call/3
2011-07-19 17:51:22 [error] CRASH REPORT Process crash with 0 neighbours crashed with reason: no match of right hand value {} in crash:handle_call/3<

A little more readable, eh? Now, there’s times when all that extra information is useful, and Lager doesn’t throw it away. Instead, it has a “crash log” where those messages go, and you’re free to dig through this file for any additional information you might need. Lager also borrows heavily from riak_err, such that printing large crash messages are safe. (I actually found a bug in riak_err, so Lager is even safer).

Now, those were messages coming out of error_logger, which is fine for legacy or library code, but Lager also has its own logging API that you can use. It’s actually implemented via a parse_transform so that Lager can capture the current module, function, line number and pid for inclusion in the log message. All this is done automatically, and the logging call in the code looks like this:


lager:error(“oh no!”)
lager:warning(“~s, ~s and ~s, oh my!”, [lions, tigers, bears])


Which will be displayed like:

2011-07-19 18:02:02 [error] @test2:start:8 oh no!
2011-07-19 18:02:02 [warning] @test2:start:9 lions, tigers and bears, oh my!

Note that you can easily see where the error occurred just by glancing at the line. Also notice that you don’t need to stick a newline on the end of the log message. Lager automatically (and happily) does that for you.

Why did I use a parse transform? I was originally going to use the traditional macro approach, capturing ?MODULE and ?LINE but I had a talk with Steve Vinoski, who also has some prior experience with Erlang logging, and he suggested a parse transform. A parse transform is handy in a couple different ways; we can do some compile time calculations, we can capture the current function name and in some ways its more flexible than a macro. Of course, Lager could also easily be extended to provide more traditional logging macros as well.

Now, Lager is architected much like error_logger in that its a gen_event with multiple handlers installed. Right now, there are only two provided: a console handler and a file handler. The file version supports multiple files at different levels, so you can have a log of only errors and above, and a log with informational messages as well. The loglevels are adjustable at runtime so you can turn the logging up/down for a particular backend:


lager:set_loglevel(lager_console_backend, debug)
lager:set_loglevel(lager_file_backend, “error.log”, warning)


The first call would tell the console to display all messages at the debug level and above, and the second tells the file backend to set the “error.log” file to log any messages warning and above. You can of course set the defaults in Riak’s app.config.

Lager keeps track of which backends are consuming which levels and will very efficiently discard messages that would not be logged anywhere (they aren’t even sent to the gen_event). This means that if you have no backends consuming debug messages, you can log a million debug messages in less than half a second; they’re effectively free. Therefore you can add lots of debug messages to your code and not have to worry they’re slowing things down if you’re not looking at them.

Lager also plays well with log rotation. You simply move or delete the log file and Lager will notice and reopen the file or recreate it. This means it will work out of the box with tools like logrotate or newsyslog. It also handles situations like out of disk space or permission errors gracefully and when the situation is resolved it will resume logging.

Some further enhancements I plan to make are:

  • Internal log rotation by size and time
  • Syslog (remote and local) backends
  • Ability to filter messages by module/pid (think enabling debug messages for just a single module instead of globally)

Needless to say, I’m pretty excited about releasing this code. Lager should be merged mainline in Riak sometime this week once the integration work has been reviewed. That means that it will be part of the next major release, as well. Please let me know what you think. As usual, patches or suggestions are welcomed and encouraged.