October 10, 2010
Dedicated Riak users who were who hanging out in the IRC channel late Friday may have seen the following message from our bot that reports changes to the canonical Riak Repo on Bitbucket:
dizzyd Added tag riak-0.13.0 for changeset 957b3d601bde
That’s right! We tagged the 0.13 release of Riak this past Friday and, after of a weekend away from our laptops, are ready to announce it officially. As you’re about to learn, this was a banner release for a whole host of reasons. Here is a rundown of the noteworthy changes and enhancements in this release:
As most people know, we have been hard at work on Riak Search for several months and, after releasing it to a handful of users as part of a limited beta, we are finally ready to release it into the wild!
Riak Search is a distributed, easily-scalable, failure-tolerant, real-time, full-text search engine built around Riak Core and tightly integrated with Riak’s key/value layer.
At a very high level, Search works like this: when a bucket in Riak has been enabled for Search integration (by installing the Search pre-commit hook), any objects stored in that bucket are also indexed seamlessly in Riak Search. You can then find and retrieve your Riak objects using the objects’ values. The Riak Client API can then be used to perform Search queries that return a list of bucket/key pairs matching the query. Alternatively, the query results can be used as the input to a Riak MapReduce operation. Currently the PHP, Python, Ruby, and Erlang APIs support integration with Riak Search.
There is obviously much more to be written about Riak Search, and a simple blurb in this blog post won’t do it justice. It’s for this reason that we are dedicating an entire blog post to it. You can expect to see this next week. In the mean time, you can go download the code and get more usage information on the Riak Search Section.
riak_kv and riak_core
In addition to focusing on previously-unreleased bits like Search, we also strengthened the code and functionality of riak_kv, the layer that handles the key/value operations, and riak_core, the component of Riak that provides all the services necessary to write a modern, well-behaved distributed application.
Firstly, the structure of the Riak source repos has been changed such that there is a single Erlang app per repo. This permits third parties to use riak_core (the abstracted Dynamo logic) in other applications. As you can imagine, this opens up a world of possibilities and we are looking forward to seeing where this code ends up being implemented. (Check out this blog post for a primer on Riak Core.)
The consistent hash ring balancing logic has been improved to enable large clusters to converge more quickly on a common ring state. Related to this, further work has been done to improve Riak’s performance and robustness when dealing with large data sets and failure scenarios.
Also, the Riak Command Line Tools were expanded with two new commands: “ringready” and “wait-for-service.” These were put in place to help anyone administering a Riak installation to script detection of cluster convergence on a given ring and wait for components like riak_kv to become available, respectively. You can read up on all your favorite Riak Command Line Tools here.
Enhancing Riak’s MapReduce functionality is something on which we’ve been focusing intensely for the past several releases, and this one was no different.
And, we improved MapReduce job input handling. In short what this means is that you will see lower resource consumption and more stable run times when the input producer exceeds the cluster’s ability to process new MapReduce inputs.
It’s also worth mentioning that there is more MapReduce goodness in the pipeline, both code and non-code. Stay tuned, because it’s only getting better!
Bitcask, the default storage backed for Riak, also got some huge enhancements in the 0.13 release. The most noticeable and significant improvement is that Bitcask now uses 40% less memory per key entry. The net effect is that you are required to have less RAM available in your cluster (as Bitcask stores all key data in memory). And, for those of you working with large data sets, we did some work to make starting up Bitcask significantly faster.
Listing keys, something the community has been asking for us to make more efficient, got a nice speedup when using Bitcask (which should make some of your map/reduce queries a lot snappier). This work is ongoing, too, so look for this process to get even faster as we make additional incremental improvements. Also of note, Bitcask will now reclaim memory of expired key entries.
Riak 0.13 is our best release yet and we are more confident than ever that it’s the best database out there for your production needs (assuming it suits your use case, of course). In addition to the changes we highlighted above, you should also take a moment to check out the complete release notes.
After you’re done with those, drop everything and do the following:
- Download Riak 0.13 on your platform of choice
- Download Riak Search
- Join the Mailing List (if you’re not on it already)
- If you’re new to Riak, take a few minutes and do The Riak Fast Track. You won’t regret it.
As always, let us know if you have any questions and issue, and thanks for being a part of Riak!
September 29, 2010
Basho hackers will be giving quite a few presentations between now and the end of the week. And they all happen to be in Baltimore! Here is a quick rundown (in no particular order) of where we will be, who will be there, and what we will be talking about:
Rusty Klophaus at CUFP
Rusty Klophaus will be at the Commercial Users of Functional Programming (CUFP) Event taking place this weekend in Baltimore, Maryland. His talk is called “Riak Core: Building Distributed Applications Without Shared State” and it should be downright amazing.
From the talk’s description: Both Riak KV (a key-value datastore and map/reduce platform) and Riak Search (a Solr-compatible full-text search and indexing engine) are built around a library called Riak Core that manages the mechanics of running a distributed application in a cluster without requiring a central coordinator or shared state. Using Riak Core, these applications can scale to hundreds of servers, handle enterprise-sized amounts of data, and remain operational in the face of server failure.
All the details on his talk can be found here.
And, in case you haven’t been following your Riak Core developments, check out Building Distributed Systems with Riak Core.
Dave Smith at the Ninth ACM SIGPLAN Erlang Workshop
Dave Smith (a.k.a “Dizzyd”) will be keynoting the ACM SIGPLAN Erlang Workshop taking place on Friday, Sept 30th, also in Baltimore. Dave’s talk is called “Rebar, Bitcask and how chemotherapy made me a better developer.”
Rebar and Bitcask are both pieces of software that Dizzy had a major hand in creating and they have played a huge role in Riak’s adoption (not to mention that Rebar has quickly become an indispensable tool for Erlang developers everywhere). Dave was also fighting follicular lymphoma while writing a lot of this code. Needless to say, this one is sure to be memorable and of immense value.
More details on his talk can be found here.
It should also be noted that newly-minted Basho Developer Scott Fritchie is the Workshop Chair for this event. He is an accomplished Erlang developer and Riak is not the only distributed key/value store about which Scott is passionate – he will also happily talk your ear off about Hibari.
Justin Sheehy at Surge
Just when you thought they couldn’t fit another conference in Baltimore on the same weekend… And this one is big: it’s the Surge Conference put on by the team at OmniTI. Basho will be there in the form of CTO Justin Sheehy.
Justin will be giving a talk about concurrency at scale, something about which every distributed systems developer should care deeply. Additionally, he will be taking part in a panel discussion – I haven’t seen an official name for it yet but rumor has it that it’s something along the lines of “SQL versus NoSQL.”
Check out the Surge site for more conference details.
As you can see, Baltimore is the place to be this weekend. Get there at all costs. And then go download Riak.
September 16, 2010
Justin Sheehy, Basho’s CTO, and Bryan Cantrill, Joyent’s VP of Engineering, are two acknowledged authorities on distributed systems and cloud computing. Riak SmartMachines offer you all the advantages of Riak running on the best cloud technology there is.
What do you get when you put Justin Sheehy, Bryan Cantrill and a microphone in a room for 45 minutes to talk about NoSQL and Riak SmartMachine performance? One of the best webinars we’ve ever had the chance to take part in!
Watch this one at least three times. Seriously. And then go download Riak.
You can download a PDF version of the slides used in the webinar here.
September 9, 2010
At long last we have all the details ironed out for the upcoming September Riak Meetup in San Francisco. The crew here in SF is quite excited about this month’s event, and here’s why:
Date: Thursday, Sept. 23rd
Location: Engine Yard Offices, located at 500 Third Street, Suite 510
- 7:15 – Riak Basics
After the first meetup, one of the attendees remarked, “Good, but looking for some basics and some hands on demo as well.” Admittedly, this is something we could have addressed a bit better. So at the beginning of this meetup (as well as all meetups moving forward) we are going to devote at least 15 minutes to discuss Riak basics. There are no stupid questions. Ask away.
- 7:30 – Riak vs Git: NOSQL Battle Royale
Presenter: Rick Olson, Github
This talk will compare and contrast Riak and Git on their merits as key/value stores, and look at how the two can work together.
- 8:00 – From Riak to RabbitMQ
Presenter: Andy Gross, Basho Technologies
This will cover using Riak to publish to RabbitMQ using post-commit hooks and gen_bunny.
- 8:30 – General Riak/Distributed Systems Conversation and Networking
Note: There is only seating for 50, so you’ll want to get there on time to secure a seat.
Basho will be providing food (pizza) and refreshments (beer, soda, etc.). And for those of you who can’t join us next Thursday, I will also be filming the talks with the goal of posting them online if everything goes to plan.
You can RSVP on the Riak Meetup Page. So go do it. Now!
Hope to see you there.
September 3, 2010
Links and link walking are two very powerful concepts in Riak. If used appropriately within an application, they can help to add another level of relationships to your data that are typically not possible when using a key/value store as your primary storage engine. It’s for this reason that we thought new users should know how to use them.
Today we are adding a Links and Link Walking tutorial to the Riak Fast Track. In about 20 minutes, you will learn what links are and how to use them. And, to cap it all off, there is a 12 minute screencast put together by Sean Cribbs to give you a more in depth look at what links are all about.We think this is pretty cool stuff and we have a feeling you’ll enjoy it, too.
As usual, we love, want and need feedback, so send an email to email@example.com with any questions or comments you might have. (I’ve also been known to send t-shirts in exchange for thoughts on how to make Riak and our documentation better…)
Enjoy, and thanks for using Riak!
August 31, 2010
This is a repost from the blog of Sean Cribbs, our Developer Advocate.
It’s been a while since I’ve blogged about a release of Ripple, in fact, it’s been a long time since I’ve released Ripple. So this post is going to dig into Ripple 0.8 (released today, August 31) and catch you up on what has happened since 0.7.1 (and 0.5 if you don’t follow the Github project).
The major features, which I’ll describe in more detail below, are:
- Supports Riak 0.12 features
- Runs on Rails 3 (non-prerelease)
- Adds Linked associations
- Adds session stores for Rack and Rails 3 apps
Riak 0.12 Features
The biggest changes here were some bucket-related features. First of all, you can define default quorum parameters for requests on a per-bucket basis, exposed as bucket properties. Riak 0.12 also allows you to specify “symbolic” quorums, that is, “all” (N replies), “quorum” (N/2 + 1 replies), or “one” (1 reply). Riak::Bucket has support for these new properties and exposes them as attr_accessor-like methods. This is a big time saver if you need to tune your quorums for different use-cases or N-values.
Second, keys are not listed by default. There used to be a big flashing warning sign on Riak::Client#bucket that encouraged you to pass :keys => false. In Ripple 0.8 that’s the default, but it’s also explicit so that if you use the latest gem on Riak 0.11 or earlier, you should get the same behavior.
Runs on Rails 3
I’ve been pushing for Rails 3 ever since Ripple was conceived, but now that the actual release of Rails 3 is out, it’s an easier sell. Thanks to all the contributors who helped me keep Ripple up-to-date with the latest prereleases.
These are HOT, and were the missing features that held me back from saying “Yes, you should use Ripple in your app.” The underlying concepts take some time to understand (the upcoming link-walking page to the Fast Track will help), but you actually have a lot more freedom than foreign keys. Here’s some examples (with a little detail of how they work):
You’ll notice only one and many in the above examples. From the beginning, I’ve eschewed creating the belongs_to macro because I think it has the wrong semantics for how linked associations work (links are all on the origin side). It’s more like you “point to one of” or “point to many of”. Minor point, but often it’s the language you choose that frames how you think about things.
Outside the Ruby-sphere, web session storage is one of Riak’s most popular use-cases. Both Mochi and Wikia are using it for this. Now, it’s really easy to do the same for your Rails or Sinatra app.
For Sinatra, Padrino and other pure Rack apps, use Riak::SessionStore:
For Rails 3, use Ripple::SessionStore.
August 20, 2010
Thank you to those who attended our webinar yesterday. Like before, we’re recapping the questions below for everyone’s sake (in no particular order).
Q: How would solve full text search with the current versions of Riak? One could also take Wriaki as an example as most wikis have some sort of fulltext search functionality.
I recommend using existing fulltext solutions. Solr has matched up well with most of the web applications I have written, and would certainly work for Wriaki as well.
Q: Where in the course of the interaction (shown on slide 18) are you defining the client ID? Don’t you need the client ID and vclock to match between updates?
On slide 42, we talk about “actors” which are essentially client IDs. Using the logged-in user as the client ID can help prevent vclock explosion and is a sensible way of structuring your updates.
August 13, 2010
Documentation is great, but playing with examples can also be a helpful way to tackle steep learning curves. To help you learn about ways of using Riak, we’d like to present “Wriaki”, an example implementation of a wiki that stores its data in Riak.
We invite you to join us for a free webinar on Thursday, August 19 at 2:00PM Eastern Time (UTC-4) about Riak in Action: Wriaki. During the presentation, Bryan Fink will cover:
- Modeling wiki data in the Riak key/value store
- Access patterns using both get/put and map/reduce
- Three strategies that Wriaki uses for dealing with eventual consistency
- how the user interface changes to accommodate Wriaki’s models
The code for Wriaki will be open-source at the time of the presentation. The presentation will last 30 to 45 minutes, with time for questions at the end.
Fill in the Sorry, registration has closed! form below to reserve your seat!
If you cannot attend, the video and slides will be made available afterward in the recap post on the blog.
August 8, 2010
This is a huge day for Basho Technologies, Riak, and our growing community of users.
We are thrilled to announce Basho’s partnership with Joyent to bring our community hosted Riak on Joyent’s Smart platform. With both open source and enterprise versions available, anyone can quickly spin up a Riak cluster and start building applications.
When we first began talking to Jason and David and the rest of the Joyent team early this year, we realized we shared a common vision for the future of infrastructure. The past several months have been spent finalizing the details, and in just a few weeks you’ll be able to go to my.joyent.com and, with a few clicks, purchase and deploy as many nodes of Riak you want, need, and can handle.
Making pre-configured Riak SmartMachines available in the Joyent cloud will enable developers to combine all the benefits of Riak with the proven, advanced hosting platform that businesses like LinkedIn, Gilt, and Backstage rely on every day.
Mark your calendar, because hosted Riak is here!
August 8, 2010
Thank you to those who attended our Rails-oriented webinar yesterday. Like before, we’re recapping the questions below for everyone’s sake (in no particular order).
Q: When you have multiple application servers and Riak nodes, how do you handle “replication lag”?
Most web applications have some element of eventual consistency (or potential inconsistency) in them by their nature. Object and view caches sacrifice immediate consistency for gains in throughput and latency, and hopefully provide a better user experience. With Riak, you can achieve acceptable data freshness by “reading your writes”. That is, use the same read quorum as your write quorum and make sure that the R+W is greater than N. For example, using R=W=DW=2 when N=3 will give a strong assurance of consistency.
Q: I find myself doing
def key; id; end. Is there any easier way to tell Ripple the key?
Currently there is not. However, I’ve found myself using this pattern frequently when I want a meaningful key that is also an attribute. There’s an issue on the tracker just for this feature. In the meantime, you could use two method aliases:
property :email, String, :presence => true
# This forces all attribute methods to be defined
alias_method :key, :email
alias_method :key=, :email=
As long as your property is a string, this should work just fine.
Q: Any tips on how to handle pagination over MapReduce queries?
The challenge with pagination in Riak is that reduce phases are not guaranteed to run only once, but instead are run in parallel as results from the previous phase come in asynchronously, and then followed by a final reduce. So in a sense, you have to treat all invocations of your reduce function as a “re-reduce”. We have plans to allow reduce phases to specify that they should be run only once, but for right now you can get around this limitation.
Reduce phases are always run on the coordinating node, so if you put a reduce phase before the one where you want to perform pagination, you are pretty much guaranteed that the whole result set is going to be available in a single application of the final reduce. A typical combination would be a “sorting” phase followed by a “pagination” phase.
Riak.reduceSlice are two built-in functions that could help accomplish this task.