Tag Archives: Ripple

Rails Rumble with Riak and Ripple – Three Developers and a Lot of Rs

October 19, 2012

This past weekend, Bryce Kerley, David Andersen, and I had the honor of participating in Rails Rumble 2012 (of which, I am proud to say, Basho was a sponsor). This event is a 48-hour competition to design, build and deploy a web application in Ruby and Rails (or one of the other Rack-based web frameworks). Developing at this pace is grueling, to say the least (especially since it took place immediately after we all flew home from Ricon 2012), but also really exciting. You get the whole product cycle experience, from design to feature releases to testing and QA, all in one intense weekend.

Our application is Brainload. It’s a social flashcard/memorization aid tool, which lets users create, share and review flash cards on various topics (useful for memorizing programming language syntax, new APIs, school exam preparations, and building foreign language vocabulary).

One of the design constraints that we undertook this weekend was to build this web app using only Riak as a back end — no traditional relational database involved. Specifically, we chose the Ripple object modeling library, which provides a Riak persistence layer for Rails models, as well as familiar ActiveModel type querying capabilities. This provided several advantages and challenges, in terms of agile web development in the accelerated timeframe of the Rumble challenge. On the plus side, since Ripple essentially treats Riak as a document store, there was no database schema management and no migrations — very helpful for rapid exploratory development. Ripple model relationships (powered by LevelDB-backed secondary index queries) provided the usual one-to-one and one-to-many capabilities required for web apps like this (users, decks, cards, categories and so on). On the challenge side, we couldn’t use some of the minor Rails features that are closely tied to ActiveRecord. For example, fixtures were not automatically loaded during rake testing tasks (the solution in that particular case was to explicitly load the needed YAML files, or to use a standalone test data generation package like Factory Girl). Fortunately, we had Bryce on our side, who was old hand at coding Rails with Ripple, to advise us and walk us through the finer points of Ripple-specific syntax.

Despite our best efforts, a last-hour commit did manage to introduce a bug (I think we were a bit ambitious with the social features we wanted to add right up to the last minute). If you try to add a card — ouch! — you get an error message. While it is frustrating that we can’t fix this bug while the judging continues, this is part of what makes the competition so exciting. Of course, now that the contest is over, we have future plans for our web app — bugfixes, features, and all.

I cannot emphasize enough what a fantastic learning experience this was. Both in terms of working with really talented developers on my team (who also turned out to have pretty amazing sysadmin and design chops), and also for getting unique hands-on insight into the benefits and rough spots of working with our own database client and object modeling libraries.

If you get the chance to next year, I highly recommend participating in the Rails Rumble (and meanwhile, looking through and voting for the other entries in the competition). It was exhilarating and educational, and I am especially proud of Basho for being a sponsor of this competition.

Welcome, New Ripple Committers

April 12, 2011

Two of the best things about working with open-source software are being part of a community and collaborating with other skilled people. When your project becomes more than just a toy, you need others to help keep it going and to infuse fresh ideas into it. You need additional maintainers. Ripple (the Ruby client driver and tools) has been no exception to this rule, so this past weekend I was thrilled to welcome two new committers to the project.

Myron Marston

Myron Marston (@myronmarston, github/myronmarston) is porting a large MySQL-backed application over to Riak at SEOmoz. Myron’s project uses the Ripple ORM fairly heavily, so he’s already fixed a bunch of things related to associations and validations.

Kyle Kingsbury

Kyle Kingsbury (@aphyrgithub/aphyr) has been working on some awesome Riak infrastructure at his day job at Vodpod. In the course of his work, he removed the ActiveSupport dependency from riak-client and created another Riak ORM called Risky.

Thank you Myron and Kyle for your continued contributions!

Sean

Ripple 0.9 Release

April 4, 2011

I’m proud to announce the Ripple family of gems (riak-client, ripple, riak-sessions) version 0.9.0 were released yesterday. This is a huge leap forward from the 0.8 series, last release of which was in December. I’m going to highlight some of the best new features in this blog post.

HTTP+SSL Support

Adam Hunter did an awesome job implementing support for HTTPS, including navigating the various idiosyncracies of HTTP client libraries. If you’ve got HTTPS turned on in Riak, or a reverse-proxy in front that provides SSL, it’s easy to set up.

“`ruby
# Turn on SSL
client = Riak::Client.new(:ssl => true)

# Alternatively, provide the protocol client = Riak::Client.new(:protocol => “https”)

# Want to be a good SSL citizen? Use a client certificate.
# This can be used to authenticate clients automatically on the server-side.
client.ssl = {:pem_file => “/path/to/pemfile”}

# Use the CA chain for server verification
client.ssl = { :ca_path => “/path/to/ca_cert/dir” }

# All three of the above options will invoke “peer” verification.
# Use “none” verification only if you’re lazy. This is the default
# if you don’t specify a client certificate or CA.
client.ssl = { :verify_mode => “none” }
“`

Adam also added HTTP Basic authentication for those who use it on their reverse-proxy servers. It can be set with the :basic_auth option/accessor as a string of “user:password”.

Protocol Buffers

Riak has had a Protocol Buffers-based client API for a long time, but the state of Protocol Buffers support in Ruby has been very bad until recently. Thanks to Blake Mizerany’s “Beefcake” library, it was really simple to add support in a cross-platform way. While not insanely faster, the decreased overhead for many operations can make a big difference in the long run. Check out these benchmarks (run on MRI 1.9.2, comparing against the Excon backend):

                               user     system      total        real
http  ping                 0.020000   0.010000   0.030000 (  0.084994)
pbc   ping                 0.000000   0.000000   0.000000 (  0.007313)
http  buckets              0.010000   0.000000   0.010000 (  0.894827)
pbc   buckets              0.000000   0.000000   0.000000 (  0.864926)
http  get_bucket           0.480000   0.020000   0.500000 (  1.075365)
pbc   get_bucket           0.170000   0.030000   0.200000 (  0.271493)
http  set_bucket           0.060000   0.000000   0.060000 (  0.660926)
pbc   set_bucket           0.030000   0.000000   0.030000 (  0.579500)
http  store_new            0.710000   0.040000   0.750000 (  2.443635)
pbc   store_new            0.630000   0.030000   0.660000 (  1.382278)
http  store_key            0.730000   0.040000   0.770000 (  2.779741)
pbc   store_key            0.580000   0.020000   0.600000 (  1.539332)
http  fetch_key            0.690000   0.030000   0.720000 (  2.014679)
pbc   fetch_key            0.410000   0.030000   0.440000 (  0.948865)
http  keys                 0.300000   0.090000   0.390000 ( 78.455719)
pbc   keys                 0.530000   0.020000   0.550000 (  0.828484)
http  key_stream           0.200000   0.010000   0.210000 (  0.689116)
pbc   key_stream           0.530000   0.010000   0.540000 (  0.833347)

Adding Protocol Buffers required a breaking change in the Riak::Client class, namely that the port setting/accessor was split into http_port and pb_port. If you still have this setting in a configuration file or your code, you will receive a deprecation warning.

“`ruby
# Use Protocol Buffers! (default port is 8087)
client = Riak::Client.new(:protocol => “pbc”, :pb_port => 8087)

# Use HTTP and Protocol Buffers in parallel, too! (Luwak and Search require HTTP)
client.store_file(“bigpic.jpg”, “image/jpeg”, File.open(“images/bigpic.jpg”, ‘rb’))
“`

**Warning**: Because some operations (namely get_bucket_props and set_bucket_props) are not semantically equivalent on both interfaces, you might run into some unexpected problems. I have been assured that these differences will be fixed soon.

MapReduce Improvements

Streaming MapReduce is now supported on both protocols. This lets you handle results as they are produced by MapReduce rather than waiting for all the results to be accumulated. Unlike the traditional mode, you will be passed a phase number in addition to the data from the phase. Like key-streaming, just give a block to the run method.

“`ruby
# Make a MapReduce job like usual.
Riak::MapReduce.new(client).
add(“people”,”sean”).
link(:tag => “friend”).
map(“Riak.mapValuesJson”, :keep => true).
run do |phase, data| # Streaming!
puts data.inspect
end
“`

MapReduce key-filters were available in beta releases of 0.9. This is a new feature in Riak 0.14 that lets you reduce the number of keys fed to your MapReduce query by some criteria of the key names. Here’s an example:

“`ruby
# Blockish builder syntax for key-filters
Riak::MapReduce.new(client).
filter(“posts”) do
tokenize “-“, 1 # Split the key on dashes, take the first token
string_to_int # Convert the token to an integer
eq 2011 # Only pass the ones from 2011
end

# Same as above, without builder syntax
Riak::MapReduce.new(client).
add(“posts”, [["tokenize", "-", 1],
["string_to_int"],
["eq", 2011]])
“`

Ripple::Document Improvements

Ripple::Document models got a lot of small improvements, including:

  • Callback ordering was fixed.
  • Documents can be serialized to JSON e.g. for API responses.
  • Client errors bubble up when saving a Document.
  • Several association proxy bugs were fixed.
  • The datetime serialization format defaults to ISO8601 but is also configurable.
  • Mass-attribute-assignment protection was added, including protecting :key by default.
  • Embedded documents can be compared for equality, which amounts to attribute equality when under the same parent document.
  • Documents can now have observer classes which can also be generated by the ripple:observer generator.

Testing Improvements

In order to make sure that the client layer is sufficiently independent of transport semantics and that the lower layers comply with the “unified” backend API, there is a new suite of integration tests for riak-client that covers operations that are supported by both transport mechanisms. This should make it much easier to implement new client backends in the future.

The Riak::TestServer was made faster and more reliable by a few changes to the Erlang bits that power it.

Onward to 1.0

Recently the committers and some members of the community joined me in discussing some key features that need to be in Ripple before it reaches “1.0” status. Some of them will be really incredible, and I’m anxious to get started on them:

  • Enhanced Document querying (scopes, indexing, lazy loading, etc)
  • User-defined sibling-resolution policies, with automatic retries
  • Enhanced Riak Search features
  • Platform-specific Protocol Buffers drivers (MRI C, JRuby, Rubinius C++)
  • Server templates for creating development clusters (extracted from Riak::TestServer)

To that end, I’ve created a 0.9-stable branch which will only receive bugfixes going forward. All new development for 1.0 will be done on master. We’re likely to break some legacy APIs, so we will try to add deprecation notices to the 0.9 series where possible.

Enjoy this latest release of Ripple!

Sean and contributors

Adam Hunter Has Joined The Basho Team

March 24, 2011

Adam Hunter is the newest Basho Developer Advocate!

Adam first got involved with Riak when he used it in conjunction with Ripple, Riak’s Ruby library, to build several applications at his previous position. (In addition to being a deeply-skilled Ruby developer, Adam has also spent some happy years writing PHP for fun and profit.) In the process, he started contributing patches and features to Ripple, and we liked his code and enthusiasm for the project so much that we extended him committer rights. Since then he has become an active and visible member of the Riak community, so we were quite pleased when he accepted the offer to come aboard.

Home base for Adam is Charlotte, North Carolina, so be sure to look him up if you’re in the area and are interested in getting an earful about Riak and distributed systems. You can also find him on Twitter and on GitHub as adamhunter.

Welcome, Adam!

The Basho Team

Ripple 0.8 is Full of Good Stuff

August 31, 2010

This is a repost from the blog of Sean Cribbs, our Developer Advocate. 

It’s been a while since I’ve blogged about a release of Ripple, in fact, it’s been a long time since I’ve released Ripple. So this post is going to dig into Ripple 0.8 (released today, August 31) and catch you up on what has happened since 0.7.1 (and 0.5 if you don’t follow the Github project).

The major features, which I’ll describe in more detail below, are:

  • Supports Riak 0.12 features
  • Runs on Rails 3 (non-prerelease)
  • Adds Linked associations
  • Adds session stores for Rack and Rails 3 apps

Riak 0.12 Features

The biggest changes here were some bucket-related features. First of all, you can define default quorum parameters for requests on a per-bucket basis, exposed as bucket properties. Riak 0.12 also allows you to specify “symbolic” quorums, that is, “all” (N replies), “quorum” (N/2 + 1 replies), or “one” (1 reply). Riak::Bucket has support for these new properties and exposes them as attr_accessor-like methods. This is a big time saver if you need to tune your quorums for different use-cases or N-values.

Second, keys are not listed by default. There used to be a big flashing warning sign on Riak::Client#bucket that encouraged you to pass :keys => false. In Ripple 0.8 that’s the default, but it’s also explicit so that if you use the latest gem on Riak 0.11 or earlier, you should get the same behavior.

Runs on Rails 3

I’ve been pushing for Rails 3 ever since Ripple was conceived, but now that the actual release of Rails 3 is out, it’s an easier sell. Thanks to all the contributors who helped me keep Ripple up-to-date with the latest prereleases.

Linked associations

These are HOT, and were the missing features that held me back from saying “Yes, you should use Ripple in your app.” The underlying concepts take some time to understand (the upcoming link-walking page to the Fast Track will help), but you actually have a lot more freedom than foreign keys. Here’s some examples (with a little detail of how they work):

You’ll notice only one and many in the above examples. From the beginning, I’ve eschewed creating the belongs_to macro because I think it has the wrong semantics for how linked associations work (links are all on the origin side). It’s more like you “point to one of” or “point to many of”. Minor point, but often it’s the language you choose that frames how you think about things.

Session stores

Outside the Ruby-sphere, web session storage is one of Riak’s most popular use-cases. Both Mochi and Wikia are using it for this. Now, it’s really easy to do the same for your Rails or Sinatra app.

For Sinatra, Padrino and other pure Rack apps, use Riak::SessionStore:

For Rails 3, use Ripple::SessionStore.

Webinar Recap – Riak with Rails

August 8, 2010

Thank you to those who attended our Rails-oriented webinar yesterday. Like before, we’re recapping the questions below for everyone’s sake (in no particular order).

Q: When you have multiple application servers and Riak nodes, how do you handle “replication lag”?

Most web applications have some element of eventual consistency (or potential inconsistency) in them by their nature. Object and view caches sacrifice immediate consistency for gains in throughput and latency, and hopefully provide a better user experience. With Riak, you can achieve acceptable data freshness by “reading your writes”. That is, use the same read quorum as your write quorum and make sure that the R+W is greater than N. For example, using R=W=DW=2 when N=3 will give a strong assurance of consistency.

Q: I find myself doing def key; id; end. Is there any easier way to tell Ripple the key?

Currently there is not. However, I’ve found myself using this pattern frequently when I want a meaningful key that is also an attribute. There’s an issue on the tracker just for this feature. In the meantime, you could use two method aliases:

“`ruby
class User
include Ripple::Document
property :email, String, :presence => true

# This forces all attribute methods to be defined
define_attribute_methods
alias_method :key, :email
alias_method :key=, :email=
end
“`

As long as your property is a string, this should work just fine.

Q: Any tips on how to handle pagination over MapReduce queries?

The challenge with pagination in Riak is that reduce phases are not guaranteed to run only once, but instead are run in parallel as results from the previous phase come in asynchronously, and then followed by a final reduce. So in a sense, you have to treat all invocations of your reduce function as a “re-reduce”. We have plans to allow reduce phases to specify that they should be run only once, but for right now you can get around this limitation.

Reduce phases are always run on the coordinating node, so if you put a reduce phase before the one where you want to perform pagination, you are pretty much guaranteed that the whole result set is going to be available in a single application of the final reduce. A typical combination would be a “sorting” phase followed by a “pagination” phase. Riak.reduceSort and Riak.reduceSlice are two built-in functions that could help accomplish this task.

Sean and Grant

Calling all Rubyists – Ripple has Arrived!

February 11, 2010

The Basho Dev. Team has been very excited about working with the Ruby community for some time. The only problem was we were heads down on so many other projects that it was hard to make any progress. But, even with all that work on our plate, we were committed to showing some love to Rubyists and their frameworks.

Enter Sean Cribbs. As Sean details in his latest blog post, Basho and the stellar team at Sonian made it possible for him to hack on Ripple, a one-of-a-kind client library and object mapper for Riak. The full feature set for Ripple can be found on Sean’s blog, but highlights include a DataMapper-like API, an easy builder-style interface to Map/Reduce, and near-term integration with Rails 3.

And, in case you need any convincing that you should consider Riak as the primary datastore for your next Rails app, check out Sean’s earlier post, “Why Riak should power your next Rails app.”

So, if you’ve read enough and want to get your hands on Ripple, go check it out on GitHub.

If you don’t have Riak downloaded and built yet, get on it.

Lastly, you are going to be seeing a lot more Riak in your Ruby. So stay tuned because we have some big plans.

Best,

Mark