Author Archives: basho

RICON Is Sold Out…But Don’t Worry

October 8, 2012

At this point you probably know that for the past several months the Basho Team has been planning a two-day conference dedicated to both Riak and distributed systems. We’ve christened this conference RICON and it’s happening this Wednesday in San Francisco.

As of the middle of last week, RICON is officially sold out. A few days from now, hundreds of hackers, technologists, and executives from around the globe (just north of 350 total attendees to be exact) will descend on the W Hotel in SOMA for two full days of talks from some of the industry’s leading minds on distributed systems. (There will also be copious hacking and a big party; more details on the latter below.)

If you will not be joining us in SF this week, there is no need to be upset. In the spirit of sharing and open-source, we’ve gone through great lengths to include you:

RICON Live

Once the conference starts, we’ll be making all the proceedings available from the RICON Live site.

  • First and foremost, we’re live-streaming all the talks, starting at 9AM Pacific on Wednesday. This is thanks to the brilliant folks at Fastly (who, in a number of days, built the ability to stream specially for RICON).
  • Keep an eye on the #RICON2012 tweets and bump the RICON playlist in your home, office, coffee shop, or house boat.
  • Track all the code, slides, and other resources that are released at RICON.

To everyone attending Wednesday, we look forward to seeing you there. To those of you who couldn’t make it, enjoy RICON Live. And make sure to register for RICON2013.

Thanks for being a part of Riak and RICON.

The Basho Team

Riak Docs Get an Overhaul

October 8, 2012

I’m proud to announce that today, Basho has delivered an overhaul to our existing documents to docs.basho.com. We streamlined the navigation, look and feel, and content. We also made some changes that will allow us to easily document future versions of Riak, without crushing legacy documents.

When I first investigated Riak for my book, I was impressed by the technical elegance of the project, but wished that the documentation better conveyed the beauty of the project. Today’s launch takes us one step closer to helping Riak users, and the merely curious, to discover the beauty I had — without resorting to reading the source code.

Crafting documentation isn’t the sexiest piece of an open source project, but it’s one of the vital ones. You can have amazingly well designed code (we do) that’s well tested (it is), but if users don’t know how to work it, your project will never grow beyond a curiosity for insiders and dedicated enthusiasts.

There is a lot to get excited about these changes. Here are a few of the gems:

Ground-up Information Architecting

The top goal of this rebuild was to make it easier to quickly find the information you’re looking for. We organized our existing wiki docs (which were growing a bit wild) into categories which would be of interest to various end users: beginners, developers and operators, and separated guides from references.

We have new avenues for finding docs, beyond the obvious primary navigation on the left. As you dive into the pages you may notice subtler forms. At the top of some longer docs, there exists a table of contents for fast navigation. And floating to the right is a version bar. This allows you to see versions of this document that apply to older versions of Riak.

TOC and Versions

Once you read through a document, most pages have a secondary “These May Also Interest You” list of links you might also want to read up on.

Interests

Most docs are also tagged with keywords. If you click one, you’ll be taken to an index of all documents with that same keyword for fast orthogonal navigation. Click on

Keywords

Version History

One of the realities of managing docs is wrangling changes to the application between versions. If you’re not careful, your documentation can easily become out of date, or filled with warnings (note! this only applies to 1.2.x!). We were inspired by the way that Node.js managed their docs, keeping every previous version since the beginning of time and pointing the latest directory to the newest version.

This allows you to not only navigate back in time to older versions, but also helps reduce the chance of broken links and allows us to evolve the design over time without confusing those who are used to older document versions.

Complete Redesign

The docs don’t merely reflect Basho’s newer styles, but also an intent to give you the information you need to get things done. Collapsible nav is useful when reading out side-by-side docs (check out our Riak to Dynamo comparison work in progress).

A responsively designed site will shrink/hide items to take full advantage of smaller screens like iPads and smartphones.

Mobile Design

The design has other goodies, too, like a straightforward landing page, clear code samples, and easy to read type and font sizes.

Landing Page

The navigation look and feel should also help clarify where you are in the documentation at all times, reducing the chance of getting lost.

Technology

Though not a primary concern for most Riak users, these docs were all build using Middleman 3, plus many of custom plugins and other magi-hackery.

Similar to the Riak wiki, the Riak documentation and code is open source, and is open for anyone to contribute. I also created a new FAQ Markup Language, called FML. All of our FAQs are written in this markup. The rest is largely markdown and Middleman’s frontmatter.

Riak Search

This isn’t out yet, but look for it within the week.

The old wiki site used Google to search through the static site, so we thought: why not use Riak?

We’re test driving the newest Riak Search functionality for some good reasons. Importantly, we use Riak Search to eat our proverbial dog food. But there’s a better reason: because the new search ended up being the best method for searching content with high uptime.

We wanted to search with facets, by version ranges for example. We wanted to extract metadata from each document’s frontmatter, so we created a custom extractor. Clearly we wanted full-text search. We wanted it to be fast and distributed, a core feature of the new search.

The new Riak Search leverages the real power of SOLR, with the benefit of Riak KV as a storage engine. One installation. One cluster to manage. No mess, no fuss.

We’re sorry to have to play teaser to this new search, but it will be out before you know it. (If you’re at RICON this week, you’ll be learning all about it.)

Thanks

So many people made these docs possible, there are literally too many to name them all here. But special thanks go to John Newman for his amazing design skills, Seth Benton and Brian Shumate for amazing work and eyes to detail. Justin Pease and the CliServ team for their hand in a realistic view of what users actually need; Mark Phillips, Shanley Kane, and the rest of Marketing for a strategic vision for these docs; Sean Cribbs and the Basho Engineering Team for working on version 1.0 of the docs; Ryan Zezeski for the new search, and the rest of Basho and our community for supporting this project.

This isn’t the Oscars, so I’ll just add, thanks go to everyone who had a hand in making these, including the community who worked to build Riak docs v1.0. Writing documentation might seem easy, but it’s a massive undertaking—and this is just the beginning.

Eric Redmond

Basho Japan Opening Ceremony

October 5, 2012

The opening of Basho’s Tokyo office, has been an exciting endeavor for everyone involved. We felt that Basho’s arm in Japan should embrace the traditions and culture of Japanese business whole-heartedly, so the opening ceremony for Basho Japan was held in traditional, Japanese “Shinto” fashion.

Following Japanese business tradition, a Shinto priest was invited from the local shrine Suga-Jinjya to perform a ceremony that, by the Shinto tradition, will help Basho continute to do well in Japan.

Following is the video of the opening ceremony in Japan. Enjoy!

Basho Japan Opening Ceremony 26 Sept 2012 from Basho Technologies on Vimeo.

Basho UK Launch Party

October 1, 2012

tl;dr – RSVP to our London Launch Party on Wednesday, 3 October 2012.

At the beginning of this year, Ian Plosker and I were tasked to bring the quality service Basho is known for in the US to our customers in Europe. We came to London in March to find the future office of Basho’s European headquarters. We found a comfy office in the heart of the thriving startup community in Shoreditch.


Since May we have decorated, hired, started the Riak London Meetup, attracted attention and added new customers – our European operations are running strong. This week, it is time for us to take step back and celebrate this milestone in Basho’s history.

This Wednesday, the 3rd of October at Corbet Place (The Old Truman Brewery) in Shoreditch at 18:30 BST, we are having a party and you should be there. The space is limited to 200 people, so please RSVP. Come celebrate with Basho executives, employees, investors, customers, and Riak users – registration, beer, and canapes are free.

Slides – Searching and Accessing Data in Riak

September 24, 2012

Last week we aired a live broadcast from our sunny San Francisco offices on searching and accessing data in Riak. This is one of the top things we get questions about – especially from people moving from databases that use other data models or people building new applications on top of Riak.

Slides are below – unfortunately we had some audio quality issues with the video and need to rerecord it for upload. The webinar covers Riak’s full-text search, object tagging and Map Reduce capabilities including:

  • use cases and features
  • when NOT to use
  • query examples
  • user scenarios
  • high-level architecture and configuration
  • hybrid architectures

Check out slides below and make sure to read our docs on the subject.

SLIDES

In the talk, we mention a blog post from Riak user Clipboard on how they optimized Riak Search for their application, which is an awesome way to save and share online content. You can read that blog post here.

Cluster Management Improvements in Riak 1.2

September 13, 2012

Last month we released Riak 1.2 , with a number of improvements in Riak stats, the protobufs API, LevelDB backend and repair/recovery capabilities. Riak 1.2 also features a new strategy for making cluster changes like adding and removing nodes. With the new approach, Riak allows you to stage changes, view the impact on the cluster, and then commit or abort changes. The increased visibility lets Riak operators make more informed decisions about when and how to scale up, scale down and upgrade or replace nodes. Additionally, you can now make multiple changes, like adding a number of nodes, at the same time – critical for large-scale clusters.

Pre 1.2 Cluster Management

In prior versions of Riak, users made changes to the cluster using commands under the “riak-admin” syntax. To add or remove a node to the cluster, you would simply call “riak-admin join” or “riak-admin leave,” and the Riak cluster would immediately begin to handoff data and ownership as appropriate. While this approach was simple, it did raise two issues we’ve tried to address with the new cluster management capabilities:

  • Coordinating cluster changes: Prior to Riak 1.2, there was no way to group changes together. Changes were entered sequentially, and if there was more than one change (e.g. joining multiple nodes to a cluster), the first change would happen in a single transition and the remaining changes (e.g. the rest of the joins) would occur together in a second transition. In the case of multiple joins, in the first transition, data is transferred from the cluster to the new node. Then, in the second transition, some of the data transferred to the first new node is then transferred to the other new nodes, wasting network bandwidth and disk space. This proved particularly problematic for production deployments in which nodes were frequently added or removed.
  • Planning: The pre-1.2 approach to cluster management didn’t give you visibility into how your changes would affect the cluster before you made them. For instance, the only way to know how many transfers a join would take would be to start the join and then run “riak-admin ring-status”. Likewise, you couldn’t know what ownership would look like until after the join.

Staged Clustering

We addressed both of the above issues with a new approach we’re calling ‘Staged Clustering’.

In Riak 1.2, instead of joins, leaves, etc. taking place immediately, they’re first staged. After staging cluster changes, you can view how the changes will affect the cluster, seeing how ring ownership will change and how many transfers between nodes will need to occur to complete the transition. After looking at the plan, you can then add or remove changes staged to be committed, scrap the plan, or execute it as is.

Staged Clustering Riak

Staged Clustering High Level Process

The ‘Staged Clustering’ interface is implemented in Riak’s command line tool, riak-admin, under the ‘cluster’ command. Underneath the ‘cluster’ command are subcommands used to stage, view, and commit cluster changes (e.g. to join the current cluster to node dev1, you’d use: ‘riak-admin cluster join dev1’ ). You can read more about the new syntax in the Riak Wiki. Currently, the new approach to cluster management is not implemented in Riak Control, our open-source management and monitoring GUI, but is planned for a later release.

Example

Let’s take a look at how the new cluster management strategy would work in a scenario where we wanted to add three nodes to an existing node (dev1) to form a four-node cluster.

1. View the Current Member Status

First, we call ‘riak-admin member_status’ to get a view of the current cluster, the nodes in it and their current ring ownership:

Member Status Riak

2. Stage Joining New Nodes

Next, we’ll join three nodes (dev2, dev3, dev4) to the cluster using the cluster command.

bash
dev2: riak-admin cluster join dev1
dev3: riak-admin cluster join dev1
dev4: riak-admin cluster join dev1

The joins are now staged for commit.

3. View How Staged Changes Will Affect the Cluster

Now we can use the new ‘riak-admin cluster plan’ command to see the impact of the joins on the cluster, viewing changes to ring ownership and transfers that need to occur.

bash
riak-admin cluster plan

Cluster Management Plan Riak

In the output, we see: the changes staged for commit, the number of resulting cluster transitions (1), how the data will be distributed around the ring after transition (25% on each node), and the number of transfers the transition will take (48 total).

4. Commit Changes

If we want to commit these changes, we use the commit command:

bash
riak-admin cluster commit

These changes start taking place immediately. If we run ‘riak-admin member_status’, we can see the status of the transition. Additionally, we’ve fleshed out the ‘riak-admin transfers’ command to give you much more visibility into active transfers in Riak 1.2.

Other Resources

For more in-depth information on the new cluster management stuff in Riak 1.2, check out this recorded webinar with Basho engineer Joseph Blomstedt and the updated docs.

In-Depth on Riak 1.2 [video]

September 12, 2012

In case you missed the announcement, last month we released Riak 1.2 into the wild. This release was many months in the making and contains bug fixes, performance enhancements, as well as upgrades to operations and cluster management.

Last week at the San Francisco Bay Area Riak Meetup, Andy Gross and Shanley Kane discussed what’s changed in Riak 1.2 and why you should care. In case you were not able to make it, the video of their talk is now online. If you’ve been on the fence about either upgrading to 1.2 or putting a fresh Riak install into production, then this video is for you. The talk is 30 minutes long–consider it a worthwhile investment.

If after watching this video you’re still not convinced, come to RICON on October 10-11. There will be plenty of Riak users and distributed systems experts on hand to answer all your tough questions. Tickets won’t last long, so register now.

Thanks for being a part of Riak.

Tom

Lingua RICON – A Guide For Language Enthusiasts

August 29, 2012

tl;dr – There will be no shortage of language-specific content at RICON when it comes to building Riak-backed applications. If you and your team working on a Riak application and have specific questions or needs around your language or framework of choice, you should be at RICON. Register here. The early bird price ends this Friday.

We are billing RICON as a “distributed systems conference dedicated to developers.” We mean this in two ways:

  1. We are raising awareness and strengthening a community around what it takes to build “distributed systems”; in which a set of physical resources that are spread over unpredictable networks cooperate to run a service in production with little or no downtime. Riak is one of a wide set of technologies that make this possible.
  2. We are delivering on a promise to simplify how developers interact with distributed systems at the language level. This is largely focused on Riak, but not entirely.

A brief look at the RICON schedule will make it quickly apparent that there is plenty of bonafide distributed systems knowledge and experience to go around. What may not be completely obvious (as was pointed out to me a few days ago by a prospective attendee and trusted advisor) is the depth of language-specific knowledge and experience that is represented in RICON’s schedule. I wanted to make sure we cleared this up.

For those of you interested what it takes to build applications with Riak (at the language level), here are the details of what will be represented in the talks. (Keep in mind that the listed speakers constitute but a tiny subset of knowledge that will be present.)

Java and the JVM

  • Comcast contributed the first ever Riak Java client some time around the beginning of 2010. Though that code has changed immensely over the past three years, Riak has spread to various teams who are now using it in production, mostly with Java on the front-end. Michael Bevilacqua-Linn’s Big Data in the Small talk will give valuable insight on how to build JVM-based services that talk to Riak.
  • George Reese’s Migrating from MySQL to Riak session will highlight their work using the Java-based Dasein persistence framework alongside Riak.
  • Brian Roach and Russell Brown, primary maintainers of the Java client, will be wandering the crowd. There will also be several community members using Riak in production with Clojure and Scala that have experience to share.

Erlang/OTP

  • Riak is written in Erlang. And it follows OTP principles in that it’s composed of various Erlang applications and extensions like riak kv and riak_core. To that end, Bryan Fink’s talk on Riak Pipe, Ryan Zezeski’s Riak and Solr session, and a few other talks from the Basho Team will highlight how to build Erlang applications with Riak.
  • OpenX is using riak_core to do all sorts of crazy, amazing things. Anthony Molinaro’s talk about how he and his team are serving trillions of ads per year will go deep on building Erlang services with Riak.

Node.js

  • Gary Flake is giving Day Two’s opening keynote. He and his team at Clipboard have put Riak through its paces and built a social network fronted by Node.js. He will have much advice and wisdom to pass along.
  • Matt Ranney and Voxer operate one of the biggest Node.js applications known to man. They recently open-sourced their Riak node.js client and, along with real-world experience about running Riak clusters that are creeping towards petabytes of data, his talk will be invaluable to anyone building an application with Riak and Node.

Ruby and Rails

  • The (not-yet-announced) talk from Ines Sombra and Michael Brodhead of EngineYard will include a non-trivial amount of Riak and Ruby production knowledge.
  • Sean Cribbs, original author of Riak’s Ruby client, will be on-hand, along with a handful of community members who have Ruby/Rails applications in production.

Python

  • The team at Bump is full of talent, and they are steeped in Python experience. The first application they wrote when they switched from MongoDB to Riak was Python-based, and their talk about building a transaction log on Riak will touch on their Python usage, too.
  • Various community members who have contributed to and use the Riak Python Client will be in attendance, ready to answer questions and debate implementation details.

Haskell

  • Bump’s talk will be valuable to Haskell fans, too, as they will be detailing using Riak with a custom, open-source Haskell proxy that handles client-side resolution.
  • There are a few other known applications running Riak with Haskell in the wild. They, too, will be represented among the crowd.

C

  • In addition to being Riak Core experts, OpenX wrote a custom C backend for Riak that will be highlighted in their talk.
  • Andy Gross, primary author of the still-beta Riak C Client, will be at RICON and is expecting to share his plan for the future with would-be contributors.

What Other Languages Enthusiasts Should Attend?

Just because there isn’t a “Building a Blog with Riak and OCaml” talk on the schedule doesn’t mean that fans of OCaml should shy away from RICON. (In fact, Dave Parfitt has been hacking on an OCaml client and I’m sure he would love your input.) Fans of languages like Perl, Clojure, Go, and Smalltalk are encouraged to join. I have no doubt that you’ll leave feeling more confident about building applications that scale in your specific domain (and as I’ve said before we’ll happily refund your admission price if leave RICON feeling less-than-enriched.)

It’s also worth noting that, along with the massive power of the 100s of non-Basho attendees, nearly every member of the Basho Team that writes code – Engineering, Developer Advocates, Architects, Evangelists – will be at RICON as both eager onlookers and Riak authorities.

Join us for RICON. We’re looking forward to seeing you in October.

Mark

RICON Talks Announced; Brewer, Hellerstein, Flake To Keynote

August 23, 2012

Over the last few weeks we’ve been rolling out the talk details for RICON2012, Basho’s first-ever developer conference dedicated to Riak and the future of distributed systems in production. Taking place in San Francisco this October, it should be invaluable to anyone working in, around or with large-scale systems. While there are some talks we have yet to announce, the majority are live.

tl;dr – Go buy your tickets for RICON. Do it right now. The early-bird price of $250 ends in just over a week, and for that reasonable fee you’re not only getting access to more than 20 amazing talks on distributed systems and Riak in production, but you’ll be part of a two day celebration that includes parties, lightning talks, amazing swag, talented developers and ops professionals from around the world, hack sessions, and much more. (Watch the blog and the @basho for details on the “much more”. )

The full schedule is here. Some of the highlights are below.

And if anyone needs convincing about why they should come to RICON, feel free to email me directly – mark@basho.com. I’m happy to answer any and all questions about why you and your teams should join us.

Mark

Keynotes

There will be three keynotes, and they will all be exceptional.

  • Opening, Day 1: Dr. Eric Brewer, VP, Infrastructure at Google and Professor of Computer Science at UC Berkeley (who also happens to be the creator of the CAP Theorem), will be kicking things off. RICON is more or less taking place on the anniversaries of both the CAP Theorem and the release of Amazon’s “Dynamo Paper”. This talk should be memorable.
  • Opening, Day 2: Gary Flake, CEO of Clipboard, and ex-Technical Fellow at Microsoft, will be sharing details on how and why he and his team chose Riak for a social application.
  • Closing, Day 2: UC Berkeley Professor Joseph Hellerstein will be closing the conference with a talk titled Keep CALM and Query ON.

Riak In Production

If you’re looking for reasons to run Riak in production from honest, articulate, experienced users, there will be no shortage. A portion of what’s planned:

  • Voxer’s Matt Ranney will be on hand to tell the crowd what he both loves about Riak and would like to see fixed. They have 100s of TBs of data in Riak spread over more than 60 machines.
  • The Bump team started with a MongoDB to Riak migration some months ago and have since built various apps and features on it. They will be presenting on how they built a Transaction-Logs based protocol with Riak and Haskell.
  • Various teams at Comcast rely on Riak in production. Michael Bevilacqua-Linn, a Principal Engineer, is joining us to share details on how Riak integrates with a custom in-memory system to power xfinity.net.
  • EnStratus CTO George Reese is presenting on the work he and his team are doing to move from MySQL to Riak in an effort build high-availability into their platform.
  • Dietrich Featherston, an engineer on the Boundary Team, is giving at talk called Modern Radiology for Distributed Systems that will highlight their work with Riak and show you why it’s important to take a “radiological view” of your entire system.

Distributed Systems In Production

Riak isn’t the only focus of RICON. Some of the talks that will touch on other distributed systems:

  • Dana Contreras from Twitter will be talking about how they managed to rebuild their massive infrastructure on the fly.
  • Mandi Walls’ talk about using Chef for distributed systems will be invaluable to anyone tasked with managing N machines at scale.
  • The inestimable Theo Schlossnagle is going in-depth on how you should approach monitoring clusters of machines in production.

Basho On Riak

A handful of Basho Engineers are giving talks on certain components of Riak and what future functionality you can expect from us and the community. Some of the highlights:

  • Sean Cribbs and Russell Brown are going deep on data structures in Riak. (Yes, actual data structures.)
  • Bryan Fink will shed some light on Riak Pipe, Riak’s MapReduce Engine, and how it can be used for large-scale data processing.
  • Kelly McLaughlin and Reid Draper are two of the primary engineers working on Riak CS, Riak’s cloud storage extension. They’ll be walking you through how and why we built it and why it’s valuable.
  • Ryan Zezeski is talking about the future of distributed search with Riak and Solr

Riak Cloud Storage – New Features and Product Update

August 21, 2012

Earlier this year we launched Riak CS – simple, available cloud storage built on Riak. We gave it an S3-compatible API, made it multi-tenant, and added per-user reporting on network and storage utilization. Riak CS provides the core features to build public or private clouds that are distributed, fault-tolerant and easy to scale.

Since then, we’ve rolled out a startup program to make Riak CS affordable for earlier-stage shops. And we’ve talked to a number of companies who wanted to license Riak CS based on the amount of storage they need, rather than the number of nodes, so now you can do that too. (Get in touch if you want a quote.) Riak CS docs are now also publicly available on our wiki.

Today we’re announcing some new features and additions to Riak CS, including more options for user admin, auth and object metadata, plus improved stats and troubleshooting. We’ll walk thru the Riak CS architecture, operations and new functionality in an upcoming webcast, or read on.

New in Riak CS

Better User Administration

We’ve beefed up the Riak CS user API so admins can now list all users, issue new credentials to a user, or disable a user. We’ve also got new configuration options to restrict user creation to admins or let anyone create a user directly.

List All Objects in a Bucket Much Faster

User objects in Riak CS are stored in flat namespaces called “buckets”. We sped up one of the most common bucket operations: listing all of its objects. Riak CS now uses MapReduce to look up objects, yielding significant performance gains.

Arbitrary Metadata

One of our top customer requests. Now you can store some additional metadata with your Riak CS objects – whatever is most useful to you.

New Basic Health Check

If you’ve ever set up Riak, you’ve probably used the basic health check to test Riak nodes. Now you can send an HTTP “ping” to Riak CS nodes as well to make sure they are responsive. The “ping” will also fail if the Riak CS node can’t reach the underlying Riak database (Riak CS nodes have a 1×1 mapping onto Riak nodes).

Better Inspection and Troubleshooting

We have added DTrace probing in Riak CS and we are working on SmartOS packaging. We will also be working on packaging for other platforms that support DTrace. Once these packages are available it should help operators who use DTrace platforms troubleshoot any issues with Riak CS and have more visibility into their stack. We also now expose more stats information on the Riak CS runtime by HTTP request.

Improved S3 API Coverage

We also added support for query parameter authentication, increasing compliance with S3’s REST Authentication scheme. This means that Riak CS users can now authenticate thru a request header or a URL-encoded query string parameter.

Stay tuned because we’re hard at work on the next Riak CS features and improvements. What do you want to see? We’d love to hear from you.