October 31, 2013
If you attended RICON West, we’d love to hear your feedback! Please fill out the survey here.
Thanks to everyone who helped make the sold-out RICON West a huge success! RICON has come a long way in just one year and we are excited to see how it grows and evolves in the future.
RICON West featured over 25 speakers from academia and industry, including speakers from Basho, Google, Microsoft Research, Netflix, Salesforce, Seagate, The Weather Company, and Twitter. Over two days, they discussed the theory, practice, and importance of running distributed systems in production as well as some predictions on what’s in store for the future. Over the next few weeks, we will be posting slides and videos from all of the talks on both ricon.io and the blog.
In case you missed it, we also received some great press. Here’s a quick recap:
- “Salesforce’s data-center design: ‘Go for web scale, and build it out of s**t!’“
- “What do we want? Strong consistency! When do we… oh, it’s in Riak v2“
What’s next for RICON? With three conferences already under our belt, we are excited to get to work on RICON Europe (our first international conference!) and continue RICON East. Keep an eye on Ricon.io and our blog for more details.
RICON West was also paired with a one-day Riak training. We plan on making these a more regular occurrence all over the country.
October 30, 2013
Today is the second day of RICON West, Basho’s distributed systems conference. If you weren’t able to make it out to San Francisco for the conference, tune in to our live stream to watch all of the great talks.
Day One of RICON West was an exciting one. Basho announced the availability of the Riak 2.0 Technical Preview (which Eric Redmond and Joseph Blomstedt described in more detail during their talks) and James Hughes of Seagate did the first live demo of the Kinetic Open Storage platform and Riak.
We also heard some amazing sessions from Pat Helland (Salesforce), Lindsey Kuper (Indiana University), Miles O’Connell (StackMob), Justin Shoffstall (Basho), Charlie Voiselle (Basho), Jeff Hodges (Twitter), Peter Bailis (UC Berkeley), Ryland Degnan (Netflix), and Derek Murray (Microsoft Research). We wrapped up Day One with Lightning Talks from across the industry and celebrated at Twenty Five Lusk (thanks for the drinks, Tower3 and Github!).
Today is another full day of amazing talks. Check out talks from Justin Sheehy (Basho), Diego Ongaro (Stanford University), Jeremy Ong (TBA), Jordan West (Basho), Susan Potter (Finsignia), Jason Brown (Netflix), Michael Bernstein (Code Climate), Sam Elliott (Basho), Andrew Thompson (Basho), Raja Selvaraj (The Weather Company), and Arvinda Gillella (The Weather Company). Additionally, Google Fellow, Jeff Dean, will be delivering the closing keynote on “The Tail at Scale: Achieving Rapid Response Times in Large Online Services.”
October 29, 2013
Throughout RICON West, we will be discussing many of the Riak 2.0 features (both in track sessions or during lightning talks), so keep your eyes on the live stream over the next two days. Videos of all sessions will also be made available after the conference.
Here is a look at some of the major enhancements available in Riak 2.0:
- Riak Data Types. Building on the eventually consistent counters introduced in Riak 1.4, Riak 2.0 adds sets and maps as new distributed data types. These Riak Data Types simplify application development without sacrificing Riak’s availability and partition tolerance characteristics.
- Strong Consistency. Developers have the flexibility to choose whether buckets should be eventually consistent (the default Riak configuration today that provides high availability) or strongly consistent, based on data requirements.
- Full-Text Search Integration with Apache Solr. Riak Search is completely redesigned in Riak 2.0, leveraging the Apache Solr engine. Riak Search in 2.0 supports the Solr client query APIs, enabling integration with a wide range of existing software and commercial solutions.
- Security. Riak 2.0 adds the ability to administer access rights and utilize plug-in authentication models. Authentication and Authorization is provided via client APIs.
- Simplified Configuration Management. Riak 2.0 continues to improve Riak’s operational simplicity by changing how, and where, configuration information is stored in an easy-to-parse and transparent format.
- Reduced Replicas for Multiple Data Centers. Riak Enterprise 2.0 can optionally store fewer copies of replicated data across multiple data centers to better maintain a balance between storage overhead and availability.
Ready to get started? Download the Technical Preview.
Please note that this is only a Technical Preview of Riak 2.0. This means that it has been tested extensively, as we do with all of our release candidates, but there is still work to be completed to ensure it’s production hardened. Between now and the final release, we will be continuing manual and automated testing, creating detailed use cases, gathering performance statistics, and updating the documentation for both usage and deployment.
Riak 2.0 Technical Preview: Deep Dive
Riak Data Types
In distributed systems, we are forced to trade consistency for availability (see: CAP Theorem) and this can complicate some aspects of application design. In Riak 2.0, we have integrated cutting-edge research on data types known as called CRDTs (Conflict-Free Replicated Data Types) pioneered by INRIA to create Riak Data Types. By adding counters, sets, maps, registers, and flags, these Riak Data Types enable developers to spend less time thinking about the complexities of vector clocks and sibling resolution and, instead, focusing on using familiar, distributed data types to support their applications’ data access patterns.
A more detailed overview of Riak Data Types is available that examines implementation considerations and the basics of usage.
In all prior versions, Riak was classified as an eventually consistent system. With the 2.0 release, Riak now lets developers choose when operations should be strongly or eventually consistent. This gives developers a choice between these semantics for different types of data. At the same time, operators can continue to enjoy the operational simplicity of Riak. Consistency preferences are defined on a per bucket type basis, in the same cluster.
A RICON West 2012 talk entitled, Bringing Consistency to Riak, shares much of the initial thinking behind this effort. In addition, the pull request that adds consistency to
riak_kv provides detailed information about related repositories and the implementation approach.
Redesigned Full-Text Search
Riak is a key/value store and the values are simply stored on disk as binary. With previous versions of Riak Search, Riak developers have long been able to index the content of these stored values. In Riak 2.0, Riak Search (code-named Yokozuna) has been completely redesigned and now uses the Apache Solr full-text document indexing engine directly. Together, Riak and Solr provide a reliable full-text context indexing solution that is highly available and built for scale. In addition, Riak Search 2.0 also fully supports the Solr client query APIs, which enables integration with existing software solutions (either homegrown or commercial).
The Basho engineers responsible for Yokozuna have created a resources page that includes recorded talks, Solr documentation links, and books on the topic.
Basho designed Riak with critical data in mind. Whether it’s data that affects revenue, user experience, or even a patient’s health (as is the case with the NHS), Riak ensures that this critical data is always available. However, often this critical data is also sensitive data. Riak 2.0 adds security to this data through the ability to administer access rights and plug-in various secure authentication models commonly used today.
The initial RFC that describes the security effort, including related Pull Requests, is available at github.com/basho/riak/issues/355.
Simplified Configuration Management
At Basho, we pride ourselves on providing operationally friendly software that functions smoothly when dealing with the challenges of a distributed system. In the past, configuration of Riak occurred in two files:
vm.args. Riak 2.0 changes how and where configuration information is stored. It no longer uses Erlang-specific syntax but, rather, provides a layout more suited for all operators and automated deployment tools. This layout is easy to parse and transparent for Riak administrators.
More information on the vision and specific implementation considerations are contained in the repository at github.com/basho/cuttlefish.
In versions of Riak prior to 2.0, keys were made up of two parts: the bucket they belong to and a unique identifier within that bucket. Buckets act as a namespace and allow for similar keys to be grouped. In addition, they provide a means of configuring how previous versions of Riak treated that data.
In Riak 2.0, several new features (security and strong consistency in particular) need to interact with groups of buckets. To this end, Riak 2.0 includes the concept of a Bucket Type. In addition to allowing new features without special prefixes in Bucket names, Riak developers and operators are able to define a group of buckets that share the same properties and only store information about each Bucket Type, rather than individual buckets.
More information about Bucket Types can be found in the Github Issue at github.com/basho/riak/issues/362. This issue describes the planned functionality, discussions about implementation, and includes related pull requests.
Change in Defaults for Sibling Resolution
Riak has always supported both application-side and timestamp and vector clock-based Last Write Wins server-side resolution. Prior to Riak 2.0, vector clock-based Last Write Wins has been the default. Moving forward, new clusters will hand off siblings to applications by default. This is the safest way to work with Riak, but requires developers to be aware of sibling resolution.
More Efficient Use of Physical Memory
Riak nodes are designed to manage the changing demands of a cluster as it experiences network, hardware, and other failures. To do this, Riak balances each node’s resources accordingly. Riak 2.0 has vastly improved LevelDB’s use of available physical memory (RAM) by allowing local databases to dynamically change their cache sizes as the cluster fluctuates under load.
In the past, it was necessary to specify RAM allocation for different LevelDB caches independently. This is no longer the case. In Riak 2.0, LevelDB databases that manage key/value or active anti-entropy data share a single pool of memory, and administrators are free to allocate as much of the available RAM to LevelDB as they feel is appropriate in their deployment. Detailed implementation documentation can be found in the basho/leveldb wiki.
Riak Ruby Vagrant Project
If you are interested in testing Riak 2.0, in a contained environment with the Riak Ruby Client, Basho engineer Bryce Kerley has put together the Riak-Ruby-Vagrant repository. In addition, this environment can be easily adapted to usage with other clients for testing the new features of Riak 2.0.
October 7, 2013
RICON West, Basho’s distributed systems conference, is quickly approaching at the end of October. This event will feature speakers from both academia and industry, presenting on a wide variety of distributed systems topics. This installment of RICON will be the largest to date and it would not be possible without our amazing sponsors.
Similar to the RICON speaker lineup, the sponsors stem from a variety of different industries. Current sponsors include Seagate, Engine Yard, Yammer, Google, SoftLayer, and Tower3. Additionally, this year RICON has its first media sponsorship from The Register. The Register’s Jack Clark has put together a list of the sessions that he’s most excited about attending in his article, “Distributed Systems Boffins Flock to RICON West.”
RICON West will be at the St. Regis in San Francisco from October 29-30th. In addition to the conference, Basho will be hosting a one-day Riak training the day before (October 28th). Be sure and grab tickets to both before they sell out!
October 1, 2013
On October 29-30th, RICON West will take over the St. Regis in San Francisco. RICON is Basho’s distributed systems conference that brings together engineers, developers, scientists, and architects. You can purchase tickets for this almost sold-out event here: ricon-west-2013.eventbrite.com/
This year’s keynote speaker is Jeff Dean, Google Fellow at Google Inc. His talk entitled, “The Tail at Scale: Achieving Rapid Response Times in Large Online Services,” will describe a collection of techniques and practices that lower response times in large distributed systems whose components run on shared clusters of machines, where pieces of these systems are subject to interference by other tasks, and where unpredictable latency hiccups are the norm, not the exception. He will also share examples of how these techniques are used in various pieces of Google’s systems infrastructure and in various higher-level online services.
RICON West also features speakers from academia and industry, including: Peter Bailis (UC Berkeley), Justin Sheehy (Basho), Pat Helland (Salesforce.com), Jeff Hodges (Twitter), Diego Ongaro (Stanford University), Susan Potter (Finsignia), Ryland Degnan and Jason Brown (Netflix), Miles O’Connell (StackMob), Derek Murray (Microsoft), Raja Selvaraj and Arvinda Gillella (The Weather Company), and many others.
If you’ll be in San Francisco on Oct. 28th, we will also be hosting a full-day Riak training. This training will teach you everything you need to know to start building highly available, scalable systems on Riak. Tickets to both the training and RICON are still available.
Be sure to grab tickets to RICON West before they sell out and see you in San Francisco!
September 30, 2013
While the biggest event of October is Basho’s distributed systems conference, RICON West, we will still be traveling the world to attend many other events this month. Here’s a look at where you can find us during the weeks leading up to RICON.
Monktoberfest: Basho’s Director of Marketing, Tyler Hannan, will be speaking at Monktoberfest on “Medieval Art, Collective Intelligence, and Language Abuse – The Ethos of Distributed Systems.” Monktoberfest will take place in Portland, ME from Oct. 3-4.
Erlang Factory Lite: Basho will have speakers at both the Chicago event (Oct. 4th) and the Berlin event (Oct. 16th). Check out talks from Chris Meiklejohn and Steve Vinoski to learn more about Riak, Erlang, and distributed systems.
CloudConnect Chicago: Basho is a sponsor and exhibitor of CloudConnect Chicago, taking place Oct. 21-23. Basho engineer, John Burwell, will also be speaking about building private clouds with Apache CloudStack and Riak CS.
O’Reilly Strata: Basho will be exhibiting and speaking at the upcoming O’Reilly Strata conference in New York from Oct. 28-30. Stop by our booth and find out why we will all be using distributed systems in the future.
August 8, 2013
RICON West, Basho’s distributed systems conference, will take place in San Francisco from October 29-30th. We’ve already started rolling out the speaker list, with 15 of the 23 speakers announced. Here are some of the highlights:
- Peter Bailis (UC Berkeley) – Bad As I Wanna Be: Coordination and Consistency in Distributed Databases
- Jeff Hodges (Twitter) – Practicalities of Productionizing Distributed Systems
- Ryland Degnan (Netflix) – Yuki: Functional Data Structures for Riak
- Lindsey Kuper (Indiana University) – LVars: Lattice-based Data Structures for Deterministic Parallelism
- Sam Elliott (Basho Technologies) – CRDTs: An Update (or maybe just a PUT)
- Miles O’Connell (StackMob) – More Than Just Data: Using Riak Core to Manage Distributed Services
- Michael Bernstein (Paperless Post) – Distributed Systems Archeology
- Charlie Voiselle and Justin Shoffstall (Basho Technologies) – The Seven-Layer Burrito; Troubleshooting a Distributed Database in Production
Be on the lookout next week when we announce the remaining speakers and the keynote. Until then, be sure and grab a ticket before they sell out – the early bird tickets are only available through August 29th – and make yourself known on the RICON West Lanyrd directory.
We look forward to seeing you in October.
June 17, 2013
RICON East, Basho’s distributed systems conference, took place last month in New York. Hundreds of developers and academics gathered for two days to learn how distributed systems are being used in production and where they’ll be in the future.
Over the next few weeks, we will be posting the videos of the talks on the RICON East Archive. These videos are open to anyone and feature speakers from various distributed systems backgrounds. Slides for all of the talks are also available in the Archive.
The first six videos are already available on the site. These videos are:
- “Automatically Scalable Computation” by Dr. Margo Seltzer, Herchel Smith Professor of CS at Harvard SEAS
- “Why is my Cache so Dumb? Smarter Caching with Pequod” by Neha Narula, PhD Candidate at MIT
- “Bloom: Big Systems from Small Programs” by Neil Conway, PhD Candidate at UC Berkeley
- “Large Scale Data Service as a Service” by Brian Akins, Senior Principal Architect at Turner Broadcasting System
- “Optimizing LevelDB for Performance and Scale” by Matthew Von-Maszewski, Software Engineer at Basho Technologies
- “Just Open a Socket – Connecting Applications to Distributed Systems” by Sean Cribbs, Software Engineer at Basho Technologies
Basho is also hosting another distributed systems conference, RICON West, in San Francisco on October 29-30th. We already have some great speakers lined up, including Jeff Dean (Google Fellow), Kate Matsudaira (Founder and CTO of Pop Forms), Peter Bailis (PhD Candidate at UC Berkeley), Justin Sheehy (CTO at Basho Technologies), Jeff Hodges (Distributed Systems Engineer at Twitter), and Diego Ongaro (PhD Candidate at Stanford University). Early bird tickets are on sale now.
Be on the lookout for more videos coming soon and we’ll see you at RICON West!
May 30, 2013
Basho’s distributed systems conference, RICON East, was chock full of amazing talks from academics and professionals from all industries. Luckily, for everyone who couldn’t attend, all of the talks were recorded and will be available on the RICON site soon.
In the meantime, we have posted the slides from the closing keynote, presented by Basho Chief Architect, Andy Gross. His talk, entitled, “Lessons Learned and Questions Raised (from building distributed systems),” goes over his experience building distributed systems and how the space is changing. Check out his slides below and be on the lookout for the complete video, which will be posted soon.
Tickets are now on sale for RICON West, which will take place October 29-30 in San Francisco. You can get early bird pricing now through August 29th.
May 23, 2013
Thank you to all who attended or tuned into the live-stream for RICON East this past week. We hope you had fun and learned a thing or two; we sure did.
We’ll be publishing the videos from RICON East over the coming weeks. Keep an eye on ricon.io/archive/2013/east.html for updates.
While RICON East may have just ended, we’re already busy working on RICON West.
RICON West will take place October 29-30 in San Francisco at the St. Regis Hotel. This will be our largest conference to date and we hope you’ll join us once again.
Tickets are on sale now, with early bird discounts through August. Each attendee will also get a personalized conference track jacket and a ticket to the after party at Twenty Five Lusk.
We even have a few speakers lined up already! Jeff Dean, Google Fellow at Google; Kate Matsudaira, Founder and CTO of Pop Forms; and Peter Bailis, PhD student at UC Berkeley will be speaking about their work with distributed systems, alongside Basho engineers.
If you’d like to present at RICON West, email email@example.com to submit a talk. We are accepting proposals through July 1st about anything distributed systems-related.
For more details, head on over to ricon.io/west.html
See you all in San Francisco in October.