At the most recent San Francisco Riak meetup, we had the pleasure to invite Uber engineer & past OSCON speaker Jeff Wolski (GitHub) to discuss his more recent work.
You may know Uber as the popular on-demand car service, but they’re so much more than that. Uber is innovating at the intersection of lifestyle and logistics at a rapid pace. To do so, they architect some of the most fascinating distributed architectures I know of.
A newer part of this ever-evolving distributed system is project Ringpop (also on GitHub). As Jeff puts it:
“Ringpop is an open-source Node.js library developed at Uber that brings application-layer sharding to many of their dispatching platform services.”
This additional abstraction layer, maintained through a consistent hashing ring familiar to any Riak enthusiast, provides a means by which Jeff can add additional dispatching services without service interruption.
To leverage this scalability while providing stability, Ringpop keeps in mind that no distributed network is always reliable. Jeff dedicates a portion of his talk to exploring these complexities and how SWIM gossip protocol is implemented to handle bad actors:
“Service instances that behave erratically, slow or otherwise, wreak havoc on the rest of the cluster by causing frequent and persistent changes to the state of the ring and ultimately, inconsistent hash ring lookups. In this talk you’ll hear about Ringpop, its implementation, and how we’ve had to employ a flap damping technique to suss out these bad actors to achieve higher levels of reliability for our services.”
The presentation, in the context of Ringpop, also shows how Uber relies on Riak for high availability. Riak acts as a persistent data store for a portion of new dispatching services as well as some functional extensions on top of Ringpop. For example, as objects are generated that require persistence, such as a new driver coming online and their associated mailbox of potential trips, these IDs are stored as keys within Riak.
These layers are used for further services, which rely on data stored in Riak, including:
- Stateful HTTP long-poll services
- Client/server sync services
- Rate limiters
- Geospatial services
It’s insanely interesting to see how Uber continues to scale as one of the most respected software companies today. To do so, Jeff notes a list of research that informs the design of Ringpop and will continue to be important to their development. What is often forgotten in our productivity-obsessed dev culture, is the importance of practice. In one Q&A, Jeff responds to a comment on the concern of creating a consistent hash on top of a consistent hash of Riak, he says “if it’s wrong I’ll go delete the repo right now.” His willingness to improve, even if it means deleting month’s worth of code, is refreshing to me.
There is a great deal of learning to be done if you’re looking to design a similar resilient set of services.
If you enjoy the video, you’ll love the documents mentioned. The Dynamo paper is close to our hearts at Basho since Riak is also based upon its goal. For further reading, check out BGP route flap damping, SWIM protocol, and Uber’s code on tchannel.
Keep sharing, learning, building and re-building,
March 24, 2014
When selecting a NoSQL solution, there are many options to choose from, each different and with their own benefits depending on your use case. To help you decide what the right choice for your needs may be, there are two amazing events this week where many NoSQL providers (including Basho) will be speaking.
The first is in conjunction with Ad:Tech in San Francisco. For advertisers to stay competitive in the modern landscape, the need to crunch massive amounts of consumer profile data and enable real-time bidding has made NoSQL the gold standard in database technology. That’s why Basho partner, GoGrid, will be hosting the panel, “NoSQL: Digital Advertising’s ‘Bad Boy’ Database Comes of Age” at 111 Minna Gallery. Speakers from Basho, Couchbase, DataStax, and MongoDB will be there to discuss how NoSQL is helping advertisers push the envelope now, and what is to come in 2014. This panel will take place on Wednesday, March 26th at 5:30pm. Registration is free and tickets are still available.
The other is hosted by the New York Software Engineers. This meetup, “The Battle of Distributed Databases – Data Modeling in the Enterprise Ecosystem,” will address some of the challenges the NoSQL community faces in enterprise adoption. Casey Rosenthal, Director of Professional Services at Basho, will be speaking about Riak and its adoption with 30% of the Fortune 50. This meetup will take place on Wednesday, March 26th at 7pm at Foursquare’s office. Be sure and register for this free event.
To see how enterprises are using Riak, check out the Users Page.
In addition to these meetups, Basho will be at multiple other events and conferences. A complete list can be found on our Events Page.
March 5, 2014
This month features a wide range of developer events, major conferences, and various meetups. Here’s a look at some of where we’ll be in March.
Erlang Factory: Erlang Factory SF brings together the rapidly growing community that uses Erlang in order to showcase the language and its various application to today’s distributed environments. Come hear talks by Basho’s own Tom Santero, Joseph Blomstedt, Chris Meiklejohn, and Joe DeVivo. We will also have a table set up to answer any questions about Riak. Erlang Factory takes place March 6-7 in San Francisco.
Game Developer’s Conference: GDC is the largest gaming industry event and will take place March 17-21 in San Francisco. Eric Liaw (Co-Founder and Developer, Quark Games) and Seth Thomas (Technical Evangelist, Basho) will be speaking on “Riak for Gaming” on Thursday, March 20th at 11:30am. Basho will also be exhibiting, so stop by and grab a t-shirt.
Clojure/West: Clojure/West takes place March 24-26 in San Francisco. Basho engineer, Reid Draper, will be speaking on “Powerful Testing with simple-check” and the Basho team will be available to answer questions about Riak.
Meetups: On March 12th, Hector Castro (Basho Technical Evangelist) will talk about Riak CS at OpenStack Boston. Both Tom Santero (Basho Technical Evangelist) and Hector Castro will talk about Riak CS and the underlying architecture, Riak, at OpenStack Connecticut on March 18th. Tom Santero will present on Highly Available Applications in the Cloud on March 19th at OpenStack New York. Finally, Hector Castro will talk about Riak CS and Riak at OpenStack Philly on March 20th.
For a full list of where we’ll be, check out the Events Page.
February 3, 2014
2014 is an exciting year for Basho and, as usual, we are traveling the world to let you know what we’re up to. Here’s a look at where we’ll be this February.
LA Ruby Conf 2014: Basho is a proud sponsor and we will be in LA to chat Riak and answer any questions you may have. LA Ruby Conf takes place February 6-8.
New York Meetup: On February 10th, Basho Technical Evangelist, Hector Castro, will present on “Supporting Riak and Riak CS Deployments with Chef” at DigitalOcean. He will explore the history of maintaining the Riak/Riak CS cookbooks and discuss how people are using them. He will also discuss how Basho is planning to revamp the cookbooks to take advantage of features coming in Riak 2.0.
O’Reilly Strata: Basho is a proud sponsor of O’Reilly Strata in Santa Clara (February 11-13). Be sure to stop by our booth to learn more about Riak and grab some swag.
Big Ruby Conf: Basho Technical Evangelist, Hector Castro, will be presenting “Throw Some Keys On It: Data Modeling for Key/Value Data Stores by Example” on February 21st. Big Ruby Conf takes place in Dallas, TX from February 20-21.
Code PaLOUsa: Code PaLOUsa takes place in Louisville, KY from February 24-26. On February 24th at 9am, we will be hosting a Riak workshop. In addition, Alex Moore (Basho Client Services Engineer) will speak on “Scaling Your Data Safely for Fun and Profit with Riak,” Sean Cribbs (Basho Software Engineer) will speak on “In Search of the Software Ursatz,” and John Daily (Basho Technical Evangelist) will speak on “Erlang, or How I Learned to Stop Worrying and Let Things Fail.”
Open Source Conference: Open Source Conference takes place from February 28-March 1 in Tokyo, Japan. Kaz Suzuki from Basho will be presenting an introduction to Riak and Riak CS. He will also be demoing Riak CS at the Basho booth.
For a full list of where we’ll be, check out the Events Page.
August 5, 2013
This month, Basho will be at Meetups and other events all over the world – presenting on all things Riak and distributed systems. Here are some of the highlights of where we’ll be in August:
Boston Meetup: Tonight, David Bishop (Lead Systems Administrator at Synacor) will discuss his experience using Riak Enterprise to power a cross-datacenter video metadata product behind some of the biggest ISP’s portals in the business.
DC Meetup: This Meetup will be focused on using different technologies to power mobile apps. It will feature case studies from Rovio and Zope Corporation. This event will take place on August 8th and you can register here.
Los Angeles Meetup: Learn about the new features and updates available with Riak 1.4, including eventually consistent counters. You can register for this August 13th event here.
Berlin Meetup: This eBay Meetup will feature two speakers, including Basho Chief Architect Andy Gross. Andy will speak about the resurgence in interest in both theoretical and applied distributed systems, explore new areas of promising research, and provide practical advice for dealing with systems in our new distributed world. You can register for this August 15th event here.
PuppetConf: Basho will be attending PuppetConf in San Francisco from August 22-23. Come visit our booth and meet our Director of Community, Mark Phillips and our Solutions Engineer, Pavan Venkatesh. Bring any Riak questions and get some great swag.
This is just a few of the events we’ll be at throughout August. For a complete list, check out the Events Page.
July 8, 2013
This month, the Basho team will be attending and speaking at a wide variety of conferences, as well as hosting many meetups all over the world. If you’re going to be at any of these events, we’d love to meet you! Feel free to contact us if you’d like to meet or if you have any questions about Riak.
Below are some highlights of where we’ll be:
Lambda Jam: On July 8-10 in Chicago, IL, Basho will be speaking at Lambda Jam. John Daily will be speaking on “Distributed Programming with Riak Core and Pipe,” Steve Vinoski will be speaking on “Addressing Network Congestion in Riak Clusters,” and Sean Cribbs and Chris Meiklejohn will be speaking on “Functional Web Applications with Webmachine.”
O’Reilly OSCON: Eric Redmond will be speaking on “Distributed Patterns in Action” at O’Reilly OSCON in Portland, OR (July 22-26).
DevCon5: Sarah Drasner will be speaking about using “CSS Animation to Tell a Story” at DevCon5 in New York City (July 24-25).
For a full list of where we’ll be throughout the year, visit our Events Page.
April 23, 2013
During the rest of April, Basho will be speaking and sponsoring events around the United States and internationally. If you want to meet up with a Basho team member at one of these events, contact us to set up a time, or send us a note on Twitter. Below are some of the highlights:
NY Tech Day: Basho will be exhibiting at NY Tech Day (April 25) in New York, a massive science fair where entrepreneurs can exhibit their startups to thousands of consumers, investors, first adopters, job seekers, major companies, press and media.
NoSQL Matters: Basho is sponsoring NoSQL Matters (April 26-27) in Cologne, Germany. Additionally, Basho engineers, Sean Cribbs and Eric Redmond, will be speaking about Riak Technologies.
RailsConf: Basho will be attending RailsConf (April 29-May 2) in Portland. It is the largest gathering of Rails developers (and most of the time, Rubyists) in the world, drawing world-class developers and companies together to see the state of the art in Rails and web development.
We hope to see you at one of these events! For a full list of events this month and in upcoming months, visit our Events Page.
March 11, 2013
Nearly each day this month, we will be speaking at conferences, hosting meetups, and sponsoring events. For a full list of events, visit our Events Page. If you want to meet up with a Basho team member at one of these events, contact us to set up a time. Below are some of the highlights:
GigaOM Structure: Basho will be speaking at two different sessions at GigaOM Structure (March 20-21) in New York. Come hear Basho CTO, Justin Sheehy, and Technical Evangelist, Tom Santero, speak, stop by our booth, or attend our cocktail reception on March 20th.
Game Developer Conference 2013: Basho Chief Architect, Andy Gross, will be speaking at the Game Developer Conference at a session titled “Gaming on NoSQL: Building Available, Fast Services with Riak.” GDC will be held March 25-29 in San Francisco. Check out our session and booth to learn more about how gaming platforms can use Riak.
Meetups: This month, we are hosting a number of meetups all over the country. If you’re in Austin, come visit us at BlackLocus on March 11th, if you’re in Seattle, visit us on March 13th at Blue Box Group, if you’re in Chicago, visit us on March 14th at Braintree, or if you’re in Boston, check us out on March 27th at Basho’s Cambridge office. We’ll also be at Riot Games in LA on March 19th and in Portland on March 28th at NedSpace.
Sponsored Events: Basho will be sponsoring Erlang Factory 2013 in San Francisco (March 18-22), Clojure/West in Portland (March 18-20), Open Analytics Summit in Arlington, VA (March 25), and Monitorama in Boston (March 28-29).
Hope to see you soon!