At the most recent San Francisco Riak meetup, we had the pleasure to invite Uber engineer & past OSCON speaker Jeff Wolski (GitHub) to discuss his more recent work.
You may know Uber as the popular on-demand car service, but they’re so much more than that. Uber is innovating at the intersection of lifestyle and logistics at a rapid pace. To do so, they architect some of the most fascinating distributed architectures I know of.
A newer part of this ever-evolving distributed system is project Ringpop (also on GitHub). As Jeff puts it:
“Ringpop is an open-source Node.js library developed at Uber that brings application-layer sharding to many of their dispatching platform services.”
This additional abstraction layer, maintained through a consistent hashing ring familiar to any Riak enthusiast, provides a means by which Jeff can add additional dispatching services without service interruption.
To leverage this scalability while providing stability, Ringpop keeps in mind that no distributed network is always reliable. Jeff dedicates a portion of his talk to exploring these complexities and how SWIM gossip protocol is implemented to handle bad actors:
“Service instances that behave erratically, slow or otherwise, wreak havoc on the rest of the cluster by causing frequent and persistent changes to the state of the ring and ultimately, inconsistent hash ring lookups. In this talk you’ll hear about Ringpop, its implementation, and how we’ve had to employ a flap damping technique to suss out these bad actors to achieve higher levels of reliability for our services.”
The presentation, in the context of Ringpop, also shows how Uber relies on Riak for high availability. Riak acts as a persistent data store for a portion of new dispatching services as well as some functional extensions on top of Ringpop. For example, as objects are generated that require persistence, such as a new driver coming online and their associated mailbox of potential trips, these IDs are stored as keys within Riak.
These layers are used for further services, which rely on data stored in Riak, including:
- Stateful HTTP long-poll services
- Client/server sync services
- Rate limiters
- Geospatial services
It’s insanely interesting to see how Uber continues to scale as one of the most respected software companies today. To do so, Jeff notes a list of research that informs the design of Ringpop and will continue to be important to their development. What is often forgotten in our productivity-obsessed dev culture, is the importance of practice. In one Q&A, Jeff responds to a comment on the concern of creating a consistent hash on top of a consistent hash of Riak, he says “if it’s wrong I’ll go delete the repo right now.” His willingness to improve, even if it means deleting month’s worth of code, is refreshing to me.
There is a great deal of learning to be done if you’re looking to design a similar resilient set of services.
If you enjoy the video, you’ll love the documents mentioned. The Dynamo paper is close to our hearts at Basho since Riak is also based upon its goal. For further reading, check out BGP route flap damping, SWIM protocol, and Uber’s code on tchannel.
Keep sharing, learning, building and re-building,
April 28, 2014
On Monday, May 5th, Basho is co-hosting an open source happy hour and networking meetup. Partnering with GoGrid, we will bring together some of the greatest minds in open source to discuss current trends and where open source will be in the future. Drinks and appetizers will be provided. This meetup will take place from 6pm-8pm on May 5th at the 111 Minna Gallery in San Francisco. Space is limited, so be sure and RSVP quickly.
This meetup is also in conjunction with the Open Business Conference. Open Source has become ubiquitous in the enterprise and in the business layer, as more and more organizations are reaping its considerable benefits, including speed, efficiency and cost savings. Open Business Conference 2014 brings together the people who are building and deploying the latest in enabling technologies and solutions and teaches businesses how to put their data to use. Basho will be available to answer any questions about Riak and open source at the GoGrid booth. Open Business Conference takes place May 5-6 at the Palace Hotel in San Francisco.
To learn more about Basho’s partnership with GoGrid (and other partnerships), visit our Partners Page.
February 5, 2014
At the recent meetup for the New York Linux Users Group (NYLUG), Basho Technical Evangelist, Tom Santero, presented “An Introduction to Basho’s Riak.” In this talk, Tom explains how Riak addresses the challenges of concurrent data storage at scale. He discusses the various design decisions, tradeoffs made, and theories at work within Riak. He also provides guidance as to how you might deploy Riak in production and why.
In addition to introducing the basics of Riak and its key/value data model, Tom presents some of the exciting features being introduced with Riak 2.0. Riak Data Types adds counters, sets, and maps to Riak – allowing for better conflict resolution. They enable developers to spend less time thinking about the complexities of vector clocks and sibling resolution and, instead, focusing on using familiar, distributed data types to support their applications’ data access patterns.
You can watch Tom’s full talk below:
For more information about Riak and how it differs from traditional databases, check out the whitepaper, “From Relational to Riak.”
To see where Basho will be presenting next, visit the Events Page.
January 8, 2014
Online payments leader, Klarna, has been on a multi-year process to upgrade its monolithic legacy purchase system to a next generation, distributed, service-oriented system. Klarna’s goal has been to build a horizontally-scalable, masterless, purchase taking front-end system that can easily handle Klarna’s rapid business growth ambitions. Klarna deployed Riak as the distributed database underlying their new purchasing system.
Recently, Malcolm Matalka, programmer at Klarna, presented on using Riak. Calling Riak a “joy to use,” Malcolm discusses how important it was to build a system that would enable them to “not be afraid of losing machines.” He explains, in detail, the challenges and benefits of moving from a huge monolithic, transaction-based system that has served Klarna well for the past eight years.
Early in the project, Klarna recognized that they wanted a system that matched the aspirations of Amazon. Malcolm notes that Riak is built on top of “very strong theoretical foundations,” citing the Amazon Dynamo paper.
Some of the highlights from his presentation include:
- Klarna’s 9-node cluster running Riak already manages 40% of Klarna’s purchasing traffic
- As a distributed system, Riak has delivered “infinite availability”
- Rolling upgrades of Riak have been smooth. A heterogeneous cluster works fine and there is no need to worry about versioning
- In addition to Riak, Klarna’s next gen purchasing system is supported by Splunk, Graphite, and Op5 for logging and monitoring
October 14, 2013
Kivra is a Swedish company that provides secure digital mailboxes, allowing users to securely receive, upload, and store all postal mail. Their mailboxes help users organize bills and notifications while eliminating the environmental footprint created from paper mail.
At a Riak Meetup in Dublin earlier this year, Kivra CTO, Bip Thelin, spoke about how they currently use Riak. Originally, Kivra tried to build their platform on a SQL database. However, they quickly outgrew this system and decided to move their infrastructure to Erlang and Riak because of its scalability and resiliency. You can watch Bip’s full presentation below to learn more about why they chose to switch to Riak, how they built six Riak clusters, and some of the lessons they’ve learned.
Bip’s slides can also be viewed here:
For more companies that have switched from SQL to Riak, check out our Users Page.
September 16, 2013
At Gilt Groupe, each day at 12pm is peak site traffic time and their main user store receives 100,000 requests per minute. To handle these traffic spikes, they need a durable database that will stay up no matter what. They decided to look into using Riak for this main user store but, before they could bring it into production, they needed to performance test it.
With the help of two Basho engineers, Seth Thomas and Steve Vinoski, they were able to create a Riak test system in a week. They stress tested Riak and Riak passed with high marks. Riak also continued to perform well during a number of staged failure scenarios.
To ensure that Riak was actually as solid as the testing seemed to prove, they even simulated some extreme read and write loads. Zeusbench executed an extreme number of HTTP requests against the test service and reported metrics on performance. They also wrote a program that executed 1,500 insertions into Riak per second. Even with both these running, read/write performance in Riak never suffered.
Gilt Groupe was also able to kill multiple nodes in Riak while maintaining the production load. Riak continued to shine.
For more information about how Gilt Groupe tested Riak, check out their blog. Additionally, if you’re in New York tomorrow (September 17th), register for the Riak Meetup at Gilt Groupe’s office. Jim Englert (Gilt Groupe Lead Software Engineer) will be speaking about his experiences stress testing Riak and Basho’s Tom Santero will talk about what’s in store for the future of Riak.
March 21, 2013
Temetra is the most widely used meter management system for water in Ireland. They provide utility companies with a cloud service to store and manage meter data for utilities such as water, gas and electricity.
Each time one of Temetra’s utility customers reads a meter, that reading is stored by Temetra and made available to the utility. As Temetra found success at home and overseas, it resulted in huge amounts and variety of data: from the meter readings themselves, through meter location information and utility staff work schedules, to audit logs.
With the switch to smart meters, where one meter can now provide 35,000 readings per year instead of just four, Temetra needed to improve the scalability and availability of their platform. They chose Riak as their primary data store and use it to collect and process all meter data.
Paul Barry, Temetra’s CEO, told the London Riak meet-up why they chose Riak and how they went from their initial evaluation to deploying it in production. Check out his interesting talk here.
February 7, 2013
Basho and our community have a handful of events lined up for February 13th. We have official meetups/group hacks in at least seven cities in the US.
We hope to see you next week. If you can’t attend an official Meetup, throw a Riak hack or drink up in your city and email email@example.com to tell us about it.
Thanks for being a part of Riak.
- Speaker: Weston Jossey, Software Engineer, Tapjoy
- Talk Title: Huge Data Migrations to Riak Made Easy(er)
- Details and RSVP
- Speaker: Sean Cribbs, Software Engineer, Basho Technologies
- Talk Title: The Deep Riak
- Details and RSVP
New York City
- Speaker: Aaron Brown, Lead Systems Engineer, ideel
- Talk Title: Riak at ideel
- Details and RSVP
- Speaker: Adron Hall and You
- Talk Title: Riak Hack & Brew
- Details and RSVP
- Speaker: Robert Zuber, Co-Founder, Copious
- Talk Title: Riak in a Multi-Datastore Strategy at Copious
- Details and RSVP
- Speaker 1: Pavan Venkatesh, Technical Evangelist, Basho Technologies
- Talk Title 1: From Relational to Riak
- Speaker 2: Sajith Kizhakkiniyil, Software Infrastructure and Backend Architecture Support, Apollo Group
- Talk Title 2: Riak at Apollo
- Details and RSVP
- Speaker: Adron Hall and You
- Talk Title: Nerd Lunch and The Start of Seattle Riak
- Details and RSVP
July 19, 2010
Basho is growing. Fast. We are adding customers and users at a frenetic pace, and with this growth comes expansion in both team and locations. As some of you may have noticed, the Basho Team is not only becoming larger but more distributed. We now have people in six states scattered across four time zones pushing code and interacting with clients everyday.
First Order of Business
To bolster this growth and expansion, we did what any self-respecting tech startup would do: we opened an office in San Francisco. Several members of the Basho Team recently moved into a space at 795 Folsom, a cozy little spot a mere five floors below Twitter. (Proximity to the Nest was a requirement when evaluating office space.) We are calling it “Basho West.” There are four of us here, and we are settling in quite nicely.
If you are in the area and want to talk Riak, Basho, open source, coffee, etc., stop in and pay us a visit any time. Seriously. If you walk through the door of Suite 1028 with a Mac Book in hand and have a question about how to model your data in Riak, we’ll get out the whiteboard and help you out.
Second Order of Business
To make an immediate impact in the Bay Area, we thought it would be a great idea to get the first regularly scheduled Riak Meetup off the ground. We heard a rumor that there were a lot of people using or interested in databases out here, so we feel obliged to join the conversation. Here is the link to the San Francisco Riak Meetup group. If you’re in the Bay Area and want to meet with other like-minded developers and technologists to discuss Riak (and other database technologies) in every possible capacity, please join us.
Third Order of Business
Pop quiz: When did Basho Technologies open source Riak? We asked ourselves this the other day. As far we can tell, it was sometime during the first week and a half of August last year. “Huh,” we thought. “Wouldn’t it be great to have a little gathering to commemorate this event?” It sure would, so that’s what we are doing.
I mentioned above that we are starting a regularly scheduled Riak Meetup. To us, it made perfect sense to combine the inaugural Meetup with the event to celebrate Riak’s One Year Anniversary of being a completely open source technology.
The date of this gathering is Monday, August 9th. The exact time and location still needs to be solidified. We’ll be announcing that within the next few days. But put it on your calendar now, as you will not want to miss this. In addition to food, drink, and exceptional overall technical discussion and fireworks, here is what you can expect:
- A talk from Dr. Eric Brewer, Basho Board Member and Father of the CAP Theorem
- A few words from the team at Mochi Media about their experiences running Riak in production
- A short talk from Basho’s VP of Engineering, Andy Gross, on the state of Riak and the near term road map
If you have any other suggestions about what you would like to see at this event, just leave us a message or an idea on the Meetup page linked above.
- Come visit the new Basho Office at 795 Folsom, Suite 1028
- Join the Riak Meetup Group
- Come be a part of the Riak One Year Anniversary Celebration
And stay tuned, because things are only going to get more exciting from here.