April 8, 2015
At the beginning of 2015, Adam Wray, our CEO and president, made a bold statement in a post entitled Basho is Back! Record Year and a Strong Start to 2015 he claimed:
At Basho we are focused on establishing product value and trust, while projecting a vision that our customers and community can invest in long term. In 2014 we built a strong foundation for growth in 2015 and beyond. This year one of our core objectives is to be seen by the marketplace as the leader in unstructured data! With this team and our product vision, I fully believe we can become the #1 NoSQL provider in the space.
In my role as the VP of Product and Marketing, I have the opportunity to shape our product based on customer and partner feedback as well as research into market direction. We are committed to providing the best multi-model solution for Big Data applications that leverage unstructured data in their active workloads. In fact, Basho has led the industry in adoption of multi-model solutions since beginning to offer key/value and object storage in 2013.
Over the last week or so you have seen us release Riak CS 2.0 and updated, Basho supported client libraries for Node.js and .NET. We will also release Riak 2.1 in the next few days with key performance enhancements. Basho is, presently, the leader in high availability and scale for distributed, active workloads and our increased focus on performance will result in performance enhancements for both Riak KV and Riak CS throughout 2015.
The updates to Riak 2.1 include numerous changes driven by our perspective on market trends and direction. Chief among these is the emphasis on performance and simplification for both developers and operations.
Enhancements to Riak 2.1 have increased write speeds by more than 2x for write-heavy workloads.
Riak 2.1 introduces the concept of “write once” buckets, buckets whose entries are intended to be written exactly once, and never updated or over-written. These write once buckets optimize Riak performance for immutable data which is a key design pattern for many Big Data applications.
The write_once property is applied to a bucket type and may only be set at bucket creation time. Once a bucket type has been set with this property and activated, the write_once property may not be modified.
This capability is extremely important for our customers, partners, and prospects who are writing and deploying IoT applications and whose data model includes immutable data workflows. We will continue to invest in performance in 2015 to drive speeds for write-heavy and other common workloads.
Basho Supported Clients
Basho has always maintained a series of supported client libraries for popular languages. With Riak 2.1, we have broadened the support by adding support for additional key languages used in the development of business applications. We are pleased to announce the inclusion of Basho-supported client libraries for Node.js and .NET. In addition, we have enhanced our support for PHP enabling easier integration for those building real-time web applications.
New Monitoring Statistics & Integrations
Once a Big Data application itself has been built, it is necessary to ensure that the cluster can be actively monitored. The addition of more than 200 supplementary Riak statistics enables fine-grained monitoring of individual node and cluster health. For example, you can monitor statistics for each Riak Data Type (CRDTs) measuring Get, Put, Update and Merge times at multiple percentiles. In addition, you can measure index and query latency alongside throughput for Riak Search (Solr). These statistics enable you to monitor the impact your application design has on the cluster. In addition, Basho has integrated these monitoring statistics with Nagios, New Relic, and Zabbix further expanding integrations with both hosted and on-premise monitoring solutions.
OS X Installers
In addition to clients and monitoring, we have invested in several new and/or updated installation options for Riak. Many application developers use OS X as their primary development machine. Basho already provides a simple project, riak-dev-cluster, for quickly getting started with a 5 node Riak Cluster. Now we are making it even easier by offering an OS X installer that lets you locally deploy a single node of Riak, for development purposes, with a series of simple clicks.
We continue our commitment to our community by working with the open-source contributors to our Chef, Puppet, and Ansible tools to ensure they are optimized for use with this release. In fact, improvements to the puppet-riak module make it one of the first to be built on Puppet 4.0, the latest release from Puppet Labs. To ensure clarity, and broader commitment to open-source development, we have arranged repositories driven by community contribution into the Basho Labs organization on Github. While our core codebase remains in the Basho organization, and undergo a rigorous review process, the Basho Labs invites community commitment and is actively monitored.
As if this wasn’t enough, we have also worked closely with Cloudsoft to release tested, optimized Riak blueprints. These blueprints enable the deployment of applications faster, and easier, across a variety of cloud service provider including AWS and SoftLayer. One-click, multiple providers.
Cloudsoft AMP blueprints are available to spin up a Riak cluster, a Riak cluster with an example application and Riak clusters in a multi-datacenter configuration.
Riak CS 2.0
It is with some pleasure that we are able to announce that Riak CS 2.0 is now generally available. This represents a major milestone in the lifecycle and development of Basho’s object storage offering. Riak provides the only true multi-model platform for the persistence and storage of a variety of unstructured data. With Riak CS 2.0, we have achieved seamless integration with the underlying Riak 2.0 codebase. This results in all the operational benefits of Riak 2.0 being included in Riak CS.
It would be remiss to not highlight that Riak CS 2.0 now provides enhanced conflict resolution that simplifies development, making it easier to reduce the likelihood of data conflicts and sibling growth in an eventually consistent system. This is achieved by leveraging the dotted version vector system introduced in Riak 2.0 enabling drastically simplified operational effort. This approach is coupled with the simplified configuration management presented initially in Riak 2.0 allowing for human-readable, and machine-parseable configuration files that are easily integrated with the orchestration tools that the enterprise prefers.
Getting started with Riak is easier than ever before thanks to the effort in simplifying the installation process for OS X. Designing and implementing a system for active workloads, whether a new design or replacement for existing infrastructure, often begins with a conversation with a member of our Solution Architecture team. They are available for onsite or remote discussions to educate your team on the practical considerations of implementing Riak for unstructured workloads and Big Data applications.
Vice President, Product & Marketing
November 12, 2013
The first email:
Occasionally, riak seems to not store an object I try to save. I have run tcpdump on the node receiving the request to ensure it is receiving the http packets with the correct JSON from the client. When the issue occurs the node is in fact receiving the request with the correct JSON.
Riak is designed to accommodate server and network failures without ever losing committed writes, so this led to a quick response from Basho’s engineers.
After some discussion, a vital piece of information was revealed:
One other thing that might be worth mentioning here is the writes I’m mentioning are actually updates to existing objects. The object exists, an attempt to write an update for the object appears to be received by a node, but the object maintains it’s original value.
Riak was dropping updates rather than writes, which is a horse of a different color. To see why updates are much more problematic for any distributed database, read on.
In a database that runs on a single server, setting aside any complications introduced by transactions or locks, the second of two updates to the same record will overwrite the first. Last write wins.
With Riak’s simplest conflict resolution behavior, the second of two updates to the same object may or may not overwrite the first, even if those two updates are spaced far apart. Last write wins, except when it doesn’t, but even then it does.
The problem is simple: there is no reliable definition of “last write”; because system clocks across multiple servers are going to drift.
On a single server, there’s one canonical clock, regardless of accuracy. The system can always tell which write occurred in which order (assuming that the clock is always increasing; setting a clock backwards can cause all sorts of bad behavior).
So, back to our original problem with lost updates:
The nodes were a bit out of synch (up to 30 seconds… looking into why ntp wasn’t working!). So far it appears this was the issue.
If two updates to the same object occur within 30 seconds in such an environment, the end result is unpredictable.
Taming the Beast
The conclusion drawn from the discussion was to implement (and, hopefully, to monitor) time synchronization. This is a step in the right direction, and one that every distributed system should implement, but there are more powerful and instructive lessons to impart.
Some of this discussion requires awareness of siblings, vector clocks, and related arcana. If you wish to read more about these topics, Basho’s earlier blog post Understanding Riak’s Configurable Behaviors: Part 1 provides sufficient context. (You can find links in the epilogue to the full series, but part 1 covers the necessary background for this post.)
If instead you decide you’d like to avoid reading about and dealing with such complexities entirely, skip over the Nitty Gritty section to The Land of Milk and Honey.
One approach that should generally be employed when writing Riak applications is to supply vector clocks with each update. It’s not clear in this particular scenario that it would have helped, but it certainly can’t hurt. Giving Riak more information to track causal history is never a bad thing.
Forcing the Last Write to Win
A rather non-obvious approach is to take the default last write wins conflict resolution one step further.
As discussed in part 1 of the configurable behaviors blog series, there are two closely-related configuration parameters that determine how Riak approaches conflict resolution:
last_write_wins. The former indicates whether Riak should keep all conflicts for the client to resolve; the latter is our concern at the moment.
allow_mult is set to
true will instruct Riak to always overwrite existing objects, ignoring the timestamps stored with them.
So, nominally, this achieves what we earlier implied to be impossible: the last write truly does win, regardless of clock consistency.
The problem is that we’ve just punted the problem down the road a bit. Yes, all servers that receive an object will blindly write it, but any servers that don’t receive it due to network partition or server failure will still retain an older value, and depending on clock consistency the older value may still win once the network or server failure is corrected.
Broadly speaking, if you’re going to have data consistency problems, it’s best for that to be obvious and easily detectable during testing stages. This “solution'; would have made the situation much harder to recognize before production.
Stopping Last Write Wins
At least in part to limit the complexity of developing applications, Basho decided to specify Riak’s default configuration as
allow_mult=false, which requires the database to resolve conflicting writes internally.
As we’ve seen, Riak isn’t exactly a genius at resolving conflicting writes. Beyond the challenges of clock consistency, Riak treats objects as opaque and has no awareness of business logic.
It’s almost always better to bite the bullet: instruct Riak to retain all conflicting updates as siblings (via
allow_mult=true) and write your application to deal with them appropriately.
Note: We are planning to change the default setting for
true in Riak 2.0, but please check the documentation and your configuration before assuming either behavior.
The Land of Milk and Honey
Distributed data types
Creating data types that can survive network partitions and self-heal has long been a goal for our engineers. With Riak 1.4, Basho introduced distributed counters; with 2.0, Riak will have a larger suite of distributed data types that can resolve conflicts internally, notably including sets and maps.
Although 2.0 is not yet released, a technical preview is available.
It is also possible to define such Riak Data Types (known formally as CRDTs) at the application layer. See the two-part blog series Index for Fun and for Profit and Indexing the Zombie Apocalypse With Riak for more information.
Also with 2.0, Riak will include the option of designating certain data as strongly consistent, meaning that the servers that hold a piece of data will have to agree on any updates to that data.
As appealing as that may sound, it is impossible to guarantee strong consistency without introducing coordination overhead and constraining Riak’s ability to continue to allow for requests when servers or networks have failed.
And aren’t low latency and high availability the reasons you’re using Riak?
The Silver(*) Bullet: Immutability
(* or at least stainless steel)
The rise of “big data” is linked to a resurgence of interest in functional programming, which is particularly well-suited for processing large data sets. (See Dean Wampler’s Lambda Jam talk Copious Data for an interesting exposition of this idea.)
One of the key tenets of functional programming is that data is immutable, meaning that destructive updates are not (typically) allowed.
The relational data model does not offer much (any?) support for immutable data, but it is a powerful concept. At Basho’s inaugural RICON conference Pat Helland gave a talk entitled Immutability Changes Everything which goes into more detail.
While it isn’t necessarily true that immutability solves everything with distributed systems, it’s a great start. Without data updates, there are no conflicts.
See the configurable behaviors epilogue (specifically, the discussion of Datomic) for a discussion of configuration tweaks to Riak to take better advantage of immutable data for low latency.
If your distributed system isn’t explicitly dealing with data conflicts, any correct behavior it exhibits is more a matter of good luck than of good design.
If your distributed database relies on clocks to pick a winner, you’d better have rock-solid time synchronization, and even then, it’s unlikely your business needs are served well by blindly selecting the last write that happens to arrive.
Riak provides powerful tools for helping address the inherent challenges of distributed data, but they have to be used to be useful.