April 8, 2015
At the beginning of 2015, Adam Wray, our CEO and president, made a bold statement in a post entitled Basho is Back! Record Year and a Strong Start to 2015 he claimed:
At Basho we are focused on establishing product value and trust, while projecting a vision that our customers and community can invest in long term. In 2014 we built a strong foundation for growth in 2015 and beyond. This year one of our core objectives is to be seen by the marketplace as the leader in unstructured data! With this team and our product vision, I fully believe we can become the #1 NoSQL provider in the space.
In my role as the VP of Product and Marketing, I have the opportunity to shape our product based on customer and partner feedback as well as research into market direction. We are committed to providing the best multi-model solution for Big Data applications that leverage unstructured data in their active workloads. In fact, Basho has led the industry in adoption of multi-model solutions since beginning to offer key/value and object storage in 2013.
Over the last week or so you have seen us release Riak CS 2.0 and updated, Basho supported client libraries for Node.js and .NET. We will also release Riak 2.1 in the next few days with key performance enhancements. Basho is, presently, the leader in high availability and scale for distributed, active workloads and our increased focus on performance will result in performance enhancements for both Riak KV and Riak CS throughout 2015.
The updates to Riak 2.1 include numerous changes driven by our perspective on market trends and direction. Chief among these is the emphasis on performance and simplification for both developers and operations.
Enhancements to Riak 2.1 have increased write speeds by more than 2x for write-heavy workloads.
Riak 2.1 introduces the concept of “write once” buckets, buckets whose entries are intended to be written exactly once, and never updated or over-written. These write once buckets optimize Riak performance for immutable data which is a key design pattern for many Big Data applications.
The write_once property is applied to a bucket type and may only be set at bucket creation time. Once a bucket type has been set with this property and activated, the write_once property may not be modified.
This capability is extremely important for our customers, partners, and prospects who are writing and deploying IoT applications and whose data model includes immutable data workflows. We will continue to invest in performance in 2015 to drive speeds for write-heavy and other common workloads.
Basho Supported Clients
Basho has always maintained a series of supported client libraries for popular languages. With Riak 2.1, we have broadened the support by adding support for additional key languages used in the development of business applications. We are pleased to announce the inclusion of Basho-supported client libraries for Node.js and .NET. In addition, we have enhanced our support for PHP enabling easier integration for those building real-time web applications.
New Monitoring Statistics & Integrations
Once a Big Data application itself has been built, it is necessary to ensure that the cluster can be actively monitored. The addition of more than 200 supplementary Riak statistics enables fine-grained monitoring of individual node and cluster health. For example, you can monitor statistics for each Riak Data Type (CRDTs) measuring Get, Put, Update and Merge times at multiple percentiles. In addition, you can measure index and query latency alongside throughput for Riak Search (Solr). These statistics enable you to monitor the impact your application design has on the cluster. In addition, Basho has integrated these monitoring statistics with Nagios, New Relic, and Zabbix further expanding integrations with both hosted and on-premise monitoring solutions.
OS X Installers
In addition to clients and monitoring, we have invested in several new and/or updated installation options for Riak. Many application developers use OS X as their primary development machine. Basho already provides a simple project, riak-dev-cluster, for quickly getting started with a 5 node Riak Cluster. Now we are making it even easier by offering an OS X installer that lets you locally deploy a single node of Riak, for development purposes, with a series of simple clicks.
We continue our commitment to our community by working with the open-source contributors to our Chef, Puppet, and Ansible tools to ensure they are optimized for use with this release. In fact, improvements to the puppet-riak module make it one of the first to be built on Puppet 4.0, the latest release from Puppet Labs. To ensure clarity, and broader commitment to open-source development, we have arranged repositories driven by community contribution into the Basho Labs organization on Github. While our core codebase remains in the Basho organization, and undergo a rigorous review process, the Basho Labs invites community commitment and is actively monitored.
As if this wasn’t enough, we have also worked closely with Cloudsoft to release tested, optimized Riak blueprints. These blueprints enable the deployment of applications faster, and easier, across a variety of cloud service provider including AWS and SoftLayer. One-click, multiple providers.
Cloudsoft AMP blueprints are available to spin up a Riak cluster, a Riak cluster with an example application and Riak clusters in a multi-datacenter configuration.
Riak CS 2.0
It is with some pleasure that we are able to announce that Riak CS 2.0 is now generally available. This represents a major milestone in the lifecycle and development of Basho’s object storage offering. Riak provides the only true multi-model platform for the persistence and storage of a variety of unstructured data. With Riak CS 2.0, we have achieved seamless integration with the underlying Riak 2.0 codebase. This results in all the operational benefits of Riak 2.0 being included in Riak CS.
It would be remiss to not highlight that Riak CS 2.0 now provides enhanced conflict resolution that simplifies development, making it easier to reduce the likelihood of data conflicts and sibling growth in an eventually consistent system. This is achieved by leveraging the dotted version vector system introduced in Riak 2.0 enabling drastically simplified operational effort. This approach is coupled with the simplified configuration management presented initially in Riak 2.0 allowing for human-readable, and machine-parseable configuration files that are easily integrated with the orchestration tools that the enterprise prefers.
Getting started with Riak is easier than ever before thanks to the effort in simplifying the installation process for OS X. Designing and implementing a system for active workloads, whether a new design or replacement for existing infrastructure, often begins with a conversation with a member of our Solution Architecture team. They are available for onsite or remote discussions to educate your team on the practical considerations of implementing Riak for unstructured workloads and Big Data applications.
Vice President, Product & Marketing
Riak shines when scaling horizontally with commodity hardware. A question frequently arises: should a redundant array of independent disks (RAID) be used, and if so, how?
Key reasons RAID is used:
- Disk-level fault tolerance
Let’s take a look at each in turn.
Particularly when using a hardware controller, RAID can offer significant read and write performance enhancements.
Optimizing writes involves striping, which reduces fault tolerance. In general, striping also provides excellent read performance. Mirrored configurations reduce performance on writes, but tend to provide excellent read performance. A balance can be struck by mixing stripes and mirrors.
Below is a table to help make the trade-offs clearer:
|RAID Level||Type||Read||Write||Fault Tolerance||Disk Minimum||Usable Space*|
|RAID 10||Stripe + Mirror||Excellent||Good||Good||4||1000GB|
*Usable space calculations assume all disks are
Tools such as
basho_bench can help you evaluate whether such a boost is useful for your environment and application needs.
Disk-level fault tolerance
Riak provides a layer of fault tolerance by distributing data to 3 servers by default. If a disk or server fails, it can be replaced as a matter of course without worrying about the data that was on it.
However, a RAID setup that allows for disk failures without crashing the file system (or entire system) has its benefits. A node utilizing RAID can still participate in reads and other cluster-wide operations. It can also ease the burden of replacing failed hardware so that the node can be reinserted into the cluster faster.
An example repair workflow:
- Disk failure detected by RAID
- Leave node up until a spare disk is acquired
- Shut down Riak and take the node offline
- Rebuild the disks completely
- Bring the node back online and start Riak
What’s the downside to RAID?
A hardware RAID controller is a single point of failure, and RAID (either software or hardware) adds a layer of complexity to configuration and troubleshooting. Drives are not necessarily portable to other RAID controllers (even similar controllers with different firmware revision numbers).
Even if a node does not fail completely, a degraded RAID array can significantly impact node performance and have an impact on the cluster. For example, if a node is configured with
RAID 5 and a disk is damaged, then writes on that node will likely suffer.
Flavors of RAID
Hardware RAID has a few benefits over software RAID. A major one is that hardware RAID allows you to open only a single I/O stream to write to multiple disks, leading to potentially less I/O wait and fewer blocking threads. With software RAID, an I/O stream has to be opened to each disk in the RAID set, causing multiple blocking threads.
Some other important considerations when selecting a hardware RAID device:
- Do you need a battery-backed cache? This aids in avoiding data loss during power
interruptions. Flash-based caches are also an option.
- How much non-volatile random-access memory (NVRAM) do you need?
- What processor speed do you need? This is important for replica rebuilds and parity calculation.
- Do you care for out-of-band management via something like a web interface?
All of these decisions impact cost. You can get away with a cheap RAID device and still get decent performance for
RAID 1/0/10, but once you throw parity into the mix, battery backups and caches have a noticeable impact.
On Linux, the most popular software RAID solution is mdadm. It can create, assemble, report on, and monitor arrays.
# RAID 0
$ mdadm --create /dev/md0 --level=0 --raid-devices=2 /dev/sdb /dev/sdc
# RAID 1
$ mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
# RAID 10
$ mdadm --create /dev/md0 --level=10 --raid-devices=4 /dev/sdb /dev/sdc /dev/sdd /dev/sde
# RAID 5
$ mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb /dev/sdc /dev/sdd
In a failure scenario, assuming a RAID level greater than zero, replacing a drive can be as simple as:
$ mdadm /dev/md0 --fail /dev/sdb
# Swap out the failed drive
$ mdadm --add /dev/md0 /dev/sdb
On Solaris or FreeBSD, the most common solution is ZFS.
# Striped zpool
$ zpool create tank /dev/dsk/c0t4d0 /dev/dsk/c0t5d0
# Mirrored zpool
$ zpool create tank mirror /dev/dsk/c0t4d0 /dev/dsk/c0t5d0
# Striped and mirrored zpool
$ zpool create tank mirror /dev/dsk/c0t4d0 /dev/dsk/c0t5d0
$ zpool add tank mirror /dev/dsk/c0t6d0 /dev/dsk/c0t7d0
$ zpool create tank raidz /dev/dsk/c0t4d0 /dev/dsk/c0t5d0 /dev/dsk/c0t6d0
Replacing a drive in a
$ zpool replace tank /dev/dsk/c0t5d0
Or, if you are replacing a device in a
zpool with a disk in a different physical location:
$ zpool replace tank /dev/dsk/c0t5d0 /dev/dsk/c0t6d0
There’s no universally correct answer to the question of RAID or no RAID. The needs of your application should drive the decision of whether RAID will be useful, and if so, at what level.
Write heavy, high performance applications should probably use
RAID 0 or avoid RAID altogether and consider using a larger
n_val and cluster size. Read heavy applications have more options, and generally demand more fault tolerance with the added benefit of easier hardware replacement procedures.
Either way, Riak provides redundancy by generating replicas of data and automatically spinning up fallback vnodes. In addition, having at least 5 nodes in a cluster decreases the likelihood of any single disk/node failure causing a service degradation or disruption.