Tag Archives: Riak Control

Riak 1.4: Riak Control

July 16, 2013

Riak Control is a web-based administrative console for inspecting and manipulating Riak clusters.

Although Riak Control is maintained as a separate application, the necessary code for Control ships with Riak 1.1 and above and requires no additional installation steps. For details on setting up Riak Control, check out our docs.

Those are things you may already know about Control. Now let’s look at the changes in 1.4.

Cluster Management with Staging

The riak-admin command-line tool has offered staged clustering since Riak 1.2. Riak 1.4 brings that functionality to Control.

The new Cluster Management interface allows you to stage cluster node additions and removals. Once the changes have been reviewed, they can be committed to the cluster. After being committed, Control displays partition transfers and memory utilization changes as they occur.

Staged changes to the cluster:

Staged Changes

Changes committed; transfers active:

Changes Committed

Cluster stabilizes after changes:

Cluster Stabilized

Standalone Node Management Interface

Because the Cluster Management interface now operates on staged changes, actions that cannot be staged have been moved to the Node Management interface. Here, changes to individual nodes, such as stopping or marking them as down, can be applied.

Node Management

Contributing to Riak Control

Riak Control’s user interface is built using Ember.js, and for persistence, Ember Data. The backend is written in Erlang using Webmachine.

Riak Control’s modular design prevents users from having to understand every detail of its existing functionality to contribute. If you’re interested in contributing, we have outlined the process of setting up a development environment, as well as some basic rules for contribution.

Check out our Riak 1.4 announcement to learn what else is included in this release.

Hector Castro

Basho Announces Availability of Riak 1.4

July 10, 2013

We are excited to announce the launch of Riak 1.4. With this release, we have added in more functionality and addressed some common requests that we hear from customers. In addition, there are a few features available in technical preview that you can begin testing and will be fully rolled out in the 2.0 launch later this year.

The new features and updates in Riak 1.4 include:

  • Secondary Indexing Improvements: Query results are now sorted and paginated, offering developers much richer semantics
  • Introducing Counters in Riak: Counters, Riak’s first distributed data type, provide automatic conflict resolution after a network partition
  • Simplified Cluster Management With Riak Control: New capabilities in Riak’s GUI-based administration tool improve the cluster management page for preparing and applying changes to the cluster
  • Reduced Object Storage Overhead: Values and associated metadata are stored and transmitted using a more compact format, reducing disk and network overhead
  • Handoff Progress Reporting: Makes operating the cluster, identifying and troubleshooting issues, and monitoring the cluster simpler
  • Improved Backpressure: Riak responds with an overload message if a vnode has too many messages in queue

This 1.4 launch also adds quite a few performance enhancements to Riak Enterprise’s multi-datacenter replication that include:

  • Replication in Riak 1.4 supports SSL, NAT, and full sync scheduling
  • Availability of cascading real-time writes gives operators the choice as to whether or not all writes are replicated to all datacenters
  • Optional use of Active Anti-Entropy during replication in Riak 1.4 to significantly decrease data transfer times is available in Technical Preview

These updates improve the performance of Riak and provide greater functionality and management for both clusters and multiple datacenters. You can download Riak 1.4 at docs.basho.com/riak/latest/downloads.

For a full list of what’s in Riak 1.4, check out our code at Github.com/basho or review the release notes. To learn even more, join our live webcast, “What’s New in Riak 1.4” on July 12th and look for a series of more detailed blog posts over the coming weeks.

We will also be launching Riak CS 1.4 shortly. Keep an eye on our blog for more information.

Basho

Introducing Riak 1.3: Active Anti-Entropy, a New Look for Riak Control, and Faster Multi-Datacenter Replication

February 21, 2013

Today we are excited to announce the latest version of Riak. Here is a summary of the major enhancements delivered in Riak 1.3:

  • Introduced Active Anti-Entropy. Riak now has active anti-entropy. In distributed systems, inconsistencies can arise between replicas due to failure modes, concurrent updates, and physical data loss or corruption. Pre-1.3 Riak already had several features for repairing this “entropy”, but they all required some form of user intervention. Riak 1.3 introduces automatic, self-healing properties that repair entropy on an ongoing basis.
  • Improved Riak Enterprise’s multi-datacenter replication performance. New advanced mode for multi-datacenter replication capabilities, with better performance, more TCP connections and easier configuration. Read more in this write up from GigaOM.
  • Improved graphical user experience. Riak Control, the user interface for managing and monitoring Riak, has a brand new look.
  • Expanded IPv6 support. IPv6 support in Riak now is supported by all interfaces.
  • Improved MapReduce. Riak MapReduce has improved back-pressure to reduce the risk of overwhelming endpoint processes during large tasks.
  • Simplified log management. Riak can now optionally send log messages to syslog.

Ready to get started or upgrade? Download the new release here, check out the official release notes, or read on for more details. Documentation for all products and releases is available on the documentation site. For an introduction to Riak and what’s new in Riak 1.3, sign up for our webcast on Thursday, March 7.

More on What’s in Riak 1.3

Active Anti-Entropy
A key feature of Riak is its ability to regenerate lost or corrupted data from replicated data stored on other nodes. Prior to this release, Riak provided two methods to repair data:

  • Read Repair: Riak compares the replies from all replicas during a read request, repairing any replica that is divergent or missing data. (K/V data only)
  • Repair Command via Riak Console: Introduced in Riak 1.2, the repair command enables users to trigger a repair of a specific partition. The partition is rebuilt based on a subset of data stored on adjacent nodes in the Riak ring. All data is rebuilt, not just missing or divergent data. (K/V and Search data)

Riak 1.3 introduces active anti-entropy, a continuous background process that compares and repairs any divergent, missing, or corrupted replicas (K/V data only). Unlike read repair, which is only triggered when data is read, the active anti-entropy system ensures the integrity of all data stored in Riak. This is particularly useful in clusters containing “cold data”: data that may not be read for long periods of time, potentially years. Furthermore, unlike the repair command, active anti-entropy is an automatic process, requiring no user intervention and is enabled by default in Riak 1.3.

Riak’s active anti-entropy feature is based on hash tree exchange, which enables differences between replicas to be determined with minimal exchange of information. Specifically, the amount of information exchanged in the process is proportional to the differences between two replicas, not the amount of data that they contain. Approximately the same amount of information is exchanged when there are 10 differing keys out of 1 million keys as when there are 10 differing keys out of 10 billion keys. This enables Riak to provide continuous data protection regardless of cluster size.

Additionally, Riak uses persistent, on-disk hash trees rather than purely in-memory trees, a key difference from similar implementations in other products. This allows Riak to maintain anti-entropy information for billions of keys with minimal additional memory usage, as well as allows Riak nodes to be restarted without losing any anti-entropy information. Furthermore, Riak maintains the hash trees in real time, updating the tree as new write requests come in. This reduces the time it takes Riak to detect and repair missing/divergent replicas. For added protection, Riak periodically (default: once a week) clears and regenerates all hash trees from the on-disk K/V data. This enables Riak to detect silent data corruption to the on-disk data arising from bad disks, faulty hardware components, etc.

New Look for Riak Control
Riak Control is a UI for managing and monitoring your Riak cluster. Riak Control lets you start and re-start Riak nodes, view a “health check” for your cluster, see all nodes and their current status, and have visibility into their partitions and services. Riak Control now has a brand new look and feel. Check out the Riak Control Github page to get up and running.

Expanded IPv6 Support
While Riak’s HTTP interface has always supported IPv6, not all of its interfaces have been as current. In Riak 1.3, the protocol buffers interfaces can now listen on IPv6 or IPv4 addresses. Riak handoff (which is responsible for data transfer when nodes are added or removed, and for handing off update responsibilities when nodes fail) also supports IPv6. It should also be noted that community member Tom Lanyon started the work on this feature. Thanks, Tom!

Improved Backpressure in Riak MapReduce
Riak has Javascript and Erlang MapReduce for performing aggregation and analytics tasks. Backpressure is an important aspect of the MapReduce system, keeping processes from being overwhelmed or memory consumption getting out of control. In Riak 1.3, tunable backpressure is extended to the MapReduce sink to prevent these types of problems at endpoint processes.

Riak Enterprise: Advanced Multi-Datacenter Replication Capabilities
With hundreds of companies using Riak Enterprise, a commercial extension of Riak, we’ve been lucky to work with many teams pushing the limits of multi-datacenter replication performance and resiliency. We’ve learned a lot and are excited to announce these capabilities are now available in advanced mode.

  • Previously, multi-datacenter replication had one TCP connection over which data was streamed from one cluster to another. This could create a performance bottleneck, especially when run on nodes constrained by per-instance bandwidth limits, such as in a cloud environment. In the new version of multi-datacenter replication, multiple concurrent TCP connections (approximately one per physical node) and processes are used between sites.
  • Configuration of multi-datacenter replication is easier. Use a shell command to name your clusters, then connect both clusters using a simple ip:port combination.
  • Better per-connection statistics for both full-sync and real-time modes.
  • New ability to tweak full-sync workers per node and per cluster, allowing customers to dial-in performance.

The new replication improvements are already used in production by customers and yielding significant performance improvements. For now, the new replication technology is available in advanced mode: it’s optional to turn on. It currently doesn’t have all of the features of the default mode – including SSL, NAT support and full-sync scheduling. Both default and advanced modes are available in the 1.3 release and function independently. In the future, “advanced mode” will become the default.

For more details about multi-datacenter replication, download our whitepaper, “Multi-Datacenter Replication: A Technical Overview.”

Basho

Riak Control

February 22, 2012

Riak Control is Basho’s new OSS, REST-driven, user-interface for Riak. The code has been available for a few months now, but it’s officially supported in Riak 1.1, so we wanted share some details on what it’s about and why you should be excited about it.

Lowering the Barrier for Entry

Once a Riak cluster is up and running we want it to be as hands-free to administer as possible. Things should “just work,” like plumbing. But we’ll be the first to admit that a new user’s initial welcoming with Riak isn’t always as pleasant as it should be.

Some steps are unavoidable: downloading, installing, and/or building from source, etc. But once the initial work is done, the experience should be as inviting as possible. Riak is a very powerful database with numerous options and commands. Riak Control allows you to easily manage/inspect your cluster while ignoring many of these until needed.

Empowering Riak Administrators

When we first sat down to decide what the more important features for any Riak interface should be, one theme stood out above all the others: cluster management. We wanted to give developers and administrators the ability to quickly build a cluster, inspect nodes, and diagnose the health of their cluster. And we wanted it to happen fast.

Riak is about large datasets and clusters replicating that dataset for maximum availability and persistence. We’re working hard to help companies that write many, many GBs per day to clusters containing 50+ nodes. Riak Control is a tool that brings issues and risks front-and-center. And it gives customers the ability to take action in real-time.

The Two-Minute Tour

Riak Control is currently broken up into nested levels of detail. Each page in Riak Control is designed to give you just as much information as you need, nothing more. As you navigate the UI, you’ll gradually be taken deeper into the rabbit hole.

The Overview/Snapshot

Snapshot

The Snapshot is what you’ll see when you first fire up Riak Control. It should give you a warm-fuzzy feeling when everything is A-okay: an unmistakable, beautiful green check mark.

For times when things aren’t perfect, you will be presented with a list of concern areas. Each will have links to other pages of Riak Control where you can take a closer look at the problem.

The Cluster

Custer

The cluster page is where you can get a quick look at all of the nodes in your cluster and manage membership.

With a glance you can see which nodes are partitioned from the rest of the cluster or offline, which are leaving or joining the cluster, view partition ownership, monitor memory, and more. And with a click you can add nodes to the cluster, take nodes offline for maintenance, and leave the cluster.

The Ring

Ring

One level deeper than the cluster view is the ring page. This is where you can see the health of each partition. Most of the time, your ring will be too large to really manage from the ring view. But with the filters you can immediately find which partitions are owned by which nodes, partitions whose primary nodes are unreachable, current handoffs, and more.

What’s Next?

Riak Control is not standing still. Riak 1.1 includes Riak Control in its early stages so we can begin to gather feedback. We want to know what it does right and what it does wrong. Your feedback and ideas are encouraged. Additionally, we have a list of features and functionality slated for future releases. None of these are set in stone, but here is a list of what we have planned…

Pluggable

While Riak Control is – at its heart – a simple REST API, we’re working to modularize it in a way that allows you to write your own modules/plugins. We want to see Riak Control become a collection of pieces that all snap and work together, empowering you to manage your cluster in the way that best fits your needs.

Event streaming

Currently Riak Control uses a pull model to gather information about the cluster. While this isn’t a performance issue, we very much want to make it a push-system. As things happen to the cluster, the cluster should notify Riak Control of the changes, which in-turn will notify the user.

Node Statistics

Clicking on a node name from anywhere should take you to a page giving details specifically about that node, similar to the data you would get from a riak-admin status command.

Bucket & Object Inspection

While low-level object manipulation isn’t designed to be a primary feature of Riak Control, it is a very handy tool to have, and extremely valuable when initially setting up Riak for the first time. More importantly, Riak buckets will be available to create and inspect.

MapReduce Queries

Riak Control will feature a powerful interface for creating MapReduce queries. You will be able to debug, save, load, and execute previously saved queries with ease.

Customer Support Tools

In addition to the general tools provided for manipulation of the cluster and data, we also are planning for improve monitoring tools.

  1. View the log files of individual nodes
  2. See graphs of load, memory latency, disk usage, etc.
  3. Coalesce and bundle data for support tickets
  4. File support tickets

Any Comments, Questions, or Feature Requests?

Anything you’d like to share or ask? Join the Riak-Users Mailing List and tell us what you think. The other option is to fork the code and make your opinions known with a pull request or by filing an issue. You can also find some formal documentation on the Riak Wiki.

Thanks for being a part of Riak.

Jeff Massung