Tag Archives: Riak

Getting Started with Riak in the Cloud

April 10, 2013

Earlier this year, we announced that hosted Riak is now available on the Engine Yard platform. Ines Sombra, Lead Data Engineer at Engine Yard, has put together a talk to help you get started using Riak in cloud environments. This talk introduces Riak’s overall architecture, some common use cases, and goes over some questions to consider when choosing a database. It also discusses what you need to know to run Riak in the cloud and how it differs from traditional hardware installation.

You can view the full talk below:

For more information about Riak on Engine Yard, check out our blog post or get started now with 500 hours free on their platform.

Basho

Syslog and Riak 1.3

April 8, 2013

Logs are an important part of understanding the health of any system. They provide historic records of events with information that may not be seen when running ad-hoc commands like riak-admin or reviewing the data gathered and trended from riak-admin status or http://$riakNode/stats.

When used correctly, logs can forewarn of an issue as well as provide forensic evidence to use after the event.

By default, Riak’s logging is split sensibly into different files for different types of log data: errors, crash and console. Take a look in /var/log/riak or /var/log/riak-cs to explore further.

Since you may have five to 50 nodes in your Riak cluster (we recommend that you have at least five nodes), reviewing those logs by hand would be a timely and repetitive task. This makes it unlikely that you’d catch any events as they’re happening.

Gathering logs centrally for analysis is the best way to resolve this and make this easier. Riak has Syslog support through its logging framework – Lager.

Lager

Lager is another great open source project from Basho. We recommend checking out the project page for full details, but a summary of its features are:

  • Custom formatting
  • Runtime changes
  • Internal log rotation (yes, sysadmin, we already have you covered)
  • AMQP support
  • SMTP support
  • Syslog Support
  • Loggly Support
  • Tracing

Configuring Lager to use Syslog is as easy as opening your /etc/riak/app.config then navigating down to Lager’s configuration section. Here, you can replace the existing handlers section with

{handlers, [
{lager_syslog_backend, ["riak", local1, info]},
] },

Then, in your local syslog.conf you may add the following: (Please remember Syslog likes tabs between sections, not spaces. If you copy and paste, you will need to edit the line accordingly.)

local1.info             /var/log/riak_info.log

Both Riak and Syslog will require restarting to apply the changes.

You can add multiple handlers for the various levels you wish to configure. You can also use a hybrid approach, leaving Lager managing some levels while Syslog manages others. Check out the Lager Syslog project page for more information.

The next step in your logging solution would be to redirect the logs to your central Syslog server. Doing so is outside the scope of this post, but there are many great guides available that describe the process.

Analysis Tools

There are some excellent tools out there for log file anaylsis. These include: Splunk and Logstash, which can both be run locally, and Loggly and Papertrail, which are Software as a Service.

What am I seeing?

When using one of these tools, we recommend that you set up alerts for common, frequent, and important messages. Check our online documentation for descriptions of common messages to get started.

Further Help

If you need further assistance, please don’t hesitate to contact the team and community:

IRC – irc.freenode.net #riak

Email – http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Or, if you’re an enterprise customer, contact us through your support account.

Team Client Services

Boundary Uses Riak to Handle Massive Data Scale

April 4, 2013

Boundary provides innovative application monitoring for new IT architectures: one-second app visualization, cloud-compatible, and only a few minutes from setup to results. They use Riak to store all of their historic data.

Due to the nature of Boundary’s service, they take in a massive amount of data at a high resolution and volume. After experimenting with a number of different options, Riak was selected for historic data due to its scalability, simplicity, high availability, and transparency. According to Boundary Founder, Cliff Moon, “Riak’s transparency and simplicity gives us peace of mind while it’s in production. We know what it’s doing and never have to worry about it.”

Altogether, they handle nearly 3TB of writes per day, store 200-300 million keys, and have three trillion separate observations stored in their cluster at any given point.

To learn more about how Boundary uses Riak, check out the complete case study.

Basho

Slides: Introduction to Riak

March 27, 2013

Yesterday, we presented an introductory webcast for those new to Riak. We’ve posted the slides below. They cover the key design goals of Riak and its architecture, operations, interfaces, and querying features. They also include new features from the recent 1.3 release and a few interesting user stories of Riak in production.

Our next webcast will be Tuesday, April 2 and will focus on our newly open sourced, cloud storage product, Riak CS. You can sign up for it here.

Basho

Spotlight: Riak for Retail, Advertising, Mobile, and Gaming

March 25, 2013

We have a few new pages on our site to help you start building retail, advertising, mobile, or gaming apps and services on Riak. On each of these pages, you will find industry use cases for Riak, relevant case studies from companies already using it in production, and information on Riak’s key/value model and querying features. Here’s a quick overview:

Retail

Riak lets retailers provide always-on, highly scalable eCommerce and mobile services. Retail use cases for Riak include:

  • Shopping carts
  • Product catalogs
  • Mobile applications/sites

The retail page also looks at how Best Buy, ideeli, and Copious use Riak. For more details, visit the Retail Users Page.

Advertising

Advertisers need to serve data reliably and quickly. Common ways advertisers leverage Riak include:

  • Serving ad content
  • Session storage
  • Mobile experiences

OpenX and Velti are two examples of advertisers using Riak. For more details, visit the Advertising Users Page.

Mobile

Riak provides the scalable, small object storage required by mobile applications and platforms. Riak can be used for many different mobile needs, including:

  • User and session data storage
  • Text and multimedia storage
  • Social authentication

The mobile page also highlights case studies from popular applications Bump and Voxer. For more details, visit the Mobile Users Page.

Gaming

Gaming platforms and applications benefit from Riak’s scale-out, low-latency design, and flexible storage capabilities. Some common use cases include:

  • Player data
  • Session data
  • Social information
  • Gaming assets

This page looks at some gaming companies using Riak, including Kiip, Mochi Media, and Hibernum, and why they chose Riak. For more details, visit the Gaming Users Page.

You can find user stories from other industries here. And make sure to sign up for our Introduction to Riak webcast, tomorrow at 11:00 am PT / 2:00 pm ET.

Basho

Riak at Temetra

March 21, 2013

Temetra is the most widely used meter management system for water in Ireland. They provide utility companies with a cloud service to store and manage meter data for utilities such as water, gas and electricity.

Each time one of Temetra’s utility customers reads a meter, that reading is stored by Temetra and made available to the utility. As Temetra found success at home and overseas, it resulted in huge amounts and variety of data: from the meter readings themselves, through meter location information and utility staff work schedules, to audit logs.

With the switch to smart meters, where one meter can now provide 35,000 readings per year instead of just four, Temetra needed to improve the scalability and availability of their platform. They chose Riak as their primary data store and use it to collect and process all meter data.

Paul Barry, Temetra’s CEO, told the London Riak meet-up why they chose Riak and how they went from their initial evaluation to deploying it in production. Check out his interesting talk here.

Basho

Riak Community Survey

We’d sincerely appreciate it if you could take some time to complete our first quarterly Riak community survey. The goal of this survey is to help us better understand how you are using Riak. Our hope is that your insights will help us make better decisions moving forward with the future of Riak.

All contributors to the survey will be provided Basho swag, as well as a discount for RICON tickets. One lucky contributor will be pseudorandomly selected to receive a free ticket to RICON | East!

If you have any questions or concerns, please don’t hesitate to hit up te@basho.com.

Light up a Riak Cluster with AWS, A Few Notes…

This is a cross post from compositecode.com written by Adron Hall, one of the Basho Technical Evangelists. In it he walks through one of the methods of setting up and configuring a cluster on AWS. Other options are enumerated in a post entitled Riak on AWS – Deployment Options


March 14, 2013

I wanted to write up an intro to getting Riak installed on AWS, even though the steps are absurdly simple and already available on the Basho Docs site, there are a few extra notes that can be very helpful for a few specific points during the process.

Start off by logging into AWS. At this point you can take two different paths that are almost identical. You can follow the path of using the pre-built AWS Marketplace image of Riak, or just start form scratch. The difference is a total of about 2 steps: installing & setting some security port connections. I’m going to step through without using the prebuilt image in these instructions.

Security Group

First thing you’ll need to get a security group with the correct permissions setup. For that, you’ll need to make a security group.

NOTE: No, I didn’t mean to misspell Riak, but it’s in there now. ;)

Security Group

Before adding the ports, go to the security group details tab and copy the security group id. I’ve pointed it out in the image above.

Create Security Group

Now add the following three and assign the security group to the ports; 4369, 8099 & 6000-7999. For the source set it to the security group id. Once you get all three added the list should look like this (below). For each rule click the Add Rule button and remember to click the Apply Rule Changes. I often forget this because the screen on some of the machines I use only shows to the bottom of the Add Rule button, so you’ll have to scroll down to find the Apply Rule Changes button.

Apply Rule Changes

Now add the standard port 22 for SSH. Next get the final two of 8087 and 8098 setup and we’re ready for moving on to creating the virtual machines.

Server Virtual Machines

For creating virtual machines I just clicked on Launch Instance and used the classic wizard. From there you get a selection of items. I’ve used the AWS image to do this, but would actually suggest using a CentOS image of your choice or Red Hat Enterprise Linux (RHEL). Another great option is to use the Ubuntu 12.04 LTS. Really though, use whatever Linux version or distro you like, there are 1-2 step instructions for installing Riak on almost every distro out there.

Virtual Machines

Next just launch a single instance. We’ll be able to launch duplicates of these further along in the process. I’ve selected a “Micro” here but I’m not intending to do anything with a remotely heavy load right now. At some point, I’ll upgrade this cluster to larger instances when I start putting it under a real load. I’ll have another blog entry to describe exactly how I do this too.

Request Instances

Continue again until you can select the security group that we created above.

SecurityGroupRequestImages

Now keep hitting that continue button, until you get to launch, and launch this thing. Once the instance is launched launch your preferred SSH connection tooling. The easiest way I’ve found for getting the most current private IP to connect to with the appropriate command is to right click on the instance in the AWS Console and click on Connect. There you’ll find the command to connect via SSH.

Connect to Instance

Paste that in and hit enter in your SSH App, you’ll see something akin to this.
$ cd Codez/working-content/
$ ssh -i riaktionz.pem root@ec2-54-245-201-97.us-west-2.compute.amazonaws.com
The authenticity of host 'ec2-54-245-201-97.us-west-2.compute.amazonaws.com (54.245.201.97)' can't be established.
RSA key fingerprint is 31:18:ac:1a:ac:fc:6e:6d:55:e8:8a:83:9a:8f:c7:5f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-54-245-201-97.us-west-2.compute.amazonaws.com,54.245.201.97' (RSA) to the list of known hosts.
Please login as the user "ubuntu" rather than the user "root".

Enter yes to continue connecting. For some instance types, like Ubuntu you’ll have to do some teaks to log into as “ubuntu” vs. “root” and the same goes for the AWS image or others. I’ll leave that to you, dear reader to get connected via ole’ SSH.

One of the other things, that you may have to do some tweaking about and googling, is figuring out the firewall setups on the various virtual machine images. For the RHEL you’ll want to turn off the firewall or open up the specific connection ports and such. Since the AWS firewall does this, it isn’t particularly important for the OS to continue running its firewall service. In this case, I’ve turned off the OS firewall and just rely on the AWS firewall. To turn off the RHEL firewall, execute the following commands.

$ service iptables save
$ service iptables stop
$ chkconfig iptables off

Now is a perfect time to start those other instances. Navigate into the AWS Console again and right click on the virtual machine instance you’ve created. On that menu select Launch More Like This.

Launch More

Go through and check the configuration on each of these, make sure the firewall is turned off, etc. Then move on to the next step and install Riak and cluster them. So it’s time to get to the distributed, massively complex, extensive list of steps to install & cluster Riak. Ok, so that’s sarcasm. ;)

Step 1: Install Riak

Install Riak on each of the instances.

package=basho-release-6-1.noarch.rpm &&
wget http://yum.basho.com/gpg/$package -O /tmp/$package &&
sudo rpm -ivh /tmp/$package
sudo yum install riak

NOTE: For other installation methods, such as directly downloading the RPM or other Linux OSes, check out the http://docs.basho.com/riak/latest/tutorials/installation/Installing-on-RHEL-and-CentOS/.

Step 2: Setup the Cluster

On the first instance, get the IP. You won’t need to do anything to this instance, just keep the IP handy. Then move on to the second instance and run the cluster command.

sudo riak-admin cluster join riak@

Do this on each of the instances you’ve added, using that first node. When you’ve added them all, on that last instance (or really any of them) then run the plan. This will get you a display plan of what will take place when the cluster is committed.

sudo riak-admin cluster plan

If that looks all cool. Commit the plan.

sudo riak-admin cluster commit

Get a check of the cluster.

sudo riak-admin member_status

That’s it; all done. You now have a Riak Cluster. For more operations to try out on your cluster, check out this list of basic API Operations.

Building Gaming Applications and Services with Riak

March 13, 2013

For a complete overview, download the whitepaper, “Gaming on Riak: A Technical Introduction.” To see how other gaming companies are using Riak, visit us at the Game Developers Conference at Booth #202!

As discussed in our previous post, “Gaming on Riak: A Brief Overview and User Case Studies,” Riak can provide a number of advantages for gaming platforms. Content agnostic, an HTTP API, many client libraries, and a simple key/value data model, Riak is a flexible data store that can be used for a variety of different use cases in the gaming industry. This post looks at some common examples and how to start building them in Riak.

Use Cases

Player and Session Data: Riak can serve and store key player and session data with predictable low latency, and ensures it is available even in the event of node failure and network partition. This data may include user and profile information, game performance, statistics and rankings, and more. In Riak, all objects are stored on disk as binaries, providing flexible storage for many content types. Since Riak is schema-less, applications can evolve without changing an underlying schema, providing agility with growth and change.

Social Information: Riak can be used for social content such as social graph information, player profiles and relationships, social authentication accounts, and other types of social gaming data.

Content: Riak is often used to store text, documents, images, videos and other assets that power gaming experiences. This data often needs to be highly available and able to scale quickly to attract and keep users.

Global Data Locality: Gaming requires a low-latency experience, no matter where the players are located. Riak Enterprise’s multi-datacenter replication feature means data can be served to global users quickly.

Data Model

Below are some common approaches to structuring gaming data with Riak’s key/value model:

Riak offers robust additional functionality on top of the fundamental key/value model. For more information on these options as well as how to implement them, their architecture, and their limitations, check out the documentation on searching and accessing data in Riak.

Riak Search
Riak Search is a distributed, full-text search engine. It provides support for various MIME types & analyzers, and robust querying including exact matches, wildcards, range queries, proximity searches, and more.
Possible Use Cases: Searching player and game information.

Secondary Indexing
Secondary Indexing (2i) gives developers the ability, at write time, to tag an object stored in Riak with one or more queryable values. Indexes can be either integers or strings and can be queried by either exact matches or ranges of an index.
Possible Use Cases: Tagging player information with user relationships, general attributes, or other metadata.

MapReduce
Developers can leverage MapReduce for analytic and aggregation tasks. It offers support for both JavaScript and Erlang MapReduce.
Possible Use Cases: Filtering game data by tag, counting items, and extracting links to related data.

To learn more about how your gaming platform can benefit from Riak, download “Gaming on Riak: A Technical Introduction.” For more information about Riak, sign up for our webcast on Thursday, March 14.

Basho

Gaming on Riak: A Brief Overview and User Case Studies

March 12, 2013

Riak provides low-latency, highly available storage to power gaming platforms and applications. Gaming companies use Riak to store player and game data, session and social information, and a variety of gaming content and events. This post offers a quick look at the advantages of Riak and some user case studies. Later this week, we’ll publish an in-depth look at the common gaming use cases and examples of data modeling.

For a complete overview, download the whitepaper, “Gaming on Riak: A Technical Introduction.”

Advantages of Riak

  • Support for Rapid Growth: Built for operational ease-of-use, Riak yields a near-linear performance and throughput increase as capacity is added.
  • Low-Latency Design: Riak is designed to store data and serve requests predictably and quickly, even during peak times.
  • Flexible, Reliable Storage: Riak has a flexible data model with redundancy built-in, and a number of mechanisms to maintain availability even in the event of node failure or network partition. Riak is content-agnostic, providing flexibility for document, image, video, and other storage.
  • Multi-Datacenter Replication: Riak Enterprise’s multi-datacenter replication provides disaster recovery and data locality.

Case Studies

Hibernum is a creator and developer of unique gaming experiences that combine the latest in social gaming, top quality visuals and animations, and cutting edge design. They switched from a relational database to Riak due to its high availability, ability to scale to peak loads, and predictable operational cost. Riak is used to store user game information for one of their most popular social games. For more information about how Hibernum uses Riak, check out the complete case study.

Kiip is a platform that lets brands provide rewards to mobile gamers for in-game achievements. Kiip replaced MongoDB with Riak in order to achieve low read/write latencies and horizontal scalability. Kiip uses Riak for session and device data. To learn more about Kiip’s experience selecting Riak, check out this video by two of their engineers.

To learn more about how your gaming platform can benefit from Riak, download “Gaming on Riak: A Technical Introduction.” For more information about Riak, sign up for out webcast on Thursday, March 14.

Basho