Tag Archives: data modeling

Riak for Gaming

April 16, 2014

The world of gaming can be unpredictable. It can be hard to judge if a game is going to be the next Angry Birds and experience exponential, global growth. Riak is designed to help gaming platforms handle this uncertainty with ease. Its focus on high availability means that all data remains accessibility, even during node failure. Its flexible data model and redundant, fault-tolerant design easily allows gaming platforms to store any type of data needed. Riak is also built for operational simplicity at scale, so Riak will seamlessly grow with data and popularity. Finally, the option for multi-datacenter replication means that gamers all over the world will get the same low-latency experience across multiple devices.

Top Use Cases for Riak in Gaming

  • Player Data: Riak provides low-latency, highly available data storage for key player data, including user and profile information, game performance, statistics and rankings, and more. Riak also provides many different tools for querying and indexing this data, such as Riak Search, Secondary Indexing, and MapReduce.
  • Session Storage: Riak is frequently used to store and serve session data with predictable low-latency – necessary for game play. Riak imposes no restrictions on the type of content stored (since all objects are stored on disk as binaries), so session data can be encoded in many ways and can evolve without administrative changes to schemas.
  • Social Information: Riak provides flexible, robust storage for social data such as social graph information, player profiles and relationships, and social authentication tokens.
  • Global Data Locality: When gaming, players require a low-latency experience, regardless of where they’re physically located. Otherwise, interrupted or slow game play can lead to poor user experience and possible user abandonment. Riak Enterprise’s multi-datacenter capabilities allow game data to be physically close to players and serve them data no matter where they happen to be.

Riak in Production

Riak is already in production by many top gaming platforms. Here’s a look at a few that have switched to Riak.

Rovio
Rovio is the creator of the popular mobile game, Angry Birds. Since user growth can be hard to predict, they needed an infrastructure that could support unexpected viral growth without failing or causing downtime. They selected Riak due to its ease-of-scale and fault tolerance. Riak now powers their new cartoon series, Angry Birds Toons, and new mobile games. Learn more about why they moved to Riak in this case study and video from GDC.

Hibernum
Hibernum is a creator and developer of unique gaming experiences that combine the latest in social gaming, top quality visuals and animations, and cutting edge design. They switched from a relational database to Riak due to the high availability, ability to scale to peak loads, and predictable operational cost. Riak is used to store user game information for one of their most popular social games. Check out the complete case study, Hibernum Selects Riak for User Data Storage.

Kiip
Kiip is a platform for building rewards and achievements into your games. Kiip replaced MongoDB with Riak in order to achieve low read/write latencies and horizontal scalability. Kiip uses Riak for storing and serving session and device data. Learn more from the video on scaling Riak to 25MM Ops/Day.

Riot Games
Riot Games is the creator of League of Legends and faced some challenges with supporting millions of concurrent players at any given moment. They switched to Riak from MySQL for their next generation stats system, which tracks gameplay statistics and stores terabytes of data that gets aggregated and presented to players in near real-time. More information on how they use Riak and why they selected it can be found here.

Data Modeling in Riak

Riak has a “schemaless” design. Objects are comprised of key/value pairs, which are stored in flat namespaces called buckets. Here are some common approaches to structuring gaming data with Riak’s key/value design:

Data Type Key Value
Player Data Login, Email, UUID Player Attributes (often stored as a JSON document); Player Rewards and Stats
Social Data Login, Email, UUID Player Profiles, Social Graph Information, Facebook/Twitter Tokens
Session Information User/Session ID Session Data
Image or Video Content Content Name, ID or Integer .JPG .PNG, .GIF or other image format; .MOV, .MPG, .MP4 or other video file format

To learn more about how gaming platforms can use Riak for their data needs, check out the complete overview, “Gaming on Riak: A Technical Introduction.” To get started with Riak, Contact Us or download it now.

Basho

Riak for Advertisers

April 14, 2014

Modern day advertisers are faced with many new challenges to ensure they can provide highly available, low latency experiences to thousands of clients and partners, and millions of users. They are also tasked with serving large amounts of data all over the world and can experience significant traffic spikes. That is why advertisers are switching to Riak for their database solution. Riak’s redundant, fault-tolerant design ensures that advertising companies can serve data reliably and quickly. Riak is also built for operational simplicity at scale and helps advertisers quickly grow to meet peak loads.

Top Use Cases for Riak in Advertising

  • Serving Ad Content: Riak’s rapid storage and content agnosticism makes it ideal for storing ad content and handling influxes of ad traffic.
  • Session Storage: This type of data is naturally a good fit for Riak’s key/value model. This data can also be encoded in many different ways and can evolve without any administrative changes to the schema.
  • Mobile Experiences: Riak is ideal for the low-latency, always-available small object storage needed to power mobile experiences across platforms.
  • Global Data Locality: Riak Enterprise’s multi-datacenter capabilities allow advertisers to maintain a global data footprint while providing an always-on, low-latency experience, anywhere in the world.

Riak in Production

Riak is already in production at many top advertising and marketing organizations. Here’s a look at a few that have switched to Riak.

Tapjoy
Tapjoy is a mobile advertising and monetization platform that is available on over one billion devices across the world. They selected Riak due to its high availability, low-latency, and multi-datacenter replication. They store 48TB of data in Riak and operate hundreds of thousands of reads/writes per second. Learn more about why Tapjoy selected Riak from the case study.

OpenX
OpenX is an ad technology platform that serves trillions of ads. They use Riak for user and trafficking data behind their data services API. OpenX also uses Riak’s multi-datacenter replication across several data centers. Watch Anthony Molinaro (Infrastructure Architect at OpenX) talk about how they use Riak for their serve-time data needs.

Velti
Velti is a mobile marketing and advertising technology provider. They use Riak for their interactive mobile platform, including letting people interact with their TV by voting, giving feedback, participating in contests, etc. Velti runs 18 nodes across two data centers, which provides them with scale, durability, and availability. Their case study goes into more detail about the process of moving to Riak.

JBA
JBA is a digital consultancy that specializes in developing customer understanding and behavioral targeting. They use Riak as a core part of their behavioral analysis and remarketing tool. They store over 10 million objects in Riak and can easily scale up to account for holiday sales cycles or new product releases as needed. Learn more about why they selected Riak from the beginning from their case study.

Moz
Moz provides analytics software to track all of a website’s inbound marketing efforts on one platform. They support over 27,000 customers and 300,000 community members worldwide. Moz uses Riak to store customer campaign search engine rankings data. Learn more about how Riak outperformed Cassandra at Moz in the case study.

Data Modeling in Riak

Riak has a “schemaless” design. Objects are comprised of key/value pairs, which are stored in flat namespaces called buckets. Here are some common approaches to structuring advertising data with Riak’s key/value design:

Data Type Key Value
Advertisement Campaign ID Ad Content
User Data Login, Email, UUID User Attributes (often stored as a JSON document)
Image or Video Content Content Name, ID or Integer .JPG, .PNG, .GIF or other image format; .MOV, .MPG, .MP4 or other video file format
Session Information User or Session ID Session Data

To learn more about how advertisers can use Riak for their data needs, check out the complete overview, “Advertisers on Riak: A Technical Introduction.” To get started with Riak, Contact Us or download it now.

Basho

Relational to Riak – Tradeoffs

November 18, 2013

This series of blog posts will discuss how Riak differs from traditional relational databases. For more information about any of the points discussed, download our technical overview, “From Relational to Riak.” The previous post in the series discussed High Availability and Cost of Scale.


Eventual Consistency

In order to provide high availability, which is a cornerstone of Riak’s value proposition, the database stores several copies of each key/value pair.

This availability requirement leads to a fundamental tradeoff: in order to continue to serve requests in the presence of failure, we do not force all data in the cluster to stay in sync. Riak will allow writes and reads no matter how many servers (and their stored replicas) are offline or otherwise unreachable.

(Incidentally, this lack of strong coordination has another consequence beyond high availability: Riak is a very, very fast database.)

Riak does provide both active and passive self-healing mechanisms to minimize the window of time during which two servers may have different versions of data.

The concept of eventual consistency may seem unfamiliar, but if you’ve ever implemented a cache or used DNS, those are common examples of the idea. In a large enough system, it’s effectively the default state of all data.

However, with the forthcoming release of Riak 2.0, operators will be able to designate selected pieces of data to require coordination and maintain strong consistency over high availability. Writing such data will be slower and subject to failure if too many servers are unreachable, but the overall robust architecture of Riak will still provide a fast, highly available solution.

Data Modeling

Riak stores data using a simple key/value model, which offers developers tremendous flexibility to define access models that suit their applications. It is also content-agnostic, so developers can store arbitrary data in any convenient format.

Instead of forcing application-specific data structures to be mapped into (and out of) a relational database, they can simply be serialized and dropped directly into Riak. For records that will be frequently updated, if some of the fields are immutable and some aren’t, we recommend keeping the immutable data in one key/value pair and the rest organized into a single or multiple objects based on update patterns.

Relational databases are ingrained habits for many of us, but moving beyond them can be liberating. Further information about data modeling, including sample configurations, are available on Use Cases section of the documentation.

Tradeoffs

One tradeoff with this simpler data model is that there is no SQL or SQL-like language with which to query the data.

To achieve optimal performance, it is advisable to take advantage of the flexibility of the key/value model to define simple retrieval patterns. In other words, determine the most useful queries and write the results of those queries as the data is being processed.

Because it is not always possible to know in advance what questions will need to be asked of your data, Riak offers added functionality on top of the key/value model. Tools such as Riak Search (a distributed, full-text search engine), Secondary Indexing (ability to tag objects with queryable metadata), and MapReduce (leveraged for aggregation tasks) are available to perform ad hoc queries as needed.

For many users, the tradeoffs of moving to Riak are worthwhile due to the overall benefits; however, it can be a bit of an adjustment. To see why others have chosen to switch to Riak from both relational systems and other NoSQL databases, check out our Users Page.

Basho

Popular Use Cases for Mobile Platforms on Riak

March 5, 2013

Mobile platforms need to provide always available, low-latency experiences that can scale to millions of users and support highly concurrent access. Riak’s redundant and fault-tolerant design ensures mobile data can be served quickly and reliably, and Riak is run in production by many popular mobile applications. For a full overview, check out the whitepaper “Mobile on Riak: A Technical Introduction.” Below are a few key mobile use cases and basic approaches to modeling them in Riak:

User Data: Storing user accounts, profile information, and events is a common use case for Riak. Mobile apps often store this data in JSON documents, using a UUID or other identifier as the key. Data can be queried through Riak features such as secondary indexes, MapReduce, and full-text search.

Session Data: Since session IDs are commonly stored in cookies, or otherwise known at lookup time, they are a natural fit for Riak’s key/value model and Riak can serve these requests at predictably low-latency. Session data can also be encoded in many different ways and evolve without any administrative changes to schema.

Text & Multimedia Storage: Since Riak is content agnostic, mobile platforms can easily store a variety of different types of data, including audio, text, photos, video, etc. to power mobile experiences.

Social Authentication: Many mobile applications have users sign in via their Facebook or Twitter accounts. Riak’s key/value scheme makes it easy to store both registered accounts and the tokens that make it possible for users to authenticate with their social accounts.

Global Data Locality: Riak Enterprise’s multi-datacenter capabilities mean mobile data can be stored in physical proximity to users and served at low-latency no matter where they happen to be.

Here is a chart with possible ways these applications and services can be modeled using Riak’s key/value design. Of course, your application should be structured in a way appropriate to its access and query patterns, among other factors – this is just to get you started. For more information on designing applications with Riak, check out our documentation.

To learn more about how mobile platforms can use Riak for their data needs, check out the complete overview, “Mobile on Riak: A Technical Introduction.” For more details about Riak and the latest 1.3 release, sign up for our webcast on March 7th.

Basho

Building Retail and eCommerce Services with Riak

January 31, 2013

This is the second in a series of blog posts covering Riak for retail and eCommerce platforms. To learn more, join our “Retail on Riak” webcast on Friday, February 8th or download the “Riak for Retail” whitepaper.

In our last post, we looked at three Riak users in the eCommerce/retail space. In this post, we will look at some common use cases for Riak and how to start building them with Riak’s key/value model and querying features.

Use Cases

  • Shopping Carts: Riak’s focus on availability makes it attractive to retailers offering shopping carts and other “buy now” functionality. If the shopping cart is unavailable, loses product additions, or responds slowly to users, it has a direct impact on revenue and user trust.
  • Product Catalogs: Retailers need to store anywhere from thousands to tens of thousands of inventory items and associated information – such as photos, descriptions, prices, and category information. Riak’s flexible, fast storage makes it a good fit for this type of data.
  • User Information: As mobile, web, and multi-channel shopping become more social and personalized, retailers have to manage increasing amounts of user information. Riak scales efficiently to meet increased data and traffic needs and ensures user data is always available for online shopping experiences.
  • Session Data: Riak provides a highly reliable platform for session storage. User/session IDs are usually stored in cookies or otherwise known at lookup time, a good fit for Riak’s key/value mode.

Data Modeling

In Riak, objects are comprised of key/value pairs, which are stored in flat namespaces called “buckets.” Riak is content-type agnostic, and stores all objects on disk as binaries, giving retailers lots of flexibility to store anything they want. Here are some common approaches to modeling the data and services discussed above in Riak:

Querying

Riak provides several features for querying data:

Riak Search: Riak Search is a distributed, full-text search engine. It provides support for various MIME types & analyzers, and robust querying.
Possible Use Cases: Searching product information or product descriptions.

Secondary Indexing: Secondary Indexing (2i) gives developers the ability, at write time, to tag an object stored in Riak with one or more values, queryable by exact matches or ranges of an index.
Possible Use Cases: Tagging products with categories, special promotion identifiers, date ranges, price or other metadata.

MapReduce: Riak offers MapReduce for analytic and aggregation tasks with support for JavaScript and Erlang.
Possible Use Cases: Filtering product information by tag, counting items, and extracting links to related products.

Check out our docs for more information on building applications and services with Riak.

For more details and examples of common Riak use cases, register for our “Retail on Riak” webcast on February 8th or download the “Riak for Retail” whitepaper.

Basho

Schema Design in Riak – Introduction

March 19, 2010

One of the challenges of switching from a relational database (Oracle, MySQL, etc.) to a “NoSQL” database like Riak is understanding how to represent your data within the database. This post is the beginning of a series of entries on how to structure your data within Riak in useful ways.

Choices have consequences

There are many reasons why you might choose Riak for your database, and I’m going to explain how a few of those reasons will affect the way your data is structured and manipulated.

One oft-cited reason for choosing Riak, and other alternative databases, is the need to manage huge amounts of data, collectively called “Big Data”. If you’re storing lots of data, you’re less likely to be doing online queries across large swaths of the data. You might be doing real-time aggregation in addition to calculating longer-term information in the background or offline. You might have one system collecting the data and another processing it. You might be storing loosely-structured information like log data or ad impressions. All of these use-cases call for low ceremony, high availability for writes, and little need for robust ways of finding data — perfect for a key/value-style scheme.

Another reason one might pick Riak is for flexibility in modeling your data. Riak will store any data you tell it to in a content-agnostic way — it does not enforce tables, columns, or referential integrity. This means you can store binary files right alongside more programmer-transparent formats like JSON or XML. Using Riak as a sort of “document database” (semi-structured, mostly de-normalized data) and “attachment storage” will have different needs than the key/value-style scheme — namely, the need for efficient online-queries, conflict resolution, increased internal semantics, and robust expressions of relationships.

The third reason for choosing Riak I want to discuss is related to CAP – in that Riak prefers A (Availability) over C (Consistency). In contrast to a traditional relational database system, in which transactional semantics ensure that a datum will always be in a consistent state, Riak chooses to accept writes even if the state of the object has been changed by another client (in the case of a race-condition), or if the cluster was partitioned and the state of the object diverges. These architecture choices bring to the fore something we should have been considering all along — how should our applications deal with inconsistency? Riak lets you choose whether to let the “last one win” or to resolve the conflict in your application by automated or human-assisted means.

More mindful domain modeling

What’s the moral of these three stories? When modeling your data in Riak, you need to understand better the shape of your data. You can no longer rely on normalization, foreign key constraints, secondary indexes and transactions to make decisions for you.

Questions you might ask yourself when designing your schema:

  • Will my access pattern be read-heavy, write-heavy, or balanced?
  • Which datasets churn the most? Which ones require more sophisticated conflict resolution?
  • How will I find this particular type of data? Which method is most efficient?
  • How independent/interrelated is this type of data with this other type of data? Do they belong together?
  • What is an appropriate key-scheme for this data? Should I choose my own or let Riak choose?
  • How much will I need to do online queries on this data? How quickly do I need them to return results?
  • What internal structure, if any, best suits this data?
  • Does the structure of this data promote future design modifications?
  • How resilient will the structure of the data be if requirements change? How can the change be effected without serious interruption of service?

I like to draw up my domain concepts on a pad of unlined paper or a whiteboard with boxes and arrows, then figure out how they map onto the database. Ultimately, the concepts define your application, so get those solid before you even worry about Riak.

Thinking non-relationally

Once you’ve thought carefully about the questions described above, it’s time think about how your data will map to Riak. We’ll start from the small-scale in this post (single domain concepts) and work our way out in future installments.

Internal structure

For a single class of objects in your domain, let’s consider the structure of that data. Here’s where you’re going to decide two interrelated issues — how this class of data will be queried and how opaque its internal structure will be to Riak.

The first issue, how the data will be queried, depends partly on how easy it is to intuit the key of a desired object. For example, if your data is user profiles that are mostly private, perhaps the user’s email or login name would be appropriate for the key, which would be easy to establish when the user logs in. However, if the key is not so easy to determine, or is arbitrary, you will need map-reduce or link-walking to find it.

The second issue, how opaque the data is to Riak, is affected by how you query but also by the nature of the data you’re storing. If you need to do intricate map-reduce queries to find or manipulate the data, you’ll likely want it in a form like JSON (or an Erlang term) so your map and reduce functions can reason about the data. On the other hand, if your data is something like an image or PDF, you don’t want to shoehorn that into JSON. If you’re in the situation where you need both a form that’s opaque to Riak, and to be able to reason about it with map-reduce, have your application add relevant metadata to the object. These are created using X-Riak-Meta-* headers in HTTP or riak_object:update_metadata/2 in Erlang.

Rule of thumb: if it’s an abstract datatype, use a map-reduce-friendly format like JSON; if it’s a concrete form, use its original representation. Of course, there are exceptions to every rule, so think carefully about your modeling problem.

Consistency, replication, conflict resolution

The second issue I would consider for each type of data is the access pattern and desired level of consistency. This is related to the questions above of read/write loads, churn, and conflicts.

Riak provides a few knobs you can turn at schema-design time and at request-time that relate to these issues. The first is allow_mult, or whether to allow recording of divergent versions of objects. In a write-heavy load or where clients are updating the same objects frequently, possibly at the same time, you probably want this on (true), which you can change by setting the bucket properties. The tradeoffs are that the vector clock may grow quickly and your application will need to decide how to resolve conflicts.

The second knob you can turn is the n_val, or how many replicas of each object to store, also a per-bucket setting. The default value is 3, which will work for many applications. If you need more assurance that your data is going to withstand failures, you might increase the value. If your data is non-critical or in large chunks, you might decrease the value to get greater performance. Knowing what to choose for this value will depend on an honest assessment of both the value of your data and operational concerns.

The third knob you can turn is per-request quorums. For reads, this is the R request parameter: how many replicas need to agree on the value for the read to succeed (the default is 2). For writes, there are two parameters, W and DW. W is how many replicas need to acknowledge the write request before it succeeds (default is 2). DW (durable writes) is how many replica backends need to confirm that the write finished before the entire write succeeds (default is 0). If you need greater consistency when reading or writing your data, you’ll want to increase these numbers. If you need greater performance and can sacrifice some consistency, decrease them. In any case, your R, W, and DW values must be smaller than n_val if you want the request to succeed.

What do these have to do with your data model? Fundamentally understanding the structure and purpose of your data will help you determine how you should turn these knobs. Some examples:

  • Log data: You’ll probably want low R and W values so that writes are accepted quickly. Because these are fire-and-forget writes, you won’t need allow_mult turned on. You might also want a low n_val, depending on how critical your data is.
  • Binary files: Your n_val is probably the most significant issue here, mostly depending on how large your files are and how many replicas of them you can tolerate (storage consumption).
  • JSON documents (abstract types): The defaults will work in most cases. Depending on how frequently the data is updated, and how many you update within a single conceptual operation with the application, you may want to enable allow_mult to prevent blind overwrites.

Sean Cribbs