February 21, 2013
Today we are excited to announce the latest version of Riak. Here is a summary of the major enhancements delivered in Riak 1.3:
- Introduced Active Anti-Entropy. Riak now has active anti-entropy. In distributed systems, inconsistencies can arise between replicas due to failure modes, concurrent updates, and physical data loss or corruption. Pre-1.3 Riak already had several features for repairing this “entropy”, but they all required some form of user intervention. Riak 1.3 introduces automatic, self-healing properties that repair entropy on an ongoing basis.
- Improved Riak Enterprise’s multi-datacenter replication performance. New advanced mode for multi-datacenter replication capabilities, with better performance, more TCP connections and easier configuration. Read more in this write up from GigaOM.
- Improved graphical user experience. Riak Control, the user interface for managing and monitoring Riak, has a brand new look.
- Expanded IPv6 support. IPv6 support in Riak now is supported by all interfaces.
- Improved MapReduce. Riak MapReduce has improved back-pressure to reduce the risk of overwhelming endpoint processes during large tasks.
- Simplified log management. Riak can now optionally send log messages to syslog.
Ready to get started or upgrade? Download the new release here, check out the official release notes, or read on for more details. Documentation for all products and releases is available on the documentation site. For an introduction to Riak and what’s new in Riak 1.3, sign up for our webcast on Thursday, March 7.
More on What’s in Riak 1.3
A key feature of Riak is its ability to regenerate lost or corrupted data from replicated data stored on other nodes. Prior to this release, Riak provided two methods to repair data:
- Read Repair: Riak compares the replies from all replicas during a read request, repairing any replica that is divergent or missing data. (K/V data only)
- Repair Command via Riak Console: Introduced in Riak 1.2, the repair command enables users to trigger a repair of a specific partition. The partition is rebuilt based on a subset of data stored on adjacent nodes in the Riak ring. All data is rebuilt, not just missing or divergent data. (K/V and Search data)
Riak 1.3 introduces active anti-entropy, a continuous background process that compares and repairs any divergent, missing, or corrupted replicas (K/V data only). Unlike read repair, which is only triggered when data is read, the active anti-entropy system ensures the integrity of all data stored in Riak. This is particularly useful in clusters containing “cold data”: data that may not be read for long periods of time, potentially years. Furthermore, unlike the repair command, active anti-entropy is an automatic process, requiring no user intervention and is enabled by default in Riak 1.3.
Riak’s active anti-entropy feature is based on hash tree exchange, which enables differences between replicas to be determined with minimal exchange of information. Specifically, the amount of information exchanged in the process is proportional to the differences between two replicas, not the amount of data that they contain. Approximately the same amount of information is exchanged when there are 10 differing keys out of 1 million keys as when there are 10 differing keys out of 10 billion keys. This enables Riak to provide continuous data protection regardless of cluster size.
Additionally, Riak uses persistent, on-disk hash trees rather than purely in-memory trees, a key difference from similar implementations in other products. This allows Riak to maintain anti-entropy information for billions of keys with minimal additional memory usage, as well as allows Riak nodes to be restarted without losing any anti-entropy information. Furthermore, Riak maintains the hash trees in real time, updating the tree as new write requests come in. This reduces the time it takes Riak to detect and repair missing/divergent replicas. For added protection, Riak periodically (default: once a week) clears and regenerates all hash trees from the on-disk K/V data. This enables Riak to detect silent data corruption to the on-disk data arising from bad disks, faulty hardware components, etc.
New Look for Riak Control
Riak Control is a UI for managing and monitoring your Riak cluster. Riak Control lets you start and re-start Riak nodes, view a “health check” for your cluster, see all nodes and their current status, and have visibility into their partitions and services. Riak Control now has a brand new look and feel. Check out the Riak Control Github page to get up and running.
Expanded IPv6 Support
While Riak’s HTTP interface has always supported IPv6, not all of its interfaces have been as current. In Riak 1.3, the protocol buffers interfaces can now listen on IPv6 or IPv4 addresses. Riak handoff (which is responsible for data transfer when nodes are added or removed, and for handing off update responsibilities when nodes fail) also supports IPv6. It should also be noted that community member Tom Lanyon started the work on this feature. Thanks, Tom!
Improved Backpressure in Riak MapReduce
Riak Enterprise: Advanced Multi-Datacenter Replication Capabilities
With hundreds of companies using Riak Enterprise, a commercial extension of Riak, we’ve been lucky to work with many teams pushing the limits of multi-datacenter replication performance and resiliency. We’ve learned a lot and are excited to announce these capabilities are now available in advanced mode.
- Previously, multi-datacenter replication had one TCP connection over which data was streamed from one cluster to another. This could create a performance bottleneck, especially when run on nodes constrained by per-instance bandwidth limits, such as in a cloud environment. In the new version of multi-datacenter replication, multiple concurrent TCP connections (approximately one per physical node) and processes are used between sites.
- Configuration of multi-datacenter replication is easier. Use a shell command to name your clusters, then connect both clusters using a simple ip:port combination.
- Better per-connection statistics for both full-sync and real-time modes.
- New ability to tweak full-sync workers per node and per cluster, allowing customers to dial-in performance.
The new replication improvements are already used in production by customers and yielding significant performance improvements. For now, the new replication technology is available in advanced mode: it’s optional to turn on. It currently doesn’t have all of the features of the default mode – including SSL, NAT support and full-sync scheduling. Both default and advanced modes are available in the 1.3 release and function independently. In the future, “advanced mode” will become the default.
For more details about multi-datacenter replication, download our whitepaper, “Multi-Datacenter Replication: A Technical Overview.”
February 19, 2013
Hibernum is a creator and developer of unique gaming experiences that combine the latest in social gaming, top quality visuals and animations, and cutting edge design. They use Riak to store user game information for one of their most popular social games.
Currently, Hibernum’s Riak installation serves thousands of requests per second to more than a million monthly active users. User data is stored in Riak as JSON objects, and Hibernum uses Riak’s HTTP interface, a perfect fit for their Node.js-based application server. As the game grows in popularity, millions of new entries are generated and stored in Riak, as well as any updates or modifications that may occur during gameplay. Mario Lefebvre, IT Specialist at Hibernum, has said that Riak is “managing this load like a charm and is a stable and rock solid solution.”
Originally, Hibernum was using a relational database, however, they found the manual sharding required to scale was operationally intensive and inefficient. They needed something that could better handle their significant growth and started looking for a cost-efficient solution that could support the large amount of requests, as well as a solution that allowed for easy scalability. After testing multiple solutions, Riak was chosen for its high availability, ability to scale to peak loads, and predictable operational cost.
To learn more about how Hibernum uses Riak, check out the complete case study.
February 14, 2013
Advertisers need to provide highly available, low latency experiences to thousands of clients and partners and millions of users. They also need to serve large amounts of data all over the world and can experience significant traffic spikes. To meet these needs, more advertisers are considering distributed data solutions. This post looks at common use cases for Riak in the advertising space, and the stories of two existing advertising users. For a full technical overview, download our whitepaper on Riak for advertisers.
Top Use Cases for Riak in Advertising:
- Serving Ad Content: Riak’s rapid storage and content agnosticism makes it ideal for storing ad content and handling influxes of ad traffic. For more information on serving ad content with Riak, check out our documentation.
- Session Storage: This type of data is naturally a good fit for Riak’s key/value model. This data can also be encoded in many different ways and can evolve without any administrative changes to the schema. You can find more information on building a session store with Riak here.
- Mobile: Riak is ideal for the low-latency, always-available small object storage needed to power mobile experiences across platforms.
- Global Data Locality: Riak Enterprise’s multi-datacenter capabilities allow advertisers to maintain a global data footprint while providing an always-on, low-latency experience, anywhere in the world.
OpenX, the global leader in digital and mobile advertising technology, serves trillions of ads each year. They use Riak for handling user and trafficking data storage behind their data services API. Riak was selected due to its highly available, low-latency, redundant architecture. OpenX also uses Riak Enterprise’s multi-datacenter replication across several data centers. For more details about how OpenX uses Riak, check out the video of Anthony Molinaro, OpenX engineer, speaking at RICON2012, Basho’s 2012 developer conference.
Velti is a global marketing and advertising technology provider. Velti’s interactive subscriber services provide television broadcast audiences the ability to interact with programs using their mobile phone– voting on people or things, giving feedback, or participating in contests. They selected Riak because it is distributed, scalable, and highly available with the ability to handle large volumes of traffic. To minimize any potentially catastrophic outages, they also opted to build two geographically separated, mirrored sites using Riak Enterprise’s multi-datacenter replication feature. For more information on Velti’s use of Riak check out the complete case study.
To learn more about how advertisers can use Riak for their data needs, check out the complete overview, “Advertisers on Riak: A Technical Introduction,” or stay tuned for future blogs posts on data modeling and querying for advertising services built on Riak.
February 13, 2013
Bump, one of the most popular mobile apps of all time, makes it easy for users to share their contact information, photos and other objects by simply “bumping” their smartphones. Bump uses Riak to store user data including events, communications sent and received, handset information and tokens needed to authenticate using social networks.
Bump chose Riak for its operational ease-of-use, ability to scale writes, and availability under failure conditions.
“It’s a relief that we don’t need to spend time thinking about whether or not Riak is working,” said Will Moss, Server Engineer at Bump. “It does what it’s supposed to do; nodes can go down but Riak will still work. It’s great to be able to deal with node failures the next day instead of at 3am.”
Recently, Bump expanded their mobile app offerings and launched Flock, a photo-sharing app. For more information on how Flock uses Riak, including their data model, watch Bump’s presentation at RICON2012, Basho’s 2012 developer conference. Bump is now running 25 nodes on Riak and storing around 3TB of data.
You can also check out the complete case study.
February 12, 2013
OmniTI is a provider of web infrastructures and applications for companies that require scalable, high-performance, mission critical solutions. They specialize in providing complex, high-transaction, and large-volume data applications. One of their customers is Viggle, a mobile app that rewards people for checking into the television shows they are watching. Viggle has more than a million users, and its advertisers include Pepsi, Kraft, and Capital One.
OmniTI designed the server architecture for Viggle’s mobile app and also designed the internal APIs that connect Viggle’s multiple back-end services, creating a cohesive, fault-tolerant system. OmniTI chose Riak as an integral part of this system, providing high availability and low latency during peak times.
“For this architecture, near-zero downtime and sustaining high throughput with low latency are critically important,” said Theo Schlossnagle, CEO of OmniTI. “We needed Viggle’s key components to remain available and responsible under sudden floods of user traffic, which made Riak the perfect fit. Riak has not disappointed us and has performed exactly as needed for this application.”
For more information on how Viggle has used OmniTI to design and support their system, check out their full announcement.
February 11, 2013
We are excited to announce Datapipe’s Stratosphere, a globally available, high-performance managed cloud computing platform, leverages Riak Cloud Storage (CS). Riak Cloud Storage provides Datapipe and its customers with highly available, low-latency and S3-compatible storage.
Datapipe offers a single provider solution for managing and securing mission critical IT services, including cloud computing, infrastructure as a service, platform as a service, managed hosting, and colocation.
Stratosphere is Datapipe’s globally available managed cloud computing platform. With the launch of Riak CS to support cloud object storage, Datapipe customers can now access cloud object storage from any solution hosted with Datapipe and adjacent to existing solutions in any Datapipe data center. Stratosphere is designed for enterprise high I/O production environments and can also be used for development, testing and QA environments. Use cases include large-scale marketing campaigns, brand sites and analytics; applications with variable peak demand times and other dynamic workloads; and cloud disaster recovery and geographic redundancy.
Datapipe delivers services from the world’s most influential technical and financial markets including New York metro, Silicon Valley, London, Hong Kong and Shanghai.
Why Riak Cloud Storage at Datapipe?
Datapipe selected Riak Cloud Storage for its low-latency, highly available object storage, operational ease-of-use, and multi-site replication capabilities. After extensively testing solutions from a variety of vendors in the space, Datapipe selected Riak Cloud Storage for a few core reasons:
- Built on years of developing Riak, Riak CS is designed to provide simple, available, distributed cloud storage at any scale.
- Riak CS is compatible with major cloud object storage clients and applications with its S3-based API.
- Riak CS meets the high performance requirements of the Stratosphere cloud-computing platform.
“Riak CS provides the high-performance, distributed datastore we need to deliver a sound foundation for our cloud storage needs now and for many years into the future,” said Ed Laczynski, VP Cloud Strategy, Datapipe.
Be on the lookout for upcoming documentation about using Riak CS-backed functionality on Stratosphere at Datapipe. Riak CS is now available with Datapipe in a limited beta, with an upcoming full release.
For a developer trial of Riak CS, sign up here.
February 7, 2013
Basho and our community have a handful of events lined up for February 13th. We have official meetups/group hacks in at least seven cities in the US.
We hope to see you next week. If you can’t attend an official Meetup, throw a Riak hack or drink up in your city and email firstname.lastname@example.org to tell us about it.
Thanks for being a part of Riak.
- Speaker: Weston Jossey, Software Engineer, Tapjoy
- Talk Title: Huge Data Migrations to Riak Made Easy(er)
- Details and RSVP
- Speaker: Sean Cribbs, Software Engineer, Basho Technologies
- Talk Title: The Deep Riak
- Details and RSVP
New York City
- Speaker: Aaron Brown, Lead Systems Engineer, ideel
- Talk Title: Riak at ideel
- Details and RSVP
- Speaker: Adron Hall and You
- Talk Title: Riak Hack & Brew
- Details and RSVP
- Speaker: Robert Zuber, Co-Founder, Copious
- Talk Title: Riak in a Multi-Datastore Strategy at Copious
- Details and RSVP
- Speaker 1: Pavan Venkatesh, Technical Evangelist, Basho Technologies
- Talk Title 1: From Relational to Riak
- Speaker 2: Sajith Kizhakkiniyil, Software Infrastructure and Backend Architecture Support, Apollo Group
- Talk Title 2: Riak at Apollo
- Details and RSVP
- Speaker: Adron Hall and You
- Talk Title: Nerd Lunch and The Start of Seattle Riak
- Details and RSVP
TED to Leverage Deep Relationships with Enterprise Companies to Accelerate Adoption of Riak Throughout Japan
CAMBRIDGE, MA and YOKOHAMA, JAPAN – February 7, 2013 – Basho Technologies, Inc. and Tokyo Electron Device Limited (TED) announced a strategic partnership and distribution agreement under which TED will resell Basho products throughout Japan and has become a strategic equity investor in Basho. Basho Technologies specializes in distributed systems technologies and is the creator of Riak, the industry leading distributed database and cloud storage software. TED provides world-class products and solutions that deliver competitive advantages to its customers. The strategic partnership enables Basho and TED to capitalize on the comprehensive resources of TED to open up new opportunities for Basho in the Japanese market. As part of the partnership, TED will build and maintain dedicated sales support and post-sales support resources specifically around Riak, Riak CS and future new products from Basho.
“Basho is very excited to enter into a long-term strategic partnership with TED,” said Sam Takagi, general manager of Basho Japan and Asia Pacific. “TED is highly regarded throughout Japan for its expertise in storage infrastructure, data backup and protection, and data warehouse design and operations. Riak’s strengths around high-availability, scalability and predictability are highly complementary to TED’s expertise and will provide an important new and innovative database and storage solution for TED’s customers. With Riak, Japanese businesses can meet demanding Internet, social and mobile requirements, as well as build highly-competitive public clouds and secure, high-performance private clouds.”
“Riak’s inherent distributed data capabilities offer a unique solution for companies building next generation applications, and cloud computing platforms that require high scalability, no downtime, and low cost operations,” said Vic Amano, VP & GM CN Business of Tokyo Electron Device Limited. “Our highly complementary expertise in data storage and our large and established network of commercial and industrial customers position us well to quickly speed adoption of Basho’s technology throughout Japan. The strategic nature of our partnership with Basho allows us to collaborate on future customer requirements and on product directions, allowing TED to maintain a competitive advantage for the next generation of database and storage technologies.”
“Through this strategic partnership, Basho is further building its global presence and particularly in the important and large market of Japan,” said Greg Collins, Basho’s president and CEO. “Our partnership with TED further underscores Basho’s commitment to the Japanese market. We are committed to building strong local capabilities and leveraging partnerships that have strong business networks and local-market expertise. TED is a terrific match for Basho. We look forward to working with TED and its customers for many years to come.”
Today’s announcement follows Basho’s recent opening of its Tokyo Office, officially launched on September 27, 2012.
About Tokyo Electron Device (TED) CN Business:
Tokyo Electron Device (TED) is a technical trading firm with a “trading business” function that provides semiconductor products and business solutions as well as a “development business” function that performs commissioned designing and the development of own-brand products. The Computer Network (CN) Business Section handles a wide range of storage systems, network-related equipment, and middleware products and provides them as part of its business solutions in the era of cloud computing. It has marketing functions in Japan and overseas to pick up on trends in the world’s advanced technologies ahead of others in order to offer products and services that cover processes that span everything from implementation to support.
For more information, visit: http://cn.teldevice.co.jp/english/.
About Basho Technologies
Basho Technologies is the leader in highly-available, distributed database technologies used to power scalable, data-intensive Web, mobile, and e-commerce applications and large cloud computing platforms. Basho customers, including fast-growing Internet-based businesses and large Fortune 500 enterprises, use the company’s flagship product, Riak, to deliver and manage digital media and unstructured data, implement multi-device user activity and sessions stores, to aggregate large amounts of data for logging, search and analytics, and to build scalable cloud storage platforms. The company is based in Cambridge, Massachusetts and operates regional offices in London, San Francisco, Tokyo and Washington DC.
Basho Technologies Medica Contact:
Bobby Patrick Chief Marketing Officer, Basho Technologies
Tokyo Electron Device Media Contact:
Yoichiro Hotta, Yoko Fukui Corporate Communications Department, Tokyo Electron Device Limited
Contact form: https://www.teldevice.co.jp/eng/contact_form_news.html
For inquiries regarding Basho Technologies and Riak Tokyo Electron Device Limited
Tsuyoshi Yoshi Tanaka, 1-510-624-3463
CN Business Contact form: http://cn.teldevice.co.jp/company/tea/form.html
February 4, 2013
Basho is proud be to be sponsoring Rubyfuza, happening this week in Cape Town, South Africa. We’ve sponsored and spoke at a few events in Cape Town over the past several years, and Riak has seen successful adoption among various South Africa-based companies like Praekelt and Mad Mimi. We’re happy to support the tech community in and around “The Tavern of the Seas.”
Sadly, as was the case with PyConZA, we won’t be able to send an official Basho envoy to be a part of it. That’s why we’re excited to announce that Jeremy Thurgood, a member of the Praekelt engineering team, will be there on our behalf as the Riak Ambassador.
Jeremy and his team have been using Riak in production for various projects, and have contributed some useful code to the community along the way. Though Python is his language of choice, he’s been brushing up on his Ruby and will be on-hand to answer any Riak questions that attendees might have. There will also be some Basho and Riak swag floating around to make sure that no one goes home empty-handed.
Enjoy Rubyfuza and make sure to say his to Jeremy if you have a moment. Also, if you’re attending a conference and want to serve as an extension of the Basho Team as a Riak Ambassador, shoot an email to email@example.com. We want to talk to you.
February 4, 2013
Today I’m happy to announce the 3rd pre-release of Yokozuna. It’s light on new features but has some good performance improvements and added robustness. Here are the highlights:
- Allow store/retrieval of schemas via HTTP.
- Upgrade to Solr 4.1.0 and the latest Riak.
- Improve write/index throughput by disabling Solr’s “realtime get” and switching from XML update to JSON.
- Added robustness around AAE and default index creation.
- Listen on ‘solr//select’ to more easily work with existing clients out of the box.
New for this release is the addition of a source package. I hope this might encourage those who are scared off by the process of building from git to give Riak/Yokozuna a try. These four steps below will produce a ready-to-run node under ‘rel/riak’.
wget http://s3.amazonaws.com/yzami/pkgs/src/riak-yokozuna-0.3.0-src.tar.gz tar zxvf riak-yokozuna-0.3.0-src.tar.gz cd riak-yokozuna-0.3.0-src make stage