Thomas Pillmayr Lead Architect at LineMetrics
"We are impressed by the scalability so far. It is intended by design but it just amazes us time and time again to see no matter how much data is in our data store there literally is no noticeable difference in performance."
Thomas Pillmayr Lead Architect at LineMetrics

LineMetrics is an IoT platform for the B2B market which facilitates continuous improvement processes based on real world Displaying LineMetrics_Factory.jpgsensor data. We have a great variety of customers in many different fields: small, medium, and big sized companies covering metal manufacturing, plastics manufacturing, energy consultants, power supply companies, retail chains like banks, retailers, hotels, etc. LineMetrics integrates seamlessly into the continuous improvement process: creating transparency, defining objectives/measures and controlling achievements.

It’s as simple as it can get: an easy to use tool for optimization processes based on sensor data. I am the lead architect of the system architecture and the software stack of the LineMetrics product.

Time series

Cassandra is used to store raw sensor data and aggregated data with a time reference. For example we are collecting and storing sensor data like energy consumption, temperature, CO2 levels, production output, stoppages of machines, etc. We take this data and transform it into different time series data (e.g. sums per month, averages, peaks, etc.). All this data is held within Cassandra.

Our time series data was stored in a MySQL database before. But the schema restrictions and data size has forced us to move to another storage system quickly. To be honest there was not much debate about other alternatives. Cassandra simply does the job we need to be done really really well.

We required continuous real-time analytics based on sensor data means working with a huge amounts of data. Within our first 12 months our data storage passed 1 billion data points. Right when we started to design our product we knew we needed a database perfectly suited for our needs. Our data mostly is structurally simple time series data. 

We quickly discovered that relational databases did not meet the performance we set as goal for our application. That was somewhat expected but our technical background was mostly in relational databases. Naturally we looked for alternatives and Cassandra was the obvious choice for us. If I remember correctly we read about Facebook using Cassandra for a specific part of their data at the time. Cassandra perfectly handles our data in the way we need it.

Linear scale

We are impressed by the scalability so far. It is intended by design but it just amazes us time and time again to see no matter how much data is in our data store there literally is no noticeable difference in performance. Optimization for quick data access is just superb with Cassandra. Our query time constantly is below 10ms. No matter what we do.

At the moment we are running two nodes. Data is stored on two geographically separated locations. Our test environments are mirroring the data structure from our live production system but are filled with automatically generated dummy data.

Understand your data model

Most important step in the beginning is to model the data structure to fit your future queries. That is the key for constantly quick queries. In our case we designed an index that basically only has two levels: which datastream at which aggregation level. The second level specifies the time range (date from/to).

From what we have seen there is a thriving community all around the world. We are looking forward to join one or another live event in the future. Due to the Runtastic founders being business angels at LineMetrics our knowledge exchange mostly happened with the awesome Runtastic team. They have done great things with Cassandra and we are happy to learn from them as much as we can.

We are just grateful for what Cassandra and the people behind it are giving us. We will make sure to one day return the favor.

We Shall Have Order!

March 2, 2015

By 

Aaron Ploetz,Lead Database Engineer at AccuLynx.com
Aaron is the Lead Database Engineer for AccuLynx, where he implemented Cassandra to scale part of their cloud-based business management system. He is active in the Cassandra tags on StackOverflow, and recently became the first recipient of a tag-specific badge in CQL. Aaron holds a B.S. in Management/Computer Systems from the University of Wisconsin-Whitewater, a M.S. in Software Engineering and Database Technologies from Regis University, and was selected as a DataStax MVP for Apache Cassandra in 2014.

Introduction

One of the more misunderstood aspects of Cassandra data modeling is that of how to properly order your data.  In this article we will explore a typical modeling journey from “first attempt” to “end solution.”  We will discuss the limitations of the ORDER BY clause, as well as explain how Cassandra clustering order works – and how to take advantage of that order when building a data model.

A common problem that new Cassandra users will attempt to solve is that of modeling time series data.  One of the more typical early mistakes made in time series data modeling is designing a table that is dependent on a time as its primary key, like this:

This is commonly accompanied by a question such as:

“How can I sort my result set by posttime?”

Of course – for this problem – designating posttime as a single primary key is not going to have the desired effect.  Cassandra will place (DataStax 2015a) “data on each node according to the value of the partition key and the range that the node is responsible for.”  It is important to understand that the order in which partitioned rows are returned, depends on the order of the hashed token values and not on the key values themselves.  So if the goal is to order the queried results by posttime then the primary key definition is going to have to be different.

Some new users make the mistake of using the Byte Ordered Partitioner (BOP) to sort data by partition key.  This is discouraged, given its propensity for hot spots and load balancing difficulties.  Some have even gone so far as to identify its use as an anti-pattern.  It is this author’s opinion that no new clusters should be built with the BOP.  Often, a use case which may appear to be a “good fit” for the BOP can be solved with proper data modeling.

The correct way to model this is to first find a different column by which to partition the data.  Proper partitioning and clustering keys will allow the data to be sorted and queried in the desired order.  The column to designate as the partition key largely depends on the requirements of the application and the particular query you are trying to solve.  Be mindful of the cardinality of your potential partition key.  If it is too low you will get “hot spots” (poor data distribution), and if it is too high you will negate the benefits of the “wide row” data model (too little data to make ordering worthwhile).

Another incorrect path which new users can find themselves going down is to designating a “dummy” partition key.  In this solution, the “dummy” key always has the same value (lowest possible cardinality), and its use can fool the user into thinking that the problem is solved.  The two main issues with this are that of hot spots and (Strickland 2014) unbounded row growth.  As the (constant) partition key value will be hashed the same on each insert, all of the data will be written to the same partition, thus creating hot spots of data.  Additionally, Cassandra can only support (McFadin 2014) two billion columns per partition.  With all data being written to that one partition, the chances of reaching that limit will increase over time.  As these two points are significant potential issues, I cannot stress enough that using a “dummy” partition key is a terrible idea.

For our example, we will decide to partition our data by userid with a new table specifically designed to serve the query, like this:

This particular table uses a compound primary key.  The first column listed in the primary key definition is the partition key (userid) and the second column is a clustering column (posttime).  After the partition key(s), data for each partition (DataStax 2015b) is “clustered by the remaining column or columns of the primary key definition.”  Essentially, the clustering columns determine the on-disk sort order within each partition.  Note that this table definition also uses the WITH clause, which indicates that the data should be clustered in descending order by posttime.

However, this can lead to another common incorrect assumption.  Consider the following query and question:

“I am trying to query the 100 most-recent posts by user.  Why is my data in random order even though CLUSTERING ORDER BY is specified in postsbyuser?”

In the example table above, queries will return data that is sorted according to the clustering key(s) only when a partition key is also specified.  Without a partition key specified in the WHERE clause, the actual order of the result set then becomes dependent on the hashed values of userid.  This is apparent when postsbyuser is queried without specifying a WHERE clause and using the token() function on userid:

In any case, to serve the above query we need qualify it with the partition key (userid).  This query will return the 100 most-recent posts where userid is equal to 2:

While we are on the right track, one last issue that may still be encountered with this solution is (again) unbounded row growth.  Depending on the frequency of user postings, this solution may be at risk for hitting the limit of two billion columns per partition (probably not, but we will assume so for the purposes of this example).  A possible solution for this is to add an additional partition key.  Let’s say that we know that the postsbyuser data will not exceed the limit of two billion columns per partition in a single calendar year.  Then, we could solve this issue by using year as an additional partition key:

Now we can query the top most-recent posts by userid for a specific year, in descending order by posttime:

Note that the ORDER BY clause does not need to be specified in the SELECT, as the data will already be sorted based on the CLUSTERING ORDER specified in the table definition.  The ORDER BY clause could be used if we wanted to alter the sort direction (“ASCending” vs. “DESCending”) of the result set.

In summary, when it comes to ordering your result sets in Cassandra, remember these points:

  • The clustering column(s) determine the data’s on-disk sort order only within a partition key.

  • Do not model your table with a “dummy” partition key (key that always has the same value).

  • It is not necessary to specify the ORDER BY clause on a query if your desired sort direction (“ASCending/DESCending”) already matches the CLUSTERING ORDER in the table definition.

  • On a CREATE TABLE command, Cassandra will specify CLUSTERING ORDER for you in the table definition if you did not specify it yourself.  The default CLUSTERING ORDER on a table is comprised of your clustering keys in the “ASCending” sort direction.

  • Be mindful of the limit of two billion columns per wide row (partition), as well as the problems that accompany unbounded row growth.

  • For tables supporting queries for recent, time-based data, you may want to specify a “DESCending” sort direction in your table definition.

  • Do not create a new cluster using the Byte Ordered Partitioner.  If still you think that you really need to do this, contact someone at DataStax support first.

Happy modeling!

 

References

DataStax (2015a).  Consistent Hashing.  Retrieved from: http://www.datastax.com/documentation/cassandra/2.1/cassandra/architecture/architectureDataDistributeHashing_c.html

DataStax (2015b).  Compound Keys and Clustering.  Retrieved from: http://www.datastax.com/documentation/cql/3.1/cql/ddl/ddl_compound_keys_c.html

Denis M. (2012).  Apache Cassandra Anti-Patterns.  Slide 13.  Strange Loop 2012.  St. Louis, MO.  Retrieved from: http://www.slideshare.net/mattdennis/strangeloop-2012-apache-cassandra-anti-patterns (slide 13).

McFadin P. (2014).  Getting Started With Time Series Data Modeling.  Retrieved from: http://planetcassandra.org/blog/getting-started-with-time-series-data-modeling/

Strickland R. (2014).  Cassandra High Availability.  Packt Publishing Ltd.  Birmingham, UK.  (pp. 144-45).

Brent Ryan Enterprise Architect at Cvent
"The main reason we needed Cassandra was for scalability. We wanted something that could handle very high volume traffic with large spikes and at the same time allow us to add more nodes over time if we needed to scale out over time."
Brent Ryan Enterprise Architect at Cvent

Cvent builds a SAAS platform for event planners and management. So if you’re planning a conference and need to send out marketing materials, setup agendas, pick hotel venues, send out emails, setup a registration website, send out surveys then Cvent is what you’d want to use.

My role here as the enterprise architect is to enhance our architecture so that we can continue to scale our products for the next 10-20 years.

Cassandra at Cvent

We currently use Apache Cassandra in a variety of ways.

 - We use it as part of DataStax Enterprise SOLR offering to help horizontally scale some of our search needs that allow customers to perform free text search through RFPs.

 - We use it as a key/value store to offload most of the work from our OLTP (SQL Server) database in support of our Survey product. We make heavy use of secondary indexes as well with this use case.

 - We use it as a datastore for our in-house analytics engine where we dump all of our analytical “counter” data. We then use spark streams to perform aggregations of this data and store the results back for further reporting.

In all 3 of our use cases we’re using DataStax Enterprise.

Horizontal scale

The main reason we needed Cassandra was for scalability. We wanted something that could handle very high volume traffic with large spikes and at the same time allow us to add more nodes over time if we needed to scale out over time. Many of our use cases as a SAAS event management product have very large high volume spikes while most of the day has only moderate volume. The other nice thing about Cassandra was the secondary indexes. This comes in handy for our second use case where we’re storing a large JSON blob, but then needed to find the row key via a secondary index. This effectively turns our Cassandra cluster into a document database and it’s worked out nicely.

Evaluating the NoSQL space

The evaluation process involved both prior experience and future needs for our business. We’ve evaluated the following databases: MemSQL, NuoDB, Redis, Memcache, Membase, MongoDB, Couchbase, Postgres, MySQL.

We’ve also briefly looked in VoltDB, Riak, FoundationDB, and Aerospike. Ultimately, many of these databases could have solved our use cases with varying degrees of success. We choose Cassandra primarily because of it’s ease of setup, good documentation, price and support from DataStax. I wanted to choose a technology that had good commercial support and was easy to run operationally.

Deployment stats

We’re currently using a 3 node DataStax Enterprise cluster running version 3.1.4 and another 3 node cluster is being evaluated for our analytics that’s running DataStax enterprise 4.5.1. We currently are not running a multi-data center deployment since most of our data can be replicated from other data sources in the event of a catastrophic failure.

The performance of Cassandra has been excellent. We have spikes up to 5000 read/write per second without issues in production. During performance testing and evaluation we were easily able to get up to 20,000 per second. The 2nd thing that was really nice was the ease of setup “yum install dse” and then there’s lots of nice blogs and documentation around properly tuning your production cluster.

Getting started tips

The best way to get started is to just download the software, install it and start playing with the demos. I highly recommend using something like Vagrant to allow you to quickly iterate on running a local cluster. Once you get it all running then I like to start playing around with the SDK client apis that interface with the database because this gives me a good feel of how it will work for real scenarios. I can also quickly write up some quick performance tests and iterate as I figure out how the technology works.

When you’re evaluating NoSQL technology you must make sure to understand both your read use cases and your write use cases. With many NoSQL technologies it’s easy to scale out write volume, but then the performance of reads suffers a lot unless you use the primary key. With the Cassandra secondary indexes we get both the flexibility of adding them later as well as the ability to query for our data like a document database. The performance of doing this has allowed us to have great performance on the read and the write side of our queries.

The community behind Cassandra has been great. The Cassandra mailing list is very active and everyone is very helpful. I’ve also gotten lots of use out of the DataStax forums and support channels, Netflix blogs, eBay blogs, etc. There’s tons of useful articles out there on Cassandra.

Juho Mäkinen Head of Technical Operations at Unity
"The old MongoDB had bigger latency distribution, but now our 95% percentile latency is really flat and low... Latency translates directly into better end user experience, plus it gives us confidence that the database cluster has plenty of room to scale."
Juho Mäkinen Head of Technical Operations at Unity

Unity is a game development ecosystem: a powerful rendering engine fully integrated with a complete set of intuitive tools and rapid workflows to create interactive 3D and 2D content; easy multiplatform publishing; thousands of quality, ready-made assets in the Asset Store and a knowledge-sharing community. My role is to lead the Technical Operations for the new Unity Ads and Everyplay community services. The Unity Ads gives game developers the ability to show video ads in games and both monetise their players and promote new games.

Migrating from MongoDB

The datastore has few hard requirements: It needs to scale linearly to several terabytes of storage, it can’t have any single point of failures, it needs to support distributing data in an active master-master way into multiple different datacenters and it needs to support TTL for each record.
3d
We were using MongoDB before as our key database but we ran out from its comfortability zone a long ago when our active dataset grew. We weren’t happy with its sharding and it doesn’t really support master-master setup in multiple datacenters. As some of our operators had previous Cassandra experience from the past it was a natural candidate as a replacement. We installed a six node Cassandra cluster and modified our core services to duplicate all data requests to it and we ran this setup half a year in parallel with the original MongoDB. During this time the MongoDB database was still the production database and the data from Cassandra cluster wasn’t used for anything else than to just test the cluster performance. After we were confident it suited our use case we switched our application to use the Cassandra cluster as the source of truth and we’ll be soon decommissioning the old MongoDB database.

As the database contained around 500 million documents we first spent some time thinking how we should model it in Cassandra. We ended up choosing a time series style list of events per each tracked object. In our data model each object has an unique id which works nicely as the partition key. Then, each tracked event is placed as rows into the partition with the timestamp as the second primary key. This was already an improvement over the old MongoDB model because we felt confident we could store more detailed data in Cassandra than MongoDB. We also used a few static columns to store common properties for each object. Each time the the database is used the frontend node.js server does a query into the MongoDB to get a document, modifies it and saves the updated data back to MongoDB.

We used a shadow launch method, which essentially means that we implemented as much of the desired functionality into production as possible, but without actually relying on it. First we altered our node.js frontend code to save a copy of each data into Cassandra. This allowed us to watch closely how the incoming data sets into the new Cassandra cluster and track its performance. Once we were satisfied we altered the frontend code to also do queries to the Cassandra cluster, but not to use those query responses in any way. This meant that the production was still running from the old MongoDB database. We let the system do this for couple of months, which resulted with around one terabyte of data in the Cassandra cluster. During this phase we practiced Cassandra operations by removing and adding nodes, running repairs with different methods, created network topology splits, rolling upgrades to a new minor version and verified that we have good monitoring and understanding on every part of the cluster.

After this we truncated the Cassandra database and modified our front-end code to start doing reads to Cassandra and to use the data from Cassandra and to fallback to read from MongoDB if Cassandra didn’t contain the data. After this the data was always saved to Cassandra. As the migration started the miss ratio from Cassandra quickly fell from 100% to around 30% within a couple of days and then gradually decreased as more and more documents were migrated. Eventually we can just disable the old MongoDB code path and conclude that the migration is fully completed.

This kind of shadow launch and gradual migration is usually pretty easy to implement assuming the code accessing the database tier is structured with good programming practices and patterns. This also allowed us to keep 100% uptime during the entire migration. Another option would have been to migrate data offline from MongoDB to Cassandra, but this could have easily taken days so we didn’t even consider that.

Improved latency for ads

Our biggest single user is our Unity Ads which stores history data on shown ads in the entire ad network. This is a real time need and the database is touched each time ads are shown in a game and the data is used to calculate which ads are shown for the player. We’re heavily using the time to live (TTL) feature in Cassandra to clean the database from old records.

Our biggest benefit received is better request times from the database tier. The old MongoDB had bigger latency distribution, but now our 95% percentile latency is really flat and low. There’s still some optimisations to be done to gain smoother 99% percentile but we’re already really happy with the performance. Latency translates directly into better end user experience, plus it gives us confidence that the database cluster has plenty of room to scale. The programmers are happy with the CQL language. It’s still doesn’t have many features when compared to a full SQL implementation, so there’s still a lot of room ease the CQL usage by implementing more SQL like features.

Deployment

We are currently running 15 c3.2xlarge nodes with 2.0.11 in a single AWS region with replication factor three. We did try i2.xlarge instances, but they were lacking CPU power for our data model. Because the CPU-SSD ratio is un-optimal in c3.2xlarge instances for Cassandra usage, we run parts of our middle tier in the same machines, which benefit from the reduced latency to the database. We run everything inside Docker containers which hasn’t been any problem for Cassandra.

We’re in the process of migrating some smaller datasets from MongoDB to Cassandra so that we gain the ability to distribute our production into multiple data centers (both AWS and private cloud). The end goal is to use geoip DNS to direct users to the closest data center. We’re using OpsCenter community edition to get an overview of the cluster and the metrics-graphite plugin to get instrumentation into our Graphite monitoring setup. We monitor latencies from both Cassandra side and our application side for each different table.

It’s known that there are several Cassandra clusters with well over one magnitude bigger than what we have, so I have no reason to believe that we can scale well into the future. Also I’m excited on the multiple master-master data center deployment which should give us even better uptime and faster response times. Also the recently released 2.1.0 contains several features which we’re hoping to use in the future.

Cassandra tradeoffs & community

Cassandra is a great choice if the dataset is larger than what a few single big machines could handle with some other database. Everything smaller can usually be handled with just a single machine, and if you can have only a single machine then there are other databases with more rich feature sets available; as Cassandra has chosen to support linear scalability and master-master deployments across data centers, there are some tradeoffs with its design.

The community is really helpful. I’ve been most active in the Apache hosted Jira which is used to track all Cassandra development. All my bug reports have been answered fast and the actual bugs have been usually fixed for the next revision. The discussion is really helpful and it gives the confidence that the product is under active development.

Walid Darwish Senior Software Engineer at Alcatel-Lucent
"We needed a solution to scale up and down to properly support our small and big customers and their corresponding needs. Other things we like and we couldn't get with Oracle, was the throughput; the amount of data we could push through, and quickly able to get out."
Walid Darwish Senior Software Engineer at Alcatel-Lucent

Alcatel-Lucent is a French company. Our group in particular is a network management group. We provide network management software solutions to ALU customers. When a customer buys ALU’s network hardware, they would then require a software management tool to manage and operate the network, monitor its efficiency, and provide necessary information needed to support informed decision-making.

Cost effective scale

Cassandra’s story started at ALU close to two years ago. Now, I know at least three, or four, other teams inside ALU using it, or at least considering its use. There is a lot of data flowing in a network. ALU customers want to make use of this data. With Cassandra, that’s now possible and it’s scalable. Interactions between teams are now beginning to emerge and this will allow everyone to explore ways to maximize the potential. ALU is trying to build up a solid Cassandra foundation for the whole network management solution.

At this stage we are concerned with network stats collection. ALU’s network management solution initially collected and served the data with Oracle. While it stayed with Oracle for a long time, Oracle just couldn’t scale up at a reasonable cost. We had to look at other solutions that would make our solution cost-appealing to our customer.

During this journey, there’s been a lot of learning, a tremendous amount of information and also a complete mind-shift on how we think of the network data. We really had to switch the way we looked at things before. It was when we realized we needed to change the way we perceive data and legacy approaches that we realized what Cassandra can offer to us in terms of scalability.

Do your homework

ALU did its homework and looked at different options, Cassandra being one of them. We evaluated solutions like: HBase, MongoDB, and other similar NoSQL solutions, and we even looked at Oracle to see what they could offer in that domain.

A lot of things led us to decide to go with Cassandra. We needed a solution to scale up and down to properly support our small and big customers and their corresponding needs. Other things we like and we couldn’t get with Oracle, was the throughput; the amount of data we could push through, and quickly able to get out. Those were key factors. As well, how easy it was for Cassandra to maintain those rivers of data we are getting from the network and how the old unneeded data was aging out and disappearing without us worrying about it.

Another consideration was the uniqueness of our use case because we embed Cassandra as part of our product and then ship it to customers. So, we needed to be able to maintain many different use cases by many different customers with the same exact solution. As such we had to be able to provide a solution that will cater to all of them at the same time, which I think makes our relation to Cassandra different than a lot of other companies. Such flexibility of Cassandra was also a key factor.

Deployment

ALU is using the community version of Cassandra right now, we start with our customers with a three to five nodes cluster. That’s just a starting point; bigger customers, like tier 1 operators, would definitely need more than this, and we are able to change that at a moments notice. The customer has the ability to add in the cluster to the solution; moving away from Oracle and add nodes so they can just simply scale. They can easily say, “I want to scale now”, we add additional nodes, and they will be able to instantly meet their load demands. We also help by giving them tools to figure out when it is necessary to do so.

I think adding Apache Spark could be the next step for ALU; our focus in this stage, is to figure out how to get the data in, and to be able to get it out in the most traditional ways that we and our customers are used to. Because we have this data structured in a different way with Cassandra, we have more ideas on how to get it out, and visualize it.

From SQL to CQL

CQL at first glance, it looks like you can use similar concepts to SQL and relational databases, build tables with primary keys ..etc. In the beginning, we were very disappointed that there is a little support for relational concepts such as joins and referential integrity and we first thought, “how can that be possible!!”. Then, over time, we learned that it’s even better to do it this way, the structure of the data is much tighter; even with performance aside, it’s much better to do it this way. Cassandra has all been a learning exercise for us, we are very happy about the end result, but there’s still a lot of homework to do.

The value Cassandra adds is definitely for ALU and its customers as well. It’s a very shared value. They buy our products, our professional services, because they want their networks to efficiently operate and scale. What we do, and Cassandra helps us with, is we give them the backbone that will make it possible.

 

1 2 3 147