December 4th, 2013


Galo Navarro: Software Engineer at Midokura

Christian Hasker: Editor at Planet Cassandra, A DataStax Community Service


TL;DR: Midokura’s Midonet is a network virtualization solution closely integrated with Openstack.  Midonet simplifies network management in the same way that virtualizing computing resources simplifies hardware management.


When looking at candidates for Midonet’s storage they required high availability and certain consistency guarantees.  Originally they used Memcached, which gave them performance but no persistence, and most importantly created a single point of failure, which traditional SQL databases would have not fixed. So they looked into various NoSQL alternatives; running an internal benchmark between Cassandra, Hazlecast, Voldemort, Kyoto Tycoon, and ZooKeeper among others.

To solve their dilemma, Midonet selected Cassandra to power their state management solution. Now powered by Cassandra, Midokura considers one of Midonet’s key strengths is having distributed architecture with no single point of failure.


Hello everyone, I am joined for today’s Apache Cassandra use case by Galo of Midokura. Why don’t you start off, Galo, by telling us what Midokura does and what your role is there?

I am a software engineer for Midokura at the Barcelona office and work on the network controller for Midonet, a network virtualization solution closely integrated with Openstack. It is targeted at service providers and private clouds providing a range of virtualized network services.


Okay, great. As a software engineer over there, are you primarily focused on Cassandra or you’re doing other stuff as well?

I am part of the team responsible for the storage cluster, which involves everything including development and operations for Cassandra as well as other storage systems. I also work on its main client: the network agent deployed on physical hosts that runs distributed simulations of network processes according to a logically centralized view of the network topology and state that is held in that storage cluster.


For those people out there that may not know too much about virtualizing your network, what are the benefits?

Products like Midonet simplify network management in the same way thatvirtualizing computing resources simplifies hardware management.To put this in context, most of us are familiar with going to aservice provider such as Amazon EC2, a hosting service, or our owncompany’s internal IT department to provision virtual resources ondemand.  However, providing network connectivity to all these virtualresources is a complex problem. For example, you may have a physicalmachine hosting VMs that belong to different companies and thereforerequire isolation, separate network segments, and so forth.Traditional solutions were not scalable and complex to manage.Virtualized networks solve those problems, and are able to offeradditional services such as software-based firewalls, load balancers,and so on. One of Midonet’s key strength is a distributedarchitecture with no single point of failure, which is where Cassandracomes in.


Thank you for the explanation. Now, let’s focus on Apache Cassandra. How are you using Apache Cassandra and what made you choose it? Did you look at anything else?

We use Cassandra to store the network state. An example is NATmappings: essentially a way of translating one space of networkaddresses to another, for example on your home router,which creates mappings between the private IPs of your devices to apublic IP facing the Internet. These mappings are typically stored inthe physical device’s memory, in our case it is a distributed virtualdevice potentially running on one of many hosts.


This should give a rough idea of the requirements we had when looking at candidates for our storage: we needed high availability, certain consistency guarantees, and very low latency given that we’re processing network packets.


Originally we used Memcached, which gave us performance but nopersistence, and most importantly created a single point of failure, which traditional SQL databases would have not fixed. So we lookedinto various NoSQL alternatives.


What version of Apache Cassandra you’re running?

1.1.8, on the current stable version. For the next one we’ll migrate to 1.2.


Was that an easy choice to come to Cassandra or did you look and evaluate some other options as well?

We ran an internal benchmark between Cassandra, Hazlecast,Voldemort, Kyoto Tycoon, ZooKeeper and others. We also considered other factors, for example, TTL (time to live) support which meant we could let the storage itself deal with ephemeral state. Also specific to our use case is that the storage cluster is deployed on machines that run in the client’s datacenters, we don’t run them. This made us value stability and simplicity when maintaining or scaling the system.


Cassandra proved to have great performance for our use case, with a good feature set that was actively maintained, and seemed both robust and easy to maintain. All this has proven right so far.


We chose not to use Cassandra to store network topology, which is donein ZooKeeper right now mainly for the reason that we wanted agents to set watchers on certain data so they got notified if the network topology changed. Cassandra didn’t offer this at the time, so we had to send part of the data to ZooKeeper. We’re planning to evaluate options to put that data also into newer versions of Cassandra.


Let’s talk a little bit about how it was picking up Apache Cassandra. How did you find the transition to that technology and any advice you would pass along to other people coming new to it?

As part of the evaluation of candidates we abstracted the client-side code from the storage and reaching feature parity on the storage logic, which in our case was relatively simple. This allowed us to adapt gradually, and separately tune both the client-side code and the database itself as we gained experience with it. This is something I’d recommend as a good principle: migrate incrementally so you can learn in small steps, make mistakes, and understand in practical terms the tradeoffs you have to pay for the benefits you’re getting, etc. For those with a background in traditional databases there is a period of brain rewiring that needs to come from experience. For example, handling certain cases of node failure inside the client library caused problems in our that our agent partly because they involved things that you usually just take for granted dealing with traditional databases.


How’s the Cassandra community in Barcelona? Are you involved and how is the community helped as you’ve learned Apache Cassandra?

The Cassandra community in Barcelona is still a bit of niche. We had a meet-up last month where we gathered around 30 people and everybody was very enthusiastic and curious. There are a few organisations here that could benefit from Cassandra, so I think starting to get together and getting to know each other is definitely going to help us to start collaborating and facilitate adoption. I’d say it’s a good moment for the local community, even if still nascent.



Are there any features that you’re looking forward to in the future with Cassandra?

One of the most interesting to us is triggers, which are out in 2.0 as I understand. We’re considering attempting an implementation that takes all the data which requires update notifications out of ZooKeeper and puts it into Cassandra. We still need to dig into the triggers implementation, how it works, and how that would fit in our architecture. But, that’s an interesting use case for us.


Another area that would be really valuable for us is simplifying all the operational stuff around Cassandra. Mainly for the reason I mentioned about deploying the whole of Midonet in client’s clouds. We can’t count on a dedicated team maintaining and operating the Cassandra cluster, as you would have in more typical deployments.Therefore, for us the ideal would be to treat it pretty much as a commodity database. We install it and it just works. Anything thats implifies upgrades, scaling up/down, maintenance, monitoring, all that range of features would be extremely important for us.