April 30th, 2013

We managed to reduce our test time from over 30 minutes to now running in just over 2. It also saved us massively in developer productivity as well as the un-quantified savings in developer sanity.

As a company who processes billions of events a month and stores large amounts of critical data for large-scale travel companies, our integration and acceptance testing is the life and soul for quality deployments. We deploy several times a day providing new features and patches without a single minute of downtime. However as everyone knows the key to good continuous integration and testing is fast feedback as the faster you get to identify the problem the faster you can fix it.

When we first started, our tests were quick. However as the product grew the amount of tests increased resulting in the CI build taking longer and longer. This cause was simple. Our integration and acceptance tests were testing how our code integrated with an embedded Cassandra. During each test we created a keyspace, column families and imported test data required for the test. Isolation was key and this happened over and over again. As you can imagine this resulted in massive disk IO for each test.

The solution we came up was to run the embedded Cassandra using a 1GB Ramdisk. We simply changed the embedded cassandra.yaml data directories to point to the newly mounted Ramdisk. (You could also use /dev/shm on Ubuntu but we prefer keeping control af the amount of RAM allocated to the ramdisk)

Create Ramdisk

mkdir -p /tmp/ramdisk
sudo mount -t tmpfs -o size=1024M tmpfs /tmp/ramdisk/


# Data stored in the target directory for this project
data_file_directories: /tmp/ramdisk/embeddedCassandra/data
commitlog_directory: /tmp/ramdisk/embeddedCassandra/commitlog
saved_caches_directory: /tmp/ramdisk/embeddedCassandra/saved_caches

This reduced the IO on the disk to zero and fundamentally increased the speed of the tests. We also noticed another benefit from the CI build duration decreasing, it was now far easier to diagnose the cause of build failures faster as there was a reduction in change sets per CI builds hence less noise.

The great thing about this is now we are seeing developers adopting the same approach and they are now running the full suite of tests locally and more often. This is also having a knock on effect as we are seeing that our CI server rarely fails enabling us to push to production more often per day.

For details of setting up a Ramdisk and why see

For running an embedded Cassandra for testing see