February 28th, 2013

Like all good 1.0 releases IronCount was destined to be made ‘good’er by a 2.0 release.

The package layout was re-factored. This was cosmetic but it makes hacking at the project easier.

The MessageHandler interface was given a new method stop(). Though the message handler interface will remain as stable as possible, stop was needed. The framework attempts to execute stop() whenever a workload is paused or removed. This is useful if a process is batching up writes that are not yet committed to their destination. Stop() can allow you to clean up.

A new feature was added to Workload as well.

Workloads could be added and removed dynamically, however the Java classpath was loaded at start time and could not change. This means that it might have taken a shut down and restart to update a handler or add new ones. With this feature the list of URLs is used by the WorkerThread to construct a URLClassLoader. This construct gives us the ability to Hot Deploy handlers without restarting the entire application.

The other option for doing this considered was the method hadoop uses. Essentially building Fat Jars and shipping them along with your job. I found that problematic over the years.  I felt the URLClassLoading system gives a maven-esc feel. Theoretically if (when 🙂 IronCount catches on big time, users can leave handlers on public web servers, and other users can use them with the URLClassloader. Also across a network managing jars on a set of web servers is trivial, or you could list jars in your existing maven repository.

The next feature that is a big game changer is JMX. I considered adding a web interface to manage everything, but I am tired of web interfaces. When deploying a Workload users used to have to start an instance of WorkloadManager to leverage the API and talk to Zookeeper. IronCount 2.0 exposes the applyWorkload(String) method through JMX. This makes it easy to start and stop workloads.

UUID of nodes running the job.

We also have statistics of messages processes and time spent processing.

I had a requirement to do some real time calculations of city and state from our event logs. IronCount to the rescue! This is a perfect use case for Cassandra counter feature. Read a line and update counters. We apply some batching because there is no reason to send one increment per source row.

Lets look at some results.

Sweet! IronCount 2.0 strikes again! Super easy to build aggregations on real time data. Now, it is easier to manage and deploy with JMX and URL class loading.

An alternative to writing handlers from scratch is using a Domain Specific Language along the lines of HStreaming and Countandra. There is feature mismatch and overlap between IronCount and these three projects, but IronCount focuses on being the lightest framework possible.