October 29th, 2013


Not too long ago, I was tasked with getting an automated regression test in place for DataStax Enterprise’s bundled version of Apache Solr. After taking a bit of time to explore the various options available, it was decided that SolrMeter would be an excellent fit for the project, except for the fact that it did not have a headless mode of operation and so controlling it from the command line would not be an easy task. After spending a few hours digging into SolrMeter’s code I came to the conclusion that, owing to the excellent architecture of SolrMeter itself, it would not be too difficult of a job to create a patch that would allow it to run in headless mode and be drivable from the command line, which is what we needed for our project. So that is exactly what I did, and in this post I will show you how to get a headless SolrMeter up and running with very little effort. I hope that you will find this useful and possibly even be able to use this in your own automation projects.

The setup

For the examples that follow, I used a single node DataStax Enterprise cluster on EC2. The single and only node of this cluster has been started up as a solr/search node, and I have taken some extra steps to load data into Solr that we will access later with SolrMeter. The steps I took were as follows.

Note: The above steps can also found in the “Running The Solr Demo” section of DataStax’s “Getting Started with Cassandra and DataStax Enterpise” documentation. Here is a link to that section of the docs for additional context: Running the Solr Demo

After having followed the steps above, we can verify that everything went well by having a look at the admin page for the wiki.solr core. Mine was available at the following URL. Yours will of course be different.


From the screenshot below we can see that the wiki.solr core is now available and ready for us to use.

wiki.solr screen shot

So that’s it for the initial setup, let’s now turn our focus towards getting the headless version of SolrMeter up and running.

Patching SolrMeter

I’ve submitted the patch that I created to the SolrMeter project for discussion/review. You can obtain it directly from the issue tracker which is found here: https://code.google.com/p/solrmeter/issues/detail?id=94. Make sure to get the most recent version of the patch: solrmeter-R318-headless-v4.patch. It should apply cleanly to revision 318 of SolrMeter, like so:

After successfully applying the patch, change into the “sources” directory and run “mvn package” like so:

That done, we can now run SolrMeter in GUI mode just as we could with the non-patched version:

Solrmeter in GUI mode

With the patch applied and verification that SolrMeter starts up successfully in GUI mode, it’s time to get things setup to run in headless mode, which just takes a few more steps.

Setting up for Headless Operation

The first thing that we need to do to run in headless mode is get a valid SolrMeter configuration file. We can get that easily by running SolrMeter in GUI mode, editing the settings to point to the correct query and update URLs and to the correct query and update files, and then exporting the settings to a configuration file on the file system. We will use this config file later to start up in headless mode. Here are some screenshots of that process:

Query Settings
Solrmeter Query Settings

Update Settings
Solrmeter Update Settings

Saving the config
Solrmeter Saving Config File

You’ll notice in the screenshots above that we are using a file named queries.txt for the queries file and one named updates.txt for the updates file. Below are the contents of those two files. Note that headless mode does not require this setup. Any of the normal operating modes of SolrMeter will work in headless mode too. Fields, filter, and facet files can all be used in headless mode as well.

Contents of queries.txt and updates.txt

As you can see, we are using some very simple queries and updates in this example for demonstration purposes only. In practice, we use a more involved setup to achieve more useful results. OK, we now have everything we need in place to run SolrMeter in headless mode. Next, we’ll see exactly how to do that.

Running SolrMeter in Headless Mode

Running SolrMeter in headless mode from the command line is easily done by specifying and setting values for two properties: solrmeter.runMode and solrmeter.configurationFile. We’ll set the latter to point to the configuration file that we created earlier. The former, we’ll set to “headless” which will cause SolrMeter to start up in headless mode. So let’s go ahead and start it up now and see what happens!

The startup process appears pretty normal. Usual text gets output to the console as follows:

Take note, however, that SolrMeter is now using a couple of new modules that it didn’t used to: SolrMeterRunModeModule and HeadlessModule. That’s your indication that SolrMeter has found your configuration file and is running correctly in headless mode. Well, that and you will surely notice that the GUI doesn’t appear on the screen!

By default SolrMeter will execute 100 queries and then exit. It should take about 5 minutes to complete with the default of 20 queries per minute.(Both of these settings are configurable so feel free to run 5,000+ queries at 1,000 per minute, for examaple) By default no updates will be performed and no commits or optimizes either.

OK, 5 minutes hence and the process has finished and returned us to the console. We have successfully run SolrMeter in headless mode. But wait! That doesn’t do us much good if we can’t see the results of the operations that have been performed like we could in GUI mode, right? Where are the results?

The headless output directory
Unless the headless.outputDirectory property has been changed in the configuration file, running SolrMeter in headless mode will result in a directory being created named solrmeter-headless in the current working directory. This is where you will find all the statistics that you are used to seeing in GUI mode. Let’s have a look in there now.

If you are familiar with SolrMeter, you will notice that the directory structure mostly mirrors that of the GUI component structure/layout. Most every component from the GUI version of SolrMeter has a counterpart in headless mode that maps conceptually as follows:

Main Window => solrmeter-headless
Query Console => solrmeter-headless/query_console
Query Statistics => solrmeter-headless/statistics/query_statistics

The information that you are used to seeing in GUI mode is available to you on the file system now in headless mode. The information in this directory refreshes continuously as operation results become available. You may have noticed that there is no update_console file. This is because no updates were performed. Let’s update our config file so that updates will be performed along with queries the next time we run SolrMeter in headless mode:

Above are the settings relevant to headless mode that you can adjust.

Edit my-solrmeter-config.smc.xml and change performUpdateOperations from “false” to “true”. Now, we run SolrMeter again in headless mode just like we did before. Another 5 minutes or so later, we inspect the solrmeter-headless directory again:

Largely the same as before but this time you’ll notice that there are two new files present: update_console and commit_console. These two new files contain the results of those operations (updates and commits).

Everything seems to be operating as it should be. Let’s take a look now at the contents of some of the files in the output directory to see what information is available to us.

As you can see from above, the headless mode of operation attempts to make the same information that is available to you in GUI mode available to you in headless mode too. Excellent!

Final Words

Well, that about covers it, but there is just one last operation that we haven’t talked about yet. How do you perform an optimize in headless mode, like you do in GUI mode? In GUI mode, when you want to perform an optimize operation you click the “optimize now” button. In headless mode there are no buttons to push but we can still achieve our goal of ‘on-demand’ optimize operations as follows. When in headless mode, SolrMeter will listen for changes to the “directives” directory. This is our way of communicating with SolrMeter. When you want to perform an optimize operation in headless mode create a file in that directory named “optimize”. Example:

mallen-> touch directives/optimize

SolrMeter will then schedule on optimize operation to be performed and delete that file. You can repeat this process as often as you’d like to. Take note that after an optimize operation has been performed you will then see a new file created inside the solrmeter-headless directory named (you guessed it!) optimize_console, which is the counterpart of the “Optimze Console” panel from the GUI version.

I hope that you have found the information in this post useful and/or interesting. We have been using the headless mode of SolrMeter for a few months now with good results in our test harness here at DataStax. On a personal note, I enjoyed working on the patch for SolrMeter and I want to thank the members of that project for providing such a useful tool and a well designed system, which ultimately made my job easier!