+ =

Splunk is a popular, enterprise-grade tool for IT infrastructure monitoring. If you haven’t come across it before, I’d take 5 minutes to check out this introductory video. In a nutshell, Splunk indexes any form of time-series data, and provides an interface for searching, analysing and reporting on these data.

The enterprise version of Splunk can be used to provide real-time info on all aspects of the operation of entire data clusters, and as you can imagine it has an enterprise-sized price tag to match. However, the good news is that there’s also a version which is free to use if you’re indexing less than 500MB of data a day and don’t require some of the advanced functions such as real-time alerts. This makes Splunk a great tool for small shops and early-stage startups to use for web-app monitoring and performance analysis.

Splunk can consume pretty much any logs your server can produce: syslog, database, Apache, mail, log4j, top – you name it, if it’s a plain-text time-stamped log file, Splunk will index it. However, in this introductory post, I’m going to keep things simple and describe how to get Splunk up and running to provide real-time monitoring of the log4j logs produced by a Grails application.

Splunk is fairly lightweight, so can usually be run on the same server as your Grails app. However, I’m going to take a slightly different approach and run Splunk on my development machine, to avoid having to install any new software on my app server.

First thing is to download and install Splunk. Head over to this page, and choose the appropriate distribution for your machine. Installation is painless and only takes 5 minutes, but if you run into trouble, check out the help pages on the Splunk site. That’s another great thing about Splunk: the documentation is really, really good. Extremely comprehensive and well-written. They even run a Q&A forum using the StackOverflow API.

Once you’re installed, start up Splunk by running

/opt/splunk/bin/splunk start --accept-license

(on Ubuntu/Debian – your system may differ). Load up the web front-end by going to http://localhost:8000/, and log in as admin (password “changeme”).

Okay, now Splunk is up and running we need some data to feed it. The simplest way is to point Splunk at a log file and tell it to tail it. If you’re running Splunk on the same machine as you’re app, this is simple. If not, we need to do an extra step to get the log file onto our development machine. I decided to go with tried-and-trusted cron and rsync:

* * * * * rsync -e ssh -avlq {remote-user}@{remote-host}:{absolute-path-to-remote-log-file} {absolute-path-to-local-log-file} >/dev/null 2>&1

(replace text in curly brackets as appropriate). Thus my application log will be sync’d to my development machine every minute. Remember, this method will only work if you have your development machine registered as an allowed host on your production server.

Now, go to http://localhost:8000/manager/search/data/inputs/monitor and click ‘New’ to add your log file as a new input. Here are the options you want:

• Source: {absolute-path-to-local-log-file}
• Host: {remote-host} // but fine to leave as the default
• Source type: Automatic // Splunk will auto-detect the log4j format
• Index: default
• Advanced options: Follow tail: checked

Click ‘Save’, and Splunk will now begin to index the log file, and monitor it for changes.

Now that the set-up is complete, the fun begins. Head over to http://localhost:8000/app/search/dashboard – this is where you can get a visualization of the contents of your log file, filter the contents, search for events, and build reports. The guys at Splunk do a better job of demonstrating these features than I could, so at this point I’ll direct you to one of the many demo videos on the Splunk site. Alternatively, check out the search tutorial in the docs.

Hopefully I’ve showed you how easy it is to set up Splunk and get it analysing the logs of your running Grails app – without having to make a single change to your production server!