Splunk, timestamps, and the DateParserVerbose internal logs - Part 1

Splunk is a pretty powerful piece of software. There is the obvious search and analytic capabilities it has but there is some robustness under the covers as well. One of those under-the-cover capabilities is detecting and understanding timestamp data. Its the sort of thing that as users of the software we simply accept and generally speaking don't spend a whole lot of time thinking about. From an admin perspective as you start to put some effort into understanding your deployment and making sure things are working correctly one of the items to look at is the DateParserVerbose logs. Why you ask? I've recently had to deal with some timstamp issues. These internal logs generally document problems related to timestamp extraction and can tell you if, for example, there are logs being dropped for a variety of timestamp related reasons. Dropped events are certainly worthy of some of your time! What about logs that aren't being dropped but for one reason or another Splunk is assigning a timestamp that isn't correct? In this writeup I will share a query you can use to bring these sorts of events to the surface and distill some quick understanding.

The unrecognized APT story....

2 hobbits sneaking into Mordor.


Somehow I don't see me using that analogy in the executive boardroom though.

Playing Sudoku in your metrics

This has been percolating in my head for a bit so figured I might as well get it out =). One of the best apps for the iPhone is Sudoku2 by fingerarts. If you aren’t familiar with the Sudoku here is a screenshot.

Each of the smaller boxes needs to have just one of the numbers 1 – 9 while each line vertical and horizontal line can also only have just one of the numbers 1 – 9. It really becomes a process of elimination based on the number you are trying to solve for and where it can go. For example, when I took the screenshot I was solving for 1s. Take a look at the bottom left box. Based on the intersection of other 1s, the 1s in this box have to be in the first vertical column although it isn’t known which of the two it might be (yellow cirlce). Because the ones are in that column though if you look at the top left box there is only one open box where the 1 could go (black 1). By filling that in there is only one open square in the top right square (red 1). And on you go through all the numbers.

There was always a tickle in the back of my mind that there was a security analogy with Sudoku but it didn’t really hit me until a few months ago when I was trying to figure out a way to create a metric-y type view of some virus and incident data.