Splunk, timestamps, and the DateParserVerbose internal logs - Part 1

Splunk is a pretty powerful piece of software. There is the obvious search and analytic capabilities it has but there is some robustness under the covers as well. One of those under-the-cover capabilities is detecting and understanding timestamp data. Its the sort of thing that as users of the software we simply accept and generally speaking don't spend a whole lot of time thinking about. From an admin perspective as you start to put some effort into understanding your deployment and making sure things are working correctly one of the items to look at is the DateParserVerbose logs. Why you ask? I've recently had to deal with some timstamp issues. These internal logs generally document problems related to timestamp extraction and can tell you if, for example, there are logs being dropped for a variety of timestamp related reasons. Dropped events are certainly worthy of some of your time! What about logs that aren't being dropped but for one reason or another Splunk is assigning a timestamp that isn't correct? In this writeup I will share a query you can use to bring these sorts of events to the surface and distill some quick understanding.

A change in log format for Splunk UF 6.x relative to tracking apps using the Deployment Server

I realized two things yesterday as I was troubleshooting various Splunk things. The first relates to having multiple input configs sent to a centralized syslog server. The second relates to changes to the internal 6.x UF logs as it relates to tracking apps that have been installed or removed.

Some queries related to Splunk administration/Deployment Server

Once again time flys /sigh. At times I wish there was more hours in the day but that would probably just translate into more hours working. Hopefully will get over the bubble soon (yeah right). At any rate we had our first Columbus Splunk user group the other day. Was neat to see others in the local area and talk Splunk...at least in as much as we could. The location was a bit noisy. Figured I'd share here a few of the queries I put together in a slide deck related to using (or mostly administrating) Splunk's Deployment Server. I guess they aren't specifically related to the DS as much as general Universal Forwarder (local Splunk agent) health which you can control with the DS

ArcSight's PartitionInfo.log

Move over James Patterson – Kerry “the database queen” has posted the third installment of her series on reviewing the partitioninfo.log file. Its a real page turner…well if you printed it out on multiple pages. This is a file you should be generating at least twice a month as part of a backup process even if you don’t review it (KB article 572 if you need instructions). Kerry has been the proud recipient of several “coolest chick in the world” minutes given out by me based on her decisive, insightful, and quick treatment of database related trouble tickets. She would probably score 6+ thumbs up on the Homer Simpson thumbs up scale. Actually my impression of the database support team is that they are a pretty knowledgeable lot. If you are like me and don’t know what to look for in this file Kerry’s series is good stuff.

To get to her blog go to the ArcSight community site and either go to the Browse button at the top and select blog or there is a series of buttons on the middle right side. You do regularly check the ArcSight Protect 724 community site don’t you?