Splunk, timestamps, and the DateParserVerbose internal logs - Part 1

Splunk is a pretty powerful piece of software. There is the obvious search and analytic capabilities it has but there is some robustness under the covers as well. One of those under-the-cover capabilities is detecting and understanding timestamp data. Its the sort of thing that as users of the software we simply accept and generally speaking don't spend a whole lot of time thinking about. From an admin perspective as you start to put some effort into understanding your deployment and making sure things are working correctly one of the items to look at is the DateParserVerbose logs. Why you ask? I've recently had to deal with some timstamp issues. These internal logs generally document problems related to timestamp extraction and can tell you if, for example, there are logs being dropped for a variety of timestamp related reasons. Dropped events are certainly worthy of some of your time! What about logs that aren't being dropped but for one reason or another Splunk is assigning a timestamp that isn't correct? In this writeup I will share a query you can use to bring these sorts of events to the surface and distill some quick understanding.

Splunk - troubleshooting remote agents with the phonehome logs

An issue popped up the other day that was pretty interesting (from a Splunk admin perspective) so figured I would share. This will likely be pretty long but hopefully someone will benefit. We had a number of servers with Splunk universal forwarders stop sending logs but in doing a spot check on their firewalls the server owner noticed traffic still going to our Splunk infrastructure backend. What had happened? The answer to the question lies in the UF phone home logs - do you know where they are and how to read them?

Having Splunk wrap long fields

Splunk generally does a good job of autosizing column widths in dashboard panels. One of the things that sort of bugs me though is when you want to display just a few fields in a Splunk dashboard panel but the length of one or two values in a field push downstream fields off to the side requiring you to do some horizontal scrolling. I ran across a relatively old post on the Splunk boards addressing this issue (here).

I figure though if you are going to pass something via a macro why limit yourself to a static character limit. Paste the following into an appropriate macros.conf file. The first argument is the field you want to wrap (eg., email message string) and the second is the string length you want the string to be/split on.

[line_breaker(2)]

args = field,len

definition = rex max_match=100 field="$field$" "(?<split__regex>.{0,$len$}(?:\s|$)|[^\s]+)" | rename split__regex as "$field$"

Splunk trick to display a one to many relationship in a table with granular and aggregate values

Another oil change and another blog post. Good thing there isn’t always a relationship between those two things or my cars wouldn’t be running. Figured I would spend a minute talking about one of my new favorite Splunk tricks. I think I ran across this in the Implementing Splunk book but then could never find again. Was pleased that it showed up in the Advanced Searching and Reporting class I took at .conf2013 (great class BTW). The trick relates to formatting data and covers a variety of use cases mostly related to displaying a one to many relationship at various levels of granularity. 

A change in log format for Splunk UF 6.x relative to tracking apps using the Deployment Server

I realized two things yesterday as I was troubleshooting various Splunk things. The first relates to having multiple input configs sent to a centralized syslog server. The second relates to changes to the internal 6.x UF logs as it relates to tracking apps that have been installed or removed.