Events of Interest from a Splunk Admin perspective

As our deployment has grown from basically myself performing administrative duties to adding a second body as well as running into a couple other things we've wanted increased visibility when certain activities have taken place. Over time we've merged these into a rather ugly query. It has proved itself a time or three though so figured I'd share. In putting something like this into place you need to figure out a few things. How frequently it will run, who it goes to, etc. You might find you want certain aspects to run more frequently than others which would require more than one query. YMMV

Splunk DateParserVerbose logs - Part 2

In part 1 of this subject we talked about what Splunk's DateParserVerbose internal logs are and I gave an example query that at its heart attempts to rollup and summarize timestamp related issues. In this post I'll present a query for taking the sourcetypes Splunk is having issues with from a timestamp perspective and display the relevant props configs. What we've done is thrown both queries into the same dashboard to make things easier to work though. I should note a couple things here. The first is the foreach command is only available in Splunk 6 (I believe). The second is the REST endpoint I'm getting the config data from is likely only available in 6.

With that out of the way here is the query:

Detecting OpenSSL version data in Splunk

I won't go into the HeartBleed details as you likely already know them. From a Splunk perspective there are any number of ways to try to get your arms around this issue but are highly dependent on the types of data you are collecting. That said, if you are using the Splunk Linux TA and have the package script enabled and/or the Windows TAs and have the InstalledApps_Windows script turned on you could use the following queries to extract the OpenSSL version. You could also combine the queries but for the purposes of posting them here that would make it harder to read /shrug. Obviously adjust based on changes you've made (ie sourcetype)

Linux
sourcetype=package  | multikv  | search NAME=openssl  | dedup host ARCH  | eval HBconcern = case(match(VERSION,"(^0\.\d\.\d|^1\.0\.0)"), "Too Low", match(VERSION,"^A"), "HP (Not familiar)", match(VERSION,"^1\.0\.1[a-f]"), "Potentially Susceptible", match(VERSION,"^1\.0\.1[g-z]"), "Patched", match(VERSION,"^1\.0\.2-beta"), "Potentially Susceptible", 1=1, "fixme") | table host NAME VENDOR GROUP VERSION ARCH Hbconcern | sort host

Windows

sourcetype=InstalledApps_Windows DisplayName=openssl | rex  "DisplayName=OpenSSL\s+(?<VERSION>\S+)\s+\((?<ARCH>[^\)]+)"| dedup host ARCH | eval HBconcern = case(match(VERSION,"(^0\.\d\.\d|^1\.0\.0)"), "Too Low", match(VERSION,"^A"), "HP (Not familiar)", match(VERSION,"^1\.0\.1[a-f]"), "Potentially Susceptible", match(VERSION,"^1\.0\.1[g-z]"), "Patched", match(VERSION,"^1\.0\.2-beta"), "Potentially Susceptible", 1=1, "fixme") | table host VERSION ARCH HBconcern | sort host

What I'm not sure on is the regex for the 1.0.2-beta as I haven't actually seen that version installed. I'm guessing it shows up like that. 

Splunk, timestamps, and the DateParserVerbose internal logs - Part 1

Splunk is a pretty powerful piece of software. There is the obvious search and analytic capabilities it has but there is some robustness under the covers as well. One of those under-the-cover capabilities is detecting and understanding timestamp data. Its the sort of thing that as users of the software we simply accept and generally speaking don't spend a whole lot of time thinking about. From an admin perspective as you start to put some effort into understanding your deployment and making sure things are working correctly one of the items to look at is the DateParserVerbose logs. Why you ask? I've recently had to deal with some timstamp issues. These internal logs generally document problems related to timestamp extraction and can tell you if, for example, there are logs being dropped for a variety of timestamp related reasons. Dropped events are certainly worthy of some of your time! What about logs that aren't being dropped but for one reason or another Splunk is assigning a timestamp that isn't correct? In this writeup I will share a query you can use to bring these sorts of events to the surface and distill some quick understanding.

Splunk - troubleshooting remote agents with the phonehome logs

An issue popped up the other day that was pretty interesting (from a Splunk admin perspective) so figured I would share. This will likely be pretty long but hopefully someone will benefit. We had a number of servers with Splunk universal forwarders stop sending logs but in doing a spot check on their firewalls the server owner noticed traffic still going to our Splunk infrastructure backend. What had happened? The answer to the question lies in the UF phone home logs - do you know where they are and how to read them?