Splunk admin tasks after you start getting data in...

I had the rather unique privilege to post a 3 part blog series on Splunk's official site recently. The focus was on some administration tasks Splunk admins should work into their routine. There is a level of assumption when users search in Splunk - these hosts are really these hosts and events that are observed within a time range really happened then. The series talks through a couple methodologies to validate those assumptions

• Part 1 - Validating host field values: link
• Part 2 - Validating agent host's system time: link
• Part 3 - Getting a feel for data ingestion latency: link

Splunk Apps: Forwarder Health

It is long past time I actually wrote a few posts on the Splunk apps I've created. Woke up far too early for a Saturday morning and in an effort to avoid anything around the house I will rationalize this as productivity at a general level and feel I've accomplished much! Who knows - it might be of value to my ... ones ... of readers! =)

Actually it was VERY cool to have a guy come up after my presentation at the 2014 Splunk user conference and mention having read my blog while working with ArcSight and now while working with Splunk (thanks Joe!).

Forwarder Health

So our environment has currently some 2,200+ forwarders which is certainly not the largest environment out there but is likely much larger than the average. While there are apps like Splunk on Splunk and Fire Brigade to help identify issues with your indexers and search heads there wasn't something that helps identify issues with forwarders. Admittedly this is a hefty task as there are innumerable issues a forwarder can have. I wondered though if there was a way to generically detect if an agent was having issues. The sage like advice from the Verizon breach reports bubbled up in my mind - start by looking at the size of the haystacks. What if you were to compare the number of internal logs a forwarder was generating and compare it to the average? A couple hours later the bones of the app were in place.

Splunk DateParserVerbose logs - Part 2

In part 1 of this subject we talked about what Splunk's DateParserVerbose internal logs are and I gave an example query that at its heart attempts to rollup and summarize timestamp related issues. In this post I'll present a query for taking the sourcetypes Splunk is having issues with from a timestamp perspective and display the relevant props configs. What we've done is thrown both queries into the same dashboard to make things easier to work though. I should note a couple things here. The first is the foreach command is only available in Splunk 6 (I believe). The second is the REST endpoint I'm getting the config data from is likely only available in 6.

With that out of the way here is the query:

Detecting OpenSSL version data in Splunk

I won't go into the HeartBleed details as you likely already know them. From a Splunk perspective there are any number of ways to try to get your arms around this issue but are highly dependent on the types of data you are collecting. That said, if you are using the Splunk Linux TA and have the package script enabled and/or the Windows TAs and have the InstalledApps_Windows script turned on you could use the following queries to extract the OpenSSL version. You could also combine the queries but for the purposes of posting them here that would make it harder to read /shrug. Obviously adjust based on changes you've made (ie sourcetype)

Linux
sourcetype=package  | multikv  | search NAME=openssl  | dedup host ARCH  | eval HBconcern = case(match(VERSION,"(^0\.\d\.\d|^1\.0\.0)"), "Too Low", match(VERSION,"^A"), "HP (Not familiar)", match(VERSION,"^1\.0\.1[a-f]"), "Potentially Susceptible", match(VERSION,"^1\.0\.1[g-z]"), "Patched", match(VERSION,"^1\.0\.2-beta"), "Potentially Susceptible", 1=1, "fixme") | table host NAME VENDOR GROUP VERSION ARCH Hbconcern | sort host

Windows

sourcetype=InstalledApps_Windows DisplayName=openssl | rex  "DisplayName=OpenSSL\s+(?<VERSION>\S+)\s+\((?<ARCH>[^\)]+)"| dedup host ARCH | eval HBconcern = case(match(VERSION,"(^0\.\d\.\d|^1\.0\.0)"), "Too Low", match(VERSION,"^A"), "HP (Not familiar)", match(VERSION,"^1\.0\.1[a-f]"), "Potentially Susceptible", match(VERSION,"^1\.0\.1[g-z]"), "Patched", match(VERSION,"^1\.0\.2-beta"), "Potentially Susceptible", 1=1, "fixme") | table host VERSION ARCH HBconcern | sort host

What I'm not sure on is the regex for the 1.0.2-beta as I haven't actually seen that version installed. I'm guessing it shows up like that. 

Splunk, timestamps, and the DateParserVerbose internal logs - Part 1

Splunk is a pretty powerful piece of software. There is the obvious search and analytic capabilities it has but there is some robustness under the covers as well. One of those under-the-cover capabilities is detecting and understanding timestamp data. Its the sort of thing that as users of the software we simply accept and generally speaking don't spend a whole lot of time thinking about. From an admin perspective as you start to put some effort into understanding your deployment and making sure things are working correctly one of the items to look at is the DateParserVerbose logs. Why you ask? I've recently had to deal with some timstamp issues. These internal logs generally document problems related to timestamp extraction and can tell you if, for example, there are logs being dropped for a variety of timestamp related reasons. Dropped events are certainly worthy of some of your time! What about logs that aren't being dropped but for one reason or another Splunk is assigning a timestamp that isn't correct? In this writeup I will share a query you can use to bring these sorts of events to the surface and distill some quick understanding.

Splunk - troubleshooting remote agents with the phonehome logs

An issue popped up the other day that was pretty interesting (from a Splunk admin perspective) so figured I would share. This will likely be pretty long but hopefully someone will benefit. We had a number of servers with Splunk universal forwarders stop sending logs but in doing a spot check on their firewalls the server owner noticed traffic still going to our Splunk infrastructure backend. What had happened? The answer to the question lies in the UF phone home logs - do you know where they are and how to read them?

Having Splunk wrap long fields

Splunk generally does a good job of autosizing column widths in dashboard panels. One of the things that sort of bugs me though is when you want to display just a few fields in a Splunk dashboard panel but the length of one or two values in a field push downstream fields off to the side requiring you to do some horizontal scrolling. I ran across a relatively old post on the Splunk boards addressing this issue (here).

I figure though if you are going to pass something via a macro why limit yourself to a static character limit. Paste the following into an appropriate macros.conf file. The first argument is the field you want to wrap (eg., email message string) and the second is the string length you want the string to be/split on.

[line_breaker(2)]

args = field,len

definition = rex max_match=100 field="$field$" "(?<split__regex>.{0,$len$}(?:\s|$)|[^\s]+)" | rename split__regex as "$field$"

Splunk trick to display a one to many relationship in a table with granular and aggregate values

Another oil change and another blog post. Good thing there isn’t always a relationship between those two things or my cars wouldn’t be running. Figured I would spend a minute talking about one of my new favorite Splunk tricks. I think I ran across this in the Implementing Splunk book but then could never find again. Was pleased that it showed up in the Advanced Searching and Reporting class I took at .conf2013 (great class BTW). The trick relates to formatting data and covers a variety of use cases mostly related to displaying a one to many relationship at various levels of granularity. 

A change in log format for Splunk UF 6.x relative to tracking apps using the Deployment Server

I realized two things yesterday as I was troubleshooting various Splunk things. The first relates to having multiple input configs sent to a centralized syslog server. The second relates to changes to the internal 6.x UF logs as it relates to tracking apps that have been installed or removed.

A search on the Splunk mug is wrong!!

For those that haven't seen it the Splunk mug is a neat little piece of practical schwag that contains queries for things ranging from finding happiness to finding Waldo and even tracking a zombie infestation. However! I've discovered an issue with one of the searches. 

The first thing to understand, if you don't already, is that the asterisk is a wildcard in Splunk. A neat little trick is that when you combine it with a field as in field=* your search will return events where that field contains a value. This makes it a great little inclusive search and potentially you won't have to use a usenull=f as part of your chart or timechart further in your search for filtering out events where the field isn't populated. 

I want more time to play!

I find myself in a somewhat strange place today where because I'm going to be at the Splunk conference next week I don't have much scheduled that needs to be done (or staged to be done this weekend). This reminds me of a line that has come up a few times as we've been going through the interview and candidate selection process for two open slots we have in the office. We have all been working way too many hours and want some 'free time' back in our normal routine. I'm not talking about a mental health break or time away from the office as much as having a pocket or two of time where we can explore/investigate/work on little side projects/quality-of-life-things that need to be done. They, generally speaking, aren't hard or long things to do but get sidelined because of higher priorities. 

So I'm monkeying around with a few things in Splunk and two rabbit holes later come up with a query that quite frankly doesn't return a whole lot of hits for me over the last month. What it DOES show is a server that wasn't able to install some config packages I was pushing from my deployment server.

index=_internal source=*metrics.log component="DeploymentMetrics" status="failed" | stats max(_time) as time by hostname event scName appName fqname | convert ctime(time)

This event is created on your deployment server. Not sure what fqname stands for exactly but in my case it was showing me the path the server was trying to install the app to (fully qualified path name is where my mind goes but doesn't fit the data). scName is likely server class name and appName is obviously the app itself - both are references to your serverclass.conf file contents. With over 1k agents deployed the fact that this found issues with only 1 server is pretty cool I suppose. Will likely bake this into the app I'll never create re: first paragraph =)

Gearing up for Splunk .conf13

We had the Columbus Splunk user group meeting today which got me looking forward to Splunk's annual user group conference. Of course as I think that I reminded myself that instead of writing this I should be working on my own preso for the conference so will keep this short. I find it easier to make an argument for securing travel and training funds when you get a discount by speaking. My talk will be around how we went from a 350GB license to 2TB in less than a year and the associated growing pains/lessons learned. Hopefully folks will find a nugget or two of value. 

For those on the fence I would pull the trigger. Lots of information to walk away with that will help you plan, adjust, tweak, create, and simply expand your vision of what's possible. Some of that won't come from the sessions as much as talking with other folks and seeing how they are using the tool. Wondering what the free gift for attendance might be this year. Last year it was a zip up Splunk hoodie that was great - primarily because the conference area was chilly! 

To circle back to an issue raised at the local user group meeting. While I'm full time on our Splunk effort (and in need an extra body) many folks are Splunk admins plus wear any number of other hats. There would probably be high interest in a session devoted to running a Splunk environment with a skeleton crew as it were. I'll try to put some thought into that.

Anyone have any thoughts on that or tips to share?

Solve for 80% - find logs needing work in Splunk

There are a couple of sayings, maxims if you will, that I try to keep in the back of my mind as I do things

• Most times coincidence is God acting anonymously
• Activity != Accomplishment
• Effectiveness and efficiency are two different concepts
• Solve for 80%

Some queries related to Splunk administration/Deployment Server

Once again time flys /sigh. At times I wish there was more hours in the day but that would probably just translate into more hours working. Hopefully will get over the bubble soon (yeah right). At any rate we had our first Columbus Splunk user group the other day. Was neat to see others in the local area and talk Splunk...at least in as much as we could. The location was a bit noisy. Figured I'd share here a few of the queries I put together in a slide deck related to using (or mostly administrating) Splunk's Deployment Server. I guess they aren't specifically related to the DS as much as general Universal Forwarder (local Splunk agent) health which you can control with the DS

Back in the saddle...and using Splunk!

Been a long time since writing and lots has changed! New house, cataract surgery (at 39!), new job, developing and implementing the log management and event correlation program at a large edu, etc. On that front a large university is very different than a large corporation. Take all of those issues you have with decentralization and multiply that by about 50. I debated between believable and hyperbole there and ended up somewhere in the middle I guess.

One of the bigger changes is using Splunk for the first time. At some point I should write some comparisons between ArcSight, Symantec MSSP, and Splunk. Granted it has been a few years since I’ve used ArcSight’s Enterprise Security Manager (ESM) or Logger for that matter though I remember ESM fondly and often wistfully. It still amazes me when I hear people say things like ArcSight is too hard. I maintain that for most of those cases they somehow had the idea (or were told by the sales people) that it was some magical plug and play cross between a one armed bandit and magic 8-ball. I also hear in the latest update to Symantec’s MSSP they have implemented a number of the dozens of changes I suggested (am sure others as well!) and it is now much better than what it was 6 months ago.

If there is one piece of advice I could give it would be install Splunk today. You may not use it a ton in an active way but at the very least I wish I had had it for all of those times I had large Excel spreadsheets to massage in an effort of extracting value.

Welp as was the case when I first started writing this was written while waiting for the oil to be changed in one of my vehicles. That is complete and this sort of went in a different direction than planned. In my next post I plan to talk about a few ways I’ve found in Splunk to look for fields that don’t exist. As an aside it is somewhat interesting to think about writing Splunk stuffs in somewhat of the same way I used to write about ArcSight stuffs. The main driver back then was there wasn’t much out there at all for ArcSight and LM/SIEM in general and the ArcSight forums were gated to the general public. That isn’t the case with Splunk (gated community) and sadly the act of running the Splunk implementation and general ho-ha leaves me far less time to actually DO something in Splunk than I’d like.

Starting to Splunk!

Well I’m off to the Splunk conference.  Having only started using Splunk just over a month ago I can say there is quite a lot to digest and frankly I’m feeling a little overwhelmed. The challenge is starting from step 0 with a good bit of unstructured data knowing in the next year or so that will grow to an estimated 700GB/day. Part of the challenge is not knowing the full implication of choosing different methods to actually do things in splunk, like field extraction, in a way that doesn’t artificially limit or cause issues down the road. This is all while developing a program to handle it all toward multiple ends. In some respects I’m going from using an MSS to being an MSS. New job + new tool + new house (that we are doing renovation work on) = good times. And just for kicks I'm building, and have talked a few others into, building a plywood canoe that we can race each other in. 

At any rate I’m excited to be going and hope to accelerate the learning curve dramatically. If anyone has any Splunk tips I’d be interested in em!