What I meant to say was...

Well for good or ill I flew through my presentation at the Splunk conference lol. The upshot is I think it went ok. The downside is there are a few things I didn't say that I had planned to. Guess I'll take the opportunity now to cover a few....

A search on the Splunk mug is wrong!!

For those that haven't seen it the Splunk mug is a neat little piece of practical schwag that contains queries for things ranging from finding happiness to finding Waldo and even tracking a zombie infestation. However! I've discovered an issue with one of the searches. 

The first thing to understand, if you don't already, is that the asterisk is a wildcard in Splunk. A neat little trick is that when you combine it with a field as in field=* your search will return events where that field contains a value. This makes it a great little inclusive search and potentially you won't have to use a usenull=f as part of your chart or timechart further in your search for filtering out events where the field isn't populated. 

I want more time to play!

I find myself in a somewhat strange place today where because I'm going to be at the Splunk conference next week I don't have much scheduled that needs to be done (or staged to be done this weekend). This reminds me of a line that has come up a few times as we've been going through the interview and candidate selection process for two open slots we have in the office. We have all been working way too many hours and want some 'free time' back in our normal routine. I'm not talking about a mental health break or time away from the office as much as having a pocket or two of time where we can explore/investigate/work on little side projects/quality-of-life-things that need to be done. They, generally speaking, aren't hard or long things to do but get sidelined because of higher priorities. 

So I'm monkeying around with a few things in Splunk and two rabbit holes later come up with a query that quite frankly doesn't return a whole lot of hits for me over the last month. What it DOES show is a server that wasn't able to install some config packages I was pushing from my deployment server.

index=_internal source=*metrics.log component="DeploymentMetrics" status="failed" | stats max(_time) as time by hostname event scName appName fqname | convert ctime(time)

This event is created on your deployment server. Not sure what fqname stands for exactly but in my case it was showing me the path the server was trying to install the app to (fully qualified path name is where my mind goes but doesn't fit the data). scName is likely server class name and appName is obviously the app itself - both are references to your serverclass.conf file contents. With over 1k agents deployed the fact that this found issues with only 1 server is pretty cool I suppose. Will likely bake this into the app I'll never create re: first paragraph =)

Gearing up for Splunk .conf13

We had the Columbus Splunk user group meeting today which got me looking forward to Splunk's annual user group conference. Of course as I think that I reminded myself that instead of writing this I should be working on my own preso for the conference so will keep this short. I find it easier to make an argument for securing travel and training funds when you get a discount by speaking. My talk will be around how we went from a 350GB license to 2TB in less than a year and the associated growing pains/lessons learned. Hopefully folks will find a nugget or two of value. 

For those on the fence I would pull the trigger. Lots of information to walk away with that will help you plan, adjust, tweak, create, and simply expand your vision of what's possible. Some of that won't come from the sessions as much as talking with other folks and seeing how they are using the tool. Wondering what the free gift for attendance might be this year. Last year it was a zip up Splunk hoodie that was great - primarily because the conference area was chilly! 

To circle back to an issue raised at the local user group meeting. While I'm full time on our Splunk effort (and in need an extra body) many folks are Splunk admins plus wear any number of other hats. There would probably be high interest in a session devoted to running a Splunk environment with a skeleton crew as it were. I'll try to put some thought into that.

Anyone have any thoughts on that or tips to share?

Solve for 80% - find logs needing work in Splunk

There are a couple of sayings, maxims if you will, that I try to keep in the back of my mind as I do things

• Most times coincidence is God acting anonymously
• Activity != Accomplishment
• Effectiveness and efficiency are two different concepts
• Solve for 80%