Having Splunk wrap long fields

Splunk generally does a good job of autosizing column widths in dashboard panels. One of the things that sort of bugs me though is when you want to display just a few fields in a Splunk dashboard panel but the length of one or two values in a field push downstream fields off to the side requiring you to do some horizontal scrolling. I ran across a relatively old post on the Splunk boards addressing this issue (here).

I figure though if you are going to pass something via a macro why limit yourself to a static character limit. Paste the following into an appropriate macros.conf file. The first argument is the field you want to wrap (eg., email message string) and the second is the string length you want the string to be/split on.

[line_breaker(2)]

args = field,len

definition = rex max_match=100 field="$field$" "(?<split__regex>.{0,$len$}(?:\s|$)|[^\s]+)" | rename split__regex as "$field$"

Splunk trick to display a one to many relationship in a table with granular and aggregate values

Another oil change and another blog post. Good thing there isn’t always a relationship between those two things or my cars wouldn’t be running. Figured I would spend a minute talking about one of my new favorite Splunk tricks. I think I ran across this in the Implementing Splunk book but then could never find again. Was pleased that it showed up in the Advanced Searching and Reporting class I took at .conf2013 (great class BTW). The trick relates to formatting data and covers a variety of use cases mostly related to displaying a one to many relationship at various levels of granularity. 

A change in log format for Splunk UF 6.x relative to tracking apps using the Deployment Server

I realized two things yesterday as I was troubleshooting various Splunk things. The first relates to having multiple input configs sent to a centralized syslog server. The second relates to changes to the internal 6.x UF logs as it relates to tracking apps that have been installed or removed.

What I meant to say was...

Well for good or ill I flew through my presentation at the Splunk conference lol. The upshot is I think it went ok. The downside is there are a few things I didn't say that I had planned to. Guess I'll take the opportunity now to cover a few....

A search on the Splunk mug is wrong!!

For those that haven't seen it the Splunk mug is a neat little piece of practical schwag that contains queries for things ranging from finding happiness to finding Waldo and even tracking a zombie infestation. However! I've discovered an issue with one of the searches. 

The first thing to understand, if you don't already, is that the asterisk is a wildcard in Splunk. A neat little trick is that when you combine it with a field as in field=* your search will return events where that field contains a value. This makes it a great little inclusive search and potentially you won't have to use a usenull=f as part of your chart or timechart further in your search for filtering out events where the field isn't populated. 

I want more time to play!

I find myself in a somewhat strange place today where because I'm going to be at the Splunk conference next week I don't have much scheduled that needs to be done (or staged to be done this weekend). This reminds me of a line that has come up a few times as we've been going through the interview and candidate selection process for two open slots we have in the office. We have all been working way too many hours and want some 'free time' back in our normal routine. I'm not talking about a mental health break or time away from the office as much as having a pocket or two of time where we can explore/investigate/work on little side projects/quality-of-life-things that need to be done. They, generally speaking, aren't hard or long things to do but get sidelined because of higher priorities. 

So I'm monkeying around with a few things in Splunk and two rabbit holes later come up with a query that quite frankly doesn't return a whole lot of hits for me over the last month. What it DOES show is a server that wasn't able to install some config packages I was pushing from my deployment server.

index=_internal source=*metrics.log component="DeploymentMetrics" status="failed" | stats max(_time) as time by hostname event scName appName fqname | convert ctime(time)

This event is created on your deployment server. Not sure what fqname stands for exactly but in my case it was showing me the path the server was trying to install the app to (fully qualified path name is where my mind goes but doesn't fit the data). scName is likely server class name and appName is obviously the app itself - both are references to your serverclass.conf file contents. With over 1k agents deployed the fact that this found issues with only 1 server is pretty cool I suppose. Will likely bake this into the app I'll never create re: first paragraph =)

Gearing up for Splunk .conf13

We had the Columbus Splunk user group meeting today which got me looking forward to Splunk's annual user group conference. Of course as I think that I reminded myself that instead of writing this I should be working on my own preso for the conference so will keep this short. I find it easier to make an argument for securing travel and training funds when you get a discount by speaking. My talk will be around how we went from a 350GB license to 2TB in less than a year and the associated growing pains/lessons learned. Hopefully folks will find a nugget or two of value. 

For those on the fence I would pull the trigger. Lots of information to walk away with that will help you plan, adjust, tweak, create, and simply expand your vision of what's possible. Some of that won't come from the sessions as much as talking with other folks and seeing how they are using the tool. Wondering what the free gift for attendance might be this year. Last year it was a zip up Splunk hoodie that was great - primarily because the conference area was chilly! 

To circle back to an issue raised at the local user group meeting. While I'm full time on our Splunk effort (and in need an extra body) many folks are Splunk admins plus wear any number of other hats. There would probably be high interest in a session devoted to running a Splunk environment with a skeleton crew as it were. I'll try to put some thought into that.

Anyone have any thoughts on that or tips to share?

Solve for 80% - find logs needing work in Splunk

There are a couple of sayings, maxims if you will, that I try to keep in the back of my mind as I do things

• Most times coincidence is God acting anonymously
• Activity != Accomplishment
• Effectiveness and efficiency are two different concepts
• Solve for 80%

Hidden value of Windows 673/4769 events - updated for Splunk

So I took some time today and converted something I had somewhat forgotten - the post I made just over 3 years ago about being able to detect an infected Windows machine exhibiting wormlike behavior into Splunk search language. Don't judge me - a lot has happened between then and now.

The beauty of this detection is you just need to get logs from your domain controllers.

Some queries related to Splunk administration/Deployment Server

Once again time flys /sigh. At times I wish there was more hours in the day but that would probably just translate into more hours working. Hopefully will get over the bubble soon (yeah right). At any rate we had our first Columbus Splunk user group the other day. Was neat to see others in the local area and talk Splunk...at least in as much as we could. The location was a bit noisy. Figured I'd share here a few of the queries I put together in a slide deck related to using (or mostly administrating) Splunk's Deployment Server. I guess they aren't specifically related to the DS as much as general Universal Forwarder (local Splunk agent) health which you can control with the DS

Back in the saddle...and using Splunk!

Been a long time since writing and lots has changed! New house, cataract surgery (at 39!), new job, developing and implementing the log management and event correlation program at a large edu, etc. On that front a large university is very different than a large corporation. Take all of those issues you have with decentralization and multiply that by about 50. I debated between believable and hyperbole there and ended up somewhere in the middle I guess.

One of the bigger changes is using Splunk for the first time. At some point I should write some comparisons between ArcSight, Symantec MSSP, and Splunk. Granted it has been a few years since I’ve used ArcSight’s Enterprise Security Manager (ESM) or Logger for that matter though I remember ESM fondly and often wistfully. It still amazes me when I hear people say things like ArcSight is too hard. I maintain that for most of those cases they somehow had the idea (or were told by the sales people) that it was some magical plug and play cross between a one armed bandit and magic 8-ball. I also hear in the latest update to Symantec’s MSSP they have implemented a number of the dozens of changes I suggested (am sure others as well!) and it is now much better than what it was 6 months ago.

If there is one piece of advice I could give it would be install Splunk today. You may not use it a ton in an active way but at the very least I wish I had had it for all of those times I had large Excel spreadsheets to massage in an effort of extracting value.

Welp as was the case when I first started writing this was written while waiting for the oil to be changed in one of my vehicles. That is complete and this sort of went in a different direction than planned. In my next post I plan to talk about a few ways I’ve found in Splunk to look for fields that don’t exist. As an aside it is somewhat interesting to think about writing Splunk stuffs in somewhat of the same way I used to write about ArcSight stuffs. The main driver back then was there wasn’t much out there at all for ArcSight and LM/SIEM in general and the ArcSight forums were gated to the general public. That isn’t the case with Splunk (gated community) and sadly the act of running the Splunk implementation and general ho-ha leaves me far less time to actually DO something in Splunk than I’d like.