Drilling into the OTHER category in Splunk

So what has broken my 3 year blog posting hiatus you might ask? Some nerd-like delight in working through a Splunk dashboard capability I didn't realize was there!

Several days ago some fellow Splunk users asked if there was a way to drill into the "OTHER" category. They had an overview dashboard with a bar chart viz allowing the user to pivot to a more detailed interactive dashboard. The challenge was the overview graphic leveraged Splunk's ability to show the top N results with the rest of the results up as OTHER. The interactive dashboard didn't like receiving OTHER as that wasn't a value in the data. I tried a few different approaches but they honestly didn't work. Through that effort though I stumbled upon the ability to set a condition match in XML.

Wait whuuut? I've known about condition match from adjusting navigation bars in Splunk and it turns out this capability is also available in dashboards themselves. Could I use this mechanism for the use case at hand?

Adjusting Splunk forwarder phonehome / throughput

I was in the process of writing up a few things for a new EDU that is going to be spinning up a larger scale Splunk environment and figured if I was going to the effort it might as well be placed here for others to see. In working with my own environment today I realized I was making some adjustments that I take for granted but that we had to learn and bake in. For this installment these items are focused on the following:

1. Adjusting the forwarder to deployment server phone home interval
2. Allowing forwarders to send more than 256 kbps

Find saved searches in Splunk that are failing

I hope to circle back to this eventually. Until then --- enjoy:

index=_internal log_level=ERROR SavedSplunker | stats count as Count by host message | rex field=message "savedsearch_id=\"(?<Author>[^;]+);(?<App>[^;]+);(?<Search>[^\"]+)\"(?:, message=)?(?<Message>.+)" | table host App Search Author Message Count | eventstats sum(Count) as total by host | eventstats sum(Count) as foo by host App | sort -total -foo -Count | fields - total foo

Splunk admin tasks after you start getting data in...

I had the rather unique privilege to post a 3 part blog series on Splunk's official site recently. The focus was on some administration tasks Splunk admins should work into their routine. There is a level of assumption when users search in Splunk - these hosts are really these hosts and events that are observed within a time range really happened then. The series talks through a couple methodologies to validate those assumptions

• Part 1 - Validating host field values: link
• Part 2 - Validating agent host's system time: link
• Part 3 - Getting a feel for data ingestion latency: link

Moving toward Splunk's CIM

For those that don't know, for some time Splunk has been moving toward a Common Information Model (CIM). They are using this both a data normalization effort - what should you name fields from particular data sources - as well as a layer of abstraction placed over your data to indicate what the data IS. In the end this is a worthwhile effort though the devil is in the details for those of us with 1) older - by whatever definition - Splunk instances with local extractions and 2) larger - again by whatever definition - Splunk instances with 3) a large - sensing a trend? - number of sourcetypes. Frankly I'm big time scared of the performance implications of using what was a second or third class citizen in Splunk (tags) as my primary source of querying across 1k sourcetypes and 15B logs per day. Martin Mueller had a great talk at .conf15 looking into performancy sorts of things which brings a lot to that aspect of the discussion (search for his name here for a link to the slides. His name is actually spelled Martin Müeller and a direct link to the pdf is >here<).

Performance concerns aside the question is how do you go about a discovery effort to figure out which of your sourcetypes should map to which CIM based data model? In theory and based on the number of sourcetypes you have you could do this by manually reviewing a list. That might work for some percentage of sourcetypes but perhaps not all. At any rate some of this is addressable by the new Splunk commands: pivot and datamodel. The challenge with those is they are essentially searching across your data with the fields contained within the model in a one off basis (one DM at a time) and if the fields don't match then there simply is no results. What I was wanting was a way to take all of the fields from my data and throw that up against all of the fields in all of the models with a side of fuzzy string comparison. I *think* I have found a way.