Splunk DateParserVerbose logs - Part 2

In part 1 of this subject we talked about what Splunk's DateParserVerbose internal logs are and I gave an example query that at its heart attempts to rollup and summarize timestamp related issues. In this post I'll present a query for taking the sourcetypes Splunk is having issues with from a timestamp perspective and display the relevant props configs. What we've done is thrown both queries into the same dashboard to make things easier to work though. I should note a couple things here. The first is the foreach command is only available in Splunk 6 (I believe). The second is the REST endpoint I'm getting the config data from is likely only available in 6.

With that out of the way here is the query:

| rest /servicesNS/-/-/configs/conf-props | eval sourcetype = if(isnull(sourcetype), title, sourcetype) | rename eai:appName as App | search App!=system App!=learned [search index=_internal DateParserVerbose | rex "source::(?<Source>[^\|]+)\|host::(?<Host>[^\|]+)\|(?<sourcetype>[^\|]+)" | dedup sourcetype | fields sourcetype]  | eval SHOULD_LINEMERGE = if(SHOULD_LINEMERGE=0, "False", "True / Default") | dedup App sourcetype | table sourcetype App TIME_FORMAT TIME_PREFIX MAX_TIMESTAMP_LOOKAHEAD LINE_BREAKER TZ SHOULD_LINEMERGE MAX_DAYS_AGO TRUNCATE | foreach * [eval <<FIELD>> = if(isnull(<<FIELD>>),"Null",<<FIELD>>)] | foreach * [eval <<FIELD>> = '<<FIELD>>'."##".sourcetype."##".App."||"] | stats values(*) as *| transpose | makemv delim="|| " "row 1" | mvexpand "row 1" | search column!=App column!=sourcetype | rex field="row 1" "^(?<value>[^\#]+)##(?<sourcetype>[^\#]+)##(?<app>[^\#]+)" | eval value = if(value="Null","-",value) | table app sourcetype column value | sort app sourcetype column | stats list(column) as setting list(value) as value by app sourcetype

The upper case props attributes call undo attention to themselves =). The blue portion of the query is a subquery that if you notice is the same as what the query in the previous post started with. The reason is since we are pulling out config data we might as well just concentrate on the ones relevant to the sourcetypes Splunk is having issues with. There is a whole other conversation to be had on using the REST query to show configs w/o having to go to the command line - but that is something for another day. 

So what is this query doing? It starts by hitting the config-props REST endpoint and then does some monkeying around and renames the app name field to make it easier to work with and ultimately is self trimming the results based on the sourcetypes that are having issues (subquery). I'm doing eval on line merging because sometimes you want to line merge in your events but that is also the default setting so you may have never actually set it. If you have ever seen the Splunk preso on getting data in correctly you will recognize the 7 fields in the table and have added max days ago as it can often show up from the previous query. If you haven't seen it try to get a hold of the .conf2012/.conf2013 slide deck. Make an effort to get these populated. For the moment we will ignore the irony of many sourcetype attribute settings in the Splunk Exchange TAs not being set. The rest of the query is somewhat harder to explain and would suggest running the query pipe by pipe. Frankly speaking a good bit of the query is based around my preference to have results in the format these end up in =).