Tuesday, May 26, 2020

Drilling into the OTHER category in Splunk

So what has broken my 3 year blog posting hiatus you might ask? Some nerd-like delight in working through a Splunk dashboard capability I didn't realize was there!

Several days ago some fellow Splunk users asked if there was a way to drill into the "OTHER" category. They had an overview dashboard with a bar chart viz allowing the user to pivot to a more detailed interactive dashboard. The challenge was the overview graphic leveraged Splunk's ability to show the top N results with the rest of the results up as OTHER. The interactive dashboard didn't like receiving OTHER as that wasn't a value in the data. I tried a few different approaches but they honestly didn't work. Through that effort though I stumbled upon the ability to set a condition match in XML.

Wait whuuut? I've known about condition match from adjusting navigation bars in Splunk and it turns out this capability is also available in dashboards themselves. Could I use this mechanism for the use case at hand?

I should mention that I'm really not a programmer nor have much programming background. I do employ the second most common programming language though - steal. As in I'm going to find some examples and steal them. Incidentally the most common programming language is of course Excel - unless you are a Splunk user and bring those spreadsheets into Splunk for analysis.

Queue up the A-Team inspired, Mandalorian spaceship rebuilding, Google searching montage.

Ok so searching done and trial and error completed I have success! As I didn't find anything along the lines of this use case I figured I'd write it up. To demonstrate I'm going to use the internal Splunk logs using the 'component' field to group by. I want to start with my top 10 component categories. If a user clicks on any one of them I want the drill down to take them to the interactive dashboard. If they click on the OTHER category I want the drill down to take them to a dashboard that shows them categories 11 through N. They can then select one of these categories which will take them to the interactive dashboard. Something like this




The initial visualization is populated with the following search

index=_internal component!=metrics | top component useother=1

I'm not showing the metrics log/component because it totally skews the chart =).

Being the relatively lazy person I am I'd like to have Splunk figure out pieces of this. The first thing I do is use the UI to figure out the drill down from my visualization to the interactive dashboard. That results in something like this which I pull out from the XML view

<drilldown>
  <link target="_blank">/app/playground/splunk_logging_component_investigator?form.component_drilldown=$click.value$</link>
</drilldown>

Back to the UI view I reset the drill down to my dashboard showing items 11 and beyond. That results in the following

<drilldown>
  <link target="_blank">/app/playground/components_11_to_n</link>
</drilldown>

Incidentally the query I use to populate that dashboard is

index=_internal sourcetype=splunkd component!=metrics NOT [search index=_internal sourcetype=splunkd component!=metrics | top component | fields component] | stats count by component | sort -count

Now - flip back to the XML view and putting the pieces together you do something like this

<drilldown>
  <condition match="'click.value'= &quot;OTHER&quot;">
     <link target="_blank">/app/playground/components_11_to_n</link>
  </condition>
  <condition match="'click.value'!= &quot;OTHER&quot;">
     <link target="_blank">/app/playground/splunk_logging_component_investigator?form.component_drilldown=$click.value$</link>
  </condition>
</drilldown>

The first condition checks to see if the clicked value is OTHER, the second looks for everything else.

Insert self congratulatory shout in my makeshift office due to Coronavirus lol.

So what about something different? I hadn't ever played with hidden panels. What if the user clicked OTHER a hidden panel appeared that showed the 11th+ categories on the same dashboard? Depending on what your initial dashboard looks like that might be better UX.

Insert another Googling montage.

Before setting up the hidden panel you first have to layout the dashboard pieces. My initial dashboard now looks like this where the first chart works as outlined above. The second one shows the same data but if you click Other it will unhide the third panel.



It appears the main way you hide/show a panel is to use the "depends" condition on the panel itself. There are likely several (better) ways to do this.

<panel depends="$your_token_here$">

To get it to show then we need to set the token within the correct condition segment. Since the logic seems to be "does this token have a value?" I put in "yep".

<drilldown>
  <condition match="'click.value'= &quot;OTHER&quot;">
     <set token="show_OTHER_items">yep</set>
  </condition>
  <condition match="'click.value'!= &quot;OTHER&quot;">
     <link target="_blank">/app/playground/splunk_logging_component_investigator?form.component_drilldown=$click.value$</link>
  </condition>
</drilldown>

With the third panel's XML looking like

  <row>
    <panel depends="$show_OTHER_items$">
      <title>The "Other" Components</title>
      <table>
        <search>
          <query>index=_internal sourcetype=splunkd component!=metrics NOT [search index=_internal sourcetype=splunkd component!=metrics | top component | fields component] | stats count by component | sort -count</query>
          <earliest>-4h@m</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="count">20</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">row</option>
        <option name="percentagesRow">false</option>
        <option name="rowNumbers">false</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <drilldown>
          <link target="_blank">/app/playground/splunk_logging_component_investigator?form.component_drilldown=$click.value$</link>
        </drilldown>
      </table>
    </panel>
  </row>

I've always appreciated Splunk creating the Splunk Dashboard Examples app (link) though would be great if they had both a condition and hidden panel example. Perhaps in a future version.

Hopefully someone will find this helpful!

No comments:

Post a Comment