Finding Log Volume Ingestion Anomalies in Splunk

 

This is for my man Destry who I met recently in person. He was giving me a bit of good-natured fun at not posting more frequently. So Destry, this is for you!

I’m doing a Splunk tips & tricks workshop this week with some folks who, among other things, had asked for a query to identify log volume anomalies. Ahh volume anomalies. So many variations of this. Several apps can be found on Splunkbase which have been developed by the user community. One might ask why Splunk hasn’t incorporated more of this sort of thing in the Monitoring Console /shrug.

My normal recommendation to folks is run a few queries to capture log volume (internal index license log) and event counts (tstats) in a ‘summary index’ for long term retention and quicker analysis. Some of that is likely found in the introspection index but I’ve not done a deep dive there TBH. The workshop I’m doing is with folks in a multi-tenant environment where each would like to do their own quick analysis.

So let’s define a few goals

• When a host is sending abnormally more or less of a data type compared to other hosts
• When a host is sending abnormally more or less of a data type compared to itself
• One query to do both comparisons to keep compute down and not have intermediate steps (like populating or reading from a lookup) for simplicity

New Managers: Hiring Process

The bug to write has bit again. 

There aren’t a ton of quick resources nor do many orgs place a lot of emphasis on training for new managers. I’ve done a good bit of candidate prospecting and hiring over the last year so I’ll write through a bit through that lens across a few posts. I’m no expert but do have some thoughts on the subject.

You’re at a point where you must hire someone. What does that process look like and where do you even start?

I’d begin by contemplating the following

• Who you hire represents a $500k - $1M investment assuming they will be with you for several years. Make sure you give the process appropriate time & energy.
• “We hire people based on what they know and fire them for who they are.” Unknown original source but really like it. Don’t be so focused on skills that you miss warning signs of cultural fit and work ethic.  

After giving those a good think, here are some additional framework-y things to do some mental gymnastics on before you start. You will inevitably have to adjust as you go but limit how much of the plane you are building after it is in the air.

Framework Compliance: Does activity live on a spectrum or an iron triangle?

Whenever I see an article title like “<insert framework>: One framework to rule them all” as recently discussed here I’m simultaneously reminded of the following xkcd cartoon, which touches on the N+1 issues of framework consolidation, and my grandmother saying "you can’t change the direction of a parked car." Relative to achieving framework compliance I’m not sure if those two thoughts live on the same spectrum or make up 2 corners of a compliance related iron triangle. Not sure what the 3rd iron triangle corner would be though.

Splunking Game Data - Darkfire Heroes part 2

In part of one of this series I went down the path of Splunking game data from a 2 day, single challenge event. In this installment I'm going to dive into a bit of data from an event that encompassed several different challenges. 

In looking back at the dashboard I created, how I was working with some of the data, and the sheer volume of data across all of the challenges - the dashboard itself might not have much value from a game analytic perspective lol. I also realized how much I focused on one challenge - waves. In looking back I think that was largely due to the numbers being smaller. The approaches to displaying the data will work across all of the events so at some level it would be a lot of copy and paste as well. At any rate hopefully folks will take something away from the underlying queries and some of the challenges with trying to display the data/volume of data. TBH this is the more primary focus of this overall effort. 

Data was captured across 4 challenges, 3 player level brackets, and listed scores for the top 100 positions (other than the clan boss fight).

1. Waves - 5 separate groups of mobs attack your heroes
2. Pacifism - multi-level, player vs environment (PvP) track where you couldn't bring damage dealers
3. No Healers - multi-level, PvE track where you couldn't bring a healer
4. Boss - big bad dude that everyone in the clan attacked

Splunking Game Data - Darkfire Heroes

(Part 2 Link)

So there is somewhat of an eclectic mix of drivers going into this post. As I talk with individual Splunk users, Splunk administrative teams, and larger teams that have access to Splunk (security, operations, etc) I sometimes come across a subset of folks who have a tough time trying to figure out how to take advantage of the data they have access to. In some cases it is because they lack imagination, vision, or strategy relative to what they are trying to accomplish. Sometimes it is being so heads down in the daily grind they have trouble 'looking up' so to speak and collecting their thoughts. More often than not though they are wearing multiple hats and could use a hand getting over the Splunk language learning curve. This post is designed to help a bit - some Splunk Processing Language (SPL) exposure mixed with a bit of dialogue on why I made the dashboard/dashboard panels the way I did.

To start with I need data. Turns out I've started playing a mobile game called Darkfire Heroes. It isn't quite in a beta state but isn't available globally quite yet. The other day, Dec 8 & 9 to be exact, they had a 2 day Player vs Player (PvP) contest. Individual match wins were tracked as well as aggregate win total if you belonged to a group within the game; a clan. As a total side note it is interesting to see people's gaming pedigree based on descriptions they use. Are some of the bad guys/monsters you fight creeps, mobs, trash, etc and are you a part of a clan, guild, or alliance? Outside of this mini event there is a clan rewards chest that also gives varying level of rewards based on aggregate PvP wins over a week. PvP matches won during this 2 day event will count towards that reward chest.