Back in the saddle...and using Splunk!

Been a long time since writing and lots has changed! New house, cataract surgery (at 39!), new job, developing and implementing the log management and event correlation program at a large edu, etc. On that front a large university is very different than a large corporation. Take all of those issues you have with decentralization and multiply that by about 50. I debated between believable and hyperbole there and ended up somewhere in the middle I guess.

One of the bigger changes is using Splunk for the first time. At some point I should write some comparisons between ArcSight, Symantec MSSP, and Splunk. Granted it has been a few years since I’ve used ArcSight’s Enterprise Security Manager (ESM) or Logger for that matter though I remember ESM fondly and often wistfully. It still amazes me when I hear people say things like ArcSight is too hard. I maintain that for most of those cases they somehow had the idea (or were told by the sales people) that it was some magical plug and play cross between a one armed bandit and magic 8-ball. I also hear in the latest update to Symantec’s MSSP they have implemented a number of the dozens of changes I suggested (am sure others as well!) and it is now much better than what it was 6 months ago.

If there is one piece of advice I could give it would be install Splunk today. You may not use it a ton in an active way but at the very least I wish I had had it for all of those times I had large Excel spreadsheets to massage in an effort of extracting value.

Welp as was the case when I first started writing this was written while waiting for the oil to be changed in one of my vehicles. That is complete and this sort of went in a different direction than planned. In my next post I plan to talk about a few ways I’ve found in Splunk to look for fields that don’t exist. As an aside it is somewhat interesting to think about writing Splunk stuffs in somewhat of the same way I used to write about ArcSight stuffs. The main driver back then was there wasn’t much out there at all for ArcSight and LM/SIEM in general and the ArcSight forums were gated to the general public. That isn’t the case with Splunk (gated community) and sadly the act of running the Splunk implementation and general ho-ha leaves me far less time to actually DO something in Splunk than I’d like.

Full of hate...

Point productivity solutions in an enterprise environment drive me crazy.

Starting to Splunk!

Well I’m off to the Splunk conference.  Having only started using Splunk just over a month ago I can say there is quite a lot to digest and frankly I’m feeling a little overwhelmed. The challenge is starting from step 0 with a good bit of unstructured data knowing in the next year or so that will grow to an estimated 700GB/day. Part of the challenge is not knowing the full implication of choosing different methods to actually do things in splunk, like field extraction, in a way that doesn’t artificially limit or cause issues down the road. This is all while developing a program to handle it all toward multiple ends. In some respects I’m going from using an MSS to being an MSS. New job + new tool + new house (that we are doing renovation work on) = good times. And just for kicks I'm building, and have talked a few others into, building a plywood canoe that we can race each other in. 

At any rate I’m excited to be going and hope to accelerate the learning curve dramatically. If anyone has any Splunk tips I’d be interested in em!

A cappella log management

A curious thought hit me the other day in church; hopefully I can articulate it well enough to be understood – even by me. For those that aren’t familiar there is often full congregational signing as part of the worship service. As you might expect this includes musicians playing various instruments, several folks up on the stage signing as part of the choir, and in the olden days a hymnal in your hands that has the words. I’m going to murder terms but to translate it in my head you have the technical musicians translating symbols on a page of music into sounds on their instruments, the next layer up are the signers who combine the lyrics with the symbols to know how to sing the words, and then you have a presentation layer where the rest of the congregation is able to see the words and take their singing cues from the music and choir. More often than not these days those words are presented on a projector screen without the musical score which works for me since I can’t read that bit anyway. At any rate there is a cumulative layering effect that allows folks like me who aren’t musically inclined or trained to participate. (Is there a musical OSI equivalent?)

The difference a couple weeks ago was the music was done a cappella and only a small number of the choir was singing. While I could see the words on the projector I was much slower to catch on to the “tune.” I and many others didn’t sing as loudly that day (this is a good thing in my case).

So what’s the link you ask? Going through the process of leaving my job to pursue another I have been somewhat introspective in thinking of what I could have done differently to try to explain the need for log management and where that fits into larger discussions of security strategy, incident detection, incident investigation, monitoring, etc. By and large I think the message wasn’t understood the higher you go up the management chain. And while I’m somewhat saddened and disappointed by that fact I’m not going to beat myself up over it. I tried several times and several different ways to communicate the need and why certain paths were better than others not only for immediate needs but also for where that positioned us 6 months, a year, several years down the road.

If you are reading this blog you, like me, can at least at a conceptual level look at something like an incident detection use case and probably see all or at least a lot of what must go into what is required for the alert to be tripped from a log collection perspective, what needs to happen once it goes off, how that use case fits into the larger picture, etc - much like my wife can look at a sheet of music and hear the music in her head. The challenge I pose to us all is - are we communicating our message a cappella in that our audience is perhaps just seeing “the words.” Something to think about the next time you do a presentation perhaps.

Statistical vs Rule based Threat Detection

A number of different discussions have led me to think about the difference between log management and SIEM when it comes to their use and play in threat detection. Of the many items that could be discussed what I came to is the difference between statistical and rule based threat detection.

An oft used analogy, even referenced in the Verizon DBIR, is the difference between looking at haystacks and needles in haystacks. A statistical detection methodology might be to review the top N of X activity within Y timeframe. The point of this exercise is twofold. The first is simply to look at the “curve” of the numbers involved in the top 5 relative to the top 10 relative to the top 30 or just a spike in a line graph showing the volume of logs collected. All of which might indicate something has happened and might be worth diving deep on. The second is to help dial in your tools by whitelisting systems performing normal activity. If you are looking for outbound SMTP traffic sorted by volume in a day, you should be able to easily spot your email gateways. Whitelist them and the next time the report is run the top of the list might contain compromised systems. By and large many log management systems should be able to accomplish this sort of activity.

At some point though you will want to focus in on specific threat activity. Take today’s SANs Diary update on Run Forest Run or Sality, Tidserv, whatever. In this case you have specific information and want to receive an alert when one of your internal systems hits an IP, URL, or multi-step pattern of activity. This is your rule based needle finding capability. Generally speaking this requires a rule engine of varying level of sophistication located within some point solution or product. Statistical detection won’t really be able to get you this.

The challenge is many point solutions, by design or omission, aren’t able to factor in the larger view in reviewing the needles found. MSSPs are notorious (too strong?) for this but then so are things like IPS. “We saw this and this so we wrapped it up in a pretty bow for you” ….great, but I need more context. The SIEM technology space, in general, was supposed to fill this gap. Not only can you develop specific rules to find needles in the overall stream of log consciousness but to a greater or lesser extent, based on vendor/tool/administrator, use them in a more statistical way. I think a lot of “Mah SIEM sux” mentality comes from how you approach your SIEM relative to this overall issue. That is a rabbit hole that I don’t want to go down though.

Especially when you first start out, if you dive directly to a rules based approach you will have a harder time seeing the forest for the trees and depending on the tool used will be frustrated that you can’t move that lens backward. In other words dealing with individual infections IS key but if you are so focused on the individual detections you lose sight of the bigger picture you can do yourself a disservice. On the other hand if you just do a statistical approach and never grow you are going to miss the needles you need to find.

I would argue there is a direct correlation between shop maturity and the ability to full leverage rule based technology. If you are just starting out I suggest you will see more value with a log management, statistical threat detection methodology. This will allow you to get to know your data – as strange as that might sound – which will allow you to better dial in your rule based solutions.