A cappella log management

A curious thought hit me the other day in church; hopefully I can articulate it well enough to be understood – even by me. For those that aren’t familiar there is often full congregational signing as part of the worship service. As you might expect this includes musicians playing various instruments, several folks up on the stage signing as part of the choir, and in the olden days a hymnal in your hands that has the words. I’m going to murder terms but to translate it in my head you have the technical musicians translating symbols on a page of music into sounds on their instruments, the next layer up are the signers who combine the lyrics with the symbols to know how to sing the words, and then you have a presentation layer where the rest of the congregation is able to see the words and take their singing cues from the music and choir. More often than not these days those words are presented on a projector screen without the musical score which works for me since I can’t read that bit anyway. At any rate there is a cumulative layering effect that allows folks like me who aren’t musically inclined or trained to participate. (Is there a musical OSI equivalent?)

The difference a couple weeks ago was the music was done a cappella and only a small number of the choir was singing. While I could see the words on the projector I was much slower to catch on to the “tune.” I and many others didn’t sing as loudly that day (this is a good thing in my case).

So what’s the link you ask? Going through the process of leaving my job to pursue another I have been somewhat introspective in thinking of what I could have done differently to try to explain the need for log management and where that fits into larger discussions of security strategy, incident detection, incident investigation, monitoring, etc. By and large I think the message wasn’t understood the higher you go up the management chain. And while I’m somewhat saddened and disappointed by that fact I’m not going to beat myself up over it. I tried several times and several different ways to communicate the need and why certain paths were better than others not only for immediate needs but also for where that positioned us 6 months, a year, several years down the road.

If you are reading this blog you, like me, can at least at a conceptual level look at something like an incident detection use case and probably see all or at least a lot of what must go into what is required for the alert to be tripped from a log collection perspective, what needs to happen once it goes off, how that use case fits into the larger picture, etc - much like my wife can look at a sheet of music and hear the music in her head. The challenge I pose to us all is - are we communicating our message a cappella in that our audience is perhaps just seeing “the words.” Something to think about the next time you do a presentation perhaps.

Statistical vs Rule based Threat Detection

A number of different discussions have led me to think about the difference between log management and SIEM when it comes to their use and play in threat detection. Of the many items that could be discussed what I came to is the difference between statistical and rule based threat detection.

An oft used analogy, even referenced in the Verizon DBIR, is the difference between looking at haystacks and needles in haystacks. A statistical detection methodology might be to review the top N of X activity within Y timeframe. The point of this exercise is twofold. The first is simply to look at the “curve” of the numbers involved in the top 5 relative to the top 10 relative to the top 30 or just a spike in a line graph showing the volume of logs collected. All of which might indicate something has happened and might be worth diving deep on. The second is to help dial in your tools by whitelisting systems performing normal activity. If you are looking for outbound SMTP traffic sorted by volume in a day, you should be able to easily spot your email gateways. Whitelist them and the next time the report is run the top of the list might contain compromised systems. By and large many log management systems should be able to accomplish this sort of activity.

At some point though you will want to focus in on specific threat activity. Take today’s SANs Diary update on Run Forest Run or Sality, Tidserv, whatever. In this case you have specific information and want to receive an alert when one of your internal systems hits an IP, URL, or multi-step pattern of activity. This is your rule based needle finding capability. Generally speaking this requires a rule engine of varying level of sophistication located within some point solution or product. Statistical detection won’t really be able to get you this.

The challenge is many point solutions, by design or omission, aren’t able to factor in the larger view in reviewing the needles found. MSSPs are notorious (too strong?) for this but then so are things like IPS. “We saw this and this so we wrapped it up in a pretty bow for you” ….great, but I need more context. The SIEM technology space, in general, was supposed to fill this gap. Not only can you develop specific rules to find needles in the overall stream of log consciousness but to a greater or lesser extent, based on vendor/tool/administrator, use them in a more statistical way. I think a lot of “Mah SIEM sux” mentality comes from how you approach your SIEM relative to this overall issue. That is a rabbit hole that I don’t want to go down though.

Especially when you first start out, if you dive directly to a rules based approach you will have a harder time seeing the forest for the trees and depending on the tool used will be frustrated that you can’t move that lens backward. In other words dealing with individual infections IS key but if you are so focused on the individual detections you lose sight of the bigger picture you can do yourself a disservice. On the other hand if you just do a statistical approach and never grow you are going to miss the needles you need to find.

I would argue there is a direct correlation between shop maturity and the ability to full leverage rule based technology. If you are just starting out I suggest you will see more value with a log management, statistical threat detection methodology. This will allow you to get to know your data – as strange as that might sound – which will allow you to better dial in your rule based solutions.

Couple random thoughts strung together

Oyie – I look up again and I haven’t written anything in a couple months. Amazing how your environment will (or won’t) influence you. I have had a few things on my mind for a bit recently though and figured I would try to knock out brief blurbs on them now.

• One thousand points of light….and no illumination
• A visual of effort vs value/return
• Integration is key

What's the value of chasing alerts and other musings

I don’t know what your thoughts on this are but I’m trying to work out an illustration on how the churn related to the hamster wheel of your run of the mill incident detection and response doesn’t really lead to a whole lot of increased security posture or reduced risk – at least not directly or by itself. Don’t get me wrong, that work needs to be done and isn’t a trivial component of your overall program. At the same time I think this is one of those things where activity doesn’t necessarily indicate/translate/equal accomplishment. Great – you cleaned X machines with Y malware. Next week it will be N machines with Z malware. Soooo….does that mean you are more or less secure than the month before?

I’m of the general opinion that an improved security posture/reduced risk/reduced exposure is a by-product of doing analysis on the information gathered from your incidents and using that to drive change in either various configuration settings or policies (or both). Not rocket science or an original thought really. Hopefully that makes some sense.

Where do you sit in the boat?

pic found on Google images

I started writing this a week and a half ago but was laid out from being sick so it sort of fell off the plate. Figured I would go ahead and post it even though it isn’t as timely now.

I once took a sailing trip with my dad and 2 family friends across Pamlico Sound to the Outer Banks in NC. The sound is relatively shallow and on this particular day we had some pretty good sized rolling waves going on. I tried to find a picture to help convey the scene and this was the closest I could find. We were heeling the boat over pretty good though not quite to the extreme shown in the picture. At one point I was sitting about where the guy in the red pants has his left foot. I’m inches from the water, the edge of the boat is underwater similar to the picture, and the rest of my view is blocked by the sail – basically a 1 foot square view with water RIGHT there. With my limited view and position every time a wave came along from the other side of the boat it felt like we were going to capsize. After a couple of minutes of sitting there staring into the water it started to become hypnotically depressing. I finally stood up and by doing just that it totally changed my perspective. Nothing about how we were sailing the boat had changed but now I could see the sky, the horizon, and the waves before they hit the boat.

Incremental changes and long term benefits

For those that don’t know, my wife and I have lead several groups though Dave Ramsey‘s Financial Peace University (FPU). Very cool stuff; YOU should look into it. That something like 70% of Americans, regardless of income level, live paycheck to paycheck is horrible. The group that we just finished up with had two folks in the medical field. At one point we got to talking about the long term cumulative effects incremental changes and small efforts can have (positive or negative) in your life from both a health and financial perspective.

That got me to thinking about InfoSec and if the line of thought and other FPUisms translated over. I came up with a couple ideas.

The mortar between your defenses

The other day I read a bit by Andreas M. Antonopoulos on Networkworld about how to be an effective security buyer. Of course when it came to finding the article again when I wanted to write this….I couldn’t find it. +1 to the Interwebs though because Mike Rothman over at Securosis mentioned it in Wednesday's Incite 4 U. Andreas’ advice seems to be when you are buying security tools to not buy something designed to fulfill a singular function. Instead go for multi-purpose tools that can cover down on multiple areas. I think the idea somewhat boils down to knocking out two birds with one stone + it sucks to have to look at one dashboard for each tool you have. Enterprise resource scaling aside though I tend to agree with Mike’s take. What really stood out to me was an analogy Andreas used:

Is ArcSight harder to use?

Interestingly enough (if I read Google Analytics correctly) the post I created back at the beginning of November related to the question of is ArcSight hard to use was the most visited page over the last month. In fact I think it has been on or towards the top of the list since I posted it. Since I have some time on my hands I figured I would write a sort of follow up to that post. Once again I will state that my only SIEM experience is with ArcSight so maybe someone will speak up for the other SIEM engines out there and/or tell me if I’m way off base.

I still maintain asking if something is “hard”, generally speaking, often only gets you only so far. Having some context implied in your question might get you an answer closer to what you really are looking for. Is calculus hard? Depends on who you are talking to. Is calculus harder than basic algebra? Yup. Is “ArcSight” hard to use? Depends on who you are talking to. Is ArcSight harder to use than some other SIEM? Ahh now we are getting somewhere. I did have sort of an interesting thought while reading the SIEM implementation book I wrote a review of back in January (at least it was interesting to me).