Full of hate...

Point productivity solutions in an enterprise environment drive me crazy.

Starting to Splunk!

Well I’m off to the Splunk conference.  Having only started using Splunk just over a month ago I can say there is quite a lot to digest and frankly I’m feeling a little overwhelmed. The challenge is starting from step 0 with a good bit of unstructured data knowing in the next year or so that will grow to an estimated 700GB/day. Part of the challenge is not knowing the full implication of choosing different methods to actually do things in splunk, like field extraction, in a way that doesn’t artificially limit or cause issues down the road. This is all while developing a program to handle it all toward multiple ends. In some respects I’m going from using an MSS to being an MSS. New job + new tool + new house (that we are doing renovation work on) = good times. And just for kicks I'm building, and have talked a few others into, building a plywood canoe that we can race each other in. 

At any rate I’m excited to be going and hope to accelerate the learning curve dramatically. If anyone has any Splunk tips I’d be interested in em!

A cappella log management

A curious thought hit me the other day in church; hopefully I can articulate it well enough to be understood – even by me. For those that aren’t familiar there is often full congregational signing as part of the worship service. As you might expect this includes musicians playing various instruments, several folks up on the stage signing as part of the choir, and in the olden days a hymnal in your hands that has the words. I’m going to murder terms but to translate it in my head you have the technical musicians translating symbols on a page of music into sounds on their instruments, the next layer up are the signers who combine the lyrics with the symbols to know how to sing the words, and then you have a presentation layer where the rest of the congregation is able to see the words and take their singing cues from the music and choir. More often than not these days those words are presented on a projector screen without the musical score which works for me since I can’t read that bit anyway. At any rate there is a cumulative layering effect that allows folks like me who aren’t musically inclined or trained to participate. (Is there a musical OSI equivalent?)

The difference a couple weeks ago was the music was done a cappella and only a small number of the choir was singing. While I could see the words on the projector I was much slower to catch on to the “tune.” I and many others didn’t sing as loudly that day (this is a good thing in my case).

So what’s the link you ask? Going through the process of leaving my job to pursue another I have been somewhat introspective in thinking of what I could have done differently to try to explain the need for log management and where that fits into larger discussions of security strategy, incident detection, incident investigation, monitoring, etc. By and large I think the message wasn’t understood the higher you go up the management chain. And while I’m somewhat saddened and disappointed by that fact I’m not going to beat myself up over it. I tried several times and several different ways to communicate the need and why certain paths were better than others not only for immediate needs but also for where that positioned us 6 months, a year, several years down the road.

If you are reading this blog you, like me, can at least at a conceptual level look at something like an incident detection use case and probably see all or at least a lot of what must go into what is required for the alert to be tripped from a log collection perspective, what needs to happen once it goes off, how that use case fits into the larger picture, etc - much like my wife can look at a sheet of music and hear the music in her head. The challenge I pose to us all is - are we communicating our message a cappella in that our audience is perhaps just seeing “the words.” Something to think about the next time you do a presentation perhaps.

Statistical vs Rule based Threat Detection

A number of different discussions have led me to think about the difference between log management and SIEM when it comes to their use and play in threat detection. Of the many items that could be discussed what I came to is the difference between statistical and rule based threat detection.

An oft used analogy, even referenced in the Verizon DBIR, is the difference between looking at haystacks and needles in haystacks. A statistical detection methodology might be to review the top N of X activity within Y timeframe. The point of this exercise is twofold. The first is simply to look at the “curve” of the numbers involved in the top 5 relative to the top 10 relative to the top 30 or just a spike in a line graph showing the volume of logs collected. All of which might indicate something has happened and might be worth diving deep on. The second is to help dial in your tools by whitelisting systems performing normal activity. If you are looking for outbound SMTP traffic sorted by volume in a day, you should be able to easily spot your email gateways. Whitelist them and the next time the report is run the top of the list might contain compromised systems. By and large many log management systems should be able to accomplish this sort of activity.

At some point though you will want to focus in on specific threat activity. Take today’s SANs Diary update on Run Forest Run or Sality, Tidserv, whatever. In this case you have specific information and want to receive an alert when one of your internal systems hits an IP, URL, or multi-step pattern of activity. This is your rule based needle finding capability. Generally speaking this requires a rule engine of varying level of sophistication located within some point solution or product. Statistical detection won’t really be able to get you this.

The challenge is many point solutions, by design or omission, aren’t able to factor in the larger view in reviewing the needles found. MSSPs are notorious (too strong?) for this but then so are things like IPS. “We saw this and this so we wrapped it up in a pretty bow for you” ….great, but I need more context. The SIEM technology space, in general, was supposed to fill this gap. Not only can you develop specific rules to find needles in the overall stream of log consciousness but to a greater or lesser extent, based on vendor/tool/administrator, use them in a more statistical way. I think a lot of “Mah SIEM sux” mentality comes from how you approach your SIEM relative to this overall issue. That is a rabbit hole that I don’t want to go down though.

Especially when you first start out, if you dive directly to a rules based approach you will have a harder time seeing the forest for the trees and depending on the tool used will be frustrated that you can’t move that lens backward. In other words dealing with individual infections IS key but if you are so focused on the individual detections you lose sight of the bigger picture you can do yourself a disservice. On the other hand if you just do a statistical approach and never grow you are going to miss the needles you need to find.

I would argue there is a direct correlation between shop maturity and the ability to full leverage rule based technology. If you are just starting out I suggest you will see more value with a log management, statistical threat detection methodology. This will allow you to get to know your data – as strange as that might sound – which will allow you to better dial in your rule based solutions.

Excelification meter

From a tool perspective it is hard to beat Microsoft Excel. Love or hate the Microsoft toolset or Microsoft in general I think you have to acknowledge Excel is a something that simply works and works for and at a variety of levels and ends. To maybe say it a different way, the developers have recognized the usefulness of the program ISN’T the program (on the surface). What makes Excel useful is that you can take data from a variety of sources and manipulate it to some end that provides value. I’ve heard it said Excel is the most widely known programming language and I agree. Good or bad, that is how I use it.

Contrast Excel’s ability to manipulate data with just about any IT related tool out there. To be fair Excel does have more than a slightly higher user base. And while I don’t expect a hammer and saw to be interchangeable I’m tired of using Enterprise scale tools whose vendors have created in such a way that they believe the tool itself is the end state. They don’t acknowledge – or worse don’t even comprehend – that their tools also need to be a means to an end. Generally speaking the tool might collect data and the vendors have tried to think about various use cases in order to support them with alerts, reports, etc. The problem is on a day to day basis this content isn’t sufficient to support deeper investigation or asks by management. What is turned to? Exporting the data to Excel.

We spend hundreds of thousands of dollar on all our various tools…and end up doing analysis and initial correlation in Excel. Crazy. Frustrating. I think we almost need some sort of Excel meter or term to gauge things when they come in the shop – “so what percent of the time are we going to have to use Excel to actually derive value from this $500k investment?”

Couple random thoughts strung together

Oyie – I look up again and I haven’t written anything in a couple months. Amazing how your environment will (or won’t) influence you. I have had a few things on my mind for a bit recently though and figured I would try to knock out brief blurbs on them now.

• One thousand points of light….and no illumination
• A visual of effort vs value/return
• Integration is key

Treating the symptom

Today in the car I heard a credit card debt consolidation commercial that sort of drove me crazy. While there is a place for those companies the last line got to me - “If credit card debt is the problem, we are the solution.” Heads up there high speed - the real problem is you can't stop buying stuff you can't afford! Your credit card debt is just the visible symptom. Treating the symptom instead of the problem only lands you back in the same spot. This goes hand in hand with the diet pills that basically say take this magical pill to lose weight and that you don't even have to change your daily habits....like over eating and getting no exercise...which is what got you where you are.

News flash – IP addresses aren’t computers

Crazy thought I know but it isn’t hard not to get caught up in that mentality. I was trying to think of a way to tell the story of resources/logs needed to be able to be able to identify sources of badness in incident response of one flavor or another. Visually I was drawing that out somewhat like IP ~ Computer Name ~ User Name all at the top level and branching under that you have various logs like DNS, DHCP, asset management, authentication, etc. All of which play a part in being able to answer questions related to what computers are infected and which users are doing ‘bad things.’ Anyway, it wasn’t until I put that down on the whiteboard that the thought hit me in that IP addresses are a supporting factor in identifying a particular computer and not equal to it. Funny what tool limitations will do to your thinking.

If none of that makes sense to anyone other than me I blame the Nyquil.

The unrecognized APT story....

2 hobbits sneaking into Mordor.


Somehow I don't see me using that analogy in the executive boardroom though.

What's in a name?.....Strategy

Follow me for a moment on a random thought. If vision begets strategy and strategy begets roadmaps and on down we go; wouldn’t common language and labels be a byproduct of that activity? If that is true, would it be somewhat safe to say if you don’t have common language and labels for groups or efforts it indicates a breakdown further up the chain?

The trick is identifying where the chain is broken....and fixing it.

International Conference on Cyber Security 2012 and "hunters"

Overall impression….meh. It was decent for a first year conference. Oh wait this is its 3rd. That there was a Cyber Security Tutorial before the first actual day of the conference should have been a little telling. Presenters were also only given 30 minutes so not much ability to dive deeply. I think there was more of a focus on law enforcement cyber concepts and general cyber research than anything else which absolutely has a place…..but not for where I am and what I’m doing. If it required a clearance it would probably be a conference I’d like to have happen twice a year given the areas the speakers came from. Ah well. I also hate to knock presenters as on some level it takes some stones to stand up before folks and speak. At one point I did see someone take their iPad, take a picture of one of the presenters as they droned on and on and used some movie special effects app to blow them up. Indeed sir, indeed.

One thing I did find myself thinking about at the airport was the idea of a Cyber Warfighter