ArcSight - The SIEM Lego Set

(Or for Chris' sake the Erector set!)

This post came mostly out of a comment and thread post on the internal ArcSight forums. The same person made them both as it turns out (Vini is the man).

Here’s the issue. I believe one of the design goals for the ESM is to move to more of a Trend based Dashboard system vs Active Channels. If that isn’t one of the official goals it is something I am at least working on developing content around at least because it just makes sense. Here is the scenario – let’s say you wanted to track failed logins over X period of time. Your choices are really a Data Monitor or Trend (or a report that runs against the raw events I guess). If you go the data monitor route you can quickly get to an active channel to see the base events…but you also have to wait for the active channel to open against the entire length of time the data monitor covers and your analysis is based on what you can visually parse line by line in the active channel. On top of that you are limited on what/how you can display with the top value counts data monitor (which is the data monitor I would use for this sort of thing) nor can you create a report based on the DM itself. Something like so which is basically an unmodified stock data monitor:

I want control over what fields to display in my initial table/view. In this case raw counts and number of unique host machines. Because this is really a query viewer I can then define a number of other query viewer drill downs to extract more data out of these numbers. This starts to get into my other post about wanting to quickly be able to query Trends or other structures in ESM like you can starting in Logger 4.5 because there will always be some new way to look at the data.

 
 
 
 
 
 
 
 
The thing is because it is a Lego set I can build it to suit my environment and needs - note the several custom drill downs.The problem is once you elevate a level of content towards a Dashboard in the client (don’t get me started on the thing they call a web console) you get locked into what Trend Dashboards can deliver. At this point that is mainly drill downs which are powerful but in a sense, limited. There aren’t bridges in place to move from a Dashboard to an Active Channel for instance (many issues relative to field mapping) nor are there variables allowing you to reach into Trends like you can with Active Lists (unknown level of difficulty) nor is there currently a way to quickly move content from the backend of one Dashboard to the next (very ambiguous developmental request). Because it is a Lego set Lord only knows how people will or have developed their content which I’d have to believe makes it just a wee bit challenging for the developers to develop all these things I want.
 
After all that is said and done the REAL issue (which I sort of intimated in the previous paragraph) is who cares about failed logins or "A"piece of content! What you really need is a way to answer the NEXT question – how does that username/system/port/source/destination/pattern relate to all my other content? I feel like it is like a scene from NCIS.

DiNozzo: Hey boss, here is a list of the top 10 usernames with failed logins from the last 24 hours! I printed it off the Dashboard (huge goofy grin on face; holding the paper forward like it is the answer to world hunger)
Gibbs: …… What systems were they on?
DiNozzo: …uhmm (grin starts to fade)
Gibbs: And how do they relate to the dropped outbound connections on the firewall….and while you are at it what internal machines have they touched? This is a SIEM that is supposed to correlate events from all our event streams right?
DiNozzo: working it boss! (runs away; grin totally gone)

The problem is putting the Legos together so that the content isn’t done in a linear, dead end fashion that resembles spokes on a wheel – always moving AWAY from the rest of the content. And to make things more interesting the data is dynamic – the top 10 x in time period Y will likely always be different. The data needs to be actionable; only "actionable" will likely change from incident to incident as the threatscape is so fluid. This is where and why you need things like filters for Trends, the ability to do conditional statements in Trend queries, etc. Why? Because the data I want is already captured! I just have a new way I need to look at it. I don’t want to have to recreate a Trend or Active List every time a new use case comes around (MAJOR PITA and time sink since I have to recreate all downstream content) and I don’t want to create another Trend to duplicate the storage. I have one rule that basically writes the same data to 3 different Active Lists and the event firing itself is captured in 2 different Trends. I don't want to have to do that.

I feel like that one sort of got away from me a bit.

I just need the pieces of my ArcSight ESM Lego set to fit together better to allow me to use a piece from this set and one or two from that set. To understand next week's issue better I will probably have to put another piece on or reorder the first 3 pieces.

Tis the season and if ArcSight was a toy what would it be?

....to not update one’s blog much. At least for me. Have been wrapped up with a number of things at work to the point where I have trouble mentally disengaging. I have said more than once that we are going to the kid’s Christmas presentations at their school when I have meant programs. At one point my wife jokingly asked if I thought they were going to have a slide deck associated with them. While I don’t brief much I do have slide decks and metrics on the brain along with everything else going on so maybe that is where it is coming from.

Of course then things slow down and hit me so that I see the magnitude of Christmas relative to itself and intertwined with Easter which is at once the other side of the coin and at the same time a transparent overlay.

Actually I have had a few thoughts about SIEM, LM, etc but they are micro thoughts more than anything else or are too contextually close to what is going on at work such that they haven’t born themselves out into blogable items. I should keep a journal or something so I can circle back to them. One that did hit me the other day - if ArcSight was a toy what would it be?

Is ArcSight hard to use?

This is a question I have a tough time with and frankly drives me a little crazy. Actually, the question can be legit; its the various offshoots of the question implying ArcSight IS hard to use that I have trouble with. Outside of a conversation of GUIs and interface ease of use issues (which certainly can make a huge difference) I mainly have 2 thoughts about the whole thing.

SIEM needs a new name

In my mind there is a point where a SIEM becomes something more than being constrained by the “S” especially as you move into a context of smaller enterprise environments. No doubt the majority of SIEM purchasers have done so with the idea of using it for security and/or compliance of one flavor or another – at least initially. Maybe another way to approach this is if SIEM is sort of the next gen implementation of log management do you have to constrain yourself to just a security based use of the product to the exclusion of the other groups within the IT shop? At the end of the day you have a box/system/appliance/magical holder of a whole lot of events. It is the security mentality you bring to those events that defines how you use those logs. If you are already tracking user movement in and out of ACL groups so that you could do security minded things like overlay that data with successful and failed logins to certain systems or keep an eye on particularly sensitive groups why not spend just a few minutes to slice and dice the data so into something a program manager can use? Alternatively if you track the source address where people have logged in you have a vestigial IDM which can be used for security stuffs but it isn’t a hard stretch to cross into BI or resource usage tracking.

Does it really take a "cyber security" mindset?

Here is the theme of a conversation between a couple of us and a couple users associated with a particular computer.

Us: It looks like this computer keeps getting infected by an infected USB drive
Them: ….
Us: Is someone taking a USB drive home, plugging it into their home box, and then plugging it back into the work PC?
Them: Well that’s weird because no one is taking a USB drive home. I don’t know what it could be.
(pause)
Them: Now that I think about it, we do have a USB drive that we use between this computer and some stand alone lab machines. Do you think that could be source?
Us: Do those machines have an anti-virus program loaded on them?
Them: …….No
Us: /smack forehead

I’m all for user training (which we do) but outside of that at some point you would think people might put two and two together when every time they plug in a particular thumb drive that they get a popup from the AV running on the box. Cracks me up everytime I think of this.

SIEM dashboards and what time slices you use to display data

This isn’t what I was going to post but I’m still toying with that one. There are some discussions, topics, and people that make me just a lil frustrated and full of hate so instead of possibly tipping my hand too far I’ll divert and post this since it seems like forever since I have posted anything. Lots going on at work and at home (I know I’m unique there right?).

One of the challenges of being in a smaller shop is having tools but maybe not having them open all the time. Anyone in a smaller shop have their SIEM console open all day while at work? One thing I struggle with for dashboards covering things like firewall drops or failed logins is what timeframe should the information cover. If you aren’t looking at the dashboard but once a day having it focus on the last hour’s events doesn’t make a whole lot of sense to me. One option is to have the timeframe cover midnight to the current time. Another, maybe more common(?), is to cover all of yesterday. The third is a continuously rolling timeframe of say the last 24hr hours. All have their strengths. Ideally you would have a multi column approach where the first shows the current count going back however far you like with subsequent columns covering previous time slices. Last hour, previous 24hrs, average per day over the last week for example. This would allow for a quick visual on a rise or fall relative to its own average. That’s a little beyond tricky to do in ArcSight (or other/many/some/lots of other SIEMs).

In the mean time one thing you can do if you have ArcSight ESM 4.5 or beyond is develop these sorts of dashboards to cover whatever time slice ya please and then develop and link a series of query viewers to perhaps break up those counts by hour over the last 24hrs. That quick drill down capability would at least allow you to dive into what otherwise is just a number.

I would be curious how others have tackled this.

Protect 10 and Logger 4.5

What does this picture have to do with the ArcSight Protect 10 user conference? Absolutely nothing. This is the mental image I had going in today to get the stints taken out from my sinus surgery last Friday. (picture is from the movie Total Recal if you aren't familar). The most amazing thing happened later this morning - I was eating something and didn't have to stop chewing and crack my mouth open to get a breath of air. Lord love a duck (not sure where that came from); its been almost 6 months since I could breathe like this.

Back from ArcSight Protect 10

ArcSight Protect 10 was a blast. Would like to write more but feeling like crap. Got CT scan and saw the ENT today; going in for sinus surgery tomorrow. Lovely.

Of Logs...and crap. Or is that the crappyness of logs?

The problem with looking through logs is its a little like looking through a whole lot of poop. Metaphorically speaking. I mean I have never really spent a whole lot of time looking through or pondering poop so I’m reaching here a bit. That isn’t to say there isn’t value in looking at it. Once you get a baseline, changes in color, volume, frequency, consistency etc can all point to a person’s general health. There comes a point though where all it is really saying is someone ate something sometime. The point is logs generally tend to fall into the same category. They are evidences of things that have happened. The problem is “what happened” doesn’t always translate well to “why something happened”. That might sound a bit crazy but walk with me a bit. The sales pitch of a SIEM vendor generally goes like this. “What if a user goes to a malicious site…or gets an infected file…or the user plugs in an infected USB, gets infected, and then the computer starts doing X, Y, and Z. Wouldn’t you want to see/alert on that?” Of course. But while the scenario sounds good what you have heard, even on a subconscious level, is the SIEM will be able to work backwards and tell you why something happened. In reality not only do you not (generally) start out knowing the computer is infected (aka why it generated the logs), the events it does create are drowning in a steaming cesspool…cessocean of crap from all over. The damn dingleberrys (ahem..sorry) are hiding in millions and millions of events.

ArcSight to be purchased by HP

Just got an email a few minutes ago saying HP and ArcSight entered a "definitive agreement" state where HP will purchase ArcSight (news release linky on AS' site). Will be interesting to see how this changes or doesn't change the user conference this weekend.  Longer term it will be interesting to see how and when this changes the ArcSight software suite.

Ironically I was RIFed by HP soon after the HP / Compaq merger years ago. Was the lowest man on the totem relative to time in the group. If it wasn't for that I wouldn't be here /shrug. (and what a long and winding road it has been from that event to "here")

Wrapping your arms around Trends: Part 3 (with a side dish of custom parameters)

What a week. Found a couple issues with our 5.0 install with varying degrees of critically. One is being fast tracked so will see what we see. While I'm confident they will be addressed, I'm also the type of guy who would just assume they are fixed yesterday. However, I also waited tables in what seems like a past life. I try to be very conscious about blaming the waiter for a burned steak. There is also a bit of thoughtful reflection as I sit here in Chick-fil-A listening to the music and chill'n while the family assault vehicle is getting an oil change across the street. It isn't the music that is causing the reflection (though it certainly can and hopefully will have that effect) as much as I wonder at the possible connection, if any, between this post and these (1 & 2) that talk about ArcSight tentatively looking for a buyer. Only time will tell what the long term effect acquisition will mean.

At any rate, as a short continuation of this "series" here is a quick report taking the concept of the first article with the Trend created from the second. Since you are pulling from this new Trend you can see all the data in one report - when it started, ended, how long it took, insert count, and ultimately if it was successful or not. Unless you enjoy mentally converting milliseconds into minutes I would recommend creating 2 variables that will convert the time to minutes. The first is to divide the milliseconds by 60,000. The next uses the round variable to convert the first variable to the closest whole minute. Is it exact? No, but unless you have a Trend that is bumping up against the hard coded time limitations for Trend query runs does it really matter?

Wrapping your arms around ArcSight Trends - Part 2

I ended part 1 of this Trend mini series by giving a quick and dirty way to see when your Trends start. While the report does have value it doesn’t show things like how long the Trend ran, how many insertions there were, or whether or not it was ultimately successful. A report that had all that data would be ideal. The problem, as such, is ArcSight doesn’t show when the Trend actually started in the event generated when the run ends. DeviceCustomDate1 shows the start of that run’s interval though if that is of value to you.

There is a common event key found in all three events (start x1, end x2 based on success or failure) which is found in deviceCustomString5. This appears to be a combination of epoch time + perhaps the number of actions performed by the ESM that day. So now you just need a rule to add the endTime + dCS5 to an active list and reference that list in a rule that looks for a Trend run completion event. Only…..you can’t pull the creation date field from an Active List in an AL lookup variable….and you can’t aggregate on the endTime field in order to add that to a field in an Active List (through 4.5 – dunno about 5.0 yet).

Wrapping your arms around ArcSight Trends - Part 1

Anyone else’s drive into work getting a little darker? I feel like the summer has FLOWN by. Before too long it will almost be back to a military 0’dark 30 to 0’dark 30 - at least relative to the sun; I'm not working additional hours :)

There are a number of things rattling around in my head I’d like to write about and even some things I indicated I would write about in previous posts. Work, family, summer, and the cumulative lost sleep effects from fighting through some weird sinus issue for the last 6 months have conspired to work against this goal. 60 back pokes and 22 shots in the arm have at least indicated it not being allergies. Good times.

For S&Gs I think this will be the start of a short series of posts/articles/whatever on Trends. My next post may or may not be the continuation lol. If it isn’t I WILL come back to it.

A little ArcSight conference breakout session self promotion

I figure since I have a blog upon which to write why not do a little promotion for the breakout session I am leading at the ArcSight 2010 User Conference.

I knew going into the 2009 User Conference that based on our daily event count ArcSight thought of us as being a medium sized company – and on the lower end at that. Certainly, I thought, we weren’t really “that” small. In short, we were. The reality is even though our event count has increased since then, we still are. At last year’s conference I was quickly struck by the number of very large companies who use this product and their multiple hundred million EPD. By and large the historic ArcSight tool and mindset has been with this sort of company in mind. One can generalize they have a 24x7 SOC with staff ready to receive alerts and respond in (near) real time. (accurate?) As a relatively small shop (average or mean sized?) we simply don’t have the same level of staffing resources. This has led us to a somewhat non-traditional alerting system.

ArcSight 2010 User Conference breakout session list has been posted

The ArcSight 2010 User Conference breakout session list is out for those that haven't seen it yet (Conference information can be found here). I really enjoyed last year's conference and am certainly looking forward to this one. If you are on the fence about going and haven't gone to one yet - send someone. Actually send 2 someones; a manager and an ArcSight admin type. This way, in theory, you can absorb and later plan at the strategic and tactical level. If you send 2 people see if your company can spring for travel for a 3rd person by tapping into ArcSight's "BOGO" deal - send 2, get the third ticket free. If you are going to be deploying or have just deployed you REALLY need to send someone(s). I mentioned to my boss last year that if we had gone it would have escalated our development/grasp of the tool by at least 3 or 4 months since the conference happened between when we made the decision to go with ArcSight and when professional services came out to install it. Granted that curve gets smaller the longer you have had the product in house and how much time/FTE you have dedicated to the tool.

Reports and what you put in them

I don't know about you but I sometimes have mixed emotions when it comes to reports. Most tend to fall into one of two buckets. Either they are a one page "executive" type summary with colored eye candy charts or they are a ridiculous number of pages showing the gory details of every port, path, or transaction between end points that is so far into the weeds you can lose sight of the bigger picture. In either case the report would probably cause you to open your SIEM in order to do further investigation because in either case a piece of the puzzle for the initial series of follow up questions are probably missing. My top whatever is this, this, and that. Great - is that distributed or focused to just one or a hand full of machines?

So what can you do to make the report itself more stand alone or actionable?

Tactically implementing Strategic goals

Yesterday my kids went to the pool. With that as the background here is a snippet of conversation between my 8 year old son and me.


Son: These Doritos chips taste hotter than normal.
Me: Are your lips chapped?
Son: ...no...my tongue is on fire!
What does that have to do with SIEM stuff? Not a thing but I find the story funny as crap. That is right up there when I asked him years ago how much he had just thrown up; he said "10." He and I did have a conversation the other day though that was slightly more relevant and has led me to the following analogy.

ArcSight's indexOf variable - how I love thee

When we first were wading through the initial honeymoon period of having ArcSight and creating content we were often stymied by trying to compare two strings. This was back in the “olden” days of 4.0 when you couldn’t just throw 2 fields into the conditions editor like you can in 4.5 (field1 = field2; field1 != field2). What about instances when you wanted to compare two fields where something like a computer name in field1 was domain\computer name and the other was COMPUTERNAME$.

Monitor vs Alert

Several weeks ago now I took some money out of the ATM and the bottom of the transaction receipt showed a balance that seemed a bit low to me. I brought it up to my wife at lunch and watched her mentally parse a multitude of variables to include the day of the month, what bills had been paid, and the times and amounts of money that move out of checking and into sub accounts. About 5 seconds later she said the amount sounded about right. This is the difference between monitoring and alerting.

Don't get me wrong. My wife isn't the type that pulls up the account information every day to check the ebb and flow of cash but by doing the bills and interacting with the accounts she had a familiarity for how and when the money does flow. The trick in my mind with a SIEM is to figure out the balance for both types of actions. Not only do you have to balance how and where analysts get data but also things like how much should be simply monitored vs alerted on, if you create reports is there too much or too little data in them to be actionable, if the report is on a particular data source can you give it some historical context or cross reference it against other data sources.

To many of those ends my new favorite report template in ArcSight is one that comes out of the box. 4 charts on one page and then a table. Most of the daily reports we have been creating were of the one table variety with a level of detail in the information - source, destinations, times, counts, etc. While we humans are good at visually picking out patterns and nuance things out of spreadsheets putting what amounts to a rolled up summary of top information with the multiple charts at the top of the report is great. Now within the report not only have you framed aspects of the data but since more detailed information is in the table section readers can pull out an IP/user name/whatever from the top page and then search for it within the report.

Converting Windows failure codes to something readable

I posted this in the content section on the ArcSight user forums (and if you are ArcSight client you are reading them right...?) but I haven't been able to take anything with antihisimine for the last week and won't be able to until after my alergy test on Monday so feeling mis-er-a-ble.

Hopefully your SIEM is capturing the reason/status/substatus codes when it comes to failed Windows login events. In ArcSight that data is generally found in deviceCustomString4. We recently had to create a series of reports that were being passed to some folks outside the more technical group who would normally just look at the codes.

To make the reports more readable I did a copy/paste job from the GREAT resource over at Ultimate Windows Security - the event encyclopedia. Then created an active list with 2 string fields making sure to index the one with the code. You do that so in your report query you can use the variable allowing you to grab a value from an active list basically doing a join. Since this calls for a string comparison with letters I chose to make the data in the AL lower case and then use another variable to take the data from customString4 and make it lower case as well. Then in your query field selection if you look under the variable section you should see BOTH fields from the Active List. Select the normal fields you want and include the unindexed field from the active list and just clean it up using the alias field in the report.

Worked like a champ though given my current state it might read like a train wreck.

Arcsight '10 User Conference - Presentation Idea

Sadly its not looking good for the presentation idea I submitted for the ArcSight '10 User Conference. Don't get me wrong - I know I am a little fish in a big pond but I really think there is some value. How much value there is relative to other items on the plate is another topic and one ultimately they will decide.

What I had hoped to talk about was the framework/system I designed to let anomalous activity "bubble up" to the surface without elaborate and extensive use cases. Anomalous activity and systems almost in a sense will triage themselves. While many large, 24x7 SOC operations probably have people watching events scroll by and react in near real time there are many SMB types out there that simply can't sustain that type of op / op tempo. Of the few companies and individuals I have interacted with who also use ArcSight or another SIEM/LM tend to fall into the category where if they don't have a 24x7 shop they spit out daily reports that are reviewed in the morning w/o really leveraging all that their SIEM can provide. There aren't a whole lot who have comfortably found the middle ground. Granted my data set for that observation is fairly small.

Don't get me wrong - this isn't a silver bullet or an especially magical use of the product. It does however provide an extensible framework that will alert users and then provides quick access to pertinent live and historically siloed data when they open the ESM console without having to rout around.

Of course one of my hidden agendas was for other experts to tell me how much better their systems are. That would give me additional insight and help me add to and refine my own. I also hoped it would spark discussion along the lines of best SIEM/ArcSight use for SMB types.

Update: I gave up hope too soon. ArcSight has accepted the presentation idea and asked me to run one of the breakout sessions. Am excited!

"Good enough" ArcSight/Use Cases

There have been a couple articles that have popped up here and there that seem to have had their base in Duncan Hoopes’ FUDsec article relative to things being “good enough” – well if not their base then a similar theme.

Last week I took some time to drill into several Win2k8 failed login events and how ArcSight was parsing them. For event 4625 (which replaced 10 Win2k3 events) I was surprised to find a rather key piece of data - sub status code – stuck in a field you can’t query on and isn’t consistent with where the same data was parsed and dumped in the corresponding Win2k3 events. What is key about these codes is it lets the reader know WHAT the condition was surrounding this particular login failure – user account doesn’t exist, account was locked out, account is currently disabled, etc. It would be easy to question why hasn’t ArcSight “fixed” this issue. The better thing to ask though to me is why hasn’t anyone over the last 2 years plus that the OS been out actually brought the issue up to ArcSight in the first place?

Maybe I look at our service contract different than others and honestly you can make an override relatively easy; bypass the issue and move on. Why not bring it up and try to get the thing fixed at a macro level….unless people don’t have content that involves that data and aren’t looking at it anyway? Is this a case of simply getting the ball on the green and wrapping your arms around just the event is good enough? Don’t get me wrong I don’t have a ton of content built around each of the sub messages but I do throw them into the Trend that is tracking failed logins where they eventually show up in multiple reports. I mean wouldn’t you want to differentiate between 600 failed login attempts but that they were because some idiot used his domain credentials on a service account and didn’t change the password vs 600 failed login attempts for an account that didn’t exist?

Hidden value of Windows 673/4769 events in your SIEM

If you are like me you are always (or in stages) on the lookout for “noise” events to filter out of the SIEM. Windows event 673 is fairly tempting in that regard. However, depending on what sources you are pulling in you can leverage these events, which are recorded on your DCs, to see PCs hitting other PCs. There are two main limitations to these events. The first is you can’t see what network resource on the destination the source is trying touch. The second is you can’t see if the attempt was successful or not. If you REALLY needed that information though you probably have the appropriate level of logging turned on at the destination anyway.

The complete guide to SIEM use cases

I started looking for information on the web relative to SIEM use cases over a year ago. Almost a self search for just-in-time learning as it were. Unfortunately the list doesn’t exist. I think I have come to terms with that fact /wipes tear. The reality is everyone’s environment is different. Different tools, different event sources, different size shops, different foci, etc etc.

Don’t get me wrong. There are nuggets “out there” just…spread around. Hopefully I can throw a few things out there as well as this progresses. Always looking for ideas whether complete or still in concept phase. Would also be interested in getting a feel for good classes out there. (hit me at m j runals at gmail dot com if you don’t want to discuss in the comments).

At any rate I spent some time yesterday marrying a couple different items we created within our ArcSight ESM. I was left with two thoughts:
1. I used a good bit of Query Viewers and variables to pull this off (looking forward to global variables with 5.0) and was left with a feeling like I was building an application within an application. The feeling was very strange though I can’t really put my finger on why….could use more sleep I guess.
2. I want a way to combine multiple queries into one trend or chart.

A brief FUD and SIEM thought

A buddy at work ran across a Wall Street Journal article that mentioned "Arcsight’s profits were up over 300% last year despite the recession and sales were up over 30%." This was basically attributed to FUD.

Of the many things you could say about those stats the one that is most on my mind is that I hope the companies that bought the software didn't just buy it to "do SIEM" or check a box and deploy it in a manner that will simply lead to failure or under utilization. At the very least I hope they have or are willing to set aside most of an FTE for care and feeding of the system.

I had a conversation with an ArcSight VP who mentioned one of their observations/frustrations is going back to a company two years after their professional services helped with an install and basically seeing no new content. To that end I think there is a very real effort within the company to make things easier and introduce some scaling (downward) for medium and smaller customers. The reality is though, imho, SIEMs aren't an InfoSec product like other point specific InfoSec solutions. In other words you buy AV, IPS, FW, etc to address a specific issue in your environment. A SIEM is really more of a platform you can use to survey normalized event data from across your network. To that end it requires effort to build and maintain content as well as effort to (crazy thought here) respond to the issues found.

Monitoring logs and SIEMs

So there are several SIEM articles that tackle SIEMs vs Log Management, reasons you might want to use one in your environment, and what you might hope to see with one. I’m kinda a simple guy though and while I have recently tried to wrap my head around the nuances between monitoring log management style vs “just” alerting and how you might use a SIEM to both ends I was hit with a rather simplistic realization - the SIEM is already monitoring the logs.

See, I said it was simplistic. Here is what I mean though. I currently have multiple tickets open with ArcSight support and having reviewed multiple sets of manager logs one of the techs was kind enough to open an additional ticket based on some things she saw. I had three takeaways:
1. I should monitor the ArcSight logs. (I’m insightful aren’t I)
2. If I had been monitoring them I would have picked up on the warnings/errors/whatever that was seen
3. My manager logs rolled every 2 hours – used to anyway.

This, to me, makes the manager logs a perfect candidate for insertion into the SIEM. Granted part of the “perfect” is that the logs are being generated by something designed to suck in logs so it doesn’t seem to make a whole lot of sense to have something other than itself actually monitor…itself. The other side of this is while there is value in actually looking at the logs periodically I think this is also a perfect case of being able to just set up alerts.

The real takeaway then is maybe an adjustment to the argument that SIEMs are an “analyst in a box.” While they aren’t really an analyst they are at least some/part/multiple/an FTE who is, or at least capable of, monitoring logs. The next takeaway is according to the 2009 Verizon Data Breach Investigations Report we should be looking at application logs anyway. I mean the OS logs I am pulling in from the server hosting the ESM certainly aren’t telling me anything about these “guardoverflow” events that I should apparently search for in my logs in order to fix some data monitors.

Tickets and more tickets

I think I overheard my ordinal placement by ticket count relative to other ArcSight customers. I don't know if I should be proud or ashamed lol.

No I'm not in the number one position.

The irony, if there is one, is that we are on the lower side of a mid range company (by events per day). Not all of mine are overly deep/hard/indepth issues but I'd like to think I don't have a whole lot in the "how is the weather in Cupertino" category either.

Random ArcSight stuff from the week

Random stuffs from the week

Apparently there aren’t a whole lot of ArcSight customers with Win2k8 DCs. I say this because when I started looking at the event stream from when we spun ours up I was shocked at how many domain (vs local) events didn’t have the client side IP in the sourceAddress field. Lovely. Granted I am sure there are probably some high speed ArcSight admins out there who have done modifications to the Unified Connector to overcome this but this last week and a half has been crazy. I also figure by sending the events into ArcSight for them to adjust the connector I have helped the overall client base (aka I’m lazy).

I HAVE to carve out time to go through the Win2k8 events anyway as I think there are some slight differences in the fields data goes into. Noticed this on some content tracking AD group membership. In many cases I tried being preemptive and added the 2008 event ID to the appropriate filters. Yeah….not so much. If you aren’t familiar with those events I highly recommend looking up Randy Franklin Smith’s Ultimate Windows Security site. The quick and dirty way to know what the new event ID will be is to add 4096 to the 2003 event.

Trends. A Trend timeout will occur at the lesser of 50% of the scheduled run time or 2 hours. I could have sworn it was 4 hours in 4.0. Actually I should be looking at a pair of reports I created to do some analysis on my Trends start, run length, and insert counts due to some issues we are seeing but the magic of the flat panel TVs displaying the menus the Tim Hortons I am in has captured my attention. Is anyone else using this technology for this ends? Is cool. Been 2 years since I have stepped inside one. Probably more on Trends in my next post assuming I haven’t wrapped my arms around some post ideas on monitoring vs alerting and SIEM use.

The oil change on the family assault vehicle is probably done and I should get out of here before I get some ice cream (Tim Hortons + Cold Stone = win (or fail depending on your point of view)).

Tip of the Day

By a pair/box of latex gloves before you change the brakes on your car. What a difference!

ArcSight's PartitionInfo.log

Move over James Patterson – Kerry “the database queen” has posted the third installment of her series on reviewing the partitioninfo.log file. Its a real page turner…well if you printed it out on multiple pages. This is a file you should be generating at least twice a month as part of a backup process even if you don’t review it (KB article 572 if you need instructions). Kerry has been the proud recipient of several “coolest chick in the world” minutes given out by me based on her decisive, insightful, and quick treatment of database related trouble tickets. She would probably score 6+ thumbs up on the Homer Simpson thumbs up scale. Actually my impression of the database support team is that they are a pretty knowledgeable lot. If you are like me and don’t know what to look for in this file Kerry’s series is good stuff.

To get to her blog go to the ArcSight community site and either go to the Browse button at the top and select blog or there is a series of buttons on the middle right side. You do regularly check the ArcSight Protect 724 community site don’t you?

Of SIEMs and Gymnastics

In thinking about SIEMs overall relative to a thought that has been building over the last month or so I had a flashback to my cheerleading days.

Don’t laugh. For a slightly introverted guy, cheerleading at a college with a girl to guy ratio of 7:1 = win.

One of the differences between a back hand spring and a back tuck is the general ability to transfer momentum from one movement into the next. This is a lot easier with a back hand spring since the motion is front to back vs the tuck’s up and down. It is one of the main reasons why you are much more apt to see a series end in a back tuck vs start with one and transition to other things.

My problem is I learned how to do a back tuck – the finishing move – first. This translated into me not really being able to do multiple back hand springs in a series.

So, how does this relate?

If you look at the maturity models from the last post, SIEMs are toward the right side. IMO then and based on the language used by the vendors I have talked with you have use cases all about alerting on X in your environment. SIEMs are (should be?) the finishing move back tucks of the InfoSec world. Based on your environment though how are you using it? Starting at alerting and backing into monitoring? Moving from monitoring into alterting? Can your SIEM do both or maybe you need just one or the other? I really think the difference between monitoring and alerting is more than just semantics.

I’m the newb here - am I off my rocker?

The maturity of your ops

I stumbled across Anton Chuvakin’s blog some time ago and have been reading it since. Back in early February he posted a maturity scale that at the time I sort of dismissed. One of the commenters posted a link to a similar, though more verbose, article they had written back in 2008.

Anton Chuvaki’s: here
Raffael Marty’s: here

(seriously…read them!)

What I had missed in the simplicity of Anton’s graph was all the soft processes, tuning, and trial and error behind each of the steps. I liken it to a 10 speed bike. You shift up too many gears and if you aren’t already going fast enough you bog down. Of course the opposite could also happen where you shift down too many gears and while you are peddling your heart out you really aren’t moving forward (the whole activity vs accomplishment issue). A lot of that is perspective and proximity to the tool at hand I suppose.

I started to write a couple other things but maybe they will show up in future posts. At any rate, the purpose of this post was really just to point people to those other two articles lol.

Query Viewers as a replacement for Active Channels?

Could you use a Query Viewer to (generically speaking) replace some of the initial visibility you get from an Active Channel. The TLDR summary is below.

As the primary content creator for our ArcSight install and since my ArcSight users don’t always have their consoles open, I often think about how best to get them to actionable data and to the point where they can dynamically interact with it in the shortest amount of time. You generally have data in Active Channels and data in Dashboards – with a sort of gulf between them. 4.5 brought an interesting feature with Query Viewers. Among other uses is the ability to tie multiple queries into a QV. Most often we use this for linking multiple buckets (Trends) of data. I see Bob’s computer/IP doing something in this bucket and with a simple right click and investigate I can see if that same IP shows up in another bucket.

The more I use ACs now though, the more I want them to act or at least have the functionality of QV. To that end I went about creating a QV that pulled 3 hours worth of events and didn’t update as a first pass at a replacement mechanism for an AC. My thought was while I would lose some of the extensibility I could at least link a number of the “buckets” I wanted to have visibility into.

TLDR Summary

I found the following issues impeded the overall goal. Others could be added to the list but then you get to the point where you might as well use an AC anyway given all that you CAN do with them.
• When you close a QV investigation panel you aren’t returned to your initial DB/QV – you are taken to the last tab in the Navigator pane on the right. This is kinda a pain if you are trying to do a quick drilldown and then get back to what you are looking at.
• QVs don’t have a horizontal scroll bar. This becomes an issue if you are trying to resize fields with long strings or if you have more than about 5 fields you want to pull up (note last bullet)
• You can’t adjust the relative position of QV drilldowns. The first one you put in is the first on the list etc. Is a pain if after you throw in a couple and want to organize it.
• The real kicker is if you have more fields in your QV then you are initially showing if you go to add them to the display while you are viewing the QV the console will lock up. At least that is what mine does. For example, I will always want to see fields like source/dest IP. I sometimes want to see zone name. My thought was to add the zone names to the QV but not initially display them.

It will be interesting to see what happens to QV in the next installment of the ESM. Hopefully some of these issues will be addressed.

ArcSight Use Cases and....Golf?

A couple weeks ago my wife was watching the Masters and I happened to see Phil Mickelson hit his second eagle in a row. The amazing part to me are all the vectors in play that led the ball to eventually fall into a hole with a 4.25 inch diameter: gravity, ball spin, force upon landing, multiple slopes on the green, etc.

I wondered if there was an analogy between that and ArcSight/SIEM use cases. The problem is I am not sure which side of the analogy they fall on.

On one hand the SIEM content creator could be the golfer and the badness you are trying to capture is the ball. You massage your content so that you see this, followed by that, followed by this other thing and then WHAM the trap springs and you have IDed badness – red flags go up and the alarm gong sounds. In that sense I often think of ArcSight as building a cantilevered mouse trap.

On the other hand the “bad guy” could be both the golfer and the golf ball. The content creator is really just focused on people who hit the green and doesn’t really care about how they got there. If the ball lands in the hole then things are really really bad but the key here is proximity to the hole is enough to alert on/be reviewed.

At a bigger picture level the two ways of looking at the analogy probably represent a SIEM centric vs Log review approach.

First Post

I entered both the InfoSec and SIEM realm about 18 months ago when I made a lateral move to be the primary administrator, innovator, and content creator for the ArcSight software we had just purchased. While I came to the table with a decent mix of IT skills they were mostly based around close desktop and network support though in a variety of environments. I have spent a lot of time searching the Interwebs for SIEM-esque material to help get over the learning curve of all that goes into enterprise level InfoSec and then turn around and create actionable ArcSight content. While I found a number of blogs and the odd resource here and there, at the end of the day there isn’t a whole lot of boiled down information out there for folks – at least with my background.

So what’s this blog about? While I’m not a SIEM expert I’d like to think I have something to bring to the bigger conversation. The other part of this is going through the mental gymnastics of converting thoughts to text. What remains to be seen is how long this little experiment will last. I have some high hopes...though I have to also learn how to navigate this blogging thingamajig