This post came mostly out of a comment and thread post on the internal ArcSight forums. The same person made them both as it turns out (Vini is the man).
Here’s the issue. I believe one of the design goals for the ESM is to move to more of a Trend based Dashboard system vs Active Channels. If that isn’t one of the official goals it is something I am at least working on developing content around at least because it just makes sense. Here is the scenario – let’s say you wanted to track failed logins over X period of time. Your choices are really a Data Monitor or Trend (or a report that runs against the raw events I guess). If you go the data monitor route you can quickly get to an active channel to see the base events…but you also have to wait for the active channel to open against the entire length of time the data monitor covers and your analysis is based on what you can visually parse line by line in the active channel. On top of that you are limited on what/how you can display with the top value counts data monitor (which is the data monitor I would use for this sort of thing) nor can you create a report based on the DM itself. Something like so which is basically an unmodified stock data monitor:
I want control over what fields to display in my initial table/view. In this case raw counts and number of unique host machines. Because this is really a query viewer I can then define a number of other query viewer drill downs to extract more data out of these numbers. This starts to get into my other post about wanting to quickly be able to query Trends or other structures in ESM like you can starting in Logger 4.5 because there will always be some new way to look at the data.
The thing is because it is a Lego set I can build it to suit my environment and needs - note the several custom drill downs.The problem is once you elevate a level of content towards a Dashboard in the client (don’t get me started on the thing they call a web console) you get locked into what Trend Dashboards can deliver. At this point that is mainly drill downs which are powerful but in a sense, limited. There aren’t bridges in place to move from a Dashboard to an Active Channel for instance (many issues relative to field mapping) nor are there variables allowing you to reach into Trends like you can with Active Lists (unknown level of difficulty) nor is there currently a way to quickly move content from the backend of one Dashboard to the next (very ambiguous developmental request). Because it is a Lego set Lord only knows how people will or have developed their content which I’d have to believe makes it just a wee bit challenging for the developers to develop all these things I want.
After all that is said and done the REAL issue (which I sort of intimated in the previous paragraph) is who cares about failed logins or "A"piece of content! What you really need is a way to answer the NEXT question – how does that username/system/port/source/destination/pattern relate to all my other content? I feel like it is like a scene from NCIS.
DiNozzo: Hey boss, here is a list of the top 10 usernames with failed logins from the last 24 hours! I printed it off the Dashboard (huge goofy grin on face; holding the paper forward like it is the answer to world hunger)
Gibbs: …… What systems were they on?
DiNozzo: …uhmm (grin starts to fade)
Gibbs: And how do they relate to the dropped outbound connections on the firewall….and while you are at it what internal machines have they touched? This is a SIEM that is supposed to correlate events from all our event streams right?
DiNozzo: working it boss! (runs away; grin totally gone)
The problem is putting the Legos together so that the content isn’t done in a linear, dead end fashion that resembles spokes on a wheel – always moving AWAY from the rest of the content. And to make things more interesting the data is dynamic – the top 10 x in time period Y will likely always be different. The data needs to be actionable; only "actionable" will likely change from incident to incident as the threatscape is so fluid. This is where and why you need things like filters for Trends, the ability to do conditional statements in Trend queries, etc. Why? Because the data I want is already captured! I just have a new way I need to look at it. I don’t want to have to recreate a Trend or Active List every time a new use case comes around (MAJOR PITA and time sink since I have to recreate all downstream content) and I don’t want to create another Trend to duplicate the storage. I have one rule that basically writes the same data to 3 different Active Lists and the event firing itself is captured in 2 different Trends. I don't want to have to do that.
I feel like that one sort of got away from me a bit.
I just need the pieces of my ArcSight ESM Lego set to fit together better to allow me to use a piece from this set and one or two from that set. To understand next week's issue better I will probably have to put another piece on or reorder the first 3 pieces.
