Sunday, August 15, 2010

Wrapping your arms around ArcSight Trends - Part 2

I ended part 1 of this Trend mini series by giving a quick and dirty way to see when your Trends start. While the report does have value it doesn’t show things like how long the Trend ran, how many insertions there were, or whether or not it was ultimately successful. A report that had all that data would be ideal. The problem, as such, is ArcSight doesn’t show when the Trend actually started in the event generated when the run ends. DeviceCustomDate1 shows the start of that run’s interval though if that is of value to you.

There is a common event key found in all three events (start x1, end x2 based on success or failure) which is found in deviceCustomString5. This appears to be a combination of epoch time + perhaps the number of actions performed by the ESM that day. So now you just need a rule to add the endTime + dCS5 to an active list and reference that list in a rule that looks for a Trend run completion event. Only… can’t pull the creation date field from an Active List in an AL lookup variable….and you can’t aggregate on the endTime field in order to add that to a field in an Active List (through 4.5 – dunno about 5.0 yet).

The solution – or at least “a” solution - to this is found in Raju’s Tricks and Tips presentation from the 09 ArcSight User Conference; SN59. In short you create your rule to throw the start data to an Active List, package and export the rule, rename the .arb to .zip, edit the XML to include endTime, save and rename back to .arb, and import the package. I suggest putting a dummy field in the aggregation and action tabs of the rule to make it easier to do a find/replace in the XML for endTime. I also never quite get the whole import override part right so I just delete the originals before I import the adjusted package. Now create your rule looking for the Trend run completion, bounce that against your Active List, and through the use of an Active List lookup variable based on dCS5 bring down the start time and assign it to your field of choice. Aggregate on other fields of interest like dCN1 and dCN2 which contain insert count and query run length respectively. The whole process is kinda a pain but now all the data is in one event. As far as line items in your Active List I simply set my TTL to 3 hours. You could address the issue by having part of the second rule action remove the line but each Trend run is unique and the longest any 1 Trend is going to run is 2 hours - any longer and the ESM will automatically cause it to fail. I just let them fall of the list /shrug.

So now what? Well at this point I suggest creating a Trend to suck in all of these events allowing you to query things like your Trend activity on a historical basis, the total insert count of events into your Trends, etc. To be fair, there is a similar Trend that comes with the stock content located along the path of /All Trends/ArcSight Administration/ESM/System Health/Resources/Trend Queries but it doesn’t have the start time data. Some more specific report ideas to be covered in the next installment as this one is already long enough.

Still looking for input on things people have done to wrap their arms around their Trends. Data below to help you find the events in question.

Trend Started
deviceEventCategory = /Trend/Run/Started  (or)
deviceEventClassId = trend:100

Trend Ended
deviceEventCategory = /Trend/Run/Success | /Failure  (or)
deviceEventClassId = trend:101 | trend:102


  1. I found the QV discussion informative. I worked with ArcSight 3.0 V upgraded to 3.5 GA. I am an independent security consultant in India. Currently interviewing for a SOC position.

    You have stated QV was introduced in v 4.5
    Could you bring me up to speed on the current version of the ESM. eg Corr engine - Conditions - AGG- Actions- Threshold, I understand this at v 3.5
    most restrictive condition first to reduce engine cpu usage etc.
    Current Corr metrics?

    How is it dealt with now?

    Is smartagent flexagent the same as smartconnector and flex connector?

    what are actors ? is there any thing as actor?

    Have assets been modified?

    Pattern disc any changes.

    Please demystify this magic.

    A lot of questions, but Please post if you have the time.

    Thank you for reading and your time.

    1. Veerendra,

      Thanks for stoping by and glad you've gotten value out of some of the stuff here. Trying to figure out what has changed between 3.5 and whatever the current version is would likely be a long conversation. At the end of the day I haven't used ArcSight for almost 3 years. I'm afraid at best some of my answers would be guesses and at worst wrong so I hesitate to answer them.

      If you have access to the ArcSight user forums that would be a place to start though to get access you need to be associated with an ArcSight account. I do hope your interview goes well and while the timing is bad/short I believe the HP security user conference is set for the middle of September. Perhaps there is an option of attending.


    2. Mark,

      Thanks for the reply. Havent got exactly what I was looking for so I've been exploring SPLUNK and going through ur blog.

      I find Arcsight being difficult funny as it is a piece of software and if someone cant utilize its potential, its a big waste of money.

      Keep posting, its an interesting read.

      Pardon the grammar, im not very good at it.