Wrapping your arms around ArcSight Trends - Part 2

I ended part 1 of this Trend mini series by giving a quick and dirty way to see when your Trends start. While the report does have value it doesn’t show things like how long the Trend ran, how many insertions there were, or whether or not it was ultimately successful. A report that had all that data would be ideal. The problem, as such, is ArcSight doesn’t show when the Trend actually started in the event generated when the run ends. DeviceCustomDate1 shows the start of that run’s interval though if that is of value to you.

There is a common event key found in all three events (start x1, end x2 based on success or failure) which is found in deviceCustomString5. This appears to be a combination of epoch time + perhaps the number of actions performed by the ESM that day. So now you just need a rule to add the endTime + dCS5 to an active list and reference that list in a rule that looks for a Trend run completion event. Only…..you can’t pull the creation date field from an Active List in an AL lookup variable….and you can’t aggregate on the endTime field in order to add that to a field in an Active List (through 4.5 – dunno about 5.0 yet).

The solution – or at least “a” solution - to this is found in Raju’s Tricks and Tips presentation from the 09 ArcSight User Conference; SN59. In short you create your rule to throw the start data to an Active List, package and export the rule, rename the .arb to .zip, edit the XML to include endTime, save and rename back to .arb, and import the package. I suggest putting a dummy field in the aggregation and action tabs of the rule to make it easier to do a find/replace in the XML for endTime. I also never quite get the whole import override part right so I just delete the originals before I import the adjusted package. Now create your rule looking for the Trend run completion, bounce that against your Active List, and through the use of an Active List lookup variable based on dCS5 bring down the start time and assign it to your field of choice. Aggregate on other fields of interest like dCN1 and dCN2 which contain insert count and query run length respectively. The whole process is kinda a pain but now all the data is in one event. As far as line items in your Active List I simply set my TTL to 3 hours. You could address the issue by having part of the second rule action remove the line but each Trend run is unique and the longest any 1 Trend is going to run is 2 hours - any longer and the ESM will automatically cause it to fail. I just let them fall of the list /shrug.

So now what? Well at this point I suggest creating a Trend to suck in all of these events allowing you to query things like your Trend activity on a historical basis, the total insert count of events into your Trends, etc. To be fair, there is a similar Trend that comes with the stock content located along the path of /All Trends/ArcSight Administration/ESM/System Health/Resources/Trend Queries but it doesn’t have the start time data. Some more specific report ideas to be covered in the next installment as this one is already long enough.

Still looking for input on things people have done to wrap their arms around their Trends. Data below to help you find the events in question.

Trend Started
deviceEventCategory = /Trend/Run/Started  (or)
deviceEventClassId = trend:100

Trend Ended
deviceEventCategory = /Trend/Run/Success | /Failure  (or)
deviceEventClassId = trend:101 | trend:102