Back in the saddle...and using Splunk!

Been a long time since writing and lots has changed! New house, cataract surgery (at 39!), new job, developing and implementing the log management and event correlation program at a large edu, etc. On that front a large university is very different than a large corporation. Take all of those issues you have with decentralization and multiply that by about 50. I debated between believable and hyperbole there and ended up somewhere in the middle I guess.

One of the bigger changes is using Splunk for the first time. At some point I should write some comparisons between ArcSight, Symantec MSSP, and Splunk. Granted it has been a few years since I’ve used ArcSight’s Enterprise Security Manager (ESM) or Logger for that matter though I remember ESM fondly and often wistfully. It still amazes me when I hear people say things like ArcSight is too hard. I maintain that for most of those cases they somehow had the idea (or were told by the sales people) that it was some magical plug and play cross between a one armed bandit and magic 8-ball. I also hear in the latest update to Symantec’s MSSP they have implemented a number of the dozens of changes I suggested (am sure others as well!) and it is now much better than what it was 6 months ago.

If there is one piece of advice I could give it would be install Splunk today. You may not use it a ton in an active way but at the very least I wish I had had it for all of those times I had large Excel spreadsheets to massage in an effort of extracting value.

Welp as was the case when I first started writing this was written while waiting for the oil to be changed in one of my vehicles. That is complete and this sort of went in a different direction than planned. In my next post I plan to talk about a few ways I’ve found in Splunk to look for fields that don’t exist. As an aside it is somewhat interesting to think about writing Splunk stuffs in somewhat of the same way I used to write about ArcSight stuffs. The main driver back then was there wasn’t much out there at all for ArcSight and LM/SIEM in general and the ArcSight forums were gated to the general public. That isn’t the case with Splunk (gated community) and sadly the act of running the Splunk implementation and general ho-ha leaves me far less time to actually DO something in Splunk than I’d like.

Full of hate...

Point productivity solutions in an enterprise environment drive me crazy.

Starting to Splunk!

Well I’m off to the Splunk conference.  Having only started using Splunk just over a month ago I can say there is quite a lot to digest and frankly I’m feeling a little overwhelmed. The challenge is starting from step 0 with a good bit of unstructured data knowing in the next year or so that will grow to an estimated 700GB/day. Part of the challenge is not knowing the full implication of choosing different methods to actually do things in splunk, like field extraction, in a way that doesn’t artificially limit or cause issues down the road. This is all while developing a program to handle it all toward multiple ends. In some respects I’m going from using an MSS to being an MSS. New job + new tool + new house (that we are doing renovation work on) = good times. And just for kicks I'm building, and have talked a few others into, building a plywood canoe that we can race each other in. 

At any rate I’m excited to be going and hope to accelerate the learning curve dramatically. If anyone has any Splunk tips I’d be interested in em!

A cappella log management

A curious thought hit me the other day in church; hopefully I can articulate it well enough to be understood – even by me. For those that aren’t familiar there is often full congregational signing as part of the worship service. As you might expect this includes musicians playing various instruments, several folks up on the stage signing as part of the choir, and in the olden days a hymnal in your hands that has the words. I’m going to murder terms but to translate it in my head you have the technical musicians translating symbols on a page of music into sounds on their instruments, the next layer up are the signers who combine the lyrics with the symbols to know how to sing the words, and then you have a presentation layer where the rest of the congregation is able to see the words and take their singing cues from the music and choir. More often than not these days those words are presented on a projector screen without the musical score which works for me since I can’t read that bit anyway. At any rate there is a cumulative layering effect that allows folks like me who aren’t musically inclined or trained to participate. (Is there a musical OSI equivalent?)

The difference a couple weeks ago was the music was done a cappella and only a small number of the choir was singing. While I could see the words on the projector I was much slower to catch on to the “tune.” I and many others didn’t sing as loudly that day (this is a good thing in my case).

So what’s the link you ask? Going through the process of leaving my job to pursue another I have been somewhat introspective in thinking of what I could have done differently to try to explain the need for log management and where that fits into larger discussions of security strategy, incident detection, incident investigation, monitoring, etc. By and large I think the message wasn’t understood the higher you go up the management chain. And while I’m somewhat saddened and disappointed by that fact I’m not going to beat myself up over it. I tried several times and several different ways to communicate the need and why certain paths were better than others not only for immediate needs but also for where that positioned us 6 months, a year, several years down the road.

If you are reading this blog you, like me, can at least at a conceptual level look at something like an incident detection use case and probably see all or at least a lot of what must go into what is required for the alert to be tripped from a log collection perspective, what needs to happen once it goes off, how that use case fits into the larger picture, etc - much like my wife can look at a sheet of music and hear the music in her head. The challenge I pose to us all is - are we communicating our message a cappella in that our audience is perhaps just seeing “the words.” Something to think about the next time you do a presentation perhaps.

Couple random thoughts strung together

Oyie – I look up again and I haven’t written anything in a couple months. Amazing how your environment will (or won’t) influence you. I have had a few things on my mind for a bit recently though and figured I would try to knock out brief blurbs on them now.

• One thousand points of light….and no illumination
• A visual of effort vs value/return
• Integration is key

Where do you fit on the chart?

It’s been too long since I have written. The problem, as it were, is I have had some ideas to write about but they are either incomplete or they are still percolating in mah head. I guess that isn’t too far from being the same thing. At any rate, the other day….actually the other week now that I think about it…I was working on some visualization and metric stuffs and had the thought to create the following tongue in cheek quadrant chart.

Tickets and more tickets

I think I overheard my ordinal placement by ticket count relative to other ArcSight customers. I don't know if I should be proud or ashamed lol.

No I'm not in the number one position.

The irony, if there is one, is that we are on the lower side of a mid range company (by events per day). Not all of mine are overly deep/hard/indepth issues but I'd like to think I don't have a whole lot in the "how is the weather in Cupertino" category either.

Random ArcSight stuff from the week

Random stuffs from the week

Apparently there aren’t a whole lot of ArcSight customers with Win2k8 DCs. I say this because when I started looking at the event stream from when we spun ours up I was shocked at how many domain (vs local) events didn’t have the client side IP in the sourceAddress field. Lovely. Granted I am sure there are probably some high speed ArcSight admins out there who have done modifications to the Unified Connector to overcome this but this last week and a half has been crazy. I also figure by sending the events into ArcSight for them to adjust the connector I have helped the overall client base (aka I’m lazy).

I HAVE to carve out time to go through the Win2k8 events anyway as I think there are some slight differences in the fields data goes into. Noticed this on some content tracking AD group membership. In many cases I tried being preemptive and added the 2008 event ID to the appropriate filters. Yeah….not so much. If you aren’t familiar with those events I highly recommend looking up Randy Franklin Smith’s Ultimate Windows Security site. The quick and dirty way to know what the new event ID will be is to add 4096 to the 2003 event.

Trends. A Trend timeout will occur at the lesser of 50% of the scheduled run time or 2 hours. I could have sworn it was 4 hours in 4.0. Actually I should be looking at a pair of reports I created to do some analysis on my Trends start, run length, and insert counts due to some issues we are seeing but the magic of the flat panel TVs displaying the menus the Tim Hortons I am in has captured my attention. Is anyone else using this technology for this ends? Is cool. Been 2 years since I have stepped inside one. Probably more on Trends in my next post assuming I haven’t wrapped my arms around some post ideas on monitoring vs alerting and SIEM use.

The oil change on the family assault vehicle is probably done and I should get out of here before I get some ice cream (Tim Hortons + Cold Stone = win (or fail depending on your point of view)).

Tip of the Day

By a pair/box of latex gloves before you change the brakes on your car. What a difference!