Events of Interest from a Splunk Admin perspective

As our deployment has grown from basically myself performing administrative duties to adding a second body as well as running into a couple other things we've wanted increased visibility when certain activities have taken place. Over time we've merged these into a rather ugly query. It has proved itself a time or three though so figured I'd share. In putting something like this into place you need to figure out a few things. How frequently it will run, who it goes to, etc. You might find you want certain aspects to run more frequently than others which would require more than one query. YMMV

Actions this query captures:

• Splunk forwarder directory out of space (or possibly an inode issue)
• When the Splunk delete command has been used
• Failed login attempts (webui)
• When the serverclass.conf file is referencing an app/config package that doesn't exist
• Local Splunk accounts that have been created or deleted

What administrative events is the query missing?

(index=_internal "No space left on device") OR (index=_audit "|  delete" NOT "index=*_audit") OR (index=_audit action="login attempt" info=failed sourcetype="audittrail") OR (index=_internal source=*splunkd.log component=serverclass warn NOT "machineTypes in app * is deprecated") OR (index=_audit action=edit_user (operation=create OR operation=remove)) | eval Alert = case(action="edit_user" AND operation="create", "User account created", action="edit_user" AND operation="remove", "User account deleted", match(_raw, "Unable to load application"), "Serverclass.conf issue", match(_raw, "delete"), "Delete used", action="login attempt" AND info="failed", "Failed local login", match(_raw,"No space left on device"), "No space on device", 1=1, "fix me" ) | eval Message = case(Alert="User account deleted", "User: " .user. "  Deleted: " .object, Alert="User account created", "User: " .user. "  Created: " .object, Alert="Failed local login", "User: " .user, Alert="Delete used", "User: " .user. "  Search: " .search, Alert="Serverclass.conf issue", message. " (Probably a spelling issue)", Alert="No space on device", "Diskspace or inodes issues",  1=1, "fix me") | eval a_time = strftime(_time,"%m/%d/%y %k %p") | stats count by a_time host Alert Message