|
I want to build a dashboard that lists alert strings for administrators and creates some basic statistics on these generated alerts. Here's the details. My event stream comes from a bunch of agents that periodically report values for some key settings. So, I have an index full of daily updates from many agents consisting of: agentId = , settingA = foo I want to generate an alert when I detect that settingA has flipped from "foo" to "bar". I'd like to be able to generate a message that says: "Agent X has changed its setting from 'Foo' to 'Bar'." Since I have a large number of events, I need to run this alert generation report as a scheduled background report. Moreover, I'd like to be able to assign a severity code to each alert message and do a "stats count by severity". In other words, I want to generate some complex stats and charts based on the number of alerts and their severity. Any recommendations on how to implement this in an application? I know its a broad question but I know someone has implemented something similar. Thanks, Tim |
|
If you want to enrich existing data with new values (e.g. severity) you may want to explore lookup files. Your search can reference an event by a unique field (or combination of fields that would be unique). This unique identifier would be appended to a lookup file along with the new data you wish to associate with it. You would then use that look-up file when searching through that data to find its associated severity and/or other interesting metadata you've added. Thanks for the quick reponse. Here's a few thoughts on that approach: The alert messages would need to be parameterized and would contain data unique to the search result. I think it's possible to dynamically generate lookup files. Any recommendations or best practices on how to do this? One other consideration: this alert lookup file will become really big and most of the entries are only relevant for 1-2 weeks. It would be better to save the results in the a Splunk index and apply JOINs as needed. Or, I could just append the message text and severity as a field on one of the events.
(06 Oct '10, 23:22)
Tim
|
