I need to track hosts as they move through a state diagram. Specifically, show only events that indicate state changes. Please help on the below approach, or suggest an alternate.
Say there are only two states, X and Y. Eventtype=X-Y matches events which trigger a state transition from X to Y, and similarly for eventtype=Y-X. These eventtypes are set as priority 1, so they appear first if the event matches multiple eventtypes. They also all get tag=state, such that a search for tag=state yields all events which might indicate a state change. The below finds only the state changes:
The isnull results in the first seen event triggering a transition (no previous knowledge of state).
However, what if there are more than two states, say X, Y, and Z? Now instead of simply omitting sequential events with matching eventtypes (the above search could be simplified but I was setting the stage), the actual state of the hosts has to be tracked, such that an X-Y event only triggers a transition to Y if the host is in X (omit the event as before if the host is already in Y, but also if it is in Z). Can this be done with the splunk search language?
We wrote a custom command to do this, but it is unacceptably slow. So I am pursuing a search language approach per the race results at Slow running custom search command on Splunk Answers (and this command seems inherently non-streaming). http://www.cs.sandia.gov/~jrstear/ras/ gives more info on the actual application.
Using streamstats seems insufficient, because the state decision needs to be made prior to streamstats (so the state can be passed downstream).
Please correct if wrong, or provide other ideas. Thanks.
answered 28 May '11, 10:42
I'm not completely sure I'm following, but you could collapse fields and use regular expressions in order to detect state change. As an example, I can detect systems which have recently come online via something like this:
Basically, take multiple "status" values, collapse them to a single string which looks like "Up Up Down Down Up", and then just match against that. Obviously, the regular expression can be as complex as needed (a regular expression literally being a state machine).
Hope that helps. Credit to gkanapathy for making this possibility known to me in the first place.
answered 30 May '11, 06:39