I have a dataset I just created using transaction
that shows when a particular service is down by pulling in the "service down" message and the "service up" message.
How do I plot when the service is up vs down on a graph over time?
Similar to the question here.
It just so happens the important service is my splunk indexing, and I'm tracking when the queues become so full the network ports are shut off.
I found a way to do this, but it requires a set time range, because you have to hack _time
a few times in the search. The idea is to turn the single "on" events into four events - the beginning and end times of the "on" status - and the "off" status just before and after the "on". Then use "connect points" to create the shape!
The idea is this:
starttime
and endtime
from. _time
and _time + duration
to use.starttime
and endtime
and snap them down to the 5 minute boundary.earlier
time by subtracting one 5-minute time period from starttime
and a later
time by adding one 5-minute period to endtime
earlier
and later
with 0s, and starttime
and endtime
with 1s. makemv
, and explode into different events using mvexpand
search:
index=_internal sourcetype=splunkd listening queues *blocked
| transaction host startswith="stopping" endswith="started"
| where duration>=300
| eval starttime=_time
| bucket starttime span=5m
| eval earlier=starttime-300
| eval endtime=_time+duration
| bucket endtime span=5m
| eval later=endtime+300
| eval values=earlier + "," + host + ",0|" + starttime + "," + host + ",1|" + endtime + "," + host + ",1|" + later + "," + host + ",0"
| table values
| makemv delim="|" values
| mvexpand values
| table values
| rex field=values "(?<_time>[^,]+),(?<host>[^,]+),(?<value>[01])"
| table _time host value
| timechart span=5m max(value) by host
Click image for full size view: