We are trying to monitor the hosts to ensure they have not stopped logging events. The search being used is
*|stats max(EVENT_TIMESTAMP) AS LastTime by WEB_SERVER
|convert timeformat="%Y%m%d %H:%M:%S" mktime(LastTime)
|eval MinutesAgo=now()-LastTime
|convert timeformat="%M:%S" ctime(MinutesAgo)
|fields WEB_SERVER, MinutesAgo
|sort -MinutesAgo
|table WEB_SERVER, MinutesAgo
I expect that there is a much more efficient way of doing this, probably going after some of the metadata that is stored in Splunk along with the log event.
Any ideas?
this search looks at the recentTime and provides a table of hosts sorted by the last time they were heard from:
| metadata type=hosts | sort recentTime desc | convert ctime(recentTime) as Recent_Time
or, you could enable the Deployment Monitor app and configure it to alert you when forwarders haven't been heard from:
http://www.splunk.com/base/Documentation/latest/Deploy/Troubleshootyourdeployment
this search looks at the recentTime and provides a table of hosts sorted by the last time they were heard from:
| metadata type=hosts | sort recentTime desc | convert ctime(recentTime) as Recent_Time
or, you could enable the Deployment Monitor app and configure it to alert you when forwarders haven't been heard from:
http://www.splunk.com/base/Documentation/latest/Deploy/Troubleshootyourdeployment