I want to be alerted when Splunkd goes down, how can I be notified?
See the previous answer about setting up an email alert if Splunk is down in production.
I'm going to try creating a cronjob that can monitor Splunks status, but as from what I can see, there doesn't seem to be a perfect (risk free) way of doing this.
All other applications I have had to deploy in my lifetime had alarming capabilities for this purpose, as i kinda makes sense to have it.
Well played sir, well played
Splunk for everything 😛
But why use Splunk? You'd be better using a dedicated monitoring solution or some other cronjob to monitor for it
Without a second system watching the first you won't notice the first system disappearing entirely, for instance when the machine goes boom or when someone takes Splunk down by using a sledgehammer...
Why use another instance? Why not just do it the usual way of a cronjob checking every X minutes and firing an email if its down/trying to restart it?
If they both go down simultaneously you likely have a larger problem.
What if they both go down?
Just run a second Splunk on a tiny system somewhere (VM?) and have each monitor each other, raising hell when one goes missing.
Yes I understand that, but there should be an alarm that's raised and is able to notify Operations when its down...
When Splunk is down there is nothing Splunk can do because, well, it's down.
Yes but that means there isn't anything application specific ready to provide a user with alerts or notifications if and when Splunk does go down.