Hi All ,
So I have two indexers in a cluster with CM
Two SH's in a cluster with a deployer
SH cluster is connected to CM .
I see Indexer's having high CPU alerts.
Sometimes Indexer01 have 100% CPU alerts
Sometime IDX02 but not both collectively.
So I went to DMC , saw that searches are causing this issue.
So i opened two indexer's via putty (command line) and ran the top command to view the CPU utilization.
whenever i see a Indexer hitting 100%
I opened the dispatch folder at
/opt/splunk/var/run/splunk/dispatch
and ran
find . -name "alive.token"
What i found out is , whenever the acceleration searches are running the cpu is hitting 100% on that particular indexer.
My question is :
1.Why my acceleration searches are only running on only one of two indexer's why not both collectively?
If i see they are running on IDX01 , the IDX02 dispatch directory doesnot have any alive searches or cpu usage is very low.
If i see they are running on IDX02 ,the IDX01dispatch directory doesnot have any alive searches or cpu usage is very low.
2.I am trying to draft a search to count the number of jobs ran on any Indexer.
I took the search_id from the dispatch folder and searched in splunk.
I got events from _audit and _internal , the problem is I dont see any field saying on which indexer the search job has run.
[as the _audit and _internal indexes are replicated among the cluster , i cant differentiate the internal logs of the indexer's]
Please give your thoughts.
P.S : it is a multisite cluster .
SH01 and IDX01 are on site1
SH02 and IDX02 are on site2
I thought search affinity is the problem.
But as per search affinity , if the search is triggered by SH01 it will run on IDX01 mostly.
But here i see search jobs are triggered by both SH01 and SH02 [i have known this from the search jobs naming convention in the dispatch folder ]and running only on either of the indexers. (edited)
... View more