Deployment Architecture

How to sync the cluster members of a search head?

vrmandadi
Builder

I have message popping on my UI messages " Search head cluster member (sh3) is having problems pushing configurations to the search head cluster captain (sh4). Changes on this member are not replicating to other members. We have 4 search heads with dynamic captain

I have ran the ./splunk resync shcluster-replicated-config command and also ran the /splunk clean kvstore --local . Still it is showing the error

Any other workaround for this

0 Karma

amitm05
Builder

Can you try reducing the value of conf_replication_max_push_count and give it some time to catch up with the syncing in small pushes only.

At below post, its been resolved this way -
https://answers.splunk.com/answers/539335/why-is-a-search-head-cluster-member-not-replicatin.html

Please accept as answer if it resolves your case.
Thanks

0 Karma

vrmandadi
Builder

Where can I change the conf_replication_max_push_count ,should I do It on the captain and push the changes?

0 Karma

amitm05
Builder

you can set it at server.conf. A direct manual update to the .conf file would not be replicated but however through Web OR CLI OR REST they should be.
Check this for SHC replication -
https://docs.splunk.com/Documentation/Splunk/7.3.0/DistSearch/HowconfrepoworksinSHC

0 Karma

lmvmandadi
Engager

I see the following message in the logs

ERROR ConfReplicationThread - Error pushing configurations to captain=https://hesplsrhc003com:8089, consecutiveErrors=173 msg="Error in acceptPush: Non-200 status_code=400: ConfReplicationException: Cannot accept push with outdated_baseline_op_id=d211555c171b47ccc8c7fe5f5588856096680c0e; current_baseline_op_id=c6235cd928e2196f63bb3cb292a55a2687467c46": Search head clustering: Search head cluster member (https://hesplsrhc001.com:8089) is having problems pushing configurations to the search head cluster captain (https://hesplsrhc003.com:8089). Changes on this member are not replicating to other members.

0 Karma

Vijeta
Influencer

@vrmandadi Check the splunkd logs for the resync if it happened correctly or there was some error.
Also this post although is for captain but may help - https://answers.splunk.com/answers/525189/how-to-resolve-error-error-pulling-configurations.html

0 Karma
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...