Thanks @isoutamo . Unfortunately that didn't help. Still stuck with these errors on my indexer logs GET /services/admin/bundles/master ---  404 This on my Master -  Upload bundle="/data/splunk/var/run/master-1001264746-1-1061363192-1605502167.bundle" to peer name=index-hio-1001264746-5-1061744048 uri=---------------- failed; http_status=400 http_description="BaseException" And in the distibuted search panel i see a similar message for all indexers. Error [00000100] Instance name "index-hio-1001264746-2-1061744028" REST interface to peer is not responding. ... View more

Yes @isoutamo , the symmkeys are the same across the all nodes. I see these errors on the search node and indexers Searchhead erver index-hio-1001264746-6-1061744054[https://10.236.18.41:8089/services/admin/bundles/search-1001264746-3-1062710635] does not support bundle version listing. Probably an older version. Giving up due to error code 404. 11-14-2020 02:58:43.205 +0000 WARN DistributedPeer - Peer:https://10.236.18.41:8089 Unable to get bundle list Master Error [00000100] Instance name "index-hio-1001264746-6-1061744054" REST interface to peer is not responding. Check var/log/splunk/splunkd_access.log on the peer. Last Connect Time:2020-11-14T03:02:48.000+00:00; Failed 11 out of 11 times.   11-14-2020 03:04:43.161 +0000 WARN DistributedPeerManager - Unable to distribute to peer named index-hio-1001264746-6-1061744054 at uri=10.236.18.41:8089 using the uri-scheme=https because peer has status=Down. Verify uri-scheme, connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information. 11-14-2020 03:04:48.169 +0000 WARN GetBundleListTransaction - Server index-hio-1001264746-6-1061744054[https://10.236.18.41:8089/services/admin/bundles/master-1001264746-1-1061363192] does not support bundle version listing. Probably an older version. Giving up due to error code 404.   ... View more

@isoutamo  On the peer/indexer i see these messages related to the master although i didn't see this continuously. 11-11-2020 02:34:54.913 +0000 ERROR CMSlave - heartbeat failure (reason: failed method=POST path=/services/cluster/master/peers/02697231-3BE7-4C09-8FB1-408B223A9466/?output_mode=json master=10.236.16.204:8089 rv=0 gotConnectionError=1 gotUnexpectedStatusCode=0 actual_response_code=502 expected_response_code=2xx status_line="Error connecting: Connection refused" socket_error="Connection refused" remote_error=) 11-11-2020 02:34:57.922 +0000 WARN CMMasterProxy - Master is down! Make sure pass4SymmKey is matching if master is running. 11-11-2020 02:35:06.644 +0000 WARN CMMessages - got genid thats invalid or out of range, setting to INVALID_GENID, jn=18446744073709552000 11-11-2020 09:04:53.119 +0000 ERROR CMSlave - heartbeat failure (reason: failed method=POST path=/services/cluster/master/peers/02697231-3BE7-4C09-8FB1-408B223A9466/?output_mode=json master=10.236.16.204:8089 rv=0 gotConnectionError=1 gotUnexpectedStatusCode=0 actual_response_code=502 expected_response_code=2xx status_line="Error connecting: Connection refused" socket_error="Connection refused" remote_error=) 11-11-2020 09:04:56.130 +0000 WARN CMMasterProxy - Master is down! Make sure pass4SymmKey is matching if master is running. 11-11-2020 09:05:07.773 +0000 WARN CMMessages - got genid thats invalid or out of range, setting to INVALID_GENID, jn=18446744073709552000    On the master I see this related the peer, 11-13-2020 06:17:10.991 +0000 WARN GetBundleListTransaction - Server index-hio-1001264746-7-1061744060[https://10.236.18.39:8089/services/admin/bundles/master-1001264746-1-1061363192] does not support bundle version listing. Probably an older version. Giving up due to error code 404. 11-13-2020 06:17:10.991 +0000 WARN DistributedPeer - Peer:https://10.236.18.39:8089 Unable to get bundle list 11-13-2020 06:18:10.994 +0000 WARN GetBundleListTransaction - Server index-hio-1001264746-7-1061744060[https://10.236.18.39:8089/services/admin/bundles/master-1001264746-1-1061363192] does not support bundle version listing. Probably an older version. Giving up due to error code 404. 11-13-2020 06:18:10.994 +0000 WARN DistributedPeer - Peer:https://10.236.18.39:8089 Unable to get bundle list 11-13-2020 06:19:10.990 +0000 WARN GetBundleListTransaction - Server index-hio-1001264746-7-1061744060[https://10.236.18.39:8089/services/admin/bundles/master-1001264746-1-1061363192] does not support bundle version listing. Probably an older version. Giving up due to error code 404. 11-13-2020 06:19:10.990 +0000 WARN DistributedPeer - Peer:https://10.236.18.39:8089 Unable to get bundle list   Thanks. ... View more

@isoutamo   splunk show config server | egrep '^(clustering|master_uri|mode)' master_uri=https://<master-ip>:8089 mode=searchhead master_uri=https://<master-ip>:8089 master_uri=https://<license-server-ip> ... View more