Search results may be incomplete, peer 's search ended prematurely

[kaizidorfa]

Hello,

I'm occasionally getting the above error on splunk web but I'm not sure where to start troubleshooting it. Any tips on what could be causing it?

Thanks

[svasan_splunk]

kaizidorfa,

Are you using clustering on ec2? We have noticed some weird clock behaviour on ec2 which was causing some problems. (The peers were thought to have timed out when the clock skips backwards which it seems to do every now and then. The peers then have to re-add themselves and this forces them to reject searches with the old generation).

This is fixed in an upcoming 5.0.x version (5.0.5 i think)

[lspringer]

We had this issue just recently and it turned out to be a problem where time was drifting too far apart on the Cluster Peers. Check the status of ntp on the servers.

Specfic error:

Search results may be incomplete, peer splunksearch01's search ended prematurely. Error = master/searhhead needs to fixup/re-synchronize generation state before this peer=A64795F9-1196-4D42-FF8F-B98A2E71A719 can participate in this search [ gen=245 baseGen=246 ]

[sloshburch]

I am also seeing this but NTP looks right. Any other ideas?