Getting an error banner "[HTTP 404] (...) [{'text':'Unknown sid.','code': None, 'type': 'FATAL'}] when searching in Splunk Web and no results

[hexx]

This error banner is displayed every time a search is run, whether from the search bar or for searches embedded in views :

No search results are displayed from the UI, but CLI searches are working.

What is going on here? How can this be resolved?

[charles_cui]

how can approach to the problem ?

[mwong]

I have a similar case about that issue. Changing the time of the machine and close to NFS server time difference. The issue has gone.

[mzax]

More detailed command to use from the NFS location:

touch var/run/splunk/dispatch/test ;  ls -l --time-style=full var/run/splunk/dispatch/test ; date

On Solaris it's:

touch var/run/splunk/dispatch/test ; ls -l -E var/run/splunk/dispatch/test ; date

[hexx · Accepted solution]

This error means that the search artifact (the file package containing the search results) requested by Splunk Web could not be found in $SPLUNK_HOME/var/run/splunk/dispatch for the search that was just dispatched.

In the error show above, the path to the real-time search artifact requested should be $SPLUNK_HOME/var/run/splunk/dispatch/rt_1302277839.33, but there is nothing there.

This problem commonly happens when the $SPLUNK_HOME/var/run/dispatch directory is hosted on a network device with a time setting behind the system clock of the operating system where splunkd is running.

What happens then is that the search artifacts are created in the dispatch directory with a modification time behind the system time known to splunkd :

[root@splunk-search-head dispatch]# pwd

/opt/splunk/var/run/splunk/dispatch

[root@splunk-search-head dispatch]# touch test ; ls -l test ; date

-rw-r--r-- 1 root root 0 Apr 22 22:16 test      <=== creation time assigned to the file by 

                                                        the NAS device : 22:16pm

Fri Apr 22 22:39:58 GMT 2011                    <=== system time at which the file was

                                                     created : 22:39pm

As you can see here, any file newly-created in the dispatch directory has a modification time 23 minutes behind the system time.

This means that any new search artifact created by an ad-hoc search will be evaluated by splunkd as having exceeded the default TTL (Time To Live) of 15 minutes and will therefore be immediately deleted.

For that reason, it is very important to ensure that the clock of any NAS device hosting the search dispatch directory is in synch with the clock of the system on which splunkd is running.

[Masa]

hexx: This is update;

limits.conf.spec says;

[search]

ttl =
* How long search artifacts should be stored on disk once completed, in seconds.
* Defaults to 600, which is equivalent to 10 minutes.