Getting an error banner "[HTTP 404] (...) [{'text':'Unknown sid.','code': None, 'type': 'FATAL'}] when searching in Splunk Web and no results

This error means that the search artifact (the file package containing the search results) requested by Splunk Web could not be found in $SPLUNK_HOME/var/run/splunk/dispatch for the search that was just dispatched.

In the error show above, the path to the real-time search artifact requested should be $SPLUNK_HOME/var/run/splunk/dispatch/rt_1302277839.33, but there is nothing there.

This problem commonly happens when the $SPLUNK_HOME/var/run/dispatch directory is hosted on a network device with a time setting behind the system clock of the operating system where splunkd is running.

What happens then is that the search artifacts are created in the dispatch directory with a modification time behind the system time known to splunkd :

[root@splunk-search-head dispatch]# pwd
/opt/splunk/var/run/splunk/dispatch

[root@splunk-search-head dispatch]# touch test ; ls -l test ; date
-rw-r--r-- 1 root root 0 Apr 22 22:16 test      <=== creation time assigned to the file by 
                                                        the NAS device : 22:16pm
Fri Apr 22 22:39:58 GMT 2011                    <=== system time at which the file was
                                                     created : 22:39pm

As you can see here, any file newly-created in the dispatch directory has a modification time 23 minutes behind the system time.

This means that any new search artifact created by an ad-hoc search will be evaluated by splunkd as having exceeded the default TTL (Time To Live) of 15 minutes and will therefore be immediately deleted.

For that reason, it is very important to ensure that the clock of any NAS device hosting the search dispatch directory is in synch with the clock of the system on which splunkd is running.