How can I improve the performance of Splunk monitoring hundreds of active files?

When installing Splunk, the default settings may not account for usage outside the norm. Monitoring many or hundreds of active files falls in this category.

There are 2 settings you can adjust in limits.conf to increase the indexing throughput when a large number of active files is involved:

[inputproc]

max_fd = <integer>
* Maximum number of file descriptors that Splunk can use in the Select Processor.
* The maximum value honored is half the current number of allowed file descriptors per process. (ulimit -n /setrlimit NOFILES)
* If a value chosen is higher than the maximum allowed value, the maximum value is used instead.
* Defaults to 32.

time_before_close = <integer>
* Modtime delta required before Splunk can close a file on EOF.
* Tells the system not to close files that have been updated in past <integer> seconds.
* Defaults to 5.

For example, these settings can increase the number of files Splunk actively monitors while reducing the rate at which Splunk recycles file descriptors:

[inputproc]
max_fd = 256
time_before_close = 2

A more in-depth discussion on Splunk’s file monitoring system follows.

In order to understand Splunk file monitoring it is useful to know:

• Each Splunk instance has a single monitoring thread
• One file descriptor is used to per source
• File descriptors are recycled once EOF is reached
• The default number of file descriptors used by Splunk is 32 (in limits.conf: max_fd = 32)
• For most Unix file systems, the max fds allocated to a single program is 1024

Splunk monitors files using a sliding window. At startup, Splunk will create the configured number of file descriptors in order to save some overhead in opening and closing fds. From this pool of fds, Splunk will begin monitoring the configured data inputs. When a fd reaches EOF, the fd is returned to the pool and immediately begins monitoring the next source in the queue.

In past versions, Splunk created one thread per source. The overhead of managing the threads and context switching defeated the performance gains of monitoring files in parallel. Ultimately, Splunk is still constrained by I/O. By using a single thread, the context switching can be avoided and Splunk can better maximize the I/O throughput.

The number of file descriptors and throughput is inversely proportional. The higher the number of fds, the lower the throughput per file descriptor. Therefore, increasing max_fd beyond a certain point will invoke diminishing returns. We believe this point to be about 256.

Please Note: File monitoring improvements in Splunk 4.1 will deliver a significant performance increase. It is not clear if this tuning will be required in 4.1.

Also Note: This tuning does not affecting indexing of gzip files. If you have many gzip files, then consider uncompressing them first to take advantage of Splunk's multi-threaded file monitoring. Splunk handles gzip files sequentially.