|
I need to index several hundred gigs of historical logs. i have a machine that is dedicated for this purpose. i installed the universal forwarder and have used the [monitor] stanza in inputs.conf to start the indexing. it is working, but it seems REALLY slow. since this server is dedicated to this purpose is there any way i can force the forwarder to use more system resources to chug through the logs at a faster pace? I'm also open to alternative solutions to this problem. |
|
There could be a few reasons as to the speed, disk IO on the forwarder, cpu, etc. However the first thing I would look at is the limits.conf file. The universalForwarder has limits for how much data it can send at a time, this may be the cause of the perceived slowness. http://www.splunk.com/base/Documentation/4.2.2/Admin/Limitsconf [thruput] maxKBps = <integer> If specified and not zero, this limits the speed through the thruput processor to the specified rate in kilobytes per second. To control the CPU load while indexing, use this to throttle the number of events this indexer processes to the rate (in KBps) you specify. On a universal forwarder, this is set to 256 by default. you guys are both right- it was my thruput. as soon as i bumped it up i could process logs way faster
(13 Sep '11, 15:00)
tpsplunk
|
|
Did you raise the http://www.splunk.com/base/Documentation/latest/Admin/Limitsconf |
