I try to look into the performance of my index using SoS. I found aggregation queue seems is the bottleneck in my environment. Anyone knows what's aggregation queue about and how can we tune it effectively?
asked 27 Feb '12, 22:52
As the data travels through Splunk it goes through the following pipelines (among others): aggQueue::Merging -> typingQueue::Typing -> indexQueue::indexerPipe. If the aggregation queue (aggQueue) is the bottleneck that means that any of the processor pipelines that come after it MAY be experiencing heavy load and therefore spending too much time processing the data. The pipelines listed above are responsible for the following and tuning any of the attributes/parameters (listed alongside them) may help increase performance:
Responsible for: TCP/SYSLOG output, Block Signing, Writing to disk
Given that you indicate a problem with aggregation queue, I would start by investigating Line Merging and Timestamp Extraction. For example, a couple of settings that will give you a performance boost are:
Hope this helps.
the aggregation queue or aggQueue is were for example time extraction and line merging happens. So if it happens to be that you have a lot of strange time stamps the aggQueue has to work hard to match them. Best thing to do is check for time stamp differences and clean/fix them.
same would be for line merging.
btw: if you click the 'learn more' button on the upper right you can get some help in the S.O.S app ;)
answered 28 Feb '12, 05:56