Getting Data In

Questions about Splunk Queues.

Genti
Splunk Employee
Splunk Employee

Why do they become blocked?
How are they related to each other? What is the hierarchy?
What does it mean for a queue to be blocked?
Does increasing queue size help?
How do queues being blocked relate to hardware on the machine?
Is there documentation anywhere on splunk.com where we can find all relevant information in one page?

Tags (2)
1 Solution

Stephen_Sorkin
Splunk Employee
Splunk Employee
  1. Queues become blocked when some (index-time) component in the system can't service data as fast as data is entering the system. For example, if date parsing can only operate at 30K E/s, and we could read data from files at >30K E/s, all upstream queues from date parsing will block.
  2. The standard index time queues and processors are: [inputs] -> parsingQueue -> [utf8 processor, line breaker, header parsing] -> aggQueue -> [date parsing and line merging] -> typingQueue -> [regex replacement, punct:: addition] -> indexQueue -> [tcp output, syslog output, http output, block signing, indexing, indexing metrics].
  3. If a queue is blocked, that means that data can't be added into the queue and the upstream pipeline will wait until it's empty.
  4. No, in steady state changing a queue's size will not materially impact when it gets blocked or system throughput characteristics. Blockage is simply caused by a mismatch of input to output rate.

View solution in original post

bong0004
Engager

Some questions are answered here:

http://wiki.splunk.com/Community:HowIndexingWorks

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee
  1. Queues become blocked when some (index-time) component in the system can't service data as fast as data is entering the system. For example, if date parsing can only operate at 30K E/s, and we could read data from files at >30K E/s, all upstream queues from date parsing will block.
  2. The standard index time queues and processors are: [inputs] -> parsingQueue -> [utf8 processor, line breaker, header parsing] -> aggQueue -> [date parsing and line merging] -> typingQueue -> [regex replacement, punct:: addition] -> indexQueue -> [tcp output, syslog output, http output, block signing, indexing, indexing metrics].
  3. If a queue is blocked, that means that data can't be added into the queue and the upstream pipeline will wait until it's empty.
  4. No, in steady state changing a queue's size will not materially impact when it gets blocked or system throughput characteristics. Blockage is simply caused by a mismatch of input to output rate.

hugocvg
Explorer

what is the recommendable relation between queues size?

0 Karma

rroberts
Splunk Employee
Splunk Employee

Is event typing also handled by the typingQueue?

0 Karma

Genti
Splunk Employee
Splunk Employee

The idea is that i would like an answers page where one can go and get most "general" questions regarding queues in one place, instead of checking multiple answers pages to get the big picture. I know that a doc is being worked on answering these questions and when completed hopefully a link to this page will be given.

0 Karma

Simeon
Splunk Employee
Splunk Employee

Thats a lot of questions at once... you should break out your questions or have a more direct one.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...