I have a report which searches through an index of extracted database rows and performs transaction on them as well as some other eval processing, such as flattening multivalue fields and establishing a new column based on the result. The database is big and grows day by day with the new changes being fed into the index through a 30-min cron job on splunk.
The search was taking quite long so I moved the transaction command to another process as detailed in http://answers.splunk.com/answers/187124/collecting-results-from-a-transaction-command-does.html but the search is still taking a while:
This search has completed and has returned 10,000 results by scanning 363,488 events in 212.374 seconds.
Execution costs
Duration (seconds) Component Invocations Input count Output count
22.146 command.eval 288 2,907,896 2,907,896
0.033 command.fields 36 363,487 363,487
11.998 command.presort 1 363,487 293,488
94.876 command.search 288 2,544,414 2,907,901
85.51 command.search.kv 32 - -
7.297 command.search.filter 252 - -
1.431 command.search.rawdata 32 - -
0.036 command.search.fieldalias 32 363,488 363,488
0.036 command.search.typer 36 363,488 363,488
0.035 command.search.tags 36 363,488 363,488
0.032 command.search.calcfields 32 363,488 363,488
0.032 command.search.lookups 32 363,488 363,488
0.02 command.search.summary 36 - -
0 command.search.index.usec_1_8 2 - -
0 command.search.index.usec_4096_32768 8 - -
0 command.search.index.usec_512_4096 23 - -
0 command.search.index.usec_64_512 5 - -
0 command.search.index.usec_8_64 15 - -
0.199 command.sort 1 1,550,569 10,000
15.166 command.table 1 363,487 1,914,056
0.014 dispatch.check_disk_usage 14 - -
0.054 dispatch.createProviderQueue 1 - -
13.166 dispatch.emit_prereport_files 32 - -
0.323 dispatch.evaluate 1 - -
0.329 dispatch.evaluate.search 8 - -
0.008 dispatch.evaluate.eval 8 - -
0.001 dispatch.evaluate.sort 1 - -
0.001 dispatch.evaluate.table 1 - -
116.744 dispatch.fetch 37 - -
50.53 dispatch.preview 7 - -
24.696 dispatch.preview.command.table 7 840,261 4,027,761
23.473 dispatch.preview.command.presort 7 840,261 675,262
0.875 dispatch.preview.command.sort 7 3,170,000 70,000
0.397 dispatch.preview.write_results_to_disk 7 - -
0.001 dispatch.reduce 1 - -
116.736 dispatch.stream.local 36 - -
2.191 dispatch.writeStatus 42 - -
0.094 startup.handoff 1 - -
as you can see the dispatch is taking the lion's share of processing time. How can I optimise this?
... View more