Refine your search:

Am curious what the performance difference is between sorted and unsorted lookups (sorting by the primary search key of course), or if there is any.

asked 15 Jan '11, 00:19

hulahoop's gravatar image

hulahoop ♦
2.5k3239
accept rate: 40%


2 Answers:

Lookups are indexed (either in memory or on disk), so I doubt that there is any advantage to presorting the CSV file.

link

answered 15 Jan '11, 00:43

araitz's gravatar image

araitz ♦♦
7.0k2516
accept rate: 38%

Lookups are only indexed over a certain size:
http://splunk-base.splunk.com/answers/8326/are-lookup-tables-indexed
http://splunk-base.splunk.com/answers/10160/at-what-point-do-very-large-lookup-files-csv-get-indexed

I tested using a sorted and then randomized lookup table. The results are below.

Sorted lookup table:

Duration (seconds)      Component       Invocations     Input count     Output count
147.425                     command.lookup  302             104             104

Randomized lookup table:

Duration (seconds)      Component       Invocations     Input count     Output count
199.059                     command.lookup  301             104             104

Caveats:

  • The original lookup table was created using outputlookup
  • The random version was created using sort -R
  • For some reason, my Splunk instance has NOT indexed these lookup tables, even though they are 15 MB (over the limit)
  • Both files are gzipped
  • I ran the test several times, results were always close this output
link

answered 15 Nov '11, 08:21

supersleepwalker's gravatar image

supersleepwa...
633
accept rate: 40%

Post your answer
toggle preview

Follow this question

Log In to enable email subscriptions

RSS:

Answers

Answers + Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×188
×147
×33

Asked: 15 Jan '11, 00:19

Seen: 1,162 times

Last updated: 15 Nov '11, 08:21

Copyright © 2005-2012 Splunk, Inc. All rights reserved.