Refine your search:

What are the pros and cons to using an external lookup script vs a custom search command when the purpose is simply to augment your results with additional fields based on a given field?

Here is my scenario: I have a field that contains a hexadecimal value that contains several bit-level encoded fields (5 single bit flags, some multi-bit lookups, and a multi-bit value). I've written a python function that will take in the hex field and return a dictionary of new fields, and now I'm wondering which approach is better.

asked 22 Jun '10, 21:25

Lowell's gravatar image

Lowell ♦
11.1k91289
accept rate: 41%


One Answer:

I've tried both approaches and found the following:

External lookups:

  • Pro: Seems to be slightly faster.
  • Pro: Less inputs for the script to handle (since only unique values are passed to the script)
  • Con: Can't natively handle multi-valued field. (You can return a ";" and then split them using eval, but that may not always work.)
  • Con: Less flexible. For example, input to output must be deterministic (or static); which works for my given scenario.
  • Pro: You can setup a new external lookup script without restarting splunk. (I had to to trick splunk into reloading the metadata file to pick up my [searchscript/my_lookup.py] entry, since I don't think you can setup these permissions via the UI yet.)
  • Pro: Lookup can be setup to automatically extracted based on source/sourcetype/...

Custom Search Command

  • Pro: Full flexibility.
  • Pro: Can return multi-value fields
  • Con: Speed is a tad bit slower. (I found that enabling "streaming" did improve performance by 6x on my test query, but it's still slightly slower than the "lookup" approach)
  • Con: User doesn't have to remember the "lookup" command syntax.
  • Con: Can't be setup to run automatically.
  • Con: Fields added by the script don't seem to show up in the field picker if the search command is followed by a transforming command (like stats).
  • Con: You have to deal with getting everything setup properly via config files. Enabling getinfo does let you do more of this without as many restarts.

Please let me know if you have additional thoughts or if you find any mistakes in either of these lists.

link

answered 22 Jun '10, 21:42

Lowell's gravatar image

Lowell ♦
11.1k91289
accept rate: 41%

Post your answer
toggle preview

Follow this question

Log In to enable email subscriptions

RSS:

Answers

Answers + Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×57
×52
×34

Asked: 22 Jun '10, 21:25

Seen: 2,009 times

Last updated: 06 Jul '10, 22:23

Copyright © 2005-2012 Splunk Inc. All rights reserved.