Splunk Search

Lookup table Limits

carmackd
Communicator

Is there a row or column limit for a lookup table. I currently have a lookup that has 25 columns, and 350k rows, which returns no results for the output field, but, if I reduce to two columns, and run the same search, I return results.

Tags (1)
0 Karma
1 Solution

steveyz
Splunk Employee
Splunk Employee

There is not supposed to be a limit.

Now, once a lookup table file reaches a certain size (by default about 10MB), we change the way that we index the lookup table for more efficient matching. So it is possible that there is a bug with how we index larger lookup tables.

Have you also tried reducing the lookup to say 10k rows, but still 25 columns?

View solution in original post

Genti
Splunk Employee
Splunk Employee

When i was trying and testing with lookup tables, i was under the impressino that something was not working either. The field extraction that was being done from the lookup tables were not happening.

However, if i gave splunk enough time to catch up and index the lookup table, then the fields would catch up.

This was not the behavior that i was seeing with small sized lookup tables, the fields were being shown immediately.

As a sidenote, my lookup table was on the order of 300MB, so i doubt there is a limit, however it might just require splunk a little time to catch up..

0 Karma

steveyz
Splunk Employee
Splunk Employee

There is not supposed to be a limit.

Now, once a lookup table file reaches a certain size (by default about 10MB), we change the way that we index the lookup table for more efficient matching. So it is possible that there is a bug with how we index larger lookup tables.

Have you also tried reducing the lookup to say 10k rows, but still 25 columns?

carmackd
Communicator

I see no sub directories in the lookups directory, only csv files (lookup tables). Currently in $SPLUNK_HOME/etc/system/lookups

0 Karma

steveyz
Splunk Employee
Splunk Employee

do you see that .index directory next to your lookup file?

0 Karma

carmackd
Communicator

I increased the max_memtable_bytes=200000000, which is roughly 190MB, but still couldn't the 350K row, 25 column, 100MB lookup file to work as it should. However, I trimmed the lookup down to 10 columns, but still kept the 350K row (40MB), and it worked.

0 Karma

steveyz
Splunk Employee
Splunk Employee

Yes, in limits.conf, under the [lookup] stanza, change max_memtable_bytes to a larger number.

Another thing to try is to use the original large file, and look at the directory with your lookup file. See if there is a subdirectory called .index

And see if there is any *.tsidx files in that directory. I've seen cases where the generated index files disappear for unknown reasons. You can try deleting that .index directory and running the search again and it should to re-generate an index file.

0 Karma

carmackd
Communicator

I'll try reducing row count. The original file size is around 100MB, but when I reduce the lookup to two columns, the file size is around 9MB. Is there any way to increase the 10MB size, or is that hard coded?

0 Karma

steveyz
Splunk Employee
Splunk Employee

It would also help to see how you defined your lookup in transforms.conf and props.conf (if automatically applied)

0 Karma

ftk
Motivator

Can you please post the searches you are using, both the one that works and the one that doesn't? And if possible please also post the first two or three rows, including the header row of the lookup table.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...