Is there a row or column limit for a lookup table. I currently have a lookup that has 25 columns, and 350k rows, which returns no results for the output field, but, if I reduce to two columns, and run the same search, I return results.
There is not supposed to be a limit.
Now, once a lookup table file reaches a certain size (by default about 10MB), we change the way that we index the lookup table for more efficient matching. So it is possible that there is a bug with how we index larger lookup tables.
Have you also tried reducing the lookup to say 10k rows, but still 25 columns?
When i was trying and testing with lookup tables, i was under the impressino that something was not working either. The field extraction that was being done from the lookup tables were not happening.
However, if i gave splunk enough time to catch up and index the lookup table, then the fields would catch up.
This was not the behavior that i was seeing with small sized lookup tables, the fields were being shown immediately.
As a sidenote, my lookup table was on the order of 300MB, so i doubt there is a limit, however it might just require splunk a little time to catch up..
There is not supposed to be a limit.
Now, once a lookup table file reaches a certain size (by default about 10MB), we change the way that we index the lookup table for more efficient matching. So it is possible that there is a bug with how we index larger lookup tables.
Have you also tried reducing the lookup to say 10k rows, but still 25 columns?
I see no sub directories in the lookups directory, only csv files (lookup tables). Currently in $SPLUNK_HOME/etc/system/lookups
do you see that
I increased the max_memtable_bytes=200000000, which is roughly 190MB, but still couldn't the 350K row, 25 column, 100MB lookup file to work as it should. However, I trimmed the lookup down to 10 columns, but still kept the 350K row (40MB), and it worked.
Yes, in limits.conf, under the [lookup] stanza, change max_memtable_bytes to a larger number.
Another thing to try is to use the original large file, and look at the directory with your lookup file. See if there is a subdirectory called
And see if there is any *.tsidx files in that directory. I've seen cases where the generated index files disappear for unknown reasons. You can try deleting that
I'll try reducing row count. The original file size is around 100MB, but when I reduce the lookup to two columns, the file size is around 9MB. Is there any way to increase the 10MB size, or is that hard coded?
It would also help to see how you defined your lookup in transforms.conf and props.conf (if automatically applied)
Can you please post the searches you are using, both the one that works and the one that doesn't? And if possible please also post the first two or three rows, including the header row of the lookup table.