MV fields being stored as strings would make sense, or an object containing strings and a hidden delim field at least. I wish one of the architects would be able to give a technical talk on how this stuff works more behind the scenes so we could help with more pointed debugging info.  I considered not having strict typing on the kvstore collection as being the issue, but regardless it shouldn't be returning the wrong value instead of nulls. That would bring it back to the lookup command being the issue.  I also think I have found a bug with mvmap that exhibits very similarly to this one. But since it was such an edge case I didn't even make a blog post about it. Hoping a splunk employee will see this and track down the issue behind the scenes.  ... View more

I wouldn't expect it to be an issue in a dynamically typed language either. In my other response to scelikok, I included a search where the typeof() command either isn't showing the real type of the field, or confirming it is a memory allocation/referencing issue because I think the c++ tostring() function re-allocates the variables (its been a while so I could be wrong though).  Do you know if typeof() is returning the actual type of the variable? Or is it all stored as an object behind the scenes so the best typeof() can do is guess based on the contents? Even when I am casting tostring() Splunk still calls it a number. Not calling it a float or an int makes me think it is taking a best guess based on object contents. Reposting the other response's search below: | inputlookup kvstore_560k_lines_long max=15000 | stats count by secondUID | where count=1 AND match(secondUID, "^\d+$") | head 4000 | eval initialType=typeof(secondUID) | eval secondUID=tostring(secondUID) | eval subsiquentType=typeof(secondUID) | where like(initialType, subsiquentType) ```all of the types still read as "numbers" so no events are removed``` My biggest clue that this is abnormal is the event count affecting the % of lookup error in the initial search. If I run the same events through the search that have errors initially but with a smaller search event count, they return the correct results. ... View more

Hey @scelikok,  Thanks for jumping in to look at this.  I agree that part of the problem is the typing here because it only returns the wrong lookup uID values for secondUID values that could be reasonably interpreted as numbers. And maybe this comes down to poor error handling.  I wouldn't be satisfied just writing this off to a typing issue though for multiple reasons: First (opaque), this would be completely opaque to the end user.  Maybe you have enough experience to tell me, but if it is a typing issue, then the typeof command does not return the actual types of the variables. I have to believe it returns what Splunk expects it would be. For instance in the search below, the last |like command does not remove any results.  | inputlookup kvstore_560k_lines_long max=15000 | stats count by secondUID | where count=1 AND match(secondUID, "^\d+$") | head 4000 | eval initialType=typeof(secondUID) | eval secondUID=tostring(secondUID) ```this line causes the search to return different results``` | eval subsiquentType=typeof(secondUID) | where like(initialType, subsiquentType) Second (inconsistent), casting the secondUID with tonumber() makes more wrong results be returned. So whatever is happening is not consistently happening for secondUIDs that are integers. I initially found this by rexing a dashboard token list of secondUIDs so it isn't a matter of how the variables are being initially loaded from the lookup table. This is also shown by needing the eval command to cause the error which is distilled logic from an if statement. And as far as the inconsistency goes, there is no pattern to the numbers that are treated as numbers vs strings. Its not like all numbers starting with 0s have this happen or whitespace being around some.  Third (unexpected lookup function), if you looked at any of the events they would look correct. The values for secondUID would be correct and they have results returned for uID. Unless this is just another quirk of Splunk where wrong lookups are possible without warning, I would expect the |lookup to output a null() value when it is unable to use the key it is given.  Fourth (noticing the error), because Splunk doesn't give a warning and outputs a result, anybody using dedup or stats without a predetermined output expected would likely never know this exists. The only way I noticed it was because I knew there should be a 1-1 relation between my keys and values and there wasn't at the end of the search.  Fifth (event count dependent), since there is a higher % of errors happening when the initial event count goes up without a pattern in the secondUID values being passed in other than being "\d+", this makes me worried there is a memory allocation/referencing bug behind the scenes. Something which would probably also be fixed by tostring if this was the case. Counterpoint being that it reliably returns the same wrong lookup values. Maybe it really is just poor error handling in |lookup combined with typeof() being a lie. But I don't know how to come to a definitive conclusion without having access to the Splunk code (c/c++?) behind the scenes, or analyzing Splunk while it is running to rule out an allocation/referencing issue in absence of the |lookup code... Maybe you would have other ideas on how to narrow it down. Or maybe I'm just crazy         ... View more

I know it has been a while since you asked this @axelmunoz but this might be able to help others since I couldn't find any results either and had to make something. If I made any bad assumptions please correct me. Alternate definition for an artifact: "An artifact is a saved search job that has not reached its ttl yet per search,user,app" This returns all of the artifacts SIDs then makes an event for each and uses the map command to loop through them loading in the results.  | rest /services/search/jobs | rename eai:acl.app as app | table author label app defaultTTL ttl diskUsage dispatchState isDone id isFailed isSavedSearch delegate published sid ```extra fields for human``` | where isDone=1 AND like(label,"PutSavedSearchNameHere") ```update this with the name of the search you want. probably smart to add app and user too``` | stats values(sid) as sid by label app author ```this version will be limited by values() max. Use count and mvrange instead if you need over values() max``` | mvexpand sid | map maxsearches=50000 search="| loadjob $sid$" ... View more