In a datasource that uses single quotes as the event delimiter, like so:
field1='value1' field2='value2' field3=''
Splunk will correctly extract value1 and value2 as just that, without the single quotes. Thus, I am able to find events that contain field1='value1' by running the search field="value1", that is, with double quotes.
Searching for field3=''
works too, but the search field3=""
fails.
Is there a setting that controls this somehow? Or do I really have to use different queries for different log types, if I want to search for an empty field value?
Do you know what splunk thinks the value of the field is? What if you use exportcsv, what's in the field? That should narrow down the nature of the fail.
My guess is that we aren't stripped the single quotes if the resulting values is empty. Therefore, the value for field3 that we store is literally ''. When it comes to searching for terms, ' is not a special character, but " is. So field3="" would search for event with a blank value for field3, but the way we extract it, it is not blank. Try using a manual regex to extract out the value of field3 and see if it works with that.
Ah fair enough. So the problem no longer is searching for the empty string, but rather how to extract it so that I can later search for it. Still seems like I can't get it done properly. I either end up with '' as a value with AUTO_KV set to auto and with AUTO_KV set to "none" it will simply not recognize that the event with the empty field have a field-value pair at all.
Sure - so the value of error_message isn't empty. It's literally two single quotes. Extraction problem.
search: "error_message='' | eval x=len(error_message)" gives the value 2 for x. For the more eclectic 2-character delimiters I was using above, it gives 4.
An easy way to test whether the field value is truly empty is to use eval or where. For example, "... | where len(field) > 0" or "... | eval length = len(field)"
As a test, I changed the delimiter completely to something Splunk would not understand by default, and set up a REGEX in transforms.conf to capture it. It captures all non-empty strings fine, but chokes on the empty string, even though the capture regex clearly captures the empty string if I run it manually over a field-value pair with an empty value field. The regex in question looks like this: REGEX = ([^\s]+)=°'(.*?)'°
The following would make the value blank:
...| replace '' with "" | ...
Or your could use field extractions instead of auto kv to make the empty field whatever you want; however, S&I will tell you that searching for field=value where "field=value" is actually in the raw event is much better.
I don't understand why field3="" would work unless the value was a literal double double-quote. What is the purpose of the actual search, maybe I could help more?
What about this: ... | replace "'*'" with "*" | ...
This has the advantage that it should strip out all the leading and trailing single-quotes for empty and non-empty fields.
In my case, I have Splunk indexing a log which contains the state changes of a type of object. That object will have certain fields that change over time, and the empty string is a perfectly valid value for these fields to assume.
One field in particular will when set to the empty string signify an important change in the object, and the most convenient way of finding that state is to search for when that field value assumes the empty string.
I am unsure if |replace is a good idea here, both due to performance and the fact that I'd need to reuse the trick manually if the need ever reappears