'NOT' not working properly since 4.3

Question

After upgrading to 4.3 I noticed one of my timecharts was not working correctly:

searchterm NOT port=16 | timechart foo

UPDATE CLARIFICATION: The above worked fine on 4.0, 4.1 AND 4.2 without any changes to configuration being required. My question is about new behaviour as shown by 4.3

I eventually figured out that this was because of the NOT port=16 part.

In both the advanced charting & standard search views, the above term is removing far, far, far more events than just those where the field port is equal to the value 16.

Any suggestions about why this is the case?

Here is an example:

Searching for index=hosttype host=hosttypel gives

_raw,_time,date_hour,date_mday,date_minute,date_month,date_second,date_wday,date_year,date_zone,eventtype,host,index,linecount,port,punct,retry_indicator,source,sourcetype,splunk_server,station,success_indicator,tag::host,time_taken,timeendpos,timestartpos
"                station1,10/01/2012-00:06:56,063,1,0",1326154016,0,10,6,january,56,tuesday,2012,local,poll_result,hosttype1,hosttype,1,1,"________________,//-::,,,",1,/home/splunk/logs/hosttype1/port1.stat,source-stat,vmhost-splunk,station1,0,"hosttype1,polling_local,polling_total",63,50,31
"                    station2,10/01/2012-00:06:27,037,0,0",1326153987,0,10,6,january,27,tuesday,2012,local,poll_result,hosttype1,hosttype,1,7,"____________________,//-::,,,",0,/home/splunk/logs/hosttype1/port7.stat,source-stat,vmhost-splunk,station2,0,"hosttype1,polling_local,polling_total",37,50,31
"                    station3,10/01/2012-00:06:15,041,0,0",1326153975,0,10,6,january,15,tuesday,2012,local,poll_result,hosttype1,hosttype,1,1,"____________________,//-::,,,",0,/home/splunk/logs/hosttype1/port1.stat,source-stat,vmhost-splunk,station3,0,"hosttype1,polling_local,polling_total",41,50,31
"               station4,10/01/2012-00:05:32,043,1,0",1326153932,0,10,5,january,32,tuesday,2012,local,poll_result,hosttype1,hosttype,1,1,"_______________,//-::,,,",1,/home/splunk/logs/hosttype1/port1.stat,source-stat,vmhost-splunk,station4,0,"hosttype1,polling_local,polling_total",43,50,31
"                    station5,10/01/2012-00:05:30,057,1,1",1326153930,0,10,5,january,30,tuesday,2012,local,poll_result,hosttype1,hosttype,1,7,"____________________,//-::,,,",1,/home/splunk/logs/hosttype1/port7.stat,source-stat,vmhost-splunk,station5,1,"hosttype1,polling_local,polling_total",57,50,31
"              station6,10/01/2012-00:05:29,036,1,0",1326153929,0,10,5,january,29,tuesday,2012,local,poll_result,hosttype1,hosttype,1,5,"______________,//-::,,,",1,/home/splunk/logs/hosttype1/port5.stat,source-stat,vmhost-splunk,station6,0,"hosttype1,polling_local,polling_total",36,50,31
"                station7,10/01/2012-00:05:06,037,1,0",1326153906,0,10,5,january,6,tuesday,2012,local,poll_result,hosttype1,hosttype,1,8,"________________,//-::,,,",1,/home/splunk/logs/hosttype1/port8.stat,source-stat,vmhost-splunk,station7,0,"hosttype1,polling_local,polling_total",37,50,31

Searching for index=hosttype host=hosttypel NOT port=16 gives no results.

EDIT: So as suggested in the first answer, 'port!=16' seems to work fine. Why doesn't 'NOT'? Its still in the documentation? Is there anywhere I can file a bug?

Accepted Answer

3

hexx was on the right track - the problem is that you are extracting the port from the source file path. By default, Splunk will look for the value of the extracted field in the event's text. In cases where it is not in the event text, you need to use fields.conf:

[port]
INDEXED_VALUE=false

This tells Splunk that the value of port is not in the indexed data. Once you do this, you still might need to change the field name wherein the port is extracted from the event itself - perhaps call it port_number. Then your search could be:

searchterm NOT port=16 NOT port_number=16 | eval port=if(isnotnull(port), port, port_number) | timechart foo

link

answered 12 Jan '12, 09:34

araitz ♦
7.9k●3●9●25
accept rate: 46%

edited 13 Jan '12, 10:18

So why doesn't this still work properly with the data where port was extracted from the event itself? Why does != work but NOT doesn't? I'm going to vote you down because of this until I know better. Your "answer" involves too much work which should be unnecessary in an ideal world. @milestulett is currently in the lead, even if you voted him down.

(13 Jan '12, 01:08) vaijpc

2

You have the right to your own opinion, but your comment above doesn't make much sense. fields.conf has been required for fields extracted from non-indexed data for several versions of Splunk.

(13 Jan '12, 09:09) araitz ♦

So why does NOT port=16 behave differntly to port!=16? Why did I not need to do what you are suggesting in 4.0, 4.1 OR 4.2? This is the first time it has been an issue with identical setup.

(13 Jan '12, 09:22) vaijpc

Also, I'm not sure you should accept your own answer on my question. I've undone your acceptance until we can get the details worked out.

(13 Jan '12, 09:23) vaijpc

2

The reason it behaves differently, as I previously stated, is because you have some values of the port field extracted from the source, which is not an indexed value, and some extracted from indexed values. This is the way Splunk is expected to work.

I was just able to confirm this on 4.3 - if I have fields extracted from either indexed values (exclusively) or non-indexed values (exclusively), both NOT and != function as expected. If I have a field that is a mishmash of indexed and non-indexed values, it does not work, as expected. If this worked before 4.3, I don't think it should have.

(13 Jan '12, 10:18) araitz ♦

Aha right, thanks for the explanation. So I guess there's a good possibility that I will lose out in the future with more complex searches involving NOT where != isn't good enough. Not exactly the most ideal situation. I'll go for the 'indexed=false' and see how things go. Thanks for the help.

(13 Jan '12, 13:50) vaijpc

3

Put another way:

By default, Splunk is looking for the term "16" in the index, then looks for the field "port" with value "16". This is why fields.conf is required - to tell Splunk "don't bother looking for the term 16 in the index".

The search "port!=16" tells Splunk just to look for events with the field "port" with values not equal to "16", in effect achieving the same effect as fields.conf.

I also considered that bloom filters (new in 4.3) might be the cause, but running './splunk cmd searchtest "search NOT port=15"' did not indicate that the filters were excluding any buckets.

(13 Jan '12, 14:04) araitz ♦

1

You could also use "NOT port=16 OR port!=16" which should cover you for both cases without having to edit conf files. Just in case you're interested :)

(15 Jan '12, 13:12) milestulett

Thanks @araitz, that explains the behaviour. I do remember that "feature" coming up before and wondering whether it was the correct word between my quotes.. Thanks for the help!

(16 Jan '12, 01:13) vaijpc

showing 5 of 9 show 4 more comments ▼

Answer 2

1

Perhaps try port!=16. This is shown in some of the training videos (which use 4.3).

ie: searchterm port!=16 | timechart foo

link

answered 11 Jan '12, 22:55

milestulett
31●4
accept rate: 14%

edited 12 Jan '12, 19:37

This seems to work... I might accept it later but for now I'm a little miffed that 'NOT' doesn't work despite it being in the documentation. Is this in the documentation? Can I rely on it?

(12 Jan '12, 01:16) vaijpc

It might be that one is interpretive, while the other one is blind? Maybe I'm making things up, but that's the only explanation I can come up with at the moment..?

(12 Jan '12, 19:43) milestulett

'NOT' not working properly since 4.3

Follow this question

Splunk Resources

Related questions