I am having a problem with the CLEAN_KEYS property.
Here is my raw message that gets appended to my log file. Note: the keys are consist of just numeric values.
Raw Message:
8=FIX.4.4,9=125,35=D,56=BLPUS,49=FMRFIBOOM,52=20120502-08:43:02,34=5662,54=1,55=GOOG3,38=15200,44=20,11=Order1,21=2,60=20120502-08:43:02,40=2,10=219,
It appears that the fields do not get recognized as they are all numeric.
We set the CLEAN_KEYS property to false expecting that we Splunk would successfully extract the numeric values (8, 9, 56 etc) as fields. However, it does not do this. Are our expectations wrong?
We have noted that if we change the message to include an character after the numeric value, i.e.
8a=FIX.4.4,9b=125,35c=D,56d=BLPUS,49e=FMRFIBOOM,etc. - splunk will recognize a, b, c as fields
I just tried something similar to what you did (just a regular REPORT-extraction), and it worked fine. As you can see, the the numeric field is not in the event per se, but deliberately set to a numeric name in the extraction process. Getting the log level from lines like;
02/May/2012 12:22:41:925 [CRITICAL] some text comes here
02/May/2012 13:02:35:257 [ERROR] some text comes here
in props.conf
REPORT-blaha = unclean_99
in transforms.conf
[unclean_99]
CLEAN_KEYS = 0
MV_ADD = 0
REGEX = \s\[(?<99>\w+)
UPDATE:
I don't believe that CLEAN_KEYS
should go into props.conf
.
See the docs for transforms.conf and field extraction in general
You could also do it inline in the search with extract.
UPDATE AGAIN:
As I said before, you should not put CLEAN_KEYS
in props.conf
.
Do as above, but for the transforms.conf
settings instead do;
[unclean_99]
CLEAN_KEYS = 0
DELIMS = ",", "="
This tells Splunk that key/value pairs are separated from each other by a comma, and that keys are separated from their values by an 'equals' character. The rest should go automatically, but you might get strange results if your events contain commas elsewhere (like in timestamps).
This worked fine for me with the following log file data (though I used semicolons instead of commas since the timestamp has a comma).
2012-05-03 15:55:17,89; 66=21963; 55=590621239; 44=FX.74
2012-05-03 15:55:17,90; 66=29886; 55=97211346; 44=FX.8411
2012-05-03 15:55:17,90; 66=27491; 55=3176432533; 44=FX.1282
2012-05-03 15:55:17,91; 66=31710; 55=74392336; 44=FX.29486
This gives me fields 66
, 55
and 44
Hope this helps,
Kristian
I just tried something similar to what you did (just a regular REPORT-extraction), and it worked fine. As you can see, the the numeric field is not in the event per se, but deliberately set to a numeric name in the extraction process. Getting the log level from lines like;
02/May/2012 12:22:41:925 [CRITICAL] some text comes here
02/May/2012 13:02:35:257 [ERROR] some text comes here
in props.conf
REPORT-blaha = unclean_99
in transforms.conf
[unclean_99]
CLEAN_KEYS = 0
MV_ADD = 0
REGEX = \s\[(?<99>\w+)
UPDATE:
I don't believe that CLEAN_KEYS
should go into props.conf
.
See the docs for transforms.conf and field extraction in general
You could also do it inline in the search with extract.
UPDATE AGAIN:
As I said before, you should not put CLEAN_KEYS
in props.conf
.
Do as above, but for the transforms.conf
settings instead do;
[unclean_99]
CLEAN_KEYS = 0
DELIMS = ",", "="
This tells Splunk that key/value pairs are separated from each other by a comma, and that keys are separated from their values by an 'equals' character. The rest should go automatically, but you might get strange results if your events contain commas elsewhere (like in timestamps).
This worked fine for me with the following log file data (though I used semicolons instead of commas since the timestamp has a comma).
2012-05-03 15:55:17,89; 66=21963; 55=590621239; 44=FX.74
2012-05-03 15:55:17,90; 66=29886; 55=97211346; 44=FX.8411
2012-05-03 15:55:17,90; 66=27491; 55=3176432533; 44=FX.1282
2012-05-03 15:55:17,91; 66=31710; 55=74392336; 44=FX.29486
This gives me fields 66
, 55
and 44
Hope this helps,
Kristian
Thanks Kristian We got it working
the stanza in transforms.conf was not been correctly referenced in props.conf !
Hi Kristian
Thanks for the update
The inline search command extract worked perfectly, i.e.
Ideally we would want this working in a non-inline mode.
We have tried editing the props.conf in the $SPLUNK_HOME/etc/system/local folder
E.g.
[source::/
CLEAN_KEYS = false
But this does not result in the numeric keys being recognized.
Just wondering if you can see anything that jumps out which would indicate why the CLEAN_KEYS is not working
Thanks
Matt
see update above. /k
Hi Kristan
Thanks for your response.
In our case we want CLEAN_KEYS disabled for all the keys in our event. In your example you appear to be setting CLEAN_KEYS to false for a particular name/value pairing in the event.
We have tried setting CLEAN_KEYS=false in $SPLUNK_HOME/etc/system/local/props.conf but this does not work, i.e. numeric fields are not recognized as fields, e.g. 8,9,35 from my original example
Please and Thank you for any future help
Mat