We index a large volume of financial logs, which are in the Financial Information eXchange (FIX) format. These are not really in an easily human readable format as they contain a bunch of numeric codes for fields and values, so I am trying to get Splunk to translate these logs so when my users search for them, they can understand them without having to reference their FIX documentation.
FIX messages contain multiple codes for field names and values in each event, that can be translated like this:
FIXcode,translation 38=,OrderQty= 39=0,OrdStatus=New 39=1,OrdStatus=Partially filled
My FIX people currently manually look up these codes in a reference manual, or run raw log text through a java translation app.
Has anyone managed to get Splunk to take care of this by itself?
(If not, I'm approaching the problem from a particular angle here: http://answers.splunk.com/questions/886/what-is-the-procedure-to-build-your-own-splunk-search-related-function)
I wrote a simple (20 line, and could be shortened) Python script that is referenced by $SPLUNK_HOME/etc/system/local/commands.conf to become a custom search command that can be used in Splunk Web. Any search for FIX logs can now just be piped to "translatefix" for human readable logs.
IMHO it's much simpler than using a large number of lookup tables (one for each of the 1139 possible fields in FIX5.0), and configuring each in props/transforms. The script take a single config file, which is a list of strings to match and replace.
It's also better, as it strips out the unprintable ASCII characters, such as the field separator SOH or \x01, making it much easier to read.
Agreed, the config file will potentially be quite long once all codes are in there (we are just using a "top 500" selection at the moment), but at least it is all contained in the one place, and does not require a Splunk restart (or any Splunk config changes at all) when new fields are added.
If anyone else would find this useful in their environment, please contact me at glenn.sinclair at igindex dot co dot uk
answered 01 Apr '10, 08:57
Well, I'm not completely sure, but it looks to me like you can pretty easily use a combination of Splunk lookup tables and field aliases to decode these, search, and report on on them.
You'll need to define set
Then, you can alias the field number to names with a series of FIELDALIAS commands. Finally, you can define a series of lookup tables, one for each field type probably. These are just CSV files that map the numeric values to some other value.
All of these functions can be set up in props.conf for your sourcetype so they run automatically whenever data from that sourcetype is displayed. Let us know if you would like more details, but all of these are standard functions that should be described in the docs, the
answered 26 Mar '10, 15:44
Can you share your python script to me ? We also want to use Splunk to index FIX logs (for securities and futures trading system).
I use props.conf and transforms.conf to extract some fields in FIX logs, but I think your python script is much better and easy to use.
answered 21 Jun '10, 14:56