We provide mobile data analytics reporting to mobile operators and we are increasingly being asked to take input in the form of proprietary transaction logs from a range of vendor solutions e.g. WAP gateways, mobile data optimization nodes, Cisco RDRs, DPI nodes, passive probes, etc.
Our solution has a standard input log format so we are having to write parsing scripts to convert these different 3rd party logs to our format.
Can we use Splunk as a generic parsing tool to facilitate the ingestion of all of these different logging formats?
Michael Stone Amethon Solutions
asked 23 Sep '11, 02:48
Well, I'd assume that if the logs in question are text files - then yes - that is pretty much what Splunk does. It will read and index events from any text file, and allow you to run searches, produce reports etc.
Bear in mind that you would most likely have to write your own "parsing" code in order to fully benefit from using Splunk. E.g. Splunk will happliy index events in a proprietary format like;
but you'd probably want to make that data into more usable information by "parsing" it, so that you interpret "BOBBY" as the hostname generating the logs, "RRR" as the type of device, "000988:2231" as the transactionID, "11,22" as the kB transferred, "ZZZZ" as cell tower ID etc etc etc.
This is not hard to do, as long as the structure of events from a (type of) file is fairly consistent.
This allows to search for events and produce reports such as "all customers in the Toronto area making more than 5 phone calls per hour on average" or "total bandwitdh used for Twitter messages from iPhones last week". As long as that kind of information is in the logs. Mathematical/statistical operations can be performed on any numeric data.
There is (as far as I know) no hard limit in the amount of different types of log a splunk instance can handle, so ... well I could go on and on... you should just download the Trial version for free and try it out.
Familiarize yourself with the concept of 'sourcetypes' and 'field extraction' and you should be on your way of making sense of the logs.
UPDATE: I re-read your question and realized that you weren't looking for an alternative to your existing solution. I'm not too familiar with using Splunk for that purpose, but it may be possible, although Splunk is not really a file conversion tool.
Hope this helps,