I have log files from a custom app we wrote that is entirely in hex.
To splunk it, I understand I might be able to create a custom command that converts hex-to-ascii and then pipe to it at search time and then pipe again to "search some ascii terms"
Could I also, though, create a custom hex-to-ascii module or component and insert it into pipeline.xml after input step and before the indexing step? I believe that approach was supported in Splunk v2.x or maybe it was v3.x.
Also, does that component need to be written in C/C++ or can it be a script instead?
You can not do the conversion at search time. Data presented to Splunk at index time must be text data, as Splunk fundamentally indexes text.
There is currently (4.1.5) no support for creating your own pipeline to insert between the file monitor and the rest of the Splunk indexing queue. The recommended solution currently is either:
monitor
or batch
inputs.Unfortunately both solutions have the disadvantage that you will have to code all file-tracking logic on your own in your program, rather than being able to use the Splunk file input monitor to do this.