|
Is it possible to use external commands in for automatic search-time field extractions ? Reason: I've got a case where performing a search-time extraction using regex isn't an option (well.. it is, but it'll need numerous heavy regexes and the performance would be awful), so the only option is using an external command to do the extractions. That works fine, but the end users have to pipe their searches through a command and that appears to be inconvenient and presents a performance penalty on the searches that have majority of the data coming from sources that don't require to be extracted with the external command. Data in question comes from Arcsight. The particular problem is with the "custom strings" - stuff that isn't in Arcsight's schema, so it comes as a pair csN={data} csNLabel={key name}. For example: The custom script extracts those pairs, so the data from the example above will be parsed as: It wouldn't be too hard to write a regex to extract that, if only the data would have been ordered in some way. But it isn't - csN and csNLabel can come anywhere in the sting, the data can contain csNLabel with no csN and so on... For example: |
|
Yes you can, there's a bit of work you need to do: http://www.splunk.com/base/Documentation/latest/SearchReference/Aboutcustomsearchcommands A built-in example that does something similar to what you probably is the But as for whether you can set up automatic extractions, the answer is no, but it is a good enhancement request. The search command already exists, I've even shared it as an app. The question was really about automatic extractions, sorry for not being clear on that. I'll file an RFE with support. Any idea if it may be implemented before 4.3?
(22 Apr '11, 09:12)
IgorB
|
Have you considered parsing the fields out at index time rather than at search time? Any chance we could see a sample of the data so the gurus here can possibly assist further with the extractions?
I have considered that, but then run into the usual problem - I've run into a new type of data that wouldn't have been caught by the extraction code I had in place. So if I've parsed the data at index time I would have to reindex whatever I have acquired by then :(
Data comes from Arcsight. I'll add an explanation to the question body - don't have enough characters left in the comment