We have indexes per environment (e.g. prod, qa, dev), with all logs from instances of an application in a particular environment being forwarded to that index. In this way, we can find errors in production in a single index. As user requests originate in an externally facing application, and are shuttled about various back-end services, we can find it all in one Splunk index.
Still, the majority of our searches are isolated to one application. We have a search-time field extracted for the application: "app". It's actually determined via lookup from the parent folder of the log file. For example, C:/logs/Admin/My.Admin.log has parent folder "Admin", which we look up to be "app" value "admin."
Since it's rare that we have to search across applications, our searches usually focus on a single app, like:
index=prod app=admin SomeText
This is so frequent that my knee-jerk reaction is to think, "I should make that an index-time field!". But every resource I come across, including this other answer, say not to do this. They all hint that there are reasons one might actually want an index-time field, but are vague about those reasons.
So, without assuming an index-time field is necessary, what is the best way to ensure that such common searches are as efficient as possible?
... View more