What's the best way to track down props.conf problems?

Splunk provides a few utilities that can greatly speed up the process of tracking down props-related config problems. The first two can even be used to check configuration changes without restarting splunkd. (Although, you need to restart splunkd once you get your configuration issues resolved.)

Splunk "test sourcetype" utility:

This utility will show you all of the relevant props config entries for a given log file. This let's you quickly test your source to sourcetype matching rules are working properly. (Or, it test that any rules:: or delayedrule:: are functioning properly.) It will also reveal simple issues of file permission. I've often found this useful in detecting overlapping source rules or issues with stanza priorities.

Example:

splunk test sourcetype /var/log/your_log_file

Splunk's "btool" utility

This utility can be used to dump a nested configuration for a specific config file or a specific stanza within a config file. You can optionally specify which application you would like to test this from. (Normally you will want to specify whichever application your have your inputs.conf setup to use.)

$SPLUNK_HOME/bin/splunk cmd btool props list <sourcetype>

Splunk | file search command

The file search command in splunk lets you view a local log file as though it were indexed. Not all props settings are honored during this process, but you can often use this to test your setup before adding a new input and catch minor (and sometimes major) problems with timestamp recognition, and event breaking logic before loading anything into your index.

Keep in mind that any settings changes do require a splunkd restart in order to take effect here. (While it's a pain to do so, it's still faster than indexing events improperly and then having to dump your indexes and reload, or try to find and delete incorrectly indexed events.)

Example: (Type this into the search bar)

| file /var/log/your_log_file

If you want to due additional searching on your log file, you can do so with a post-search command, like so:

| file /var/log/your_log_file | search my-search-term NOT unwanted-search-term ...

Splunk's | extract reload=T search command:

This trick is useful once you have your events indexed correctly and would like to setup field extractions. This is helpful if you edit your props.conf or transforms.conf files my hand. You should be able to stick it in your search and have your config changes be reloaded which is much faster and less problematic that restarting splunk. (Oh, and don't forget to remove this command before you save your search--you only want to use this interactively on an as-needed basis)

Additional resources:

• Splunk Docs:
  • Troubleshooting configurations - More in-depth examples using btool.
  • Configuration file precedence - Understanding Splunk's layered configurations.
  • Attribute precedence within a single props.conf file - Discussion on using priority and matching based on souce:: vs host:: vs sourcetype.
• Splunk Wiki:
  • Where do I configure my Splunk settings? - This is a great resource that provides insight into how splunk processes your events, how event processing changes for light vs heavy forwarders and which config entries are used in different processing phases. (A big thanks to Gerald for point out this link. It's extremely helpful.)