Possible to rewrite the sourcetype twice?

[Fonzie2k]

Hi,

I am trying to get the Splunk_TA_esxilogs app to work in our Splunk Enviroment, but cant get it working together with our app that rewrites index and sourcetype. I suspect that one Splunk Enterprice instance cannot rewrite the sourcetype and index more that one time.

The ESXi logs are allready collected at an syslog server, and forwarded to the Heavy Forwarder.

At the HF we use "rewrite app" with an regex to change the sourcetype from "syslog" to "esxi", based out of the hostname, like this:

props.conf:
[syslog]
TRANSFORMS-force_vmware = force_sourcetype_vmware, force_ix_vmware

transforms.conf:
[force_sourcetype_vmware]
SOURCE_KEY = MetaData:Host
REGEX = ^host::(10\.24[1289]\.70\.\d+|10\.243\.12\.\d+|10\.25[01]\.70\.\d+|10\.252\.198\.50|10\.30\.209\.19[5-6]|10\.36\.1[128]\.\d+|10\.37\.12\.\d+|10\.45\.[12]\.\d+|10\.6[23]\.12.\d+|10\.63\.10\.20|10\.65\.(0|64)\.\d+|10\.65\.65\.65)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::vmw-syslog

[force_ix_vmware]
SOURCE_KEY = MetaData:Sourcetype
REGEX = ^sourcetype::(?i)vmw-syslog$
DEST_KEY = _MetaData:Index
FORMAT = vmware-esxilog

So far, so good. This rewrite app does its job. The data now has index "vmware-esxilog" and sourcetype "vmw-syslog".


Now the Splunk_TA_esxilog app should in theory start baking the data:

props.conf:
####### INDEX TIME EXTRACTION ##########
[vmw-syslog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?:.*?(?:[\d\-]{10}T[\d\:]{8}(?:\.\d+)?(?:Z|[\+\-][\d\:]{5})?)\s[^ ]+\s+[^ ]+\s+[^\->])|([\r\n]+)(?:.*?\w+\s+\d+\s+\d{2}:\d{2}:\d{2})(?:\s+[^ ]+\s+)+[^\->]
TZ = UTC
DATETIME_CONFIG = /etc/apps/Splunk_TA_esxilogs/default/syslog_datetime.xml

TRANSFORMS-nullqueue = vmware_generic_level_null
TRANSFORMS-vmsyslogsourcetype = set_syslog_sourcetype,set_syslog_sourcetype_4x,set_syslog_sourcetype_sections
TRANSFORMS-vmsyslogsource = set_syslog_source

But it doesnt. The data gets indexed without beeing touched by the Splunk_TA_esxilogs app.

It works IF i disable the HF rewrite app, and change the stanza in Splunk_TA_esxilogs from [vmw-syslog] to [syslog], but that will hit way to wide.

The name of the HF rewrite app starts with "05", so its configuration comes before the app named "Splunk_TA_esxilogs".


Any suggestions is highly appreciated 🙂

[isoutamo]

Hi

Why you don't use syslog to write VMware logs to own log files and then put those sourcetype and index into inputs.conf? Then you don't need that second transforms here.

One event can go trough parsing phase only once. If you want send it again to that you should use CLONE_SOURCETYPE for that and then drop all events from old one or you have duplicate events with two different source types.

See dataflow here: https://community.splunk.com/t5/Getting-Data-In/Diagrams-of-how-indexing-works-in-the-Splunk-platfor...

r. Ismo

[Fonzie2k]

Thanks anyways. My suspicions about rewriting the sourcetype twice was impossible, seems to be correct.
So gotta find another way of solving this.

[Fonzie2k]

Hi,

How exactly could that be done?

I would need a [monitor://] stanza for each ESXi host, and we have hundereds. Thats why we define the esxi hosts with regrex, in the HF rewrite app.

If we had just a few, it could be solved by the following in inputs.conf:
[monitor:///data/logs/esxihost1]
disabled = false
host_segment = 3
sourcetype = vmw-syslog

Was this your suggestion, or did i misunderstand?

[isoutamo]

Almost that. You can write those e.g. /data/logs/vmware/esx/<hostname>/log-file

and then use wildcard like

[monitor:///data/logs/vmware/esx/*/vmw-syslog]
disabled = false
host_segment = 5
sourcetype = vmw-syslog
index = <vmware index>