Hello Splunkers. I would like to ask for some advice from you, as  we are planning to replace a lot of rsync scripts that we use to distribute apps to all of our deployment servers. We have an architecture of 5 different tenants, that are pretty much completly isolated from eachothers. Because of that, we have one deployment server in each tenant. To centrally manage all this tenants, we have one "master" server, where we keep all our splunk configuration (apps, serverclasses etc.), and uses scripts based on rsync to push them out to the other deployment servers. I have an impression of that using tools like ansible or puppet etc. has become the "industry standard" of the way of handling such big Splunk multi-tenant enviroments. Found this presentation from CONF19, held by Splunk themself, that shows how to utilize ansible to achieve this: FN2048.pdf (splunk.com) As of what i understand, the alternative to using an 3rd party tools (ie. ansible) for this, would be to use a "Master/Slave" configuration for the deployment servers, having the master deployment server to push apps to "/opt/splunk/etc/deployment-apps/" to other slave deployment servers with such config: [serverClass:secondaryDeploymentServersDeploymentApps] targetRepositoryLocation = $SPLUNK_HOME/etc/deployment-apps (source: https://community.splunk.com/t5/Deployment-Architecture/How-to-set-up-Multiple-Deployment-Servers-Configuration/m-p/45392 ) We want to get rid of all theese scripts for syncing indexers, standalone search heads, search head clusters and UF's, so we are trying to find the best way. My question is, is there any advantages or disadvantages with theese two models? The "splunk only" method of doing this, doesnt seem to be nearly as popular using ansible? In advance, a bit thank you for any advice 🙂 ... View more

Hi, I am trying to get the Splunk_TA_esxilogs app to work in our Splunk Enviroment, but cant get it working together with our app that rewrites index and sourcetype. I suspect that one Splunk Enterprice instance cannot rewrite the sourcetype and index more that one time. The ESXi logs are allready collected at an syslog server, and forwarded to the Heavy Forwarder. At the HF we use "rewrite app" with an regex to change the sourcetype from "syslog" to "esxi", based out of the hostname, like this: props.conf: [syslog] TRANSFORMS-force_vmware = force_sourcetype_vmware, force_ix_vmware transforms.conf: [force_sourcetype_vmware] SOURCE_KEY = MetaData:Host REGEX = ^host::(10\.24[1289]\.70\.\d+|10\.243\.12\.\d+|10\.25[01]\.70\.\d+|10\.252\.198\.50|10\.30\.209\.19[5-6]|10\.36\.1[128]\.\d+|10\.37\.12\.\d+|10\.45\.[12]\.\d+|10\.6[23]\.12.\d+|10\.63\.10\.20|10\.65\.(0|64)\.\d+|10\.65\.65\.65) DEST_KEY = MetaData:Sourcetype FORMAT = sourcetype::vmw-syslog [force_ix_vmware] SOURCE_KEY = MetaData:Sourcetype REGEX = ^sourcetype::(?i)vmw-syslog$ DEST_KEY = _MetaData:Index FORMAT = vmware-esxilog So far, so good. This rewrite app does its job. The data now has index "vmware-esxilog" and sourcetype "vmw-syslog". Now the Splunk_TA_esxilog app should in theory start baking the data: props.conf: ####### INDEX TIME EXTRACTION ########## [vmw-syslog] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+)(?:.*?(?:[\d\-]{10}T[\d\:]{8}(?:\.\d+)?(?:Z|[\+\-][\d\:]{5})?)\s[^ ]+\s+[^ ]+\s+[^\->])|([\r\n]+)(?:.*?\w+\s+\d+\s+\d{2}:\d{2}:\d{2})(?:\s+[^ ]+\s+)+[^\->] TZ = UTC DATETIME_CONFIG = /etc/apps/Splunk_TA_esxilogs/default/syslog_datetime.xml TRANSFORMS-nullqueue = vmware_generic_level_null TRANSFORMS-vmsyslogsourcetype = set_syslog_sourcetype,set_syslog_sourcetype_4x,set_syslog_sourcetype_sections TRANSFORMS-vmsyslogsource = set_syslog_source   But it doesnt. The data gets indexed without beeing touched by the Splunk_TA_esxilogs app. It works IF i disable the HF rewrite app, and change the stanza in Splunk_TA_esxilogs from [vmw-syslog] to [syslog], but that will hit way to wide. The name of the HF rewrite app starts with "05", so its configuration comes before the app named "Splunk_TA_esxilogs". Any suggestions is highly appreciated 🙂 ... View more