SH clustering is bit complex (especially if you have Enterprise Security)
The three elements to do properly a SH cluster are
Staging Server - this is a standalone system whereby you produce all the code and make your changes
Deployer - This can be same as Staging SErver, but the location of the apps should be in "etc/shcluster/apps" and will push to all sh-members
SH members - The final location of apps
So when you create a new app yourself, I tend to put the code in "local". So the app you create , you create in "Staging Server". Pass to deployer and deploy to SH-members. But when it comes to SH-members, it will appear into "default" , no matter you make the changes in "local" in your staging-server. (This is a pain for version-control though). Ensure you pass "-preserve_lookups true" while deploying from deployer, so that the lookups in the SH-members are not overwritten.
So SH-cluster flow in large environments is for your myapp would be:
Staging Server ($SPLUNK_HOME/etc/apps/myapp/) -> deployer ($SPLUNK_HOME/etc/shcluster/apps/myapp) -> SH-members ( $SPLUNK_HOME/etc/apps/myapp)
Going into the file level configurations # I agree this is BAD, but this is how Splunk works
Staging Server ($SPLUNK_HOME/etc/apps/myapp/local/server.conf ) => deployer ($SPLUNK_HOME/etc/shcluster/apps/myapp) => SH-members ($SPLUNK_HOME/etc/apps/myapp/default/server.conf
... View more