Splunkforwarder 9.2.0.1 install fails with conflict with splunk package

[Egyas]

I have a current Splunk install in my production environment, all running RedHat Linux.  I have a single server w/ Splunk Enterprise installed on it, as well as SplunkForwarder.  I have 100+ other servers w/ SplunkForwarder installed on them all pushing logs to the Splunk Enterprise server.  All servers had v9.1.2 of the forwarder installed, and the Enterprise server was also this version.

I recently updated the Splunk Enterprise server, as well as the Splunk Forwarders on all servers, to version 9.2.0.1 successfully.  With one exception.  The forwarder installed on my Splunk Enterprise server (named "splunkenter1") fails.  It displays the error listed below where it says that the splunkforwarder package is conflicting with the splunk install.

I have another Splunk Enterprise install (using the same set-up) in another environment, and I did not run into this issue.  That upgrade worked without issue.

I've tried Google'ing the issue, but haven't found much.  Anyone have any ideas on what could be causing this, or has anyone seen this before?

[root@splunkenter1 ~]# dnf update splunkforwarder
Last metadata expiration check: 0:01:36 ago on Mon 22 Apr 2024 04:47:07 PM UTC.
Dependencies resolved.
========================================================================================================
 Package                     Architecture       Version                          Repository        Size
========================================================================================================
Upgrading:
 splunkforwarder             x86_64             9.2.0.1-d8ae995bf219             splunk-repo              44 M

Transaction Summary
========================================================================================================
Upgrade  1 Package

Total download size: 44 M
Is this ok [y/N]: y
Downloading Packages:
splunkforwarder-9.2.0.1-d8ae995bf219.x86_64.rpm                          41 MB/s |  44 MB     00:01    
--------------------------------------------------------------------------------------------------------
Total                                                                    41 MB/s |  44 MB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction test error:
  file /usr/lib/.build-id/03/f57acc2883000e6b54bf75c7e67d1a07446919 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/06/a82be30cc16ea5bea39f78f8056447e18beb15 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/1a/b0b8e873c6d668dcd3361470954d12004926cd from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/1e/8edb02a946c645cd20558aa8a6b420792f5541 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/35/e87a7fb154de7d5226e5a0a28c80ffd0c1be48 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/3a/3aac493bff5bb22e02b8726142dd67443dd03c from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/42/abc0f2a26bfb13b563104e87287312420c707e from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/44/6a270f1de8d26f47bf9ff9ae778e1fd3332403 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/64/b2324ff715d30c8a91dee6a980d63c291648d8 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/65/274a42201dd21f83996ba7c8bd0ba0dc3894c8 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/6d/dd008477651e7c8febce4699a739aaf188b0ae from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/88/cbe6deabd44a4766207eebf7c5e74f7ed53120 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/8a/6ee8699fb74fb883874a1123d91acf0b0d98a6 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/94/ea2865a21761f062a2db312845c535d5429bfc from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/95/d5fe61c313d8a5616f8a45f6c7d05151283ab6 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/96/b9463c40fc6541345a4b87634e8517281f8d4d from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/99/93008fdae763af21c831956de21501bb09e197 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/9b/2a882e45910da32603baf28a13b1630987184e from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/9f/b5fd366b32867d537caa84d4b2b521f5c21083 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/a0/1ae9032915dce67a58e8696c3c9fe195193d77 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/a1/616e140409dc54f0db2bf02ed7e114f07490af from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/b6/6dd3d33542916fff507849621dac5f763a98a2 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/b6/fd3c259a4c6e552d9b067f39e66c03cc134895 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/b7/e3d0b70694caa826df19d93b7341de0decdad3 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/bc/f1c9c6878bb887ef6869012b79c97546983b83 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/c8/d218675e02086588c28882c28b3533069d505c from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/d0/be01f291a5b978e02dcdd0069b82ce8a764dbf from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/d3/7dcf7bcf859ed048625d20139782517947e6e0 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/d7/30a0409850e89f806f3798ca99b378c335b7a5 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/dc/259ac038741ecbd76f6052a9fa403bc5ab5ab3 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/de/294f4dd1fa80d590074161566f06b39b9230fb from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/e0/0ee3712cdbd590286c2b8da49724fdaf6dee15 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/e6/7f07efdda1fcfe82b6ceb170412f22e03d2ab5 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/ec/dc3eeaba4750e657f5910fa2adb21365533f27 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/ee/6addfc324fb4bf57058df3adf7ea55dff4953f from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/f1/0b5a5bc3bcb996183924bd6029efba8290c71a from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/f2/c0dd88030fc9e343f6d9104a5015938cfe3503 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/f3/61ef732e036606eef3d78bb13f6d6165bcd927 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/f4/c1fc01304f2796efaabefd2a6350ba67cc9edc from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/f9/3cf5828d46fbdd6e82b2d18a4a5c650b84c185 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/fa/a370a95319b4a8ce1bd239652457843a09c15e from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64
  file /usr/lib/.build-id/fd/201b0799acb29720c90a6129be08800ce4b7e5 from install of splunkforwarder-9.2.0.1-d8ae995bf219.x86_64 conflicts with file from package splunk-9.2.0.1-d8ae995bf219.x86_64

[burwell]

So there's a bug with installing Splunk Enterprise 9.2.x and the universal forwarder on the same server, something that should work. I have opened a case with Splunk and requested them to document the issue in the known issues. They have not done that yet. 

[deepakc]

Yes it bit painful,  if you have made lots of /local/ based configs in your apps backup the /opt/splunk/etc/apps folder at minimum, this way you ar least have your app config's backed up and can restore those apps after your re-install Splunk to keep it clean. 

[deepakc]

You say
"I have a single server w/ Splunk Enterprise installed on it, as well as SplunkForwarder"

Why did you install a forwader onto the Splunk Server instance as well - There is no need to do this, its not a normal thing to do, hence why you are most likley getting those conflict fails. 

I suspect you wanted to collect logs etc from the Splunk instance, hence you may have done this, but the full splunk instance will have this functionality built it in.

Keep the Splunk instance clean (only installs apps/ta's etc). 

Install the forwarder for your target hosts that you want to monitor for your logs etc.

If you now uninstall the the forwarder from the Splunk server, you may get all sorts of errors, and then need to re-install the Splunk server as you may have overwritten various Splunk server files etc...messy. 

[MC5]

Splitting up Splunk Enterprise and OS level log collection is a good idea. Including OS log collection with Splunk Enterprise forwarding creates some issues. Logs being ingested by an indexer may be handled differently than local files. For example, settings applied to inputs.conf on the indexer, for the sake of indexed files, might be applied everywhere. Thought a nuisance, things like this can be handled with careful configuration.

But, from a management perspective, if you want to have a baseline set of OS log collection in an enterprise, applying rules across all of your systems if you have Indexer clusters, search head clusters, deployment servers, heavy forwarders, etc... all the different types of system, can be cumbersome to the point of not workable. If you do this using the deployment server managed UF, baseline log collection becomes far more manageable. This can be important if baseline log collection changes regularly.

Also, note that splunk recently changed the UF to use the 'splunkfwd' user, while the 'splunk' user is for Splunk Enterprise. This leads me to believe the Splunk is already moving the direction of splitting up local log collection and log indexing.

[Egyas]

Huh.  Guess I was just assuming that it needed both, and that's the way I've always done it.  Now I'll have to play around in the lab and see what happens when I remove it.

Thanks!

[lpino]

Hi @Egyas,

I just run into the same issue trying to upgrade a Splunk UF 9.1.2 -> 9.2.1 installed on a server with a Splunk Enterprise instance (just upgraded to 9.2.1).

Did you find any workaround/solution except removing one of them?

Thanks in advance!

[Egyas]

Not yet.  I'll be trying to work this out in the lab,

If anyone else finds a solution other than the apparently painful process of removing the forwarder, please let me know!  Thanks!

[Egyas]

Still working on this one.  II uninstalled SplunkForwarder form the Splunk Enterprise server, and that seems to have been a BAD move.  Seems to have caused some config changes and permissions changes, and Splunk Enterprise segfaults when I try to start it now.

Still trying to work out a fix.

[deepakc]

Yes, I suspected that would happen, maybe try:

1. Stop Splunk if you can
2. Backup /opt/splunk/etc/apps folder  (So you have your App configs at least)
3. For your data if you are using the default in $SPLUNK_HOME/var/lib/splunk folder - you can be move to a temp folder as well, but if you had a seperate volume even better - it wont get touched.
4. Re-install Splunk over the current broken install  and see if that works (I suspect not) but worth a go

5. If it works restore the /opt/splunk/etc/apps folder and your data .
(make sure you set the splunk permissions - chown -R splunk:splunk etc to the Splunk and data folders

If that all fails, then may be wipe it clean and start again, if you keep it as it is not going to bode well for the future as you will have other upgrades to do in the future  and it will always cause some kind of problem, so better to sort it all out now and make it clean.

If it was me, I would start clean again, less issues in the longer run.