I found the issue..  1) I got confused as the previous instance DNS pointer is not updated to the new FQDN, so when I tried to hit the DNS name it did not hit the new instance  2) The new instance was configured with web.conf ->  enableSplunkWebSSL = 1  to enforce SSL but I think there is an issue with the common name   ... View more

from splunkd.log : 06-28-2023 12:12:09.162 -0400 WARN HttpListener - Socket error from <ip> while idling: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http reques   ... View more

I don't have Firewalld installed      ... View more

    The GUI means the Splunk dashboard where we log in and run a query Are you referring: https://www.splunk.com/en_us/blog/tips-and-tricks/selinux-and-splunk.html ... View more

You have updated old env to 8.0.3 - The current(old) Splunk version is 8.0.3. In the destination(new instance), the version is the same version and I am planning to do in-place upgrade to 8.1  to~> 9.0.5 ( once migrated successfully ) You have installed RH 8 into the new server- ( our company provides the instances the new one is RedHat8 the old was 7)  You have rsync /opt/splunk/ and db dir to the new server - correct You have changed a needed conf files etc - correct to contain the new server name (FQDN) You have installed splunk 8.0.3 into the new server- correct You have started splunk on new sever -correct now we are here? Yes,sir  ... View more

for the ownership I ran  chown -R splunk:splunk /app/splunk-app/($SPLUNK_HOME) and same for /app/splunk-data ($SPLUNK_DB) there is a symlink /app/splunk-app to /opt/splunk (to enable RPM installs and updates). for the OS just installed the RPMs with yum localinstall <RPM>   in regards to  other errors : ERROR DatabaseDirectoryManager - Getting size on disk: Path for bid=<index>~5614~E1F35BEE-29E7-4CCB-9E16-F000A9A1741F cannot be located. caller=serialize_SizeOnDisk migration log: /opt/splunk/var/log/splunk/migration.log.2023-06-26.17-32-08: Migrating to: VERSION=8.0.3 BUILD=a6754d8441bf PRODUCT=splunk PLATFORM=Linux-x86_64   Copying '/opt/splunk/etc/myinstall/splunkd.xml' to '/opt/splunk/etc/myinstall/splunkd.xml-migrate.bak'.   Checking saved search compatibility...   Handling deprecated files...   Checking script configuration...   Copying '/opt/splunk/etc/myinstall/splunkd.xml.cfg-default' to '/opt/splunk/etc/myinstall/splunkd.xml'. Deleting '/opt/splunk/etc/system/local/field_actions.conf'.   The following apps might contain lookup table files that are not exported to other apps:   splunk_monitoring_console   Such lookup table files could only be used within their source app.  To export them globally and allow other apps to access them, add the following stanza to each /opt/splunk/etc/apps/<app_name>/metadata/local.meta file:   [lookups] export = system   For more information, see http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/SetPermissions#Make_objects_globally_available.   Checking for possible UI view conflicts...  App "splunk_monitoring_console" has an overriding copy of the "alerts.xml" view, thus the new version may not be in effect. location=/opt/splunk/etc/apps/splunk_monitoring_console/default/data/ui/views  App "splunk_monitoring_console" has an overriding copy of the "dashboards.xml" view, thus the new version may not be in effect. location=/opt/splunk/etc/apps/splunk_monitoring_console/default/data/ui/views  App "splunk_monitoring_console" has an overriding copy of the "reports.xml" view, thus the new version may not be in effect. location=/opt/splunk/etc/apps/splunk_monitoring_console/default/data/ui/views Removing legacy manager XML files... Removing legacy nav XML files... DMC is not set up, no need to migrate nav bar. Removing System Activity dashboards... Removing splunkclouduf XML file... Removing splunkclouduf view XML files... Distributed Search is not configured on this instance Removing legacy search.xml file from splunk_instrumentation... Deleting '/opt/splunk/share/splunk/search_mrsparkle/modules'. Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.   Checking for the modules related files and folders that should not be present after upgrade.   Checking for the Advanced XML dashboard templates that should not be present after upgrade.   Checking for the 'Getting Started' app that should not be present after upgrade.     It seems that the Splunk default certificates are being used. If certificate validation is turned on using the default certificates (not-recommended), this may result in loss of communication in mixed-version Splunk environments after upgrade.   "/opt/splunk/etc/auth/ca.pem": already a renewed Splunk certificate: skipping renewal "/opt/splunk/etc/auth/cacert.pem": already a renewed Splunk certificate: skipping renewal Clustering migration already complete, no further changes required.   Generating checksums for datamodel and report acceleration bucket summaries for all indexes. If you have defined many indexes and summaries, summary checksum generation may take a long time. Processed 1 out of 12 configured indexes. Processed 2 out of 12 configured indexes. Processed 3 out of 12 configured indexes. Processed 4 out of 12 configured indexes. Processed 5 out of 12 configured indexes. Processed 6 out of 12 configured indexes. Processed 7 out of 12 configured indexes. Processed 8 out of 12 configured indexes. Processed 9 out of 12 configured indexes. Processed 10 out of 12 configured indexes. Processed 11 out of 12 configured indexes. Processed 12 out of 12 configured indexes. Finished generating checksums for datamodel and report acceleration bucket summaries for all indexes. [App Key Value Store migration] NOTE: Lock file is not empty. If migration fails, stop the service (if it is running), delete the lock file, and run the migration again. [App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds. [App Key Value Store migration] Migration is not required. [App Key Value Store migration] Checking if migration is needed. Upgrade type 2. This can take up to 600 seconds. [App Key Value Store migration] Migration is not required. [DFS] Performing migration. [DFS] Finished migration.     ... View more

I am still not able to see the GUI  ... View more

I did rsync both SPLUNK_HOME and SPLUNK_DB ( entirely ) I had to change the server name in some of the *.conf  files to the new one ... View more

Thank you, everyone. I decided to migrate to the same version 8.0.3 in source and destination following @richgalloway advice and followed @isoutamo  migration steps: - setup new host - rsync (1st - Nth time) data + configs - yum install splunk - test with new version w/o alarms, emails etc. - stop old - final rsync with delete option - start new   After I started the Splunkd service in the new instance I got some  ERRORS few about SSL certificate which I think got resolved (as I don't see it in /ops/splunk/var/splunk/splunk.log)  by generating a new SSL certificate and signing them using the organization CA  and .. ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/opt/splunk/var/log/splunk/migration.log.2023-06-26.17-32-08). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info. 06-27-2023 14:18:08.891 -0400 ERROR IntrospectionGenerator:resource_usage - RU - Mount '/opt/splunk' (ext4) is not interesting, iostats will not be collected. And  -0400 INFO LMStackMgr - license_warnings_update_interval=auto has reached the minimum threshold 10. Will not reduce license_warnings_update_interval beyond this value   ... View more