|
An attempt was made to upgrade Splunk 4.2.2 to 4.3. However, the incorrect Splunk_Home was used, namely the Splunk location was entered as /opt/splunk when it should have been just /opt. Therefore the upgrade went into the directory structure /opt/splunk/splunk. I had a backup of the original structure, but for some reason I wasn't able to remove the upgrade using rpm -e. I would get a message saying that it couldn't be found, yet I couldn't reinstall it because it was already there. So, it was suggested by another that I copy the /opt/splunk/splunk structure to a temporary location, remove the /opt/splunk/splunk structure, and then copy the temporary structure back to /opt/splunk. That was fine, but I then didn't have my data in the /opt/splunk/etc directory. So the suggestion was made to delete the /opt/splunk/etc directory and then copy that directory from my backup from 4.2.2. This was done. It was only later after going through the 4.3 tutorials that I realized that there had been 4.3 specific data in the /opt/splunk/etc directory that I now didn't have. So, the question is how to proceed from this point. If I could remove the upgrade using rpm -e, then I coould just delete the /opt/splunk directory, and copy it back from my backup and then reapply the upgrade rpm. I also see that there is not nearly as much data indexed as there was previously, so that things being searched for are no longer there. Can I reindex the data? What would have caused data that had been indexed to no longer be indexed? I tried submitting a support ticket but the support page always hangs upon submit (in any browser.) |
|
Useful info on this topic can be found here: Moving Splunk indexer from one host to another host on Splunk Answers Specific to your situation, I would uninstall Splunk, then do a new install of 4.3, and copy over the configuration from $SPLUNK_HOME/etc/apps/<appname> that were relevant, as well as the $SPLUNK_HOME/etc/system/local folders. You can also copy the indexed data from $SPLUNK_HOME/var/lib/splunk/<indexname> on the backup to the new instance in the same location under the $SPLUNK_HOME because the data format didn't change from 4.2 to 4.3. However, you need to be careful to avoid conflicting bucket ids. For details on that, see: Moving indexes to a new Splunk server on Splunk Answers If you do run into bucket id conflicts, see: How can I find all duplicate bucket id's that are causing conflicts in my index? on Splunk Answers If you're having trouble submitting a support case, you may want to try calling support(presuming you have a enterprise support contract). This could be a tricky thing to do, so if you need to call, feel free. The contact number can be found here: We hadn't created any new apps, but had only modified the default search app. If memory serves me right there were only modifications to conf files, but I don't remember which ones. Since we are only using the default search app, if I were to copy that app into 4.3 then I would lose the 4.3 related search features. I see from documentation that after performing a search the screen has additional items on it for 4.3 than for 4.2.2. (In multiple parts because of character count limitations)
(29 Feb '12, 06:28)
RVDowning
The only file copied into $SPLUNK_HOME/var/lib/splunk/ when the $SPLUNK_HOME copy was made was appserver. I don't know why other files in $SPLUNK_HOME/var/lib/splunk/ would not have copied. Perhaps I should have used a compare utility to check the before and after images. Is this why the users have lost their data? Others have also indicated problems when attempting to submit a support case. They said it was quite common for the application to hang when the Submit key was pressed. They did say that after multiple attempts it would usually go through.
(29 Feb '12, 06:29)
RVDowning
While I understand why you feel hesitatant about upgrading, I don't think it should stop you. 1) you don't need to copy the whole search app, just the changes you made to $SPLUNK_HOME/etc/app/search/local. You can be more surgical with your approach by copying the local folders, if you'd prefer. 2)I don't know why other folders wouldn't have been copied, either your indexes exist in a custom location or something went wrong with the copy. Presuming something went wrong with the copy, this is why your data is missing. I still suggest you open a support case on the issue for guidance.
(29 Feb '12, 08:10)
jbsplunk ♦
It turns out that the database is not in the $SPLUNK_HOME hierarchy in $SPLUNK_HOME/var/lib/splunk but rather in the top level directory /splunk. Only appserver is in the original location. So I don't understand why older data is not appearing. There had been over 1.5 million transactions whereas now there are only 300,000. Older data is no longer appearing.
(29 Feb '12, 10:00)
RVDowning
I would guess, and its just a guess, that the data isn't appearing because the config points to /var/lib/splunk instead of /splunk. Without going through all of your config it is difficult to say. If at all possible, get in touch with Support
(29 Feb '12, 10:10)
jbsplunk ♦
|
