We have Universal Forwarders installed on Windows 2003 & 2008 Servers, plus a heavy forwarder on Windows 2008...
We updated to 4.3.2 on all forwarders in April, and converted all but one system configured as heavy forwarders to universal forwarders. Most of the systems were previously running 4.2.4 heavy forwarders, though a few were running 4.3.1 Universal Forwarders.
Last week I noticed, 11 of my 15 Windows forwarders displayed the "Splunk could not get the description for this event" message in 4,647 events for a 24 hour period, excluding domain controller security logs (in which case it goes into the millions). In the case of the domain controller, cycling the SplunkForwarder service once or twice usually clears up the messages from the WinEventLog:Security, though I'll continue to get the error message on the DCs in the Application and System Logs.
05/08/2012 01:19:29 PM
LogName=System
SourceName=Service Control Manager
EventCode=7040
EventType=4
ComputerName=DC2.hersheymed.net
User=SYSTEM
Sid=S-1-5-18
SidType=1
TaskCategory=None
OpCode=None
RecordNumber=211980
Keywords=None
Message=Splunk could not get the description for this event. Either the component that raises this event is not installed on your local computer or the installation is corrupt.
FormatMessage error: The handle is invalid.
Got the following information from this event:
Windows Modules Installer
demand start
auto start
TrustedInstaller
All 11 are Windows 2008(32-bit, 64-bit, and R2), the other four are all Windows 2003. The number of messages in the System and Application logs that display this behavior far exceeds the number of messages that do not. Indexes are 4.3.2 on RedHat, in case it matters. There are no (or very few if they're buried in the data) events with this behavior prior to updating the forwarder on any given host.
I've had a support case open since late last week, but I thought I'd ask the community if they can think of anything to check while I'm waiting... we're continuing to pull in corrupt (well, incomplete anyway) log data from these Windows forwarders so the delay in the back-and-forth-by-email isn't appealing.
Hm, I answered this, but the comment system ate it whole. Trying again.
This is indeed a bug in 4.3.2 WinEventLog data acquisition code. It's now been identified and we can fix for the next release.
The code flow was altered in order to address performance concerns, which was a success. The performance for rapidly acquiring eventlog data should improve by around a factor of 2, which relieves stress on the wineventlog service (which was the bottleneck). The changes have to do with cacheing handles for data providers of windows eventlog strings.
Of course that's a poor consolation for correct operation. Unfortunately, the set of events we tested with did not have the distribution of data providers, so the problem wasn't identified internally.
For now please use 4.3.1 forwarders (or earlier) to acquire this data type.
Please note that this error message does not have a one to one correlation with this misbehavior. Other scenarios such as loading EVT files without the corresponding availble DLLs that provide the messages, or reading eventlogs for an application which has been subsequently uninstalled could (and do) produce the same message.
Look at your windows event logs locally and I will bet you are getting the same message. If it is your security log you are probably missing the msaudite.dll file under system32 folder along with security subkey under the hklmsystemcurrentcontrolsetserviceseventlogsecurity.
If it is in the app or system event log you are missing the registry hives for those events. You can just copy them over from a working machine.
Thank you Jeff but the above is not the only resolution as I have just worked through this same issue myself and it was not a bug with us as I rolled back to 4.3.1 and it did not fix the issue. What did fix the issue was that we were missing essential dll's and registry keys. I just wanted to provided more info for others where this solution may not be the answer. Thanks again and next time before giving negative points ensure that you have the proper knowledge.
No, the event viewer view was fine looking at the same messages... it's a bug in 4.3.2 (SPL-51312 ) and is resolved in 4.3.3. You should read through the entire post - this was mentioned above.
Where can i get Universal forwarder 4.3.1 ??
Done!! Thank you!!! (Older versions link in the same download web ....)
Instead of the forwarder 4.3.1 the message of the event is not showed. Same message 'Splunk could not get ...'
Any idea?
Hm, I answered this, but the comment system ate it whole. Trying again.
This is indeed a bug in 4.3.2 WinEventLog data acquisition code. It's now been identified and we can fix for the next release.
The code flow was altered in order to address performance concerns, which was a success. The performance for rapidly acquiring eventlog data should improve by around a factor of 2, which relieves stress on the wineventlog service (which was the bottleneck). The changes have to do with cacheing handles for data providers of windows eventlog strings.
Of course that's a poor consolation for correct operation. Unfortunately, the set of events we tested with did not have the distribution of data providers, so the problem wasn't identified internally.
For now please use 4.3.1 forwarders (or earlier) to acquire this data type.
Please note that this error message does not have a one to one correlation with this misbehavior. Other scenarios such as loading EVT files without the corresponding availble DLLs that provide the messages, or reading eventlogs for an application which has been subsequently uninstalled could (and do) produce the same message.
I don't see this bug on the "know issues" list, and I don't see it on the list of things fixed in 4.3.3. Am I just not seeing it, or....?
Rolling back to 4.3.1 UF did not work for me...any other suggestions?
In May I was told 4.3.3 was targeted for July.
For searching purposes, this issues is listed as SPL-51312.
Any timeline for the release of 4.3.3?
I had the same issue, and now I'm downgrading the universal forwarder: hope it can solve the bug 🙂
Thank you
@coreindustries - http://www.splunk.com/page/previous_releases
Where can I download 4.3.1?
I meant the message as it should have been gathered; for example as it appears with 4.3.1 or in event log viewer. I'm working on the bug -- and more information will help.
; well the problem turned out to have not much relationship with the message, so nevermind.
You mean, other than the sample message in the post?... if you read through the rest of the post you'll see that I downgraded to 4.3.1 on key forwarders and the symptoms went away. Support is entering a bug report for me, as of late Friday afternoon.
The likely relevant change was from 4.3.1 to 4.3.2; I would recommend using 4.3.1 for now. An example specific message which behaved that way would be useful.
Have you considered downgrading the forwarder? We're not seeing this issue with UF 4.2.5.
Our site has also encountered this. Smells like a bug as it has only occurred with the 4.3.2 UF, not with the previous version of UF (4.2.x)
Hello,
what the SPL number for this?
Kind Regards,
Jens
yeah, support is finally filing a bug report...