|
I've verified that the indexer (receiver) is the same or later version of Splunk as the forwarder. What log or configuration files can I look at to troubleshoot this problem? |
|
You can do the command "splunk list forward-server" to see if the forward-server is active on the forwarder. If it's inactive, it usually means you have not enabled the receiver to receive forwarded data. Go to the receiver and then browse to the "Manager > Forwarding and receiving > under receive data select Receive data from forwarder. The port specified here should be the same port the forwarders are configured to send data. So if you're receiver is set to receive forwarded data to port 8889, then you should have this listed when you do the "splunk list forward-server" command: splunkserver:8889 |
|
On the forwarder check: sysinfo.txt (verify general system info) outputs.conf (verify settings) metrics.log (search for tcpout_connections, destport=xxxx) splunkd.log (search for Error and WARN "failed to make connections") Configuration/log files to check on indexer: inputs.conf (search for splunktcp:\xxxx) metrics.log (search for data coming from forwarder) splunkd.log (search for Error, tcpin_connections (look for forwarder hostname/IP)) Also, you can try running a search on the indexer to see if data came in from the forwarder. |
|
I would begin by confirming basic connectivity. I will assume we are on linux and using the default forwarding port of 9997 (no ssl): Look for your receiving port to be open on the indexer:
**This should return an active TCP listener on 9997 Look for your receiving port to be connected to from the forwarder:
**This should return an active TCP connection TO port 9997 on your indexer If neither of the above are operational, then fowarding will not work. You should review if you have properly configured receiving and forwarding. Note that you may need to restart to enable forwarding. Next, you should run a search to find the forwarder connection on the indexer:
You should see an event very similar to below with your forwarder IP address:
If you see positive values for tcp_Kprocessed, that means your forwarder is connected and has transferred data. If you do not see the above event in your metrics.log file (_internal index), you should then refer to the splunkd.log on your indexer and forwarder. Splunk will log an entry in the splunkd.log file when a forwarder has connected. |
|
Of config files, Before all that, though, be sure to check network connectivity with |
