I've verified that the indexer (receiver) is the same or later version of Splunk as the forwarder. What log or configuration files can I look at to troubleshoot this problem?
On the forwarder check: sysinfo.txt (verify general system info) outputs.conf (verify settings) metrics.log (search for tcpout_connections, destport=xxxx) splunkd.log (search for Error and WARN "failed to make connections")
Configuration/log files to check on indexer: inputs.conf (search for splunktcp:\xxxx) metrics.log (search for data coming from forwarder) splunkd.log (search for Error, tcpin_connections (look for forwarder hostname/IP))
Also, you can try running a search on the indexer to see if data came in from the forwarder.
You can do the command "splunk list forward-server" to see if the forward-server is active on the forwarder. If it's inactive, it usually means you have not enabled the receiver to receive forwarded data.
Go to the receiver and then browse to the "Manager > Forwarding and receiving > under receive data select Receive data from forwarder. The port specified here should be the same port the forwarders are configured to send data. So if you're receiver is set to receive forwarded data to port 8889, then you should have this listed when you do the "splunk list forward-server" command:
splunkserver:8889
I would begin by confirming basic connectivity. I will assume we are on linux and using the default forwarding port of 9997 (no ssl):
Look for your receiving port to be open on the indexer:
> netstat -an | grep 9997
**This should return an active TCP listener on 9997
Look for your receiving port to be connected to from the forwarder:
> netstat -an | grep 9997
**This should return an active TCP connection TO port 9997 on your indexer
If neither of the above are operational, then fowarding will not work. You should review if you have properly configured receiving and forwarding. Note that you may need to restart to enable forwarding.
Next, you should run a search to find the forwarder connection on the indexer:
index=_internal source=*metrics.log tcpin_connections
You should see an event very similar to below with your forwarder IP address:
04-23-2010 23:00:36.887 INFO Metrics - group=tcpin_connections, 192.1.168.10:36924:9997, connectionType=cooked, sourcePort=36924, sourceHost=forwarder.splunk.com, sourceIp=10.8.240.201, destPort=9997, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=2.08, _tcp_Kprocessed=10078.00, _tcp_eps=0.00
If you see positive values for tcp_Kprocessed, that means your forwarder is connected and has transferred data. If you do not see the above event in your metrics.log file (_internal index), you should then refer to the splunkd.log on your indexer and forwarder. Splunk will log an entry in the splunkd.log file when a forwarder has connected.
Here's a starting point: http://www.splunk.com/wiki/Community:TroubleshootingForwarding
You can do the command "splunk list forward-server" to see if the forward-server is active on the forwarder. If it's inactive, it usually means you have not enabled the receiver to receive forwarded data.
Go to the receiver and then browse to the "Manager > Forwarding and receiving > under receive data select Receive data from forwarder. The port specified here should be the same port the forwarders are configured to send data. So if you're receiver is set to receive forwarded data to port 8889, then you should have this listed when you do the "splunk list forward-server" command:
splunkserver:8889
If you are sending data to a specific index. It has to be created on the indexer first. You can see if an index contains data (and when the first and last events arrived) in the web gui.
splunkd.log
would be a good start; please check it on the forwarder first, then indexer; look for ERROR
lines.
Of config files, outputs.conf
on the forwarder is of interest.
Before all that, though, be sure to check network connectivity with ping(8)
or ping
.
On the forwarder check: sysinfo.txt (verify general system info) outputs.conf (verify settings) metrics.log (search for tcpout_connections, destport=xxxx) splunkd.log (search for Error and WARN "failed to make connections")
Configuration/log files to check on indexer: inputs.conf (search for splunktcp:\xxxx) metrics.log (search for data coming from forwarder) splunkd.log (search for Error, tcpin_connections (look for forwarder hostname/IP))
Also, you can try running a search on the indexer to see if data came in from the forwarder.