Summary
Not all logs are being forwarded for indexing by my splunkforwarders.
Situation
I have 4 instances that run 3 processes I am interesting in.
Each process outputs logs that I am forwarding to Splunk via a splunkforwarder on the instance.
These logs are rotated by logrotate.d.
On some instances all logs are being forwarded, on some instances only some logs are being forwarded.
Problems
I believe the relevant error from the logs is this one (others below):
splunkd.log:09-22-2017 01:30:04.522 +0000 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_search-1.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
Possible Solutions?
- Increase the initCrcLen?
- WHAT ELSE SHOULD I TRY?
- DO THE OTHER ERRORS IN THE LOG MATTER (ERROR JsonLineBreaker or ERROR TcpOutputProc)
Reference
Splunk Forwarder Config
Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/etc/system/local# cat inputs.conf
[default]
host = bowman-<myserver>1
[monitor:///home/ubuntu/logs/json-bowman-<myserver>1*.log]
disabled = 0
sourcetype = boeinglogjson
index = prod-boeing
Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/etc/system/local# cat outputs.conf
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = splunk.myotherserver.com:9997
[tcpout-server://splunk.myotherserver.com:9997]
Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/etc/system/local# cat props.conf
TRUNCATE = 2000000
[boeinglogjson]
INDEXED_EXTRACTIONS = json
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = info.created
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3Q
category = Custom
disabled = false
Other errors from Splunk Logs
Env=prodb|Role=bowman|root@bowman-<myserver>1:/opt/splunkforwarder/var/log/splunk# grep ERROR *.log
splunkd.log:09-21-2017 23:44:53.585 +0000 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.
splunkd.log:09-22-2017 01:30:04.522 +0000 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_booking-1.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
splunkd.log:09-22-2017 01:30:04.522 +0000 ERROR TailReader - File will not be read, is too small to match seekptr checksum (file=/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_search-1.log). Last time we saw this initcrc, filename was different. You may wish to use larger initCrcLen for this sourcetype, or a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.
splunkd.log:09-26-2017 08:51:46.621 +0000 ERROR JsonLineBreaker - JSON StreamId:11681658046189288813 had parsing error:String value too long - data_source="/home/ubuntu/logs/json-bowman-<myserver>1-bowman-worker_default-1.log", data_host="bowman-<myserver>1", data_sourcetype="boeinglogjson"
Example logrotate.d conf..
/<log file path and name>/
{
size 250M
missingok
rotate 3
compress
delaycompress
notifempty
create 664 ubuntu ubuntu
su ubuntu ubuntu
sharedscripts
postrotate
service bowman-worker_booking-1 restart
endscript
}
Hi @markconlin,
If you have same header (First 256 bytes) in all files which splunkforwarder is trying to read then please change inputs.conf on your forwarder something like this.
[monitor:///home/ubuntu/logs/json-bowman-<myserver>1*.log]
disabled = 0
crcSalt = <SOURCE>
sourcetype = boeinglogjson
index = prod-boeing
Additionally props.conf which you have defined on splunk universal forwarder will not work because splunk universal forwarder will not do any parsing those parsing will be done by either heavy forwarder (if you are using it) or indexer. So apply that props.conf configuration on heavy forwarder or indexer whichever comes first in your environment.
I hope this helps.
Thanks,
Harshil
@harsmarvania57 I have concerns with crcSalt and log rotate, because of this comment in the Splunk documentation:
Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed
after it has rolled.
In this case you need to increase initCrcLength
so that splunk forwarder will indentify that these are different file. It looks like in current scenario first 256 bytes of all your log files are same.