Deployment Architecture

Why am I not receiving any Data from the Linux Server?

JarrettM
Path Finder

No data being received from Linux client.

Running Splunk Enterprise 7.0.3 on Windows Server 2012 R2. Receiving data across 7 indexes from 36 Windows Universal Forwarder clients and two syslog servers. Installed Splunk Universal Forwarder 7.0.3 on Redhat 7 server. Client phones home and makes good SSL connection:

5/10/18 8:53:54.613 AM 10..x.x.x - - [10/May/2018:08:53:54.613 -0400] "POST /services/broker/phonehome/connection_10..x.x.x_8089_ SPLUNK_CLIENT _ SPLUNK_CLIENT _784C7242-6306-403F-B877-11B04B59FFC7 HTTP/1.1" 200 832 - - - 0ms

05-10-2018 08:53:49.446 -0400 INFO  Metrics - group=tcpin_connections, 10.x.x.x:48460:9997, connectionType=cookedSSL, sourcePort=48460, sourceHost=10.x.x.x, sourceIp=10.x.x.x, destPort=9997, kb=72.01, _tcp_Bps=2378.52, _tcp_KBps=2.32, _tcp_avg_thruput=2.77, _tcp_Kprocessed=3107.70, _tcp_eps=3.55, _process_time_ms=0, evt_misc_kBps=0.06, evt_raw_kBps=1.03, evt_fields_kBps=1.13, evt_fn_kBps=0.84, evt_fv_kBps=0.29, evt_fn_str_kBps=0.06, evt_fn_meta_dyn_kBps=0.00, evt_fn_meta_predef_kBps=0.03, evt_fn_meta_str_kBps=0.68, evt_fv_num_kBps=0.03, evt_fv_str_kBps=0.23, evt_fv_predef_kBps=0.00, evt_fv_offlen_kBps=0.00, evt_fv_fp_kBps=0.00, build=fa31da744b51, version=7.0.3, os=Linux, arch=x86_64, hostname=SPLUNK_CLIENT, guid=784C7242-6306-403F-B877-11B04B59FFC7, fwdType=full, ssl=true, lastIndexer=y.y.y.y:9997, ack=false destPort =9997 host =SPLUNK_SERVER     support-server hostname =SPLUNK_CLIENT log_level =INFO source =E:\Splunk\var\log\splunk\metrics.log sourceHost =10.x.x.x sourcePort =   48460 tag =support-server

Deployment App shows in Client's Apps directory:
#var-logs
[monitor:///var/log/*.log]
disabled=false
index=linux
sourcetype=var-logs

Index and sourcetype are correct. 

These are the only lines the the Clients splunkd that indicate any sort of problem:

05-10-2018 09:04:03.532 -0400 WARN  LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 12323 - data_source="/home/USER_NAME/splunk/var/log/splunk/.splunkd.log.swp", data_host=" SPLUNK_CLIENT ", data_sourcetype="swp-too_small"

05-10-2018 09:04:03.533 -0400 WARN  TailReader - Insufficient permissions to read file='/home/USER_NAME/splunk/var/log/splunk/.splunkd.log.swx' (hint: No such file or directory ,                            UID: 474401269, GID: 474400513).

05-10-2018 09:04:06.536 -0400 WARN  FileClassifierManager - The file '/home/USER_NAME/splunk/var/log/splunk/.splunkd.log.swp' is invalid. Reason: binary

Any ideas?

Thanks

0 Karma

JarrettM
Path Finder

Succeeded in getting data by using the the following command on the client:

./splunk add monitor /var/log

however it still will not work by using a deployed app from the deployment server.

0 Karma

JarrettM
Path Finder

Never Mind!

Here was the problem:

Default instead of default

Doh!

0 Karma
Get Updates on the Splunk Community!

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...

Adoption of Infrastructure Monitoring at Splunk

  Splunk's Growth Engineering team showcases one of their first Splunk product adoption-Splunk Infrastructure ...

Modern way of developing distributed application using OTel

Recently, I had the opportunity to work on a complex microservice using Spring boot and Quarkus to develop a ...