Hey everyone! I've successfully set up a link from Splunk Connect for Kubernetes on our OpenShift environment. It outputs to a local Heavy forwarder, which then splits the data stream and sends to our on-prem Splunk instance and a proof of concept Splunk Cloud instance (which we're hopefully going to be moving towards in the future). I have the system setup so that it sends most of its logs to an index called "test_ocp_logs". This covers cases in the format of [ocp:container:ContainerName]. However, I am getting a strange log into our root "test" index, which I have set up as the baseline default in the configuration.  These have the following info: source = namespace:splunkconnect/pod:splunkconnect-splunk-kubernetes-logging-XXXXX sourcetype = fluentd:monitor-agent These look like some kind of report on what the SCK system grabbed and processed, but I can't seem to find any kind of definition anywhere. Here's what one of the events looks like :   { [-] emit_records: 278304 emit_size: 0 output_plugin: false plugin_category: filter plugin_id: object:c760 retry_count: null type: jq_transformer }     So I have a few main questions: What is this log, and is it something we should care about? If we should care about this, what do the fields mean? If we should care about this, how do I direct where it goes so that I keep all my SCK/OpenShift events kept in the same index (at least for now)? For reference, this is the contents of my values.yaml for the helm chart to build SCK:   global: logLevel: info splunk: hec: host: REDACTED port: 8088 token: REDACTED protocol: indexName: test insecureSSL: true clientCert: clientKey: caFile: indexRouting: kubernetes: clusterName: "paas02-t" prometheus_enabled: monitoring_agent_enabled: monitoring_agent_index_name: serviceMonitor: enabled: false metricsPort: 24231 interval: "" scrapeTimeout: "10s" additionalLabels: { } splunk-kubernetes-logging: enabled: true logLevel: fluentd: # Resticting to APP logs only for the proof of concept path: /var/log/containers/*APP*.log exclude_path: - /var/log/containers/kube-svc-redirect*.log - /var/log/containers/tiller*.log - /var/log/containers/*_kube-system_*.log # ignoring internal Openshift Logging generated errors - /var/log/containers/*_openshift-logging_*.log containers: path: /var/log pathDest: /var/lib/docker/containers logFormatType: cri logFormat: "%Y-%m-%dT%H:%M:%S.%N%:z" refreshInterval: k8sMetadata: podLabels: - app - k8s-app - release watch: true cache_ttl: 3600 sourcetypePrefix: "ocp" rbac: create: true openshiftPrivilegedSccBinding: true serviceAccount: create: true name: splunkconnect podSecurityPolicy: create: false apparmor_security: true splunk: hec: host: port: token: protocol: indexName: test_ocp_logs insecureSSL: clientCert: clientKey: caFile: journalLogPath: /run/log/journal charEncodingUtf8: false logs: docker: from: journald: unit: docker.service timestampExtraction: regexp: time="(?<time>\d{4}-\d{2}-\d{2}T[0-2]\d:[0-5]\d:[0-5]\d.\d{9}Z)" format: "%Y-%m-%dT%H:%M:%S.%NZ" sourcetype: kube:docker kubelet: &glog from: journald: unit: kubelet.service timestampExtraction: regexp: \w(?<time>[0-1]\d[0-3]\d [^\s]*) format: "%m%d %H:%M:%S.%N" multiline: firstline: /^\w[0-1]\d[0-3]\d/ sourcetype: kube:kubelet etcd: from: pod: etcd-server container: etcd-container timestampExtraction: regexp: (?<time>\d{4}-\d{2}-\d{2} [0-2]\d:[0-5]\d:[0-5]\d\.\d{6}) format: "%Y-%m-%d %H:%M:%S.%N" etcd-minikube: from: pod: etcd-minikube container: etcd timestampExtraction: regexp: (?<time>\d{4}-\d{2}-\d{2} [0-2]\d:[0-5]\d:[0-5]\d\.\d{6}) format: "%Y-%m-%d %H:%M:%S.%N" etcd-events: from: pod: etcd-server-events container: etcd-container timestampExtraction: regexp: (?<time>\d{4}-[0-1]\d-[0-3]\d [0-2]\d:[0-5]\d:[0-5]\d\.\d{6}) format: "%Y-%m-%d %H:%M:%S.%N" kube-apiserver: <<: *glog from: pod: kube-apiserver sourcetype: kube:kube-apiserver kube-scheduler: <<: *glog from: pod: kube-scheduler sourcetype: kube:kube-scheduler kube-controller-manager: <<: *glog from: pod: kube-controller-manager sourcetype: kube:kube-controller-manager kube-proxy: <<: *glog from: pod: kube-proxy sourcetype: kube:kube-proxy kubedns: <<: *glog from: pod: kube-dns sourcetype: kube:kubedns dnsmasq: <<: *glog from: pod: kube-dns sourcetype: kube:dnsmasq dns-sidecar: <<: *glog from: pod: kube-dns container: sidecar sourcetype: kube:kubedns-sidecar dns-controller: <<: *glog from: pod: dns-controller sourcetype: kube:dns-controller kube-dns-autoscaler: <<: *glog from: pod: kube-dns-autoscaler container: autoscaler sourcetype: kube:kube-dns-autoscaler kube-audit: from: file: path: /var/log/kube-apiserver/audit.log timestampExtraction: format: "%Y-%m-%dT%H:%M:%SZ" sourcetype: kube:apiserver-audit openshift-audit: from: file: path: /var/log/openshift-apiserver/audit.log timestampExtraction: format: "%Y-%m-%dT%H:%M:%SZ" sourcetype: kube:openshift-apiserver-audit oauth-audit: from: file: path: /var/log/oauth-apiserver/audit.log timestampExtraction: format: "%Y-%m-%dT%H:%M:%SZ" sourcetype: kube:oauth-apiserver-audit resources: requests: cpu: 100m memory: 200Mi buffer: "@type": memory total_limit_size: 600m chunk_limit_size: 20m chunk_limit_records: 100000 flush_interval: 5s flush_thread_count: 1 overflow_action: block retry_max_times: 5 retry_type: periodic sendAllMetadata: false nodeSelector: node-role.kubernetes.io/app: '' affinity: {} extraVolumes: [] extraVolumeMounts: [] priorityClassName: kubernetes: securityContext: true splunk-kubernetes-objects: enabled: false splunk-kubernetes-metrics: enabled: false   ... View more

Hey everyone! I have what I would consider a complex problem, and I was hoping to get some guidance on the best way to handle it. We are attempting to log events from an OpenShift (Kubernetes) environment. So far, I've successfully gotten raw logs coming in from Splunk Connect for Kubernetes, into our heavy forwarder via HEC, and then into our indexer. The data that is being ingested has a bunch of metadata about the pod name, container name, etc. from this step. However, the problem is what to do with it from there. In the case of this specific configuration, the individual component logs are getting combined into a single stream, with a few fields of metadata attached to the beginning. After this metadata, the event 100% matches up with what I'd consider a "standard" event, or something Splunk is more used to processing. For example: tomcat-access;console;test;nothing;127.0.0.1 - - [08/Dec/2021:13:25:21 -0600] "GET /idp/status HTTP/1.1" 200 3984 This is first semicolon-delimited, and then space delimited, as follows: "tomcat-access" is the name of the container component that generated the file. "console" indicates the source (console or file name) "test" indicates the environment. "Nothing" indicates the User Token And everything after this semicolon is the real log. In this example, it is a Tomcat-access sourcetype. Compare this to another line in the same log: shib-idp;idp-process.log;test;nothing;2021-12-08 13:11:21,335 - 10.103.10.30 - INFO [Shibboleth-Audit.SSO:283] - 10.103.10.30|2021-12-08T19:10:57.659584Z|2021-12-08T19:11:21.335145Z|sttreic shib-idp is the name of the container component that generated the log idp-process.log is the source file in that component test is the environment nothing is the user token And everything after that last semicolon is the Shibboleth process log. Notably, this part uses pipes as delimiters. The SCK components, as I have them configured now, ship all these sources to "ocp:container:shibboleth" (or something like that). When they are shipped over, metadata is added for the container_name, pod_name, and other CRI-based log data. What I am aiming to do I would like to use the semicolon-delimited parts of the event to tell the heavy forwarder what sourcetypes to work with. Ideally, I would like to cut down on having to make my own sourcetypes and regex, but I can do that if I must. So for the tomcat-access example above, I'd want: All the SCK / Openshift related fields to stick with the event. The event to be chopped up into 5 segments The event type to be recognized by the first 2 fields (there is some duplication in the first field, so the second field would be the most important) The first 4 segments to be appended as field information (like "identifier" or "internal_source") The 5th segment to be exported to another sourcetype for further processing (in this case, "tomcat_localhost_access" from "Splunk_TA_tomcat"). All the other fields would stick with the system as Splunk_TA_tomcat did its field extractions. If this isn't possible, I could make a unique sourcetype transform for each event type - the source program has 8 potential sources. But that would involve quite a bit of duplication.   Even as I type this out, I'm getting the sinking feeling that I'll need to just bite the bullet and make 8 different transforms. But one can hope, right?   Any help would be appreciated. I've gotten through Sysadmin and data admin training, but nothing more advanced than that. I suspect I'll need to use this pattern in the future for other Openshift logs of ours, but I don't know at this stage. ... View more

Hey all! I've inherited a Splunk instance that has been running for about 8 years now. There are instances of Splunk_TA_windows all over it - most are 4.8.3, but a couple are 8.0.0 and 8.1.2. (The overall Splunk instance is running at 7.2 currently). In the process of investigation, I have discovered that our Active Directory controllers had Universal Forwarders installed on them using the GUI installer. In the process, they were set to collect Windows event logs, but no other configuration was made. As a result, a ton of logging is flowing into our "main" index. In fact, the only thing in the "inputs.conf" file is the IP address of the host. Thanks to the help and pointers of many, I've determined that this is definitely "not good" and instead I should have some filters/blacklists in place. I've gotten the controllers in question hooked up to our deployment server, so I want to push some apps to them via that. My question is: Should I deploy the entire Splunk_TA_windows app to the domain controllers? Or should I just push custom apps that contain the filtering/settings I want, and leave Splunk_TA_windows to the Heavy Forwarders, Indexers, and Search Heads we plan on using? Or should I do both? I've consulted a few other resources, such as https://community.splunk.com/t5/All-Apps-and-Add-ons/Is-it-a-best-practice-to-use-the-Splunk-Add-on-for-Microsoft/td-p/427679 (Best practice to use Splunk_TA_windows) https://docs.splunk.com/Documentation/WindowsAddOn/8.1.2/User/AbouttheSplunkAdd-onforWindows (Deploy and use documentation) https://www.splunk.com/en_us/blog/tips-and-tricks/working-with-active-directory-on-splunk-universal-forwarders.html (working with AD on Splunk Universal Forwarders) Digging around, I'm seeing that some Windows logging is being put into the "ActiveDirectory" sourcetype already, but not from any configuration I can find applying to the system, so I assume it is just recognizing them as AD events. My biggest concern is that I want to build a "baseline" that is easy to maintain going forward. I know from my Data Admin training that deployed add-ons are evaluated in reverse-lexicographical  order (IE "Splunk_TA_Windows" has lower priority than "institution_windows_core"), so I should be able to stack things... but again, I just want to make sure I'm following what people recommend. ( May also be using this forum as a "Rubber ducky" situation. 😄 ) ... View more

Hey everyone! I'm in the process of investigating a Splunk instance that I have inherited. I've got a decent handle on things, but I am seeing that the majority of our index is being eaten up by logs from our multiple Active Directory controllers. Digging around, I see that the local inputs.conf file for the universal forwarder on the DCs is empty, and btool confirms they are not pulling in config from other places. There is, however, a deploymentclient.conf file, with a single targetUri in it. What's interesting, though, is that the listed TargetUri is not a server name that is present in our environment. It's close, but not exact. Further, I see no signs that this particular domain controller has ever checked in with our deployment server. I know for a fact that we manually installed the Universal Forwarder on the domain controller. I also know that the correct Deployment Server and Indexer were provided at install time. So what might have caused the targetUri to change? I'm thinking it may be something in the deployment server itself, but I don't know where to look for that setting or how the deployment server might have updated it. I'm still getting my head wrapped around just what the deployment server itself is doing, in fact. But I am worried that with a full throttle, out of the box universal forwarder, we are likely collecting way more information than we actually want.   ... View more