Why is there Missing data from logs?

[SBadams]

Hello, I have an issue with web and syslog indexes not being logged properly. I believe that I will need to change the settings of the Spunk Forwarders and I need help with modifying the UF configs so that I can correct the data that needs to be logged. We have a deployment server set up and I think this is probably the route to go. What does the process look like for doing this?

[PickleRick]

Ok. Your question is so general that it can't be answered reasonably. It's as if you asked "my car is not working properly, what should I do?".

There are several key pieces of information missing here.

1. How are you ingesting your logs? (UFs installed on the source hosts in case of web logs or syslog-pushed logs? In case of syslogs - are they being pushed directly to the UFs or do you have any intermediate receiver? And so on)

2. What kinds of logs do you have? From what solutions, in what format? What are your settings for those sourcetypes?

3. How do you know they are not indexed properly?

And deployment server is just a tool to manage configs on UFs but the configs themselves must be done properly in the first place (and a lot of those settings are not configurable on UFs but on indexers).

So there's much more to it than meets the eye.

[SBadams]

We have hundreds of servers in the environment running IIS. All servers have different logging levels set for IIS which causes different results in Splunk when searching our web index. We are looking for a solution to level set the logging level for IIS so we can build proper detections. Note - we also have this issue with apache logs. But we can concentrate on IIS for this. 

[PickleRick]

OK. So firstly, you need to make sure you get your logging configured consistently across your whole environment (unless you really want it to be set differently on some servers).

Then you need to ingest the files properly. There is an add-on for IIS logs - https://splunkbase.splunk.com/app/3185 Install it, configure inputs as described in the docs (also verify that your IIS logging is configured properly according to the docs), then check if all the files are getting indexed properly.

[SBadams]

The goal of this project is to create consistent logging across all servers in the environment. What tools exist on Splunk to achieve this? We are already ingesting existing logs properly.

[PickleRick]

It's not up to Splunk to configure your logging. Typically if you download an add-on from splunkbase it has a docs page which describes how to configure source to produce relevant logs.

[SBadams]

Would changing each server's iis logging settings using a GPO be the recommended option for solving this issue?

[PickleRick]

The Add-on docs provide docs how the single IIS should be configured so that it logs the proper data.

How to deploy that configuration in your environment is something you have to consult with your admins and check with your local policies. We can't tell you if in your case GPO will be the appropriate solution. It might be (I'm not sure if you can configure those settings with GPO) but there can be other ways to do it (for example if you used any third party automation solution you could use that instead of deploying settings via GPO.

The reqiurements for the Add-on regarding IIS configuration are described here - https://docs.splunk.com/Documentation/AddOns/released/MSIIS/Hardwareandsoftwarerequirements#Microsof... but how to apply them properly is up to you and your infrastructure team.

[gcusello]

Hi @SBadams,

your question is really very vague!

you should share some additional info:

• what do you mean with missing data: are arrived and now stopped to arrive or never arrived?
• logs are incomplete?
• are qyou speaking of syslogs or logs from a Forwarder?
• which kind of logs?
• logs are correctly parsed (with special attention to the timestamp)?

Ciao.

Giuseppe