Deployment Architecture

How do I remove "missing" forwarders from Splunk Deployment Monitor 4.3.1?

Cagey
Engager

Every time I go into deployment monitor it tells me I have 65 missing forwarders. In all cases these forwarders are listed as an IP address. In some cases the IP address corresponds to a "active" forwarder which is reported by the servers name. In other cases the forwarder is actually no longer in service and needs to be removed from the list of forwarders. I have read other comments regarding this and they mention a forwarder as going "quiet" or deployment monitor have a "remove missing forwarders" button. In my case neither of these is present.

As I see it this is actually two problems:
1. making splunk correlate the IP address of the "missing" forwarder to the DNS name for the associated "active" forwarder.
2. remove actual "missing" forwarders from the list of forwarders.

gpullis
Communicator

What worked for me was using the Rebuild forwarder assets... button in Monitoring Console > Settings > Forwarder Monitoring Setup.

See: https://docs.splunk.com/Documentation/Splunk/7.1.1/DMC/Configureforwardermonitoring

richaGindodia
Path Finder

Not sure of this. But you could actually add a ping script to your forwarders which would ping your server at regular intervals.

0 Karma

Cagey
Engager

Thank you for your response Rich but this would not solve my problem. All the forwarders report to an indexing server which keeps track, via a database or something, of all the forwarders and when they last reported into the indexer. Now my problem (which actually has two parts) is that I cannot acknowledge the missing forwarders so that they stop showing up in the list of forwarders.

To further explain the first part of my problem, suppose I have a forwarder with a DNS name of "forwarder1" and an IP address of "1.2.3.4". My indexer is reporting that "forwarder1" is active but IP address "1.2.3.4" is missing. This is not possible since they are the same device. Obviously this is a problem with the actual code or database which is used to report the forwarders.

The second part of the problem is that I DO actually have some forwarders which are no longer in service and they are rightly being reported as missing. However, I know this and would like to acknowledge this to the application and stop having them reported as missing. The problem is, there is no way to do this so every time I go into the application I am once again informed about the missing forwarders. However, if there are any new ones listed it is hard to pick them out from the large list of 65.

So, still two problems:

  1. Code (or database) needs fixing to correlate the IP with the DNS name.
  2. Acknowledgement function required to remove actual "missing" forwarders from the database.
0 Karma

gpullis
Communicator

Yeah. Same. We're logging VDI machines that are pretty ephemeral, so my production indexer is complaining about 4758 "missing" forwarders. Some of those are legit, but it's pretty painful to try to figure out which ones.

0 Karma
Get Updates on the Splunk Community!

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...