Does the indexer acknowledgement queue/list persist across Splunk restarts?

[Glenn]

I'd like to user indexer acknowledgement in my HA setup when forwarding from a primary indexer which receives events from forwarders, to a secondary indexer (despite the horrible proliferation of duplicate events it can cause, but that's another issue).

I'd like to know whether the queue or list of unacknowledged events maintained on the primary indexer will persist if the primary indexer is restarted (while the secondary is still unavailable).

If it doesn't, we could easily lose the queue and have gaps in our secondary index, breaking HA.

[sdaniels]

You could use persistent queues on the forwarder so that the data is there after restart, otherwise you'll lose that data in memory.

http://docs.splunk.com/Documentation/Splunk/4.3.1/Data/Usepersistentqueues

[Glenn]

Thanks, this is helpful.

[yannK · Accepted solution]

If I am correct, the ACK=true option (on the forwarder) will cause the forwarder to wait for an acknowledgement from the indexer that the event has been written to disk.
So If the indexer goes down, the forwarder will retry.
As you can see, at the end it will not cause gaps, only accidental duplicates.

Edit :

In the case of a forwarder

About tailing :

• File Tailing queues, splunk keeps track of the position it was reading a file, and if it is restarted, it will restart from that point.
• Scripted inputs/network inputs (example syslog on port 514), a splunk instance will store thoses queues in memory, and cannot recover them. An easy workaround is to use a syslog-ng/rsyslog server to write the log to file, and have splunk monitor the files (equivalent to disk buffer).

About queues :

• Memory queues, used by default. A forwarder crashing/shutdown while queueing data will lose the data waiting in the memory queue.
• Persistent queues, you can configure your forwarders to keep part of the data queued on disk instead of memory, they will be resumed once restarted. see http://docs.splunk.com/Documentation/Splunk/4.3.1/Data/Usepersistentqueues

About HA acknowledgement :

• you can also add another level of security, by using acknowledgments, this is more costly in traffic and speed. http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Protectagainstlossofin-flightdata

[youngsuh]

Here is the updated link for community convenience:

Use persistent queues to help prevent data loss - Splunk Documentation 

https://docs.splunk.com/Documentation/Splunk/8.2.2/Forwarding/Protectagainstlossofin-flightdata

[cphair]

@yannK, do you have specific measures of the cost of enabling HA acknowledgement beyond what's in the document you linked? I understand the memory usage on the forwarder side would increase, but I'd like to know the effect on the indexer side as well.

[Glenn]

Looks good thanks. I think persistant queues is what I was looking for.

[yannK]

edited above.

[Glenn]

I should have been more clear. I mean, what happens to the queue on the forwarder, if it goes down while the indexer is already down. ie. does the forwarder's queue still have the same data after it is restarted?