OK, I have found the issue. A custom-made app (implemented long before I took control of the Splunk Instance) is setting up the environment to enable the forwarding of local instance data and metrics to a SOC layer. That worked just fine before the latest upgrades (from 6.5 to 7.2 to 8.1). When I disable the forwarding, the audit data is injected back in the local _audit index, and it breaks again as soon as I enable the forwarding. Now that forwarding is necessary, it can't be disabled, but now I'm trying to figure out how to save a copy of the audit data without impacting the rest of the saved searches and forwarded data. I know I could select the option to make a local copy of the forwarded data. Still, I don't know what the consequences could be since it has been working for a while without the option "store a local copy of forwarded events" enabled.  Any suggestions? ... View more

Verify your role has access to the _audit index. > Yes, the role has access. Make sure the searches are specifying the index name explicitly rather than depending on _audit to be on the list of default indexes (which can change). >Yes, search specify the _audit index Verify the audit.log file is being monitored.  Use splunk list monitor to check > After running 'splunk list monitor' it shows the ...splunk/audit.log is being monitored. The last event indexed in _audit was from 16 days ago. I did try deleting the _audit index data and generating a new one, and it didn't work. ... View more