Does splunk list monitor accurately reflect files being monitored

Question

When I run command

bin/splunk add monitor '/var/mqm/qmgrs/*/errors/AMQERR01.LOG' -follow-only True

to monitor specific log file under several directories e.g. $ ls /var/mqm/qmgrs/*/errors/AMQERR01.LOG /var/mqm/qmgrs/QM1/errors/AMQERR01.LOG
/var/mqm/qmgrs/QM2/errors/AMQERR01.LOG
/var/mqm/qmgrs/QM3/errors/AMQERR01.LOG

bin/splunk list monitor

returns the following

    /var/mqm/qmgrs/*/errors/AMQERR01.LOG
            /var/mqm/qmgrs/@SYSTEM
            /var/mqm/qmgrs/QM1
            /var/mqm/qmgrs/QM1/@app
            /var/mqm/qmgrs/QM1/@app/esem
            /var/mqm/qmgrs/QM1/@app/esem/uateai07
            /var/mqm/qmgrs/QM1/@app/isem
            /var/mqm/qmgrs/QM1/@app/isem/uateai07
            /var/mqm/qmgrs/QM1/@app/msem
            /var/mqm/qmgrs/QM1/@app/msem/uateai07
            /var/mqm/qmgrs/QM1/@app/shmem
            .. another 4000 subdirectories

However despite what list monitor says - I do see the error files I am looking for in Splunk

Is this a bug?

Answer 1

Yes, the list monitor command does have some enhancements coming down the pipeline, right now it doesn't give complete information on every file & directory being monitored, and it lists files that are not being monitored due to whitelist/blacklist settings. Also, it's possible that it could take a long time to complete if you are monitoring hundreds of thousands of files & directories.

There is a way you can get this information, but it's in the early stages of development and not as pretty as it could be. If you hit this endpoint with your browser - https://<servername>:8089/services/admin/inputstatus/TailingProcessor%3AFileStatus - you will get a lot of XML back detailing the current status of your monitored files. Again, this is not fully developed yet so it just gives you a lot of bare-bones info. Remember to insert your servername and splunkd management port into the URL.

Answer 2

Thanks for that. I did find that using the wild card wasn't the smartest thing as the splunkd process used a bunch of CPU - I assume scanning the file system for file matches repeatedly. It would be nice to separate the polling for files to monitor (say only poll for new file every 5,10,60 etc. minutes) versus polling the files themselves.

splunk add monitor '/var/mqm/qmgrs/*/errors/AMQERR01.LOG' -follow-only True

Anyway I switched and ran this instead

ls /var/mqm/qmgrs/*/errors/AMQERR01.LOG | xargs -i -t bin/splunk add monitor {} -follow-only True

which adds each found file separately. Cpu dropped dramatically.

I will just script to it to run that daily to pick up any new files.

Does splunk list monitor accurately reflect files being monitored

Follow this question

Splunk Resources

Related questions